Commit graph

133 commits

Author SHA1 Message Date
2d324b77eb
Rename DistanceMatrix into Path
- a `Path` is a better description for an instance of the model
- the `Location`s en route are renamed into `.waypoints`
- generic `assoc` is renamed into `path` in the test suite
2021-09-12 17:33:48 +02:00
2d08afa309
Upgrade sqlalchemy
Adapt code to prevent new warnings and errors (as of SQLAlchemy 1.4):
- Overlapping foreign key columns could be set in a conflicting way
  => This is prevented by the application logic
  => Ignore the warning by setting a `overlaps` flag
- Transaction already rolled back
  => This only happens when tests cause an `IntegrityError` on purpose
  => Filter away the corresponding warning in the fixture
- Query returns `Row` objects and not scalars
  => Add genexpr to pull out `primary_id`
2021-09-12 16:52:51 +02:00
3bef9ca38d
Remove flake8-expression-complexity ...
... from the dev dependencies.

Longer queries in SQLAlchemy get flagged even though they are not
complicated. Other expressions are generally not that complicated.
2021-09-12 16:52:44 +02:00
1c19da2f70
Solve all issues detected by PyCharm
- as of September 2021, PyCharm is used to write some of the code
- PyCharm's built-in code styler, linter, and type checker issued
  some warnings that are resolved in this commit
  + spelling mistakes
  + all instance attributes must be specified explicitly
    in a class's __init__() method
    => use `functools.cached_property` for caching
  + make `tuple`s explicit with `(...)`
  + one test failed randomly although everything is ok
    => adjust the fixture's return value (stub for Google Directions API)
  + reformulate SQL so that PyCharm can understand the symbols
2021-09-12 16:51:12 +02:00
1268aba017
Pin the dependencies ...
... after upgrading:
- alembic
- geopy
- googlemaps
- psycopg2
- rpy2
- dev dependencies
  + coverage       + darglint      + flake8      + flake8-annotations
  + flake8-black   + flake8-comprehensions       + flake8-docstrings
  + flake8-pytest-style            + pre-commit  + pytest
  + pytest-cov     + pytest-mock   + sphinx      + sphinx-autodoc-typehints
- research dependencies
  + jupyterlab     + matplotlib    + numpy       + pandas
- transient dependencies
  + argcomplete    + argon2-cffi   + attrs       + babel
  + bleach         + certifi       + cffi        + cfgv
  + chardset-normalizer            + colorlog    + debugpy
  + decorator      + defusedxml    + distlib     + geographiclib
  + gitdb          + gitpython     + greenlet    + identify
  + idna           + importlib-resources         + ipykernel
  + ipython        + jinja2        + json5       + jupyter-client
  + kiwisolver     + mako          + markupsafe
  + nbclient       + nbconvert     + nbformat    + nodeenv
  + notebook       + parso         + pathspec    + pbr
  + pillow         + platformdirs  + pluggy      + prometheus-client
  + prompt-toolkit + pycodestyle   + pydocstyle  + pyflakes
  + pygments       + pyristent     + python-dateutil
  + pywin32        + pywinpty      + pyzmq       + regex
  + requests       + scipy         + send2trash  + six
  + smmap          + sphinxcontrib-htmlhelp
  + sphinxcontrib-serializinghtml  + stevedore   + terminado
  + textfixtures   + testpath      + traitlets   + typed-ast
  + typing-extensions              + tzdata      + tzlocal
  + urllib3        + virtualenv    + zipp
2021-09-12 16:51:12 +02:00
6636e56ec8
Upgrade flake8-pytest-style
The newly introduced "P023" error code must be disabled explicitly.
2021-09-12 16:51:11 +02:00
fa3b761054
Remove pytest-randomly from the dev dependencies
- problem: because of the randomization of test cases, every once in a
  while, the schema holding the test db already exists and cannot be
  created a second time (background: we run all tests against a db
  created with `metadate.create_all()` by SQLAlchemy at once and
  a db created by running all incremental Alchemy migration scripts)
- quick fix: possible, we could find a way to run all tests against
  one of the two test db's in random order and then against the other
  => in the interest of time, we simply do not randomize the test cases
2021-09-12 16:51:11 +02:00
322ce57062
Make mypy a bit stricter 2021-09-12 16:51:11 +02:00
2ba4914af7
Ignore PyCharm's .idea/ folder 2021-09-12 16:51:10 +02:00
db715edd6d
Add constructor for the DistanceMatrix class
- `DistanceMatrix.from_addresses()` takes a variable number of
  `Address` objects and creates distance matrix entries for them
- as a base measure, the air distance between two `Address`
  objects is calculated
- in addition, an integration with the Google Maps Directions API is
  implemented that provides a more realistic measure of the distance
  and duration a rider on a bicycle would need to travel between two
  `Address` objects
- add a `Location.lat_lng` convenience property that provides the
  `.latitude` and `.longitude` of an `Address` as a 2-`tuple`
2021-09-12 16:51:10 +02:00
5e9307523c
Add ordered-set to the dependencies 2021-09-12 16:51:10 +02:00
cc75307e5a
Add DistanceMatrix class
- the class stores the data of a distance matrix between all addresses
  + air distances
  + bicycle distances
- in addition, the "path" returned by the Google Directions API are
  also stored as a JSON serialized sequence of latitude-longitude pairs
- we assume a symmetric graph
2021-09-12 16:51:10 +02:00
28368cc30a
Add geopy to the dependencies 2021-09-12 16:51:09 +02:00
3dd848605c
Add googlemaps to the dependencies 2021-09-12 16:51:09 +02:00
6c03261b07
Merge branch 'main' into develop 2021-03-01 14:45:24 +01:00
f5ced933d6
Merge branch 'tactical-forecasting' into main
Some checks failed
CI / fast (without R) (push) Has been cancelled
CI / slow (with R) (push) Has been cancelled
2021-03-01 14:13:21 +01:00
9d6de9d98c
Create tactical demand forecasts
- the first notebook runs the tactical-forecasts command
- the second notebook describes the tactical demand forecasting process
  + demand aggregation on a per-pixel level
  + time series generation: horizontal, vertical, and real-time time series
  + STL decomposition into seasonal, trend, and residual components
  + choosing the most promising forecasting model
  + predicting demand with various models
- fix where to re-start the forecasting process after it was interrupted
- enable the heuristic for choosing the most promising model
  to also work for 7 training weeks
2021-02-09 17:06:37 +01:00
21d012050c
Add visualization scripts for customers/restaurants
- the two notebook files are helpful in visualizing all relevant
  pickup (red) or delivery (blue) locations from the point of view
  of either e restaurant or a customer
2021-02-04 16:19:12 +01:00
40471b883c
Add infos on tactical demand forecasting 2021-02-04 16:12:25 +01:00
d494a7908b
Merge branch 'gridification' into main
Some checks failed
CI / fast (without R) (push) Has been cancelled
CI / slow (with R) (push) Has been cancelled
2021-02-04 15:39:46 +01:00
a89b9497f2
Visualize the pixel grids
- add notebook that runs the plotting code
- add three visualizations per city:
  + all addresses, colored by zip code
  + all restaurants, incl. the number of received orders
  + all restaurants on a grid with pixel side length of 1000m
2021-02-04 15:37:16 +01:00
1e263a4b98
Run the gridification script 2021-02-04 15:29:42 +01:00
28a7c7451c
Rename existing notebooks using order numbers 2021-02-04 15:10:14 +01:00
915aa4d3b4
Merge branch 'release-0.3.0' into develop 2021-02-04 13:23:39 +01:00
241e7ed81f
Merge branch 'release-0.3.0' into main
Some checks failed
CI / fast (without R) (push) Has been cancelled
CI / slow (with R) (push) Has been cancelled
2021-02-04 13:18:32 +01:00
d4ca85b55a
Finalize release 0.3.0 2021-02-04 13:13:26 +01:00
0da86e5f07
Pin the dependencies ...
... after upgrading:
- alembic
- matplotlib
- pandas
- rpy2
- sqlalchemy
- statsmodels
- dev dependencies
  + coverage     + factory-boy    + faker                 + nox
  + packaging    + pre-commit     + flake8-annotations    + pytest
  + pytest-cov   + sphinx
- research dependencies
  + numpy        + pyty
- transient dependencies
  + astpretty    + atomicwrites   + bleach                + chardet
  + colorlog     + darglint       + flake8-comprehensions + gitpython
  + identify     + ipykernel      + ipython               + jedi
  + jinja2       + jupyter-client + jupyter-core          + mako
  + nbformat     + nest-asyncio   + notebook              + parso
  + pluggy       + prompt-toolkit + ptyprocess            + pygments
  + pyyaml       + pyzmq          + requests              + smmap
  + terminado    + textfixtures   + snowballstemmer       + typed-ast
  + urllib3      + virtualenv

- fix SQL statements written in raw text
2021-02-04 13:12:47 +01:00
50b35a8284
Add CLI script to run tactical forecasting heuristic 2021-02-04 12:05:43 +01:00
23391c2fa4
Adjust OrderHistory.choose_tactical_model() heuristic
- use the `HorizontalSMAModel` for low demand
- use the `TrivialModel` for no demand
2021-02-02 15:20:02 +01:00
3f5b4a50bb
Rename Forecast.training_horizon into .train_horizon
- we use that shorter name in `urban_meal_delivery.forecasts.*`
  and want to be consistent in the ORM layer as well
2021-02-02 13:09:09 +01:00
6fd16f2a6c
Add TrivialModel
- the trivial model simply predicts `0` demand for all time steps
2021-02-02 12:45:26 +01:00
015d304306
Add HorizontalSMAModel
- the model applies a simple moving average on horizontal time series
- refactor `db.Forecast.from_dataframe()` to correctly convert
  `float('NaN')` values into `None`; otherwise, SQLAlchemy complains
2021-02-02 12:40:53 +01:00
af82951485
Add OrderHistory.choose_tactical_model()
- the method implements a heuristic from the first research paper
  that chooses the most promising forecasting `*Model` based on
  the average daily demand in a `Pixel` for a given `train_horizon`
- adjust the test scenario => `LONG_TRAIN_HORIZON` becomes `8`
  as that is part of the rule implemented in the heuristic
2021-02-02 11:29:27 +01:00
8926e9ff28
Fix nox session for slow CI tests
- when running tests marked with "r" we still must not run tests
  marked with "db" on the CI server
2021-02-01 22:00:47 +01:00
cb7611d587
Add OrderHistory.avg_daily_demand()
- the method calculates the number of daily `Order`s in a `Pixel`
  withing the `train_horizon` preceding the `predict_day`
2021-02-01 21:50:42 +01:00
67cd58cf16
Add urban_meal_delivery.forecasts.models sub-package
- `*Model`s use the `methods.*.predict()` functions to predict demand
  given an order time series generated by `timify.OrderHistory`
- `models.base.ForecastingModelABC` unifies how all `*Model`s work
  and implements a caching strategy
- implement three `*Model`s for tactical forecasting, based on the
  hets, varima, and rtarima models described in the first research paper
- add overall documentation for `urban_meal_delivery.forecasts` package
- move the fixtures in `tests.forecasts.timify.conftest` to
  `tests.forecasts.conftest` and adjust the horizon of the test horizon
  from two to three weeks
2021-02-01 20:39:52 +01:00
796fdc919c
Add Forecast.from_dataframe() constructor
- this alternative constructor takes the `pd.DataFrame`s from the
  `*Model.predict()` methods and converts them into ORM models
2021-02-01 15:50:30 +01:00
b8952213d8
Add extrapolate_season.predict() function
- the function implements a forecasting "method" similar to the
  seasonal naive method
  => instead of simply taking the last observation given a seasonal lag,
     it linearly extrapolates all observations of the same seasonal lag
     from the past into the future; conceptually, it is like the
     seasonal naive method with built-in smoothing
- the function is tested just like the `arima.predict()` and
  `ets.predict()` functions
  + rename the `tests.forecasts.methods.test_ts_methods` module
    into `tests.forecasts.methods.test_predictions`
- re-organize some constants in the `tests` package
- streamline some docstrings
2021-02-01 11:32:10 +01:00
1d63623dfc
Add Forecast.__repr__() 2021-01-31 21:57:05 +01:00
47ef1f8759
Add OrderHistory.first/last_order() methods
- get the `datetime` of the first or last order within a pixel
- unify some fixtures in `tests.forecasts.timify.conftest`
2021-01-31 21:46:20 +01:00
7b824a4a12
Shorten a couple of names
- rename "total_orders" columns into "n_orders"
- rename `.make_*_time_series()` methods into `.make_*_ts()`
2021-01-31 20:20:55 +01:00
d45c60b764
Add OrderHistory.time_step property 2021-01-31 20:06:23 +01:00
fd404e2b89
Adjust Pixel.__repr__() a tiny bit 2021-01-31 19:34:05 +01:00
63e8e94145
Add Pixel.restaurants property
- the property loads all `Restaurant`s from the database that
  are within the `Pixel`
2021-01-31 19:29:18 +01:00
08b748c867
Move decomposition module into methods sub-package
- move the module
- unify the corresponding tests in `tests.forecasts.methods` sub-package
- make all `predict()` and the `stl()` function round results
- streamline documentation
2021-01-31 18:54:58 +01:00
a5b590b24c
Add Forecast.actual column 2021-01-31 18:29:53 +01:00
6429165aaf
Add statsmodels to the dependencies 2021-01-31 18:24:03 +01:00
4b6d92958d
Add functionality for drawing folium.Maps
- this code is not unit-tested due to the complexity involving
  interactive `folium.Map`s => visual checks give high confidence
2021-01-26 17:07:50 +01:00
605ade4078
Add Pixel.northeast/southwest properties
- the properties are needed for the drawing functionalitites
2021-01-26 17:05:36 +01:00
ca2ba0c9d5
Fix missing dependencies in test session 2021-01-24 19:18:09 +01:00