Commit graph

57 commits

Author SHA1 Message Date
28a7c7451c
Rename existing notebooks using order numbers 2021-02-04 15:10:14 +01:00
0da86e5f07
Pin the dependencies ...
... after upgrading:
- alembic
- matplotlib
- pandas
- rpy2
- sqlalchemy
- statsmodels
- dev dependencies
  + coverage     + factory-boy    + faker                 + nox
  + packaging    + pre-commit     + flake8-annotations    + pytest
  + pytest-cov   + sphinx
- research dependencies
  + numpy        + pyty
- transient dependencies
  + astpretty    + atomicwrites   + bleach                + chardet
  + colorlog     + darglint       + flake8-comprehensions + gitpython
  + identify     + ipykernel      + ipython               + jedi
  + jinja2       + jupyter-client + jupyter-core          + mako
  + nbformat     + nest-asyncio   + notebook              + parso
  + pluggy       + prompt-toolkit + ptyprocess            + pygments
  + pyyaml       + pyzmq          + requests              + smmap
  + terminado    + textfixtures   + snowballstemmer       + typed-ast
  + urllib3      + virtualenv

- fix SQL statements written in raw text
2021-02-04 13:12:47 +01:00
50b35a8284
Add CLI script to run tactical forecasting heuristic 2021-02-04 12:05:43 +01:00
23391c2fa4
Adjust OrderHistory.choose_tactical_model() heuristic
- use the `HorizontalSMAModel` for low demand
- use the `TrivialModel` for no demand
2021-02-02 15:20:02 +01:00
3f5b4a50bb
Rename Forecast.training_horizon into .train_horizon
- we use that shorter name in `urban_meal_delivery.forecasts.*`
  and want to be consistent in the ORM layer as well
2021-02-02 13:09:09 +01:00
6fd16f2a6c
Add TrivialModel
- the trivial model simply predicts `0` demand for all time steps
2021-02-02 12:45:26 +01:00
015d304306
Add HorizontalSMAModel
- the model applies a simple moving average on horizontal time series
- refactor `db.Forecast.from_dataframe()` to correctly convert
  `float('NaN')` values into `None`; otherwise, SQLAlchemy complains
2021-02-02 12:40:53 +01:00
af82951485
Add OrderHistory.choose_tactical_model()
- the method implements a heuristic from the first research paper
  that chooses the most promising forecasting `*Model` based on
  the average daily demand in a `Pixel` for a given `train_horizon`
- adjust the test scenario => `LONG_TRAIN_HORIZON` becomes `8`
  as that is part of the rule implemented in the heuristic
2021-02-02 11:29:27 +01:00
cb7611d587
Add OrderHistory.avg_daily_demand()
- the method calculates the number of daily `Order`s in a `Pixel`
  withing the `train_horizon` preceding the `predict_day`
2021-02-01 21:50:42 +01:00
67cd58cf16
Add urban_meal_delivery.forecasts.models sub-package
- `*Model`s use the `methods.*.predict()` functions to predict demand
  given an order time series generated by `timify.OrderHistory`
- `models.base.ForecastingModelABC` unifies how all `*Model`s work
  and implements a caching strategy
- implement three `*Model`s for tactical forecasting, based on the
  hets, varima, and rtarima models described in the first research paper
- add overall documentation for `urban_meal_delivery.forecasts` package
- move the fixtures in `tests.forecasts.timify.conftest` to
  `tests.forecasts.conftest` and adjust the horizon of the test horizon
  from two to three weeks
2021-02-01 20:39:52 +01:00
796fdc919c
Add Forecast.from_dataframe() constructor
- this alternative constructor takes the `pd.DataFrame`s from the
  `*Model.predict()` methods and converts them into ORM models
2021-02-01 15:50:30 +01:00
b8952213d8
Add extrapolate_season.predict() function
- the function implements a forecasting "method" similar to the
  seasonal naive method
  => instead of simply taking the last observation given a seasonal lag,
     it linearly extrapolates all observations of the same seasonal lag
     from the past into the future; conceptually, it is like the
     seasonal naive method with built-in smoothing
- the function is tested just like the `arima.predict()` and
  `ets.predict()` functions
  + rename the `tests.forecasts.methods.test_ts_methods` module
    into `tests.forecasts.methods.test_predictions`
- re-organize some constants in the `tests` package
- streamline some docstrings
2021-02-01 11:32:10 +01:00
1d63623dfc
Add Forecast.__repr__() 2021-01-31 21:57:05 +01:00
47ef1f8759
Add OrderHistory.first/last_order() methods
- get the `datetime` of the first or last order within a pixel
- unify some fixtures in `tests.forecasts.timify.conftest`
2021-01-31 21:46:20 +01:00
7b824a4a12
Shorten a couple of names
- rename "total_orders" columns into "n_orders"
- rename `.make_*_time_series()` methods into `.make_*_ts()`
2021-01-31 20:20:55 +01:00
d45c60b764
Add OrderHistory.time_step property 2021-01-31 20:06:23 +01:00
fd404e2b89
Adjust Pixel.__repr__() a tiny bit 2021-01-31 19:34:05 +01:00
63e8e94145
Add Pixel.restaurants property
- the property loads all `Restaurant`s from the database that
  are within the `Pixel`
2021-01-31 19:29:18 +01:00
08b748c867
Move decomposition module into methods sub-package
- move the module
- unify the corresponding tests in `tests.forecasts.methods` sub-package
- make all `predict()` and the `stl()` function round results
- streamline documentation
2021-01-31 18:54:58 +01:00
a5b590b24c
Add Forecast.actual column 2021-01-31 18:29:53 +01:00
4b6d92958d
Add functionality for drawing folium.Maps
- this code is not unit-tested due to the complexity involving
  interactive `folium.Map`s => visual checks give high confidence
2021-01-26 17:07:50 +01:00
605ade4078
Add Pixel.northeast/southwest properties
- the properties are needed for the drawing functionalitites
2021-01-26 17:05:36 +01:00
1bfc7db916
Make Grid.gridify() use only pickup addresses
- ensure a `Restaurant` only has one unique `Order.pickup_address`
- rework `Grid.gridify()` so that only pickup addresses are assigned
  into `Pixel`s
- include database migrations to ensure the data adhere to these
  tighter constraints
2021-01-24 19:04:39 +01:00
0c1ff5338d
Check if predict_at/day is in .totals
- this is a minor sanity check
2021-01-24 18:40:08 +01:00
de3e489b39
Adjust flake8 to not consider constants magic 2021-01-24 18:31:02 +01:00
f37d8adb9d
Add confidence intervals to Forecast model
- add `.low80`, `.high80`, `.low95`, and `.high95` columns
- add check contraints for the confidence intervals
- rename the `.method` column into `.model` for consistency
2021-01-20 16:57:39 +01:00
64482f48d0
Add wrappers for R's "arima" and "ets" functions 2021-01-20 13:06:32 +01:00
98b6830b46
Add stl() function
- `stl()` wraps R's "stl" function in Python
- STL is a decomposition method for time series
2021-01-11 16:10:45 +01:00
b0f2fdde10
Add rpy2 to the dependencies
- add a Jupyter notebook that allows to install all project-external
  dependencies regarding R and R packages
- adjust the GitHub Action workflow to also install R and the R packages
  used within the project
- add a `init_r` module that initializes all R packages globally
  once the `urban_meal_delivery` package is imported
2021-01-11 16:06:58 +01:00
84876047c1
Return the resulting time series as pd.Series 2021-01-10 16:11:40 +01:00
9196c88ed4
Remove pylint from the project 2021-01-09 17:47:45 +01:00
100fac659a
Add OrderHistory.make_real_time_time_series()
- the method slices out a real-time time series from the data within
  an `OrderHistory` object
2021-01-09 17:30:00 +01:00
5330ceb771
Add OrderHistory.make_vertical_time_series()
- the method slices out a vertical time series from the data within
  an `OrderHistory` object
2021-01-09 17:28:55 +01:00
b61db734b6
Add OrderHistory.make_horizontal_time_series()
- the method slices out a horizontal time series from the data within
  an `OrderHistory` object
2021-01-09 16:34:42 +01:00
65d1632e98
Add OrderHistory class
- the main purpose of this class is to manage querying the order totals
  from the database and slice various kinds of time series out of the
  data
- the class holds the former `aggregate_orders()` function as a method
- modularize the corresponding tests
- add `tests.config` with globals used when testing to provide a
  single source of truth for various settings
2021-01-09 16:29:58 +01:00
d5b3efbca1
Add aggregate_orders() function
- the function queries the database and aggregates the ad-hoc orders
  by pixel and time steps into a demand time series
- implement "heavy" integration tests for `aggregate_orders()`
- make `pandas` a package dependency
- streamline the `Config`
2021-01-07 23:35:13 +01:00
e8c97dd7da
Add Forecast model to ORM layer
- the model handles the caching of demand forecasting results
- include the database migration script
2021-01-07 12:59:30 +01:00
54ff377579
Add CLI script to gridify all cities
- reorganize `urban_meal_delivery.console` into a sub-package
- move `tests.db.conftest` fixtures into `tests.conftest`
  => some integration tests regarding CLI scripts need a database
- add `urban_meal_delivery.console.decorators.db_revision` decorator
  to ensure the database is at a certain state before a CLI script runs
- refactor the `urban_meal_delivery.db.grids.Grid.gridify()` constructor:
  - bug fix: even empty `Pixel`s end up in the database temporarily
    => create `Pixel` objects only if an `Address` is to be assigned
       to it
  - streamline code and docstring
  - add further test cases
2021-01-06 16:17:05 +01:00
daa224d041
Rename _*_id columns into just *_id 2021-01-05 22:37:12 +01:00
776112d609
Add Grid.gridify() constructor
- the purpose of this constructor method is to generate all `Pixel`s
  for a `Grid` that have at least one `Address` assigned to them
- fix missing `UniqueConstraint` in `Grid` class => it was not possible
  to create two `Grid`s with the same `.side_length` in different cities
- change the `City.viewport` property into two separate `City.southwest`
  and `City.northeast` properties; also add `City.total_x` and
  `City.total_y` properties for convenience
2021-01-05 18:58:48 +01:00
a1cbb808fd
Integrate the new Location class
- the old `UTMCoordinate` class becomes the new `Location` class
- its main purpose is to represent locations in both lat-long
  coordinates as well as in the UTM system
- remove `Address.__init__()` and `City.__init__()` methods as they
  are not executed for entries retrieved from the database
- simplfiy the `Location.__init__()` => remove `relative_to` argument
2021-01-04 20:33:10 +01:00
2e3ccd14d5
Use globals for the database connection
- remove the factory functions for creating engines and sessions
- define global engine, connection, and session objects to be used
  everywhere in the urban_meal_delivery package
2021-01-04 20:23:55 +01:00
f996376b13
Add ORM models for the pixel grids
- add Grid, Pixel, and AddressPixelAssociation ORM models
- each Grid belongs to a City an is characterized by the side_length
  of all the square Pixels contained in it
- Pixels aggregate Addresses => many-to-many relationship (that is
  modeled with SQLAlchemy's Association Pattern to implement a couple
  of constraints)
2021-01-03 19:33:36 +01:00
6cb4be80f6
Add Address.x and Address.y coordinates
- the Address.x and Address.y properties use the UTMCoordinate class
  behind the scenes
- x and y are simple coordinates in an x-y plane
- the (0, 0) origin is the southwest corner of Address.city.viewport
2021-01-02 16:29:50 +01:00
6f9935072e
Add UTMCoordinate class
- the class is a utility to abstract working with latitude-longitude
  coordinates in their UTM representation (~ "cartesian plane")
- the class's .x and .y properties enable working with simple x-y
  coordinates where the (0, 0) origin is the lower-left of a city's
  viewport
2021-01-02 14:31:59 +01:00
78dba23d5d
Re-factor the ORM tests to use randomized fake data
- create `*Factory` classes with fakerboy and faker that generate
  randomized instances of the ORM models
- add new pytest marker: "db" are the integration tests involving the
  database whereas "e2e" will be all other integration tests
- streamline the docstrings in the ORM models
2020-12-29 15:40:32 +01:00
3e0300cb0e
Disable too-few-public-methods error in pylint 2020-12-16 11:04:43 +01:00
c1064673aa
Isolate configuration related code better
- create the global `config` object inside the
  `urban_meal_delivery.configuration` module
- streamline documentation and comments
2020-12-14 15:15:08 +01:00
9ee9c04a69
Remove python-dotenv from the dependencies
zsh-dotenv automatically loads the environment variables upon entering
the project's root.
2020-12-14 14:26:12 +01:00
a16c260543
Add database migrations
- use Alembic to migrate the PostgreSQL database
  + create initial migration script to set up the database,
    as an alternative to db.Base.metadata.create_all()
  + integrate Alembic into the test suite; the db_engine fixture
    now has two modes:
    * create the latest version of tables all at once
    * invoke `alembic upgrade head`
    => the "e2e" tests are all run twice, once in each mode; this
       ensures that the migration scripts re-create the same database
       schema as db.Base.metadata.create_all() would
    * in both modes, a temporary PostgreSQL schema is used to create the
      tables in
    => could now run "e2e" tests against production database and still
       have isolation
- make the configuration module public (to be used by Alembic)
- adjust linting rules for Alembic
2020-08-11 10:29:58 +02:00