Commit graph

132 commits

Author SHA1 Message Date
0c1ff5338d
Check if predict_at/day is in .totals
- this is a minor sanity check
2021-01-24 18:40:08 +01:00
f36fffdd4d
Add pytest-mock to the dev dependencies 2021-01-24 18:32:07 +01:00
de3e489b39
Adjust flake8 to not consider constants magic 2021-01-24 18:31:02 +01:00
a1da1e9af8
Add matplotlib to the dependencies 2021-01-21 17:15:39 +01:00
2339100371
Add folium to the dependencies 2021-01-21 11:47:22 +01:00
f37d8adb9d
Add confidence intervals to Forecast model
- add `.low80`, `.high80`, `.low95`, and `.high95` columns
- add check contraints for the confidence intervals
- rename the `.method` column into `.model` for consistency
2021-01-20 16:57:39 +01:00
64482f48d0
Add wrappers for R's "arima" and "ets" functions 2021-01-20 13:06:32 +01:00
98b6830b46
Add stl() function
- `stl()` wraps R's "stl" function in Python
- STL is a decomposition method for time series
2021-01-11 16:10:45 +01:00
b0f2fdde10
Add rpy2 to the dependencies
- add a Jupyter notebook that allows to install all project-external
  dependencies regarding R and R packages
- adjust the GitHub Action workflow to also install R and the R packages
  used within the project
- add a `init_r` module that initializes all R packages globally
  once the `urban_meal_delivery` package is imported
2021-01-11 16:06:58 +01:00
84876047c1
Return the resulting time series as pd.Series 2021-01-10 16:11:40 +01:00
9196c88ed4
Remove pylint from the project 2021-01-09 17:47:45 +01:00
100fac659a
Add OrderHistory.make_real_time_time_series()
- the method slices out a real-time time series from the data within
  an `OrderHistory` object
2021-01-09 17:30:00 +01:00
5330ceb771
Add OrderHistory.make_vertical_time_series()
- the method slices out a vertical time series from the data within
  an `OrderHistory` object
2021-01-09 17:28:55 +01:00
b61db734b6
Add OrderHistory.make_horizontal_time_series()
- the method slices out a horizontal time series from the data within
  an `OrderHistory` object
2021-01-09 16:34:42 +01:00
65d1632e98
Add OrderHistory class
- the main purpose of this class is to manage querying the order totals
  from the database and slice various kinds of time series out of the
  data
- the class holds the former `aggregate_orders()` function as a method
- modularize the corresponding tests
- add `tests.config` with globals used when testing to provide a
  single source of truth for various settings
2021-01-09 16:29:58 +01:00
d5b3efbca1
Add aggregate_orders() function
- the function queries the database and aggregates the ad-hoc orders
  by pixel and time steps into a demand time series
- implement "heavy" integration tests for `aggregate_orders()`
- make `pandas` a package dependency
- streamline the `Config`
2021-01-07 23:35:13 +01:00
e8c97dd7da
Add Forecast model to ORM layer
- the model handles the caching of demand forecasting results
- include the database migration script
2021-01-07 12:59:30 +01:00
54ff377579
Add CLI script to gridify all cities
- reorganize `urban_meal_delivery.console` into a sub-package
- move `tests.db.conftest` fixtures into `tests.conftest`
  => some integration tests regarding CLI scripts need a database
- add `urban_meal_delivery.console.decorators.db_revision` decorator
  to ensure the database is at a certain state before a CLI script runs
- refactor the `urban_meal_delivery.db.grids.Grid.gridify()` constructor:
  - bug fix: even empty `Pixel`s end up in the database temporarily
    => create `Pixel` objects only if an `Address` is to be assigned
       to it
  - streamline code and docstring
  - add further test cases
2021-01-06 16:17:05 +01:00
daa224d041
Rename _*_id columns into just *_id 2021-01-05 22:37:12 +01:00
078355897a
Fix missing unique constraint drop 2021-01-05 22:32:24 +01:00
992d2bb7d4
Adjust flake8 ...
... to not complain about implementation details when testing.
2021-01-05 19:08:52 +01:00
776112d609
Add Grid.gridify() constructor
- the purpose of this constructor method is to generate all `Pixel`s
  for a `Grid` that have at least one `Address` assigned to them
- fix missing `UniqueConstraint` in `Grid` class => it was not possible
  to create two `Grid`s with the same `.side_length` in different cities
- change the `City.viewport` property into two separate `City.southwest`
  and `City.northeast` properties; also add `City.total_x` and
  `City.total_y` properties for convenience
2021-01-05 18:58:48 +01:00
a1cbb808fd
Integrate the new Location class
- the old `UTMCoordinate` class becomes the new `Location` class
- its main purpose is to represent locations in both lat-long
  coordinates as well as in the UTM system
- remove `Address.__init__()` and `City.__init__()` methods as they
  are not executed for entries retrieved from the database
- simplfiy the `Location.__init__()` => remove `relative_to` argument
2021-01-04 20:33:10 +01:00
2e3ccd14d5
Use globals for the database connection
- remove the factory functions for creating engines and sessions
- define global engine, connection, and session objects to be used
  everywhere in the urban_meal_delivery package
2021-01-04 20:23:55 +01:00
f996376b13
Add ORM models for the pixel grids
- add Grid, Pixel, and AddressPixelAssociation ORM models
- each Grid belongs to a City an is characterized by the side_length
  of all the square Pixels contained in it
- Pixels aggregate Addresses => many-to-many relationship (that is
  modeled with SQLAlchemy's Association Pattern to implement a couple
  of constraints)
2021-01-03 19:33:36 +01:00
6cb4be80f6
Add Address.x and Address.y coordinates
- the Address.x and Address.y properties use the UTMCoordinate class
  behind the scenes
- x and y are simple coordinates in an x-y plane
- the (0, 0) origin is the southwest corner of Address.city.viewport
2021-01-02 16:29:50 +01:00
6f9935072e
Add UTMCoordinate class
- the class is a utility to abstract working with latitude-longitude
  coordinates in their UTM representation (~ "cartesian plane")
- the class's .x and .y properties enable working with simple x-y
  coordinates where the (0, 0) origin is the lower-left of a city's
  viewport
2021-01-02 14:31:59 +01:00
755677db46
Add utm to the dependencies 2021-01-01 17:59:15 +01:00
556b9d36a3
Add shapely to the dependencies 2020-12-30 17:37:51 +01:00
78dba23d5d
Re-factor the ORM tests to use randomized fake data
- create `*Factory` classes with fakerboy and faker that generate
  randomized instances of the ORM models
- add new pytest marker: "db" are the integration tests involving the
  database whereas "e2e" will be all other integration tests
- streamline the docstrings in the ORM models
2020-12-29 15:40:32 +01:00
416a58f9dc
Add geopy to the dev dependencies 2020-12-28 15:52:08 +01:00
3e0300cb0e
Disable too-few-public-methods error in pylint 2020-12-16 11:04:43 +01:00
2ddd430534
Add Faker to the dev dependencies 2020-12-15 19:07:14 +01:00
8345579b6c
Add factory_boy to the dev dependencies 2020-12-15 12:23:45 +01:00
0aefa22666
Integrate pytest-randomly into the test suite
As a lot of the integration tests populate the database with test data,
it is deemed safer to run the tests in random order to uncover potential
dependencies between distinct test cases.
Because of how the `db_session` fixture is designed, this should already
be taken care of.
2020-12-15 11:35:05 +01:00
b9c3697434
Move notebooks into the research folder 2020-12-14 16:56:27 +01:00
671d209cc5
Move submodule with demand-forecasting paper into research folder 2020-12-14 16:21:12 +01:00
86ad139c7b
Fix --require-hashes mode in GitHub Actions
- GitHub Actions complains about missing hashes in poetry's export
  of pinned dependencies
- as an example, see https://github.com/webartifex/urban-meal-delivery/runs/1550750320
2020-12-14 15:26:57 +01:00
c1064673aa
Isolate configuration related code better
- create the global `config` object inside the
  `urban_meal_delivery.configuration` module
- streamline documentation and comments
2020-12-14 15:15:08 +01:00
9ee9c04a69
Remove python-dotenv from the dependencies
zsh-dotenv automatically loads the environment variables upon entering
the project's root.
2020-12-14 14:26:12 +01:00
570cb0112e
Pin the dependencies ...
... after upgrading:
- dev dependencies
  + packaging
  + pre-commit
  + pytest
  + sphinx
- research dependencies
  + pandas
- transient dependencies
  + appnode   + argcomplete        + babel         + bandit
  + certifi   + cffi               + colorlog      + darglint
  + identify  + ipykernel          + jupyter-core  + nest-asyncio
  + pathspec  + prometheus-client  + py            + pygments
  + pywin32   + pyzmq              + regex         + requests
  + restructedtext-lint            + stevedore     + testfixtures
  + urllib3   + virtualenv
2020-12-14 13:54:26 +01:00
143ecba98e
Update submodule for demand-forecasting paper
The paper is now published.
2020-12-14 13:46:45 +01:00
51bb7e8235
Adjust the branch reference fixer task's logic ...
... to assume a feature branch if the branch name does not start with
'release' or 'research' and change all references into 'develop'.
2020-11-07 16:42:35 +01:00
03e498cab9
Pin the dependencies ...
... after upgrading:
- sqlalchemy
- dev dependencies
  + darglint
  + flake8(-comprehensions)
  + pre-commit
  + pytest
  + sphinx(-autodoc-typehints)
- reseach dependencies
  + jupyterlab
  + numpy
  + pandas
  + pytz
- transient dependencies
  + attrs
  + colorama
  + gitpython
  + identify
  + iniconfig
  + ipython
  + nbclient
  + nbconvert
  + nbformat
  + nest-asyncio
  + notebook
  + pandocfilters
  + pbr
  + prompt-toolkit
  + pygments
  + regex
  + testfixtures
  + toml
  + tornado
  + traitlets
  + urllib3
  + virtualenv
2020-11-07 16:25:18 +01:00
af5d54f159
Upgrade poetry to v1.1.0
The order of keys in the poetry.lock file is changed.
2020-11-07 16:23:27 +01:00
f8fd9c83bd
Add submodule for demand forecasting paper 2020-11-07 12:51:09 +01:00
88a9b8101c
Merge branch 'research-clean-data' into develop 2020-09-30 13:52:01 +02:00
6e852c8e06
Merge branch 'research-clean-data' into main
Some checks failed
CI / tests (push) Has been cancelled
2020-09-30 13:48:51 +02:00
6d9e5ffcef
Add info about the data cleaning 2020-09-30 13:43:00 +02:00
6333f1af1e
Clean the raw data
- clean the raw data given by the undisclosed meal delivery platform:
  + keep data only for the three target citis:
    * Bordeaux
    * Lyon
    * Paris
  + merge duplicates
    * it appears as redundant addresses were created
      for each order by the same customer
      =>  significant reduction in the number of addresses
    * propagate the merges to the other tables
      that reference records merged away
  + cast data types and keep their scopes narrow
  + normalize the data
  + remove obvious outliers
  + adjust/discard unplausible values
- map the cleaned data onto the ORM models
- store the cleaned data in a new database schema
2020-09-30 13:39:48 +02:00