- Sep 30, 2020
-
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
- clean the raw data given by the undisclosed meal delivery platform: + keep data only for the three target citis: * Bordeaux * Lyon * Paris + merge duplicates * it appears as redundant addresses were created for each order by the same customer => significant reduction in the number of addresses * propagate the merges to the other tables that reference records merged away + cast data types and keep their scopes narrow + normalize the data + remove obvious outliers + adjust/discard unplausible values - map the cleaned data onto the ORM models - store the cleaned data in a new database schema
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
.. after upgrading a couple of packages
-
Alexander Hess authored
-
Alexander Hess authored
- change references to temporary branches (e.g., "release-*" and "publish") to point to the 'main' branch - add --branch=BRANCH_NAME option to the nox session so that one can pass in a target branch to make all references point to - run "fix-branch-references" as the first pre-commit hook as it fails the fastest - bug fix: allow dots in branch references (e.g., "release-0.1.0")
-
- Aug 11, 2020
-
-
Alexander Hess authored
-
Alexander Hess authored
- dependencies used to run the Jupyter Lab environment that are not required by the `urban-meal-delivery` package itself are put into an installation extra called "research" - this allows to NOT install the requirements, for example, when testing the package in an isolated environment
-
Alexander Hess authored
- run "format" and "lint" separately => remove the nox session "pre-commit" - execute the entire "test-suite" before merges + rename "pre-merge" into "test-suite" + do not "format" and "lint" here any more + do not execute this before pushes * allow branches with <100% test coverage to exist on GitHub (they cannot be merged into 'main' until 100% coverage) * GitHub Actions executes the test suite
-
Alexander Hess authored
- many *.py and *.ipynb files will contain links to resources on GitHub or nbviewer that have branch references in them - add a pre-commit hook implemented as the nox session "fix-branch-references" that goes through these files and changes all the branch labels to the current one
-
Alexander Hess authored
- some glob patterns in .gitignore were not correctly expanded - adapt the exclude logic to focus on the start of the excluded paths
-
Alexander Hess authored
- use Alembic to migrate the PostgreSQL database + create initial migration script to set up the database, as an alternative to db.Base.metadata.create_all() + integrate Alembic into the test suite; the db_engine fixture now has two modes: * create the latest version of tables all at once * invoke `alembic upgrade head` => the "e2e" tests are all run twice, once in each mode; this ensures that the migration scripts re-create the same database schema as db.Base.metadata.create_all() would * in both modes, a temporary PostgreSQL schema is used to create the tables in => could now run "e2e" tests against production database and still have isolation - make the configuration module public (to be used by Alembic) - adjust linting rules for Alembic
-
Alexander Hess authored
- use SQLAlchemy (and PostgreSQL) to model the ORM layer - add the following models: + Address => modelling all kinds of addresses + City => model the three target cities + Courier => model the UDP's couriers + Customer => model the UDP's customers + Order => model the orders received by the UDP + Restaurant => model the restaurants active on the UDP - so far, the emphasis lies on expression the Foreign Key and Check Constraints that are used to validate the assumptions inherent to the cleanded data - provide database-independent unit tests with 100% coverage - provide additional integration tests ("e2e") that commit data to a PostgreSQL instance to validate that the constraints work - adapt linting rules a bit
-
Alexander Hess authored
... after upgrading: - flake8-plugin-utils - sphinx
-
Alexander Hess authored
- add the following file: + src/urban_meal_delivery/_config.py - a config module is created holding two sets of configurations: + production => against the real database + testing => against a database with test data - the module is "protected" (i.e., underscore) and imported at the top level via a proxy-like object `config` that detects in which of the two environments the package is being run
-
- Aug 05, 2020
-
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
-
Alexander Hess authored
- only support Python 3.8 in the project as GitHub Actions has no 3.9 in its base image - re-order some sessions so that the order in `poetry run nox --list` is more intuitive - add extensive documentation to the noxfile.py module - add more whitespace to noxfile.py to make it easier to read - add generic maintainance tasks: + "init-project" => set up all pre-commit hooks + "clean-pwd" => ~ `git clean -X` with minor exceptions - upgrade isort to 5.3.0
-
Alexander Hess authored
- use sphinx to document the developed package - create a nox session "docs" that builds the docs - include a skeleton in the docs/ folder + how to install the package + how to use nox + license
-
Alexander Hess authored
- use xdoctest to validate all code snippets in docstrings - add xdoctest to the nox session "test"
-
- Aug 04, 2020
-
-
Alexander Hess authored
- add a nox session "safety"
-
Alexander Hess authored
- for tests/ and the noxfile.py, type annotations are not strictly enforced any more + this simplifies the way test cases and nox sessions are written + for many pytest fixtures, no types are available via a public API - put fixtures inside the classes the corresponding test cases are grouped in
-
Alexander Hess authored
- add the following file: + src/urban_meal_delivery/console.py => click-based CLI tools + tests/test_console.py => tests for the module above - add a CLI entry point `umd`: + implement the --version / -V option to show the installed package's version + rework to package's top-level: * add a __pkg_name__ variable to parameterize the package name + add unit and integration tests - fix that pylint cannot know the proper order of imports in the isolated nox session
-
Alexander Hess authored
- add pre-commit and pre-merge hooks: + run `poetry run nox -s pre-commit` on staged *.py files + run common pre-commit hooks for validations that could not be achieved with tools in the develop environment so easily + run `poetry run nox -s pre-merge` before merges and pushes - implement the "pre-commit" and "pre-merge" sessions in nox + include a little hack to deal with the positional arguments passed by the pre-commit framework - provide more documentation on the nox sessions
-
Alexander Hess authored
- use pytest as the base, measure coverage with pytest-cov + configure coverage to include branches and specify source locations + configure pytest to enforce explicit markers - add a package for the test suite under tests/ - add a `__version__` identifier at the package's root + it is dynamically assigned the version of the installed package + the version is PEP440 compliant and follows a strict subset of semantic versioning: x.y.z[.devN] where x, y, z, and N are all non-negative integers + add module with tests for the __version__ - add a nox session "test" that runs the test suite - use flake8 to lint pytest for consistent style
-
- Aug 03, 2020
-
-
Alexander Hess authored
- use flake8 as the main and pylint as the auxiliary linter - install flake8 with the following plug-ins: + flake8-annotations => enforce type annotations for functions/classes + flake8-black => ensure black would not make any changes + flake8-expression-complexity + wemake-python-styleguide, which packages the following: * darglint * flake8-bandit * flake8-broken-line * flake8-bugbear * flake8-commas * flake8-comprehensions * flake8-debugger * flake8-docstrings * flake8-eradicate * flake8-isort * flake8-rst-docstrings * flake8-string-format * flake8-quotes * pep8-naming - configure flake8 & friends in a rather explicit and strict way - isort needed to be downgraded to ^4.3.21 due to a conflict with pylint and wemake-python-styleguide: + provide TODO's to remove the parts that "fix" isort - use mypy for static type checking - add a nox session "lint" that runs flake8, mypy, and pylint - lint all source files
-
Alexander Hess authored
- (auto-)format code with: + autoflake => * remove unused imports and variables * remove duplicate dict keys * expand star imports + black => enforce an uncompromising code style + isort => enforce a consistent import style (complying with Google's Python Style Guide) - implement the nox session "format" that runs all these tools - add the following file: + setup.cfg => holds configurations for the develop tools
-
Alexander Hess authored
- set up skeletons for all planned nox sessions - provide a base configuration for all nox sessions - create two utility functions: + _begin() => run commands common to all sessions + _install_packages() => install dependencies in nox sessions respecting the versions pinned in poetry.lock - add the following file: + noxfile.py
-
Alexander Hess authored
- add the following files: + .gitignore => ignore poetry's and pyenv's artifacts + LICENSE.txt => MIT license as the project is scientific research + README.md => rough description of the project + poetry.lock + pyproject.toml + src/urban_meal_delivery/__init__.py => source code package - use a "src" layout structure: + ensure that pytest runs the tests against a packaged and pip installed version of the source code, not the *.py files in the project directory + more info: https://hynek.me/articles/testing-packaging/
-