Alexander Hess
a3a17236a2
- replace pipenv with poetry
- update the README.md:
* streamline the text
* update links to notebooks with nbviewer
* update installation notes with poetry info
- streamline the notebooks:
* use backticks in MarkDown cells to make references to
columns in DataFrames clearer
* blacken all code cells
- add MIT license
- ignore .venv/ and .python-version
70 lines
3.2 KiB
Markdown
70 lines
3.2 KiB
Markdown
# Tidy Data
|
|
|
|
The purpose of this repository is to illustrate how the data cleaning process described
|
|
in the paper "[Tidy Data](tidy-data.pdf)" by Hadley Wickham, a member of the
|
|
[RStudio](https://rstudio.com/) team, can be done in
|
|
[Python](https://www.python.org/).
|
|
|
|
The paper was published in 2014 in the [Journal of Statistical Software](https://www.jstatsoft.org/article/view/v059i10).
|
|
The author offers it for free [here](http://vita.had.co.nz/papers/tidy-data.html).
|
|
Furthermore, the original [R](https://www.r-project.org/) code is available [here](https://github.com/hadley/tidy-data).
|
|
|
|
After installing the dependencies for this project (cf., the [installation notes](https://github.com/webartifex/tidy-data#installation)
|
|
below), it is recommended to first read the paper to get the big picture and
|
|
then work through the six Jupyter notebooks listed below.
|
|
|
|
|
|
## Summary
|
|
|
|
|
|
### Definition
|
|
|
|
**Tidy** data is defined as data that comes in a table form adhering to the
|
|
following requirements:
|
|
1. each variable is a column,
|
|
2. each observation a row, and
|
|
3. each type of observational unit forms a table.
|
|
|
|
This is equivalent to [Codd's 3rd normal form](https://en.wikipedia.org/wiki/Third_normal_form),
|
|
a concept from the theory on relational databases.
|
|
A dataset that does *not* satisfy these properties is called **messy**.
|
|
|
|
|
|
### Tidying Data
|
|
|
|
The five most common problems with messy data are:
|
|
|
|
- column headers are values, not variable names
|
|
(cf., [notebook 1](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/1_column_headers_are_values.ipynb))
|
|
- multiple variables are stored in one column
|
|
(cf., [notebook 2](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/2_multiple_variables_stored_in_one_column.ipynb))
|
|
- variables are stored in both rows and columns
|
|
(cf., [notebook 3](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/3_variables_are_stored_in_both_rows_and_columns.ipynb))
|
|
- multiple types of observational units are stored in the same table
|
|
(cf., [notebook 4](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/4_multiple_types_in_one_table.ipynb))
|
|
- a single observational unit is stored in multiple tables
|
|
(cf., [notebook 5](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/5_one_type_in_multiple_tables.ipynb))
|
|
|
|
|
|
### Case Study
|
|
|
|
A case study (cf., [notebook 6](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/6_case_study.ipynb))
|
|
shows the advantages of tidy data as a standardized input to statistical functions.
|
|
|
|
|
|
## Installation
|
|
|
|
Get a local copy of this repository with [git](https://git-scm.com/).
|
|
|
|
`git clone https://github.com/webartifex/tidy-data.git`
|
|
|
|
If you are not familiar with [git](https://git-scm.com/), simply download the latest
|
|
version of the files in a zip archive [here](https://github.com/webartifex/tidy-data/archive/master.zip).
|
|
|
|
This project uses [poetry](https://python-poetry.org/docs/) to manage its dependencies.
|
|
Install all third-party packages into a [virtual environment](https://docs.python.org/3/library/venv.html).
|
|
|
|
`poetry install`
|
|
|
|
Alternatively, use the [Anaconda Distribution](https://www.anaconda.com/products/individual)
|
|
that *should* also suffice to run the provided notebooks.
|