Newer
Older
The purpose of this repository is to illustrate how the data cleaning process described
in the paper "[Tidy Data](tidy-data.pdf)" by Hadley Wickham, a member of the
[RStudio](https://rstudio.com/) team, can be done in
[Python](https://www.python.org/).
The paper was published in 2014 in the [Journal of Statistical Software](https://www.jstatsoft.org/article/view/v059i10).
The author offers it for free [here](http://vita.had.co.nz/papers/tidy-data.html).
Furthermore, the original [R](https://www.r-project.org/) code is available [here](https://github.com/hadley/tidy-data).
After installing the dependencies for this project (cf., the [installation notes](https://github.com/webartifex/tidy-data#installation)
below), it is recommended to first read the paper to get the big picture and
then work through the six Jupyter notebooks listed below.
## Summary
### Definition
**Tidy** data is defined as data that comes in a table form adhering to the
following requirements:
1. each variable is a column,
2. each observation a row, and
3. each type of observational unit forms a table.
This is equivalent to [Codd's 3rd normal form](https://en.wikipedia.org/wiki/Third_normal_form),
a concept from the theory on relational databases.
A dataset that does *not* satisfy these properties is called **messy**.
The five most common problems with messy data are:
- column headers are values, not variable names
(cf., [notebook 1](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/1_column_headers_are_values.ipynb))
- multiple variables are stored in one column
(cf., [notebook 2](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/2_multiple_variables_stored_in_one_column.ipynb))
- variables are stored in both rows and columns
(cf., [notebook 3](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/3_variables_are_stored_in_both_rows_and_columns.ipynb))
- multiple types of observational units are stored in the same table
(cf., [notebook 4](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/4_multiple_types_in_one_table.ipynb))
- a single observational unit is stored in multiple tables
(cf., [notebook 5](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/5_one_type_in_multiple_tables.ipynb))
A case study (cf., [notebook 6](https://nbviewer.jupyter.org/github/webartifex/tidy-data/blob/master/6_case_study.ipynb))
shows the advantages of tidy data as a standardized input to statistical functions.
## Installation
Get a local copy of this repository with [git](https://git-scm.com/).
`git clone https://github.com/webartifex/tidy-data.git`
If you are not familiar with [git](https://git-scm.com/), simply download the latest
version of the files in a zip archive [here](https://github.com/webartifex/tidy-data/archive/master.zip).
This project uses [poetry](https://python-poetry.org/docs/) to manage its dependencies.
Install all third-party packages into a [virtual environment](https://docs.python.org/3/library/venv.html).
Alternatively, use the [Anaconda Distribution](https://www.anaconda.com/products/individual)
that *should* also suffice to run the provided notebooks.