data consitency

A simple toy project that lets you evaluate the consistency of your tabular data in just a few clicks.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact

About The Project

Have you ever struggled to compare datasets?

In many data science or data engineering projects, at one point or another you have to validate certain concepts that are crucial to the data you're exploiting. When you go into production, it's common to translate these concepts into monitoring indicators and check that you're not drifting away from what you already know (see data drift).

This project proposes a way of quickly validating the differences and similarities between two datasets, in a word, checking the consistency of the data.

(back to top)

Built With

• 
• 
• 

(back to top)

Getting Started

To run and use the app, you have two options: either run it locally or use a docker container.

Prerequisites

• Get make: https://www.gnu.org/software/make/

Prerequisites #1 - run locally

• Get python: https://www.python.org/downloads/
• Get conda: https://docs.conda.io/projects/conda/en/latest/commands/install.html
• Create your environment:

  conda create -n streamlit-data-consistency python=3.10

• Activate your environment:

  conda activate streamlit-data-consistency

• Install libraries:

  pip install -r requirements.txt

Prerequisites #2 - run through docker

• Install Docker following the installation guide for your platform: here
• Build the docker image:

  make docker-build

Installation

1. Start the app using your local environment...

   make run

2. Or start the app using your local environment...

   make run-locally

3. Visit http://localhost:8501

(back to top)

Usage

You need to provide 3 files:

• a description of the data schema: json file representing the internal representation and data type of your data sets, a template file can be found in /demo folder.
• a first dataset (supported file type are csv and xlsx)
• a second dataset (supported file type are csv and xlsx)

For the moment, it's just a simple pairwise comparison.

Several sections are available based on the provided data type declared in the schema file :

1. Overview section:

• compare the shape, the header (columns in common, in one datasets and not in the other)...

2. Numerical section:

• for each numerical columns get the null distribution, the min/max and compute the Jensen-Shannon divergence...

3. Categorical tab:

• for each categorical columns get the null distribution, the count modalities and compute the Jaccard dissimilarity...

4. Temporal tab:

• for each temporal columns displays a Gantt Chart to visualize the span of periods within each dataset...

(back to top)

Roadmap

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the AGPL-3.0 License. See LICENSE.txt for more information.

(back to top)

Contact

Project Link: https://github.com/nielsborie/streamlit-data-consistency

(back to top)