dev-fix/Improvements for Local Development Setup and Unit Test Structure

[mhmotamedi]

PR Description:

Improvements for Local Development Setup and Unit Test Structure

This PR focuses on resolving issues with local environment setup and unit test execution in the dataprofiler library. The changes ensure a more reliable setup process, eliminate dependency conflicts, and enhance the performance and maintainability of unit tests. It also involves fixing Pre-commit Hook Errors.

Changes Introduced

1. Refactored Makefile

• Dependency Installation Order: Moved requirements-ml.txt earlier in the installation sequence to prevent conflicts with TensorFlow and shared dependencies like NumPy.

• Fresh Dependency Downloads: Added --no-cache-dir to pip install commands to ensure fresh downloads, avoiding issues caused by stale or corrupted cached files.

• Virtual Environment Isolation: Updated the test command to explicitly use venv/bin/python, ensuring tests use the correct Python interpreter and dependencies from the virtual environment.

• Python Version Check: Introduced a check-python target to ensure the specified Python version is 3.9, 3.10, or 3.11. If the Python version does not match, the Makefile exits with a clear error message, improving reliability across environments.

• Help Section: Added a help target to display all available Makefile commands with descriptions. This makes the development workflow more transparent and easier to understand for new and existing contributors.

2. Refactored Unit Tests

• Improved Test Isolation: Replaced global variables with setUpClass and tearDownClass, ensuring proper initialization and cleanup of shared resources within each test suite. (dataprofiler/tests/profilers/test_profile_builder.py and dataprofiler/tests/test_data_profiler.py)
• Faster Test Execution: Centralized resource setup to reduce redundant operations, improving test performance without relying on the --forked flag, which can be slow for local runs.

3. Fix Pre-commit Hook Errors and Finalize Configurations:

• upgraded mypy version to resolve the following error:

58
/home/runner/.cache/pre-commit/repo5__fa9id/py_env-python3/lib/python3.10/site-packages/dask/dataframe/dask_expr/_backends.py:129: error: Condition can't be inferred, unable to merge overloads
59
/home/runner/.cache/pre-commit/repo5__fa9id/py_env-python3/lib/python3.10/site-packages/dask/dataframe/dask_expr/_backends.py:136: error: Condition can't be inferred, unable to merge overloads

• refactored dataprofiler/reports/graphs.py module to resolve mypy errors raised after upgrade:

dataprofiler/reports/graphs.py:112: error: Return value expected [return-value]
dataprofiler/reports/graphs.py:183: error: Item "None" of "Optional[Axes]" has no attribute "set" [union-attr]
dataprofiler/reports/graphs.py:186: error: Item "None" of "Optional[Axes]" has no attribute "set_title" [union-attr]
dataprofiler/reports/graphs.py:187: error: Incompatible return value type (got "Optional[Axes]", expected "Axes") [return-value]
dataprofiler/reports/graphs.py:195: error: Function "matplotlib.pyplot.figure" is not valid as a type [valid-type]
dataprofiler/reports/graphs.py:195: note: Perhaps you need "Callable[...]" or a callback protocol?
dataprofiler/reports/graphs.py:219: error: Function "matplotlib.pyplot.figure" is not valid as a type [valid-type]
dataprofiler/reports/graphs.py:219: note: Perhaps you need "Callable[...]" or a callback protocol?
dataprofiler/reports/graphs.py:247: error: Return value expected [return-value]
dataprofiler/reports/graphs.py:266: error: Incompatible types in assignment (expression has type "Union[Figure, SubFigure, None]", variable has type "Figure") [assignment]

• refactored dataprofiler/labelers/regex_model.py module to resolve mypy errors raised after upgrade:

  dataprofiler/labelers/regex_model.py:168: error: Missing return statement [empty-body]
  dataprofiler/labelers/regex_model.py:168: note: If the method is meant to be abstract, use @abc.abstractmethod
  dataprofiler/labelers/column_name_model.py:177: error: Missing return statement [empty-body]

• refactored dataprofiler/profilers/profiler_options.py module to resolve mypy errors raised after upgrade:

  dataprofiler/profilers/profile_builder.py:1081: error: Dict entry 0 has incompatible type "str": "Optional[BaseDataLabeler]"; expected "str": "bool" [dict-item]
  dataprofiler/profilers/profile_builder.py:1086: error: Dict entry 0 has incompatible type "str": "Optional[BaseDataLabeler]"; expected "str": "bool" [dict-item]

• Ran pre-commit hooks (mypy, flake8, check-manifest, etc.) to validate changes across the project files.

• Force-pushed updates after resolving errors to ensure all checks pass.

Note: This PR has not fixed the override mypy errors yet. This can be fixed with another PR.

> dataprofiler/data_readers/parquet_data.py:69: error: Signature of "file_encoding" incompatible with supertype "BaseData"  [override]
> dataprofiler/profilers/numerical_column_stats.py:368: error: Signature of "profile" incompatible with supertype "BaseColumnProfiler"  [override]
> dataprofiler/profilers/float_column_profile.py:197: error: Signature of "profile" incompatible with supertype "NumericStatsMixin"  [override]
> dataprofiler/profilers/text_column_profile.py:87: error: Signature of "profile" incompatible with supertype "NumericStatsMixin"  [override]
> dataprofiler/profilers/int_column_profile.py:95: error: Signature of "profile" incompatible with supertype "NumericStatsMixin"  [override]

4. Updated version.py

• Incremented the version in the version.py file to reflect the new changes introduced in this PR.

[armaan-dhillon]

LGTM, just left one comment to understand the typing changes