import resuable modules across notebooks

[oindrillac]

Related Issues and Dependencies

Resolves #98

This introduces a breaking change

• Yes
• No

This Pull Request implements

In this PR, I have removed the duplicate function definitions across notebooks to improve maintainability of code.
We are now importing the following functions/modules across notebooks keeping the main function definition only in the primary notebook.

• decode_run_length()
• calc_flakiness_score()
• calc_consecutive_failures()
• calc_flake_edges()
• flake_edge_end()
• calc_optimal_flakiness_score()

We are using the .ipynb function to import modules from python notebooks similar to scripts.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[sesheta]

Pre-Commit Test failed! Click here

[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pycqa/pydocstyle.git.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/tomcatling/black-nb.
[INFO] Initializing environment for https://github.com/s-weigand/flake8-nb.
[INFO] Initializing environment for https://github.com/s-weigand/flake8-nb:pep8-naming.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pycqa/pydocstyle.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/tomcatling/black-nb.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/s-weigand/flake8-nb.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Tests should end in _test.py.........................(no files to check)Skipped
check BOM - deprecated: use fix-byte-order-marker........................Passed
Check for case conflicts.................................................Passed
Check docstring is first.................................................Passed
Check JSON...............................................................Passed
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Check python ast.........................................................Passed
Debug Statements (Python)................................................Passed
pydocstyle...............................................................Passed
Check Toml...............................................................Passed
Check Yaml...............................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
mypy.....................................................................Passed
black....................................................................Passed
black-nb.................................................................Failed
- hook id: black-nb
- files were modified by this hook

13 cells left unchanged.
    26 cells left unchanged.
    74 cells left unchanged.
reformatted notebooks/failure-type-classification/background/testgrid_flakiness_detection.ipynb
    1 cell reformatted, 34 cells left unchanged.
All done!
1 file reformatted, 3 files left unchanged.
    23 cells left unchanged.
    27 cells left unchanged.
reformatted notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb
    1 cell reformatted, 37 cells left unchanged.
reformatted notebooks/failure-type-classification/stage/failure_type_classifier.ipynb
    4 cells reformatted, 20 cells left unchanged.
All done!
2 files reformatted, 2 files left unchanged.

flake8-nb................................................................Failed
- hook id: flake8-nb
- exit code: 1

notebooks/failure-type-classification/background/testgrid_flakiness_detection.ipynb#In[40]:1:1: E402 module level import not at top of file
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[2]:2:1: F401 'ipynb' imported but unused
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[4]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[2]:4:1: F401 'ipynb' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_flakiness_score' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_consecutive_failures' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_flake_edges' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.flake_edge_end' imported but unused
notebooks/data-sources/TestGrid/testgrid_indepth_EDA.ipynb#In[1]:1:1: F401 'ipynb' imported but unused
notebooks/data-sources/TestGrid/testgrid_indepth_EDA.ipynb#In[2]:1:1: E402 module level import not at top of file

[oindrillac]

This is the result of the initial pre-commit test which caught some issues which slipped my attention.

notebooks/failure-type-classification/background/testgrid_flakiness_detection.ipynb#In[40]:1:1: E402 module level import not at top of file
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[2]:2:1: F401 'ipynb' imported but unused
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[4]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[2]:4:1: F401 'ipynb' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_flakiness_score' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_consecutive_failures' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.calc_flake_edges' imported but unused
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[4]:2:1: F401 'ipynb.fs.defs.testgrid_flakiness_detection.flake_edge_end' imported but unused
notebooks/data-sources/TestGrid/testgrid_indepth_EDA.ipynb#In[1]:1:1: F401 'ipynb' imported but unused
notebooks/data-sources/TestGrid/testgrid_indepth_EDA.ipynb#In[2]:1:1: E402 module level import not at top of file

• the function imports which were imported but unused were originally imported since they were being used in another function which was also a duplicate function. So importing that eliminated the need of importing the child functions, so i removed them
• seems like an explicit import of the ipynb package is not needed for the import statement, so removed that as well

However, after making the above changes, and fixing the issues, this is the current status of the pre-commit test:

notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[32]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[32]:2:1: E402 module level import not at top of file
notebooks/failure-type-classification/background/testgrid_flakiness_detection.ipynb#In[40]:1:1: E402 module level import not at top of file
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[4]:1:1: E402 module level import not at top of file

this is catching the following import statement from ipynb.fs.defs.testgrid_EDA import decode_run_length which is below the line where I set the path to the notebook to be imported which is causing the test to fail.

Any recommendations on how to set the path before the import and yet have the import statement at the top of the notebook?

cc: @MichaelClifford

[Shreyanand]

Maybe this can help: https://stackoverflow.com/questions/36827962/pep8-import-not-at-top-of-file-with-sys-path

Easiest would be this answer https://stackoverflow.com/a/38338146

[sesheta]

Pre-Commit Test failed! Click here

[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pycqa/pydocstyle.git.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/tomcatling/black-nb.
[INFO] Initializing environment for https://github.com/s-weigand/flake8-nb.
[INFO] Initializing environment for https://github.com/s-weigand/flake8-nb:pep8-naming.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pycqa/pydocstyle.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/tomcatling/black-nb.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/s-weigand/flake8-nb.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Tests should end in _test.py.........................(no files to check)Skipped
check BOM - deprecated: use fix-byte-order-marker........................Passed
Check for case conflicts.................................................Passed
Check docstring is first.................................................Passed
Check JSON...............................................................Passed
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Check python ast.........................................................Passed
Debug Statements (Python)................................................Passed
pydocstyle...............................................................Passed
Check Toml...............................................................Passed
Check Yaml...............................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
mypy.....................................................................Passed
black....................................................................Passed
black-nb.................................................................Passed
flake8-nb................................................................Failed
- hook id: flake8-nb
- exit code: 1

notebooks/failure-type-classification/background/testgrid_flakiness_detection.ipynb#In[40]:1:1: E402 module level import not at top of file
notebooks/data-sources/Sippy/stage/sippy-analysis.ipynb#In[4]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[32]:1:1: E402 module level import not at top of file
notebooks/failure-type-classification/stage/failure_type_classifier.ipynb#In[32]:2:1: E402 module level import not at top of file

[oindrillac]

Maybe this can help: https://stackoverflow.com/questions/36827962/pep8-import-not-at-top-of-file-with-sys-path

Easiest would be this answer https://stackoverflow.com/a/38338146

Thanks @Shreyanand. That seemed to ignore that particular line when i ran it locally :)

[MichaelClifford]

👍 Nice Notebook refactoring