Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate ImagingUpload and MriUploadDB to the new database abstraction #1224

Merged
merged 4 commits into from
Feb 19, 2025

Conversation

maximemulder
Copy link
Contributor

@maximemulder maximemulder commented Jan 8, 2025

Migrate ImagingUpload and MriUploadDB to the new database abstraction.

The current input checking is very complex and this should simplify it a little. The previous code seems to sometimes accept or forget to check if the DICOM archive or MRI upload is null in a few specific places, but I do not see how it can work as a whole in these conditions. I therefore assume the DICOM archive and MRI upload must not be null.

The integration test will check if this works broadly, but since this touches the argument validation, I will need to test the pipeline manually with:

  • ✅ Correct DICOM archive path and correct MRI upload ID.
  • ✅ Correct DICOM archive path.
  • ✅ Correct MRI upload ID.
  • ✅ Incorrect DICOM archive path.
  • ✅ Incorrect MRI upload ID.
  • ✅ Correct DICOM archive path but incorrect MRI upload ID.
  • ✅ Correct MRI upload ID but incorrect DICOM archive path.
  • ✅ Correct DICOM archive path but no associated MRI upload.
  • ✅ Correct MRI upload ID but no associated DICOM archive.
  • ✅ No DICOM archive path or MRI upload ID.

Tested and there does not seem to be any regression: run_dicom_archive_loader.py expects either (exclusive) --upload_id or --tarchive_path, and requires the DICOM archive to be associated with exactly one MRI upload.

@maximemulder maximemulder changed the title Migrate ImagingUpload and MriUploadDB to sqlalchemy Migrate ImagingUpload and MriUploadDB to the new database abstraction Jan 8, 2025
@maximemulder maximemulder force-pushed the sqlalchemy_mri_upload branch 4 times, most recently from 2b5499e to c26a276 Compare January 9, 2025 00:24
@maximemulder maximemulder marked this pull request as ready for review February 14, 2025 01:58
@maximemulder maximemulder added Area: BIDS PR or issue related to the handling of BIDS datasets Area: ORM PR or issue related to the SQLAlchemy integration Difficulty: Medium PR or issue that require a moderate effort or expertise to implement, review, or test labels Feb 14, 2025
Copy link
Collaborator

@cmadjar cmadjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maximemulder very nice!! A few minor suggestions/questions. I did not test it yet.

use dicom archive id instead of tarchiveid because it is prettier

ruff check
@cmadjar cmadjar merged commit ba8ca1d into aces:main Feb 19, 2025
9 checks passed
@cmadjar cmadjar added this to the 27.0 milestone Feb 26, 2025
cmadjar added a commit that referenced this pull request Mar 21, 2025
* remove trailing whitespaces (#1149)

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* SQLAlchemy proof-of-concept (#1152)

* add sqlalchemy

* fix lints

* add comment

---------

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* fix (#1168)

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* integration test (#1138)

* test

* main Loris here

* ignore Loris python

* clone Loris

* rm loris

* override test

* override test

* override RB data

* fix folder

* test php files

* test

* try

* test

* try install loris mri

* os

* test

* rm modules

* Update flake8_python_linter.yml

* Small linting improvements (#1167)

* better linting

* remove tab in comment

---------

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* Improve Python subject determination and validation error handling (#1146)

* improve subjects determination and validation

* use tuple for database query arguments

* query on single line

---------

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* Add literal regex leading r (#1176)

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* fix (#1178)

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* LORIS-MRI tooling configuration (#1170)

* add pyproject.toml

* keep flake8 (for now)

* double type checking

* Use Pyright as main type checker

* remove comment

* test

* test

* change configuration

* add python environment variables

* try factorization

* test

* test

* test

* test

* remove temporary things

* test ruff error

* ruff output github

* further test errors

* test pyright output github

* test

* test

* test

* change comment

* remove test errors

* test

---------

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* update docker install (#1171)

Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>

* add RB override tables for the MRI side (#1173)

* Remove php 8.1 and 8.2 from the LORIS tests to keep only test on PHP 8.3 (#1175)

* remove php 8.1 and 8.2 testing

* update to 8.3 in Dockerfile.test.php8

* ADD minc tools install in gitactions (#1179)

* todo install loris mri

* test

* run pytest in gitaction (#1172)

* test

* ttt

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* test

* Delete test_example.py

* rename

* done

* Delete test/Dockerfile.test.py

* test (#1187)

* unit tests poc (#1188)

* add more pytest global rules (#1194)

* Stricter linter configuration (#1193)

* stricter ruff config

* do not ignore N818

* lint for sorted imports

* add python scripts directory (#1196)

* Bunch of SQLAlchemy definitions (#1192)

* bunch of sqlalchemy definitions

* add project orm definition

* fix candidate to site query function

* optimize docker image (#1195)

* fix pytest config (#1197)

* fix python scripts (#1199)

* remove mysql-connector (#1200)

* Improve subject configuration structure (#1150)

* refacor subject config

* remove suspicious phantom code

* change naming

* fix rebase lints

* cleaner database url

* Cecile CandID path join bug fix

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

---------

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

* Integration test to check that the ORM definitions match the LORIS SQL database (#1198)

* orm sync integration test

* add docker health check

* use integration config file

* test error

* Revert "test error"

This reverts commit d1a70e6.

* Clean up Docker database scripts (#1202)

* clean docker database

* remove docker warning

* remove unused attribute (#1201)

* try fix perl ci

* Add file, file_parameter and scanner SQLAlchemy models (#1207)

* add tables

* rename file model

* rename parameter_file

* fix lints (#1210)

* Unify LORIS-MRI logging (#1191)

* new env

* change print order in database log

* create notification type if needed

* fix typing pyright update

* Integration test for `run_dicom_archive_loader.py` (#1203)

add integration test

put keys in github repo

debugging

add database check

add comments and debugging

remove s3 keys

change converter in test

check file tree

fix doc

fix docs 2

rename variable

go back to using s3 keys

fix path

try fix

* rename database sub-namepaces (#1214)

* fix warnings (#1215)

* Factorize get subject session (#1190)

* factorize subject session

* re-add site log

* Migrate "get all sites from the database" to the SQLAlchemy abstraction (#1216)

* Fix new PEP585 Ruff lint (#1218)

* Clean-up database scan type (#1139)

Complementary PR of LORIS #9304

* Migrate DICOM archive files to new database abstraction (#1217)

* wip

* use self.dicom_archive.id instead of self.tarchive_id

* fix todo

* Fix SQLAlchemy file model (#1221)

* Improve Python requirements order (#1222)

* add python documentation (#1223)

* Update SQLAlchemy bindings to match LORIS#9556 (#1229)

* trigger_tests

* try update

* fix old queries for integration tests

* Migrate `ImagingUpload` and `MriUploadDB` to the new database abstraction (#1224)

* migrate mri upload to sqlalchemy

* fix concurrency bug

* fix clean up

* display mri upload ids

use dicom archive id instead of tarchiveid because it is prettier

ruff check

* migrate scanner to new abstraction (#1232)

* Fix install script (#1234)

* fix install script and move files related to install into the install directory

* fix install script and move files related to install into the install directory

* fix install script and move files related to install into the install directory

* fix paths in imaging_install_test.sh

* fix paths in imaging_install_test.sh

* fix SQL error in install script

* fix tests

* fix tests

* fix github action

* add cpanfile

* copy cpanfile in MRI dockerfile

* add new dependency in order to be able to source the Digest::BLAKE2 in the cpanfile

* fix Digest::BLAKE2 installation though cpanfile

* move config templates to a templates directory and requirements files into a requirements directory

* remove copy

* fix test

* fix test

* fix test

* fix test

* Maxime's feedback

* specify python version (#1237)

* fix large file hash memory (#1236)

* add lib.util module (#1238)

* New DICOM study import script (#1117)

* rewrite dicom archive

* update

* remove --year and --target options

* get session from config file

* remove unused upload argument

* fix little oopsie for --session

* handle large files hash

* fix unhandled modalities

* use new lib.util module

* Add support to be able to read headers from enhanced DICOMs  (#7)

* print

* print

* print

* print

* add support for enhanced DICOMs

* add support for enhanced DICOMs

* rework associate files with series

* sort dicom series and files before inserting in the database

* handle null scanner

---------

Co-authored-by: Cécile Madjar <cecile.madjar@mcin.ca>

* refactoring of CandID on the non-ORM python side

* tests

* fix insertion in mri_protocol_violation_scans

* alphabetical order is hard sometimes

* FIx `get_candidate_id` documentation

---------

Co-authored-by: Maxime Mulder <maxime-mulder@outlook.com>
Co-authored-by: Maxime Mulder <maxime.mulder@mcgill.ca>
Co-authored-by: Shen <kongtiaowangshen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: BIDS PR or issue related to the handling of BIDS datasets Area: ORM PR or issue related to the SQLAlchemy integration Difficulty: Medium PR or issue that require a moderate effort or expertise to implement, review, or test Refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants