Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[instrument_manager] Parse Instrument Data from CSV file #9472

Open
wants to merge 25 commits into
base: main
Choose a base branch
from

Conversation

jeffersoncasimir
Copy link
Contributor

@jeffersoncasimir jeffersoncasimir commented Nov 15, 2024

Closes #9341

Brief summary of changes

Contains a new CSVParser class, extended by an InstrumentDataParser class.

A template file with the expected headers for the selected instrument can be downloaded from the new "Upload Instrument Data" panel's form in the Upload tab of the Instrument Manager module.

The uploaded .csv file will be saved and tracked in a new table called instrument_data_files.

The file's headers must match the selected instrument's expected headers.

For each row, an instrument is created using the NDB_BVL_Instrument::factory and is then saved to the database.

If an error is detected, the database transaction will be rolled back, and the list of errors will be returned to the user.

If the uploaded file exceeds a certain size (currently hard-coded to 512 MB), the task will be performed in the background as a new server_process_manager type called parse_instrument_data. This process runs the parse_instrument_data.php script. It is expected that the monitor_instrument_data.php script is run periodically to check the status of running tasks. When a task is done executing, the server_process_manager table is uploaded and the user who uploaded the file receives an email containing the "computed exit text".

  • Documentation updated

Testing instructions

  1. Select an instrument from the dropdown
  2. Download the headers template
  3. Manually add rows to the template and upload it
  4. Ensure it works as expected for files exceeding MAX_FILE_BYTES
  5. Ensure MriUploadServerProcess still works as expected

Screenshots:
Screenshot 2025-02-05 at 9 32 24 AM
Screenshot 2025-02-05 at 9 33 06 AM
Screenshot 2025-02-05 at 9 33 27 AM
Screenshot 2025-02-05 at 9 37 17 AM
Screenshot 2025-02-05 at 9 34 20 AM
Screenshot 2025-02-05 at 9 35 11 AM

Still unresolved:

  • one php lint issue
  • examiner is always uploader

@skarya22
Copy link
Contributor

From Nov 19 LORIS call:

  • Move data upload to Browse instead of Upload
  • Should store the CSV somewhere that was uploaded
  • Change from saveValues to save so that there is validation
  • TODO: Discuss further about what the behaviour should be when data is not validated correctly. Don't upload anything, or upload all without errors

@jeffersoncasimir jeffersoncasimir force-pushed the 2024_10_28_instrument_data_parser branch from b50171d to 73f26b9 Compare February 3, 2025 22:45
@jeffersoncasimir jeffersoncasimir marked this pull request as ready for review February 5, 2025 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generalized CSV importer for instrument data
2 participants