Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Verbose logs #2

Open
ludovicc opened this issue Oct 13, 2017 · 0 comments
Open

Verbose logs #2

ludovicc opened this issue Oct 13, 2017 · 0 comments
Assignees
@ludovicc

Comments

@ludovicc
Copy link
Contributor

Logs are too verbose during initialisation of df_pipeline. We should do something about it:

[2017-10-13 12:40:17,157] {models.py:168} INFO - Filling up the DagBag from /opt/airflow-scripts/data-factory-airflow-dags/df_pipelines_init.py
[2017-10-13 12:40:17,876] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_failed_processing
[2017-10-13 12:40:17,889] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_skipped_processing
[2017-10-13 12:40:17,901] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_successful_processing
[2017-10-13 12:40:17,913] {configuration.py:609} WARNING - section/key [data-factory:clm:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:17,913] {configuration.py:609} WARNING - section/key [data-factory:clm:reorganisation:dicom_reorganise/output_folder] not found in config
[2017-10-13 12:40:17,916] {df_pipelines_init.py:31} INFO - Add DAG clm_reorganise_files
[2017-10-13 12:40:17,929] {df_pipelines_init.py:31} INFO - Add DAG clm_reorganisation_scan_input_folder
[2017-10-13 12:40:17,929] {df_pipelines_init.py:76} INFO - Create pipelines for dataset CHUV CLM using scannners [''] and pipelines ['neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:17,941] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:17,941] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:17,941] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:17,943] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:17,945] {df_pipelines_init.py:31} INFO - Add DAG clm_pre_process_images
[2017-10-13 12:40:17,958] {configuration.py:609} WARNING - section/key [data-factory:ppmi:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:17,961] {df_pipelines_init.py:31} INFO - Add DAG ppmi_reorganise_files
[2017-10-13 12:40:17,977] {df_pipelines_init.py:31} INFO - Add DAG ppmi_reorganisation_scan_input_folder
[2017-10-13 12:40:17,977] {df_pipelines_init.py:76} INFO - Create pipelines for dataset PPMI using scannners [''] and pipelines ['dicom_to_nifti', 'neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:17,993] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:17,994] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:17,994] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:17,996] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:17,999] {df_pipelines_init.py:31} INFO - Add DAG ppmi_pre_process_images
[2017-10-13 12:40:18,013] {df_pipelines_init.py:31} INFO - Add DAG ppmi_ehr_to_i2b2
[2017-10-13 12:40:18,025] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:18,027] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_reorganise_files
[2017-10-13 12:40:18,040] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_reorganisation_scan_input_folder
[2017-10-13 12:40:18,040] {df_pipelines_init.py:76} INFO - Create pipelines for dataset Ng sample using scannners [''] and pipelines ['dicom_to_nifti', 'neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:18,052] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:18,052] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:18,052] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:18,055] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:18,057] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_pre_process_images
[2017-10-13 12:40:18,101] {base_task_runner.py:112} INFO - Running: ['bash', '-c', 'airflow run ng_sample_reorganisation_scan_input_folder scan_dirs 2017-10-13T12:00:00 --job_id 4085 --raw -sd DAGS_FOLDER/df_pipelines_init.py']
[2017-10-13 12:40:18,555] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:18,555] {__init__.py:57} INFO - Using executor LocalExecutor
[2017-10-13 12:40:18,626] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:18,626] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
[2017-10-13 12:40:18,647] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:18,647] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
[2017-10-13 12:40:19,021] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,021] {models.py:168} INFO - Filling up the DagBag from /opt/airflow-scripts/data-factory-airflow-dags/df_pipelines_init.py
[2017-10-13 12:40:19,884] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,883] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_failed_processing
[2017-10-13 12:40:19,897] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,897] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_skipped_processing
[2017-10-13 12:40:19,910] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,910] {df_pipelines_init.py:31} INFO - Add DAG mri_notify_successful_processing
[2017-10-13 12:40:19,924] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,924] {configuration.py:609} WARNING - section/key [data-factory:clm:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:19,924] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,924] {configuration.py:609} WARNING - section/key [data-factory:clm:reorganisation:dicom_reorganise/output_folder] not found in config
[2017-10-13 12:40:19,927] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,927] {df_pipelines_init.py:31} INFO - Add DAG clm_reorganise_files
[2017-10-13 12:40:19,942] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,941] {df_pipelines_init.py:31} INFO - Add DAG clm_reorganisation_scan_input_folder
[2017-10-13 12:40:19,942] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,942] {df_pipelines_init.py:76} INFO - Create pipelines for dataset CHUV CLM using scannners [''] and pipelines ['neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:19,955] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,955] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:19,955] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,955] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:19,955] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,955] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:19,958] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,958] {configuration.py:609} WARNING - section/key [data-factory:clm:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:19,960] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,960] {df_pipelines_init.py:31} INFO - Add DAG clm_pre_process_images
[2017-10-13 12:40:19,974] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,974] {configuration.py:609} WARNING - section/key [data-factory:ppmi:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:19,977] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,977] {df_pipelines_init.py:31} INFO - Add DAG ppmi_reorganise_files
[2017-10-13 12:40:19,991] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,991] {df_pipelines_init.py:31} INFO - Add DAG ppmi_reorganisation_scan_input_folder
[2017-10-13 12:40:19,991] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:19,991] {df_pipelines_init.py:76} INFO - Create pipelines for dataset PPMI using scannners [''] and pipelines ['dicom_to_nifti', 'neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:20,005] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,005] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:20,005] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,005] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:20,005] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,005] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:20,008] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,008] {configuration.py:609} WARNING - section/key [data-factory:ppmi:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:20,011] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,011] {df_pipelines_init.py:31} INFO - Add DAG ppmi_pre_process_images
[2017-10-13 12:40:20,026] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,026] {df_pipelines_init.py:31} INFO - Add DAG ppmi_ehr_to_i2b2
[2017-10-13 12:40:20,039] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,039] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:reorganisation:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:20,042] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,042] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_reorganise_files
[2017-10-13 12:40:20,055] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,055] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_reorganisation_scan_input_folder
[2017-10-13 12:40:20,056] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,055] {df_pipelines_init.py:76} INFO - Create pipelines for dataset Ng sample using scannners [''] and pipelines ['dicom_to_nifti', 'neuro_morphometric_atlas', 'export_features', 'catalog_to_i2b2']
[2017-10-13 12:40:20,069] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,069] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:copy_to_local/output_folder] not found in config
[2017-10-13 12:40:20,069] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,069] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:dicom_to_nitfi/output_folder] not found in config
[2017-10-13 12:40:20,069] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,069] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:mpm_maps/output_folder] not found in config
[2017-10-13 12:40:20,072] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,072] {configuration.py:609} WARNING - section/key [data-factory:ng_sample:preprocessing:mpm_maps/pipeline_path] not found in config
[2017-10-13 12:40:20,075] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,074] {df_pipelines_init.py:31} INFO - Add DAG ng_sample_pre_process_images
[2017-10-13 12:40:20,112] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,112] {models.py:1128} INFO - Dependencies all met for <TaskInstance: ng_sample_reorganisation_scan_input_folder.scan_dirs 2017-10-13 12:00:00 [queued]>
[2017-10-13 12:40:20,114] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,114] {models.py:1128} INFO - Dependencies all met for <TaskInstance: ng_sample_reorganisation_scan_input_folder.scan_dirs 2017-10-13 12:00:00 [queued]>
[2017-10-13 12:40:20,114] {base_task_runner.py:95} INFO - Subtask: [2017-10-13 12:40:20,114] {models.py:1334} INFO -
@ludovicc ludovicc self-assigned this Oct 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ludovicc ludovicc

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ludovicc