1085 Ecocounter Open Data Schema+DAG+QC plots

[gabrielwol]

What this pull request accomplishes:

• New Monthly DAG to insert/download Ecocounter data. Includes prompts to review data and check if data for all days are present. @chmnata

  • dags/ecocounter_open_data.py

• Open Data tables + functions @A-DUYVESTYN

  • open_data_daily_counts + fn open_data_daily_counts_insert , open_data_raw_counts + fn open_data_raw_counts_insert, open_data_sites

• Calibration factor supporting tables @A-DUYVESTYN

  • volumes/ecocounter/tables/sensitivity_history.sql: format sensitivity history for ease of use. We should remove sensitivity_changes when we are satisfied with changes.
  • volumes/ecocounter/views/create-view-calibration_factors.sql: merge sensitivity changes with validation studies to identify when to apply calibration factors
  • volumes/ecocounter/views/create-view-counts.sql: apply calibration factors

Does not need significant review:

• R plots

  • Interactive anomalous range ID and QC plots: volumes/ecocounter/qc/qc_shiny_app.r + qc_graph_volumes postgres function
  • Output static PDF plots: volumes/ecocounter/qc/qc_plots_pdf.r (could be useful base in the future if we want to automatically certain output QC plots each month).

• Misc.

  • Add counter to sites table (commonly used ID field in Ecocounter communications/dashboard)
    • updated ecocounter_pull.py, pull_data_from_api.py.
  • Add direction_main field to flows: easily group dominant/contraflow together
  • Fixed anomalous_range join conditions (Use only one of site_id and flow_id)
    • counts, counts_calibrated

Issue(s) this solves:

• Ecocounter: develop sensitivity/factor tables #1085

What, in particular, needs to reviewed:

What needs to be done by a sysadmin after this PR is merged

• Delete counts_old (saved the old version of counts for monitoring project dependency)
• Delete discontinuities table

[radumas]

Only commenting on the OpenData schema.

Looking pretty solid.

To summarize some change requests made elsewhere:

• There should be README updates in here for documentation relevant to the unit.
• the VIEWs the DAG are pulling from could be in open_data and named to include Miovision data. When Miovision is ready to publish, we can replace the VIEWs to have those UNIONs without having to change the DAG
• the unit of location should be the linear direction (directional centreline) to
  • handle technologies having different centrelines for each direction, or
  • handle one side of the street being faulty; and,
  • avoid the public having to join multiple location tables.
  • And we don't expose the "parent" site/sensor to the public.

[chmnata]

did we add this for mapped task naming? It was using ds before right?

[gabrielwol]

Yes, the insert_and_download_data task_group is mapped over the output of get_years. The purpose is to run the exports for last two months (which may be two separate years), since new data frequently arrives within ~30 days.

[A-DUYVESTYN]

OK to remove sensitivity_changes since it's superseded by sensitivity_history.
Not sure why flow_id = 101042942 has a stray date_range = (,) .
The readme should mention:

• sensitivity_history is manually updated based on emails with with Eco-counter Technical Support Specialist (Derek Yates as of 2024).
• what the numbered settings mean (least to most selective, or not standardized)
• Eco-counter must confirm what the current and new settings are whenever a sensitivity is adjusted, along with the date the change is applied.

[gabrielwol]

OK to remove sensitivity_changes since it's superseded by sensitivity_history.

✅

Not sure why flow_id = 101042942 has a stray date_range = (,) .

That flow_id never had any data. Deleted the entry.

The readme should mention...

Adding!

[A-DUYVESTYN]

How to separate scooter flows calibration factors from bike flows? Currently we apply the same "total volume" factor to both, but we will have separate validation factors for these veh types soon. (Not a blocker for this release)
Readme should state:

• ecocounter.calibration_factors excludes factors in ecocounter.manual_counts_info with flagged as do_not_use= true . do_not_use is entered manually on a case-by-case basis depending on which calibration study is the 'best'.
• Caution is required when assigning do_not_use to calibration studies. Avoid as much as possible applying a new factor to data that has already been calibrated and published to Open Data.

[gabrielwol]

The way to separate scooter calibration factors would be to edit ecocounter.validation_results, particularly the flow_ids CTE to link different flow_ids to those studies.

[A-DUYVESTYN]

@gabrielwol, I'm not finding this file that my review was requested for: volumes/ecocounter/views/create-view-counts_calibrated.sql

[gabrielwol]

@gabrielwol, I'm not finding this file that my review was requested for: volumes/ecocounter/views/create-view-counts_calibrated.sql

Renamed to "counts" from counts_calibrated

[A-DUYVESTYN]

@gabrielwol, I'm not finding this file that my review was requested for: volumes/ecocounter/views/create-view-counts_calibrated.sql

Renamed to "counts" from counts_calibrated

In bigdata, I see both VIEWs counts_calibrated and counts. Is the intention to only have counts which would contain calibrated counts? And to get raw data, use table counts_unfiltered?

[gabrielwol]

In bigdata, I see both VIEWs counts_calibrated and counts. Is the intention to only have counts which would contain calibrated counts? And to get raw data, use table counts_unfiltered?

@A-DUYVESTYN yes, we would keep only counts and counts_unfiltered.

• counts contains both raw and calibrated volumes, and omits anomalous ranges and unvalidated flows/sitese.
• counts_unfiltered contains all raw data.

Will work on readmes today.

[A-DUYVESTYN]

OK. I'm not clear on process for assigning flows_unfiltered.validated and sites_unfiltered.validated. Another one for documentation.