Merge branch 'deploy' into maplibre

README.md

@@ -8,20 +8,26 @@ The Travel Time Request App is a simple web application designed to help City st
 This app was originally developed as a [class project by U of T students](https://www.youtube.com/watch?v=y6lnefduogo) in partnership with the City, though it has undergone substantial development by the Data & Analytics Unit since then. 
 
 ## How to use the app
-When you [visit the app](https://trans-bdit.intra.prod-toronto.ca/traveltime-request/), you will be prompted to add/create at least one of each of the following:
-* a corridor, drawn on the map
-* a time range, given in hours of the day, 00 - 23
-* a date range (note that the end of the date range is exclusive)
-* a day of week selection
-* a selection of whether or not to include statutory holidays 
 
-The app will combine these factors together to request travel times for all possible combinations. If one of each type of factor is selected, only a single travel time will be estimated with the given parameters. 
+### Via the front-end user-interface 
+
+When you [visit the app](https://trans-bdit.intra.prod-toronto.ca/traveltime-request/), you will be prompted to create at least one of each of the following factors:
+
+| Factor | Description |
+| ----- | ----------- |
+| Corridor | Drawn on the map, it is a shortest path between two intersections of your choice. Draw it in both directions if you need both directions of travel. |
+| Time Range | Times must start and end on the hour and the app accepts integer values between 0 and 24. The final hour is _exclusive_, meaning that a range of 7am to 9am covers two hours, not three. Values of 0 and 24 both interchangeably represent midnight; a time range of 0 - 24 will return all hours of the day. A time range starting after it ends (e.g. 10pm to 4am) will wrap around midnight[^1]. |
+| Date Range | Use the calendar widget to select a date range. Note that selected ranges are displayed with an exclusive end date. |
+| Day of Week | Identify the days of week to include in the aggregation. |
+| Holiday Inclusion | Decide whether to include or exclude Ontario's statutory holidays if applicable. You can also opt to do it both ways. |
+
+The app will combine these factors together to request travel times for all valid combinations. If one of each type of factor is selected, only a single travel time will be estimated with the given parameters. 
 
 Once each factor type has been validly entered it will turn from red to green. Once one or more of each type of factor is ready, a button will appear allowing you to submit the query. Once the data is returned from the server (this can take a while when there are many combinations to process) you will be prompted to download the data as either CSV or JSON.
 
 If you have any trouble using the app, please send an email to Nate Wessel ([email protected]) or feel free to open an issue in this repository if you are at all familiar with that process.
 
-## Outputs
+#### Outputs
 
 The app can return results in either CSV or JSON format. The fields in either case are the same:
 
@@ -40,10 +46,15 @@ The app can return results in either CSV or JSON format. The fields in either ca
 | `mean_travel_time_minutes` | The mean travel time in minutes is given as a floating point number rounded to three decimal places. Where insufficient data was available to complete the request, the value will be null. |
 | `mean_travel_time_seconds` | Same as above, but measured in seconds. |
 
+### By querying the back-end API directly
+
+The front-end UI pulls all data from the backend service available at https://trans-bdit.intra.prod-toronto.ca/tt-request-backend/. This API defines several endpoints, all of which return JSON-structured data. Those endpoints are documented at the link above.
+
+Generally, the API returns much more data than is available through the UI and this allows some extended use-cases which are just starting to be documented in the [`analysis/`](./analysis) folder. These may include looking at travel time variability within a given window and conducting statistical comparisons between different time periods.
 
 ## Methodology
 
-Data for travel time estimation through the app are sourced from [HERE](https://github.com/CityofToronto/bdit_data-sources/tree/master/here)'s [traffic API](https://developer.here.com/documentation/traffic-api/api-reference.html) and are available back to about 2017. HERE collects data from motor vehicles that report their speed and position to HERE, most likely as a by-poduct of the driver making use of an in-car navigation system connected to the Internet.
+Data for travel time estimation through the app are sourced from [HERE](https://github.com/CityofToronto/bdit_data-sources/tree/master/here)'s [traffic API](https://developer.here.com/documentation/traffic-api/api-reference.html) and are available back to 2017-09-01. HERE collects data from motor vehicles that report their speed and position to HERE, most likely as a by-poduct of the driver making use of an in-car navigation system connected to the Internet.
 
 The number of vehicles within the City of Toronto reporting their position to HERE in this way has been [estimated](./analysis/total-fleet-size.r) to be around 2,000 to 3,000 vehicles during the AM and PM peak periods, with lower numbers in the off hours. While this may seem like a lot, in practice many of these vehicles are on the highways and the coverage of any particular city street within a several hour time window can be very minimal if not nil. For this reason, we are currently restricting travel time estimates to "arterial" streets and highways.  
 
@@ -59,8 +70,10 @@ We aggregate corridors together spatially as necessary into larger corridors whe
 
 ### Other means of estimating travel times
 
-The City also has [bluetooth sensors](https://github.com/CityofToronto/bdit_data-sources/blob/master/bluetooth/README.md) at some intersections which can be used to get a more reliable measure of travel time. These sensors pick up a much larger proportion of vehicles than the HERE data, making it possible to do a temporally fine-grained analysis. The sensors however are only in a few locations, especially in the downtown core and along the Gardiner and DVP expressways.  
+The City also has [bluetooth sensors](https://github.com/CityofToronto/bdit_data-sources/blob/master/bluetooth/README.md) at some intersections which can be used to get a more reliable measure of travel time. These sensors pick up a much larger proportion of vehicles than the HERE data, making it possible to do a temporally fine-grained analysis. The sensors however are only in a few locations, especially in the downtown core and along the Gardiner and DVP expressways.
 
 ## Development
 
 For information on development and deployment, see [Running the App](./running-the-app.md).
+
+[^1]: Time ranges that wrap midnight will result in some discontinuity of periods because of the interaction with the date range and day-of-week paremeters. For example, if you select the time range `[22,2)` but only have one date within your date range (e.g. `[2024-01-01,2024-01-02)`) then the aggregation will include both the period from `[2024-01-01 00:00:00, 2024-01-01 02:00:00)` and `[2024-01-01 22:00:00, 2024-01-01 00:00:00)`, averaged together. That is, both the early morning and late evening of the same day.

backend/app/getGitHash.py

@@ -0,0 +1,4 @@
+from subprocess import check_output
+
+def getGitHash():
+    return check_output(['git', 'rev-parse', 'HEAD']).decode('ascii').strip()

backend/app/get_centreline_links.py

@@ -0,0 +1,49 @@
+import json
+from app.db import getConnection
+
+links_query = '''
+WITH centreline_path AS (
+    SELECT unnest(links)::int AS centreline_id
+    FROM gis_core.get_centreline_btwn_intersections(
+        %(from_node_id)s,
+        %(to_node_id)s
+    )
+)
+
+SELECT
+    centreline_id,
+    linear_name_full_legal AS st_name,
+    ST_AsGeoJSON(geom) AS geojson,
+    ST_length(ST_Transform(geom, 2952)) AS length_m,
+    from_intersection_id,
+    to_intersection_id
+FROM centreline_path
+JOIN gis_core.centreline_latest USING (centreline_id)
+'''
+
+# returns a json with geometries of links between two nodes
+def get_centreline_links(from_node_id, to_node_id, map_version='23_4'):
+    with getConnection() as connection:
+        with connection.cursor() as cursor:
+            cursor.execute(
+                links_query,
+                {
+                    "from_node_id": from_node_id,
+                    "to_node_id": to_node_id
+                }
+            )
+
+            links = [
+                {
+                    'centreline_id': centreline_id,
+                    'name': st_name,
+                    #'sequence': seq,
+                    'geometry': json.loads(geojson),
+                    'length_m': length_m,
+                    'source': source,
+                    'target': target
+                } for centreline_id, st_name, geojson, length_m, source, target in cursor.fetchall()
+            ]
+
+    connection.close()
+    return links

backend/app/get_links.py → backend/app/get_here_links.py

@@ -28,7 +28,7 @@
 '''
 
 # returns a json with geometries of links between two nodes
-def get_links(from_node_id, to_node_id, map_version='23_4'):
+def get_here_links(from_node_id, to_node_id, map_version='23_4'):
     parsed_links_query = sql.SQL(links_query).format(
         routing_function = sql.Identifier(f'get_links_btwn_nodes_{map_version}'),
         street_geoms_table = sql.Identifier(f'routing_streets_{map_version}'),

backend/app/get_nearest_centreline_node.py

@@ -0,0 +1,48 @@
+from app.db import getConnection
+from json import loads as loadJSON
+
+SQL = '''
+WITH nearest_centreline AS (
+    SELECT
+        intersection_id,
+        geom::geography <-> ST_MakePoint(%(longitude)s, %(latitude)s)::geography AS distance
+    FROM gis_core.intersection_latest
+    ORDER BY geom <-> ST_SetSRID(ST_MakePoint(%(longitude)s, %(latitude)s), 4326) ASC
+    LIMIT 1
+)
+
+SELECT
+    intersection_id AS centreline_id,
+    ST_AsGeoJSON(geom) AS geojson,
+    distance,
+    ARRAY_AGG(DISTINCT linear_name_full_from) AS street_names
+FROM nearest_centreline
+JOIN gis_core.intersection_latest AS ci USING (intersection_id)
+GROUP BY
+    intersection_id,
+    geom,
+    distance
+'''
+
+def get_nearest_centreline_node(longitude, latitude):
+    """
+    Return the nearest node from the latest city centreline network
+
+    arguments:
+    longitude (float): longitude of the point to search around
+    latitude (float): latitude of the point to search around
+    """
+    node = {}
+    with getConnection() as connection:
+        with connection.cursor() as cursor:
+            cursor.execute(SQL, {'longitude': longitude, 'latitude': latitude})
+            centreline_id, geojson, distance, street_names = cursor.fetchone()
+            node = {
+                'centreline_id': centreline_id,
+                'street_names': street_names,
+                'geometry': loadJSON(geojson),
+                'distance': distance
+            }
+    connection.close()
+    return node
+

backend/app/get_node.py

@@ -2,21 +2,23 @@
 
 import json
 from app.db import getConnection
+from app.get_nearest_centreline_node import get_nearest_centreline_node
 
 SQL = '''
 SELECT
-    ST_AsGeoJSON(cg_nodes.geom) AS geom,
+    ST_AsGeoJSON(
+        ST_GeometryN(here_nodes.geom, 1) -- necessary because currently stored as a multi-point
+    ) AS geom,
     array_agg(DISTINCT InitCap(streets.st_name)) FILTER (WHERE streets.st_name IS NOT NULL) AS street_names
-FROM congestion.network_nodes AS cg_nodes
-JOIN here.routing_nodes_21_1 AS here_nodes USING (node_id)
-JOIN here_gis.streets_att_21_1 AS streets USING (link_id)
-WHERE node_id = %(node_id)s
+FROM here.routing_nodes_23_4 AS here_nodes
+JOIN here_gis.streets_att_23_4 AS streets USING (link_id)
+WHERE here_nodes.node_id = %(node_id)s
 GROUP BY
-    node_id,
-    cg_nodes.geom;
+    here_nodes.node_id,
+    here_nodes.geom;
 '''
 
-def get_node(node_id):
+def get_node(node_id, conflate_with_centreline=False):
     node = {}
     with getConnection() as connection:
         with connection.cursor() as cursor:
@@ -27,5 +29,11 @@ def get_node(node_id):
                 'street_names': street_names,
                 'geometry': json.loads(geojson)
             }
+            if conflate_with_centreline:
+                lon = node['geometry']['coordinates'][0]
+                lat = node['geometry']['coordinates'][1]
+                node['conflated'] = {
+                    'centreline': get_nearest_centreline_node(lon, lat)
+                }
     connection.close()
     return node

backend/app/get_travel_time.py

@@ -1,12 +1,15 @@
 """Function for returning data from the aggregate-travel-times/ endpoint"""
 
 from app.db import getConnection
-from app.get_links import get_links
+from app.get_here_links import get_here_links
 from app.selectMapVersion import selectMapVersion
+from traveltimetools.utils import timeFormats
 import numpy
 import math
 import pandas
 import random
+import json
+from app.getGitHash import getGitHash
 
 # the way we currently do it
 def mean_daily_mean(obs):
@@ -19,23 +22,53 @@ def mean_daily_mean(obs):
     # average the days together
     return numpy.mean(daily_means)
 
-def timeFormat(seconds):
-    return {
-        'seconds': round(seconds,3),
-        'minutes': round(seconds/60,3),
-        # format travel times in seconds like a clock for humans to read
-        'clock': f'{math.floor(seconds/3600):02d}:{math.floor((seconds/60)%60):02d}:{round(seconds%60):02d}'
-    }
+def checkCache(uri):
+    query = f'''
+        SELECT results
+        FROM nwessel.cached_travel_times
+        WHERE uri_string = %(uri)s AND commit_hash = %(hash)s
+    '''
+    connection = getConnection()
+    with connection:
+        with connection.cursor() as cursor:
+            try:
+                cursor.execute(query, {'uri': uri, 'hash': getGitHash()})
+                for (record,) in cursor: # will skip if no records
+                    return record # there could only be one
+            except:
+                pass
+
+def cacheAndReturn(obj,uri):
+    query = f'''
+        INSERT INTO nwessel.cached_travel_times (uri_string, commit_hash, results)
+        VALUES (%(uri)s, %(hash)s, %(results)s)
+    '''
+    connection = getConnection()
+    with connection:
+        with connection.cursor() as cursor:
+            try:
+                cursor.execute(query, {'uri': uri, 'hash': getGitHash(), 'results': json.dumps(obj)})
+            finally:
+                return obj
 
 def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_date, include_holidays, dow_list):
     """Function for returning data from the aggregate-travel-times/ endpoint"""
 
+    # first check the cache
+    cacheURI = f'/{start_node}/{end_node}/{start_time}/{end_time}/{start_date}/{end_date}/{str(include_holidays).lower()}/{"".join(map(str,dow_list))}'
+    cachedValue = checkCache(cacheURI)
+    if cachedValue:
+        return cachedValue
+
     holiday_clause = ''
     if not include_holidays:
         holiday_clause = '''AND NOT EXISTS (
             SELECT 1 FROM ref.holiday WHERE ta.dt = holiday.dt
         )'''
 
+    # if end_time is less than the start_time, then we wrap around midnight
+    ToD_and_or = 'AND' if end_time > start_time else 'OR'
+
     query = f'''
         SELECT
             link_dir,
@@ -45,8 +78,10 @@ def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_
         FROM here.ta
         WHERE
             link_dir = ANY(%(link_dir_list)s)
-            AND tod >= %(start_time)s::time
-            AND tod < %(end_time)s::time
+            AND (
+                tod >= %(start_time)s::time
+                {ToD_and_or} tod < %(end_time)s::time
+            )
             AND date_part('ISODOW', dt) = ANY(%(dow_list)s)
             AND dt >= %(start_date)s::date
             AND dt < %(end_date)s::date
@@ -55,7 +90,7 @@ def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_
 
     map_version = selectMapVersion(start_date, end_date)
 
-    links = get_links(
+    links = get_here_links(
         start_node,
         end_node,
         map_version
@@ -114,7 +149,7 @@ def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_
 
     if len(sample) < 1:
         # no travel times or related info to return here
-        return {
+        return cacheAndReturn({
             'results': {
                 'travel_time': None,
                 'observations': [],
@@ -126,7 +161,7 @@ def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_
                 'corridor': {'links': links, 'map_version': map_version},
                 'query_params': query_params
             }
-        }
+        }, cacheURI)
 
     tt_seconds = mean_daily_mean(sample)
 
@@ -140,22 +175,22 @@ def get_travel_time(start_node, end_node, start_time, end_time, start_date, end_
         p95lower, p95upper = numpy.percentile(sample_distribution, [2.5, 97.5])
         reported_intervals = {
             'p=0.95': {
-                'lower': timeFormat(p95lower),
-                'upper': timeFormat(p95upper)
+                'lower': timeFormats(p95lower,1),
+                'upper': timeFormats(p95upper,1)
             }
         }
 
-    return {
+    return cacheAndReturn({
         'results': {
-            'travel_time': timeFormat(tt_seconds),
+            'travel_time': timeFormats(tt_seconds,1),
             'confidence': {
                 'sample': len(sample),
                 'intervals': reported_intervals
             },
-            'observations': [timeFormat(tt) for (dt,tt) in sample]
+            'observations': [timeFormats(tt,1) for (dt,tt) in sample]
         },
         'query': {
             'corridor': {'links': links, 'map_version': map_version},
             'query_params': query_params
         }
-    }
+    },cacheURI)