Provenance tracking

.gitignore

@@ -109,4 +109,10 @@ tmp.py
 pilates_conda_env_setup4hima.txt
 
 *.DS_Store
-.idea
+.idea
+.aider*
+
+.apptainer/
+.singularity/
+
+pilates/database/schema/generated/

README.md

@@ -1,109 +1,174 @@
-<p align="center"><img src="logo_multi.png" width="700"></p>
+<p align="center"><img src="logo_multi.png" width="700" alt="PILATES Logo"></p>
 
-**P**latform for \
-**I**ntegrated \
-**L**anduse \
-**A**nd \
-**T**ransportation \
-**E**xperiments and \
-**S**imulation
+**PILATES** (**P**latform for **I**ntegrated **L**anduse **A**nd **T**ransportation **E**xperiments and **S**imulation) is a framework for orchestrating containerized microsimulation models to study the co-evolution of land use and transportation systems. Designed for long-term regional forecasting, PILATES enables researchers and planners to simulate complex urban dynamics over multi-decade periods by linking specialized models that operate at different time scales.
 
-PILATES is designed to facilitate the integration of various containerized microsimulation applications that together fully model the co-evolution of land use and transportation systems over time for the purpose of long-term regional forecasting.
+Rather than tightly coupling models within a single software process, PILATES orchestrates them in a containerized environment. This modular architecture allows it to leverage the behavioral sophistication of existing specialized models with minimal modification, while providing robust state management and reproducibility.
 
-The PILATES Python library is comprised primarily of the following:
-1. **run.py** -- An executable python script responisble for orchestrating the execution of all containerized applications and data-transformation steps.
-2. **settings.yaml** -- A configuration file for managing and defining high-level system parameters (e.g. geographic region, local data paths, simulation time horizon, etc.)
-3. **application-specific dirs** -- The subdirectories of `pilates/` contain application-specific I/O directories for mounting to their respective container images, as well as application-specific Python modules responsible for transforming and archiving I/O data generated/used by other component applications.
-4. **utils/** -- A subdirectory of `pilates/` containing various Python modules that might be relevant to any or all of the component applications (e.g. common geographic data transformations or http requests to the US Census API.)
+-----
 
+## Integrated Simulation Models
 
-## 1. Setting up your environment
-1. Make sure docker is running on your machine and that you've either pre-downloaded the required container images (specified in `settings.yaml`) or that you've signed into a valid docker account with dockerhub access.
-2. Change other relevant parameters in `settings.yaml` (probably only [L7-31](https://github.com/ual/PILATES/blob/v2/settings.yaml#L7-L30))
-   - UrbanSim settings of note:
-      - `region_to_region_id` must have an entry that corresponds to the name of the input HDF5 datastore (see below)
-   - ActivtySim settings of note:
-      - num_processors: adjust this number to take full advantage of multi-threaded data processing in Python. Number should be close to the total number of virtual CPUs available on your machine (threads X cores x processors per core or something like that).
-      - chunk_size: adjust this number to take full advantage of the available RAM on your machine. Trying making this number bigger until the activitysim container segfaults or is killed due to a memory error.
-4. Make sure your Python environment has `docker-py`, and `pyyaml` installed.
+PILATES integrates several leading microsimulation models to create a comprehensive forecasting platform:
 
-## 2. I/O
-PILATES only needs two local data files in order to run: 1) an archive of land use and population tables corresponding to base year data for the specified region; and 2) a table of base-year travel skims in the format of the specified travel model. Currently, these two files are organized as follows:
-1. **pilates/urbansim/data/custom_mpo_\<xxxxxxxx\>_model_data.h5** - an UrbanSim-formatted HDF5 datastore where `<xxxxxxxx>` is an 8-digit region ID corresponding to one of the IDs in the settings [L40](https://github.com/ual/PILATES/blob/master/settings.yaml#L40).
-2. **pilates/\<travel model\>/\<travel model data dir\>/\<skims filename\>** - the input skims file, where `<skims filename>` is the name of the skims file specified in settings [L30](https://github.com/ual/PILATES/blob/master/settings.yaml#L31). Currently `polaris` and `beam` are the only supported travel models/skim formats.
+  * **[UrbanSim](https://github.com/UDST/urbansim)**: Models long-term metropolitan evolution by simulating household and business location choices, real estate development, and land use changes over periods of years or decades.
 
-With those two files in those two places, PILATES should handle the rest. 
+  * **[ATLAS](https://doi.org/10.1080/03081060.2024.2353784)**: Simulates household vehicle fleet dynamics, including vehicle purchase, replacement, and technology adoption (e.g., electric vs. internal combustion vehicles).
 
-NOTE: currently all input data is overwritten in place throughout the course of a multi-year PILATES run. To avoid data loss please store a local copy of the input data outside of PILATES.
+  * **[ActivitySim](https://github.com/ActivitySim/activitysim)**: Generates daily activity-based travel demand by simulating individual travel decisions including trip purpose, destination, time of day, and mode choice for a synthetic population.
 
-## 3. Executing the full workflow
+  * **[BEAM](https://github.com/LBNL-UCB-STI/beam)**: The **B**ehavior, **E**nergy, **A**utonomy, and **M**obility framework simulates agent-based transportation on detailed road networks, modeling traffic congestion, transit operations, and emerging mobility services to produce network performance metrics.
+
+-----
+
+## Simulation Scenarios
+
+PILATES supports various simulation configurations to match different research and planning needs:
+
+1.  **BEAM Only**: Detailed network performance analysis using fixed travel plans. Ideal for studying infrastructure impacts, operational strategies, or new mobility services.
+
+2.  **ActivitySim + BEAM**: Agent-based travel demand modeling with network feedback. ActivitySim generates daily travel demand, BEAM simulates it on the network, and the resulting travel times feed back until equilibrium is reached.
+
+3.  **UrbanSim + ActivitySim + BEAM**: Long-term land use and transportation co-evolution. UrbanSim simulates multi-year changes, which inform ActivitySim/BEAM travel demand modeling. Changes in accessibility from the transport model inform the next UrbanSim period.
+
+4.  **UrbanSim + ATLAS + ActivitySim + BEAM**: Comprehensive modeling of land use, transportation, and vehicle technology co-evolution, including how system changes influence household vehicle choices and their effects on energy consumption and emissions.
+
+-----
+
+## Key Features
+
+  * **Modular Architecture**: Containerized models (Docker/Singularity) can be mixed and matched to create custom simulation workflows
+  * **Flexible Temporal Coupling**: Configure execution frequency of different models to study various feedback loops and time horizons
+  * **State Management**: Automated tracking and persistence of simulation state across model runs and time steps
+  * **Reproducibility**: Complete provenance tracking of data transformations and model executions
+  * **High-Performance Computing**: Optimized for HPC environments with parallel processing support
+  * **Database Integration**: Optional analytical database backend for improved performance and data management
+
+-----
+
+## Getting Started
+
+### Prerequisites
+
+  * Container runtime: **Docker** or **Singularity**
+  * **Anaconda** or **Miniconda** for Python environment management
+
+### Installation
+
+1.  **Clone the repository:**
+
+    ```bash
+    git clone https://github.com/LBNL-UCB-STI/PILATES.git
+    cd PILATES
+    ```
+
+2.  **Create and activate the conda environment:**
+
+    ```bash
+    conda env create -f environment.yml
+    conda activate pilates
+    ```
+
+3.  **Download input data:**
+
+    Various raw input data files are required for the different models. See [lawrencium-setup.md](lawrencium-setup.md#download-data) for instructions.
+
+### Configuration
+
+1.  Edit `settings.yaml` to configure your simulation:
+    - Set container preference: `container_manager: "docker"` or `"singularity"`
+    - Update region-specific parameters
+    - Adjust computational settings (`num_processors`, `chunk_size`, etc.)
+
+### Running a Simulation
+
+Execute the main script from the root directory. Use the `-p` flag to pull the latest container images before the first run:
+
+```bash
+python run.py -v -p
 ```
-usage: ipython [-v] [-p] [-h HOUSEHOLD_SAMPLE_SIZE] [-s] [-w] [-d DISABLE_MODEL] [-c CONFIG]
-
-optional arguments:
-  -v, --verbose         print docker stdout
-  -p, --pull_latest     pull latest docker images before running
-  -h HOUSEHOLD_SAMPLE_SIZE, --household_sample_size HOUSEHOLD_SAMPLE_SIZE
-                        household sample size (only works if land use models are disables)
-  -s, --static_skims    bypass traffic assignment altogether (i.e. use base year skims for every run)
-  -w, --warm_start_skims
-                        generate full activity plans for the base year only. useful for generating warm start skims.
-  -d DISABLE_MODEL, --disable_model DISABLE_MODEL
-                        "l" for land use, "a" for activity demand, "t" for traffic assignment. Can specify multiple (e.g. "at")
-  -c CONFIG, --config CONFIG
-                        Specify different config .yaml (other than "settings.yaml")
+
+## Advanced Features
+
+### Database Integration
+
+PILATES includes an optional database backend for improved performance and scalability. The database system supports:
+
+- Direct database input to ActivitySim, eliminating expensive H5 preprocessing
+- Dual storage architecture preserving both raw and processed data
+- Parallel data uploads for large-scale simulations
+- Complete data lineage tracking with OpenLineage metadata
+- Automatic schema documentation and data quality validation
+
+For detailed setup and usage instructions, see [docs/database-setup.md](docs/database-setup.md) and [docs/database_documentation_guide.md](docs/database_documentation_guide.md).
+
+### HPC Execution
+
+For high-performance computing environments, specialized scripts are available in the `hpc/` directory. For detailed instructions on setting up PILATES on the Lawrencium cluster, see [lawrencium-setup.md](lawrencium-setup.md).
+
+```bash
+cd hpc
+./job_runner.sh [options]
 ```
 
-## Miscellany
+### Background Execution
 
-### ActivitySim BEAM integration
-In order to have BEAM to run correctly one needs to set the following settings:
+To run PILATES as a background process:
 
-1. **skims_fname**: `gemini/10.activitySimODSkims.UrbanSim.TAZ.Full.csv.gz` The full skim file that contains all Origin Destinations pairs with ActivitySim path types.
-2. **beam_config**: `gemini/activitysim-base-from-60k-input.conf` Path to beam config. This path must be relative to `beam_local_input_folder` and `region`. The BEAM docker container is provided with this config as an input.
-3. **beam_scenario_folder**: `gemini/activitysim-plans-base-2010-cut-60k` Folder with BEAM scenario where ActivitySim output goes. Files from this folder are a scenario input for BEAM.
-4. **beam_local_input_folder**: `pilates/beam/production/` Path to BEAM input folder. This folder is going to be mapped to the BEAM container input folder.
-5. **beam_local_output_folder**: `pilates/beam/beam_output/` The BEAM output is going to be saved here. In order to have a clean run this directory should be empty before start.
+```bash
+nohup python run.py -v &
+```
 
-#### BEAM Config: saves ASIM skims, enables BEAM to reuse the previous BEAM output plans, linkstats.
+Output will be saved to `nohup.out`.
 
-BEAM config should be set in the way so that BEAM saves ActivitySim skims, linkstats and loads people plans and linkstats from the previous runs.
+-----
 
-This is the BEAM config options that enables it.
+## Architecture
 
-```hocon
-# most of the time we need a single iteration
-beam.agentsim.firstIteration = 0
-beam.agentsim.lastIteration = 0
+PILATES uses a consistent preprocessor/runner/postprocessor pattern for model execution, providing a structured approach to data preparation, model execution, and output processing. This pattern enables:
 
-beam.router.skim = {
-  # This allows to write skims on each iteration
-  writeSkimsInterval = 1
-}
+- Modular development and testing of individual components
+- Reusability across different simulation models
+- Clear separation of concerns for maintainability
+- Integrated provenance tracking for reproducibility
 
-beam.exchange{
-  output {
-    # this enables saving activitySim Skims
-    activitySimSkimsEnabled = true
-    # geo level different than TAZ (in beam taz-centers format)
-    geo.filePath = ${beam.inputDirectory}"/block_group-centers.csv.gz"
-  }
-}
+For implementation details and guidance on integrating new models, see [docs/architecture.md](docs/architecture.md).
 
-# This loads linkStats from the last found BEAM runs
-beam.warmStart.type = "linkStatsFromLastRun"
+-----
 
-# For subsequential beam runs (some data will be laoded from the latest found run in this directory)
-beam.input.lastBaseOutputDir = ${beam.outputs.baseOutputDirectory}
-# This prefix is used to find the last run output directory within beam.input.lastBaseOutputDir direcotry
-beam.input.simulationPrefix = ${beam.agentsim.simulationName}
+## Contributing
 
-# fraction of input plans to be merged into the latest output plans (taken from the beam.input.lastBaseOutputDir)
-beam.agentsim.agents.plans.merge.fraction = 0.2
-```
+We welcome contributions! Please use the [GitHub issue tracker](https://github.com/LBNL-UCB-STI/PILATES/issues) for bug reports, feature requests, and support questions.
+
+### How to Contribute
+
+1.  Fork the repository
+2.  Create a feature branch (`git checkout -b feature/my-feature`)
+3.  Commit your changes (`git commit -m 'Add new feature'`)
+4.  Push to the branch (`git push origin feature/my-feature`)
+5.  Open a Pull Request
+
+Please see our (forthcoming) `CONTRIBUTING.md` for detailed guidelines.
+
+-----
 
-#### Executing the simulation
-```shell
-nohup python run.py -v
+## Citation
+
+If you use PILATES in your research, please cite:
+
+```bibtex
+@misc{pilates_2024,
+  author       = {Needell, Zachary and Waddell, Paul and Caicedo, Juan and Laarabi, Haitam and Wang, Yuhan and Poliziani, Cristian and Lazarus, Jessica and Openkov, Dmitrii and Gardner, Max and Rezaei, Nazanin and others},
+  title        = {Platform for Integrated Land use And Transportation Experiments and Simulation (PILATES) v1.0},
+  doi          = {10.11578/dc.20240613.2},
+  url          = {https://www.osti.gov/biblio/2373117},
+  place        = {United States},
+  year         = {2024},
+  month        = {05}
+}
 ```
-nohup keeps the script working in case the user session is closed. The output is saved to nohup.out file by default.
+
+-----
+
+## License
+
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

add_missing_iterations.py

@@ -0,0 +1,89 @@
+import json
+import argparse
+import shutil
+import re
+from pathlib import Path
+
+
+def add_missing_iterations(run_info_path_str: str, no_backup: bool):
+    """
+    Parses short_names in a run_info.json file to add missing 'iteration'
+    and 'sub_iteration' keys to file records.
+    """
+    run_info_path = Path(run_info_path_str)
+
+    if not run_info_path.exists():
+        print(f"Error: File not found at {run_info_path}")
+        return
+
+    if not no_backup:
+        backup_path = run_info_path.with_suffix(run_info_path.suffix + ".bak")
+        print(f"Creating backup at {backup_path}")
+        shutil.copy(run_info_path, backup_path)
+
+    try:
+        with open(run_info_path, "r") as f:
+            data = json.load(f)
+    except json.JSONDecodeError as e:
+        print(f"Error reading JSON file: {e}")
+        return
+
+    if "file_records" not in data:
+        print("Error: 'file_records' key not found in JSON data.")
+        return
+
+    updated_count = 0
+    # Regex to find an iteration number at the end of a short_name, e.g., "name_2018_1" -> "1"
+    iter_pattern = re.compile(r"_(\d+)$")
+
+    for record in data["file_records"].values():
+        # Check if iteration is missing or null
+        if record.get("iteration") is None:
+            short_name = record.get("short_name", "")
+            match = iter_pattern.search(short_name)
+
+            if match:
+                iteration_num = int(match.group(1))
+                record["iteration"] = iteration_num
+
+                # Also ensure sub_iteration exists
+                if record.get("sub_iteration") is None:
+                    record["sub_iteration"] = 0
+
+                updated_count += 1
+                print(
+                    f"Patched record '{short_name}': set iteration to {iteration_num}"
+                )
+
+    if updated_count > 0:
+        try:
+            with open(run_info_path, "w") as f:
+                json.dump(data, f, indent=2)
+            print(f"\nSuccessfully patched {updated_count} records in {run_info_path}")
+        except IOError as e:
+            print(f"Error writing to file: {e}")
+    else:
+        print("No records needed patching for missing iteration numbers.")
+
+
+def main():
+    """Main function to parse arguments and run the patcher."""
+    parser = argparse.ArgumentParser(
+        description="Adds missing 'iteration' fields to records in a run_info.json file by parsing the 'short_name'.",
+        formatter_class=argparse.RawTextHelpFormatter,
+    )
+    parser.add_argument(
+        "run_info_file", help="Path to the run_info.json file to patch."
+    )
+    parser.add_argument(
+        "--no-backup",
+        action="store_true",
+        help="If set, a backup of the original file will not be created.",
+    )
+
+    args = parser.parse_args()
+    add_missing_iterations(args.run_info_file, args.no_backup)
+
+
+if __name__ == "__main__":
+    main()