Skip to content

CONFIGURATION.md

Latest commit

 

History

History
384 lines (272 loc) · 15.4 KB

CONFIGURATION.md

File metadata and controls

384 lines (272 loc) · 15.4 KB
Raw

Configuration & Setup Guide

This guide provides detailed instructions for configuring the MH DAQ backup system using the data_config.json configuration file, including Ecobee thermostat data integration and automatic scheduling.

Table of Contents

  1. Quick Start — Creating Configuration
  2. Network Paths Configuration
  3. Data Backup Configuration
  4. Ecobee Thermostat Setup
  5. Splinterware System Scheduler
  6. Verifying Your Setup

Quick Start — Creating Configuration

All backup scripts load their configuration from data_config.json in the repository root. This file is not committed to GitHub — each deployment creates its own copy from the provided template.

Step 1: Create Configuration File from Template

  1. In the repository root, find data_config.template.json
  2. Copy this file and name the copy data_config.json
  3. Open data_config.json in a text editor (Notepad, VSCode, etc.)

Step 2: Configure Local Source Paths

Find the "local_sources" section and update the paths for your system:

"local_sources": {
  "indoor_daq": {
    "path": "C:\\Users\\YOUR_USERNAME\\Desktop\\Task_Logger_Data",
    "description": "Task Logger output"
  },
  "outdoor_weather": {
    "path": "C:\\Users\\YOUR_USERNAME\\Desktop\\Outdoor_Data",
    "description": "Met One AIO2 weather station data"
  }
}

Change:

  • YOUR_USERNAME → Your actual Windows username (e.g., iaq, operator)
  • Verify these folders exist on your local machine

Step 3: Configure Network Destination Paths

Find the "remote_destinations" section. Configure one or more of the following:

Mission Network (Primary)

"mission": {
  "base_path": "\\\\YOUR_SERVER\\YOUR_SHARE\\...\\DAQ_Data\\raw_data",
  "folders": {
    "indoor_daq": "indoor_daq",
    "weather_station": "weather_station"
  },
  "log_file": "backup_log.txt"
}

Change:

  • YOUR_SERVER → Your actual server name (e.g., mission.el.nist.gov)
  • YOUR_SHARE and path structure → Match your network drive's actual path

EPA Shower Network (Retired — campaign ended 2026-07-16)

The EPA Shower measurement campaign has concluded. Its backup script now lives in src/completed_campaigns/ and is no longer part of the nightly run_backup.bat sequence. The epa_shower block is kept in data_config.json only so the retired script can still be run manually for a one-off re-copy; new deployments do not need it.

"epa_shower": {
  "base_path": "\\\\YOUR_SERVER\\YOUR_SHARE\\EPA_Shower\\MH DAQ",
  "folders": {
    "indoor_daq": "indoor_daq",
    "weather_station": "weather_station"
  },
  "archive_year": 2026
}

WUI Smoke Network (Active)

The NIST WUI Smoke IAQ Mitigation campaign is active (first burn 2026-07-31, no set end date). Its backup script (src/wui_smoke_file_backup.py) runs nightly in run_backup.bat and copies indoor DAQ and weather station data to the Elwood share. Only files whose leading YYYYMMDD name prefix is on or after start_date are copied; pre-deployment files are moved manually and are not handled by the script. The destination feeds the data processing in the NIST_wui-smoke-iaq-mitigation repository.

"wui_smoke": {
  "base_path": "\\\\elwood.nist.gov\\732\\internal\\IAQ\\MH_Contamination_Fall_2026\\instruments",
  "folders": {
    "indoor_daq": "MH_DAQ_indoor",
    "weather_station": "MH_DAQ_weather"
  },
  "start_date": "2026-07-31"
}

Change:

  • start_date → The campaign deployment date in YYYY-MM-DD format. Files dated before it are skipped.
  • Leave base_path and folders as shown unless the Elwood share layout changes.

Thermostat (Ecobee)

"thermostat": {
  "base_path": "\\\\YOUR_SERVER\\YOUR_SHARE\\...\\thermostat",
  "log_file": "backup_log_thermostat.txt",
  "thermostat_id": "YOUR_THERMOSTAT_ID"
}

Change:

  • YOUR_THERMOSTAT_ID → Your Ecobee thermostat's numeric ID
  • The thermostat ID can also be set via the ECOBEE_THERMOSTAT_ID environment variable (takes priority over the config value)

Step 4: Save Configuration

  1. After updating all paths, save the file
  2. Do NOT commit data_config.json to Git — it's in .gitignore to protect your paths
  3. Each deployment maintains its own copy

Network Paths Configuration

Understanding the Data Flow

All backup scripts use the same configuration file with these data flows:

Script Data Source Mission Destination
mh_daq_file_backup.py local_sources.indoor_daq.path remote_destinations.mission.base_path\indoor_daq\
mh_daq_file_backup.py local_sources.outdoor_weather.path remote_destinations.mission.base_path\weather_station\
wui_smoke_file_backup.py local_sources.indoor_daq.path (files on/after start_date) remote_destinations.wui_smoke.base_path\MH_DAQ_indoor\
wui_smoke_file_backup.py local_sources.outdoor_weather.path (files on/after start_date) remote_destinations.wui_smoke.base_path\MH_DAQ_weather\
ecobee_thermostat_backup.py Ecobee API (internet) remote_destinations.thermostat.base_path\YYYY\

The retired src/completed_campaigns/epa_shower_file_backup.py (EPA Shower campaign, ended 2026-07-16) copied the archive year of indoor DAQ and weather station data to remote_destinations.epa_shower.base_path. It is no longer run nightly.

All source and destination paths are configured in data_config.json.

Network Path Format

Windows network paths in JSON must use double backslashes (\\\\):

"base_path": "\\\\YOUR_SERVER\\YOUR_SHARE\\...\\DAQ_Data\\raw_data"

This represents the actual UNC path:

\\YOUR_SERVER\YOUR_SHARE\...\DAQ_Data\raw_data

Data Backup Configuration

The "instruments" section in data_config.json documents which sensors are installed and what variables they produce. You do not need to modify this section — it is informational reference only. The backup scripts copy entire data files and do not filter by instrument or variable.

Active instruments being backed up:

  • AIO2 — Outdoor weather station (5 channels: wind speed, direction, temperature, humidity, pressure)
  • Vaisala_HMP155 — Indoor RH/temperature at 4 locations (high-accuracy probes)
  • Vaisala_HMP45A — Indoor RH/temperature at 8 locations
  • Ecobee4 — Smart thermostat (temperature, humidity, HVAC runtime — downloaded via Ecobee API)

Instruments present in data files but currently disconnected:

  • Setra_264 — Differential pressure transducers (channels are logged as zero or noise — disconnected as of 12/31/25)

Live-File Safety (DAQ Backup Scripts)

mh_daq_file_backup.py skips the current day's data file by default. The DAQ system writes to today's file continuously throughout the day, and copying a file that is still being written to can produce a truncated or corrupt backup copy.

Each scheduled daily run captures only completed files (previous days). The current day's partial file is left alone and will be backed up at the next scheduled run after midnight, once the DAQ system has finished writing to it.

To include today's partial file (e.g., when running manually to verify recent data is visible on the network):

python <repo_path>\src\mh_daq_file_backup.py --include-today

Note: The Ecobee thermostat script (ecobee_thermostat_backup.py) is not affected by this — it always fetches the previous day's data from the Ecobee cloud API. The API only provides complete, finalized interval data for finished days, so there is no live-file risk for thermostat data.

Ecobee Thermostat Setup

Current status (Building 423 deployment): The Ecobee thermostat backup is not yet operational. An Ecobee API key has not been obtained for this account, so ecobee_thermostat_backup.py cannot authenticate and will log a token/authorization error when run_backup.bat runs. This does not affect the DAQ or weather-station backups, which run normally. The instructions below are retained so the integration can be enabled once an API key is available.

The Ecobee integration automatically downloads 5-minute interval thermostat runtime data each day and saves it as a CSV file on the mission network drive.

Overview

Each daily run fetches the previous day's full 24-hour dataset (288 five-minute intervals). Data includes indoor temperature, humidity, HVAC runtime, setpoints, and equipment state. See ecobee_thermostat_backup.py module docstring for the full list of columns.

Step 1: Enable Developer Access (One-Time)

This step requires access to the Ecobee web portal. Only one person needs to do this.

  1. Go to ecobee.com and log in with the MH project account
  2. Click your account icon in the top-right corner
  3. From the dropdown menu, select Developer
  4. Click Create New App and fill in the form:
    • Application Name: NIST MH DAQ
    • Application Summary: Automated data collection for NIST IAQ research
    • Authorization Method: Select ecobee PIN
  5. Click Create — you'll see your 32-character API Key

IMPORTANT: The API key is sensitive. Store it securely and do not commit it to the repository. It is used only during token setup (Step 2) and does not need to be stored in data_config.json.

Step 2: Token Setup (First Time on DAQ Computer)

After obtaining the API key, run the interactive token setup script on the DAQ computer:

  1. Open Command Prompt on the DAQ computer in Building 423
  2. Navigate to the scripts folder: 
    cd <repo_path>\src
  3. Run the setup script: 
    python ecobee_token_setup.py
  4. Paste your API Key when prompted
  5. The script will display a 4-character PIN
  6. Go to ecobee.com → My Apps → Add Application and enter the PIN
  7. Return to Command Prompt and press Enter — tokens will be saved to the path shown on screen (default: %USERPROFILE%\scripts\ecobee_tokens.json, or the path set in the ECOBEE_TOKEN_FILE environment variable)

Token Lifetime and Refresh

  • Access Token: Valid for ~2 hours (automatically refreshed by the script)
  • Refresh Token: Valid for ~1 year (requires re-authorization after 1 year)

If the script logs show Token refresh failed:

  1. Re-run: python ecobee_token_setup.py
  2. Follow the authorization steps again

Backfilling Missed Days

If the script fails on a given day, you can retrieve that day's data later using the TARGET_DATE_OVERRIDE environment variable:

set TARGET_DATE_OVERRIDE=2026-03-15
python <repo_path>\src\ecobee_thermostat_backup.py

The script skips dates where the CSV file already exists, so re-running it is always safe.

Ecobee Data Structure

Thermostat data is organized under the path configured in data_config.json → remote_destinations.thermostat.base_path:

<remote_destinations.thermostat.base_path>\
    ├── backup_log_thermostat.txt
    └── 2026\
        ├── 2026-01-01_thermostat.csv
        ├── 2026-01-02_thermostat.csv
        └── ...

Each CSV contains 288 rows of 5-minute interval data per day.

Splinterware System Scheduler

To run backups automatically every night, use Splinterware System Scheduler (required due to PIV smart card login on the DAQ computer). An event is already configured on the Building 423 DAQ computer; the steps below document how it was set up and how to edit it.

Why Splinterware?

The DAQ computer requires PIV smart card login, which prevents standard Windows Task Scheduler from running tasks under the iaq account. Splinterware System Scheduler works around this limitation.

Setup Steps

  1. On the DAQ computer, open Splinterware System Scheduler

  2. Create a new event using either method:

    • From the top menu, click Action → New Event..., or
    • Click the green circle with the white + sign

    This opens the Event screen on the Event tab.

  3. On the Event tab, configure these fields:

    • Event Type: Run Application
    • Title: MH IAQ DAQ Backup
    • Application: <repo_path>\scripts\run_backup.bat
    • Parameters: (leave blank)
    • Working Dir: <repo_path>\scripts
    • Sendkeys: (leave blank)
    • State: leave at the default (normal window)

  4. Click the Schedule tab

  5. Configure the schedule:

    • Frequency: Daily
    • Time: Choose a time when the DAQ computer is reliably powered on and no active experiments are running
    • Recommended: 6:00 AM

  6. (Optional) Click the Advanced tab to review additional options, such as retry-on-failure and a limit on how long the event may run. Configure these only if needed; the defaults are fine for a nightly backup.

  7. Click OK (or Save) to create the event. Confirm it appears in the main event list.

Verifying the Scheduled Event

  1. Manual test: Right-click the MH IAQ DAQ Backup event → Run (or Run Now) to verify it works before relying on the schedule
  2. First scheduled run: Check batch_output.log at the scheduled time to confirm it ran automatically
  3. Ongoing monitoring: Check batch_output.log after each scheduled run to ensure no errors

Editing or Disabling the Event

  1. Open Splinterware System Scheduler
  2. Find the MH IAQ DAQ Backup event in the event list
  3. Right-click → Edit (or Properties) to reopen the Event/Schedule/Advanced tabs
  4. Make your changes and click OK to save

Verifying Your Setup

Pre-Deployment Checklist

Before considering the system fully configured, verify:

  • Network drives are accessible (see Installation Guide)
  • Manual backup test succeeds (run_backup.bat runs — DAQ and weather-station backups complete without errors)
  • DAQ data appears on mission network drive (indoor_daq\ and weather_station\ folders)
  • Ecobee token is created and first backup succeeds (pending an Ecobee API key — not required for current deployment)
  • Splinterware scheduler event is configured and runs automatically

Manual Testing

To test the backup system:

  1. On the DAQ computer, open Command Prompt
  2. Run: 
    <repo_path>\scripts\run_backup.bat
  3. Open <repo_path>\scripts\batch_output.log to verify all backups succeeded

Viewing Backed-Up Data

  1. Open File Explorer on the DAQ computer
  2. Navigate to the configured mission network path
  3. Check for:
    • indoor_daq\ folder with .txt files (newest files should match recent dates)
    • weather_station\ folder with data files
    • thermostat\YYYY\ folder with daily CSV files

Log File Analysis

The batch_output.log file captures console output from each script run. Look for:

Success indicators:

  • ALL BACKUPS COMPLETED SUCCESSFULLY
  • ECOBEE BACKUP COMPLETED SUCCESSFULLY
  • File counts: X copied, Y unchanged, 0 errors

Error indicators:

  • Network path NOT accessible
  • ERROR lines
  • Error counts greater than 0

See the Troubleshooting Guide for common errors and solutions.

Next Steps

  • Troubleshooting issues? See the Troubleshooting Guide
  • Need help? Contact Nathan Lima with your log file and error message