Our Automated Testing System

Daysim is under continuous integration, which means revisions to the software are automatically checked against a set of test regions to ensure comprehensive coverage and that revisions are working correctly. Our test system was migrated from Jenkins to Azure Pipelines in August, 2019.

Overview

• RSG's test server contains multiple Azure build agents, each configured to build, run, and test the DaySim .Net application.
• Changes to DaySim's develop branch trigger a new Azure build on the test server. Azure pulls the updated DaySim code and builds the executable.
• Azure also pulls the test data from each of the region specific private test repositories. This data persists on the test server and is updated only when Git detects a change.
• Each test region job is then run/tested in parallel by different agents.
• Each job runs a Python program that compares the build results to the expected target data specified in each region's test repository.
• Each job also runs an R program that generates a comprehensive set of Excel reports which are then stored on the test server. Both the Excel reports and the build results persist on the server until overwritten by the next build.
• The results of the test(s) are returned to Github's Azure plugin, and the job is reported as pass or fail. The status of the job is used to set the GitHub branch status as well.
• The develop branch can only be merged to master if all test jobs pass

Test Types and Setup

There are two types of tests supported by the system:

• Simple difference - All folders generated by DaySim (working, outputs, estimation) are compared with the target folders included in the test repository. A difference in outputs is considered FAIL
• Modeling difference(s) - The main.R script generates reports on the outputs in order to review the differences compared to the target in terms of tour generation rates, trip lengths, mode shares, etc. These reports are in Excel so comparison must be manually reviewed.

Regions Automatically Tested

The regional DaySim models included in the test system are listed in the Azure console. The regional test repositories are private and therefore not accessible.