Project Meeting 2025.04.03

Agenda

• Explicit Error Terms

Technical

• Explicit Error Terms
  • Current Method of Seeding Random Number Generators (RNG) in ActivitySim
    • Each random number is drawn after 'seeding' the random number generator based on "chooser ID" and model that is being run
    • Chooser ID is related to the decision-maker who the choice is being made for
    • ActivitySim keeps reseeding the RNG to ensure consistency across the same insistences of a model (for parallel processing or on different computer)
    • For some models you need to have a sequence of RNs or multiple RNs
  • Strengths and weaknesses of Monte Carlo Simulation and Simulation with Explicit Error Terms
    • Monte Carlo simulation is computational efficient and simple to implement
    • MC with fixed RN is useful for debugging and replicability
    • When inputs change may result in odd correlation introduced between based and build outcomes
    • RN seeds should be explicitly varied by the user to reduce the correlation in choices
    • Further testing would be helpful to understand the extent of the problem
    • Simulation with Explicit Error Terms (EET) ensures choice outcomes between the base and build scenarios follow changes in systematic utility
    • EET reduces simulation variance - less 'switching' in simulation; may lead to fewer iterations required to reach convergence (requires further testing)
    • EET enables the implementation of other discrete choice models such as models that do not have closed form solution and models that explicitly account for taste variation (ex: error-component logit with panel effects)
    • EET is computationally more expensive than MC
    • EET has some ambiguity in identifying comparable choices between base and build scenarios
  • Potential software features to address weaknesses of Simulation with Explicit Error Terms
    • Addressing computational efficiency
      • Use more efficient random number generator code
      • have one RNG per model and chooser rather than re-seed the same RNG over and over
      • Use more efficient nested logit calculations through vectorization or investigate other methods
      • Use simple random sampling or MC simulation for destination choice
      • Turn off EET for certain models; provide user the option to use either MC or EET
  • Next Steps
    • Implement and test 'initial code' to assess magnitude of the problem
  • One of the goals are to get results that don't have weirdness; focus on results that are meaningful. Individual tours are not statistically meaningful. Sensitivity testing should help us understand the magnitude of changes and prevalence.
  • Even the semantic of "switching" is difficult; what about insertions of new tours and how do we summarize those? Don't need to think about switching or a single realization. Aligning single realizations between base and build does not make sense and it is not correct (not how we use the models).
  • One of the motivations was economic appraisals. Is it worthwhile to think about how economic appraisals are actually done? Appraisal methods use some aggregation (rule of half) not going through a fully disaggregate level. If so, we don't want to waste time with this comparability exercises. Interested in stability across scenarios. Framing of how detailed we get should consider how information will be consumed in economic appraisal models, which may be out of scope but useful for thinking about this moving forward.
    • Should look into how they are doing economic appraisals in Australia (also using ActivitySim). Talk to people at KPMG.
  • Will write up a testing plan; current implementation should be implemented in a current version of ActivitySim; need to think about what models to use--full or test; more detailed level of effort (cost to complete) -- needed for assigning tasks to team members.