Implement Error Tracking for Missing DHIS2 Option Codes

[mtuchi]

Background

Currently, when mapping OpenMRS encounter data to DHIS2 events, we have a fallback mechanism that attempts to use multiple sources for option values:

const matchingOption =
  opt?.["DHIS2 Option Code"] ??
  opt?.["DHIS2 Option name"] ?? 
  answer?.value?.display;

This fallback to answer.value.display masks metadata mapping issues and causes DHIS2 validation errors. The root cause is incomplete metadata configuration where DHIS2 Option Codes are not properly set in the metadata files by the MSF team.

Problem Statement

When DHIS2 Option Codes are missing from the metadata configuration, the system silently falls back to using answer.value.display, which often doesn't match DHIS2's expected option codes. This results in:

1. DHIS2 API rejecting events with invalid option values
2. Difficult-to-trace mapping errors that only surface during DHIS2 synchronization
3. Getting blocked during development and testing

Proposed Solution

Implement a robust error tracking system that:

1. Removes the fallback to answer.value.display and DHIS2 Option name
2. Collects detailed information about missing option codes
3. Logs errors to a centralized Google Sheet for MSF team review
4. Provides clear visibility into metadata gaps without blocking testing

Acceptance Criteria

Code Changes

• Remove the fallback logic: ?? opt?.["DHIS2 Option name"] ?? answer?.value?.display;
• Initialize an error tracking array at the beginning of the mapping process
• When !matchingOption, capture the following details:
  • OpenMRS Question (display name)
  • Concept External ID (answer.concept.uuid)
  • Answer Display Value (answer.value.display)
  • Answer Value UUID (answer.value.uuid)
  • DHIS2 Data Element UID (dataElement)
  • DHIS2 Option Set UID (matchingOptionSet)
  • Metadata file name (encounter.form.name or encounter.form.uuid)
  • Encounter UUID (encounter.uuid)
  • Patient UUID (encounter.patient.uuid)
  • Timestamp of error occurrence

Google Sheets Integration

• Create a new workflow step that pushes error data to the tracking spreadsheet
• Target Google Sheet: Missing OptionSets Mappings
• Sheet columns should match:

  Timestamp | OpenMRS Question | Concept External ID | Answer Display | Answer UUID | DHIS2 DE UID | DHIS2 OptionSet UID | Metadata File | Encounter UUID | Patient UUID

• Implement batch writing to avoid rate limits (group errors before writing)
• Include error count summary and URL to the google sheet in workflow logs

Logging & Monitoring

• Log the Google Sheet URL at the end of each workflow run
• Include summary statistics:
  • Total encounters processed
  • Total missing option codes found
  • Number of unique option sets affected
  • Link to error tracking sheet
• Console log format:

  ⚠️  Missing DHIS2 Option Codes Detected
  Total Errors: [count]
  Affected Option Sets: [count]
  View Details: [Google Sheet URL]

[mtuchi]

@AishaHassen this is ready for QA, See workflow run here, And here is the Missing OptionsSets Mappings Sheet

• The Missing OptionSets step will always run if we are missing option sets
• I have rename Metadata File to Metadata Form
• I have added a Source File: ....xlsx log in Missing OptionsSets step
• Googlesheet does not support batch writing to avoid rate limits, But we have a ticket in adaptors GoogleSheets adaptor should support multi-range update and throttling  adaptors#1476

Slightly unrelated, If you try to run the workflow without a cursor or specific patient i am seeing this error in create-child-teis

"Attribute: `P4wdYGkldeG`, is mandatory in program `w9MSPn5oSqp` but not declared in enrollment `pYzKK3UuAia`.",
        "errorCode": "E1018",
        "trackerType": "ENROLLMENT",
        "uid": "pYzKK3UuAia"

I will investigate further what's causing this, it's probably missing mappings

[AishaHassen]

Thanks @mtuchi , i will look at it in details but quick responses:

1. Can you also add "Source File" column to the google sheet. There should be a column for capturing the metadata file name.

2. Googlesheet does not support batch writing to avoid rate limits

• What is the impact of this? ^

3. Can you turn on the rest of the forms in the metadata file and lets also test with those forms?

[mtuchi]

@AishaHassen

1. I have added Source File column,
2. Regarding google sheet rate limits, If we send more than 60 request / minute when adding values to googlesheet. The step will fail. Also google recommends we send a maximum payload of 2mb for better performance. So If updating 100,000 rows, we should split into batches of ~5,000-10,000 rows each request. I can chunk the values to 10,000 rows so that we don't degrade performance if we have millions of rows 🤷🏽, Should we do this ?
3. I have enabled F55 to F66 are there other forms that i should enable ?

enable-froms.mp4

[AishaHassen]

@mtuchi

1. Thanks
2. Oii, i dont expect to have over 100k rows of erraneous metadata so i dont think that will be necessary
3. Thanks. This looks correct.