BFD-3809: Dataset loaded by Monday 9 AM SLO implementation checks database

[malessi]

JIRA Ticket:
BFD-3809

What Does This PR Do?

This PR implements a new Python Docker-based Lambda, ccw-manifests-verifier, that runs every Monday at 9 AM ET and reconciles the state of Incoming/Synthetic/Incoming manifests with manifest status in the database. If any S3 manifests do not exist in the database or are not marked as COMPLETED the Lambda will send an alert to configured SNS Topics. In prod, this will send alerts to Splunk On Call and to the #bfd-alerts Slack channel.

What Should Reviewers Watch For?

If you're reviewing this PR, please check for these things in particular:

What Security Implications Does This PR Have?

Please indicate if this PR does any of the following:

• Adds any new software dependencies

• Modifies any security controls

• Adds new transmission or storage of data

• Any other changes that could possibly affect security?

• I have considered the above security implications as it relates to this PR. (If one or more of the above apply, it cannot be merged without the ISSO or team security engineer's (@sb-benohe) approval.)

Validation

Have you fully verified and tested these changes? Is the acceptance criteria met? Please provide reproducible testing instructions, code snippets, or screenshots as applicable.

• terraform applying in test, verifying that it applies successfully
• Running the ccw-manifests-verifier Lambda in test with a fake unloaded manifest in S3, verifying that it sends an alert to the #bfd-internal-alerts Slack channel indicating that the manifest is unloaded

[malessi]

This change, and the related changes in ops/terraform/services/pipeline/user-data.sh.tftpl to the jq queries, is necessary as the new SSM parameter /bfd/${env}/pipeline/nonsensitive/ccw/slo/weekend_data_availability/verifier/enabled would otherwise "shadow" the value of /bfd/${env}/pipeline/nonsensitive/ccw/job/enabled as the last in the existing jq queries would result in a duplicate enabled variable in the Ansible's "extra vars".

Changing the jq to instead include the full name of the parameter (replacing / with _) to avoid this results in this variable becoming job_enabled as its full name is job/enabled. I verified that there were no other variables changed that were relevant to the bfd-pipeline Ansible.

[malessi]

See c8c60fd for relevant commit.