Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterise the catalogue-graph ingestor trigger with pipeline date #2852

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

kenoir
Copy link
Contributor

@kenoir kenoir commented Mar 14, 2025

What does this change?

This change is in order to allow ingestion runs to be namespaced and monitored on a per pipeline basis, required for monitoring. We add some stats on file size and record count to the loader that will be used in reporting and monitoring in a later PR.

How to test

  • Run the tests, do they pass?

How can we measure success?

Allow us to run multiple pipeline and monitor them seperately.

Have we considered potential risks?

The catalogue graph is not quite in production use, so we should be able to catch errors before they are in public view, mitigating risk.

kenoir and others added 2 commits March 14, 2025 15:20
This change is in order to allow ingestion runs to be namespaced and
monitored on a per pipeline basis, required for monitoring.
@kenoir kenoir requested a review from a team as a code owner March 14, 2025 15:24
Copy link

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
1846 1592 86% 0% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
catalogue_graph/src/ingestor_indexer.py 85% 🟢
catalogue_graph/src/ingestor_loader.py 83% 🟢
catalogue_graph/src/ingestor_local.py 0% 🟢
catalogue_graph/src/ingestor_trigger.py 81% 🟢
TOTAL 62% 🟢

updated for commit: 2bdae58 by action🐍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Catalogue Graph: monitoring & reliability; Ensuring the Catalogue API index is safely updated
2 participants