Skip to content

Comments

Add Openstack Importer V2 Pipeline#2154

Open
Samk1710 wants to merge 3 commits intoaboutcode-org:mainfrom
Samk1710:add-ossa-importer
Open

Add Openstack Importer V2 Pipeline#2154
Samk1710 wants to merge 3 commits intoaboutcode-org:mainfrom
Samk1710:add-ossa-importer

Conversation

@Samk1710
Copy link
Contributor

@Samk1710 Samk1710 commented Feb 8, 2026

Closes

Data Source: https://github.com/openstack/ossa
Research Notes: #374 (comment)

Importer logs:

Importing data using ossa_importer_v2
INFO 2026-02-23 19:20:33.701459 UTC Pipeline [OSSAImporterPipeline] starting
INFO 2026-02-23 19:20:33.701711 UTC Step [clone] starting
INFO 2026-02-23 19:20:33.701807 UTC Cloning `git+https://github.com/openstack/ossa`
INFO 2026-02-23 19:20:39.229018 UTC Step [clone] completed in 6 seconds
INFO 2026-02-23 19:20:39.229224 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-23 19:20:39.232283 UTC Collecting 53 advisories
INFO 2026-02-23 19:20:39.337457 UTC Progress: 11% (6/53) ETA: 1 seconds
INFO 2026-02-23 19:20:39.409824 UTC Progress: 22% (12/53) ETA: 1 seconds
INFO 2026-02-23 19:20:39.471911 UTC Progress: 32% (17/53) ETA: 1 seconds
INFO 2026-02-23 19:20:39.545581 UTC Progress: 43% (23/53)
INFO 2026-02-23 19:20:39.615647 UTC Progress: 54% (29/53)
INFO 2026-02-23 19:20:39.681759 UTC Progress: 64% (34/53)
INFO 2026-02-23 19:20:39.760755 UTC Progress: 75% (40/53)
INFO 2026-02-23 19:20:39.862910 UTC Progress: 86% (46/53)
INFO 2026-02-23 19:20:39.937117 UTC Progress: 96% (51/53)
INFO 2026-02-23 19:20:39.982854 UTC Successfully collected 53 advisories
INFO 2026-02-23 19:20:39.983122 UTC Step [collect_and_store_advisories] completed in 1 seconds
INFO 2026-02-23 19:20:39.983200 UTC Step [clean_downloads] starting
INFO 2026-02-23 19:20:39.983281 UTC Removing cloned repository
INFO 2026-02-23 19:20:39.989312 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-23 19:20:39.989466 UTC Pipeline completed in 6 seconds

Copy link
Collaborator

@ziadhany ziadhany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Samk1710, the code looks good overall, we just need to refactor a few functions to make them simpler and cleaner

self.log(f"Cloning `{self.repo_url}`")
self.vcs_response = fetch_via_vcs(self.repo_url)

def fetch(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be merged in collect_advisories function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany
Moved into get_processable_files that is called in advisories_count and collect_advisories. Let me know if this works

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also now we filter with only the file name ;>

skipped_old += 1
continue

self.processable_advisories.append(file_path)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, avoid using global variables to store all the paths, as this takes up some amount of memory.

Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
@Samk1710
Copy link
Contributor Author

Hey @ziadhany
I have refactored as per suggestions and moved parse_version_range in universe with

Kindly have a look when time and let me know if anything needs adjustments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants