Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for mirroring of the NVD via API #3175

Merged
merged 23 commits into from
Nov 13, 2023

Conversation

nscuro
Copy link
Member

@nscuro nscuro commented Nov 5, 2023

Description

Adds the ability to mirror the NVD through its API v2.0, in response to the NVD's plan to retire the file-based vulnerability feeds.

Usage of the API will be initially optional, as using it without prior configuration of an API key will make users subject to aggressive rate limiting, which in turn causes mirroring to take a long time.

Downloading of feed files can optionally be enabled in order to keep DT's internal file mirror up-to-date. This may help users with transitioning.

Mirroring of the entire NVD database with API key takes around 10min. This is also true when running on a system that previously mirrored the NVD via the legacy feeds.

Addressed Issue

Closes #1861
Partly addresses #2895

Additional Details

Frontend changes

DependencyTrack/frontend#642

Observed differences in mirrored data

  • The published and updated timestamps of vulnerabilities have a higher resolution, and now include seconds.
  • Some descriptions include wrongly decoded unicode characters; It's unclear if this is caused by the NVD directly, or the Open-Vulnerability-Project client library we use.
  • Links included in references have a different order now.
  • A good chunk of vulnerabilities now includes CWEs and CVSS scores where they previously did not; Unclear if this is due to a bug in the original NistMirrorTask, or the NVD simply provides more data via their API.

Non-functional changes

  • The Open-Vulnerability-Project client is loading a lot of data into memory in an attempt to speed up the mirroring process. This can, especially when users provide an NVD API key, cause peaks of heap usage.
    • This only affects the first-time mirror operation. All operations afterwards will load significantly less data.
  • Synchronization of Vulnerability and VulnerableSoftware, as well as AffectedVersionAttribution records with the database is now properly transactional. There is now one transaction for synchronizing the Vulnerability record, and one for synchronizing VulnerableSoftware + AffectedVersionAttribution.
    • This is in contrast to the feed-based NistMirrorTask which committed multiple transactions per synchronization operation.
  • Synchronization operations are faster now, averaging 600 ops/s on my local machine, vs <100 ops/s with the feed-based NistMirrorTask.
  • The feed-based NistMirrorTask appears to be creating duplicate entries in the VULNERABLESOFTWARE_VULNERABILITIES table. This is dealt with and automatically cleaned up by the new API-based NistApiMirrorTask.

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have added corresponding update logic
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@nscuro nscuro added this to the 4.10 milestone Nov 5, 2023
@nscuro nscuro marked this pull request as draft November 5, 2023 18:08
nscuro added 17 commits November 6, 2023 15:22
Signed-off-by: nscuro <[email protected]>
Makes debugging and evaluating test failures easier.

Signed-off-by: nscuro <[email protected]>
Signed-off-by: nscuro <[email protected]>
@nscuro nscuro marked this pull request as ready for review November 13, 2023 18:29
@nscuro nscuro merged commit bd9bfec into DependencyTrack:master Nov 13, 2023
@nscuro nscuro deleted the issue-1861 branch November 13, 2023 19:25
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate from NVD data feeds to NVD API 2.0
1 participant