fix(server): own the vuls2 db for the process instead of per request

[MaineK00n]

What did you implement:

Server mode created a vuls2 db session per request, so every request decided for itself whether the db was due for a download, fetched it, and opened bolt. This is the shared root cause behind two discussions:

• Database update restarting #2613 — a pod that came up with no db on disk had each arriving request start its own multi-gigabyte fetch into the same directory, all of them competing for bandwidth and disk until they failed and started over.
• Concurrency issue in boltdb handling #2615 — a request that did find a db opened bolt three times (shouldDownload, the metadata check in newDBConfig, and the real open) and built a read cache it threw away on the way out, so concurrent requests shared nothing and re-read the same CVEs from disk.

This PR hoists the db out of the request path.

SharedDB owns the db for the process

SharedDB opens the db once and hands every request a borrowed session over that one open handle. Sharing is safe: bolt allows concurrent read transactions on a single open handle, every vuls2 storage read is such a transaction, and the vuls2 cache is a sync.Map.

The shared session deliberately carries no read cache (WithCache: false). The vuls2 cache never evicts, so one that outlives a request would accumulate every advisory and vulnerability it ever read until the process is OOM-killed. newDBConfig gained a withCache parameter for this; the report/detect path keeps its per-run cache, where the lifetime is bounded and the cache is what collapses the O(K²) re-reads enrich would otherwise do.

Nothing on the request path fetches

Only the goroutine that prepares and refreshes the db fetches, so a fetch is single-flight by construction. Until a db is open, /health answers 503 and /vuls refuses — a request that arrives first is no longer the thing that downloads the db.

Important

Point a readiness probe at /health. A liveness probe would restart the process partway through the first fetch and start the download over from nothing.

Refreshing never disturbs a running query

A new db is installed alongside the one it replaces; the old one is closed by whichever request releases it last. Closing a db munmaps it, so swapping in place would fault a query mid-flight. A refresh that fails leaves the working db in place rather than taking it away.

Startup serves from disk before considering a download

A db on disk that is merely due for a refresh is still a db worth serving, so a process that has one comes up in the time it takes to mmap a file instead of after a full fetch. Bringing it up to date is then the refresher's job, with requests being served the whole time.

Refreshes compare the repository digest first

shouldDownload goes by timestamps, and the nightly db's LastModified is the night it was built — so a db past the staleness window looks due on every check for as long as it lives. Combined with a periodic refresh that re-fetched gigabytes hourly even when the tag had not moved. hasNewerRemote resolves the repository manifest and compares its digest against the one fetch recorded in the local db's metadata, so the download only happens when the tag has actually moved.

Two adjacent fixes

• Detection workers are sized by GOMAXPROCS rather than runtime.NumCPU(), which reports the machine's CPU count even when a cgroup quota lets far fewer of them run. On a 2-vCPU pod scheduled on an 8-core host, every request was spawning 8 workers for CPUs it would never get.
• -max-concurrency (default GOMAXPROCS) bounds concurrent detections. One detection holds every CVE it finds, so an unbounded number of them oversubscribes memory badly enough to stall the server. Requests queue on it rather than being rejected — clients with tight HTTP timeouts may see a timeout where they previously saw a slow response.

Relation to #2617 / #2618

@maxenced opened #2617 and #2618 against discussion 2613, and the analysis in both discussions is theirs — including the observation that the per-request open/close pattern is the problem, and the shape of the startup fix (fetch at startup, keep the server out of rotation until it completes). This PR arrives at the same behaviour for /health and the same single-flight guarantee.

Where it differs is scope: #2617/#2618 keep the per-request session and coordinate the fetch, which fixes 2613 but leaves 2615 untouched — the per-request bolt opens and the discarded read cache are structural to that model. This PR replaces the ownership model instead, which is what lets both discussions be closed by one change.

I'd rather not have @maxenced's work go in and then be rewritten, so I'm opening this as a draft to compare first. @maxenced — you have an environment that reproduces both problems, which I don't; would you be willing to try a build of this branch?

Type of change

• Bug fix (non-breaking change which fixes an issue)
• This change requires a documentation update

Behaviour changes

	before	after
db on disk, stale	request blocks on an 11 GB fetch	serves immediately, refreshes in the background
no db on disk	every request fetches its own copy	/health and /vuls 503, one fetch, retried every minute
db up to date but past the staleness window	re-fetched on every check	manifest resolve only
concurrent requests	own bolt handle + own cache each	one shared handle, no cache
/health	always 200 ok	503 until a db is open

How Has This Been Tested?

• make fmt, go build ./..., go vet ./...
• go test -race ./detector/...

New tests in detector/vuls2/shared_test.go (all run under -race):

• TestSharedDB_AcquireBeforeReady — a db on disk is not a db this process has open; Acquire refuses instead of opening one
• TestSharedDB_OpenLocal / TestSharedDB_Prepare — a stale db (SkipUpdate: false, LastModified old enough for a refresh to be due) is opened without reaching the registry, which is what proves the staleness rule no longer blocks startup
• TestSharedDB_OpenLocalWithoutDB — refuses rather than downloading
• TestSharedDB_PrepareCancelled — gives up on ctx rather than blocking a shutdown
• TestSharedDB_Reload — a borrowed Close() leaves the shared db open; a reload with nothing due reuses the installed handle
• TestSharedDB_InstallKeepsRetiredGenerationOpenUntilReleased — the regression test for the munmap hazard: a db swapped out while a request is reading it stays readable, and is closed when that request releases it
• TestSharedDB_ConcurrentAcquire — 8 readers against 4 concurrent installs

Test_hasNewerRemote in detector/vuls2/db_test.go covers the branches reachable offline (no digest recorded, no db file, unparsable repository). The digest-match path needs a registry and is not covered.

Not yet verified against a real deployment — I don't have an environment that reproduces either discussion's symptoms at scale.

Checklist:

• Write tests
• Write documentation — /health must be a readiness probe, and -max-concurrency queues rather than rejects. Not written yet.
• Check that there aren't other open pull requests for the same issue/feature — see the section on fix(detector/vuls2): prevent redundant parallel DB downloads in server mode #2617 / Download vuls2 database (if enabled) on startup #2618 above
• Format your source code by make fmt
• Pass the test by make test
• Provide verification config / commands
• Enable "Allow edits from maintainers" for this PR
• Update the messages below

Known gaps

Neither this PR nor #2617/#2618 addresses the fetch failure itself. In discussion 2613 the download dies at ~5 GB with stream error: PROTOCOL_ERROR; received from peer, and fetch.Fetch reads the layer as one stream with no retry and no ranged resume, so it starts over from zero every time. Fixing that belongs in MaineK00n/vuls2, along with the same digest precheck so vuls2 db fetch benefits too.

Until then, the reliable deployment is SkipUpdate = true with the db provided by an initContainer or a PVC — with this PR that path also gives the fastest startup, since OpenLocal succeeds immediately.

Is this ready for review?: NO

[maxenced]

Do you have a built image with this I can use to test ?
Tried to fork and build it but build fails with some issues in db config

[MaineK00n]

I had a look at your test/share-vuls2-db-in-server-mode branch — the build failure is from stacking, not from this PR.

That branch has this PR at the base with #2617 and #2618 rebased on top. All three rewrite newDBConfig, and the conflict resolution dropped this PR's changes to detector/vuls2/db.go, which leaves:

detector/vuls2/db.go:13:2: "oras.land/oras-go/v2/registry/remote" imported and not used
detector/vuls2/db.go:108:14: undefined: withCache
detector/vuls2/shared.go:167:24: undefined: hasNewerRemote
detector/vuls2/vuls2.go:210:54: too many arguments in call to newDBConfig

This PR is meant to replace #2617 and #2618 rather than sit on top of them, so please build the branch on its own:

git clone -b MaineK00n/share-vuls2-db-in-server-mode https://github.com/future-architect/vuls.git
cd vuls && docker build -t vuls:2628 .

I confirmed that from a fresh clone. I have also rebased the branch onto master, which incidentally drops the duplicate oras.land/oras-go/v2 entry in go.mod that you had to fix by hand.

I don't have a published image to hand you — the upstream docker-publish.yml only runs on master pushes and tags. Your fork already has a ghcr-publish.yml, so pointing that at a branch with just this commit should work.

To set expectations: I haven't verified this against a real deployment yet — I don't have an environment that reproduces #2613/#2615. It builds and the unit tests pass, but that's all I can vouch for so far, which is exactly why your testing would help a lot.

[maxenced]

So, I tested your image. The database load at startup works as expected !
But ... the software<->cve mapping is muuuuuch slower than with my images (I guess it might be related to the cache ? ).
On the same hardware (4 cores, 32Gb of ram), most of the requests we're doing are taking more than 300s, while they take a few (dozen of, at most) seconds with my image.

Hope it helps