Trivy not detecting package upgrades in Kroki hardened image

[DavidACastagna]

IDs

All of them? They're listed in the attached JSON.

Description

Creating a "hardened" verison of an image is not resulting in a clean Trivy scan - although I'm pretty sure it should. And the fix we had to make to get Trivy to see the package updates ended up erasing the base image's ENTRYPOINT and ENV blocks rendering the base image useless. So we need to figure out how to get Trivy to see the correctly installed packages so we don't need to try to use a workaround that doesn't break the container.

I'm hoping the issue is simple to resolve with a trivy option of some kind? Or with a copy of some specific content from the base image? But I'm guessing that there is something more difficult going on here.

Consider these two trivy reports:

The baseline report: kroki-0.24.1.trivy.json
The report post-hardening: kroki-0.24.1-hardened.trivy.json

The vulnerabilities section of both of these are the same. Even though this is the Dockerfile used to construct the "hardened" image:

FROM docker.io/yuzutech/kroki:0.24.1
USER root
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y bash curl libcurl4 libc6 libc-bin libgssapi-krb5-2 libk5crypto3 \
                   libkrb5-3 libkrb5support0 libnghttp2-14 libprocps8 libgnutls30 \
                   libldap-2.5-0 libpam-modules libpam-modules-bin libpam-runtime \
                   libpam0g locales login libgssapi-krb5-2 libssl3 zlib1g libssh-4 \
                   openssl passwd perl-base procps tar
HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1
USER kroki

Using this Dockerfile instead seems to fix the trivy issue (it sees the installed packages) but the ENTRYPOINT and ENV blocks are purged rendering the base image useless (that part is obviously not trivy's doing):

FROM docker.io/yuzutech/kroki:0.24.1 as base
USER root
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y bash curl libcurl4 libc6 libc-bin libgssapi-krb5-2 libk5crypto3 \
                   libkrb5-3 libkrb5support0 libnghttp2-14 libprocps8 libgnutls30 \
                   libldap-2.5-0 libpam-modules libpam-modules-bin libpam-runtime \
                   libpam0g locales login libgssapi-krb5-2 libssl3 zlib1g libssh-4 \
                   openssl passwd perl-base procps tar
HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1
FROM scratch
COPY --from=base / /
USER kroki

Reproduction Steps

The tl;dr of what's below is: To harden a base image I run a scan, install package upgrades via an enclosing Dockerfile, scan again and see no changes.  Make a breaking change to the Dockerfile and now the scan comes back clean.

---

1. I run `trivy images` on a base image (yuzutech/kroki:0.24.1) which comes up with a whole bunch of fixes to make (libraries to update) see attached JSON report.
2. Then I create a wrapper Dockerfile which uses apt-get to execute all of the suggested library updates.
3. Then I build this new Dockerfile to produce a new image.
4. I scan it and `trivy image <new-image>` produces exactly the same report as before (it sees none of the package updates).
5. To verify that the updates actually occurred I did a docker run as root and checked the stalled packages with `apt show <package>`. This shows the packages are in fact updated:

A scan of the base third-party image shows (among lots of other things) these two packages needing updates:


 │ curl               │ CVE-2023-46218 │ MEDIUM   │ fixed    │ 7.81.0-1ubuntu1.14           │ 7.81.0-1ubuntu1.15           │ curl: information disclosure by exploiting a mixed case flaw │
│ libc-bin           │ CVE-2023-5156  │ MEDIUM   │ fixed    │ 2.35-0ubuntu3.4              │ 2.35-0ubuntu3.5              │ glibc: DoS due to memory leak in getaddrinfo.c               │

To verify the image contains the right stuff I ran apt show on curl and libc-bin and got:

root@01acd61ff1b7:/# apt show curl
Package: curl
Version: 7.81.0-1ubuntu1.15
Priority: optional
. . .
root@01acd61ff1b7:/# apt show libc-bin
Package: libc-bin
Version: 2.35-0ubuntu3.6
Priority: required
. . .

Which are upgraded from the baseline and match or exceed what was in the original trivy report.

6. Then I add the "FROM scratch" and "COPY --from=base / /" lines and build again.
7. This time the trivy scan comes up clean. BUT this is not a working Docker image since it has no more ENTRYPOINT or ENV entries in it.

### Target

Container Image

### Scanner

Vulnerability

### Target OS

ubuntu:jammy

### Debug Output

```bash
$ trivy image --debug yuzutech/kroki:0.24.1
2024-03-07T10:27:56.436-0600	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-03-07T10:27:56.437-0600	DEBUG	Ignore statuses	{"statuses": null}
2024-03-07T10:27:56.480-0600	DEBUG	cache dir:  /Users/david.castagna/Library/Caches/trivy
2024-03-07T10:27:56.482-0600	INFO	Need to update DB
2024-03-07T10:27:56.483-0600	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2024-03-07T10:27:56.483-0600	INFO	Downloading DB...
42.41 MiB / 42.41 MiB [---------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 21.74 MiB p/s 2.2s
2024-03-07T10:27:59.537-0600	DEBUG	Updating database metadata...
2024-03-07T10:27:59.538-0600	DEBUG	DB Schema: 2, UpdatedAt: 2024-03-07 12:10:05.219113353 +0000 UTC, NextUpdate: 2024-03-07 18:10:05.219113113 +0000 UTC, DownloadedAt: 2024-03-07 16:27:59.53789 +0000 UTC
2024-03-07T10:27:59.538-0600	INFO	Vulnerability scanning is enabled
2024-03-07T10:27:59.538-0600	DEBUG	Vulnerability type:  [os library]
2024-03-07T10:27:59.538-0600	INFO	Secret scanning is enabled
2024-03-07T10:27:59.538-0600	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-03-07T10:27:59.538-0600	INFO	Please see also https://aquasecurity.github.io/trivy/v0.49/docs/scanner/secret/#recommendation for faster secret detection
2024-03-07T10:27:59.538-0600	DEBUG	Enabling misconfiguration scanners: [azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan]
2024-03-07T10:28:00.170-0600	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-07T10:28:00.171-0600	DEBUG	The nuget packages directory couldn't be found. License search disabled
2024-03-07T10:28:00.171-0600	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-07T10:28:00.690-0600	DEBUG	Image ID: sha256:3f28d0b4ef0088f80f7f3002563fbf9a4bce2cb6aa99fadd3e06c9cbbd832b38
2024-03-07T10:28:00.690-0600	DEBUG	Diff IDs: [sha256:256d88da41857db513b95b50ba9a9b28491b58c954e25477d5dad8abb465430b sha256:3a9073a4d18e5ed2ae6f9fd9fee81ea43774907ce603ba955bba8fc0819aa250 sha256:0762222312339e83a1dd7836d0a1784e81464629ae1a0b9958bfcbfcd7517352 sha256:5fda322ef353c4c7a77f42eefc99944f00f18123acc6888a8701988046d8995f sha256:25713e58e40729c098be036d430af0f8d0a56c47bca5acf9f6f75341c2e48629 sha256:251d59c9175f2efe1e47563a4f88dc736130a883b2b31fd829a07d6a0a14d378 sha256:1a0b3814c35e4c4e1ba109d3441f0f1f1d879cc9787f3de7e92222db5ea81f06 sha256:83821bdeddfa2dc097b8abf2aebb994b11faf9f852b706512436f11cfa9beb2c sha256:9bd07678c5129c01e170bf007cd6341cb1884b5320f2fd34d47c99d3ab61b841 sha256:c8b73d4464f22b20e2f82c54130f1b9edd436e980744d395b7a4f1455f20e60a sha256:9d76ef7cfc2486d08774e8a262bbaaf679b71748b4dc9802db0d62402f4ae200 sha256:0b29d2b0f2199a6d0d5d721dc2e8adf9a237023a45963b38e40aaac3e95f4d1b sha256:10bd4dd81c1da42aaf042046d8c92e9e1fb6648983296524c3bafe437acc62b5 sha256:8a40886051b7b68bf52a06a257720c613876327125a9eee598b7fbd4159e3b39 sha256:491724c764f5c1d0e07b1682ae03f2e511e7db08b0a6f0b39a12a2e0f6bb6c3b sha256:956a1c81919efa8f1fb99fbbc905e64b4ef0674d44e5e6950da048d6aebf8516 sha256:346c9d8e088c5f72de0b85f1f95bfdbf71d78416013c49efd8a79ee50081a942 sha256:97abba2fcde1600d470a74ce805f560c49ee152bc1e8e126ffd861af1e962e70 sha256:467c3f493e611d70bb8b4cdd9fa81a43146f1fa617afa0d8f5eb78cd9d9ae679 sha256:e8cdf4417714e9c5bd8f2e20c3563bf2028b8ce828880a5206098397802ecbc7 sha256:ebe51e3447abd6cab89e205fbef2fe828b895f4dd920a12e94671f4bb8f719c5 sha256:59bd3f6858b3eb7348409f4e01b5a76769cd0385a21023605e0e53c400edf9bd sha256:0bed2459974f6cf0f077b77e41aac0bbad7ed815c0bf7f4dfd9c7516097ea3fa sha256:c5a623bb71f2c680e4c8161ff17a2cd4339b2686a9320c2d15bf557f99fb88f2 sha256:c22955d3189840dee4498e469c0dc8207d499e1db86d6229426ca383b5dda57d sha256:5b197de034dc1bfae819b5f2f6a630e4a4d9a5b7c4f7a3da244bbe539f0658c3 sha256:746f2dde5a51a8d1db1ab66d922b4f68b7bf2e8a2fa87c4d6af4d8071dadb9d3]
2024-03-07T10:28:00.690-0600	DEBUG	Base Layers: [sha256:256d88da41857db513b95b50ba9a9b28491b58c954e25477d5dad8abb465430b]
2024-03-07T10:28:00.727-0600	INFO	Detected OS: ubuntu
2024-03-07T10:28:00.727-0600	INFO	Detecting Ubuntu vulnerabilities...
2024-03-07T10:28:00.727-0600	DEBUG	ubuntu: os version: 22.04
2024-03-07T10:28:00.727-0600	DEBUG	ubuntu: the number of packages: 126
2024-03-07T10:28:00.730-0600	INFO	Number of language-specific files: 2
2024-03-07T10:28:00.730-0600	INFO	Detecting gobinary vulnerabilities...
2024-03-07T10:28:00.730-0600	DEBUG	Detecting library vulnerabilities, type: gobinary, path: usr/bin/d2
2024-03-07T10:28:00.731-0600	INFO	Detecting jar vulnerabilities...
2024-03-07T10:28:00.731-0600	DEBUG	Detecting library vulnerabilities, type: jar, path: 


$ trivy image --debug kroki-issue:fix-attempt
2024-03-07T10:29:32.028-0600	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-03-07T10:29:32.028-0600	DEBUG	Ignore statuses	{"statuses": null}
2024-03-07T10:29:32.069-0600	DEBUG	cache dir:  /Users/david.castagna/Library/Caches/trivy
2024-03-07T10:29:32.069-0600	DEBUG	DB update was skipped because the local DB is the latest
2024-03-07T10:29:32.070-0600	DEBUG	DB Schema: 2, UpdatedAt: 2024-03-07 12:10:05.219113353 +0000 UTC, NextUpdate: 2024-03-07 18:10:05.219113113 +0000 UTC, DownloadedAt: 2024-03-07 16:27:59.53789 +0000 UTC
2024-03-07T10:29:32.070-0600	INFO	Vulnerability scanning is enabled
2024-03-07T10:29:32.070-0600	DEBUG	Vulnerability type:  [os library]
2024-03-07T10:29:32.070-0600	INFO	Secret scanning is enabled
2024-03-07T10:29:32.070-0600	INFO	If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-03-07T10:29:32.070-0600	INFO	Please see also https://aquasecurity.github.io/trivy/v0.49/docs/scanner/secret/#recommendation for faster secret detection
2024-03-07T10:29:32.070-0600	DEBUG	Enabling misconfiguration scanners: [azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan]
2024-03-07T10:29:32.078-0600	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-07T10:29:32.078-0600	DEBUG	The nuget packages directory couldn't be found. License search disabled
2024-03-07T10:29:32.078-0600	DEBUG	No secret config detected: trivy-secret.yaml
2024-03-07T10:29:32.078-0600	DEBUG	Image ID: sha256:1693ebb5dc49770846f1745850268eb136a6654a4845baa169caa5a62435d7b3
2024-03-07T10:29:32.078-0600	DEBUG	Diff IDs: [sha256:cf17226a24b25aa47bb40f6a652934c16228229a5bf81f67a8fc8866b95850b1]
2024-03-07T10:29:32.078-0600	DEBUG	Base Layers: []
2024-03-07T10:29:32.086-0600	WARN	No OS package is detected. Make sure you haven't deleted any files that contain information about the installed packages.
2024-03-07T10:29:32.086-0600	WARN	e.g. files under "/lib/apk/db/", "/var/lib/dpkg/" and "/var/lib/rpm"
2024-03-07T10:29:32.086-0600	INFO	Detected OS: ubuntu
2024-03-07T10:29:32.086-0600	INFO	Detecting Ubuntu vulnerabilities...
2024-03-07T10:29:32.086-0600	DEBUG	ubuntu: os version: 22.04
2024-03-07T10:29:32.086-0600	DEBUG	ubuntu: the number of packages: 0
2024-03-07T10:29:32.086-0600	INFO	Number of language-specific files: 2
2024-03-07T10:29:32.086-0600	INFO	Detecting gobinary vulnerabilities...
2024-03-07T10:29:32.086-0600	DEBUG	Detecting library vulnerabilities, type: gobinary, path: usr/bin/d2
2024-03-07T10:29:32.087-0600	INFO	Detecting jar vulnerabilities...
2024-03-07T10:29:32.087-0600	DEBUG	Detecting library vulnerabilities, type: jar, path:

Version

$ trivy --version
Version: 0.49.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-03-07 12:10:05.219113353 +0000 UTC
  NextUpdate: 2024-03-07 18:10:05.219113113 +0000 UTC
  DownloadedAt: 2024-03-07 16:27:59.53789 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-03-04 00:43:55.806268092 +0000 UTC
  NextUpdate: 2024-03-07 00:43:55.806267912 +0000 UTC
  DownloadedAt: 2024-03-04 17:27:27.334698 +0000 UTC

Checklist

• Read the documentation regarding wrong detection
• Ran Trivy with -f json that shows data sources and confirmed that the security advisory in data sources was correct

[DmitriyLewen]

Hello @DavidACastagna
Thanks for your report and investigation!

I created #6297 for this task.

Regards, Dmitriy