Integrate CLIPV2 build pipeline

[luke-lombardi]

---

Summary by cubic

Integrates CLIP v2: builds images with buildah and publishes index-only .clip archives, skipping the runc build flow. Adds end-to-end OCI credentials for private layers so builds and workers can pull v2 image content securely.

• New Features

  • Added CLIP v2 pipeline: render Dockerfile, build with buildah, push to a build registry, index via CreateFromOCIImage, and skip runc for build containers.
  • V2 Dockerfile rendering with standard pip (no uv-b9); coalesces pip/mamba steps; respects custom Dockerfile FROM and uses provided creds for authenticated base pulls; skips Python install for beta9 base images.
  • Deterministic image IDs updated for v2 (Dockerfile included); separate base image ID for v1.
  • Worker reads container config (env, user, workdir, entrypoint/cmd) from CLIP archive metadata for v2 and writes initial_config.json; caches source image refs; no runtime skopeo.
  • Pull path handles OCI vs S3 .clip formats; canonicalizes v2 index path; S3 fallback if local store miss.
  • Config: default clipVersion=2, new buildRegistry and buildRegistryInsecure flags; improved TLS handling for localhost registries; enable blobcache by default.
  • Ensures /workspace and /volumes exist via the overlay upper layer; hardens overlay mounts with fuse-overlayfs fallback.
  • Pass OCI registry credentials at runtime (Linear 7278): create workspace secrets during build, store on the image, scheduler attaches image_credentials, worker uses them for lazy layer pulls.
  • V2 builds support ENV lines and build secrets (as build-args) in generated/appended Dockerfiles and pass them to buildah.

• Migration

  • Ensure buildah and skopeo are installed on workers.
  • Set imageService.buildRegistry (and buildRegistryInsecure for localhost) so build/push/index works.
  • Run migration 036_add_image_credential_fields to add image credential secret references.
  • If forcing legacy behavior, set imageService.clipVersion to 1.

^{Written for commit 8001876. Summary will update automatically on new commits.}

---

Note

Integrates CLIP v2: buildah-based Dockerfile builds producing index-only archives with end-to-end OCI credentials, updated image IDs, worker/runtime support, and defaults clipVersion to 2.

• Image Build (CLIP v2):
  • Build pipeline via buildah with rendered Dockerfiles; push to build registry; create index-only .clip via CreateFromOCIImage; skip runc command execution for builds.
  • V2 Dockerfile rendering (standard pip/mamba coalescing, env/secrets, beta9 base detection); updated image ID hashing (Dockerfile/context for v2).
• OCI Credentials:
  • New pkg/registry credential utilities and providers; convert user creds; create workspace secrets; persist on images; scheduler attaches image_credentials to requests; worker uses for lazy pulls.
• Worker Runtime:
  • Pull/mount supports OCI index vs S3 archives; canonical v2 index path; derive spec (env/cmd/user/cwd) from CLIP metadata; overlay ensures /workspace and /volumes.
  • Build-or-pull: buildah + push + index for Dockerfile builds; source image indexing path.
• Services/APIs:
  • RuncImageService → ContainerImageService; verify/build flow updated for v2; proto/types add image_credentials.
• Config/Defaults:
  • Default clipVersion: 2; add buildRegistry and buildRegistryInsecure; enable blobcache; improved skopeo TLS for localhost.
• Repository/DB:
  • New image credential fields and accessors; migration 036_add_image_credential_fields; secret create/update signatures adjusted.
• Deps/Tests:
  • Add container registry/docker deps; remove old image credentials; extensive new unit tests across build, ID, builder, worker.

^{Written by Cursor Bugbot for commit 8001876. This will update automatically on new commits. Configure here.}

[cubic-dev-ai]

Reviewed changes from recent commits (found 5 issues).

5 issues found across 19 files

Prompt for AI agents (all 5 issues)

Understand the root cause of the following 5 issues and fix them.


<file name="pkg/worker/image.go">

<violation number="1" location="pkg/worker/image.go:329">
Rule violated: **Prevent Redundant Code Duplication**

The new credential-provider setup repeats the same SourceImageCreds parsing logic that you just added in createOCIImageWithProgress. Please extract this shared flow into a helper so both code paths reuse it and stay consistent with the Prevent Redundant Code Duplication guideline.</violation>
</file>

<file name="pkg/abstractions/image/image.go">

<violation number="1" location="pkg/abstractions/image/image.go:146">
Rule violated: **Prevent Redundant Code Duplication**

This block duplicates the credential-conversion logic already handled inside BuildOpts.setCustomImageBuildOptions. Because initializeBuildConfiguration ultimately calls that helper, the new code repeats the same work and any future change to credential handling must be updated in two places, violating the no-duplication guideline.</violation>

<violation number="2" location="pkg/abstractions/image/image.go:502">
Using context.Background() here (and on SetImageCredentialSecret just below) drops request cancellation, deadlines, and tracing metadata. Please keep the original ctx so secret persistence respects the caller’s lifecycle.</violation>
</file>

<file name="pkg/scheduler/scheduler.go">

<violation number="1" location="pkg/scheduler/scheduler.go:410">
Logging the decrypted OCI credentials exposes sensitive registry secrets in plaintext logs. Remove or redact the value before logging.</violation>
</file>

<file name="pkg/registry/credentials.go">

<violation number="1" location="pkg/registry/credentials.go:47">
ParseCredentialsFromEnv drops DOCKERHUB_USERNAME/DOCKERHUB_PASSWORD, so Docker Hub auth fails when credentials come from env. Please add these keys to knownCredKeys.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

Rule violated: Prevent Redundant Code Duplication

The new credential-provider setup repeats the same SourceImageCreds parsing logic that you just added in createOCIImageWithProgress. Please extract this shared flow into a helper so both code paths reuse it and stay consistent with the Prevent Redundant Code Duplication guideline.

Prompt for AI agents

Address the following comment on pkg/worker/image.go at line 329:

<comment>The new credential-provider setup repeats the same SourceImageCreds parsing logic that you just added in createOCIImageWithProgress. Please extract this shared flow into a helper so both code paths reuse it and stay consistent with the Prevent Redundant Code Duplication guideline.</comment>

<file context>
@@ -325,33 +324,36 @@ func (c *ImageClient) PullLazy(ctx context.Context, request *types.ContainerRequ
-				Msg(&quot;no image credentials provided, using default provider chain&quot;)
+		var credProvider clipCommon.RegistryCredentialProvider
+
+		if request.ImageCredentials != &quot;&quot; {
+			// Runtime container: credentials already in JSON format from secret
+			credProvider = c.createCredentialProvider(ctx, request.ImageCredentials, imageId)
</file context>

[cubic-dev-ai]

Rule violated: Prevent Redundant Code Duplication

This block duplicates the credential-conversion logic already handled inside BuildOpts.setCustomImageBuildOptions. Because initializeBuildConfiguration ultimately calls that helper, the new code repeats the same work and any future change to credential handling must be updated in two places, violating the no-duplication guideline.

Prompt for AI agents

Address the following comment on pkg/abstractions/image/image.go at line 146:

<comment>This block duplicates the credential-conversion logic already handled inside BuildOpts.setCustomImageBuildOptions. Because initializeBuildConfiguration ultimately calls that helper, the new code repeats the same work and any future change to credential handling must be updated in two places, violating the no-duplication guideline.</comment>

<file context>
@@ -138,9 +137,21 @@ func (is *RuncImageService) BuildImage(in *pb.BuildImageRequest, stream pb.Image
 	buildOptions.ClipVersion = clipVersion
 
+	// Process credentials for custom base image (if provided)
+	if buildOptions.ExistingImageUri != &quot;&quot; &amp;&amp; len(buildOptions.ExistingImageCreds) &gt; 0 {
+		baseImageCreds, err := reg.GetRegistryTokenForImage(buildOptions.ExistingImageUri, buildOptions.ExistingImageCreds)
+		if err != nil {
</file context>

[cubic-dev-ai]

Logging the decrypted OCI credentials exposes sensitive registry secrets in plaintext logs. Remove or redact the value before logging.

Prompt for AI agents

Address the following comment on pkg/scheduler/scheduler.go at line 410:

<comment>Logging the decrypted OCI credentials exposes sensitive registry secrets in plaintext logs. Remove or redact the value before logging.</comment>

<file context>
@@ -407,6 +407,7 @@ func (s *Scheduler) attachImageCredentials(request *types.ContainerRequest) erro
 		Str(&quot;image_id&quot;, request.ImageId).
 		Str(&quot;secret_name&quot;, secretName).
 		Int(&quot;credentials_length&quot;, len(secret.Value)).
+		Str(&quot;credentials&quot;, secret.Value).
 		Msg(&quot;attached OCI credentials&quot;)
 
</file context>

✅ Addressed in dce97cf

[cubic-dev-ai]

Using context.Background() here (and on SetImageCredentialSecret just below) drops request cancellation, deadlines, and tracing metadata. Please keep the original ctx so secret persistence respects the caller’s lifecycle.

Prompt for AI agents

Address the following comment on pkg/abstractions/image/image.go at line 502:

<comment>Using context.Background() here (and on SetImageCredentialSecret just below) drops request cancellation, deadlines, and tracing metadata. Please keep the original ctx so secret persistence respects the caller’s lifecycle.</comment>

<file context>
@@ -464,40 +488,32 @@ func (is *RuncImageService) createCredentialSecretIfNeeded(ctx context.Context,
+
 	// Create or update secret
-	secret, err := is.upsertSecret(ctx, authInfo, secretName, secretValue, registry)
+	secret, err := is.upsertSecret(context.Background(), authInfo, secretName, secretValue, registry)
 	if err != nil {
 		return err
</file context>

Suggested change

	secret, err := is.upsertSecret(context.Background(), authInfo, secretName, secretValue, registry)
	secret, err := is.upsertSecret(ctx, authInfo, secretName, secretValue, registry)

[cubic-dev-ai]

ParseCredentialsFromEnv drops DOCKERHUB_USERNAME/DOCKERHUB_PASSWORD, so Docker Hub auth fails when credentials come from env. Please add these keys to knownCredKeys.

Prompt for AI agents

Address the following comment on pkg/registry/credentials.go at line 47:

<comment>ParseCredentialsFromEnv drops DOCKERHUB_USERNAME/DOCKERHUB_PASSWORD, so Docker Hub auth fails when credentials come from env. Please add these keys to knownCredKeys.</comment>

<file context>
@@ -0,0 +1,709 @@
+	&quot;AZURE_CLIENT_SECRET&quot;:            true,
+	&quot;AZURE_TENANT_ID&quot;:                true,
+	&quot;DOCKER_USERNAME&quot;:                true,
+	&quot;DOCKER_PASSWORD&quot;:                true,
+	&quot;REGISTRY_USERNAME&quot;:              true,
+	&quot;REGISTRY_PASSWORD&quot;:              true,
</file context>

[cubic-dev-ai]

No issues found across 3 files

[cubic-dev-ai]

No issues found across 2 files

[cubic-dev-ai]

No issues found across 2 files

[cubic-dev-ai]

No issues found across 6 files

[cubic-dev-ai]

No issues found across 7 files

[cubic-dev-ai]

No issues found across 7 files

[cubic-dev-ai]

No issues found across 2 files

[cursor]

Bug: Plaintext logging of OCI credentials in attachImageCredentials

The attachImageCredentials function logs sensitive OCI credentials in plaintext, exposing authentication secrets in application logs and creating a security vulnerability.

 

[cursor]

Bug: Global env writes cause race condition in credentials

The CreateProviderFromEnv function uses os.Setenv for AWS credentials, which are process-global. This introduces a race condition where concurrent requests can overwrite credentials, leading to leakage or authentication failures.

 

[cubic-dev-ai]

2 issues found across 7 files

Prompt for AI agents (all 2 issues)

Understand the root cause of the following 2 issues and fix them.


<file name="pkg/worker/runc_server.go">

<violation number="1" location="pkg/worker/runc_server.go:345">
Rule violated: **Prevent Redundant Code Duplication**

The new metadata-to-spec block duplicates the existing buildSpecFromCLIPMetadata logic (env/cwd/user/entrypoint/cmd) instead of reusing a shared helper, violating the Prevent Redundant Code Duplication guideline. Please extract or reuse the existing helper so metadata field handling stays consistent across workers.</violation>
</file>

<file name="pkg/worker/lifecycle.go">

<violation number="1" location="pkg/worker/lifecycle.go:406">
Rule violated: **Prevent Redundant Code Duplication**

buildSpecFromCLIPMetadata duplicates the CLIP metadata-to-spec mapping that already exists in writeInitialSpecFromImage; please extract a shared helper to avoid divergence across worker entrypoints.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

Rule violated: Prevent Redundant Code Duplication

The new metadata-to-spec block duplicates the existing buildSpecFromCLIPMetadata logic (env/cwd/user/entrypoint/cmd) instead of reusing a shared helper, violating the Prevent Redundant Code Duplication guideline. Please extract or reuse the existing helper so metadata field handling stays consistent across workers.

Prompt for AI agents

Address the following comment on pkg/worker/runc_server.go at line 345:

<comment>The new metadata-to-spec block duplicates the existing buildSpecFromCLIPMetadata logic (env/cwd/user/entrypoint/cmd) instead of reusing a shared helper, violating the Prevent Redundant Code Duplication guideline. Please extract or reuse the existing helper so metadata field handling stays consistent across workers.</comment>

<file context>
@@ -334,45 +334,36 @@ func (s *RunCServer) RunCArchive(req *pb.RunCArchiveRequest, stream pb.RunCServi
-			// Fallback to legacy Env field if Config is not available
-			spec.Process.Env = append(spec.Process.Env, imgMeta.Env...)
+	// Try to get CLIP metadata from archive (v2 images only)
+	clipMeta, ok := s.imageClient.GetCLIPImageMetadata(instance.Request.ImageId)
+	if ok {
+		log.Info().Str(&quot;image_id&quot;, instance.Request.ImageId).Msg(&quot;using v2 image metadata from clip archive for initial spec&quot;)
</file context>

[cubic-dev-ai]

Rule violated: Prevent Redundant Code Duplication

buildSpecFromCLIPMetadata duplicates the CLIP metadata-to-spec mapping that already exists in writeInitialSpecFromImage; please extract a shared helper to avoid divergence across worker entrypoints.

Prompt for AI agents

Address the following comment on pkg/worker/lifecycle.go at line 406:

<comment>buildSpecFromCLIPMetadata duplicates the CLIP metadata-to-spec mapping that already exists in writeInitialSpecFromImage; please extract a shared helper to avoid divergence across worker entrypoints.</comment>

<file context>
@@ -381,85 +382,49 @@ func (s *Worker) readBundleConfig(request *types.ContainerRequest) (*specs.Spec,
-func (s *Worker) buildSpecFromImageMetadata(imgMeta *common.ImageMetadata) *specs.Spec {
+// buildSpecFromCLIPMetadata constructs an OCI spec from CLIP image metadata
+// This is the primary path for v2 images with embedded metadata
+func (s *Worker) buildSpecFromCLIPMetadata(clipMeta *clipCommon.ImageMetadata) *specs.Spec {
 	spec := specs.Spec{
 		Process: &amp;specs.Process{
</file context>

[cursor]

Bug: Plaintext logging of sensitive OCI credentials vulnerabilities

The attachImageCredentials function logs OCI registry credentials in plaintext, including sensitive values like passwords or tokens. This exposes confidential information in the application logs, creating a security vulnerability.

 

[cursor]

Bug: Global env mutation risks credential leakage in parallel

The CreateProviderFromEnv function sets AWS, GCP, and Azure credentials using os.Setenv(). As environment variables are process-global, this introduces a race condition and potential credential leakage in concurrent or multi-tenant environments, allowing credentials from one request to leak to another.

 

[cubic-dev-ai]

Reviewed changes from recent commits (found 1 issue).

1 issue found across 2 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="pkg/worker/image.go">

<violation number="1" location="pkg/worker/image.go:729">
SourceImageCreds can arrive in JSON form (see createOCIImageWithProgress), but buildah --creds only accepts username[:password]. Passing the JSON string here breaks pulls when structured credentials are provided. Please deserialize and convert to a user:pass tuple (or use an authfile) before invoking buildah.</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

SourceImageCreds can arrive in JSON form (see createOCIImageWithProgress), but buildah --creds only accepts username[:password]. Passing the JSON string here breaks pulls when structured credentials are provided. Please deserialize and convert to a user:pass tuple (or use an authfile) before invoking buildah.

Prompt for AI agents

Address the following comment on pkg/worker/image.go at line 729:

<comment>SourceImageCreds can arrive in JSON form (see createOCIImageWithProgress), but buildah --creds only accepts username[:password]. Passing the JSON string here breaks pulls when structured credentials are provided. Please deserialize and convert to a user:pass tuple (or use an authfile) before invoking buildah.</comment>

<file context>
@@ -723,6 +723,11 @@ func (c *ImageClient) BuildAndArchiveImage(ctx context.Context, outputLogger *sl
+		
+		// Add credentials if provided (in username:password format)
+		if request.BuildOptions.SourceImageCreds != &quot;&quot; {
+			pullArgs = append(pullArgs, &quot;--creds&quot;, request.BuildOptions.SourceImageCreds)
+		}
 
</file context>

[cubic-dev-ai]

No issues found across 2 files