Skip to content

feat: add durable git share snapshots#64

Merged
steipete merged 1 commit into
mainfrom
codex/crawlkit-share-migration
Jun 19, 2026
Merged

feat: add durable git share snapshots#64
steipete merged 1 commit into
mainfrom
codex/crawlkit-share-migration

Conversation

@steipete

Copy link
Copy Markdown
Collaborator

Summary

  • add immutable publish --tag snapshots and non-mutating historical restores with update --ref
  • move Git repository, branch, commit, push, tag, ref, and FTS escaping mechanics onto CrawlKit
  • preserve private share-repository permissions, local unpublished branches, media restore behavior, and current search semantics
  • refresh Go dependencies

Proof

  • go test ./...
  • go test -race ./...
  • go vet ./...
  • go run golang.org/x/tools/cmd/deadcode@v0.45.0 -test ./...
  • autoreview clean

@clawsweeper

clawsweeper Bot commented Jun 19, 2026

Copy link
Copy Markdown

ClawSweeper status: review started.

I am starting a fresh review of this pull request: feat: add durable git share snapshots This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatedgithub.com/​openclaw/​crawlkit@​v0.12.2 ⏵ v0.12.3-0.20260619112528-82bf1826da3f91 +1100100100100

View full report

@socket-security

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: golang modernc.org/libc is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: ?golang/github.com/openclaw/crawlkit@v0.12.3-0.20260619112528-82bf1826da3fgolang/modernc.org/libc@v1.73.4

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/modernc.org/libc@v1.73.4. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: golang modernc.org/libc is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: ?golang/github.com/openclaw/crawlkit@v0.12.3-0.20260619112528-82bf1826da3fgolang/modernc.org/libc@v1.73.4

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore golang/modernc.org/libc@v1.73.4. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e146428a0d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/cli/app.go
return err
}
} else {
manifest, err = share.ImportAt(ctx, st, opts, *ref)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid auto-updating before restoring refs

When share.auto_update is enabled and the reader archive is stale, a.openStore(cfg) has already run autoUpdateShare, which calls share.Pull and imports the latest manifest, before this --ref branch runs. In that default configuration, slacrawl update --ref <old-tag> can fast-forward the share checkout first (and an invalid ref still leaves the DB/repo updated to latest), contradicting the advertised non-mutating historical restore. Open the store without auto-update or bypass autoUpdateShare when --ref is set.

Useful? React with 👍 / 👎.

@steipete steipete merged commit b1170f0 into main Jun 19, 2026
14 checks passed
@steipete steipete deleted the codex/crawlkit-share-migration branch June 19, 2026 11:30
@steipete

Copy link
Copy Markdown
Collaborator Author

Landed in b1170f0361213acc18d604d6d5ac99a324284e0f.

Proof: full tests, race tests, vet, dead-code analysis, and autoreview passed locally. GitHub tests, lint, dependency checks, Docker, release snapshot, CodeQL, secret scanning, and Socket security passed on the merged head.

@steipete

Copy link
Copy Markdown
Collaborator Author

Post-merge verification completed against b1170f0361213acc18d604d6d5ac99a324284e0f.

  • Built the exact CLI and used a fresh temporary bare remote. Published three changed snapshots with three immutable tags on a non-default branch; duplicate-tag reuse was rejected.
  • Verified the remote stayed private (0700), publisher/subscriber repositories used 0750, and both checkouts stayed on the configured branch.
  • A fresh subscriber imported latest, then restored by tag, restored by commit, and returned to latest with expected synthetic cardinalities 3 -> 1 -> 2 -> 3. Historical restores left branch, HEAD, manifest, and working tree unchanged.
  • Ran archive-backed status, sql, messages, channels, users, and metadata smoke tests with every Slack/API source disabled. Private archive payload and values were suppressed.
  • Focused tests, full tests, race tests, vet, deadcode, module verification, vulnerability scan, Docker, CodeQL, secret scanning, and snapshot release build all passed.

The audit found one bounded omission: generated Bash/Zsh completions lacked the new flags. Fixed and landed in #65 (d1421a90f70bbba3cf31f60773c7b3deb8e1025a). Exact merged-head CI, Docker, CodeQL, and secret scanning are green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants