Added srb and srbd parameters to support storing only the response bo… #1446
Conversation
WalkthroughAdds support for storing raw HTTP response bodies: new options and CLI flags, wiring through initialization, auto-enable logic when a body directory is provided, response struct path tracking, and writer logic to create/validate directories and write body files. Changes
Sequence DiagramsequenceDiagram
participant User
participant CLI as CLI Parser
participant Runner as Runner Options
participant Crawler as Crawler Init
participant Output as Output Writer
participant FS as File System
participant Response as Response Struct
User->>CLI: provide --store-response-body(-dir)
CLI->>Runner: readFlags()
Runner->>Runner: validateOptions()
alt StoreResponseBodyDir set && not StoreResponseBody
Runner->>Runner: enable StoreResponseBody
end
Runner->>Crawler: NewCrawlerOptions (includes StoreResponseBody)
Crawler->>Output: init writer with storeResponseBody config
rect rgb(255,245,235)
Output->>FS: create/validate storeResponseBodyDir (respect NoClobber)
end
Note over Output,Response: On each write when enabled
Output->>FS: write response body file
FS-->>Output: file path
Output->>Response: set StoredResponseBodyPath
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧬 Code graph analysis (1)pkg/navigation/response.go (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (5)
pkg/types/options.go (1)
135-138: Docs polish for new fields (capitalize HTTP, pluralize “body”).Small comment grammar tweaks improve CLI/docs consistency.
- // StoreResponseBody specifies if katana should store http responses body + // StoreResponseBody specifies if katana should store HTTP response bodies - // StoreResponseBodyDir specifies if katana should use a custom directory to store http responses body + // StoreResponseBodyDir specifies a custom directory to store HTTP response bodiesinternal/runner/options.go (1)
47-50: Auto-enable behavior matches ‘sr’; drop trailing newline in log.Functionality is good. The debug message includes an unnecessary “\n”.
- gologger.Debug().Msgf("store response body directory specified, enabling \"srb\" flag automatically\n") + gologger.Debug().Msgf("store response body directory specified, enabling \"srb\" flag automatically")cmd/katana/main.go (1)
229-231: Help text grammar (plural, HTTP, article).Clarify flag descriptions for user-facing help.
- flagSet.BoolVarP(&options.StoreResponseBody, "store-response-body", "srb", false, "store http responses body"), - flagSet.StringVarP(&options.StoreResponseBodyDir, "store-response-body-dir", "srbd", "", "store http responses body to custom directory"), + flagSet.BoolVarP(&options.StoreResponseBody, "store-response-body", "srb", false, "store HTTP response bodies"), + flagSet.StringVarP(&options.StoreResponseBodyDir, "store-response-body-dir", "srbd", "", "store HTTP response bodies to a custom directory"),pkg/output/output.go (2)
225-236: Clarify error messages: say “response body”.These errors belong to the body-only storage path—message should reflect that.
- if err := fileWriter.Write([]byte(result.Response.Body)); err != nil { - return errkit.Wrap(err, "output: could not store response") + if err := fileWriter.Write([]byte(result.Response.Body)); err != nil { + return errkit.Wrap(err, "output: could not store response body")
152-164: Parity with response index (optional).You create an index.txt for StoreResponse but not for StoreResponseBody. If index is intended, consider adding it for the body dir too; if not used, consider removing it from both for consistency.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
cmd/katana/main.go(1 hunks)internal/runner/options.go(1 hunks)pkg/navigation/response.go(1 hunks)pkg/output/options.go(1 hunks)pkg/output/output.go(5 hunks)pkg/types/crawler_options.go(1 hunks)pkg/types/options.go(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
pkg/output/output.go (1)
pkg/navigation/response.go (1)
Response(30-45)
🪛 GitHub Actions: 🔨 Build Test
pkg/navigation/response.go
[error] 43-44: govet: structtag: struct field StoredResponseBodyPath repeats json tag 'stored_response_path' (also at response.go:43).
🪛 GitHub Check: Lint
pkg/navigation/response.go
[failure] 44-44:
structtag: struct field StoredResponseBodyPath repeats json tag "stored_response_path" also at response.go:43 (govet)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
- GitHub Check: Functional Test (windows-latest)
- GitHub Check: Functional Test (macOS-latest)
- GitHub Check: Functional Test (ubuntu-latest)
- GitHub Check: release-test-mac
- GitHub Check: release-test-linux
- GitHub Check: Analyze (go)
🔇 Additional comments (2)
pkg/output/options.go (1)
15-15: Wiring LGTM.Fields align with types.Options and writer usage.
Also applies to: 23-23
pkg/types/crawler_options.go (1)
81-82: Propagation LGTM.StoreResponseBody and StoreResponseBodyDir correctly flow into output.Options.
Also applies to: 86-87
1 participant
#1445
Summary by CodeRabbit