Add comprehensive URL depth filtering system #1353

[Vinayakp2001]

Implements URL filtering based on structural characteristics:

Core Features:

• Path depth filtering (-cpd): Filter URLs by number of path segments
• Query parameter filtering (-cqp): Filter URLs by query parameter count
• Subdomain depth filtering (-csd): Filter URLs by subdomain levels
• Flexible operators: >=, <=, ==, >, < for precise control
• Range syntax: Support for ranges like '2-5', '1-3' for flexible filtering
• Logic operators: --depth-filter-or flag for OR logic between filter types

Enhanced User Experience:

• Comprehensive error messages with specific guidance and examples
• Intuitive CLI integration following existing Katana patterns
• Performance optimizations with intelligent caching system
• Seamless integration with existing filters (regex, extension, scope)

Technical Implementation:

• Thread-safe caching for URL component analysis
• Efficient parsing and validation with detailed error reporting
• Memory-optimized design for large-scale crawling
• Comprehensive test coverage including unit, integration, and performance tests

Examples:
katana -u target.com -cpd '>=2' -cqp '1-3' # URLs with 2+ path depth and 1-3 params
katana -u target.com --depth-filter-or -cpd '>=4' -csd '>=1' # Deep paths OR subdomains

#1353

Summary by CodeRabbit

• New Features
  • Added depth-based URL filtering flags: --count-path-depth, --count-query-params, and --count-subdomain-depth. Supports comparisons (==, >=, <=, >, <) and ranges (e.g., 2-5).
  • Added --depth-filter-or to switch combination logic from AND to OR across depth filters.
  • Filters are validated at startup with clear error messages on invalid syntax.

[coderabbitai]

Walkthrough

Introduces depth-based URL filtering across path depth, query parameter count, and subdomain depth. Adds new CLI flags and corresponding options, integrates a DepthFilterValidator with caching, validates filters during option parsing, applies filtering in output flow before regex/DSL evaluation, and provides comprehensive tests for parsing, counting, and logic modes (AND/OR).

Changes

Cohort / File(s)	Summary of changes
CLI flags & options wiring cmd/katana/main.go, pkg/types/options.go, pkg/output/options.go	Adds CLI flags and corresponding public fields: CountPathDepth, CountQueryParams, CountSubdomainDepth, DepthFilterOrLogic. Extends exported Options types to carry these settings.
Runner validation internal/runner/options.go	Imports filters package; validates depth-related filter strings via ValidateAndSuggest during options validation, returning errors on invalid inputs.
Output integration pkg/output/output.go	Imports filters and net/url; adds depthValidator to StandardWriter; initializes it when depth filters are set; in filterOutput, parses URL and applies depth validation prior to existing regex/DSL filters.
Crawler options changes pkg/types/crawler_options.go	Adds DepthValidator field to CrawlerOptions; initializes when depth filters provided; notes that path extension validation is separate and depth filtering occurs later in output.
Depth filtering subsystem pkg/utils/filters/depth_filter.go	Implements DepthFilterValidator with parsing, evaluation (AND/OR), and caching. Adds parsing utilities, URL component counters, evaluation helpers, cache structures, and validation/suggestion API.
Tests for depth filtering pkg/utils/filters/depth_filter_test.go	Adds tests for filter parsing, component counting, AND/OR logic evaluation, and various valid/invalid scenarios.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant U as User
  participant CLI as cmd/katana
  participant R as Runner (options)
  participant O as Output.New (StandardWriter)
  participant F as filters.DepthFilterValidator

  U->>CLI: Run with depth flags
  CLI->>R: Build Options (CountPathDepth, ...)
  R->>F: ValidateAndSuggest(...) per filter
  R-->>CLI: Validation result (ok/error)
  CLI->>O: Initialize writer with Options
  O->>F: NewDepthFilterValidator(..., useOrLogic)
  F-->>O: Validator (or error)

Loading

sequenceDiagram
  autonumber
  participant C as Crawler/Emitter
  participant O as Output.filterOutput
  participant P as net/url
  participant F as DepthFilterValidator
  participant RX as Regex/DSL Filters

  C-->>O: Event(URL)
  O->>P: Parse URL
  alt parse error
    O-->>C: Filter out (true)
  else parsed
    O->>F: ValidateURL(parsed)
    alt fails depth validation
      O-->>C: Filter out (true)
    else passes
      O->>RX: Apply existing filters
      RX-->>O: Pass/Fail
      O-->>C: Final decision
    end
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

I hop through paths, count steps with care,
Subdomains stack like clouds in air.
Queries chirp—how many? Let’s see—
OR or AND, decide for me.
With caches warm and flags so neat,
I filter finds with thumpy feet. 🐇✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 61.11% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title clearly and succinctly summarizes the primary change by stating the addition of a URL depth filtering system, which aligns directly with the PR’s objectives and conveys the main feature introduced.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (7)

internal/runner/options.go (1)

63-87: Early depth-filter validation is solid; consider small DRY/UX tweaks

• Factor the three loops via a tiny helper to reduce repetition.
• Optionally strings.TrimSpace before validation for friendlier UX on inputs with stray spaces.

pkg/types/crawler_options.go (1)

40-41: Avoid double creation/caching of DepthFilterValidator

This validator is created here and again inside pkg/output/output.go. Prefer a single instance to avoid duplicate parsing and cache memory. Consider passing DepthValidator via output.Options (or constructing only in one place and reusing).

Also applies to: 147-148

pkg/output/output.go (2)

91-104: Reuse existing validator to avoid duplicate parsing/caching

A DepthFilterValidator is also created in pkg/types/crawler_options.go. Prefer injecting that instance via Options to avoid double work and memory.

---

373-388: Optional: add debug log on URL-parse failures before filtering out

Currently parse errors silently drop results. A debug-level log would aid troubleshooting malformed URLs without noisy output.

Example:

if err != nil {
    gologger.Debug().Msgf("Depth filter: failed to parse URL %q: %v", event.Request.URL, err)
    return true
}

pkg/utils/filters/depth_filter.go (3)

466-474: Avoid deprecated strings.Title usage.

strings.Title is deprecated. Use simple first-letter capitalization to avoid new deps.

- return fmt.Errorf("❌ %s filter validation failed:\n%w", 
-   strings.Title(filterType), err)
+ title := filterType
+ if len(title) > 0 {
+   title = strings.ToUpper(title[:1]) + title[1:]
+ }
+ return fmt.Errorf("❌ %s filter validation failed:\n%w", title, err)

---

170-218: Micro-optimization: precompile regexes.

parseDepthFilter recompiles regex each call. Precompile at package level to reduce allocations during option parsing.

- // inside parseDepthFilter
- rangeRe := regexp.MustCompile(`^(\d+)-(\d+)$`)
+ // at package scope
+ var (
+   rangeRe = regexp.MustCompile(`^(\d+)-(\d+)$`)
+   compRe  = regexp.MustCompile(`^(>=|<=|==|>|<)(\d+)$`)
+ )
...
- re := regexp.MustCompile(`^(>=|<=|==|>|<)(\d+)$`)
- matches := re.FindStringSubmatch(filter)
+ matches := compRe.FindStringSubmatch(filter)

---

368-376: Cache eviction is coarse; consider bounded/LRU behavior.

Clearing entire maps when urlComponents reaches maxSize can cause cache thrash and keeps filterResults growing independently. Consider:

• Track size of filterResults and bound it similarly.
• Implement simple LRU (list + map) or random eviction instead of full reset.

Also applies to: 398-406

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 64a99ad and 821f5df.

📒 Files selected for processing (8)

• cmd/katana/main.go (1 hunks)
• internal/runner/options.go (2 hunks)
• pkg/output/options.go (1 hunks)
• pkg/output/output.go (5 hunks)
• pkg/types/crawler_options.go (5 hunks)
• pkg/types/options.go (1 hunks)
• pkg/utils/filters/depth_filter.go (1 hunks)
• pkg/utils/filters/depth_filter_test.go (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (4)

internal/runner/options.go (1)

pkg/utils/filters/depth_filter.go (1)

• ValidateAndSuggest (467-474)

pkg/utils/filters/depth_filter_test.go (1)

pkg/utils/filters/depth_filter.go (2)

• DepthFilter (14-18)
• NewDepthFilterValidator (47-90)

pkg/types/crawler_options.go (1)

pkg/utils/filters/depth_filter.go (2)

• DepthFilterValidator (38-44)
• NewDepthFilterValidator (47-90)

pkg/output/output.go (2)

pkg/utils/filters/depth_filter.go (2)

• DepthFilterValidator (38-44)
• NewDepthFilterValidator (47-90)

pkg/navigation/request.go (1)

• Request (12-25)

🔇 Additional comments (4)

pkg/types/crawler_options.go (1)

174-183: Good: early extension check, depth deferred to output

Short-circuiting on extension improves performance; deferring depth filtering to output keeps discovery intact.

pkg/output/options.go (1)

31-34: LGTM: clean API for depth filters

Fields mirror CLI and keep output decoupled from goflags types.

pkg/types/options.go (1)

45-52: LGTM: options surface for depth filters is clear and aligned with CLI

Comments and naming are consistent.

cmd/katana/main.go (1)

177-181: Options fields and wiring verified All new flags map to Options struct fields and are consumed by the depth‐filter validator in both crawler and output setup.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Compile-time type mismatch: convert goflags.StringSlice to []string

output.Options fields are []string, while options.Count* are goflags.StringSlice. Add explicit conversions to avoid build errors.

-        CountPathDepth:        options.CountPathDepth,
-        CountQueryParams:      options.CountQueryParams,
-        CountSubdomainDepth:   options.CountSubdomainDepth,
+        CountPathDepth:        []string(options.CountPathDepth),
+        CountQueryParams:      []string(options.CountQueryParams),
+        CountSubdomainDepth:   []string(options.CountSubdomainDepth),
         DepthFilterOrLogic:    options.DepthFilterOrLogic,

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	CountPathDepth: options.CountPathDepth,
	CountQueryParams: options.CountQueryParams,
	CountSubdomainDepth: options.CountSubdomainDepth,
	DepthFilterOrLogic: options.DepthFilterOrLogic,
	}
	CountPathDepth: []string(options.CountPathDepth),
	CountQueryParams: []string(options.CountQueryParams),
	CountSubdomainDepth: []string(options.CountSubdomainDepth),
	DepthFilterOrLogic: options.DepthFilterOrLogic,

🤖 Prompt for AI Agents

In pkg/types/crawler_options.go around lines 99 to 103, output.Options fields
(CountPathDepth, CountQueryParams, CountSubdomainDepth, DepthFilterOrLogic) are
typed as []string but the assigned values are goflags.StringSlice, causing a
compile-time type mismatch; fix by converting each goflags.StringSlice to a
[]string before assignment (e.g., call a conversion helper or use
[]string(someStringSlice) / someStringSlice.ToSlice() as appropriate) so each
field receives a plain []string.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Also convert goflags.StringSlice when constructing the validator

filters.NewDepthFilterValidator takes []string slices; pass converted values.

-        depthValidator, err = filters.NewDepthFilterValidator(
-            options.CountPathDepth,
-            options.CountQueryParams,
-            options.CountSubdomainDepth,
-            options.DepthFilterOrLogic,
-        )
+        depthValidator, err = filters.NewDepthFilterValidator(
+            []string(options.CountPathDepth),
+            []string(options.CountQueryParams),
+            []string(options.CountSubdomainDepth),
+            options.DepthFilterOrLogic,
+        )

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	depthValidator, err = filters.NewDepthFilterValidator(
	options.CountPathDepth,
	options.CountQueryParams,
	options.CountSubdomainDepth,
	options.DepthFilterOrLogic,
	)
	depthValidator, err = filters.NewDepthFilterValidator(
	[]string(options.CountPathDepth),
	[]string(options.CountQueryParams),
	[]string(options.CountSubdomainDepth),
	options.DepthFilterOrLogic,
	)

🤖 Prompt for AI Agents

In pkg/types/crawler_options.go around lines 128-133, the call to
filters.NewDepthFilterValidator is passing goflags.StringSlice values directly;
convert those to plain []string slices before passing them. Replace the
arguments options.CountPathDepth, options.CountQueryParams and
options.CountSubdomainDepth with their []string equivalents (perform a simple
conversion or extract the underlying slice) so NewDepthFilterValidator receives
[]string types.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

countQueryParams should ignore bare keys (test exposes bug).

Case "user=admin&empty&pass=secret" expects 2 by ignoring "empty" (no '='); current implementation counts it as 3. Fix countQueryParams to only count pairs with a non-empty key before '='. See suggested fix in pkg/utils/filters/depth_filter.go Lines 326-344.

🤖 Prompt for AI Agents

In pkg/utils/filters/depth_filter_test.go around lines 110 to 121: the test
shows countQueryParams currently counts bare keys like "empty" in
"user=admin&empty&pass=secret" but should ignore them; update the implementation
in pkg/utils/filters/depth_filter.go (around lines 326-344) so the function
splits on '&', then for each part checks that it contains '=' and that the
substring before '=' is non-empty (trimmed) before incrementing the count;
ensure you skip parts without '=' or with an empty key and add/adjust unit tests
if needed.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Fix: don’t count bare keys as query parameters.

"empty" (no '=') must be ignored. Count only tokens containing '=' with a non-empty key. This aligns with tests and real URL semantics.

Apply this diff:

 func countQueryParams(query string) int {
   if query == "" {
     return 0
   }
 
   // Split by & and count valid parameters
   params := strings.Split(query, "&")
   count := 0
   for _, param := range params {
-    param = strings.TrimSpace(param)
-    // Count parameters that have a key (with or without value)
-    if param != "" && !strings.HasPrefix(param, "=") {
-      count++
-    }
+    param = strings.TrimSpace(param)
+    if param == "" {
+      continue
+    }
+    // Count only key[=value] where key is non-empty
+    if i := strings.IndexByte(param, '='); i > 0 {
+      // key is param[:i]; allow empty value
+      count++
+    }
   }
 
   return count
 }

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	// countQueryParams counts the number of query parameters in a query string
	func countQueryParams(query string) int {
	if query == "" {
	return 0
	}
	// Split by & and count valid parameters
	params := strings.Split(query, "&")
	count := 0
	for _, param := range params {
	param = strings.TrimSpace(param)
	// Count parameters that have a key (with or without value)
	if param != "" && !strings.HasPrefix(param, "=") {
	count++
	}
	}
	return count
	}
	// countQueryParams counts the number of query parameters in a query string
	func countQueryParams(query string) int {
	if query == "" {
	return 0
	}
	// Split by & and count valid parameters
	params := strings.Split(query, "&")
	count := 0
	for _, param := range params {
	param = strings.TrimSpace(param)
	if param == "" {
	continue
	}
	// Count only key[=value] where key is non-empty
	if i := strings.IndexByte(param, '='); i > 0 {
	// key is param[:i]; allow empty value
	count++
	}
	}
	return count
	}

🤖 Prompt for AI Agents

In pkg/utils/filters/depth_filter.go around lines 326 to 344, the current
countQueryParams function incorrectly counts bare keys (tokens without '=') as
parameters; update it to only count tokens that contain an '=' and have a
non-empty key (i.e., the '=' must exist and its index must be > 0). Trim the
token, skip empty tokens, ensure strings.Contains(param, "=") is true, and
verify the substring before '=' is non-empty before incrementing the count.