Skip to content

Complete native request backend parity #1588

Description

@maurosoria

The native request backend now has executable CI coverage plus GET scanning, request headers/static cookies, retries and timeouts, advanced filters, HTTP/HTTPS proxies with credentials, replay proxy support, redirect following, bounded normal-response buffering, and CPU-based Tokio worker sizing.

Merged foundation:

P0: scan control and resource bounds

  • Return native results incrementally instead of materializing every task and response for a batch.
  • Add cooperative cancellation from Python into Rust.
  • Make pause, skip, quit, --max-time, and --target-max-time interrupt an active native batch.
  • Keep the Tokio runtime and reqwest clients alive across batches instead of recreating them.
  • Bound the raw HTTP fallback reader before read_to_end.
  • Decode chunked raw responses correctly and preserve measured vs declared response length.
  • Add tests for cancellation latency, timeout enforcement, and bounded memory with large/chunked bodies.

P1: request feature parity

  • Support non-GET methods and request bodies (--data, --data-file).
  • Support Basic, Bearer/JWT, Digest, and NTLM authentication; reject only unsupported auth types.
  • Support URL-embedded Basic credentials.
  • Support SOCKS4/SOCKS5 and Tor proxies.
  • Support client certificates and keys.
  • Support --random-agent, --interface, and --ip.
  • Support --max-rate and --delay without blocking the runtime.
  • Match Python cookie/session behavior, not only a static Cookie header.
  • Enable and test gzip, deflate, Brotli, and zstd response decoding.
  • Define HTTPS behavior for malformed/raw paths; the current raw fallback is HTTP-only.
  • Apply retries consistently to raw-path requests.

P2: CLI and scheduling

  • Keep --threads as HTTP in-flight concurrency and document that runtime workers follow CPU count.
  • Benchmark whether an expert --native-workers override is useful before exposing it.
  • Add a bounded native batch/in-flight control only if incremental streaming still needs one.
  • Avoid scheduling every path in a chunk up front; maintain a bounded task set.
  • Emit results in completion order while preserving path/result association.

P3: packaging and platform coverage

  • Lower the abi3 baseline from Python 3.13 and validate native mode on every Python version supported by dirsearch, or document why the native minimum must differ.
  • Run real native integration tests on Windows and macOS in addition to Linux.
  • Add cargo clippy and dependency/security auditing to CI.
  • Exercise native wheels and standalone artifacts, not only source builds.
  • Resolve the missing v0.5.0 tag tracked by Release v0.5.0: tag is missing #1550.

Acceptance criteria

  • Native mode never silently ignores a CLI option.
  • Unsupported combinations fail during option parsing with an actionable message.
  • Python and native backends share local-server contract tests for request target, method/body, headers, proxy, redirects, auth, cookies, compression, retries, filters, cancellation, and time limits.
  • Large scans have bounded task count and bounded per-response memory.
  • Benchmarks report runtime workers and HTTP concurrency separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions