Infinite recursion on modern web apps returning same-size fallback pages

[pentest0099]

Description

When scanning modern web apps (e.g. React/Next.js/Angular) in recursive mode, dirsearch often loops infinitely because the target returns identical-sized responses (e.g. fallback index.html) for non-existent paths.
This causes repeated recursion with no new results.

Steps to Reproduce
dirsearch -u https://target.site/ -r --deep-recursive -i 200 --recursion-status 200,403
All subpaths return the same response size and dirsearch keeps scanning endlessly.

Expected Behavior

If the same (status_code, response_size) repeats several times for a base path, dirsearch should:

Stop recursion on that path.

Log a message indicating repeated identical responses

Proposed Solution

Add a limit and skip behavior:

CLI option: --max-same-size (default 5)

Internal logic example:
seen_sizes[base][(status, size)] += 1
if seen_sizes[base][(status, size)] > max_same_size:
skip_recursion(base)

Why

Modern apps return same-size fallback pages for all routes — current recursion logic misinterprets them as valid directories

[shelld3v]

--duplicate-threshold is the same as your --max-same-size. However, it will still perform recursion on paths that are found before the limitation is reached. I wonder should we stop recursion once the threshold is reached

[mojtaba13133]

Problem Analysis

Thanks @shelld3v for the response! I understand --duplicate-threshold exists, but it doesn't fully solve the SPA recursion problem because it still recurses into paths found before the threshold is reached.

Real-World Impact

When scanning modern SPAs (React/Next.js), even the first few "duplicates" are fake fallback responses. Here's what happens:

Current behavior:

/api/users     → 200 781B  ✓ recurses
/api/posts     → 200 781B  ✓ recurses  
/api/comments  → 200 781B  ✓ recurses
/api/profiles  → 200 781B  ✓ recurses
/api/settings  → 200 781B  ✓ recurses (threshold=5 reached)
/api/notifications → 200 781B  ✗ skipped

Result: 5 useless recursive scans still happen → wastes time and generates noise

Desired behavior:

/api/users     → 200 781B  (counted)
/api/posts     → 200 781B  (counted)
/api/comments  → 200 781B  (threshold reached)
[!] Stopping recursion on /api/: 781B appears 3+ times
→ No recursion happens

---

Proposed Solutions

I've identified 5 strategies to handle this, ranging from quick fixes using existing flags to new features. Here's a breakdown:

---

Strategy 1: Enhanced Duplicate Detection ⭐ (Quick Win)

What: Improve existing --duplicate-threshold behavior

Proposed new flag:

--stop-on-duplicate

Behavior:

• When used with --duplicate-threshold N:
  • Once N identical (status, size) pairs are found on a base path
  • Immediately stop ALL recursion on that path
  • Don't recurse into any of the duplicates (including previously found ones)

Example:

dirsearch -u https://target.com -r \
  --duplicate-threshold 3 \
  --stop-on-duplicate

Implementation complexity: Low (modify existing duplicate detection logic)

---

Strategy 2: Pre-Scan Fingerprinting ⭐⭐ (High Impact)

What: Auto-detect fallback patterns before the main scan

Proposed new flag:

--auto-fingerprint

Behavior:

1. Before main scan, test 3-5 random non-existent paths
2. If they all return same (status, size) → fallback detected
3. Automatically add to --exclude-sizes and log detection
4. Proceed with main scan

Example:

dirsearch -u https://target.com -r \
  --auto-fingerprint \
  --max-recursion-depth 3

Output:

[*] Fingerprinting target...
[*] Testing: /random_xyz123, /test_invalid999, /notexist_abc
[!] Fallback detected: 200 781B (text/html)
[*] Auto-excluding: 781B responses
[*] Starting main scan...

Implementation complexity: Medium (add pre-scan phase)

---

Strategy 3: Recursion Strategy Presets ⭐⭐⭐ (Best UX)

What: Predefined flag combinations for common scenarios

Proposed new flag:

--recursion-strategy [mode]

Modes:

conservative (safest, recommended for SPAs)

# Equivalent to:
--duplicate-threshold 3
--stop-on-duplicate
--exclude-content-type text/html
--max-recursion-depth 2

balanced (default)

# Equivalent to:
--duplicate-threshold 5
--exclude-content-type text/html
--max-recursion-depth 3

aggressive (thorough scan)

# Equivalent to:
--duplicate-threshold 10
--max-recursion-depth 5
--deep-recursive

api (optimized for REST APIs)

# Equivalent to:
--exclude-content-type text/html,text/plain
--recursion-status 200,201,401,403
--duplicate-threshold 5

Example usage:

dirsearch -u https://react-app.com -r --recursion-strategy conservative

Implementation complexity: Low (wrapper around existing flags)

---

Strategy 4: Size Variance Analysis ⭐⭐ (Advanced)

What: Detect fallback patterns by analyzing response size distribution

Proposed new flags:

--variance-detection
--variance-threshold [0.0-1.0]  # default: 0.8

Behavior:

• For each base path, track response size distribution
• If >80% of responses have identical size → likely fallback, stop recursion
• Otherwise → size variance indicates real endpoints, continue

Example:

dirsearch -u https://target.com -r \
  --variance-detection \
  --variance-threshold 0.75

Output:

[*] Scanning /api/...
[!] Variance check: /api/ → 15/17 responses are 781B (88%)
[!] Stopping recursion on /api/ (fallback pattern detected)

Implementation complexity: Medium (add statistical analysis)

---

Strategy 5: Batch Mode with Auto-Fingerprinting ⭐⭐⭐ (Enterprise)

What: Optimized workflow for scanning many targets

Proposed new flag:

--batch-mode

Behavior:

• When used with -l urls.txt:
  1. For each URL, run fingerprinting automatically
  2. Apply per-target exclusions
  3. Scan with smart recursion
  4. Save individual results

Example:

dirsearch -l 1000_targets.txt \
  --batch-mode \
  --auto-fingerprint \
  --recursion-strategy conservative \
  -o results/ --format json

Output:

[*] Batch mode: processing 1000 URLs
[*] [1/1000] https://site1.com → fingerprint: 781B
[*] [2/1000] https://site2.com → fingerprint: 1024B
...
[*] Results saved to: results/site1_com.json, results/site2_com.json, ...

Implementation complexity: High (workflow orchestration)

---

Strategy Comparison

Strategy	Impact	Complexity	Uses Existing Flags	Solves SPA Loop

1. Stop on Duplicate | 🟢 High | 🟢 Low | ✅ Yes | ✅ Yes
2. Auto-Fingerprint | 🟢 High | 🟡 Medium | ⚠️ Partial | ✅ Yes
3. Strategy Presets | 🟢 High | 🟢 Low | ✅ Yes | ✅ Yes
4. Variance Analysis | 🟡 Medium | 🟡 Medium | ❌ No | ✅ Yes
5. Batch Mode | 🟢 High | 🔴 High | ⚠️ Partial | ✅ Yes

---

My Recommendations

Phase 1: Quick Wins (Immediate)

Implement these using mostly existing code:

1. Add --stop-on-duplicate flag

   • Modifies --duplicate-threshold behavior
   • Stops all recursion when threshold reached
   • Effort: ~50 lines of code

2. Add --recursion-strategy presets

   • Maps to existing flag combinations
   • Improves UX significantly
   • Effort: ~100 lines of code

Phase 2: High-Impact Features (Short-term)

New functionality that solves the core problem:

3. Add --auto-fingerprint flag
   • Pre-scan detection of fallback patterns
   • Auto-excludes detected sizes
   • Effort: ~200 lines of code

Phase 3: Advanced Features (Long-term)

For power users and enterprise scenarios:

4. Add --variance-detection flag
5. Add --batch-mode flag

---

Code Examples

Using Existing Flags (Current Workaround)

# Manual fingerprinting + exclusion
curl -s -o /dev/null -w "%{size_download}\n" https://target.com/random123
# Output: 781
dirsearch -u https://target.com -r 

--duplicate-threshold 5 

--exclude-sizes 781B 

--exclude-content-type text/html 

--max-recursion-depth 3

With Proposed Flags (Future)

# Strategy 1: Enhanced duplicate detection
dirsearch -u https://target.com -r \
  --duplicate-threshold 3 \
  --stop-on-duplicate
Strategy 2: Auto-fingerprinting
dirsearch -u https://target.com -r 

--auto-fingerprint 

--max-recursion-depth 3
Strategy 3: Preset modes
dirsearch -u https://target.com -r 

--recursion-strategy conservative
Strategy 4: Variance analysis
dirsearch -u https://target.com -r 

--variance-detection 

--variance-threshold 0.8
Strategy 5: Batch processing
dirsearch -l targets.txt 

--batch-mode 

--auto-fingerprint 

--recursion-strategy conservative

---

Question for Maintainers

Should we stop recursion once the threshold is reached?

My answer: Yes, absolutely.

In SPAs, all duplicate responses are fake fallbacks. Recursing into any of them wastes resources. The --stop-on-duplicate flag would let users choose this behavior.

Would you accept PRs for:

• --stop-on-duplicate flag (Phase 1)
• --recursion-strategy presets (Phase 1)
• --auto-fingerprint flag (Phase 2)

---

Environment:

• dirsearch version: latest
• Target type: Modern SPAs (React/Next.js/Vue)
• Use case: Penetration testing with 1000+ targets