CrawlFuse monitor

Start from the template; set your URLs in crawlfuse.json before running it.

A zero-dependency Node 22 monitor for public webpages. It is designed for a scheduled GitHub Actions job, a release workflow, or a local cron task.

Use it directly from Mandrilsquad1441/crawlfuse-monitor or download the v1.0.3 ZIP.

The monitor follows at most five redirects, validates DNS before every hop, pins the connection to a validated address, bounds response size and time, and rejects private, loopback, link-local, reserved, documentation, and mixed public/private DNS answers.

Each URL produces machine-readable checks for:

• fetch success
• final HTTP status
• canonical link presence and validity
• noindex in meta robots directives
• noindex in the X-Robots-Tag response header

Run

Node 22 or later is required. There is no install step.

node src/cli.mjs --config config.example.json

The complete JSON report is written to stdout. To persist it for the next run:

node src/cli.mjs \
  --config config.example.json \
  --output current-report.json

Compare the next scan with that report:

node src/cli.mjs \
  --config config.example.json \
  --baseline current-report.json \
  --output next-report.json

The config may instead include "baseline": "./current-report.json"; that path is resolved relative to the config file. A command-line baseline takes priority.

Exit behavior

• 0: no check changed from pass to fail
• 1: one or more checks changed from pass to fail
• 2: configuration or operational error; the report has completed: false

An initial failure without a baseline is recorded in JSON but exits 0. Recoveries and changed observations are also reported without failing the process. A configuration or operational error is emitted as JSON, written to --output when possible, and exits 2 so a broken monitor cannot look green.

Configuration

urls accepts one to ten absolute public HTTP or HTTPS URLs. Entries may be strings or objects with a display name.

{
  "urls": [
    "https://example.com/",
    {
      "name": "Pricing",
      "url": "https://example.com/pricing"
    }
  ],
  "request": {
    "timeoutMs": 6000,
    "maxRedirects": 4,
    "maxBodyBytes": 524288
  }
}

Hard safety ceilings are 10 seconds per request, 5 redirects, and 1 MiB per final response. Only default HTTP and HTTPS ports are allowed.

GitHub Actions

See the public template repository and its verified first-run workflow for a complete copyable installation.

Create crawlfuse.json in your repository, then add this workflow:

name: CrawlFuse

on:
  workflow_dispatch:
  schedule:
    - cron: "17 6 * * *"

permissions:
  contents: read

jobs:
  seo-release-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Check SEO release regressions
        uses: Mandrilsquad1441/crawlfuse-monitor@v1
        with:
          config: crawlfuse.json
          output: crawlfuse-report.json
      - name: Preserve the report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: crawlfuse-report
          path: crawlfuse-report.json
          if-no-files-found: error

Copy the workflow exactly as shown for the first run. Download and review the generated report, commit an accepted copy as crawlfuse-baseline.json, then add this input below config:

          baseline: crawlfuse-baseline.json

Update that file only after intentional changes are reviewed. After trying the workflow, share whether the first report was produced in agency validation issue #1.

The major v1 tag receives compatible v1 updates. Pin a full release tag or commit SHA if your supply-chain policy requires an immutable reference.

Vendored action

The repository includes a composite action, so a project that keeps this directory can run it with uses: ./monitor.

Commit a prior successful report as monitor/baseline.json, then add a workflow step:

- name: Check SEO release regressions
  uses: ./monitor
  with:
    config: monitor/config.example.json
    baseline: monitor/baseline.json
    output: monitor/latest-report.json

Upload monitor/latest-report.json as an artifact if you want to inspect changes after each run. Update the baseline only after reviewing and accepting intentional changes.

Tests

cd monitor
npm test

Tests use Node's built-in test runner and do not require network access.

Security

The monitor rejects local, private, reserved, documentation, and mixed public/private DNS answers. It validates and pins an address before every redirect request, caps redirect traversal, blocks HTTPS downgrades, and bounds response size and time.

Report vulnerabilities privately through this repository's Security tab. See SECURITY.md; do not open a public issue for a suspected vulnerability.

The monitor sends no telemetry or reports to a CrawlFuse service. See PRIVACY.md for the local and GitHub Actions data boundary.

License

MIT. See LICENSE.