NLPM audit findings: 23 NL schema bugs, 8 security findings (score 79/100)

[xiaolai]

Automated audit: This PR was generated by NLPM, a natural language programming linter, running via claude-code-action. Please evaluate the diff on its merits.

Summary

NLPM ran a progressive audit of this repo on 2026-04-06 and found 23 NL schema bugs and 8 security findings across 103 NL artifacts. The overall NL quality score is 79/100 — solid foundations, especially the github-dev agents which score 88-95.

This issue tracks the bugs and the PRs opened to fix them. All fixes are minimal and match the existing repo style.

NLPM Methodology

NLPM scores NL artifacts (commands, agents, skills) on a 100-point scale using deterministic penalties:

• Missing name frontmatter (required for command registration): -25 pts
• Missing allowed-tools frontmatter (required for tool permissions): -5 pts
• Missing frontmatter block entirely: -50 pts
• Broken references: -5 pts

The 50 Rules of Natural Language Programming define the schema requirements. All findings below are rule violations with no design judgment involved.

Bugs Found (23 total, PR-worthy)

Priority	File	Bug
🔴 Critical	plugins/claude-tools/commands/sync-allowlist.md	Step 1 references fcakyon/claude-settings (wrong repo) — silently fetches from wrong/nonexistent repo
🟠 High	plugins/claude-tools/commands/sync-allowlist.md	Missing name frontmatter
🟠 High	plugins/claude-tools/commands/update-readme.md	Missing name frontmatter
🟠 High	plugins/claude-tools/commands/sync-claude-md.md	Missing name frontmatter
🟠 High	plugins/claude-tools/commands/load-claude-md.md	Missing name frontmatter
🟠 High	plugins/tavily-tools/commands/setup.md	Missing name + allowed-tools frontmatter
🟠 High	plugins/azure-tools/commands/setup.md	Missing name + allowed-tools frontmatter
🟠 High	plugins/gcloud-tools/commands/setup.md	Missing name + allowed-tools frontmatter
🟠 High	plugins/paper-search-tools/commands/setup.md	Missing name + allowed-tools frontmatter
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/functions.md	No frontmatter block (scores 50/100)
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/gen.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/init.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/link.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/login.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/projects.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/secrets.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/start.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/status.md	No frontmatter block
🟡 Medium	plugins/supabase-skills/skills/supabase-cli/references/commands/stop.md	No frontmatter block

Security Findings (8 total)

Severity	File	Pattern
HIGH	plugins/ultralytics-dev/hooks/hooks.json:10	Inline bash reads file_path from AI tool input and uses it in sed -i without sanitization — recommend extracting to a Python script with pathlib.Path.resolve() validation (not PR'd; requires architectural change)
MEDIUM	plugins/claude-tools/hooks/scripts/sync_marketplace_to_plugins.py	Path traversal risk: source field from marketplace.json used to construct write path with no traversal guard
MEDIUM	plugins/github-dev/hooks/scripts/git_commit_confirm.py	subprocess.run with git diff — safe list form, low risk
MEDIUM	plugins/github-dev/hooks/scripts/gh_pr_create_confirm.py	subprocess.run with gh API calls — safe list form, low risk
MEDIUM	plugins/ultralytics-dev/hooks/scripts/python_code_quality.py	File path from JSON input in subprocess.run — validated via pathlib before use, safe in practice
MEDIUM	.github/scripts/_helpers.sh	python3 -c "..." with shell variable interpolation — developer-only script, limited exposure
LOW	plugins/*/.mcp.json (5 files)	Broad MCP server permissions with no explicit scope restrictions
LOW	.github/scripts/release.sh	Glob expansion of developer-controlled scripts — developer-only context

The HIGH finding in ultralytics-dev/hooks/hooks.json requires extracting the inline bash to a Python script and adding pathlib.Path.resolve() validation before any shell operation. This is an architectural change not suitable for an automated PR.

PRs Opened

All fixes are minimal, match existing repo style, and touch only the broken fields:

PR	Fix
#166	Fix broken repo URL (fcakyon/claude-settings → fcakyon/claude-codex-settings) + add name to sync-allowlist.md
#167	Add missing name frontmatter to update-readme.md, sync-claude-md.md, load-claude-md.md
#168	Add missing name + allowed-tools to 4 setup commands (tavily, azure, gcloud, paper-search)
#169	Add missing frontmatter to 10 supabase-cli reference skill docs
#170	Add path traversal guard in sync_marketplace_to_plugins.py (Medium security fix)

What Was Not PR'd

• Quality/convention issues (description format, unused tools, non-standard fields) — no schema violation, design judgment required
• HIGH security finding in ultralytics-dev/hooks/hooks.json — architectural change required, recommend private discussion
• The _helpers.sh variable interpolation — developer-only script, risk is minimal

This is a high-quality repo overall. The github-dev agents are particularly well-structured. The main drag on the score is the supabase reference docs (10 files × -50 pts each). Fixing those alone would push the score above 90.

Thank you for maintaining this project!