-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Seed the agent knowledge base #18253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
07f970c
315dc75
a2c2734
dced8a6
9d1ab63
0093f6d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
| @@ -0,0 +1,27 @@ | ||||
| # Knowledge backlog | ||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe .BACKLOG.md or __BACKLOG.md if we don't intend agents to read this, but may not matter. |
||||
|
|
||||
| Candidate agent-only entries, grouped by contributor-docs coverage. Each lands as its own reviewed PR. | ||||
|
|
||||
| ## Not in the docs | ||||
|
|
||||
| - **casing-and-pluralization** — how API names map to Terraform names; when `api_name` is required. | ||||
| - **output-only-test-assertions** — server-populated fields must have their values asserted in tests. | ||||
|
c2thorn marked this conversation as resolved.
Outdated
|
||||
| - **pr-mined-lessons** — recurring review catches and per-service quirks surfaced from merged PRs. | ||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
(up to you) IMO cut as we'd use historical analysis to determine new rules of this type to cover, rather than the outputs of historical analysis being a single rule. |
||||
|
|
||||
| ## Overlaps the docs (entry would add the judgment layer) | ||||
|
|
||||
| - **permadiff-decision-path** — choosing between `output`, `default_from_api`, diff suppression, or a real | ||||
| fix; mechanics are in `docs/content/develop/diffs.md`. | ||||
| - **data-source-idioms** — pitfalls beyond the procedure in `docs/content/develop/add-handwritten-datasource.md`. | ||||
| - **test-adequacy-traps** — cases `docs/content/test/test.md` doesn't cover (identical-config update steps, | ||||
| missing import-and-recheck). | ||||
|
|
||||
| ## Mostly covered by the docs (revisit only if agents misread them) | ||||
|
|
||||
| - **immutability-nuances** — `docs/content/reference/field.md`, `docs/content/best-practices/immutable-fields.md`. | ||||
| - **sensitive-and-write-only** — `docs/content/reference/field.md`. | ||||
|
|
||||
| ## Lives elsewhere | ||||
|
|
||||
| - **failure-troubleshooting-catalog** — `.agents/skills/operations/troubleshooting_reference.md`; migrates | ||||
| here if it outgrows the skill. | ||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Agent knowledge base | ||
|
|
||
| The **funnel to the knowledge an agent needs**: [`index.md`](index.md) routes to the right source for a | ||
| given decision. Agents read the index at decision points and open only what the task needs — never the | ||
| whole base (loading everything degrades reasoning and wastes context). | ||
|
|
||
| Most knowledge already lives in the **contributor documentation** (`docs/content/`), and that stays the | ||
| single source of truth for everything it covers — the index points there. **Agent-only entries** exist in | ||
| this directory only for knowledge that has no home in the docs: judgment rules that were never written | ||
| down, pitfall catalogs, and (later) lessons proposed by agents from completed tasks. If something is | ||
| covered by the contributor docs but covered badly, the fix is improving the docs — not writing a copy here. | ||
|
|
||
| Planned entries: [`BACKLOG.md`](BACKLOG.md). | ||
|
|
||
| ## Entry format | ||
|
|
||
| One Markdown file, one topic, ≤120 lines, YAML frontmatter: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is the frontmatter a well-known format or did you define it? Will agents do progressive disclosure with the format if it's custom?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Having YAML frontmatter is a well-known format. Gemini skills/rules use it (name+description), and so does the GCP Open Knowledge Format. The exact contents of the frontmatter were my invention, but can be fairly flexible if we find it's better to add or remove parts
Progressive disclosure doesn't depend on the frontmatter at all in these files. Right now frontmatter-driven loading only applies to skills + rules, and those are both location and frontmatter dependent. Right now the mechanism is driven by a harness-level rule to read the index. If Antigravity adopts an official shared knowledge system (not just local memories), we could switch to whatever that format is. They'll likely reuse name+description. The rest of the custom fields are mostly for record keeping.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. By well-known format I mean like OKF or skills i.e. https://agentskills.io/specification#frontmatter!
So we expect the agent to proactively read the index into context, and then choose to follow the index for those topics on-demand? |
||
|
|
||
| ```yaml | ||
| --- | ||
| name: enums-vs-strings # kebab-case, unique, matches filename | ||
| description: Model an API enum as Enum (strict) or String (forward-compatible). # the index line; <=140 chars | ||
| topics: [field] # index section(s) that list it | ||
| task_types: [field-add, new-resource] | ||
| source: docs/content/best-practices/validation.md # provenance: doc, PR, or "authored" | ||
| status: certified # certified (human-reviewed) | draft (proposed, unreviewed) | ||
| last_verified: 2026-07-09 | ||
| --- | ||
| ``` | ||
|
|
||
| Body rules: **rule + the why**, never bare imperatives; a real example per rule; a "do NOT use for" line | ||
| where misuse is likely; reference the contributor docs rather than duplicating them. | ||
|
|
||
| ## Curation rules | ||
|
|
||
| - **Nothing enters unreviewed.** Entries land via PR like code, **one entry at a time** — each one codifies | ||
| team judgment and deserves a real review conversation. Agent-proposed entries ship as `status: draft` in | ||
| the task PR that motivated them; a human review flips them to `certified`. | ||
| - **Never bulk-rewrite.** Edit entry by entry. Do not have a model re-summarize or reorganize the base | ||
| wholesale — that is the documented failure mode (drift and self-contradiction). | ||
| - **Contradictions are quarantined.** A proposed entry that contradicts an existing one is not merged; a | ||
| human adjudicates: update or retire the old entry, or discard the proposal. | ||
| - **Keep the index true.** Index lines come from entry frontmatter (name + description). Adding or changing | ||
| an entry updates the index in the same commit. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| --- | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Entries in A question that we'll need to ask ourselves: should this exist as an agent-only knowledge entry, or promoted to the contributor documentation? I would say in most new cases, it'll be more efficient to have new topics/decision points exist as new entries outside of the contributor documentation. These entries are intended to be optimized for LLM ingestion and also fed by future session learnings. Over time, I expect the % of documentation readers to lean towards mostly agents, whether initiated by a human author or not. |
||
| name: enums-vs-strings | ||
| description: Model an API enum as Enum (strict, plan-time) or String (forward-compatible) - the deliberate trade-off. | ||
| topics: [field] | ||
| task_types: [field-add, new-resource] | ||
| source: docs/content/best-practices/validation.md + the schema-review skill's original trade-off note | ||
| status: draft | ||
| last_verified: 2026-07-09 | ||
| --- | ||
|
|
||
| # Enum vs. String | ||
|
|
||
| **The trade-off (decide deliberately, per field):** | ||
| - **`Enum`** when the value set is exhaustive for a clearly-defined domain and new values are extremely | ||
| unlikely. Buys strict plan-time validation — users fail fast instead of mid-apply. | ||
| - **`String`** (with allowed values named in the description and a link to API docs) when the API is | ||
| likely to add values. **Why:** an Enum's value list lives in the provider; when the API adds a value, | ||
| every user needs a provider upgrade before they can use it — and imports of resources using the new | ||
| value break. A String stays forward-compatible. | ||
|
|
||
| ```yaml | ||
| - name: 'severity' | ||
| type: Enum | ||
| enum_values: | ||
| - 'LOW' | ||
| - 'MEDIUM' | ||
| - 'HIGH' | ||
| ``` | ||
|
|
||
| **Rules for Enum:** | ||
| - Do **not** include `UNSPECIFIED`-style values in `enum_values`. | ||
|
c2thorn marked this conversation as resolved.
Outdated
|
||
| - Enums validate against the list automatically; a custom `validation` **overrides** that default — if you | ||
| add one, it must re-verify the enum values itself. | ||
|
|
||
| **Rule for String-instead-of-enum:** list the allowed values in the field description with a link to the | ||
| API docs — otherwise users only discover an invalid value when the API rejects it mid-apply. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # Knowledge index | ||
|
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Lightweight index for agents to deliberately read so they are aware of all existing documentation available to them. The intention is that it performs better than the agent sporadically deciding to grep for existing documentation via specific terms or phrases. It could still decide to NOT consult any documentation (without more forceful guardrails), but this should cover awareness at the very least. |
||
|
|
||
| Read this index at decision points; open only the source the task needs. Format and curation rules: | ||
| [README.md](README.md). | ||
|
|
||
| ## Contributor documentation (`docs/content/`) | ||
|
|
||
| **Reference (what the schema properties mean):** | ||
|
|
||
| - Field properties (`output`, `immutable`, defaults, `sensitive`, validation, enums, arrays, `api_name`, conflicts) — `docs/content/reference/field.md` | ||
| - Resource-level properties (URLs, timeouts, async, custom code hooks) — `docs/content/reference/resource.md` | ||
| - Resource metadata — `docs/content/reference/metadata.md` | ||
| - Samples format — `docs/content/reference/sample.md` | ||
| - Update test expectations — `docs/content/reference/update-test-changes.md` | ||
| - Make commands — `docs/content/reference/make-commands.md` | ||
|
|
||
| **Procedures (how to do a task):** | ||
|
|
||
| - Add or change a field — `docs/content/develop/add-fields.md` | ||
| - Add a resource — `docs/content/develop/add-resource.md` | ||
| - Add a handwritten data source — `docs/content/develop/add-handwritten-datasource.md` | ||
| - Add IAM support — `docs/content/develop/add-iam-support.md` | ||
| - Custom code (expanders, flatteners, hooks) — `docs/content/develop/custom-code.md` | ||
| - Fix diffs and permadiffs — `docs/content/develop/diffs.md` | ||
| - Client-side fields — `docs/content/develop/client-side-fields.md` | ||
| - Promote beta → GA — `docs/content/develop/promote-to-ga.md` | ||
| - Generate the providers / set up the environment — `docs/content/develop/generate-providers.md`, `docs/content/develop/set-up-dev-environment.md` | ||
|
|
||
| **Testing:** | ||
|
|
||
| - Write acceptance tests (create, update, import) — `docs/content/test/test.md` | ||
| - Run tests — `docs/content/test/run-tests.md` | ||
|
|
||
| **Contributing (PRs, review, release notes):** | ||
|
|
||
| - Contribution process end to end — `docs/content/contribution-process.md` | ||
| - Create a PR — `docs/content/code-review/create-pr.md` | ||
| - Write release notes — `docs/content/code-review/release-notes.md` | ||
| - Review a PR — `docs/content/code-review/review-pr.md` | ||
|
|
||
| **Documentation:** | ||
|
|
||
| - Add resource documentation — `docs/content/document/add-documentation.md` | ||
| - Handwritten docs style guide — `docs/content/document/handwritten-docs-style-guide.md` | ||
|
|
||
| **Best practices (judgment calls with team positions):** | ||
|
|
||
| - Immutable fields / ForceNew — `docs/content/best-practices/immutable-fields.md` | ||
| - Deletion policy — `docs/content/best-practices/deletion-policy.md` | ||
| - Labels and annotations — `docs/content/best-practices/labels-and-annotations.md` | ||
| - Client-side validation — `docs/content/best-practices/validation.md` | ||
| - Common resource patterns (singletons) — `docs/content/best-practices/common-resource-patterns.md` | ||
|
|
||
| ## Agent-only entries | ||
|
|
||
| - **enums-vs-strings** — Model an API enum as Enum (strict, plan-time) or String (forward-compatible): the deliberate trade-off. — [field/enums-vs-strings.md](field/enums-vs-strings.md) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
An initial scan of potentially missing knowledge entries seeded from my own personal agentic sessions and some initial PR investigations.
Each entry would be codifying potentially ambiguous or stylistic decision-making, and should be reviewed/discussed individually. I believe it to be the most important tool our team has in curbing the last ~20% of scenarios the base models+harnesses cannot do alone.