Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@

* [Daemon Lifecycle](use-dkg/run-node.md)
* [Publish & Query](use-dkg/publish-and-query.md)
* [Large Content](use-dkg/large-content.md)
* [Knowledge Asset Lifecycle CLI](use-dkg/knowledge-asset-lifecycle.md)
* [Async Publisher Wallets](use-dkg/async-publisher-wallets.md)
* [OKF Import, Export, and Verify](use-dkg/okf.md)
Expand Down
1 change: 1 addition & 0 deletions docs/use-dkg/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Use these routes when you want a node running, an agent connected, memory operat
| Install DKG, connect an agent, or run a standalone node | [Quickstart](../getting-started/quickstart.md) |
| Start, stop, and inspect the daemon | [Daemon Lifecycle](run-node.md) |
| Write, publish, and query knowledge | [Publish and Query](publish-and-query.md) |
| Publish text larger than the RDF literal limit | [Large Content](large-content.md) |
| Drive named Knowledge Asset lifecycle commands | [Knowledge Asset Lifecycle CLI](knowledge-asset-lifecycle.md) |
| Configure async publisher wallets | [Async Publisher Wallets](async-publisher-wallets.md) |
| Import, export, and verify OKF bundles | [OKF Import, Export, and Verify](okf.md) |
Expand Down
50 changes: 50 additions & 0 deletions docs/use-dkg/large-content.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Large content on the DKG

The DKG replicates RDF graphs across many independently operated nodes running
different triple-store backends. To keep every node able to store every graph,
**a single RDF literal (one text value in one triple) must stay at or below
60,000 bytes** (measured as Java modified-UTF-8, roughly: bytes of UTF-8 text).

Why this limit exists:

- One supported backend (Blazegraph) has a hard ~64 KB per-string storage
limit. A literal above it can never be stored by those nodes, so a graph
containing one would permanently diverge across the network.
- Very large literals are also a replication-health problem: every subscribed
node downloads and re-serves them on sync, so one oversized value multiplies
into network-wide bandwidth and storage cost.

## What happens if you exceed it

Writes containing an oversized literal are rejected with a structured
`OVERSIZED_RDF_LITERAL` error (HTTP 400 on the API routes). The error names the
subject, predicate, and byte size so you can locate the offending value. This
applies to publishes, updates, imports, shared-memory writes, and context-graph

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Context-graph metadata is documented as guarded, but the create path can bypass the 60 KB check

What's wrong
The new page turns the oversized-literal behavior into a public contract for context-graph names/descriptions, but the context-graph create flow does not use the 60,000-byte guard before inserting those metadata triples. That means callers can see different behavior from what the docs promise, and on mixed backends the metadata may still hit backend-specific failure behavior instead of a consistent API rejection.

Example
POST /api/context-graph/create with a 60,001-byte description follows the context-graph create path. On the embedded Oxigraph store, there is no 60,000-byte validation in that path, so it can be accepted instead of returning the documented structured OVERSIZED_RDF_LITERAL HTTP 400.

Suggested direction
Align the docs and implementation: either remove context-graph registration metadata from this guarantee, or validate generated name/description quads before insertion and map OVERSIZED_RDF_LITERAL to the same structured 400 response.

For Agents
Either narrow the docs to the write paths that actually call assertQuadLiteralsMutf8Safe, or add the same literal-size validation to context-graph create/register metadata before store.insert. Preserve the documented HTTP 400 body shape and prove it with a route-level context-graph create test using an oversized description/name.

registration metadata (names, descriptions).

## How to publish large content today

Two supported patterns:

1. **Chunk it.** Split the text into ordered parts, each below the limit, as
separate triples/resources (e.g. `part 1..N` with an order index and a
total count). Automatic chunking for large public `schema.org/text` values
is in progress.
2. **Externalize it.** Store the body outside the graph (your own storage, an
artifact store, IPFS, …) and publish a compact pointer instead: the
content **URI**, its **SHA-256 hash**, **byte size**, media type, and a
short **summary or excerpt** for discoverability. The hash in the triple
keeps the external content verifiable — anyone can fetch the body and check
it against the published hash, and for Knowledge Assets the hash is covered
by the on-chain commitment.

As a rule of thumb: triples carry *structured knowledge and pointers*; bulk
text and binary bodies belong outside the triple store, bound by hash.

## Roadmap: first-class external content storage

We plan to make pattern 2 a built-in feature: publish large content in one
call, with the node storing the body in a content-addressed blob store and
emitting the pointer triples (URI + hash + size + excerpt) for you —
SPARQL-queryable metadata, verifiable integrity, without hand-rolling the
externalization. Until then, the two patterns above are the supported paths.
Loading