From 87b3ce9293eb4e7e9309ece4f24690d0b5e09e3d Mon Sep 17 00:00:00 2001 From: Branimir Rakic Date: Wed, 8 Jul 2026 16:39:04 +0200 Subject: [PATCH] =?UTF-8?q?docs(use-dkg):=20large-content=20guidance=20?= =?UTF-8?q?=E2=80=94=20literal=20limit,=20chunk/externalize=20patterns,=20?= =?UTF-8?q?roadmap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Public-facing page for the 60,000-byte RDF literal limit (OVERSIZED_RDF_LITERAL): why it exists (cross-backend storability + replication health), the two supported patterns today (ordered chunking; externalize with URI + SHA-256 + size + excerpt), and the roadmap note for first-class external content storage. Producer rejection errors will cite this page. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/SUMMARY.md | 1 + docs/use-dkg/README.md | 1 + docs/use-dkg/large-content.md | 50 +++++++++++++++++++++++++++++++++++ 3 files changed, 52 insertions(+) create mode 100644 docs/use-dkg/large-content.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index ab60141dd8..4e1ede05af 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -37,6 +37,7 @@ * [Daemon Lifecycle](use-dkg/run-node.md) * [Publish & Query](use-dkg/publish-and-query.md) +* [Large Content](use-dkg/large-content.md) * [Knowledge Asset Lifecycle CLI](use-dkg/knowledge-asset-lifecycle.md) * [Async Publisher Wallets](use-dkg/async-publisher-wallets.md) * [OKF Import, Export, and Verify](use-dkg/okf.md) diff --git a/docs/use-dkg/README.md b/docs/use-dkg/README.md index 2a04cbd4ee..e948cd2b50 100644 --- a/docs/use-dkg/README.md +++ b/docs/use-dkg/README.md @@ -16,6 +16,7 @@ Use these routes when you want a node running, an agent connected, memory operat | Install DKG, connect an agent, or run a standalone node | [Quickstart](../getting-started/quickstart.md) | | Start, stop, and inspect the daemon | [Daemon Lifecycle](run-node.md) | | Write, publish, and query knowledge | [Publish and Query](publish-and-query.md) | +| Publish text larger than the RDF literal limit | [Large Content](large-content.md) | | Drive named Knowledge Asset lifecycle commands | [Knowledge Asset Lifecycle CLI](knowledge-asset-lifecycle.md) | | Configure async publisher wallets | [Async Publisher Wallets](async-publisher-wallets.md) | | Import, export, and verify OKF bundles | [OKF Import, Export, and Verify](okf.md) | diff --git a/docs/use-dkg/large-content.md b/docs/use-dkg/large-content.md new file mode 100644 index 0000000000..8e13b73e30 --- /dev/null +++ b/docs/use-dkg/large-content.md @@ -0,0 +1,50 @@ +# Large content on the DKG + +The DKG replicates RDF graphs across many independently operated nodes running +different triple-store backends. To keep every node able to store every graph, +**a single RDF literal (one text value in one triple) must stay at or below +60,000 bytes** (measured as Java modified-UTF-8, roughly: bytes of UTF-8 text). + +Why this limit exists: + +- One supported backend (Blazegraph) has a hard ~64 KB per-string storage + limit. A literal above it can never be stored by those nodes, so a graph + containing one would permanently diverge across the network. +- Very large literals are also a replication-health problem: every subscribed + node downloads and re-serves them on sync, so one oversized value multiplies + into network-wide bandwidth and storage cost. + +## What happens if you exceed it + +Writes containing an oversized literal are rejected with a structured +`OVERSIZED_RDF_LITERAL` error (HTTP 400 on the API routes). The error names the +subject, predicate, and byte size so you can locate the offending value. This +applies to publishes, updates, imports, shared-memory writes, and context-graph +registration metadata (names, descriptions). + +## How to publish large content today + +Two supported patterns: + +1. **Chunk it.** Split the text into ordered parts, each below the limit, as + separate triples/resources (e.g. `part 1..N` with an order index and a + total count). Automatic chunking for large public `schema.org/text` values + is in progress. +2. **Externalize it.** Store the body outside the graph (your own storage, an + artifact store, IPFS, …) and publish a compact pointer instead: the + content **URI**, its **SHA-256 hash**, **byte size**, media type, and a + short **summary or excerpt** for discoverability. The hash in the triple + keeps the external content verifiable — anyone can fetch the body and check + it against the published hash, and for Knowledge Assets the hash is covered + by the on-chain commitment. + +As a rule of thumb: triples carry *structured knowledge and pointers*; bulk +text and binary bodies belong outside the triple store, bound by hash. + +## Roadmap: first-class external content storage + +We plan to make pattern 2 a built-in feature: publish large content in one +call, with the node storing the body in a content-addressed blob store and +emitting the pointer triples (URI + hash + size + excerpt) for you — +SPARQL-queryable metadata, verifiable integrity, without hand-rolling the +externalization. Until then, the two patterns above are the supported paths.