-
Notifications
You must be signed in to change notification settings - Fork 10
docs(use-dkg): large-content guidance — RDF literal limit, chunk/externalize, roadmap #1527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
branarakic
wants to merge
1
commit into
main
Choose a base branch
from
docs/large-content-guidance
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # Large content on the DKG | ||
|
|
||
| The DKG replicates RDF graphs across many independently operated nodes running | ||
| different triple-store backends. To keep every node able to store every graph, | ||
| **a single RDF literal (one text value in one triple) must stay at or below | ||
| 60,000 bytes** (measured as Java modified-UTF-8, roughly: bytes of UTF-8 text). | ||
|
|
||
| Why this limit exists: | ||
|
|
||
| - One supported backend (Blazegraph) has a hard ~64 KB per-string storage | ||
| limit. A literal above it can never be stored by those nodes, so a graph | ||
| containing one would permanently diverge across the network. | ||
| - Very large literals are also a replication-health problem: every subscribed | ||
| node downloads and re-serves them on sync, so one oversized value multiplies | ||
| into network-wide bandwidth and storage cost. | ||
|
|
||
| ## What happens if you exceed it | ||
|
|
||
| Writes containing an oversized literal are rejected with a structured | ||
| `OVERSIZED_RDF_LITERAL` error (HTTP 400 on the API routes). The error names the | ||
| subject, predicate, and byte size so you can locate the offending value. This | ||
| applies to publishes, updates, imports, shared-memory writes, and context-graph | ||
| registration metadata (names, descriptions). | ||
|
|
||
| ## How to publish large content today | ||
|
|
||
| Two supported patterns: | ||
|
|
||
| 1. **Chunk it.** Split the text into ordered parts, each below the limit, as | ||
| separate triples/resources (e.g. `part 1..N` with an order index and a | ||
| total count). Automatic chunking for large public `schema.org/text` values | ||
| is in progress. | ||
| 2. **Externalize it.** Store the body outside the graph (your own storage, an | ||
| artifact store, IPFS, …) and publish a compact pointer instead: the | ||
| content **URI**, its **SHA-256 hash**, **byte size**, media type, and a | ||
| short **summary or excerpt** for discoverability. The hash in the triple | ||
| keeps the external content verifiable — anyone can fetch the body and check | ||
| it against the published hash, and for Knowledge Assets the hash is covered | ||
| by the on-chain commitment. | ||
|
|
||
| As a rule of thumb: triples carry *structured knowledge and pointers*; bulk | ||
| text and binary bodies belong outside the triple store, bound by hash. | ||
|
|
||
| ## Roadmap: first-class external content storage | ||
|
|
||
| We plan to make pattern 2 a built-in feature: publish large content in one | ||
| call, with the node storing the body in a content-addressed blob store and | ||
| emitting the pointer triples (URI + hash + size + excerpt) for you — | ||
| SPARQL-queryable metadata, verifiable integrity, without hand-rolling the | ||
| externalization. Until then, the two patterns above are the supported paths. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Issue: Context-graph metadata is documented as guarded, but the create path can bypass the 60 KB check
What's wrong
The new page turns the oversized-literal behavior into a public contract for context-graph names/descriptions, but the context-graph create flow does not use the 60,000-byte guard before inserting those metadata triples. That means callers can see different behavior from what the docs promise, and on mixed backends the metadata may still hit backend-specific failure behavior instead of a consistent API rejection.
Example
POST /api/context-graph/create with a 60,001-byte description follows the context-graph create path. On the embedded Oxigraph store, there is no 60,000-byte validation in that path, so it can be accepted instead of returning the documented structured OVERSIZED_RDF_LITERAL HTTP 400.
Suggested direction
Align the docs and implementation: either remove context-graph registration metadata from this guarantee, or validate generated name/description quads before insertion and map OVERSIZED_RDF_LITERAL to the same structured 400 response.
For Agents
Either narrow the docs to the write paths that actually call assertQuadLiteralsMutf8Safe, or add the same literal-size validation to context-graph create/register metadata before store.insert. Preserve the documented HTTP 400 body shape and prove it with a route-level context-graph create test using an oversized description/name.