|
| 1 | +# vgi-etf-spdr — agent notes |
| 2 | + |
| 3 | +A VGI (DuckDB) worker exposing SPDR / State Street (SSGA) US ETF data as two base **tables** — |
| 4 | +`products` (the catalog) and `holdings` (hive-partitioned) — plus one table **function**, |
| 5 | +`nav_history` (and the listed `holdings_scan` backing the holdings table). TypeScript, runs on |
| 6 | +Bun, built on `@query-farm/vgi` (the TS SDK). Keyless — no secret type, no auth. Modeled on the |
| 7 | +sibling `vgi-etf-ishares` worker; the KEY DIFFERENCE is that SSGA publishes **current holdings only**, |
| 8 | +so `holdings` has NO time travel. |
| 9 | + |
| 10 | +## Base tables (`products`, `holdings`) — two layers: registry vs listing |
| 11 | + |
| 12 | +Tables are wired via `SchemaDescriptor.tables` (`makeCatalog`'s `tables: [...]`); each |
| 13 | +`TableDescriptor` has `function: <scan>` + `arguments: new Arguments([], new Map())` and carries |
| 14 | +its docs on `tags`/`comment`/`columnComments`. Two INDEPENDENT layers matter: |
| 15 | +- **FunctionRegistry** (`registry.register(scan)`) — the *dispatch* layer. Required for a table to |
| 16 | + be scannable. |
| 17 | +- **catalog `schemas[].functions`** — the *listing* layer. Controls what shows as a callable `X()` |
| 18 | + function AND is where the extension discovers a scan's capabilities (e.g. `filter_pushdown`). |
| 19 | + |
| 20 | +`products`: backing `productsScan` is **registered but NOT listed** → exposed only as the table. |
| 21 | +`holdings`: backing `holdingsScan` MUST be **listed** (`functions: [...functions, holdingsScan]`) |
| 22 | +— an unlisted backing scan gets no `pushdown_filters` (the extension can't see its |
| 23 | +`filter_pushdown` capability), so the `fund_ticker` partition filter never reaches it. Hence a |
| 24 | +visible `holdings_scan()` is unavoidable; VGI311 is waived in `vgi-lint.toml`. |
| 25 | + |
| 26 | +## `holdings` — hive-partitioned by `fund_ticker`, CURRENT holdings only (no time travel) |
| 27 | + |
| 28 | +Query `FROM spdr.main.holdings WHERE fund_ticker = 'SPY'` (fund selector); an **unfiltered scan |
| 29 | +streams every fund** (one partition per fund). Mechanics: |
| 30 | +- **Hive partitioning + streaming queue.** `holdingsScan` is a `partitionKind: |
| 31 | + "SINGLE_VALUE_PARTITIONS"` generator — `fund_ticker` is the partition key (annotated |
| 32 | + `vgi.partition_column` in `holdingsSchema`). `onInit` reads the pushed `fund_ticker` filter (or, |
| 33 | + absent one, the whole catalog), and `queuePush`es one `{ticker}` item per fund onto a |
| 34 | + `BoundStorage` queue keyed by the execution id. `process()` pops one fund per tick, fetches its |
| 35 | + holdings `.xlsx`, and `out.emit`s a single partition batch tagged with `vgi_partition_values` |
| 36 | + (min==max==ticker). `maxWorkers` workers drain the same queue → work-stealing fan-out. `LIMIT` |
| 37 | + short-circuits the stream. |
| 38 | +- **No time travel.** SSGA has only one current holdings file per fund. There is deliberately NO |
| 39 | + `supportsTimeTravel` and NO as-of argument; `process()` never reads `p.atValue`. `as_of_date` is |
| 40 | + a real output column populated from the spreadsheet's own "As of" header row. |
| 41 | +- **404-tolerant.** A fund with no holdings file throws in `getBytes`; `process()` catches and |
| 42 | + skips to the next fund so an all-funds scan never fails on one missing file. |
| 43 | +- **`filterPushdown: true`** + LISTED → the extension pushes the `fund_ticker` filter into the scan. |
| 44 | +- **`fund_ticker` is a SEPARATE column from `ticker`** — `ticker` is the CONSTITUENT's own ticker |
| 45 | + (present for equity funds, null for bond/loan funds); `fund_ticker` is the fund's ticker, |
| 46 | + constant per fund. The scan tags every row with the requested fund ticker, upper-cased. |
| 47 | +- Constraints: `products` advisory PK `[isin]`, `holdings` `notNull [fund_ticker]`. No cross-table |
| 48 | + FK (identifier columns recur with different meanings). VGI311/807/809 waived with reasons. |
| 49 | + |
| 50 | +## Holdings/NAV are `.xlsx` — the one extra dependency |
| 51 | + |
| 52 | +SSGA's catalog is JSON, but holdings and NAV history are published **only as `.xlsx`** (no CSV/JSON |
| 53 | +— the `.csv` path 404s). So the worker adds [SheetJS `xlsx`](https://www.npmjs.com/package/xlsx) |
| 54 | +(`0.18.5`), a PURE parser (no network) — allowed in the driver layer. `readXlsxMatrix(bytes)` (the |
| 55 | +ONLY XLSX touch) decodes a byte buffer into a row matrix via `XLSX.utils.sheet_to_json(ws, |
| 56 | +{header:1})`; every other parser takes a plain matrix so the unit tests drive them without any |
| 57 | +xlsx (a couple round-trip a SheetJS-built buffer through `readXlsxMatrix`). |
| 58 | + |
| 59 | +**Column layout VARIES by fund type**, so the spreadsheet parsers are **header-driven**: find the |
| 60 | +header row, map each column by its (lowercased) label, then read data rows until the first blank |
| 61 | +row (the disclaimer footer follows). Observed layouts: |
| 62 | +- equity (SPY): Name, Ticker, Identifier, SEDOL, Weight, Sector, Shares Held, Local Currency |
| 63 | +- treasury (BIL): Name, Identifier, SEDOL, Weight, Coupon, Par Value, Market Value, Local Currency, Maturity |
| 64 | +- loan (SRLN): Name, Identifier, FIGI, Weight, Coupon, Maturity, Par Value, Market Value |
| 65 | +The union is emitted as one wide schema; unfilled columns are null per fund type. `parseHoldings` |
| 66 | +sorts by `weight_percent` DESC (NULLS last) so `... LIMIT n` returns the top holdings. |
| 67 | + |
| 68 | +## Architecture (keep this separation) |
| 69 | + |
| 70 | +- **`src/spdr.ts` — the pure driver.** URL builders + JSON/spreadsheet parsers, plus thin |
| 71 | + `fetch*` orchestrators and `resolveTicker` that take injected `get(url)` (JSON) and/or |
| 72 | + `getBytes(url)` (bytes). NO network, NO SDK import (it MAY import `xlsx`, a pure parser). This is |
| 73 | + what the unit tests exercise. All parsing is defensive: a missing field/column/row degrades to |
| 74 | + `[]`/`null`, never a throw. `resolveTicker` returns `string | null` (null = not found) rather |
| 75 | + than throwing, so this module needs no SDK import; `functions.ts` turns null into a typed |
| 76 | + `ArgumentValidationError`. |
| 77 | +- **`src/client.ts` — the only network module.** `makeSpdrClient()` returns `{ get, getBytes }`. |
| 78 | + `get` fetches JSON (and memoizes the fund-finder for 24 h); `getBytes` fetches an `.xlsx` as a |
| 79 | + `Uint8Array` (never cached). Its one job beyond `fetch` is setting the browser-like User-Agent |
| 80 | + SSGA requires (the default fetch UA gets an interstitial HTML page). No dedicated unit test for |
| 81 | + the byte path beyond a shape check; exercised live by the HTTP-transport + haybarn tests. |
| 82 | +- **`src/schema.ts` — typed Arrow schemas + batch builders.** Real typed columns |
| 83 | + (`Utf8`/`Float64`/`DateDay`), not JSON. Every calendar date is a real Arrow **DATE** (`DateDay` |
| 84 | + → DuckDB `DATE`, no timezone; a DATE cell is a JS `Date` at UTC midnight via `dateOrNull`). |
| 85 | + NOTE: dates are DATE, not TIMESTAMP (casting a UTC-midnight TIMESTAMPTZ `::DATE` shifts the day |
| 86 | + in non-UTC sessions). Percent columns carry a `_percent` suffix and hold **percent points** |
| 87 | + (SSGA's raw values: `weight_percent` 7.38 = 7.38%, `expense_ratio_percent` 0.09 = 0.09%). |
| 88 | +- **`src/functions.ts`** — three `defineTableFunction`s: `makeProductsScan` (unlisted products |
| 89 | + backing scan), `makeHoldingsScan` (`holdings_scan`, LISTED, filterPushdown, SINGLE_VALUE |
| 90 | + partitions, queue/BoundStorage streaming), and `makeNavHistoryFunction`. Each `make*` takes the |
| 91 | + whole `SpdrClient` (`{get, getBytes}`) for uniformity. |
| 92 | +- **`src/catalog.ts` / `src/worker.ts`** — catalog descriptor (no `secretTypes`) and the entry |
| 93 | + that wires the real client into the functions. |
| 94 | + |
| 95 | +## SSGA endpoint facts (why the design is what it is) |
| 96 | + |
| 97 | +Three keyless planes, all needing only the browser User-Agent: |
| 98 | + |
| 99 | +1. **Fund-finder** — `GET /bin/v1/ssmp/fund/fundfinder?country=us&language=en&role=intermediary&product=etfs&ui=fund-finder`. |
| 100 | + One ~0.8 MB object; the fund array is at `data.funds.etfs.datas` (~180 ETFs). Backs `products` |
| 101 | + and the ticker resolution in `resolveTicker`. Scalar fields come as either a bare value or a |
| 102 | + **`[display, raw]` pair** (e.g. `nav: ["$747.70", 747.704983]`); the "no data" sentinel is a |
| 103 | + `"-"` display (its paired raw is a garbage denormal `-5e-324`, so `pairNum` keys off the |
| 104 | + DISPLAY). Helpers: `pairDisp()` (display string, sentinels→null), `pairNum()` (raw number), |
| 105 | + `parseDate()` (ISO `YYYY-MM-DD` → epoch seconds). Asset class is NOT a per-fund field — it comes |
| 106 | + from `data.funds.etfs.categories` (the `assetclass` category's sub-categories carry a name + a |
| 107 | + pipe-delimited `funds` list; `assetClassMap` builds ticker→class, trimming a trailing |
| 108 | + " Sector"). ISIN/CUSIP are pulled by SHAPE from the comma-separated `keywords` string |
| 109 | + (`idsFromKeywords`: 12-char ISIN, 9-char CUSIP), because that string is positionally |
| 110 | + heterogeneous (fund names contain commas). Ticker is `fundFilter` (clean; `fundTicker` carries a |
| 111 | + `®`). AUM is reported in **millions** → `net_assets` scales it to whole USD (×1e6). |
| 112 | + |
| 113 | +2. **holdings-daily** — `GET /library-content/products/fund-data/etfs/us/holdings-daily-us-en-<ticker>.xlsx` |
| 114 | + (lower-case ticker). A real `.xlsx`; header rows (Fund Name / Ticker Symbol / "Holdings: As of |
| 115 | + DD-Mon-YYYY") then the constituent table. Current holdings only. |
| 116 | + |
| 117 | +3. **navhist** — `GET /library-content/products/fund-data/etfs/us/navhist-us-en-<ticker>.xlsx`. |
| 118 | + `.xlsx` with header `Date, NAV, Shares Outstanding, Total Net Assets`, daily back to inception. |
| 119 | + |
| 120 | +**Dates:** date ARGS are real SQL `DATE` (Arrow `DateDay`). The vgi runtime hands a DATE arg to |
| 121 | +`p.args` as a number of epoch **milliseconds**; `dateArgToEpoch` converts it (magnitude-robust: |
| 122 | +epoch-ms, JS Date, bigint, days-since-epoch, or a YYYY-MM-DD string). `parseDate` also handles the |
| 123 | +spreadsheet date shapes `DD-Mon-YYYY` (As-of / NAV Date) and `MM/DD/YYYY` (bond Maturity). |
| 124 | +`nav_history` takes `fund` + optional `start_date`/`end_date` (client-side filter; named `*_date` |
| 125 | +because `END` is reserved). `holdings` takes NO date arg (current only). |
| 126 | + |
| 127 | +## Fund identifier (`fund` arg) |
| 128 | + |
| 129 | +`resolveTicker(get, fund)`: matches the fund-finder catalog case-insensitively and returns the |
| 130 | +canonical ticker (or null = not found). It does NOT throw (spdr.ts is SDK-free); `functions.ts` |
| 131 | +`resolveOrThrow` converts null into an `ArgumentValidationError` with a "list tickers via products" |
| 132 | +hint. The `holdings`/`nav_history` URLs use the ticker directly (lower-cased) — there is no numeric |
| 133 | +portfolio id. Resolution is not cached beyond the 24 h fund-finder memo. |
| 134 | + |
| 135 | +## Commands |
| 136 | + |
| 137 | +```bash |
| 138 | +bun install |
| 139 | +bun test # 34 tests: SDK-free driver + Arrow batch builders + live HTTP-transport E2E |
| 140 | +bun run typecheck # own-source only; scripts/typecheck.sh filters node_modules errors |
| 141 | +./run_tests.sh # haybarn SQLLogic E2E: worker under real DuckDB + community vgi ext |
| 142 | +``` |
| 143 | + |
| 144 | +`run_tests.sh` sets `VGI_TEST_WORKER=bin/vgi-etf-spdr-worker` + `VGI_WORKER_CATALOG_NAME=spdr` and |
| 145 | +runs `test/sql/*.test` (DESCRIBE-based schema asserts + a few live-invariant asserts that hit |
| 146 | +SSGA). CI runs this, the reusable `ts-ci.yml`, and a `vgi-lint` gate at `--fail-on info` |
| 147 | +(currently 100/100). |
| 148 | + |
| 149 | +Typecheck must be a `bash scripts/typecheck.sh` file (not an inline package.json pipeline) — |
| 150 | +`bun run` uses Bun's shell, which mishandles the `grep -v node_modules` filter. Pin |
| 151 | +`typescript ^6.0.3` (5.x descends into SDK `.ts` source and reports external errors). |
| 152 | + |
| 153 | +## Gotchas / conventions |
| 154 | + |
| 155 | +- Emit `Date` (rich repr) for DATE columns via `batchFromColumns`; date fields go through |
| 156 | + `parseDate` (→ epoch seconds) then `dateOrNull`. |
| 157 | +- `noUncheckedIndexedAccess` is on: guard matrix/array cell reads (the parsers use `at(row, col)` |
| 158 | + and null-check before use) so destructured cells don't type as possibly `undefined`. |
| 159 | +- vgi-lint rules to keep satisfied: catalog/schema descriptions must NOT enumerate the worker's own |
| 160 | + functions (VGI173); numeric column comments should state units (VGI131 — e.g. "per share in |
| 161 | + USD", "percent points"); argument docs must NOT restate the data type (VGI313); every function |
| 162 | + needs an agent test task (VGI520 — products/holdings/holdings_scan/nav_history are covered in |
| 163 | + `catalog.ts` `vgi.agent_test_tasks`). |
| 164 | +- Don't add a secret type; this worker is keyless by design. |
| 165 | +- Keep the `holdings` current-only contract: do NOT add `supportsTimeTravel` or an as-of arg. |
| 166 | + |
| 167 | +## DuckDB (manual) |
| 168 | + |
| 169 | +```sql |
| 170 | +LOAD vgi; |
| 171 | +ATTACH 'spdr' AS spdr (TYPE vgi, LOCATION '/path/to/vgi-etf-spdr/bin/vgi-etf-spdr-worker'); |
| 172 | +SELECT ticker, net_assets FROM spdr.products ORDER BY net_assets DESC LIMIT 10; |
| 173 | +SELECT name, ticker, weight_percent FROM spdr.holdings WHERE fund_ticker = 'SPY' ORDER BY weight_percent DESC LIMIT 10; |
| 174 | +SELECT as_of_date, nav FROM spdr.nav_history('SPY', start_date := DATE '2026-01-01') ORDER BY as_of_date DESC; |
| 175 | +``` |
0 commit comments