docs: make cloud-sink examples runnable standalone via docker-compose (#947)

Author: scarmuega

examples/aws_lambda/README.md

@@ -13,20 +13,44 @@ flowchart LR
 - **Filters**
   - `SplitBlock`: breaks each block into individual transactions.
   - `ParseCbor`: decodes the raw transaction CBOR into structured records.
-- **Sink** — `AwsLambda`: invokes `function_name` in `region` with each event as the payload.
+- **Sink** — `AwsLambda`: invokes `function_name` in `region` with each event as the
+  JSON payload.
 
 ## Prerequisites
 
 - Built with the `aws` feature.
-- AWS credentials available to the process (env vars, profile, or instance role) with
-  permission to invoke the function.
-- Edit `region` and `function_name` in `daemon.toml` to match your function.
+- A running Docker engine — LocalStack runs each Lambda invocation in its own container
+  (the compose file mounts the Docker socket for this).
 
-## Run
+## Run standalone (LocalStack)
+
+The included `docker-compose.yml` starts [LocalStack](https://www.localstack.cloud/) and
+deploys `my-lambda` (a tiny function in `localstack/handler.py` that logs each event), so
+the example runs without a real AWS account:
 
 ```sh
 cd examples/aws_lambda
+docker compose up -d
+```
+
+Point the AWS SDK at LocalStack with dummy credentials, then run Oura:
+
+```sh
+export AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566
+export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1
 cargo run --features aws --bin oura -- daemon --config daemon.toml
 ```
 
 (or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)
+
+Watch the function get invoked — one `lambda.Invoke => 200` per event:
+
+```sh
+docker compose logs -f localstack | grep "lambda.Invoke"
+```
+
+## Run against real AWS
+
+Skip the compose step and the `AWS_ENDPOINT_URL` export. Provide real credentials (env
+vars, profile, or instance role) with permission to invoke the function, and set `region`
+and `function_name` in `daemon.toml` to match it.

examples/aws_lambda/docker-compose.yml

@@ -0,0 +1,16 @@
+version: "3"
+services:
+  # LocalStack emulates AWS locally, so the example runs without a real AWS account.
+  # The init script under ./localstack deploys a tiny function named `my-lambda`
+  # that logs each event it receives.
+  localstack:
+    image: localstack/localstack:3
+    container_name: localstack-lambda
+    ports:
+      - "4566:4566"
+    environment:
+      - SERVICES=lambda
+    volumes:
+      - ./localstack:/etc/localstack/init/ready.d
+      # LocalStack runs each Lambda invocation in its own container.
+      - /var/run/docker.sock:/var/run/docker.sock

examples/aws_lambda/localstack/handler.py

@@ -0,0 +1,4 @@
+def handler(event, context):
+    # Oura invokes the function once per chain event; the payload is the JSON record.
+    print("oura event:", event)
+    return {"statusCode": 200}

examples/aws_lambda/localstack/init.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+# Runs inside the LocalStack container once it is ready.
+# Packages handler.py and deploys it as the function daemon.toml invokes.
+cd /etc/localstack/init/ready.d
+python3 -c "import zipfile; zipfile.ZipFile('/tmp/fn.zip','w').write('handler.py')"
+awslocal lambda create-function \
+  --function-name my-lambda \
+  --runtime python3.12 \
+  --handler handler.handler \
+  --role arn:aws:iam::000000000000:role/lambda-role \
+  --zip-file fileb:///tmp/fn.zip

examples/aws_s3/README.md

@@ -1,30 +1,52 @@
 # AWS S3 sink
 
-Decode transactions and write each one as an object into an S3 bucket.
+Write each block's raw CBOR as an object into an S3 bucket.
 
 ## Pipeline
 
 ```mermaid
 flowchart LR
-  src[N2N source] --> f1[ParseCbor] --> sink[AwsS3 sink]
+  src[N2N source] --> sink[AwsS3 sink]
 ```
 
 - **Source** — `N2N`: mainnet relay, starting from the chain tip.
-- **Filters** — `ParseCbor`: decodes the raw transaction CBOR into structured records.
-- **Sink** — `AwsS3`: writes objects under `prefix` in `bucket` (`region`).
+- **Sink** — `AwsS3`: writes one object per block under `prefix` in `bucket` (`region`),
+  keyed by `slot.hash`. The `AwsS3` sink stores the block CBOR as-is, so the example runs
+  no filters — a `ParseCbor` / `SplitBlock` filter would change the record type and the
+  sink would reject it.
 
 ## Prerequisites
 
 - Built with the `aws` feature.
-- AWS credentials available to the process (env vars, profile, or instance role) with
-  permission to write to the bucket.
-- Edit `region`, `bucket`, and `prefix` in `daemon.toml` to match your bucket.
 
-## Run
+## Run standalone (LocalStack)
+
+The included `docker-compose.yml` starts [LocalStack](https://www.localstack.cloud/) and
+provisions the `my-bucket` bucket, so the example runs without a real AWS account:
 
 ```sh
 cd examples/aws_s3
+docker compose up -d
+```
+
+Point the AWS SDK at LocalStack with dummy credentials, then run Oura:
+
+```sh
+export AWS_ENDPOINT_URL=http://s3.localhost.localstack.cloud:4566
+export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1
 cargo run --features aws --bin oura -- daemon --config daemon.toml
 ```
 
 (or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)
+
+Inspect the objects Oura wrote:
+
+```sh
+docker exec localstack-s3 awslocal s3 ls s3://my-bucket/mainnet/
+```
+
+## Run against real AWS
+
+Skip the compose step and the `AWS_ENDPOINT_URL` export. Provide real credentials (env
+vars, profile, or instance role) with permission to write to the bucket, and edit
+`region`, `bucket`, and `prefix` in `daemon.toml` to match it.

examples/aws_s3/daemon.toml

@@ -8,8 +8,8 @@ type = "mainnet"
 [intersect]
 type = "Tip"
 
-[[filters]]
-type = "ParseCbor"
+# The AwsS3 sink stores the raw block CBOR, so the block must reach it unparsed
+# (no ParseCbor / SplitBlock filter).
 
 [sink]
 type = "AwsS3"

examples/aws_s3/docker-compose.yml

@@ -0,0 +1,13 @@
+version: "3"
+services:
+  # LocalStack emulates AWS locally, so the example runs without a real AWS account.
+  # The init script under ./localstack provisions the bucket Oura writes to.
+  localstack:
+    image: localstack/localstack:3
+    container_name: localstack-s3
+    ports:
+      - "4566:4566"
+    environment:
+      - SERVICES=s3
+    volumes:
+      - ./localstack:/etc/localstack/init/ready.d

examples/aws_s3/localstack/init.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+# Runs inside the LocalStack container once it is ready.
+# Creates the bucket that daemon.toml writes to.
+awslocal s3 mb s3://my-bucket

examples/aws_sqs/README.md

@@ -13,21 +13,43 @@ flowchart LR
 - **Filters**
   - `SplitBlock`: breaks each block into individual transactions.
   - `ParseCbor`: decodes the raw transaction CBOR into structured records.
-- **Sink** — `AwsSqs`: sends messages to `queue_url` (`region`) with the configured
-  `group_id` (FIFO queues).
+- **Sink** — `AwsSqs`: sends each event as a JSON message to `queue_url` (`region`). For a
+  FIFO queue (`queue_url` ending in `.fifo`) the configured `group_id` is used.
 
 ## Prerequisites
 
 - Built with the `aws` feature.
-- AWS credentials available to the process (env vars, profile, or instance role) with
-  permission to send to the queue.
-- Edit `region`, `queue_url`, and `group_id` in `daemon.toml` to match your queue.
 
-## Run
+## Run standalone (LocalStack)
+
+The included `docker-compose.yml` starts [LocalStack](https://www.localstack.cloud/) and
+provisions the `my-queue` queue that `daemon.toml` points at, so the example runs without a
+real AWS account:
 
 ```sh
 cd examples/aws_sqs
+docker compose up -d
+```
+
+Point the AWS SDK at LocalStack with dummy credentials, then run Oura:
+
+```sh
+export AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566
+export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1
 cargo run --features aws --bin oura -- daemon --config daemon.toml
 ```
 
 (or `oura daemon --config daemon.toml` with a binary built with the `aws` feature.)
+
+Read the messages Oura enqueued:
+
+```sh
+docker exec localstack-sqs awslocal sqs receive-message \
+  --queue-url http://localhost:4566/000000000000/my-queue
+```
+
+## Run against real AWS
+
+Skip the compose step and the `AWS_ENDPOINT_URL` export. Provide real credentials (env
+vars, profile, or instance role) with permission to send to the queue, and set `region`,
+`queue_url`, and `group_id` in `daemon.toml` to match your queue.

examples/aws_sqs/daemon.toml

@@ -17,5 +17,7 @@ type = "ParseCbor"
 [sink]
 type = "AwsSqs"
 region = "us-east-1"
-queue_url = "https://sqs.us-east-1.amazonaws.com/*****/my-queue"
+# Points at the queue created by the bundled LocalStack (see README). For a real
+# AWS queue, use its URL: https://sqs.<region>.amazonaws.com/<account-id>/my-queue
+queue_url = "http://localhost.localstack.cloud:4566/queue/us-east-1/000000000000/my-queue"
 group_id = "my_group"