Flyte Agent Loop Engineering

A fully functional loop-engineering system built on Flyte 2. Three scheduled flyte.ai.agents.Agent pipelines cooperate over a shared, durable memory to autonomously take GitHub issues all the way to reviewed pull requests — and then grade themselves.

Want to build your own agents? The flyteorg/flyte-agent-plugins repo provides the Flyte agent plugin and building blocks this project is built on.

Pipeline	Cadence	What it does
builder	every 5 min	Claims an open issue (via a "dibs" comment), implements it with tests/examples/docs, has a verifier sub-agent check the work, then opens a PR.
reviewer	every 5 min	Claims an open agent PR, reads its review comments, makes + verifies fixes, pushes them, and releases the claim for follow-ups.
distiller	every 10 min	Uses a distiller Agent to dedupe + consolidate the builder's and reviewer's run history into a compact, high-signal "lessons" memory (fed back to them as context), and publishes success-rate metrics, the memory filesystem, and per-run reasoning traces as a Flyte report.

See docs/architecture.md for the full design, or the slide deck for the tour.

Publishing the slides: the deck is a single self-contained docs/index.html (no build step). A GitHub Actions workflow (.github/workflows/pages.yml) publishes docs/ to GitHub Pages on every push to main that touches it. One-time setup: Settings → Pages → Source: GitHub Actions. GitHub then serves it at https://unionai-oss.github.io/flyte-agent-loop/.

How it works

• Dibs — before working an issue/PR, a run posts an (invisible) HTML-comment marker. Future scheduled runs parse the markers and skip anything with an active claim, so overlapping cron fires never double-work. Claims expire and can be released. Logic lives in dibs.py and is pure + unit tested.
• Verifier sub-agent — the builder/reviewer agents only propose a change plan; a separate strict verifier agent checks correctness & completeness, and the PR is opened / fixes are pushed only if it passes.
• Shared memory — one keyed flyte.ai.agents.MemoryStore holds per-run records and a consolidated "lessons" digest. Every 10 minutes the distiller pipeline runs a distiller Agent that dedupes and consolidates the newest runs into that digest (as much signal as possible per token); the builder/reviewer agents read it as context so they learn from past verifier feedback.

Layout

The package is meant to be read top-to-bottom, from the deploy entrypoint down to the low-level building blocks (src/flyte_agent_loop/__init__.py is the reading guide):

src/flyte_agent_loop/
  # 1. deploy
  deploy.py            # CLI entrypoint: deploy the env + triggers, or run one pipeline
  environments.py      # the shared Image + secrets + TaskEnvironment every task attaches to
  # 2. the pipelines (the "loop")
  builder_agent.py     # Pipeline 1 (cron */5): an issue -> a PR (or, for a spec, new issues)
  reviewer_agent.py    # Pipeline 2 (cron */5): an agent PR -> verified fixes pushed
  distiller_agent.py   # Pipeline 3 (cron */10): run history -> compact "lessons" memory + report
  pipeline.py          # run bookkeeping (record / release / finish) shared by builder & reviewer
  # 3. agent definitions
  agents.py            # the Agent factories, their prompts, and output parsers
  llm.py               # the custom call_llm callback (generous max_tokens)
  # 4. tools
  tools.py             # durable @env.task GitHub tools handed to the agents
  staging.py           # in-process "stage a proposal" tools the pipeline verifies then applies
  # 5. building blocks
  dibs.py              # pure cooperative-claim ("dibs") state machine
  github_client.py     # small, mockable GitHub REST client
  memory.py            # read/write helpers over the shared MemoryStore
  evals.py             # the RunRecord, metrics, and context compaction
  introspect.py        # read a run's durable sub-actions (its "reasoning trace")
  reports.py           # Flyte report rendering (metrics, memory, traces) + styling
  config.py            # env-var Settings
  common.py            # the current run's id/name + UTC time helpers
tests/                 # hermetic pytest suite (no cluster/network/LLM needed)
examples/run_local.py  # run one pipeline once, locally

Setup

This project uses uv to manage the environment. If you don't already have it (uv --version to check), install it:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# macOS (Homebrew)
brew install uv

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then create the environment and install the project:

uv venv --python 3.13 && source .venv/bin/activate
uv pip install -e ".[dev]"

Configure the target repo and model (see .env.example):

export FLYTE_AGENT_REPO=your-org/your-sandbox-repo   # a repo you own!
export FLYTE_AGENT_MODEL=claude-sonnet-4-5

Create a GitHub token

The agent reads and writes the target repo through a GitHub fine-grained personal access token (PAT). Create one scoped to only the repo in FLYTE_AGENT_REPO:

1. GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token (or go straight to https://github.com/settings/personal-access-tokens/new).

2. Token name — e.g. flyte-agent-loop; set an Expiration.

3. Resource owner — the user or org that owns the target repo.

4. Repository access → Only select repositories → pick the repo you want the agent to work on (the one in FLYTE_AGENT_REPO).

5. Permissions → Repository permissions — set each of these to Read and write:

   Permission	Access	Why the agent needs it
   Contents	Read and write	read repo files, create branches, commit changes
   Pull requests	Read and write	open/update PRs, read + post PR comments
   Issues	Read and write	read issues, post the "dibs" claim comments
   Discussions	Read and write	read/participate in repo discussions

   (Metadata: Read-only is selected automatically and is required.)

6. Generate token and copy the github_pat_... value — you won't see it again.

If you use a classic PAT instead, grant the repo scope (full control of private repositories). Fine-grained tokens are strongly preferred because they can be limited to the single target repo.

Run on a local Flyte Devbox

The Flyte Devbox is a full single-node Flyte cluster that runs locally in Docker — tasks execute in containers with schedules and reports, just like a remote cluster, but with nothing to provision. It's the easiest way to exercise the scheduled pipelines end-to-end on your machine.

Prerequisites: Docker running (and kubectl if you want to inspect the cluster). The flyte CLI ships with the flyte package installed above.

1. Install Docker. The devbox runs the Flyte cluster inside Docker, so you need a working Docker Engine first:

   • macOS / Windows — install Docker Desktop (or, on macOS, brew install --cask docker) and launch it.

   • Linux — install Docker Engine for your distro. A quick path on Debian/Ubuntu:

     curl -fsSL https://get.docker.com | sh
     sudo usermod -aG docker "$USER"   # then log out/in so `docker` runs without sudo

   Confirm the daemon is up before continuing:

   docker run --rm hello-world

2. Start the devbox (first run pulls the image; UI at http://localhost:30080/v2, image registry at localhost:30000):

   flyte start devbox

3. Point the CLI/SDK at it. Generate a config for the local cluster (writes ./config.yaml, which flyte and flyte.init_from_config() auto-discover):

   flyte create config \
     --endpoint localhost:30080 --insecure --builder local \
     --project flytesnacks --domain development \
     --registry localhost:30000

   (It's gitignored. Use -o ~/.flyte to write ~/.flyte/config.yaml instead.)

4. Add the secrets to the devbox cluster (same names as before; see Create a GitHub token):

   flyte create secret --project flytesnacks --domain development github-token --value github_pat_xxx
   flyte create secret --project flytesnacks --domain development anthropic-api-key --value sk-ant-xxx

5. Deploy the pipelines (uses ~/.flyte/config.yaml via flyte.init_from_config()); the local image builder builds into the devbox registry:

   export FLYTE_AGENT_REPO=your-org/your-sandbox-repo
   python -m flyte_agent_loop.deploy  # activate schedules on the devbox
   python -m flyte_agent_loop.deploy --run builder  # or trigger one run now {builder, reviewer, distiller}

   Watch executions and reports at http://localhost:30080/v2.

6. Pause / tear down when done:

   flyte stop devbox              # pause (keeps state; resume with `flyte start devbox`)
   flyte delete devbox --volume   # remove the container and its storage volume

The manual integration test can also target the devbox instead of the demo cluster — point tests/integration/config.yaml at endpoint: dns:///localhost:30080 with builder: local.

Deploy to a remote tenant (e.g. a Union tenant)

Deploying to a real, always-on tenant (a Union tenant, or any remote Flyte 2 cluster) is the same flow as the devbox — the only differences are where the CLI points, how you authenticate (device flow instead of insecure localhost), and that the image is built by the remote builder instead of a local registry. The pipeline code and the deploy command are unchanged.

1. Point the CLI/SDK at your tenant. Generate a config for the remote cluster (auth negotiation opens a browser on the first command — device flow):

   flyte create config \
     --endpoint <your-tenant>.unionai.cloud \
     --project flytesnacks --domain development \
     --builder remote \
     -o ~/.flyte            # writes ~/.flyte/config.yaml (auto-discovered)

   Or write the config by hand (mirrors tests/integration/config.yaml), setting org to your Union org:

   admin:
     endpoint: dns:///<your-tenant>.unionai.cloud
     authType: DeviceFlow      # opens a browser to authenticate
   image:
     builder: remote           # the tenant builds + hosts the task image
   task:
     org: <your-org>
     project: flytesnacks
     domain: development

2. Add the secrets on the tenant (same names as the devbox; you need permission to create secrets in the target project/domain — see Create a GitHub token):

   flyte create secret --project flytesnacks --domain development github-token --value github_pat_xxx
   flyte create secret --project flytesnacks --domain development anthropic-api-key --value sk-ant-xxx

3. Deploy the pipelines. Identical to the devbox — deploy.py uses flyte.init_from_config(), so it picks up ~/.flyte/config.yaml and the remote builder builds + pushes the image the tenant will run:

   export FLYTE_AGENT_REPO=your-org/your-sandbox-repo
   export FLYTE_AGENT_MODEL=claude-sonnet-4-5
   python -m flyte_agent_loop.deploy               # deploy + activate the three cron triggers
   python -m flyte_agent_loop.deploy --dryrun      # or plan only, without applying
   python -m flyte_agent_loop.deploy --run builder # or trigger one run now

   The deploy prints the URL of each activated trigger. Watch executions and reports in your tenant's UI.

Use a different FLYTE_AGENT_MEMORY_KEY (and/or FLYTE_AGENT_ID) per deployment if several environments share a tenant/project, so their shared memory and dibs claims don't collide. To point at a different project/domain, pass -p/-d (or change task.project/task.domain in the config).

Stop the agents (deactivate all triggers)

Deploying activates three cron triggers. To stop the agents from firing without un-deploying, deactivate each trigger with flyte update trigger <trigger-name> <task-name> --deactivate:

flyte update trigger builder_every_5m  flyte_agent_loop.builder --deactivate -p flytesnacks -d development
flyte update trigger reviewer_every_5m flyte_agent_loop.reviewer --deactivate -p flytesnacks -d development
flyte update trigger distiller_every_10m flyte_agent_loop.distiller --deactivate -p flytesnacks -d development

Use the same --config/endpoint (and -p/-d) you deployed with. Verify with flyte get trigger -p flytesnacks -d development.

• Reactivate later: rerun the commands with --activate (or just redeploy — python -m flyte_agent_loop.deploy re-activates them, since the triggers default to auto_activate=True).
• Remove entirely: flyte delete trigger <trigger-name> <task-name> -p … -d ….

Deactivating stops the schedules but leaves the tasks deployed, so any run already in flight finishes and ad-hoc --run invocations still work.

Test

pytest -q                 # unit suite (scoped to tests/unit via pyproject)

tests/unit is fully hermetic: the claim state machine, evals, and agent-output parsers are pure functions, and the GitHub client is exercised against an in-memory httpx.MockTransport. No cluster, network, or LLM key required. This is what CI runs (.github/workflows/unit-tests.yml, Python 3.11–3.13).

tests/integration runs against a live Union tenant and is manual (skipped unless RUN_INTEGRATION=1). It's intended for Union employees running against an actual tenant — tests/integration/config.yaml points at the demo tenant with device-flow auth (admin.authType: DeviceFlow), so the first run opens a browser to authenticate:

RUN_INTEGRATION=1 pytest tests/integration -m integration -s

See tests/integration/README.md for details.