README.md

play-with-claude 🎮🤖

An AI (Claude) playing PlayStation 2 against you on PCSX2 — like a coach who studies the opponent and writes the playbook before the fight.

Documentação em português: README.pt-BR.md

You play on Pad 1. Claude plays on Pad 2. Before the match it analyzes your character, counter-picks, and writes a conditional game plan — "Hwoarang tends to open with a high-kick rush, so I start blocking high and punish with d+1,2". During the round, zero API calls happen: a local executor runs the playbook at 60Hz. Every ~30s the AI receives telemetry, sees where its prediction failed, and ships a revised plan — announced on a screen overlay, like a coach shouting from the corner.

When its prediction fails, it loses. That's intentional — it's what makes the project fun: it plays like someone who prepared for the matchup, not like an aimbot.

Live benchmark: Claude vs. me in Tekken 5

The project is currently going through an internal benchmark to make Claude better at Tekken 5, run by a dedicated benchmark agent I built with my Anthropic API key — separate from the in-game planner. Each match's telemetry and post-match diagnosis feed back into the next game plans.

Human-vs-AI scoreboard so far:

Matches played	Claude wins	My wins	Claude win rate
20	8	12	40%

Eight wins in twenty matches — and its reads are visibly getting sharper with every session. I can feel its skills being honed day by day.

How it works

                 you (keyboard or controller) ──▶ Pad 1 ─┐
                                                         ├─▶ PCSX2 (e.g. Tekken 5)
   AI: virtual gamepad (uinput) ────────────────▶ Pad 2 ─┘        │
            ▲                                                     │ RAM via PINE (60Hz)
            │ executes the current plan                           ▼
      ┌─────┴──────────────────────────── bridge (Python) ───────────────┐
      │  60Hz executor  │  telemetry  │  async replanner  │  overlay     │
      └────────────────────────────────────┬─────────────────────────────┘
                                           ▼
                                      Claude API
                        (initial plan, replans, diagnosis)

API latency (1–3s per call) stays out of the game loop: Claude thinks when the clock isn't running (character select) or in the background (replans, while the old plan keeps playing). Plan swaps are atomic — the AI never freezes.

The game plan

Claude returns validated JSON (structured outputs + Pydantic) in this shape:

{
  "character": "Paul",
  "opponent_read": "Hwoarang tends to open with a high-kick rush",
  "strategy": "Block high on the opening, punish with d+1,2 and look for the deathfist",
  "macros": [
    { "name": "fast_punish", "steps": [ { "hold": ["down", "cross"], "frames": 3 } ] }
  ],
  "phases": [
    {
      "name": "defensive opening",
      "until_seconds": 5,
      "rules": [
        {
          "name": "punish after blockstun",
          "when": [ { "sensor": "distance", "op": "<", "value": 2.0 } ],
          "do": "fast_punish",
          "priority": 5,
          "cooldown_frames": 45
        }
      ]
    }
  ]
}

• Phases split the round over time (opening → pressure → adaptation).
• Rules are evaluated every frame: the highest-priority rule whose conditions (when, AND) hold fires, respecting cooldowns.
• Macros are input sequences timed in frames (1 frame = 16.7ms) — frame-perfect execution is guaranteed; choosing the moment is where Claude's prediction lives and dies.
• Sensors come from game RAM via PINE (health, position, distance) plus the built-in match_time, phase_time and frame. Without mapped sensors, plans fall back to time-based conditions only.

Why PCSX2 makes this possible

We mapped the emulator's source code before writing the bridge. The facts the architecture rests on:

Fact	Where it lives in the PCSX2 code
Keyboard and controller converge into the same funnel and become an emulated DualShock 2 — the game can't tell the source	InputManager → Pad::SetControllerState (pcsx2/SIO/Pad/Pad.cpp:546)
SDL controllers work without window focus (keyboard doesn't) — hence the uinput virtual pad	polling in VMManager.cpp:2924 → InputManager.cpp:1697, independent of Qt
The TAS system already overrides the pad live — input injection is architecturally trivial	PadData::OverrideActualController() (pcsx2/Recording/PadData.cpp:95)
PINE (IPC over unix socket) reads/writes game RAM, exposes title/serial and savestates	opcodes in pcsx2/PINE.cpp:144-163
PINE has no input or screenshot command — the full enum has 16 opcodes	same
A native OSD exists (Host::AddOSDMessage, pcsx2/Host.h:52) but isn't exposed via PINE — hence the external overlay	future upgrade via patch

The virtual gamepad presents itself as an Xbox 360 controller (vendor 045e, product 028e), so PCSX2's SDL recognizes it with a full mapping, as if it were real hardware.

Setup

1. PCSX2

1. Install PCSX2 (nightly recommended) and configure your game.
2. Enable PINE: Settings → Advanced → PINE → Enable (default slot 28011). On Linux the socket shows up at $XDG_RUNTIME_DIR/pcsx2.sock.

2. uinput permission (Linux)

The virtual pad needs access to /dev/uinput:

sudo modprobe uinput
echo 'KERNEL=="uinput", MODE="0660", GROUP="input"' | sudo tee /etc/udev/rules.d/99-uinput.rules
sudo udevadm control --reload-rules && sudo udevadm trigger
sudo usermod -aG input $USER   # log out and back in afterwards

3. Bridge

git clone git@github.com:teuzowebdeveloper9/play-with-claude.git
cd play-with-claude
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
export ANTHROPIC_API_KEY="your-key"

4. Bind the AI's pad

pwc vpad-test

With the command running, open Settings → Controllers → Pad 2 in PCSX2 and map the buttons using the "Microsoft X-Box 360 pad" device that appeared. Your controller/keyboard stays on Pad 1.

Usage

pwc identify                      # shows the running game (title, serial, status)
pwc profile                       # classifies the game via Claude and caches it in games/<serial>.json
pwc plan --opponent "Hwoarang"    # generates a game plan and prints the JSON (dry run)
pwc play --opponent "Hwoarang"    # full loop: plan + executor + replans + overlay
pwc play --opponent "Hwoarang" --character "Paul" --replan 20

The play flow: generate the plan (the AI announces its counter-pick and its read on the overlay) → pick the characters in-game → the executor plays → every --replan seconds the telemetry goes to Claude, which returns a diagnosis plus a revised plan.

Per-game sensors

The profile in games/<serial>.json accepts manually mapped RAM addresses (the PCSX2 cheats/pnach community is a great source):

{
  "sensors": {
    "p1_hp":  { "addr": "0x00A2F1C4", "width": 2 },
    "p1_x":   { "addr": "0x00A2F200", "width": 4, "scale": 0.001 },
    "p2_x":   { "addr": "0x00A2F600", "width": 4, "scale": 0.001 }
  }
}

With p1_x and p2_x present, the derived distance sensor becomes available automatically.

Project status

• PINE client (RAM, game identity, savestates, batched reads)
• uinput virtual gamepad with Xbox 360 identity
• Game plan schema + 60Hz executor with phases/priority/cooldown
• Planner via Claude API with structured outputs (plan, replan, classification)
• Telemetry + async replan with hot swap
• tkinter overlay showing the AI's "thinking"
• Full CLI + unit tests for the protocol and plan evaluation
• End-to-end validation against real PCSX2 (20 benchmark matches played in Tekken 5)
• Fully autonomous character select via screenshot (vision)
• Internal benchmark loop with a dedicated agent on the Anthropic API
• Tekken 5 sensor map (health/position via community pnach)
• Macro calibration in training mode (verifies the moves come out)
• Trigger-based replan (lost X% health) in addition to the fixed interval
• Optional PCSX2 patch: PINE opcodes for OSD and screenshots

Known limitations

• Tested on Linux (uinput is Linux-specific). The Windows equivalent would be ViGEm.
• A real-time fighting game against a human is the hard scenario by design: the AI wins when it reads you right and loses when you break its script.

License

MIT