Skip to content

README.md

Latest commit

 

History

History
341 lines (231 loc) · 16.4 KB

README.md

File metadata and controls

341 lines (231 loc) · 16.4 KB
Raw

FAST ATOMIC FLOW

PHP 8.4 Swoole 6.2 Go 1.26 NATS JetStream Redis 8.0 Jaeger 2.17
PHPStan Level 10 Distributed Semaphores Load Balancer Message Bus Binary Message: 9 bytes
PHPStan Ignore: 0 Memory Leaks: 0 Uptime
AI-assisted Vibe-coding: 0%

Atomic task orchestrator on Swoole + NATS + Go WebSocket proxy

🇬🇧 English | 🇷🇺 Русский

Fast theme - star dust mode

🌐 Live:

What is it

A demo project that visualizes semaphores and queues in a real‑world high‑load architecture.

You will see:

  • How tasks with different concurrency limits compete for resources
  • How semaphores regulate parallel execution
  • How a NATS JetStream queue works
  • All of this — in real time, via WebSocket

Design Philosophy

The project is built so each component minds its own business and doesn't poke into others' stalls.

Low Coupling

Components communicate via DTOs and NATS messages. Want to change the transport or storage layer? You won't need to rewrite business logic. The component isn't tied to one cart.

High Cohesion

Each service does one thing, but with surgical precision. The Worker processes tasks. The Proxy holds connections. The Orchestrator conducts. No mess, no confusion.

Architecture

Component Technology Purpose
API & Workers PHP 8.4 + Swoole Task intake, semaphores, processing
API (Distributed) Go 1.26 + Redis Distributed semaphores, horizontal scaling
Balancer Go 1.26 + net/http Load balancing, health checks, auto-registration
Message Bus NATS (Deez Nutz) Queues, broadcasts, persistence
WebSocket Go 1.26 + Gorilla Real‑time updates, metrics
Queue Storage NATS JetStream Durable queues with replication
Semaphore Store Redis 8.0 + Lua Distributed semaphores, TTL, atomicity
Tracing OpenTelemetry + Jaeger 2.17 Distributed trace collection and visualization

Balancer (Go Balancer)

A custom HTTP load balancer written in Go with automatic API instance registration.

  • Dynamic upstream: API instances self-register with the balancer via /register on startup. No static list, no nginx -s reload
  • Health checks: The balancer probes every instance every 5 seconds. Dead instances are excluded from rotation, revived ones are re-added automatically
  • Lock-free balancing: Atomic pointer swap on immutable lists. Readers (HTTP requests) never block writers (instance registration) (overengineering / cancelled — simple mutex is enough)
  • Round-robin: Requests are evenly distributed across all alive instances
  • Graceful degradation: If all instances are down — returns 503
Component Technology Purpose
Balancer Go 1.26 + net/http Load balancing, health checks, auto-registration

WebSocket Binary Protocol

The WebSocket proxy (Go) communicates with the frontend via a binary protocol — compact, fast, no JSON overhead.

  • Each message is packed into 9 bytes:
    • magic byte — message type (1 byte)
    • status — task status (1 byte, 8 predefined states)
    • taskId + sem — semaphore driver type (1 bit, highest bit) packed with task ID (31 bits, lower bits) into a single uint32 (4 bytes)
    • max_concurrent — concurrency limit (1 byte, 0–255)
    • progress + task_mode — completion percentage (7 bits, 0–100) + task mode (1 bit, highest bit: 0 = observation, 1 = stress) packed into a single byte
    • worker_id — worker that processed the task (1 byte, 0–255)

The binary format ensures minimal overhead (9 bytes per event vs hundreds in JSON) and strict message ordering via FIFO channels.

Hybrid Semaphore Strategy (PHP & Go)

Fast.AF features a dual-driver semaphore system, allowing you to switch between ultra-fast local locking and distributed cluster-wide synchronization. The driver is defined per Flow Theme via YAML configuration, enabling real-time performance comparison.

Drivers:

  • [PHP] PHP Atomic (Shared Memory): High-speed local semaphore using Swoole\Atomic. Best for single-node performance with near-zero latency
  • [API] Go Distributed API: A robust network-based semaphore powered by a dedicated Go microservice. It enables cluster-wide concurrency control, ensuring limits are respected across multiple physical servers

Architectural Features:

  • Auto-Release (TTL): Every distributed permit has a built-in TTL to prevent "zombie" locks if a worker crashes
  • Zero-Overhead Protocol: Internal communication uses a lean binary-ready mapping to distinguish between drivers in monitoring and visualization
  • Visual Distinction: The UI differentiates drivers in real-time (rounded squares for Go API, sharp squares for PHP Atomic)
  • RAND (Random Spam Mode): The "RAND" button triggers chaotic spam mode, firing hundreds of batches with randomized parameters: max_concurrent, task_mode, and semaphore driver change per batch. Perfect for stress testing and visual fireworks on the worker heatmap

Distributed Semaphores (Redis + Lua)

Since May 2026, the Go API uses distributed semaphores backed by Redis 8.0 and Lua scripts.

  • Atomic acquisition: Lua scripts execute atomically inside Redis, guaranteeing consistency even with 100+ API instances
  • Auto-cleanup (TTL): Each semaphore slot has an individual TTL via HEXPIRE. A crashed worker won't hold a slot forever — Redis frees it automatically
  • Distributed release: A semaphore can be released from any API instance, regardless of where it was acquired. SlotUID (a compact string like "5:3" — semaphore 5, slot 3) carries all the information needed
  • Maximum 255 slots: 1 byte for max_concurrent. Enough for any real-world scenario
  • Event‑driven slot acquisition: When no slots are available, the API instance subscribes to Redis Pub/Sub and waits for a slot‑release event. This eliminates busy‑waiting and reduces Redis load
Component Technology Purpose
Redis Redis 8.0 Semaphore store, atomic Lua scripts
SlotUID Go string Compact identifier ("mc:slotIdx")

Distributed Tracing (Jaeger)

Fast Atomic Flow includes distributed tracing powered by OpenTelemetry + Jaeger.

  • Full pipeline visibility: traces propagate from the HTTP request through NATS JetStream into Swoole Task Workers and back to WebSocket clients.
  • Context propagation: traceparent is injected into every task payload and status update, so the trace survives queue boundaries.
  • Swoole-safe isolation: each Task Worker resets its OpenTelemetry context before processing a new task, preventing cross-task trace contamination.
  • Jaeger UI: open fast.af.l3373.xyz/jaeger/ to explore traces in real time.
Component Technology Purpose
Tracing OpenTelemetry + Jaeger 2.17 Distributed trace collection and visualization
Propagation W3C Trace Context traceparent header across HTTP, NATS, Swoole
SDK Custom TraceContext service Span lifecycle, context isolation, force-flush

Jaeger UI base path

If you proxy Jaeger behind nginx at a custom location (e.g., /jaeger/), set the base path in jaeger.yaml:

jaeger_query:
  base_path: /jaeger

Benchmarks

Metric PHP Atomic Go Distributed
Avg latency 0.5 µs 380 µs
Throughput ~2M ops/sec ~2.6K ops/sec
Use case Single-node local locks Cluster-wide consistency

Go Distributed is slower but guarantees consistency across nodes.

How it works

  1. You create tasks through the interface
  2. app (PHP + Swoole) publishes them to NATS
  3. NATS stores tasks in JetStream
  4. Workers pull tasks, check semaphores, execute
  5. Statuses go via NATS to the Go proxy, and from there — to the frontend via WebSocket

Two operation modes

  • Observation mode (task_mode: observation, default): Artificial delay via Co::sleep() — 11 steps of 50-200 ms each. PrecisionProcessor.php

  • Stress test mode (task_mode: stress): Stress test buttons are visually distinct — colored background, border, or accent depending on the theme. Instead of sleep() — real CPU work: hash('sha256', $data) in a loop. HighLoadProcessor.php

The mode is passed in the POST request body when creating tasks (/api/tasks/create).

Key feature: tasks with different max_concurrent values use independent semaphores and can run in parallel without interfering with each other.

Crystal theme - observation mode

In the In Progress zone — no more tasks than the semaphore allows (the number inside the square). The rest wait in Queue or Check Lock.

Quick start

Run from pre-built images (GHCR)

git clone https://github.com/centaur-vova/fast-atomic-flow.git
cd fast-atomic-flow
cp .env.example .env
docker compose -f docker-compose.prod.yaml up -d --scale api=3

This launches 3 Go API instances, the balancer, Redis, NATS, Jaeger, and PHP Swoole

After starting, open http://localhost:9501

Local development

For those who want to dig into the code, change the workflow, and run everything locally (PHP + Go natively, NATS in Docker) — see Local Development Workflow

Configuration

NATS

Variable Default Description
NATS_HOST deez-nutz NATS server host
NATS_PORT 4222 NATS port
NATS_TOKEN alfa-omega Access token
NATS_TIMEOUT_SEC 1 Response timeout
NATS_STREAM_TASKS tasks Stream name for tasks

Swoole

Variable Default Description
SERVER_PORT 9501 HTTP API port
SERVER_WORKER_NUM 6 Number of workers
TASK_SEMAPHORE_MAX_LIMIT 255 Maximum semaphore limit

Go WebSocket Proxy

Variable Default Description
WS_PORT 8080 WebSocket port

Semaphore & Retry tuning

These settings control how tasks behave when the semaphore is busy:

Variable Default Description
TASK_LOCK_TIMEOUT_SEC 5 Maximum time a task waits for a semaphore slot before giving up
TASK_RETRY_DELAY_SEC 2 Delay before re‑queueing a task after a failed lock attempt
TASK_MAX_RETRIES 3 How many times a failed task is retried before being marked as retries_failed

⚠️ Important: These settings affect task fairness. Too many retries can overload the queue.

Technical specifications

  • Runtime: PHP 8.4, Go 1.26
  • Engine: Swoole 6.2+, Gorilla WebSocket
  • Message Bus: NATS JetStream 2.12+
  • Queue Capacity: 10,000 tasks (configurable)
  • Concurrency: 1 to 255 (configurable)
  • API Instances: 3 (scales via --scale api=N)
  • Semaphore Store: Redis 8.0 + Lua scripts
  • Balancer: Round-robin, auto-registration, health checks every 5 seconds
  • Tracing: OpenTelemetry + Jaeger 2.17

Themes

Fast Atomic Flow supports visual themes. Each theme is defined as a separate YAML file and can be switched via URL parameter ?theme=<name>.

Easy switching: Just click the theme links in the page footer — fast 🚀, crystal 💎, or sin city 🖤. No need to type URLs.

Sin City theme - noir style with RAND button

Sin City theme: noir aesthetics, RAND button active — hybrid semaphore stress test.

Built‑in themes:

  • fast — default neon style
  • fluttershy — pastel rainbow, gentle and caring. For ponies
  • crystal — icy blues and purples
  • sin-city — noir, mostly gray with red accents

Theme‑aware task buttons: Each button can specify its own semaphore driver (shared for PHP Atomic, api for Go Distributed).
The RAND button in every theme fires random batches on both drivers simultaneously — perfect for stress testing the hybrid architecture.

Custom themes: You can create your own theme by adding a new folder under themes/ with theme.yaml (colors, zone coordinates, button sets, per‑button semaphore drivers, etc.).
See the Wiki for details.

Support the Journey

If this project helps you or you'd like to support further development:

  • USDT (TRC20): TYEZ68z59jDZTiwyhAnzcBnAxym9qjEr5R
  • TON: UQAucXLX4BDU5o-DkckiwFdS-bbWq52h6T76hpvR-5D5IL63

Every contribution helps keep the code flowing.

Author

Dmitry Shmanatov (Centaur-Vova)

© 2026 l3373.xyz