Skip to content

technical_docu.md

Latest commit

 

History

History
383 lines (333 loc) · 12.3 KB

technical_docu.md

File metadata and controls

383 lines (333 loc) · 12.3 KB
Raw

Model Context Protocol (MCP) Server

1. Goals for a Production-Grade Structure

  • Maintainability: Clean separation of concerns, consistent patterns.
  • Scalability: Stateless in the hot path, horizontal scale-ready.
  • Testability: Every module unit-testable with mocks/stubs.
  • Observability: Logs, metrics, tracing from day one.
  • Extensibility: Plug-in context providers and LLM integrations without touching core logic.
  • Deployment Readiness: Configurable for local dev, staging, and prod.

2. High-Level Folder Structure

server/
├── cmd/                                   # Application entry points (Go binaries)
│   └── server/                            # Main API server binary
│       └── main.go                        # Bootstrap app, wire dependencies, start server
│
├── internal/                              # Private application code (cannot be imported externally)
│   ├── app/                               # Application layer (use cases, orchestration)
│   │   ├── orchestrator.go                # Coordinates request → context retrieval → LLM → response
│   │   ├── pipeline.go                    # Prompt enrichment & execution pipeline
│   │   ├── concurrency.go                 # Goroutine orchestration helpers for parallel context calls
│   │   └── usecases.go                    # High-level use case definitions for the app
│   │
│   ├── domain/                            # Core business logic (entities & interfaces)
│   │   ├── models.go                      # Core entities (LLMRequest, ContextBundle, etc.)
│   │   ├── interfaces.go                  # Contracts for LLM, providers, storage, etc.
│   │   └── errors.go                      # Domain-specific error types
│   │
│   ├── infrastructure/                    # External integrations (frameworks, adapters, drivers)
│   │   ├── api/                           # HTTP/WebSocket layer
│   │   │   ├── routes.go                  # Registers API routes
│   │   │   ├── http_handler.go            # REST endpoint handlers → call use cases
│   │   │   └── ws_handler.go              # WebSocket handlers for streaming
│   │   │
│   │   ├── llm/                           # LLM service implementations
│   │   │   ├── router.go                  # Chooses the best model for the request
│   │   │   ├── vllm_client.go             # vLLM integration
│   │   │   ├── mistral_client.go          # Mistral integration
│   │   │   └── model_capabilities.go      # Metadata: strengths, limits, latency
│   │   │
│   │   ├── context/                       # Context provider implementations
│   │   │   ├── provider.go                # Base interface for all providers
│   │   │   ├── weather.go                 # Weather provider example
│   │   │   ├── finance.go                 # Finance provider example
│   │   │   └── vectordb.go                # VectorDB embedding search provider
│   │   │
│   │   ├── storage/                       # Data persistence & caching
│   │   │   ├── redis_store.go             # Redis client wrapper
│   │   │   ├── pg_store.go                # PostgreSQL client wrapper
│   │   │   └── conversation_repo.go       # Repository for conversation history
│   │   │
│   │   ├── config/                        # App configuration loader
│   │   │   └── config.go                  # Loads YAML/env config into typed structs
│   │   │
│   │   ├── logging/                       # Logging system setup
│   │   │   └── logger.go                  # Configures Zap logger for structured logs
│   │   │
│   │   ├── metrics/                       # Metrics & tracing integrations
│   │   │   └── metrics.go                 # Prometheus metrics + OpenTelemetry tracing
│   │   │
│   │   └── utils/                         # Helper functions
│   │       └── http_client.go             # HTTP client with retries & timeout
│   │
│   └── di/                                # Dependency Injection wiring
│       └── container.go                   # Provides services via Uber-go/dig
│
├── pkg/                                   # Public shared libraries/utilities
│   ├── types/                             # Common type definitions
│   └── errors/                            # Error wrapping utilities
│
├── scripts/                               # DevOps/utility scripts
│   ├── run_local.sh                       # Starts app locally
│   └── migrate_db.sh                      # Runs DB migrations
│
├── configs/                               # Environment configuration files
│   ├── config.local.yaml                  # Local development config
│   ├── config.staging.yaml                # Staging environment config
│   └── config.prod.yaml                   # Production environment config
│
├── tests/                                 # Test suite
│   ├── e2e/                               # End-to-end integration tests
│   │   └── e2e_test.go                    # Tests full request/response flow
│   ├── unit/                              # Unit tests for core logic
│   │   └── orchestrator_test.go           # Tests orchestrator without external services
│   └── mocks/                             # Generated test mocks
│
├── Dockerfile                             # Multi-stage Docker build for production
├── docker-compose.yaml                    # Local stack (Postgres, Redis, MCP server)
├── go.mod                                 # Go module definition
└── go.sum                                 # Go dependency checksums

High Level Architecture Diagram

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#BB2528',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#7C0000',
      'lineColor': '#F8B229',
      'secondaryColor': '#006100',
      'tertiaryColor': '#fff',
      'lineWidth': 8,
      'fontSize':30
    }
  }
}%%
flowchart LR
    Client[React Client / API Consumer] --> API[HTTP/WebSocket API Gateway]
    API --> OR[Core Orchestrator]
    OR --> CE[Concurrency Engine]
    CE --> CP1[Context Provider: Weather]
    CE --> CP2[Context Provider: Finance]
    CE --> CP3[Context Provider: VectorDB]
    OR --> PE[Prompt Enricher]
    PE --> LR[LLM Router]
    LR --> L1[vLLM]
    LR --> L2[Mistral]
    LR --> L3[Qwen]
    OR --> Cache[Redis]
    OR --> Storage[PostgreSQL]
Loading

System Design Diagram

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#BB2528',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#7C0000',
      'lineColor': '#F8B229',
      'secondaryColor': '#006100',
      'tertiaryColor': '#fff',
      'lineWidth': 8,
      'fontSize':30
    }
  }
}%%
graph TD
    subgraph Client Layer
        RC[React Client]
    end

    subgraph API Layer
        API[HTTP/WebSocket API Gateway]
    end

    subgraph Core Logic
        OR[Core Orchestrator]
        CE[Concurrency Engine]
        PE[Prompt Enricher]
        LR[LLM Router]
    end

    subgraph Context Providers
        CP1[Weather Provider]
        CP2[Finance Provider]
        CP3[VectorDB Provider]
    end

    subgraph AI Models
        L1[vLLM]
        L2[Mistral]
        L3[Qwen]
    end

    subgraph Data Layer
        Cache[(Redis)]
        Storage[(PostgreSQL)]
    end

    RC --> API
    API --> OR
    OR --> CE
    CE --> CP1
    CE --> CP2
    CE --> CP3
    OR --> PE
    PE --> LR
    LR --> L1
    LR --> L2
    LR --> L3
    OR --> Cache
    OR --> Storage
Loading

Sequence Flow: Parallel Context Retrieval + Streaming Token Delivery

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#BB2528',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#7C0000',
      'lineColor': '#F8B229',
      'secondaryColor': '#006100',
      'tertiaryColor': '#fff',
      'lineWidth': 8,
      'fontSize':30
    }
  }
}%%
sequenceDiagram
    participant U as User
    participant API as MCP API Gateway
    participant OR as Core Orchestrator
    participant CE as Concurrency Engine
    participant CP1 as Context Provider: Weather
    participant CP2 as Context Provider: Finance
    participant CP3 as Context Provider: VectorDB
    participant PE as Prompt Enricher
    participant LR as LLM Router
    participant L1 as vLLM
    participant L2 as Mistral
    participant L3 as Qwen

    U->>API: POST /chat {prompt, model, contexts[]}
    API->>OR: Validate & forward request
    OR->>CE: Launch parallel context retrieval
    par Parallel Context Calls
        CE->>CP1: Fetch weather context
        CE->>CP2: Fetch finance context
        CE->>CP3: Fetch vector search context
    end
    CP1-->>CE: Weather data
    CP2-->>CE: Finance data
    CP3-->>CE: Vector search results
    CE->>PE: Aggregate context results
    PE->>LR: Pass enriched prompt for routing
    LR->>L1: Dispatch prompt to vLLM
    LR->>L2: Optionally dispatch to Mistral
    LR->>L3: Optionally dispatch to Qwen
    par Parallel LLM Execution
        L1-->>API: Stream tokens
        L2-->>API: Stream tokens
        L3-->>API: Stream tokens
    end
    API-->>U: Push streaming tokens to client
Loading

3. Key Production Practices

a) Dependency Injection

Use Uber-go/dig for wiring services.

func BuildContainer() *dig.Container {
    c := dig.New()
    c.Provide(NewConfig)
    c.Provide(NewLogger)
    c.Provide(NewRedisStore)
    c.Provide(NewPostgresStore)
    c.Provide(NewLLMRouter)
    c.Provide(NewOrchestrator)
    return c
}

b) Configuration Management

  • Precedence: ENV → YAML → defaults
  • Config per environment
  • No hardcoded secrets — use Vault or AWS SSM

Example YAML:

server:
  port: 8080

llms:
  - name: vllm-llama3
    endpoint: http://localhost:8000
    strengths: ["general reasoning", "long context"]
    max_tokens: 16384

c) Context Provider Interface

type ContextProvider interface {
    Name() string
    Fetch(ctx context.Context, query string) (string, error)
}

Each provider runs in its own goroutine.

d) Concurrency Model

  1. Fan-Out: Spawn goroutines for each context provider.
  2. Timeouts & Cancellations: Use context.WithTimeout to avoid hanging calls.
  3. Result Aggregation: Use channels to collect results.
  4. Fan-In: Aggregate all context into a unified bundle.
  5. Concurrent LLM Execution: Allow parallel prompt execution if multiple models are queried.
  6. Streaming Response: Forward tokens as they arrive from LLM.

Example :

func gatherContext(ctx context.Context, providers []ContextProvider) []ContextResult {
    var wg sync.WaitGroup
    results := make(chan ContextResult, len(providers))

    for _, p := range providers {
        wg.Add(1)
        go func(provider ContextProvider) {
            defer wg.Done()
            data, err := provider.Fetch(ctx, "query")
            results <- ContextResult{Name: provider.Name(), Data: data, Err: err}
        }(p)
    }

    wg.Wait()
    close(results)

    var out []ContextResult
    for r := range results {
        out = append(out, r)
    }
    return out
}

e) Prompt Enrichment Pipeline

  1. Collect all contexts concurrently.
  2. Merge into structured context bundle.
  3. Apply prompt templates.
  4. Send to selected LLM.

f) Logging & Metrics

  • Zap for structured logging
  • Prometheus for metrics:
    • LLM latency
    • Context fetch time
    • Cache hit/miss ratio
  • OpenTelemetry for tracing

g) Deployment Model

  • Containerized with multi-stage Docker build
  • Behind NGINX or Caddy for TLS
  • Horizontal scale via Kubernetes HPA or Docker swarm

4. Benefits of This Layout

  • Testable: Mockable modules
  • Maintainable: Clear module boundaries
  • Deployable: Same code works in dev & prod
  • Scalable: Stateless, horizontally scalable
  • Extendable: Drop-in new LLMs or providers