Skip to content

ARCHITECTURE_OVERVIEW.md

Latest commit

 

History

History
119 lines (100 loc) · 5.6 KB

ARCHITECTURE_OVERVIEW.md

File metadata and controls

119 lines (100 loc) · 5.6 KB
Raw

Architecture Overview

Product thesis

Traditional BI tools make users build reports. This product makes users describe outcomes.

The MVP is a local-first, structured-first AI reporting platform that accepts messy CSV/XLSX data, profiles and repairs it, builds a semantic model, captures the user's outcome in plain English, and later turns that into a deterministic, trustworthy report.

Core product rules

  • Outcome-first, not chart-first.
  • AI plans and narrates; deterministic code computes and renders.
  • Raw data is immutable.
  • Cleaned working data and transformation logs are separate, inspectable artifacts.
  • Methodology and confidence are first-class UI elements, not afterthoughts.
  • The renderer owns layout, chart selection guardrails, and export behavior.

Repository shape

  • frontend/: Next.js application for the guided workflow and future report renderer.
  • backend/: FastAPI service, contracts, profiling/repair/semantic services, and session persistence.
  • packages/contracts/: exported JSON Schemas and generated TypeScript types.
  • fixtures/golden-datasets/: seed datasets and manifest files for evals and tests.
  • docs/: implementation docs, template definitions, eval strategy, and QC checklists.
  • scripts/: bootstrap, run, verification, and contract generation helpers.
  • data/: local app metadata, uploaded files, analytical working files, and logs.

Engine map

1. Ingestion & Profiling Engine

  • Input: raw CSV/XLSX upload.
  • Responsibilities: file parsing, schema inspection, type/null/duplicate/date analysis, grain hints, candidate measures and dimensions, domain hypotheses.
  • Output: DatasetProfile.

2. Data Quality & Repair Engine

  • Input: DatasetProfile plus raw table data.
  • Responsibilities: quality scoring, issue detection, safe coercions, category normalization proposals, missing-value strategy recommendations, unusable-field exclusion proposals, and lineage capture.
  • Outputs: DataQualityRepairPlan, CleanedDatasetVersion, TransformationLog.

3. Semantic Modeling Engine

  • Input: cleaned working layer plus profile/repair context.
  • Responsibilities: business-friendly naming, measure and dimension detection, hierarchy hints, time-grain inference, KPI candidates, aliases, and descriptions.
  • Outputs: SemanticModel, KPIRegistryCandidate.

4. Intent & Brief Engine

  • Input: plain-English user goal plus semantic context.
  • Responsibilities: outcome interpretation, audience and tone capture, analysis-type classification, scope and constraints capture, minimal follow-up detection.
  • Output: IntentBrief.

5. Report Planning Engine

  • Input: IntentBrief, SemanticModel, and template metadata.
  • Responsibilities: report archetype choice, section planning, metric and dimension selection, calculation declarations, narrative requirements, caveats.
  • Output: ReportPlan.

6. Deterministic Compute Engine

  • Input: ReportPlan plus cleaned working data.
  • Responsibilities: all numerical calculations, KPI generation, comparisons, outlier scoring, chart-ready and table-ready data preparation.
  • Output: ComputedReportData.

7. Visualization Composition Engine

  • Input: ReportPlan, ComputedReportData, and design tokens.
  • Responsibilities: deterministic template rendering, chart/table variant selection inside guardrails, density handling, responsive layout, empty states, and export readiness.
  • Output: RenderedReport.

8. Trust & Audit Engine

  • Input: artifacts from every prior engine.
  • Responsibilities: show repairs, exclusions, confidence, assumptions, formulas, caveats, and how the report was built.
  • Output: AuditMethodologySummary.

9. AI Gateway & Evaluation Engine

  • Input: provider configs, model manifests, eval cases, and prompt templates.
  • Responsibilities: provider abstraction, routing, fallback, admission tests, tracing, and ongoing eval tracking.
  • Outputs: ProviderConfig, ModelCapabilityManifest, EvaluationResults.

Phase alignment

Phase 0 in this implementation

  • Lock architecture.
  • Define schema contracts.
  • Define report-template metadata and visual lint rules.
  • Build the golden dataset pack and eval strategy.
  • Provide a coded design reference through the frontend shell and template docs.

Phase 1 in this implementation

  • Upload.
  • Profile.
  • Repair proposal and approval.
  • Semantic model preview.
  • Methodology panel v1.
  • Intent capture scaffold.

Deferred to later phases

  • Report planning runtime.
  • Deterministic report computation.
  • Premium report rendering.
  • Prompt-based refinement.
  • HTML/PDF export.
  • Live provider adapters and routing.

First-slice workflow

  1. User uploads one CSV or XLSX file.
  2. Backend persists the raw file and derives a profile.
  3. Backend proposes a repair plan with clear rationale and confidence.
  4. User approves safe repair actions into a separate cleaned working layer.
  5. Backend generates a semantic preview with measures, dimensions, grain, and KPI candidates.
  6. User provides an outcome request in plain English.
  7. UI exposes methodology, confidence, repairs, exclusions, and semantic assumptions throughout the flow.

Persistence and lineage

  • Raw uploads stay immutable.
  • Cleaned working data is written separately.
  • Each approved repair becomes a log entry with before/after semantics.
  • Session metadata persists locally.
  • Analytical work is local-first and single-dataset per session at MVP.

Out of scope for the first vertical slice

  • Drag-and-drop dashboards.
  • Arbitrary chart authoring.
  • Multi-table semantic joins.
  • Collaboration and auth.
  • Scheduled refresh.
  • Full report export implementation.
  • Report rendering driven directly by model-authored UI.