ALF Test Suite — Test Case Generation (PoC)

Internal PoC for Channel Talk's ALF Test Suite. It automatically generates ALF chatbot test cases from real customer-support chat logs, so that ALF's behavior can be evaluated against realistic, reproducible scenarios.

Each generated test case describes what a customer does (instruction + opening_message) and how the agent should respond (success_criteria), written to generalize beyond the specific customer/data in the source chat.

Overview

Real ALF conversations are pulled from MongoDB / QueryPie, filtered down to meaningful single-topic chats, and passed to an LLM that extracts a structured, value-independent test case. Criteria are then verified against the customer's actual request so that the test only asserts what the customer really asked for.

• Model access: Anthropic SDK (AsyncAnthropic) via the internal PRISM gateway (base_url=https://prism.ch.dev, PRISM_API_KEY). Default model: claude-sonnet-4-6.
• Scope: Channel 1 (ALF_engaged_Channel_1).

Pipeline

MongoDB / QueryPie  →  sample_userchats  →  generate_test_cases  →  JSON + viewer
   (raw chat logs)      (filter)              (LLM extract+verify)

1. Sample & filter — notebooks/sample_userchats_channel1.ipynb

• Random-samples ALF-engaged Korean chats for Channel 1 from messages_v4 (MongoDB), then pulls full event context via a QueryPie SQL join over alf_sessions / alf_events / post_evaluation_results (includes cx_score, cx_summary).
• build_conversation reconstructs the event log into readable conversation text (MessageEvent, RagResultsSummaryEvent, ToolsEvent).
• Filters to meaningful, focused chats via is_meaningful: ≥2 user and ≥2 assistant turns, resolved (resolvedFinish) or ≥8 total turns, and ≤2 topics (topic count is judged by the LLM in count_topics).
• Output: data/input_data/ALF_engaged_Channel_1_filtered/context_<tail>.csv (~38 chats from 145 in the current run).

2. Generate — notebooks/generate_test_cases_channel1.ipynb

Runs the filtered chats in batch through TestCaseGenerator (scripts/test_case_generator.py), with per-chat exception handling and a concurrency semaphore.

TestCaseGenerator.generate is a two-step LLM flow:

1. Extract — a forced tool call (extract_test_case) produces topics, instruction, opening_message, criteria_mapping, and success_criteria. The prompt enforces key principles: describe only customer behavior; abstract away agent specifics; write for the first topic only; never embed concrete values (channel names, dates, amounts); anchor each criterion to an instruction step via (#N).
2. Verify (_verify) — a second LLM pass drops success_criteria items not grounded in the customer's actual request (agent-initiated content, off-topic items, unanswered trailing questions, unpredictable escalations), recording each drop in removed_criteria.

Task grounding: if a chat invoked an ALF task, load_tasks / extract_task_ref / build_task_context recover the task's step definitions (agent nodes, branch conditions) from task_channel_1.csv and feed them in as a rubric reference for [task] criteria. task_needs_user_profile flags cases gated on a preset user profile (membership / linked channel / service plan), which can't be reproduced without that profile.

3. Output — JSON files + viewer copies (see below).

Output format

Each test case is a JSON object:

{
  "user_chat_id": "69d5a50e...",
  "cx_score": 4.7,
  "cx_summary": "ALF ...",
  "task_id": "113",                 // null if no task ran
  "topics": ["요금", "..."],         // all customer topics, in order; only the 1st is used
  "instruction": "1. 고객은 ...\n2. ...",   // numbered customer actions
  "opening_message": "...",         // first message the simulated user sends
  "criteria_mapping": [             // planning step: instruction step -> rubric points
    { "scenario_step": 1, "rubrics": ["[resolution] ... "] }
  ],
  "success_criteria": [             // final flattened rubric, 3–5 items
    "[resolution] 새로 발생한 알림 메시지의 내용을 안내한다 (#3)"
  ],
  "removed_criteria": [             // items dropped in verify, with reasons
    { "criterion": "...", "reason": "..." }
  ]
}

Criterion tags: [resolution] (direct answer to the core need), [context] (important supporting context), [task] (correct task-flow step), [escalation] (predictable-from-instruction escalation). Each ends with a (#N) anchor to its instruction step.

The generate notebook writes several variants (suffix on data/output_data/test_cases_channel1_<tail>):

File	[task] criteria	[tag] labels	mapping / removed	Notes
.json (default)	✗	stripped	✗	anchors kept
_with_tag.json	✗	kept	✗
_task_included.json	✓	kept	✓	full verify rationale
_no_user_profile.json	✗	stripped	✗	profile-gated cases removed
_sampled.json	—	—	—	curated SELECTED_IDS subset

Copies are also written into viewer/ (test_cases*.json, data_filtered.json) so each variant is selectable in the viewer.

Viewer — viewer/

Static, single-file web viewer (index.html, no build step) for browsing generated test cases. Left sidebar lists chats by cx_score; the main pane shows the reconstructed conversation next to the generated test case (instruction, opening message, tagged criteria). A dropdown switches between the test_cases*.json datasets. Just open viewer/index.html in a browser.

How to run

Prereqs: Python 3.10+, PRISM_API_KEY in the environment (or .env), and VPN access for MongoDB/QueryPie steps.

pip install anthropic pydantic pandas python-dotenv
export PRISM_API_KEY=sk-pri-...

1. Sample & filter — run notebooks/sample_userchats_channel1.ipynb. (The MongoDB/QueryPie cells are commented out; provide/refresh data/input_data/ALF_engaged_Channel_1/context.csv, then run the filter cells to produce the filtered CSV. Set tail to date the files.)
2. Generate — run notebooks/generate_test_cases_channel1.ipynb. Set in_tail / out_tail, adjust LIMIT (None = all) and CONCURRENCY, then run all cells. Outputs land in data/output_data/ and viewer/.
3. Browse — open viewer/index.html.

Layout

scripts/test_case_generator.py   # TestCaseGenerator + prompts + task grounding (core logic)
notebooks/
  sample_userchats_channel1.ipynb    # step 1: sample + filter
  generate_test_cases_channel1.ipynb # step 2: batch generate + save
data/input_data/                 # sampled chat contexts + task_channel_1.csv
data/output_data/                # generated test case JSON variants
viewer/                          # static web viewer for generated test cases