Core engine powering Calibrate, a framework for evaluating AI agents which let you move from slow, manual testing to a fast, automated, and repeatable testing process for your entire agent stack:
Text to Text (LLMs): Evaluate the response quality and tool calling of your LLMs for multi-turn conversations and find the find LLM for your agentHuman alignment: Create LLM judges to make your evaluations scalable and reliable with human in the loop.Speech to Text (STT): Benchmark multiple providers (Google, Sarvam, ElevenLabs and more) on your dataset across 10+ indic languages using metrics optimised for agentic use casesText to Speech (TTS): Benchmark generated speech by multiple providers automatically using an Audio LLM Judge across 10+ indic languagesSimulations: Simulate realistic conversations using realistic user personas and scenarios to test failure modes for your agent (including interruptions for voice agents)
pip install calibrate-agentcalibrate-agent # Interactive main menu
calibrate-agent stt # Benchmark STT providers
calibrate-agent tts # Benchmark TTS providers
calibrate-agent llm # Interactive LLM evaluation
calibrate-agent simulations # Interactive text or voice simulationsFor the web version, see the frontend and backend repositories.
Install development dependencies once (requires uv):
uv sync --extra devRun the full test suite:
uv run pytest tests/Enable the project's git hooks so the pre-commit test
runner fires on commits to main:
git config core.hooksPath .githooksEvery contributor needs to run it once.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
