Universal token counting for 20+ LLM providers.
Count tokens accurately for any LLM—Claude, OpenAI, Llama, Mistral, and 16+ more. One API for all providers.
PyTokenCalc provides universal token counting for ANY LLM. Support for 20+ providers (cloud + local), custom models, pattern-based forward compatibility, 99%+ accuracy.
from pytokencalc import count_tokens
# Works with any model
tokens = count_tokens("Your text here", model="claude-3-opus")
print(f"Tokens: {tokens}")
# No need to specify provider—it's auto-detected
tokens = count_tokens("Your text here", model="gpt-4-turbo")Works with 20+ LLM providers:
- OpenAI (GPT-4, GPT-3.5, etc.)
- Anthropic (Claude family)
- Open-source models (Llama, Mistral, etc.)
- Local models
- Custom tokenizers
- And more
- Single API for all providers
- 99%+ accuracy
- Forward-compatible with new models
- No API keys required for counting
- Fast, offline counting
- Custom model support
Provider Support
- 20+ LLM providers (OpenAI, Anthropic, Google, Meta, etc.)
- Local models (Ollama, LLaMA, Mistral)
- Custom tokenizers
- Forward-compatible pattern matching
Accuracy
- 99%+ accuracy on standard models
- Handles edge cases (special tokens, Unicode)
- Regular updates for model changes
- Fallback strategies
Integration
- Drop-in replacement for tiktoken
- Async support
- Batch processing
- Cost estimation
- Python 3.10+
- Regex engine (built-in)
- Optional: requests for provider APIs
- Optional: transformers for local models
pip install pytokencalc
# or with uv
uv pip install tokencalc
# Verify installation
tokencalc --version- Estimate API costs before making requests
- Optimize prompts to stay under token limits
- Monitor token usage in production
- Plan batch operations
- Cost tracking and billing
from pytokencalc import count_tokens, estimate_cost
# Count tokens
text = "Your long text here"
tokens = count_tokens(text, model="claude-3-opus")
# Estimate cost
cost = estimate_cost(tokens, model="claude-3-opus")
print(f"Estimated cost: ${cost:.4f}")
# Batch counting
texts = ["text1", "text2", "text3"]
counts = [count_tokens(t, model="gpt-4") for t in texts]MIT License - See LICENSE