Report approximate cost of an event to Slack upon event completion

[jencompgeek]

Add capability to NumberCruncher agent to calculate (from Langsmith trace info) and store the approximate LLM cost (and possibly other cost factors?) for an event and report to Slack

[jencompgeek]

numberCruncher: Cost Per Event (Future Implementation)

Goal

When a conversation stops, post an estimated LLM cost for that event to the numberCruncher Slack channel.

LangSmith Cost Data

LangSmith traces contain token usage and cost estimates per run. Key points:

• LangSmith calculates cost using its own internal pricing table, not your negotiated provider rates (AWS Bedrock pricing varies by region/tier, and your actual invoice will differ).
• Cost data is available via the LangSmith API on RunPublic objects: total_cost, prompt_cost, completion_cost, total_tokens, prompt_tokens, completion_tokens.
• The trace for a conversation can be found by filtering runs by a project name and a tag or metadata field that identifies the conversation. We should tag each LangChain invocation with the conversationId at call time (e.g. config: { metadata: { conversationId } }) so we can query it later. NOTE: most if not all runs should already have this in the metadata.
• LangSmith REST API endpoint: GET /api/v1/runs with filter params.

LangSmith API Query

GET https://api.smith.langchain.com/api/v1/runs?project_name=<project>&filter=<filter>
Authorization: Bearer <LANGSMITH_API_KEY>

Filter by conversation metadata:

filter=and(eq(metadata_key, "conversationId"), eq(metadata_value, "<id>"))

Response includes per-run cost fields. Sum total_cost across all runs for the conversation.

What to Display

The card should note clearly that the figure is an estimate based on LangSmith's pricing table:

Estimated LLM cost: ~$1.47 (LangSmith estimate — actual Bedrock charges may differ)

Also display:

• Total tokens (prompt + completion) per model used
• Breakdown by agent if feasible (each agent's runs are tagged with agent name)
• Number of LLM calls

Implementation Plan

1. Tag LangChain calls with conversationId

In getModelChat.ts or wherever ChatModel.invoke() is called, pass LangSmith-compatible metadata:

await llm.invoke(messages, {
  metadata: { conversationId: this.conversation._id.toString() }
})

This allows filtering LangSmith runs by conversation after the fact. NOTE from Jen: we already have this in most getTraceMetadata methods in the agent type. Take this into account when implementing.

2. New method on numberCruncher agent: fetchConversationCost(conversationId)

• Calls LangSmith API to fetch all runs tagged with the conversationId
• Sums total_cost, total_tokens, prompt_tokens, completion_tokens
• Returns a ConversationCostData object

3. Hook into onConversationEvent('conversationStopped')

Similar to vibesAnalyst — numberCruncher registers for all public topic conversation events and posts a cost card when one stops.

async onConversationEvent(evt) {
  if (evt.type !== 'conversationStopped') return []
  const costData = await fetchConversationCost(evt.conversationId)
  if (!costData) return []
  return [{
    visible: true,
    message: `Cost summary for conversation ${evt.conversationId}`,
    messageType: 'text',
    responseKind: 'conversationCostSummary',
    renderData: costData,
    channels: this.conversation.channels
  }]
}

4. New capabilities

numberCruncher will need read: [{ type: 'allPublicTopics' }] (same as vibesAnalyst) so the agentDispatcher routes conversationStopped events to it.

5. New Slack Block Kit card: conversationCostSummary

Show:

• Header: "Cost Summary: [Event Name]"
• Estimated total cost (with caveat)
• Token breakdown table (model → prompt tokens / completion tokens / cost)
• Footer: checked at timestamp

6. Optional: ConversationCost model

For persistence and future reporting:

{
  conversationId: ObjectId
  estimatedCostUSD: number
  tokenBreakdown: { model: string, promptTokens: number, completionTokens: number, estimatedCostUSD: number }[]
  source: 'langsmith'
  capturedAt: Date
}

Do NOT store this on ConversationAnalyticsModel — that model is semantically for engagement metrics (attendance, dwell time) sourced from Matomo. Cost is a separate concern.

Known Limitations

• LangSmith cost estimates use its own pricing table, not AWS Bedrock's actual charges. The discrepancy can be significant for Bedrock (different regional pricing, potential enterprise discounts).
• If LANGSMITH_TRACING_V2 is not enabled, no trace data will exist.
• There is a small delay between conversation stop and LangSmith archiving all runs — consider a short delay or retry before fetching (similar to how vibesAnalyst retries Matomo).
• Runs that fail or are retried may inflate token counts.