Building a Cost Tracker for Multi-Model AI Workflows

A multi-model AI workflow with no cost tracking is a money leak with a progress bar. You watch tasks complete, users seem happy, and then the invoice arrives and you realize you've been running Claude Opus on tasks that GPT-4o-mini could handle, embedding every request fresh instead of caching, and firing synchronous calls where batch would cost 50% less. The damage is invisible until it isn't.

Why Multi-Model Workflows Break Simple Cost Tracking

A single agent task might touch four models. Claude 3.5 Sonnet reasons through a customer complaint. Gemini 1.5 Flash reads an attached image. text-embedding-3-small turns the resolved ticket into a vector. A voice synthesis model reads the response back to the user. Each call has a different pricing unit: input tokens, output tokens, image tiles, audio seconds, API call flat fees.

The naive approach is to log total spend per day. That tells you nothing actionable. You don't know which task type is expensive, which model is the culprit, or whether the cost is justified by the outcome. When you're running something like OneShot where agents pay for tools (voice, email, SMS, research) with USDC per call, the cost picture gets more complex still: you're mixing LLM token costs with per-action tool costs, and they need to be tracked in the same ledger.

The goal is a cost record that answers: "For task type X, what did we spend on each model and each tool, and did the task succeed?"

The Attribution Schema

Every LLM call and every tool invocation needs four fields attached before it fires:

• task_id: a UUID generated when the top-level task starts
• task_type: a human-readable label ("complaint_resolution", "product_search", "lead_research")
• tool_name: the specific tool or agent step ("reasoning", "image_parse", "embed", "voice_call")
• model: the exact model identifier, including version ("claude-3-5-sonnet-20241022", "gemini-1.5-flash-002")

You also need to capture cost at call time, not reconstruct it later. Reconstructing from logs is fragile because model pricing changes and you'll lose the price that was actually charged. Capture input tokens, output tokens, and the per-token rates at the moment of the call.

Here's a TypeScript wrapper that handles this for any OpenAI-compatible API:

import OpenAI from "openai";

interface CostRecord {
  task_id: string;
  task_type: string;
  tool_name: string;
  model: string;
  input_tokens: number;
  output_tokens: number;
  input_cost_usd: number;
  output_cost_usd: number;
  total_cost_usd: number;
  success: boolean | null; // filled in later
  timestamp: number;
}

// Prices in USD per 1M tokens — update from https://openrouter.ai/models
const MODEL_PRICES: Record = {
  "claude-3-5-sonnet-20241022": { input: 3.0, output: 15.0 },
  "gpt-4o-mini": { input: 0.15, output: 0.6 },
  "text-embedding-3-small": { input: 0.02, output: 0 },
  "gemini-1.5-flash-002": { input: 0.075, output: 0.3 },
};

async function trackedCompletion(
  client: OpenAI,
  params: OpenAI.ChatCompletionCreateParamsNonStreaming,
  meta: { task_id: string; task_type: string; tool_name: string },
  emit: (record: CostRecord) => void
): Promise {
  const result = await client.chat.completions.create(params);
  const usage = result.usage!;
  const prices = MODEL_PRICES[params.model] ?? { input: 0, output: 0 };

  const record: CostRecord = {
    ...meta,
    model: params.model,
    input_tokens: usage.prompt_tokens,
    output_tokens: usage.completion_tokens,
    input_cost_usd: (usage.prompt_tokens / 1_000_000) * prices.input,
    output_cost_usd: (usage.completion_tokens / 1_000_000) * prices.output,
    total_cost_usd:
      (usage.prompt_tokens / 1_000_000) * prices.input +
      (usage.completion_tokens / 1_000_000) * prices.output,
    success: null,
    timestamp: Date.now(),
  };

  emit(record);
  return result;
}

The emit function can write to Postgres, a time-series DB, or a simple JSONL file depending on your scale. The key thing is that cost is computed immediately from usage data and stored with full attribution. Don't defer it.

Tracking Tool Costs Alongside LLM Costs

When agents use external tools, those costs are separate from token costs and need the same attribution. The OneShot pricing model charges per tool invocation: a voice call costs differently from an email send or a web research request. These costs are in USDC, not USD, but for a unified ledger you convert at call time.

Here's a pattern that wraps a OneShot tool call and emits a cost record in the same format:

import { OneShotClient } from "@oneshot-agent/sdk";

interface ToolCostRecord {
  task_id: string;
  task_type: string;
  tool_name: string;
  model: string; // "oneshot/voice" | "oneshot/email" | etc.
  input_tokens: number;
  output_tokens: number;
  input_cost_usd: number;
  output_cost_usd: number;
  total_cost_usd: number;
  success: boolean | null;
  timestamp: number;
}

// USDC is pegged 1:1 to USD for accounting purposes
async function trackedToolCall(
  client: OneShotClient,
  tool: "voice" | "email" | "sms" | "research",
  payload: Record,
  meta: { task_id: string; task_type: string },
  emit: (record: ToolCostRecord) => void
): Promise {
  const result = await client.tools[tool].run(payload);

  // OneShot returns cost_usdc in the response envelope
  const costUSDC = (result as any).cost_usdc ?? 0;

  const record: ToolCostRecord = {
    ...meta,
    tool_name: tool,
    model: `oneshot/${tool}`,
    input_tokens: 0,
    output_tokens: 0,
    input_cost_usd: costUSDC, // 1 USDC = 1 USD
    output_cost_usd: 0,
    total_cost_usd: costUSDC,
    success: (result as any).success ?? null,
    timestamp: Date.now(),
  };

  emit(record);
  return result;
}

Now every cost event, whether it's a Claude reasoning step or a OneShot voice call, lands in the same table with the same schema. You can sum across them cleanly.

What the Data Actually Shows

Once you have a few thousand records, the breakdown is usually surprising. Based on estimates from typical agentic workflows (your numbers will vary):

• Reasoning steps on frontier models (Claude Sonnet, GPT-4o) account for roughly 60-70% of total LLM spend despite being fewer than 20% of total calls by count.
• Embedding calls are extremely cheap per call but can dominate by volume if you're not caching. Re-embedding the same document on every request at text-embedding-3-small rates ($0.02/1M tokens) costs almost nothing per call but adds up fast at scale.
• Tool costs from voice calls dwarf most LLM costs on a per-task basis. A 3-minute voice call through a provider costs roughly $0.15-0.30 in telephony alone, which is more than most reasoning steps.
• Output tokens cost 5-10x more than input tokens on frontier models. A workflow that generates long responses is paying a heavy premium. Claude Sonnet charges $3/1M input but $15/1M output.

These ratios tell you where to look first. If reasoning is 65% of your spend, you optimize model routing. If voice is 40%, you optimize resolution rate (fewer calls to achieve the same outcome). If embeddings are 15%, you add a cache layer.

The Four Optimization Levers

Model Routing

The biggest wins come from routing easy subtasks to cheaper models. A task classification step that decides which tool to invoke doesn't need Claude Sonnet. GPT-4o-mini at $0.15/1M input tokens is 20x cheaper for that call, and on simple routing decisions the quality difference is negligible.

The implementation pattern: score each task type by complexity (based on historical output token counts and error rates), set a complexity threshold, and route below-threshold tasks to a cheaper model. Track cost per task type before and after. A/B test the routing rule. Don't assume cheaper means worse without measuring.

You can browse current pricing across models at OpenRouter's model pricing page to find the right tier for each subtask. The gap between frontier and mid-tier models is large enough that even a 10% routing improvement on volume tasks moves the total bill significantly.

Caching

Semantic caching on embeddings is the easiest win. If two requests embed the same document (or very similar ones), return the cached vector. A cosine similarity threshold of 0.98 on the embedding of the input text catches near-duplicate requests effectively.

For LLM calls, prompt caching on Claude and GPT-4o can reduce input token costs by 50-90% on repeated system prompts. If your system prompt is 2,000 tokens and you make 10,000 calls per day, that's 20M tokens of input. At Claude Sonnet rates, that's $60/day just in system prompt tokens. With prompt caching, it drops to $6-12/day.

Prompt Compression

Long context is expensive. A research workflow that dumps 50,000 tokens of retrieved documents into a single call at Claude Sonnet rates costs $0.15 per call in input alone. If you're making 1,000 such calls per day, that's $150/day from one step.

Aggressive summarization before the main reasoning call, even using a cheap model, often recovers quality while cutting context by 70-80%. The summarization step might cost $0.002 (using GPT-4o-mini on 50K tokens), and the compressed context drops the main call cost from $0.15 to $0.03. Net saving: $0.118 per call, $118/day on 1,000 calls.

Batch Processing

OpenAI's Batch API charges 50% of the standard rate for asynchronous jobs. If any part of your workflow is non-real-time (nightly enrichment, bulk embeddings, background research), batch it. The tradeoff is latency: batch jobs can take up to 24 hours. For anything user-facing, batch doesn't apply. For backend processing, it's a straightforward cost cut.

Building Dashboards That Actually Help

Cost per token is the wrong primary metric. It tells you about efficiency at the model level, not at the business level. The metric that matters is cost per successful outcome.

For a complaint resolution workflow, that's cost per resolved complaint. For a checkout recovery agent, that's cost per recovered sale. For a lead research pipeline, it's cost per qualified lead delivered. These metrics connect spending to value in a way that cost per token never does.

To compute cost per successful outcome, you need to close the loop on the success field in your cost records. When a task completes, update all cost records for that task_id with the outcome:

async function closeTask(
  db: Database,
  task_id: string,
  success: boolean
): Promise {
  await db.query(
    `UPDATE cost_records SET success = $1 WHERE task_id = $2`,
    [success, task_id]
  );
}

async function getCostPerOutcome(
  db: Database,
  task_type: string,
  since: Date
): Promise<{ cost_per_success: number; success_rate: number; total_spend: number }> {
  const result = await db.query(
    `SELECT
       SUM(total_cost_usd) as total_spend,
       SUM(CASE WHEN success = true THEN total_cost_usd ELSE 0 END) as success_spend,
       COUNT(DISTINCT CASE WHEN success = true THEN task_id END) as successes,
       COUNT(DISTINCT task_id) as total_tasks
     FROM cost_records
     WHERE task_type = $1 AND timestamp >= $2`,
    [task_type, since.getTime()]
  );

  const row = result.rows[0];
  const success_rate = row.successes / row.total_tasks;
  const cost_per_success = row.total_spend / row.successes;

  return { cost_per_success, success_rate, total_spend: row.total_spend };
}

Now you can answer questions like: "Our complaint resolution cost per success went from $0.85 to $1.20 this week. Why?" Then you drill into the model breakdown for that task type during that period and find that one reasoning step started using a more expensive model, or that average context length increased because users started attaching screenshots.

For teams using LangSmith, you can pipe these cost records alongside LangSmith traces to correlate cost with trace-level debugging. LangSmith tracks token usage natively but doesn't handle multi-model attribution or tool costs in the same ledger, so the two systems are complementary.

A Concrete Scenario

Say you're running a workflow that processes 500 customer complaints per day. Your current cost breakdown (estimates):

• Complaint classification: Claude Sonnet, avg 800 input / 50 output tokens. Cost: $0.0026/call. Daily: $1.30.
• Resolution reasoning: Claude Sonnet, avg 4,000 input / 400 output tokens. Cost: $0.018/call. Daily: $9.00.
• Response generation: Claude Sonnet, avg 2,000 input / 600 output tokens. Cost: $0.015/call. Daily: $7.50.
• Voice call (if escalated, 30% of cases): OneShot voice, avg $0