LLM API Best Practices for Production Apps

LLM API Best Practices for Production Apps

Integrating LLM APIs into production applications requires more than just calling an endpoint. Streaming, caching, error handling, cost control, and prompt versioning determine whether your AI features are reliable and affordable at scale.

Streaming Responses

Always stream LLM responses in user-facing applications. Users perceive streaming as faster even when total latency is the same:

async function streamCompletion(prompt: string) {
  const response = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-api-key": process.env.ANTHROPIC_API_KEY!,
      "anthropic-version": "2023-06-01",
    },
    body: JSON.stringify({
      model: "claude-sonnet-4-20250514",
      max_tokens: 1024,
      stream: true,
      messages: [{ role: "user", content: prompt }],
    }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    const chunk = decoder.decode(value);
    // Parse SSE events and extract text deltas
    yield parseStreamChunk(chunk);
  }
}

Caching Strategies

Cache responses for identical or similar queries to reduce cost and latency:

Exact Match Cache

import { Redis } from "ioredis";
const redis = new Redis();

async function cachedCompletion(prompt: string, ttl = 3600) {
  const cacheKey = llm:\${hashPrompt(prompt)};
  const cached = await redis.get(cacheKey);

  if (cached) {
    return JSON.parse(cached);
  }

  const result = await callLLM(prompt);
  await redis.setex(cacheKey, ttl, JSON.stringify(result));
  return result;
}

Semantic Cache

For queries that are phrased differently but mean the same thing, use embedding similarity:

async function semanticCache(prompt: string, threshold = 0.95) {
  const embedding = await getEmbedding(prompt);
  const similar = await vectorDb.search(embedding, { threshold });

  if (similar.length > 0) {
    return similar[0].cachedResponse;
  }

  const result = await callLLM(prompt);
  await vectorDb.insert({ embedding, prompt, cachedResponse: result });
  return result;
}

Error Handling and Fallbacks

LLM APIs fail. Build resilience with retries and provider fallbacks:

async function resilientCompletion(prompt: string) {
  const providers = [
    { name: "anthropic", fn: () => callAnthropic(prompt) },
    { name: "openai", fn: () => callOpenAI(prompt) },
  ];

  for (const provider of providers) {
    try {
      return await withRetry(provider.fn, {
        maxRetries: 3,
        backoff: "exponential",
        retryOn: [429, 500, 502, 503],
      });
    } catch (err) {
      console.error(\${provider.name} failed:, err);
      continue; // Try next provider
    }
  }

  throw new Error("All LLM providers failed");
}

async function withRetry(
  fn: () => Promise,
  opts: { maxRetries: number; backoff: string; retryOn: number[] }
): Promise {
  for (let i = 0; i <= opts.maxRetries; i++) {
    try {
      return await fn();
    } catch (err: any) {
      if (i === opts.maxRetries || !opts.retryOn.includes(err.status)) throw err;
      await new Promise((r) => setTimeout(r, Math.pow(2, i) * 1000));
    }
  }
  throw new Error("Unreachable");
}

Cost Control

LLM API costs can escalate quickly. Implement guardrails:

Token Budgets

function estimateTokens(text: string): number {
  // Rough estimate: 1 token per 4 characters
  return Math.ceil(text.length / 4);
}

async function budgetedCompletion(prompt: string, maxCostCents: number) {
  const inputTokens = estimateTokens(prompt);
  const estimatedCost = (inputTokens / 1_000_000) * 300; // $3/MTok input

  if (estimatedCost > maxCostCents / 100) {
    throw new Error(Estimated cost $\${estimatedCost.toFixed(4)} exceeds budget);
  }

  return callLLM(prompt);
}

Per-User Rate Limits

async function checkUserBudget(userId: string) {
  const usage = await redis.get(llm_usage:\${userId});
  const dailyTokens = parseInt(usage ?? "0");

  if (dailyTokens > 100_000) {
    throw new Error("Daily token limit exceeded");
  }
}

Prompt Versioning

Treat prompts like code — version them, test them, and deploy them:

// prompts/summarize.ts
export const PROMPTS = {
  summarize: {
    v1: "Summarize the following text in 3 bullet points:",
    v2: "Provide a concise summary of the following text. Use exactly 3 bullet points. Each bullet should be one sentence.",
    current: "v2",
  },
};

function getPrompt(name: string): string {
  const config = PROMPTS[name];
  return config[config.current];
}

Monitoring

Track key metrics for your LLM integration:

Latency — time to first token and total response time.

Token usage — input and output tokens per request.

Cost — daily and monthly spend by feature.

Error rate — failed requests by error type.

Cache hit rate — what percentage of requests are served from cache.

Conclusion

Production LLM integrations need streaming for UX, caching for cost, retries and fallbacks for reliability, token budgets for cost control, and prompt versioning for maintainability. Treat your LLM integration like any other critical dependency — with monitoring, error handling, and capacity planning.