Building Resilient LLM Pipelines in Production

When you ship a large language model to production, you're not deploying a single endpoint — you're deploying a dependency stack that includes the model provider, a tokenizer, rate limiters, and your own application logic. Any link in that chain can break, and when it does, the failure is rarely graceful. Users see 500 errors. Latency spikes. Costs spiral.

After running LLM-powered features in production for two years across three teams, I've learned that resilience is not a feature you add later — it's an architecture you build from day one. This guide breaks down the four patterns that matter most.

1. Semantic Caching: The 60% Win

Before you worry about model failures, eliminate unnecessary calls entirely. Semantic caching stores embeddings of previous queries and returns cached responses when a new query is semantically similar — not just lexically identical.

In production, we found that 40–60% of user queries are semantically near-duplicates of something already cached. That's more than half your inference budget, recovered.

Here's a minimal implementation using pgvector and a similarity threshold:

pythonCopy
# semantic_cache.py — pgvector + sentence-transformers
import asyncio
from sentence_transformers import SentenceTransformer
from asyncpg import create_pool

SIMILARITY_THRESHOLD = 0.92
encoder = SentenceTransformer("all-MiniLM-L6-v2")

class SemanticCache:
    def __init__(self, pool):
        self.pool = pool

    async def get(self, query: str) -> str | None:
        embedding = encoder.encode(query).tolist()
        # Cosine similarity via pgvector's <#> operator (KNN)
        row = await self.pool.fetchrow(
            """
            SELECT response FROM cache
            WHERE embedding <#> $1 < (1 - $2)
            ORDER BY embedding <#> $1
            LIMIT 1
            """,
            embedding, SIMILARITY_THRESHOLD
        )
        return row["response"] if row else None

    async def set(self, query: str, response: str, ttl: int = 3600):
        embedding = encoder.encode(query).tolist()
        await self.pool.execute(
            """
            INSERT INTO cache (query, response, embedding, expires_at)
            VALUES ($1, $2, $3, NOW() + INTERVAL '%s seconds')
            """ % ttl,
            query, response, embedding
        )

The threshold is critical. Set it too low and you'll return wrong answers to similar-but-different questions. Set it too high and your hit rate drops to zero. 0.90–0.93 is the sweet spot for most domains — tune it against your own query logs.

2. Circuit Breakers: Stop Hitting the Button

When your LLM provider starts failing, the worst thing you can do is keep sending requests. Each failed call adds latency, consumes retries, and can trigger rate-limit penalties. A circuit breaker pattern trips after N consecutive failures and stops all outbound calls for a cooldown period.

Here's the state machine in plain terms:

State	Behavior	Transition
Closed	Requests flow normally	5 failures → Open
Open	All requests fail fast	30s timeout → Half-Open
Half-Open	1 probe request allowed	Success → Closed / Fail → Open

Using Python's pybreaker library, the implementation is straightforward:

pythonCopy
import pybreaker
from tenacity import retry, stop_after_attempt, wait_exponential

# Trip after 5 failures, reset after 30s cooldown
llm_breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=30,
    exclude=[ValueError],  # Don't trip on bad input
)

llm_breaker.add_listener(pybreaker.CircuitBreakerListener())

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=8))
@llm_breaker
async def call_llm(prompt: str, model: str = "gpt-4o") -> str:
    try:
        resp = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            timeout=30,
        )
        return resp.choices[0].message.content
    except Exception as e:
        logger.warning(f"LLM call failed: {e}")
        raise  # Let breaker track it

The key insight: a circuit breaker protects your system, but it also protects your provider. Hammering a struggling API endpoint is how you turn a 5-minute outage into a 30-minute ban. — from "Distributed Systems Patterns for ML" (2025)

3. Fallback Chains: Never Return Nothing

When your primary model fails, you need a backup — and that backup needs its own backup. A fallback chain defines an ordered list of model providers that activate sequentially as each one fails.

The chain should degrade gracefully: from your premium model down to a cheaper, faster model, and finally to a static response template.

pythonCopy
from dataclasses import dataclass
from typing import Callable, Awaitable

@dataclass
class FallbackStep:
    name: str
    call: Callable[[str], Awaitable[str]]
    max_latency_ms: int = 8000

FALLBACK_CHAIN = [
    FallbackStep("gpt-4o",       call_gpt4o,       max_latency_ms=8000),
    FallbackStep("claude-3.5",   call_claude,      max_latency_ms=6000),
    FallbackStep("llama-3-70b",  call_self_hosted, max_latency_ms=12000),
    FallbackStep("static",       call_static_template),  # last resort
]

async def resilient_generate(prompt: str) -> str:
    errors = []
    for step in FALLBACK_CHAIN:
        try:
            return await asyncio.wait_for(
                step.call(prompt), timeout=step.max_latency_ms / 1000
            )
        except Exception as e:
            errors.append(f"{step.name}: {e}")
            logger.warning(f"Fallback {step.name} failed, trying next")
            continue

    # If we get here, even static failed — log aggressively
    logger.error(f"All fallbacks exhausted: {errors}")
    return "I'm unable to process that right now. Please try again."

4. Observability: You Can't Fix What You Can't See

Resilience without observability is just guesswork. Every LLM call should emit structured logs with these fields at minimum:

• model and provider used
• latency broken into: time-to-first-token, generation time, total
• token counts (prompt, completion, total) for cost tracking
• cache hit/miss and similarity score if applicable
• circuit breaker state before and after the call
• fallback depth — which step in the chain actually served the request

Here's a minimal OpenTelemetry setup:

pythonCopy
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

tracer = trace.get_tracer("llm.gateway")

async def traced_call(prompt: str, model: str):
    with tracer.start_as_current_span("llm.inference") as span:
        span.set_attributes({
            "llm.model": model,
            "llm.provider": "openai",
            "llm.prompt_tokens": count_tokens(prompt),
            "cache.hit": False,
        })
        result = await call_llm(prompt, model)
        span.set_attribute("llm.completion_tokens", result.usage.completion_tokens)
        span.set_attribute("llm.latency_ms", result.latency_ms)
        return result

Putting It All Together

The full request flow looks like this:

1. Request arrives → generate embedding for semantic cache lookup
2. Cache hit? → return immediately (p50: ~20ms)
3. Cache miss → check circuit breaker state
4. Breaker closed → call primary model with retry + timeout
5. Primary fails → walk the fallback chain
6. Response ready → write to cache, emit telemetry, return

This architecture isn't theoretical. We've been running it in production for 14 months across a product serving ~2M daily LLM queries. The results:

Metric	Before	After	Δ
p95 latency	4,200ms	680ms	−84%
Error rate	2.1%	0.04%	−98%
Monthly inference cost	$147k	$61k	−59%
Cache hit rate	0%	52%	—

None of these patterns are exotic. Circuit breakers and caching are decades old. What's changed is that LLM inference is expensive enough and unreliable enough that these patterns are no longer optional — they're the baseline.

If you're shipping an LLM-powered feature today and you don't have all four of these in place, you have a production incident waiting to happen. The good news is that each one is a small, independent change. Start with semantic caching — it's the highest ROI and the easiest to implement.

Got questions about adapting this to your stack? Drop a comment or find me on X. I'm happy to share the full reference implementation.