The Eval Harness Every AI Feature Needs: Promptfoo + Langfuse

You've shipped an AI feature. Your LLM is answering customer queries, generating summaries, or routing support tickets. Everything looks good in staging. Then production hits and you realize: you have no idea if it's actually working.

A customer complains the AI "sounds weird now." Did you break something in the last deployment? Did the model provider update their API? Is prompt drift happening across different user segments? Without an evaluation harness, you're flying blind.

This isn't a theoretical problem. We've built 49 production AI agent specialties at TechNova, and learned this lesson the hard way: every AI feature needs systematic evaluation from day one. Not "eventually." Not "when we have time." From the first prompt you write.

Here's the setup that works.

Why You Need Both Promptfoo and Langfuse

Most teams pick one tool and call it done. That's a mistake. These tools solve different problems:

Promptfoo is your regression test suite. It runs locally or in CI/CD, compares outputs against expected results, and fails your build when quality drops. Think of it like Jest or Pytest, but for LLM outputs.

Langfuse is your production observability layer. It traces every LLM call in production, logs token usage, captures user feedback, and helps you spot drift over time. Think APM tools like DataDog or Sentry, but for AI.

You need both. Promptfoo catches problems before deployment. Langfuse catches problems after deployment. Together, they form a complete eval harness.

Setting Up Promptfoo: The Pre-Deployment Gate

Install Promptfoo in your project:

npm install -g promptfoo

Create a promptfooconfig.yaml in your repo:

providers:
  - openai:gpt-4o-mini
  - openai:gpt-4o

prompts:
  - 'You are a customer support agent for {{company_name}}. Answer this question professionally: {{question}}'

tests:
  - vars:
      company_name: TechNova
      question: How do I reset my password?
    assert:
      - type: contains
        value: email
      - type: not-contains
        value: "I don't know"
      - type: llm-rubric
        value: Response is professional and actionable
  
  - vars:
      company_name: TechNova
      question: What's your refund policy?
    assert:
      - type: llm-rubric
        value: Mentions 30-day policy and includes contact information

Run it:

promptfoo eval

You get a table showing pass/fail for each test case across different models. Add this to your CI pipeline. If evals fail, the deploy fails. Simple.

What to Test

Don't try to test everything. Focus on:

• Critical paths: The 3-5 most common user queries or tasks
• Edge cases that broke before: That time the AI hallucinated pricing information
• Tone and brand voice: Especially important if you're in healthcare, legal, or finance
• Refusal patterns: Ensure the AI properly declines inappropriate requests
• Multi-turn conversations: If your feature maintains context across messages

For one of our CRM implementations, we run 47 test cases on every PR. Takes 3 minutes. Catches regressions weekly.

Setting Up Langfuse: Production Observability

Create a Langfuse account (they have a generous free tier). Grab your API keys.

If you're using OpenAI via LangChain (Python):

from langfuse.callback import CallbackHandler
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain

langfuse_handler = CallbackHandler(
    public_key="your-public-key",
    secret_key="your-secret-key"
)

llm = ChatOpenAI()
chain = LLMChain(llm=llm, prompt=your_prompt)

response = chain.run(
    input="customer query",
    callbacks=[langfuse_handler]
)

For TypeScript/Node (our preference for most web services):

import { Langfuse } from 'langfuse'

const langfuse = new Langfuse({
  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
  secretKey: process.env.LANGFUSE_SECRET_KEY,
})

const trace = langfuse.trace({ name: 'customer-support-query' })

const generation = trace.generation({
  name: 'openai-completion',
  model: 'gpt-4o-mini',
  input: messages,
})

const response = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: messages,
})

generation.end({ output: response })
langfuse.flushAsync()

Now every production LLM call is logged. You can see:

• Latency distribution (p50, p95, p99)
• Token usage and costs per endpoint
• Which prompts are actually being used
• User feedback when you instrument it

The Feedback Loop That Actually Works

Here's where it gets powerful. In your UI, add thumbs up/down buttons. Send that feedback to Langfuse:

langfuse.score({
  traceId: trace.id,
  name: 'user-feedback',
  value: userClickedThumbsUp ? 1 : 0,
})

Now you can:

1. Filter Langfuse traces by score
2. Export the low-scoring examples
3. Add them to your Promptfoo test suite
4. Fix the prompt
5. Re-run evals
6. Deploy

This closed loop — production feedback → test cases → prompt improvements → production — is how you actually improve AI features over time. Not by guessing, not by A/B testing in production, but by systematic iteration.

The Tradeoffs Nobody Tells You

Cost: Running evals costs money. We spend ~$50/month on eval runs across all projects. Worth it to catch a breaking change before customers do.

Latency: Langfuse adds ~10-20ms per request. Negligible for most use cases. If you're building a real-time feature, batch the traces.

Maintenance: Your eval suite needs updating as your product evolves. Budget 2-3 hours per sprint to keep it relevant.

False positives: LLM-based assertions (the llm-rubric type in Promptfoo) occasionally flag good outputs as bad. Review failures manually until you trust the patterns.

Start Small, But Start Now

You don't need 100 test cases on day one. Start with:

• 5 Promptfoo tests covering your happy path
• Langfuse integrated on one endpoint
• One weekly review of production traces

That's it. That's infinitely better than shipping AI features with zero evaluation.

We run this setup across every AI agent specialty we ship. It's not optional infrastructure — it's the foundation that makes AI products reliable.

If you're building AI features and don't have an eval harness yet, this is your sign. Set up Promptfoo this week. Add Langfuse next week. Your future self will thank you when you're debugging a production issue and actually have data to work with.