/
LLM Evals
Power evals that improve quality reliably
Find headroom and ensure optimizations holistically improve user outcomes in your LLM apps.
Use your favorite LLM eval platform
Pi's scoring model works in most leading eval platforms, and is platform agnostic.
Scored.
Drop in replacement
Simply adjust the scoring type to use Pi, instead of LLM as a judge. All your Evals should still work, and only get more precise.
Consistent accurate from the start
Evaluations are consistent across similar inputs, and are accurate from the start. No system prompt or rubric necessary.
5x cheaper than LLM as a Judge
Evaluate more inputs for the same cost, avoiding sending a system prompt/rubric each time.
Designed for explainability
Evaluate with scores instead of yes/no judgments to tell how much better a response is, and why.
Minimal overhead
Score 20+ dimensions in less than 200ms. Receive scores instantly instead of waiting for tokens to generate.
Tune with user data
Improve your evals beyond prompts. Calibrate against user data to help capture your user's notion of quality.
Start with our official Promptfoo Integration
Promptfoo is free, open source, and has a large community constantly improving it.
Pi Assertion Docs Easily add your Pi assertions to your Promptfoo YAML config.
defaultTest:
assert:
- type: pi
value: Rate the accuracy of the translation of {{text}} from English to {{language}}
metric: accuracy
- type: pi
value: Rate the fluency of the translation of {{text}} from English to {{language}}
metric: fluency
- type: pi
value: Rate the grammar of the translation of {{text}} from English to {{language}}
metric: grammar
Generate missing Pi Assertions directly from your existing config.
promptfoo generate assertions -w
Evaluate and receive quick, consistent scores, for a fraction of the price.
Ready to run LLM evals with Pi? Grab a free API key to get started. Don't hesitate to reach out for enterprise needs.
Explore Pi Enterprise and interact with a dedicated quality engineer to help you integrate.
Explore Enterprise