Observability Solutions

Observability

Products

Models

Pi Scorer

Foundation models designed for scoring

Pi Ranking Alpha

Customizable cross encoders for ranking

Pi Embedding Alpha

Customizable embeddings for retrieval

Pi User Behaviour Alpha

User engagement prediction models

Solutions

Pi Copilot

AI assistant for tuning your scorer

Spreadsheets

Analyze data with your scorer

LLM Evals

Evaluate LLMs consistently

Observability

Powerful, low overhead monitoring

Resources

Tools

Featured Projects

Example scorers devs and PMs built

Handbooks

Today's best practices

Code Examples

Examples of how to use your scorer.

Company

Blog

Latest updates and research.

Discord

Our official support line

Docs

Contact

Monitor fuzzy, subjective criteria in your logs

Build metrics that capture nuanced, natural language patterns in realtime

Build scoring system API Docs

We have big upcoming plans for observability

Drop into any observability framework

See the "unseeable".

Graduate beyond traditional metrics and start measuring subjective and highly nuanced heuristics that are hard to define with code.

Obtain explainable insights.

Build reliable metrics you can make informed conclusions from. Understand what is driving your metrics, so you can make accurate conclusions from your data.

Low overhead, high power.

Measure 20+ dimensions in less than 100ms. Never worry about adding noticeable overhead to your systems when adding complex metrics - that if implemented without AI, would be computationally prohibitive.

Power your metrics with a scoring system that grows with your app. Tune it with user data, product specs, and logs to improve the accuracy of your analytics over time.

Build scoring system View Docs

Creativity

Major

0.80

Is the content creative?

Grammar

Major

0.95

Is the content free of grammatical errors?

Format

Major

0.40

Is the content formatted according to this JSON structure?

python
from typing import Any

def score(
    response_text: str,
    input_text: str,
    kwargs: dict[str, Any],
) -> dict:

    def evaluate_response(response):
        """
        Check if the JSON object specifies the price of the house.
        
        Args:
            response (str): The LLM's response text
        
        Returns:
            float: 1.0 if the JSON object specifies the price, 0.0 otherwise
        """
        import json
        
        try:
            # Attempt to parse the response as JSON
            house_data = json.loads(response)
            
            # Check if the 'price' key exists in the JSON object
            if 'price' in house_data:
                return 1.0
            return 0.0
        except json.JSONDecodeError:
            # Return 0.0 if the response is not valid JSON
            return 0.0

    final_score = evaluate_response(response_text)
    return {"score": final_score, "explanation": ""}

Usage Patterns

Major

0.40

Does the content mention sources?

Transform data into metrics

Not sure what to measure? Pi figures it out for you. Feed it any or all of your prompts, your PRDs, your user feedback, or just sit down and chat with it and it will help you figure out the best calibrated metrics for your application.

Quick, deterministic scores

Our foundation model, Pi Scorer, scores more accurately than Deepseek and GPT 4.1, but runs at the size and speed of GPT Mini and Gemini Flash. You can score 20+ custom dimensions in less than 100msec; it’s that fast.

Framework agnostic

A single Pi Scorer can be used in every part of your AI stack and existing tools: offline evals, online observability, training data quality, model optimization, agent control flows and more. Plug into any observability stack,

Upgrade your observability with a few lines of code. Grab a free API key to get started. Don't hesitate to reach out for enterprise needs.

Create an API Key Talk to an expert

Explore Pi Enterprise and interact with a dedicated quality engineer to help you integrate.

Explore Enterprise

Home Docs Pricing Support