withpi.ai

withpi.ai, developed by Pi Labs, is an advanced platform designed for developers to build sophisticated evaluation and search systems that evolve with their data. It provides a suite of tools to create tunable ranking and scoring systems, integrating both natural language and code-based criteria into any AI application. The platform's core mission is to turn subjective evaluations into precise, user-calibrated, and cost-effective signals that can be used throughout the entire AI stack.

Unlike traditional methods that rely on expensive and slow large language models (LLMs) as judges, withpi.ai offers a specialized foundation model, Pi Scorer, which is optimized for speed and accuracy in evaluation tasks. This allows developers to measure multiple custom dimensions of their AI's performance quickly and affordably, ensuring continuous alignment with user expectations and business goals.

How to use withpi.ai

Integrating withpi.ai into your workflow is straightforward and can be done with just a few lines of code. The process typically involves:

1. Sign Up & Get API Key: Register on the withpi.ai website to get your API credentials.
2. Install the Client: Install the official Python library for easy integration.
3. Define Scoring Criteria: Create a `scoring_spec` where you define the questions and criteria for evaluation. This can be based on product requirements, user feedback, or any other relevant metric. For example: `[{"question": "Is there a strong call to action?"}]`.
4. Score AI Outputs: Use the `pi.scoring_system.score()` method, passing the LLM input, the LLM output, and your defined scoring specification.
5. Integrate Scores: The returned scores are deterministic and can be used anywhere in your stack: for offline evaluations, online observability, improving training data quality, optimizing models, or controlling agent decision flows. The platform is framework-agnostic and can be easily plugged into tools like Google Spreadsheets, Promptfoo, and CrewAI.

Core Features of withpi.ai

• Pi Scorer: A highly optimized foundation model designed specifically for scoring. It's faster and more accurate than general-purpose LLMs for evaluation tasks.
• Pi Ranking: Provides customizable cross-encoders to build powerful ranking systems for search and recommendation.
• Pi Embedding: Offers customizable embeddings tailored for high-performance retrieval applications.
• User-Calibrated Systems: Continuously improve and align your scoring system by calibrating it with your own labels, user preferences, and expert feedback.
• Comprehensive Metrics: The system can evaluate both 'soft' measures (like writing style, tone, naturalness) and 'hard' measures (like code correctness, factual accuracy) simultaneously.
• Pi Copilot: An AI assistant that helps developers and product managers define, refine, and tune their scoring metrics.
• Framework Agnostic: Seamlessly integrates into any part of the AI development lifecycle, from offline evaluation to real-time production monitoring.

Use Cases for withpi.ai

withpi.ai is versatile and can be applied to a wide range of scenarios:

• LLM Evals: Consistently and objectively evaluate the quality of LLM responses against a set of predefined principles.
• RAG Optimization: Tune your Retrieval-Augmented Generation systems by scoring the relevance and quality of retrieved documents to improve final outputs.
• AI Agent Control Flow: Use scores as decision-making nodes within AI agents to determine the next best action, such as re-trying a task or proceeding with a generated plan.
• Content Quality Assurance: Automatically score generated content like blog posts, marketing copy, or meeting summaries for quality, brand voice, and factual accuracy.
• Specialized Evaluators: Build custom scorers for niche domains, such as a SQL Query Evaluator, a Log Security Analyzer, a Startup Resume Analyzer, or even a Chess Move Scorer.

Advantages of withpi.ai

The primary advantages of using withpi.ai stem from its specialized design:

• Speed and Performance: Capable of scoring over 20 custom dimensions in less than 100 milliseconds, enabling real-time feedback loops.
• Cost-Effectiveness: Up to 5 times cheaper than using large LLMs like GPT-4 for evaluation, allowing for more comprehensive and frequent testing without high costs.
• Superior Accuracy: The Pi Scorer model is trained to understand principles, not just mimic content, leading to more accurate and reliable scores than general models.
• Alignment with Human Judgment: The platform is built around a virtuous feedback loop, allowing systems to be continuously refined to match team expertise and actual user behavior.
• Holistic Evaluation: It uniquely combines qualitative and quantitative measures to provide a complete picture of an AI's performance.

Pricing and Plans

withpi.ai offers a simple and accessible pricing model designed to let developers start easily and scale as needed.

• Free Tier: Includes $10 in free credits, which is enough to cover approximately 25 million tokens. This is ideal for testing, development, and small-scale projects.
• Pay-as-you-go: After using the free credits, the cost is a flat rate of $0.40 per million tokens. This plan allows for unlimited use and scales directly with your consumption.

The company notes that pricing is still being refined and they are open to user feedback.