promptfoo

promptfoo is a professional-grade tool designed to help developers and enterprises build secure, reliable, and high-performing AI applications. It serves as a comprehensive framework for evaluating, testing, and improving the quality of prompts and the performance of various Large Language Models (LLMs). Trusted by 27 Fortune 500 companies and a large open-source community, promptfoo provides the necessary tools to ensure AI systems are robust and safe before deployment.

The core philosophy of promptfoo is to enable systematic comparison and evaluation. Users can test different prompts against multiple LLMs simultaneously, analyze the outputs side-by-side, and make data-driven decisions. This is crucial for optimizing performance, reducing costs, and selecting the best model for a specific use case. Furthermore, promptfoo places a strong emphasis on security, offering advanced features like AI-powered red teaming to proactively identify vulnerabilities such as prompt injections, data leaks, and the generation of toxic content.

How to use promptfoo

Using promptfoo is straightforward and designed for developers. The process typically involves the command-line interface (CLI) and a simple YAML configuration file.

1. Installation & Initialization: Get started by running a single command like npx promptfoo@latest init. This command interactively sets up a configuration file (promptfooconfig.yaml) in your project.
2. Configuration: Edit the promptfooconfig.yaml file. Here, you define the prompts you want to test (using variables like {{variable_name}} for dynamic inputs), specify the LLM providers (e.g., OpenAI, Anthropic, Google, or local models via Ollama), and create your test cases.
3. Define Test Cases: In the 'tests' section of the YAML file, you list various inputs (test cases) that your prompts will be tested against. You can also add 'assertions' to automatically check if the model's output meets specific criteria (e.g., does not contain certain phrases, is valid JSON, or passes an LLM-based rubric).
4. Run Evaluation: Execute the command npx promptfoo@latest eval in your terminal. promptfoo will then run all your prompts against all specified providers using every test case.
5. View Results: After the evaluation, run npx promptfoo@latest view to open a web-based UI. This interface presents a clear, side-by-side comparison of all outputs, highlighting which ones passed or failed your assertions, making it easy to analyze results and iterate.

Core Features of promptfoo

• Systematic Evaluation: Compare prompts, models, and model parameters in a structured side-by-side view to find the optimal configuration.
• AI-Powered Red Teaming: Automatically generate and run customized attacks to discover vulnerabilities like prompt injections, data leaks, insecure tool use, and toxic content generation.
• Model Quality Benchmarking: Evaluate and compare the performance, cost, and speed of over 50 LLM providers, including OpenAI, Google, Anthropic, and local models like Llama.
• Automated Assertions and Metrics: Define pass/fail criteria using various assertion types, including JavaScript expressions, Python code, and even LLM-based checks (rubrics) to grade outputs automatically.
• Developer-Friendly Workflow: A powerful CLI with features like live reloads and caching to speed up the development cycle. It's security-first, with no required SDKs or cloud dependencies for the core tool.
• Flexible Deployment: Use the open-source CLI for free, or opt for managed cloud or on-premises enterprise solutions for advanced features, collaboration, and support.

Use Cases for promptfoo

promptfoo is versatile and can be applied in various scenarios:

• Prompt Engineering: Iteratively refine prompts to achieve more accurate, consistent, and desired responses from LLMs.
• Model Selection: Benchmark different models (e.g., GPT-4o vs. Claude 3 Sonnet vs. Llama 3) on your specific data to choose the most cost-effective and performant option.
• Regression Testing: Integrate promptfoo into your CI/CD pipeline to ensure that updates to your prompts or underlying models do not degrade performance or introduce new issues.
• AI Security Audits: Proactively test your AI application for security flaws before they can be exploited in production.
• Quality Assurance for RAG: Evaluate the quality of Retrieval-Augmented Generation (RAG) systems by testing the relevance and accuracy of the generated answers.
• Content Moderation and Safety: Ensure that your AI application adheres to safety guidelines and does not produce harmful, biased, or inappropriate content.

Advantages of promptfoo

The main advantage of promptfoo is its focus on building robust and secure AI. It moves beyond simple prompt testing to a holistic quality and security assurance framework. It's open-source, highly flexible, and battle-tested at an enterprise scale. By running locally without cloud dependencies, it ensures the privacy and security of your data. The tool empowers teams to move quickly and confidently, knowing their AI applications are both effective and safe.

Pricing and Plans

promptfoo operates on a freemium model. The core command-line tool is open-source and completely free to use. For teams and enterprises requiring advanced capabilities, promptfoo offers paid solutions:

• Open-Source (Free): Includes the CLI, all evaluation features, provider integrations, and community support.
• Enterprise: Offers managed cloud or on-premises deployment, advanced red teaming features, collaboration tools, dedicated support, and more. Pricing for the enterprise plan is available upon request by booking a demo.