dagworks

Dagworks is a company dedicated to enabling developers and data scientists to build Reliable AI. At its core, Dagworks offers two powerful, open-source Python frameworks: Hamilton and Burr. These tools are designed to standardize the development process of data, ML, LLM, and agentic workflows, significantly improving productivity, maintainability, and collaboration.

The platform's mission is to embed observability and introspection as first-class citizens in AI systems. By providing a standardized way to write Python code for complex pipelines and applications, Dagworks ensures that systems are easier to debug, monitor, and scale. This approach reduces the total cost of ownership and accelerates the time-to-value for AI projects.

How to use dagworks

Using the Dagworks ecosystem involves leveraging its two core components, Hamilton and Burr, which can be used independently or together.

1. For Data & ML Pipelines (Hamilton):

• Installation: Start by installing the open-source Hamilton library in your Python environment: pip install sf-hamilton.
• Define Functions: Break down your data pipeline logic into small, pure Python functions. Each function represents a single transformation or step (a node in a Directed Acyclic Graph - DAG).
• Execute the Pipeline: Use the Hamilton driver to execute your pipeline by specifying the final outputs you need. Hamilton automatically determines the execution path (the DAG), manages data flow between functions, and computes the results.
• Integrate Observability: With a single line of code, you can integrate the Hosted Hamilton UI to get full data lineage, a visual representation of your pipeline, a data catalog, and performance metrics.

2. For RAG & Agentic Applications (Burr):

• Installation: Install the Burr library: pip install burr.
• Define States and Actions: Structure your application as a state machine. Define actions (Python functions) that transition the application between different states.
• Run the Application: Use the Burr runtime to execute your state machine. Burr manages the state, tracks the execution history, and allows for easy debugging.
• Utilize Burr Cloud: For production environments, you can use Burr Cloud (or self-host) for hosted execution, state persistence, and advanced observability, allowing you to trace and debug complex agent interactions in real-time.

Core Features of dagworks

• Hamilton (for Pipelines): A lightweight Python framework that represents pipelines as a DAG of functions. It promotes modular, reusable, and unit-testable code. It offers automatic data lineage, provenance tracking, and versioning.
• Burr (for Agents): A framework for building stateful, agentic applications. It standardizes state management, making complex RAG and multi-agent systems easier to build, debug, and observe.
• Integrated Observability: Both frameworks are designed for one-line integration with observability tools. The hosted UIs provide deep insights into code execution, data flow, and application state.
• Data Catalog & Lineage: The Hamilton UI automatically generates a data catalog from your code and provides interactive lineage graphs to understand data dependencies.
• Flexibility and Integration: The tools are lightweight and designed to integrate seamlessly with existing MLOps stacks like MLFlow, Sentry, Docker, and Pandera.
• Open-Source Core: The fundamental frameworks, Hamilton and Burr, are fully open-source, fostering community collaboration and transparency.

Use Cases for dagworks

Dagworks is versatile and trusted by companies in various sectors, from Fintech to consulting. A notable example is Kora Money, a fintech company specializing in risk underwriting.

Kora faced challenges with data lineage for compliance and standardizing their MLOps processes. They adopted both Hamilton and Burr to structure their underwriting platform. Hamilton was used to define data transformation and feature engineering pipelines, breaking them into manageable nodes. Burr was used to orchestrate higher-level workflows, linking multiple Hamilton pipelines with specific business logic. This DAG-based approach simplified data lineage and improved workflow transparency. As a result, Kora successfully migrated a legacy pipeline in just two months, significantly enhancing productivity, streamlining compliance checks, and improving team collaboration.

Advantages of dagworks

The primary advantage of Dagworks is its focus on creating **Reliable AI**. This is achieved through:

• Increased Productivity: Teams can iterate on pipelines and applications up to 4x faster.
• Reduced TCO: Standardized, modular code is easier to maintain, test, and debug.
• Enhanced Collaboration: A common framework ensures that code is understandable and reusable across teams.
• Built-in Governance: Automatic lineage and observability simplify compliance and auditing.
• Future-Proofing: The composable nature of the frameworks lays the foundation for more complex and robust AI systems.

Pricing and Plans

Dagworks operates on a freemium model:

• Open Source: The core Hamilton and Burr Python frameworks are free to use.
• Hosted Hamilton UI: This is a paid service that provides advanced observability, cataloging, and lineage visualization for Hamilton pipelines. It offers a 14-day free trial at the Team level.
• Burr Cloud: Pricing for the hosted Burr service for agentic applications is announced as 'Coming Soon'.

This model allows individual developers and small teams to get started for free, with paid options available for enterprises requiring advanced features, support, and hosting.