Metaplane

Metaplane, now part of Datadog, is a comprehensive data observability platform designed to bring software and data observability together. It empowers modern data teams to build trust in their data by proactively identifying and resolving data quality issues across the entire data stack, from source to BI tools. The platform uses advanced machine learning to detect anomalies that traditional, rigid tests might miss, accounting for seasonality and trends in your data. This ensures that alerts are relevant and noise-free, allowing teams to focus on genuine problems.

The core mission of Metaplane is to prevent silent data incidents—subtle errors in data that can go unnoticed for long periods, leading to flawed analysis and poor business decisions. By providing 100% visibility into data pipelines, Metaplane helps teams understand the flow of data, identify critical dependencies, and assess the impact of changes before they are deployed.

How to use Metaplane

Getting started with Metaplane is designed to be quick and straightforward, typically taking less than 15 minutes.

1. Integrate Your Data Stack: Connect Metaplane to your data warehouses (like Snowflake, BigQuery, Redshift), transformation tools (like dbt), and BI platforms (like Looker, Tableau, PowerBI). The integration process is seamless and requires only read-only access to metadata, ensuring your underlying data remains secure.
2. Configure Monitors: Metaplane automatically suggests monitors for your most important tables. You can also manually configure monitors for various data quality metrics, including volume, freshness, schema changes, uniqueness, nullness, and statistical distributions. Custom SQL monitors are also available for specific business logic.
3. ML Model Training: Once connected, Metaplane's machine learning models begin training on your data's metadata profiles. Within as little as three days, the system is ready to send intelligent, context-aware alerts.
4. Receive and Act on Alerts: When an anomaly is detected, Metaplane sends a targeted notification to your preferred communication channel (e.g., Slack, MS Teams, PagerDuty). The alert includes rich context, such as the nature of the issue and potential downstream impacts.
5. Investigate and Resolve: Use Metaplane's end-to-end column-level lineage to trace the issue back to its root cause. The lineage graph visualizes dependencies across your entire stack, significantly accelerating triage and resolution time.
6. Prevent Future Issues with Data CI/CD: Integrate Metaplane into your CI/CD pipeline (GitHub/GitLab). Before merging a pull request (e.g., for a dbt model change), Metaplane automatically runs regression and impact tests, forecasting which downstream tables and dashboards will be affected. This prevents data quality issues from ever reaching production.

Core Features of Metaplane

• Automated Anomaly Detection: ML-powered monitors that automatically adjust to seasonality and trends to detect issues in data volume, freshness, schema, uniqueness, nullness, and more.
• End-to-End Column-Level Lineage: Get a complete, automatically generated picture of your data stack, from sources to BI tools, to understand data flow and dependencies without manual setup.
• Data CI/CD: Prevent data quality issues by running automated regression and impact tests within pull requests. It integrates with dbt Core/Cloud and GitHub/GitLab.
• Data Insights & Usage Analytics: Discover how your data is being used, by whom, where, and how frequently. This helps in optimizing queries, reducing data debt, and prioritizing critical data assets.
• Automated & Configurable Alerts: Receive targeted, noise-free notifications in tools like Slack, MS Teams, and PagerDuty, with the context needed to resolve issues quickly.
• Schema Change Tracking: Get notified whenever databases, schemas, tables, or columns are added, renamed, or removed.
• Snowflake Native App: Deploy and run Metaplane directly within your Snowflake environment, ensuring data never leaves your warehouse and allowing you to pay with existing Snowflake credits.

Use Cases for Metaplane

Metaplane is valuable for various roles and teams within a data-driven organization:

• Data Engineering Teams: Proactively monitor data pipelines, reduce time spent firefighting data bugs, and ensure the reliability of data infrastructure.
• Analytics Engineers: Confidently develop and deploy dbt models using Data CI/CD to prevent breaking changes to downstream dashboards and analyses.
• BI and Analytics Teams: Build and maintain trust with business stakeholders by ensuring that the data powering reports and dashboards is accurate, fresh, and reliable.
• Data Platform Owners: Gain full visibility and control over the data stack, optimize warehouse costs by identifying unused assets, and enforce data governance standards.

Advantages of Metaplane

Metaplane offers several key advantages over traditional data quality solutions:

• Fast Time-to-Value: Setup takes minutes, not weeks, with actionable alerts generated within days.
• Cost-Effective Pricing: A flexible, usage-based model where you only pay for the tables you actively monitor, which scales with your needs.
• Reduced Alert Fatigue: Intelligent, ML-driven alerting that understands your data's normal patterns, significantly reducing false positives.
• Holistic Observability: It combines monitoring, lineage, usage insights, and CI/CD into a single, unified platform.
• Enterprise-Grade Security: SOC 2 Type II compliant and adheres to high privacy standards like GDPR, CCPA, and HIPAA, with a read-only metadata approach.

Pricing and Plans

Metaplane offers a flexible pricing structure designed for teams of all sizes.

• Free Plan: A forever-free plan that includes monitoring for up to 10 tables and support for 4 users. It provides core monitoring capabilities for small teams or for trying out the platform.
• Pro Plan: A usage-based plan where you pay per monitored table. It includes everything in the Free plan, plus advanced features like column-level lineage, Data CI/CD, cost & performance monitoring, and support for 12 users and unlimited viewers.
• Enterprise Plan: A custom plan for large organizations with specific needs. It includes all Pro features, plus custom integrations, SSO (Okta, SAML), PrivateLink support, and premium support with a dedicated CSM.

Additionally, users of the Snowflake Native App can pay for Metaplane using their existing Snowflake credits, simplifying procurement.