Rerun

Rerun is a specialized data stack engineered for the emerging field of Physical AI, which encompasses robotics, spatial computing, and embodied AI. It offers a comprehensive suite of open-source tools designed to handle the complexities of multimodal, time-series data. At its core, Rerun provides a powerful visualization and logging solution that enables developers and researchers to see, understand, and debug their systems with unprecedented clarity. By offering SDKs in popular languages like Python, C++, and Rust, Rerun integrates seamlessly into existing development workflows, making it an essential tool for anyone working with complex sensor data, simulations, or AI models that interact with the physical world.

The platform is built around a time-aware Entity Component System (ECS) data model, which is uniquely suited for handling streams of heterogeneous data (e.g., 3D point clouds, images, transforms, video streams) while keeping them perfectly synchronized in time. This allows for intuitive 'time-travel' debugging, where users can scrub through a recording to pinpoint the exact moment an issue occurred. The Rerun Viewer is a high-performance, interactive application that runs natively, in the browser, or can be embedded directly into Jupyter notebooks and other applications, offering maximum flexibility.

How to use Rerun

Using Rerun is designed to be straightforward, allowing you to start visualizing data in minutes. Here’s a typical workflow:

1. Installation: Install the Rerun SDK for your preferred language. For Python, it's as simple as running pip install rerun-sdk.
2. Initialization: In your application code, initialize the Rerun library and connect to a viewer. You can spawn a new viewer process, connect to a remote one, or log to a file for later viewing. Example: rr.init("my_application", spawn=True).
3. Logging Data: Use the rr.log() function to send data to the viewer. Rerun provides built-in archetypes for common data types like 3D points (rr.Points3D), images (rr.Image), 3D transforms (rr.Transform3D), and even live video streams (rr.VideoStream). You can log positions, colors, text, tensors, and more.
4. Visualization: The Rerun Viewer will automatically display your logged data. You can interact with the visualization by rotating, panning, and zooming. Use the timeline slider at the bottom to navigate through time, play back the sequence, or jump to specific events.
5. Customization and Analysis: Customize layouts and visualizations directly in the UI or programmatically via the SDK. For deeper analysis, use Rerun's query APIs to extract time-aligned data from your recordings into formats like Apache Arrow, which can be easily loaded into data analysis libraries like Pandas or Polars.

Core Features of Rerun

• Multimodal Visualization: Natively supports a wide range of data types including 3D point clouds, images, text, tensors, 3D/2D geometric shapes, and transforms.
• Time-Series Focus: A central timeline allows for intuitive scrubbing and playback of data, which is crucial for debugging dynamic systems.
• Multi-Language SDKs: Provides easy-to-use SDKs for Python, Rust, and C++, ensuring broad compatibility with most robotics and AI projects.
• High-Performance Viewer: Built from the ground up in Rust for performance, the viewer can handle large datasets smoothly. It runs natively on major OSes, in modern web browsers via WebAssembly, and can be embedded.
• Flexible Data Model: The Entity Component System (ECS) model allows for logging complex, evolving data structures without rigid schemas.
• Live Streaming & Recording: Supports both live streaming of data for real-time debugging and recording to efficient .rrd files for offline analysis and sharing.
• Robotics-Ready: Includes built-in features for robotics, such as a URDF (Unified Robot Description Format) data loader to visualize and animate robot models.
• Data Querying API: Enables programmatic extraction of clean, time-aligned datasets from messy log files, bridging the gap between debugging and model training.

Use Cases for Rerun

Rerun is versatile and used across various domains:

• Robotics: Debugging robot perception stacks, visualizing sensor fusion (LIDAR, camera, IMU), animating robot arm kinematics, and analyzing simulation logs. The Hugging Face LeRobot project uses Rerun for its visualization tools.
• Computer Vision: Visualizing the intermediate steps and final outputs of algorithms for object detection, semantic segmentation, SLAM (Simultaneous Localization and Mapping), and 3D reconstruction.
• Spatial & Embodied AI: Understanding the behavior of AI agents in simulated or real-world environments by logging their perceptions, states, and actions over time.
• Generative Media: Visualizing the evolution of generative models, such as viewing the diffusion process in image generation models frame by frame.
• Simulations: Logging and replaying complex physical or multi-agent simulations to understand emergent behaviors and debug system dynamics.

Advantages of Rerun

Rerun offers significant advantages for developers in the Physical AI space:

• Intuitive Debugging: The 'time-travel' feature transforms debugging from a tedious, print-based process into an intuitive, visual exploration.
• Accelerated Development: By making it easy to 'see' what your code is doing, Rerun drastically reduces the time it takes to identify and fix bugs in complex systems.
• Improved Collaboration: Rerun's .rrd log files are self-contained and portable, making it easy to share complex scenarios with teammates for collaborative debugging.
• Open Source and Community-Driven: The core visualization tools are free, open-source (MIT/Apache 2 licensed), and actively developed with input from a growing community.
• Scalable Architecture: The platform is designed to scale from a simple script on a laptop to a large-scale data management solution with its upcoming commercial offering.

Pricing and Plans

Rerun operates on a freemium model:

• Open Source: The core Rerun SDKs and Viewer are completely free and open source. This includes all visualization, logging, and simple log handling functionalities. It is dual-licensed under MIT and Apache 2.
• Commercial: Rerun is developing a commercial cloud platform designed for data management at scale. This will provide managed infrastructure for ingestion, storage, indexing, and streaming of large-scale Physical AI data. This product is currently under development with select design partners, and interested parties can sign up for a waitlist.