dstack

dstack is a powerful open-source container orchestrator specifically engineered to address the challenges faced by AI and Machine Learning teams. Its primary goal is to simplify the complex process of workload orchestration and significantly improve the utilization of expensive GPU resources. As a vendor-agnostic platform, dstack offers a unified compute layer that seamlessly integrates with any GPU cloud (like AWS, GCP, Azure, OCI), on-premise clusters, and a wide array of accelerated hardware, including NVIDIA, AMD, TPUs, and more. This flexibility ensures that teams are not locked into a single provider and can leverage the best hardware for their needs, wherever it is located.

The platform is designed with the developer experience at its core, abstracting away the underlying infrastructure complexities. This allows ML engineers and researchers to focus on building, training, and deploying models rather than managing servers, dependencies, and scaling. dstack is trusted by world-class ML teams at companies like Electronic Arts and Mobius Labs for its ability to scale from quick prototyping to large, multi-node distributed training jobs.

How to use dstack

Getting started with dstack is a straightforward process designed for rapid adoption:

1. Setup the Server: You can begin by installing the dstack server on your local machine using a simple command like uv tool install "dstack[all]" and running it with dstack server. Alternatively, you can deploy it anywhere using the official Docker image or sign up for dstack Sky, the managed cloud version, to avoid hosting it yourself.
2. Define Configurations: Workflows in dstack are defined using simple YAML files within your project repository. These configurations describe the environment, resources, and commands for your tasks. Key configuration types include:
   • Dev Environments: For interactive development, allowing you to connect your local IDE (like VS Code) to a powerful remote GPU machine.
   • Tasks: For scheduling batch jobs, such as pre-training or fine-tuning models. This is ideal for workloads that run to completion.
   • Services: For deploying models as secure, auto-scaling, OpenAI-compatible endpoints.
   • Fleets: For managing groups of cloud or on-premise instances as a single resource pool.
3. Apply Configurations: Once your YAML file is ready, you apply it using the command line interface: dstack apply. dstack then handles the rest: provisioning the necessary infrastructure, scheduling the job, managing auto-scaling, handling port-forwarding, and streaming logs back to your terminal. For detached execution, you can use the -d flag.

Core Features of dstack

• Unified Compute Layer: Provides a single, vendor-agnostic control plane for all your AI compute resources, whether on-cloud or on-premise.
• Broad Accelerator Support: Natively supports a wide range of hardware, including NVIDIA GPUs, AMD GPUs, Google Cloud TPUs, Intel Gaudi, and Tenstorrent accelerators.
• Developer-Centric Workflows: Offers specialized configurations like Dev Environments for interactive coding, Tasks for batch processing, and Services for easy model deployment.
• Efficient Resource Management: Features a built-in scheduler to maximize GPU utilization. It includes policies to automatically terminate underutilized instances, saving costs.
• Seamless Integration: Works smoothly with leading GPU clouds (AWS, GCP, Azure, OCI) and can run on top of existing Kubernetes clusters. SSH fleets allow connecting bare-metal servers.
• Auto-Scaling Services: Easily deploy models as production-ready services with features like auto-scaling, HTTPS, and OpenAI-compatible API endpoints.
• Data Persistence: Supports network and instance volumes to persist data, models, and caches across runs, ensuring state is not lost.
• Advanced Configuration: Allows for fine-grained control with features like retry policies for capacity issues, environment variable management, and custom Docker image support.

Use Cases for dstack

dstack is versatile and supports a wide range of ML workflows:

• Model Training and Fine-Tuning: Run single-node or distributed training jobs for large language models (LLMs) using popular frameworks like TRL, Axolotl, and DeepSpeed.
• Inference and Model Serving: Deploy optimized models for inference using high-performance serving frameworks like vLLM, SGLang, TGI, and NVIDIA NIM.
• Interactive AI Development: ML engineers can spin up powerful GPU-backed development environments in seconds, connecting their local IDE to experiment and debug code interactively.
• High-Performance Cluster Management: Set up, configure, and run tests (e.g., NCCL tests) on specialized multi-node clusters like GCP A3 Mega or AWS EFA-enabled instances.
• Cross-Cloud Cost Optimization: Effortlessly compare and utilize the most cost-effective GPU instances across different cloud providers for any given task.

Advantages of dstack

The primary advantage of dstack is its ability to dramatically simplify AI infrastructure. It empowers ML teams by letting them focus on their research and models instead of infrastructure. Key benefits include increased productivity, significant cost savings through better GPU utilization and access to spot instances, and prevention of vendor lock-in. Its open-source nature fosters transparency and community-driven development, while the developer-centric design makes it incredibly easy to define a configuration and run it without worrying about GPU availability or complex setups.

Pricing and Plans

dstack offers a flexible pricing structure to suit different needs:

• dstack (Open-Source): The core platform is open-source and free to use. You can self-host it on your own infrastructure without any licensing fees.
• dstack Sky: A managed cloud service that handles the hosting of the dstack server for you. It also provides access to a marketplace of the cheapest GPUs. It offers a free tier to get started.
• dstack Enterprise: A self-hosted version designed for larger organizations, which includes enterprise-grade features like Single Sign-On (SSO), advanced governance controls, and dedicated enterprise support. A trial can be requested for this version.

This model makes dstack accessible to individual researchers, startups, and large enterprises alike.