Creating Dagster projects

The easiest way to start building a Dagster project is by using the create-dagster CLI. This CLI tool allows you to create a special type of Python package accessible to the Dagster webserver and UI.

Prerequisites

• Python 3.10+
• If using uv as your package manager, you will need to install uv (Recommended).
• If using pip as your package manager, you will need to install the create-dagster CLI with Homebrew, curl, or pip.

For detailed instructions, see the Installation guide.

Step 1. Scaffold a new Dagster project

uv

1. Open your terminal and scaffold a new Dagster project. You can replace my-project with a different project name if you wish:

   uvx create-dagster@latest project my-project

2. Respond y to the prompt to run uv sync after scaffolding

   

3. Change to the project directory:

   cd my-project

4. Activate the virtual environment:

MacOS/Unix

source .venv/bin/activate

Windows

.venv\Scripts\activate

pip

1. Open your terminal and scaffold a new Dagster project. You can replace my-project with a different project name if you wish:

   create-dagster project my-project

2. Change to the project directory:

   cd my-project

3. Create and activate a virtual environment:

MacOS/Unix

python -m venv .venv

source .venv/bin/activate

Windows

python -m venv .venv

.venv\Scripts\activate

4. Install your project as an editable package:

   pip install --editable .

Your new Dagster project should have the following structure:

uv

.
└── my-project
   ├── pyproject.toml
   ├── src
   │   └── my_project
   │       ├── __init__.py
   │       ├── definitions.py
   │       └── defs
   │           └── __init__.py
   ├── tests
   │   └── __init__.py
   └── uv.lock

pip

.
└── my-project
   ├── pyproject.toml
   ├── src
   │   └── my_project
   │       ├── __init__.py
   │       ├── definitions.py
   │       └── defs
   │           └── __init__.py
   └── tests
      └── __init__.py

info

The create-dagster project command creates a directory with a standard Python package structure with some additions. For more information on the files and directories in a typical Dagster project, see the Dagster project file and directory reference.

Step 2. Add assets

Assets are the core abstraction in Dagster and can represent logical units of data such as tables, datasets, or machine learning models. Assets can have dependencies on other assets, forming the data lineage for your pipelines. To add assets to your project, see Defining assets.

Step 3: View assets in the UI

To start the Dagster UI, run:

dg dev

To see your assets, navigate to http://localhost:3000.

tip

See Running Dagster locally for more information on configuring and running your local Dagster instance, including creating a persistent instance and detecting when you're running locally to selectively run schedules or sensors depending on environment.

Step 4: Continue local development

Add new Python dependencies

You can specify new Python dependencies in pyproject.toml.

Add integrations

See the Integrations docs for a full list of Dagster-supported and community-supported integrations.

Use environment variables and secrets

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

For more information and examples, see Using environment variables and secrets.

Add and run unit tests

Tests can be added to the tests directory and run using pytest:

pytest tests

For more information on testing, see the following docs:

• The Testing guides have guidance on asset checks, data freshness checks, unit testing assets and ops, and testing partitioned config and jobs.
• Testing component definitions contains testing best practices for definitions scaffolded existing components.
• Testing your component has best practices for testing custom components.

Create custom components

If built-in Dagster components and integrations don't meet your needs, you can create custom components to share with your team.

Step 5: Deploy your project to production (Optional)

• OSS: Follow the steps in the OSS deployment docs to set up a production OSS deployment. You will need to add your project code to the Docker container used in the deployment.
• Dagster+ Serverless or Hybrid: Follow the steps in Configuring CI/CD in Dagster+.