Creating a new Dagster project

The easiest way to start building a Dagster project is by using the create-dagster CLI. This CLI tool allows you to create a special type of Python package, called a project, that defines a Dagster code location.

Prerequisites

• Python 3.9+
• If using uv as your package manager, you will need to install uv (Recommended).
• If using pip as your package manager, you will need to install the create-dagster CLI with Homebrew, curl, or pip.

For detailed instructions, see the Installation guide.

Step 1. Scaffold a new Dagster project

uv

1. Open your terminal and scaffold a new Dagster project. You can replace my-project with a different project name if you wish:

   uvx -U create-dagster project my-project

2. Respond y to the prompt to run uv sync after scaffolding

   

3. Change to the project directory:

   cd my-project

4. Activate the virtual environment:

MacOS/Unix

source .venv/bin/activate

Windows

.venv\Scripts\activate

pip

1. Open your terminal and scaffold a new Dagster project. You can replace my-project with a different project name if you wish:

   create-dagster project my-project

2. Change to the project directory:

   cd my-project

3. Create and activate a virtual environment:

MacOS/Unix

python -m venv .venv

source .venv/bin/activate

Windows

python -m venv .venv

.venv\Scripts\activate

4. Install your project as an editable package:

   pip install --editable .

Your new Dagster project should have the following structure:

uv

.
└── my-project
   ├── pyproject.toml
   ├── src
   │   └── my_project
   │       ├── __init__.py
   │       ├── definitions.py
   │       └── defs
   │           └── __init__.py
   ├── tests
   │   └── __init__.py
   └── uv.lock

pip

.
└── my-project
   ├── pyproject.toml
   ├── src
   │   └── my_project
   │       ├── __init__.py
   │       ├── definitions.py
   │       └── defs
   │           └── __init__.py
   └── tests
      └── __init__.py

info

The create-dagster project command creates a directory with a standard Python package structure with some additions. For more information on the files and directories in a Dagster project, see the Dagster project file reference.

Step 2. Add assets

Assets are the core abstraction in Dagster and can represent logical units of data such as tables, datasets, or machine learning models. Assets can have dependencies on other assets, forming the data lineage for your pipelines. To add assets to your project, see Defining assets.

Step 3: View assets in the UI

To start the Dagster UI, run:

dg dev

To see your assets, navigate to http://localhost:3000.

Step 4: Continue development

• Add new Python dependencies
• Add integrations
• Use environment variables and secrets
• Add and run unit tests

Add new Python dependencies

You can specify new Python dependencies in pyproject.toml.

Add integrations

See the Integrations docs for a full list of Dagster-supported and community-supported integrations.

Use environment variables and secrets

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

For more information and examples, see Using environment variables and secrets.

Add and run unit tests

Tests can be added to the tests directory and run using pytest:

pytest tests

For more information on testing, see the following docs:

• The Testing guides have guidance on asset checks, data freshness checks, unit testing assets and ops, and testing partitioned config and jobs.
• Testing component definitions contains testing best practices for definitions scaffolded existing components.
• Testing your component has best practices for testing custom components.

Next steps

• Add integrations to your project
• Create your own Dagster Components to share with your team
• Deploy your project to Dagster+ or your own infrastructure