Skip to main content

Converting an existing project

In this guide, we walk through converting an existing Dagster project that defines a Python package with a single Dagster asset, which is exposed in a top-level Definitions object in my_existing_project/definitions.py.

note

We'll cover both the case of uv with pyproject.toml, and pip with setup.py.

If you used uv to set up your Dagster project, you will have a pyproject.toml project configuration file:

tree
.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
├── pyproject.toml
└── uv.lock

2 directories, 6 files

Step 1: Create and activate a virtual environment​

First, you will need to make sure you have an activated and up-to-date virtual environment. We recommend creating the virtual environment in the project root (particularly when using uv), although it is not required.

If you don't have a virtual environment yet, run:

uv sync

Then activate it:

source .venv/bin/activate

Step 2: Install dependencies​

Next, you will need to install the dg command line tool into your project virtual environment.

uv add dagster-dg-cli

Step 3: Update project structure​

Step 3.1: Add dg configuration​

The dg command recognizes Dagster projects through the presence of TOML configuration. This may be either a pyproject.toml file with tool.dg and tool.dg.project sections, or a dg.toml file.

Since the example project already has a pyproject.toml file, you just need to add tool.dg and tool.dg.project sections:

pyproject.toml
...
[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

Configuration reference​

  • directory_type = "project": This is how dg identifies your package as a Dagster project. This setting is required.
  • root_module = "my_existing_project": This points to the root module of your project. This setting is required.
  • code_location_target_module = "my_existing_project.definitions": This tells dg where to find the top-level Definitions object in your project. Since this setting defaults to NAME_OF_ROOT_MODULE.definitions, it is not strictly necessary to set it for this example. However, if your project has the top-level Definitions object defined in a different module, this setting is required.

Step 3.2: Check configuration​

Now that the project configuration file has been updated, you can interact with your project using dg. To check the configuration and see the one asset in this project, run dg list defs:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key      ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ my_asset │ default │      │       │             │ │
│         │ └──────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────┘

Step 3.3: Add an entry point to the project for custom component types​

dg uses the Python entry point API to expose custom component types and other scaffoldable objects from user projects. In the project configuration file, the entry point specifies the location of a Python submodule where the project exposes registry modules.

note

By convention, this submodule is named <root_module>.lib, but in the case of a Dagster project, it is <root_module>.components.

Step 3.3.1: Create submodule directory​

First, create the submodule directory and __init__.py file:

mkdir my_existing_project/components && touch my_existing_project/components/__init__.py

Step 3.3.2: Add entry point to project configuration file​

Next, add a dagster_dg_cli.registry_modules entry point to your project configuration file, then reinstall the project package into your virtual environment.

Since the package metadata is in pyproject.toml, you will need to add the entry point declaration there:

pyproject.toml
...
[project.entry-points]
"dagster_dg_cli.registry_modules" = { my_existing_project = "my_existing_project.components"}
...

Step 3.3.3: Reinstall the project package​

Reinstall the package. Note that uv sync will not reinstall your package, so you will need to use uv pip install instead:

uv pip install --editable .
warning

The reinstallation step is crucial. Python entry points are registered at package installation time, so if you simply add a new entry point to an existing editable-installed package, it won't be picked up.

Step 3.4: Check configuration​

If you've done everything correctly, you should now be able to run dg list registry-modules and see the module my_existing_project.components, which you have registered as an entry point, listed in the output:

dg list registry-modules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Module                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster                        │
│ my_existing_project.components │
└────────────────────────────────┘

You can now scaffold a new component in your project, and it will be available to dg commands. First create the component:

dg scaffold component Foo

Scaffolded Dagster component at /.../my-existing-project/my_existing_project/components/foo.py.

Then run dg list components to confirm that the new component is available:

dg list components
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key                                ┃ Summary                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster.DefinitionsComponent       │ An arbitrary set of Dagster definitions.                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.DefsFolderComponent        │ A component that represents a directory containing multiple Dagster definition  │
│                                    │ modules.                                                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.FunctionComponent          │ Represents a Python function, alongside the set of assets or asset checks that  │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.PythonScriptComponent      │ Represents a Python script, alongside the set of assets and asset checks that   │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.TemplatedSqlComponent      │ A component which executes templated SQL from a string or file.                 │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.UvRunComponent             │ Represents a Python script, alongside the set of assets or asset checks that it │
│                                    │ is responsible for executing.                                                   │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ my_existing_project.components.Foo │ COMPONENT SUMMARY HERE.                                                         │
└────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘

You should see the my_project.lib.MyComponentType listed in the output.

Step 3.5: Create a defs submodule​

Part of the dg experience is autoloading definitions. This means automatically picking up any definitions that exist in a particular module. In this step, you will create a new submodule named my_existing_project.defs (defs is the conventional name of the module for where definitions live in dg) from which you will autoload definitions:

mkdir my_existing_project/defs

Step 4: Modify top-level definitions​

Step 4.1: Modify definitions.py file​

Autoloading is provided by a function that returns a Definitions object. Because there are already some other definitions in the example project, you will need to combine those with the autoloaded ones from my_existing_project.defs.

To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.

You will autoload definitions using load_from_defs_folder, then merge them with your existing definitions using Definitions.merge. You pass load_from_defs_folder the defs submodule you just created:

import dagster as dg
from my_existing_project.assets import my_asset

defs = dg.Definitions(
    assets=[my_asset],
)

Step 4.2: Add an asset to new defs submodule​

Next, add an asset to the new defs submodule. Create a file called my_existing_project/defs/autoloaded_asset.py with the following contents:

import dagster as dg


@dg.asset
def autoloaded_asset(): ...

Step 4.3: Confirm the new asset is autoloaded​

Finally, confirm the new asset is being autoloaded. If you run dg list defs again, you should see both the new autoloaded_asset and old my_asset:

dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key              ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ autoloaded_asset │ default │      │       │             │ │
│         │ ├──────────────────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ my_asset         │ default │      │       │             │ │
│         │ └──────────────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────┘

Now the project is fully compatible with dg!

Next steps​