Skip to main content

Configuring CI/CD in Dagster+

Follow the steps below to create a GitHub or GitLab CI/CD configuration file in your project to deploy and synchronize your code to Dagster+. You can also use other Git providers, or a local Git repository to run your own CI/CD process.

A note on Dagster+ Serverless

When you sign up for Dagster+ Serverless, you create a GitHub or GitLab repository that contains basic Dagster project code and a CI/CD workflow file consistent with Dagster+ best practices.

Pushing changes to the main branch of that project repository will automatically deploy them to Dagster+. Pull or merge requests will create ephemeral branch deployments that you can preview and test in the Dagster+ UI.

To add another Dagster project to your Serverless deployment, follow the steps in this guide to scaffold a GitHub or GitLab CI/CD configuration file in your new project to deploy and synchronize your code to Dagster+ Serverless. You can also use other Git providers or a local Git repository with the dagster-cloud CLI to run your own CI/CD process.

GitHub​

Prerequisites

Before following the steps in this section, you will need to:

  1. Change to the project root directory:

    cd <project-directory>

  2. Activate the virtual environment:

    shell source .venv/bin/activate

  3. Initialize a Git repository in the project directory:

    git init .

  4. Create a remote repository on GitHub to connect with the local project repository. Be sure to select Push an existing local repository to github.com when prompted:

    gh repo create

  5. Use the dg plus deploy configure CLI command to scaffold deployment configuration files for your deployment type, including a GitHub Actions workflow file:

    dg plus deploy configure --git-provider github

  6. Create a Dagster Cloud API token and set it as a GitHub Action secret for the project:

    dg plus create ci-api-token --description 'Used in my-project GitHub Actions' | gh secret set DAGSTER_CLOUD_API_TOKEN

  7. Commit and push your changes to deploy to Dagster+:

    git add . && git commit -m "Deploy to Dagster+" && git push origin main

During the deployment, the agent will attempt to load your code and update the metadata in Dagster+. When that has finished, you should see the GitHub Action complete successfully, and also be able to see the code location under the Deployment tag in Dagster+.

tip

We recommend periodically (every 6 months or so) updating the version of the Dagster GitHub Actions in your workflow files. These actions use version tags that follow the Dagster release convention (e.g., @v1.11.13). You can find the most recent Dagster release on GitHub.

GitLab​

Prerequisites

Before following the steps in this section, you will need to:

  1. Change to the project root directory:

    cd <project-directory>

  2. Activate the virtual environment:

    • MacOS/Unix: 
      source .venv/bin/activate
    • Windows: 
      .venv\Scripts\activate

  3. Initialize a Git repository in the project directory:

    git init .

  4. Commit and push your changes:

    git add . && git commit -m "Initial commit"

  5. Create a remote repository on GitLab and push your changes to it. You can either do so in the GitLab UI by navigating to the new project creation page, or on the command line by running the following command, replacing USERNAME with your GitLab username or organization workspace:

    git push --set-upstream git@gitlab.com:USERNAME/$(git rev-parse --show-toplevel | xargs basename).git $(git rev-parse --abbrev-ref HEAD)

  6. Use the dg plus deploy configure CLI command to scaffold deployment configuration files for your deployment type, including a GitLab CI/CD configuration file:

    dg plus deploy configure --git-provider gitlab

  7. Create a Dagster Cloud API token:

    dg plus create ci-api-token --description 'Used in Dagster project GitHub Actions'

  8. Set the Dagster Cloud API token as a CI/CD variable in the GitLab repo:

    • Navigate to the project page in GitLab.
    • In the left sidebar, click Settings > CI/CD.
    • On the settings page, click Variables.
    • Under Project variables, click Add variable.
    • In the Key field, enter DAGSTER_CLOUD_API_TOKEN
    • In the Value field, paste the Dagster Cloud API token.
    • Optionally update the variable type, environments, visibility, flags, and description fields as needed.
    • Click Add variable.

After following these steps, commiting and pushing additional changes to your Dagster project in GitLab will deploy them to your Dagster+ organization.

Other Git providers or local Git repository​

Prerequisites

Before following the steps in this section, you must first create a Dagster project.

If you don't want to use our automated GitHub/GitLab process, you can use the dagster-cloud command-line CLI in another CI environment or locally.

  1. First, create a new project with the create-dagster project command and activate the project virtual environment.

  2. Next, install the dagster-cloud CLI and use the configure command to authenticate it to your Dagster+ organization:

    pip install dagster-cloud
    dagster-cloud configure

You can also configure the dagster-cloud tool non-interactively; for more information, see the dagster-cloud installation and configuration docs.

  1. Finally, deploy your project to Dagster+ using the serverless command, replacing YOUR_PACKAGE_NAME with the name of your Dagster package:
dagster-cloud serverless deploy-python-executable ./my-project \
  --location-name example \
  --package-name YOUR_PACKAGE_NAME \
  --python-version 3.12
Windows variant

If you are using Windows, you will need to replace the deploy-python-executable command with deploy:

dagster-cloud serverless deploy ./my-project \
  --location-name example \
  --package-name YOUR_PACKAGE_NAME \
  --python-version 3.12