Dagster & Sigma (Component)
This feature is considered in a beta stage. It is still being tested and may change. For more information, see the API lifecycle stages documentation.
The dagster-sigma library provides a SigmaComponent which can be used to easily represent Sigma workbooks and datasets as assets in Dagster.
SigmaComponent is a state-backed component, which fetches and caches Sigma organization metadata. For information on managing component state, see Configuring state-backed components.
1. Prepare a Dagster project
To begin, you'll need a Dagster project. You can use an [existing components-ready project](/guides/build/projects/moving-to-components/migrating project) or create a new one:
uvx create-dagster project my-project && cd my-project/src
Activate the project virtual environment:
source ../.venv/bin/activate
Finally, add the dagster-sigma library to the project:
uv add dagster-sigma
2. Scaffold a Sigma component definition
Now that you have a Dagster project, you can scaffold a Sigma component definition:
dg scaffold defs dagster_sigma.SigmaComponent sigma_ingest
Creating defs at /.../my-project/src/my_project/defs/sigma_ingest.
The dg scaffold defs call will generate a defs.yaml file:
tree my_project/defs
my_project/defs
├── __init__.py
└── sigma_ingest
└── defs.yaml
2 directories, 2 files
3. Configure your Sigma organization
Update the defs.yaml file with your Sigma organization connection details. You'll need to provide your base URL, client ID, and client secret. For more information on creating API credentials, see the Sigma API documentation.
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: "{{ env.SIGMA_BASE_URL }}"
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ Orders_Dataset │ default │ │ dataset │ Sample orders dataset │ │
│ │ │ │ │ │ sigma │ │ │
│ │ ├─────────────────┼─────────┼──────┼──────────┼───────────────────────┤ │
│ │ │ Sample_Workbook │ default │ │ sigma │ │ │
│ │ │ │ │ │ workbook │ │ │
│ │ └─────────────────┴─────────┴──────┴──────────┴───────────────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────────────────┘
4. Filter Sigma content
You can filter which Sigma workbooks and datasets are loaded using the sigma_filter key. For example, you can load only workbooks from specific folders and exclude unused datasets:
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: "{{ env.SIGMA_BASE_URL }}"
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
sigma_filter:
workbook_folders:
- ["My Documents"]
include_unused_datasets: false
5. Customize Sigma asset metadata
You can customize the metadata and grouping of Sigma assets using the translation key:
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: "{{ env.SIGMA_BASE_URL }}"
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
translation:
group_name: sigma_data
description: "Sigma asset: {{ data.properties.name }}"
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ Orders_Dataset │ sigma_data │ │ dataset │ Sigma asset: Orders_Dataset │ │
│ │ │ │ │ │ sigma │ │ │
│ │ ├─────────────────┼────────────┼──────┼──────────┼──────────────────────────────┤ │
│ │ │ Sample_Workbook │ sigma_data │ │ sigma │ Sigma asset: Sample_Workbook │ │
│ │ │ │ │ │ workbook │ │ │
│ │ └─────────────────┴────────────┴──────┴──────────┴──────────────────────────────┘ │
└─────────┴───────────────────────────────────────────────────────────────────────────────────┘
Customize specific data types
You may also specify distinct translation behavior for specific data types. For example, you can add a tag to all workbooks:
type: dagster_sigma.SigmaComponent
attributes:
organization:
base_url: "{{ env.SIGMA_BASE_URL }}"
client_id: "{{ env.SIGMA_CLIENT_ID }}"
client_secret: "{{ env.SIGMA_CLIENT_SECRET }}"
translation:
for_workbook:
tags:
is_workbook: "true"
dg list defs --columns name,kinds,tags
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Kinds ┃ Tags ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ Orders_Dataset │ dataset │ │ │
│ │ │ │ sigma │ │ │
│ │ ├─────────────────┼──────────┼──────────────────────┤ │
│ │ │ Sample_Workbook │ sigma │ "is_workbook"="true" │ │
│ │ │ │ workbook │ │ │
│ │ └─────────────────┴──────────┴──────────────────────┘ │
└─────────┴───────────────────────────────────────────────────────┘