Using Dagster pipes

In this guide, we’ll show you how to use Dagster Pipes with Dagster’s built-in subprocess PipesSubprocessClient to run a local subprocess with a given command and environment. You can then send information such as structured metadata and logging back to Dagster from the subprocess, where it will be visible in the Dagster UI.

To get there, you'll:

• Create a Dagster asset that invokes a subprocess
• Modify existing code to work with Dagster Pipes to send information back to Dagster
• Learn about using Dagster Pipes with other entities in the Dagster system in the Reference section

note

This guide focuses on using an out-of-the-box PipesSubprocessClient resource. For further customization with the subprocess invocation, use open_dagster_pipes approach instead. Refer to Customizing Dagster Pipes protocols for more info.

Ready to get started?

You can get started by creating a Dagster asset that executes a subprocess.