Getting Started with Dagster and SDF

Getting started with Dagster and SDF is as easy as installing both with pip:

pip install dagster-sdf dagster-webserver

The dagster-sdf library installs both sdf-cli and dagster as python dependencies. If you’re starting from scratch, this will make the sdf CLI available to you, which you can use to interact with your SDF workspace.

To validate that you’ve installed the packages correctly, run the following commands and confirm that their output matches:

sdf --version

sdf 0.3.28

dagster-sdf --help

Usage: dagster-sdf [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                                                                                                                                                        
CLI tools for working with Dagster and sdf.                                                                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                                                                                            
 Options
  --help  -h  Show this message and exit.

 Commands
  workspace   Commands for using an sdf workspace in Dagster.

Select from one of the following to create a Dagster project alongside your SDF workspace. If you don’t have an SDF workspace on hand, create one from our samples using the following command:

  sdf new --sample moms_flower_shop_completed

• Using the dagster-sdf cli
• Author a definition file
• Bring your own Dagster project

dagster-sdf workspace scaffold --project-name ${DAGSTER_PROJECT_NAME}

Running with dagster-sdf version: 1.8.0.  
Initializing Dagster project my_project_name in the current working directory for sdf workspace  
directory /home/user/my_sdf_workspace.
Your Dagster project has been initialized. To view your SDF workspace in Dagster, run the following  
commands:  

cd ./my_project_name  
DAGSTER_SDF_COMPILE_ON_LOAD=1 dagster dev

Great! Now we have a Dagster project configured alongside our SDF workspace.

You may have noticed some new objects in your python code when generating or authoring your Dagster project. These are all made possible by and exposed through the dagster-sdf package installed in Step 1. We’ll briefly break down the key components here:

• SdfWorkspace: This is a representation of your SDF workspace in Dagster. Its core responsibility is to define the key components of a workspace, for example, the workspace and target output directories as well as which environment to prepare (i.e. compile) the workspace with.
• SdfCliResource: This is a Dagster resource that provides access to the SDF cli. It’s used by the @sdf_assets decorator to materialize your SDF workspace by running a command with the sdf-cli. It can be used to run arbitrary SDF commands, but is tested to work with the compile, run and test commands.
• @sdf_assets: This is a decorator that builds a multi-assets representation of your SDF workspace from the SDF information schema. This is used by Dagster to display your SDF workspace as an asset dag in the Dagster UI.

With your Dagster project set up, you can now materialize your SDF workspace using the Dagster UI.

Depending on the approach you took in Step 2, you can start the Dagster development server from either a:

• Dagster Project
• Dagster File

cd my_dagster_project/

DAGSTER_SDF_COMPILE_ON_LOAD=1 dagster dev

Serving dagster-webserver on http://127.0.0.1:3000 in process 70635

With the Dagster web server running, you can now navigate to the Dagster UI to view your SDF workspace as an asset dag.

From here, we can view the metadata available at compile time from SDF on a given asset, such as the asset description, raw sql, code location, and more. We can also see the Dagster assets that your SDF workspace depends on, and the assets that depend on your SDF workspace.

In Dagster, running an SDF model corresponds to materializing an asset. Each SDF model is represented as an asset in the Dagster UI. Depending on your configuration, running a model will result in it either executing locally using the SDF DB or executing remotely using the relevant configured integrations.

To see this in action, click on the Materialize all button on the asset lineage view. This will materialize all assets in the workspace, including your SDF assets.

If everything is set up correctly, you should see the assets in your SDF workspace materialize successfully. Inspecting a particular asset will reveal additional runtime metadata, such as the execution time, the materialized sql, and the table schema along with any propagated classifiers.

A common use case for using Dagster with SDF is when you have a complex data pipeline, parts of which are not authored in SDF. These assets can be upstream dependencies that your SDF workspace expects to have materialized prior to run.

SDF interprets these dependencies as source tables. Let’s take for example a snowflake warehouse which has tables that are materialized by different assets in your Dagster project.

If you have the following query in your SDF workspace:

SELECT * FROM upstream_dagster_table

Where upstream_dagster_table is not defined in SDF but is materialized in Dagster via:

import snowflake.connector
...
@asset
def build_upstream_dagster_table():
    credential = ...
    conn = snowflake.connector.connect(
        user=credential['username'],
        password=credential['password'],
        account=credential['account-id'],
        warehouse=credential['warehouse'],
        role=credential['role'],
    )
    cursor = conn.cursor()
    create_upstream_table = """
    CREATE OR REPLACE TABLE upstream_dagster_table (
        id INTEGER,
        field STRING,
    );
    """
    cursor.execute(create_upstream_table)
    cursor.close()
    conn.close()
...

Then you can define the upstream_dagster_table as a Dagster source in your SDF workspace by specifying a dagster-asset-key in the meta section of the table definition:

table:
  name: upstream_dagster_table
  meta:
    dagster-asset-key: build_upstream_dagster_table

This will have no impact on SDF’s execution, but will give Dagster the context that the upstream_dagster_table is a Dagster asset that needs to be materialized before the SDF workspace can be run.