Integrating with DBT

dbt compile will compile the DBT project and generate the manifest file. This must be run first before SDF can work.

dbt compile

Using the next command, SDF will create a workspace.sdf.yml based on your DBT project’s configuration. This file will be placed adjacent to your dbt_project.yml file and should be committed to your repository.

sdf dbt init

Notice that your workspace.sdf.yml includes block points to files within the target directory. This is because SDF deals with raw SQL directly and not DBT models. To accomplish this, SDF copies the necessary compiled DBT models into the sdf directory with target/compiled (or your configured target-path).

This command will also compile DBT snapshots into a format SDF can understand, and port over credentials stored in your profiles.yml file to the ~/.sdf/credentials directory in the root of your system. This credentials will be used by SDF later to fetch required table schemas from the cloud warehouse.

As new changes to models or snapshots are made, running sdf dbt refresh will refresh the sdf workspace to point to the latest.

If using DBT with a cloud warehouse, you’ll likely need to configure the integrations block in the workspace.sdf.yml file. This block will contain the necessary information to connect to your warehouse.

In DBT terms, this block replaces your DBT sources. As such, it enables SDF to pull down the schema information for table dependencies not defined in SQL. For example, if I had some DBT sources coming from the database my_db in Snowflake, I would use the following configuration to pull them down at compile-time:

integrations:
	- provider: snowflake
		type: database
		sources:
			- pattern: my_db.*.*

For more information, or for guides on how to configure this for other warehouses, check out our Integrations section.

Now that we have our workspace.sdf.yml file configured, we can compile SDF. This will validate your SQL, dependencies, and produce all SDF artifacts (including column-level lineage). These artifacts will be placed it in the sdftarget directory local to your DBT project.

sdf compile

Great, now that we’ve successfully compiled our models, let’s try adding some metadata.

In our jaffle shop example, we have a table created from a dbt seed called raw_customers. This table contains two columns (first_name, last_name) with personally identifiable information (PII). Let’s classify these columns as PII in SDF, and ensure any downstream usage of these columns also inherits this classification.

First, let’s define our PII classifier in the workspace.sdf.yml file.

---
classifier: 
	name: pii
	description: Personally Identifiable Information
	labels: 
		- name: name
		  description: An individual's first, middle, or last name

Next, let’s attach this to the right columns raw_customers table in the workspace.sdf.yml file.

---
table:
	name: raw_customers
	columns:
	  - name: first_name
		  classifiers:
		  - pii.name
	  - name: last_name
		  classifiers:
		  - pii.name

Great, now let’s compile SDF again and see what happens.

sdf compile --show all

Working set 6 model files, 1 .sdf file
Finished 8 models [8 reused] in 0.089 secs

Schema jaffle_shop.dbt_alice.raw_customers
+-------------+-----------+------------+
| column_name | data_type | classifier |
+-------------+-----------+------------+
| id          | bigint    |            |
| first_name  | varchar   | pii.name   |
| last_name   | varchar   | pii.name   |
+-------------+-----------+------------+

...
...

Schema jaffle_shop.dbt_alice.stg_customers
+-------------+-----------+------------+
| column_name | data_type | classifier |
+-------------+-----------+------------+
| customer_id | bigint    |            |
| first_name  | varchar   | pii.name   |
| last_name   | varchar   | pii.name   |
+-------------+-----------+------------+

...
...

Schema jaffle_shop.dbt_alice.customers
+-------------------------+-----------+------------+
| column_name             | data_type | classifier |
+-------------------------+-----------+------------+
| customer_id             | bigint    |            |
| first_name              | varchar   | pii.name   |
| last_name               | varchar   | pii.name   |
| first_order             | date      |            |
| most_recent_order       | date      |            |
| number_of_orders        | bigint    |            |
| customer_lifetime_value | bigint    |            |
+-------------------------+-----------+------------+

You’ll notice the classification is not only attached to the raw_customers table, but also to the downstream customers table and others. This is because SDF is able to infer the lineage between these two tables and propagate the classification.