sdf compile
.
Checks are simply small SQL queries that run against this information schema and check for the existence of certain properties which you want to assure. Some examples of effective checks are:
Create a Sample Workspace
Created hello_with_pii/.gitignore
Created hello_with_pii/checks/code_check.sql
Created hello_with_pii/models/main.sql
Created hello_with_pii/workspace.sdf.yml
Welcome to your new SDF Workspace! To help you on your journey:
💡 Join the SDF Community Slack -> https://sdf.com/join
📚 Read the Docs to Get Started -> https://docs.sdf.com/
Finished new in 0.312 secs
Include Checks in Workspace
woskspace.sdf.yml
.Run the Check
sdf check
. During compile, SDF runs all its static analysis on the workspace, creating lineage, metadata, and ultimately executing the checks.Hurray! The check passes.Working set 1 model file, 1 .sdf file
Compiling hello.pub.main (./models/main.sql)
Working set 1 check file, 1 .sdf file
Checking hello.pub.code_check (./checks/code_check.sql)
Finished 1 model [1 succeeded], 1 check [1 passed] in 2.987 secs
[Pass] Check hello.pub.code_check
Add a Classifier
sdf check
again will result in a failed test. The output will look like this:Working set 1 model file, 1 .sdf file
Compiling hello.pub.main (./models/main.sql)
Working set 1 check file, 1 .sdf file
Checking hello.pub.code_check (./checks/code_check.sql)
Finished 1 model [1 succeeded], 1 check [1 failed] in 3.049 secs, for details see below.[Fail] Check hello.pub.code_check
┌────────────┬─────────────┬─────────────┐
│ table_name ┆ column name ┆ classifiers │
╞════════════╪═════════════╪═════════════╡
│ main ┆ column_2 ┆ [PII.name] │
└────────────┴─────────────┴─────────────┘
1 rows.-------
Summary 1 model [1 succeeded], 1 check [1 failed] in 3.049 secs.
-------
md5()
hash to column_2 (the name column) to obfuscate it. This is called a reclassification since we are transforming one classifier into another.
Add a Reclassifying Function
workspace.sdf.yml
create the following function block.Create a New Model
Update Check
and c.table_name like '%.sink'
. It should now look like the check below.
Now, the Check specifically checks the sink table for the PII.name classifier.sdf check
again and voila! All checks pass. The source table has PII.name, the sink table does not.Working set 2 model files, 1 .sdf file
Compiling hello.pub.main (./models/main.sql)
Compiling hello.pub.sink (./models/sink.sql)
Working set 1 check file, 1 .sdf file
Checking hello.pub.code_check (./checks/code_check.sql)
Finished 2 models [2 succeeded], 1 check [1 passed] in 3.080 secs
[Pass] Check hello.pub.code_check