Creating a Workspace
To create a new SDF Workspace, run the following command: Created hello/.gitignore
Created hello/models/main.sql
Created hello/workspace.sdf.yml
Welcome to your new SDF Workspace! To help you on your journey:
💡 Join the SDF Community Slack -> https://sdf.com/join
📚 Read the Docs to Get Started -> https://docs.sdf.com/
Finished new in 0.289 secs
You can view and modify the source code for all our examples directly in our GitHub repository.
hello
. The workspace will contain the following files and folder structure:
.
├── models
│ └── main.sql
└── workspace.sdf.yml2 directories, 2 files
workspace.sdf.yml
file, which is the primary configuration file for your project. It contains the following YML:
workspace:
name: hello
edition: “1.3”
description: “A minimal workspace” includes:
- path: models
main.sql
, which contains the following SQL:
select ‘Hello World!’ as message
SDF uses a cache to fingerprint outputs and accelerate recomputation. This cache is by default located in the
sdftarget/
directory.
The cache is machine specific and should not be checked in to git. An appropriate .gitignore
file is created as part of the sdf new
command.We refer to SQL statements in SDF as
models
. Models are SQL statements that will materialized in your data warehouse, or locally with the SDF DB. They differ from tables
as they can be materialized as tables, views, and more based on the configuration. Furthermore, they can be templatized with jinja and SDF SQL variables. SDF recommends specifying one model per file, as each model receives a fully qualified name (database.schema.table
) that can correspond nicely to a directory structure.
See our indexing documentation for more.Exploring Your Project
Let’s see just how easy it is to set up SDF and run your first query.1
Static Analysis with `sdf compile`
First, we’ll run the core command of SDF: sdf compile.The Your output should look like:As you can see from the output, SDF has statically analyzed the query and determined there’s a single non-nullable column named
--show
flag allows you to modulate SDF’s output and the all
option indicates that we would like to see all schemas from all models referenced in the workspace.Working set 1 model file, 1 .sdf file
Compiling hello.pub.main (./models/main.sql)
Finished 1 model [1 succeeded] in 0.871 secsSchema hello.pub.main
┌─────────────┬──────────────────┬────────────┬─────────────┐
│ column_name ┆ data_type ┆ classifier ┆ description │
╞═════════════╪══════════════════╪════════════╪═════════════╡
│ message ┆ varchar not null ┆ ┆ │
└─────────────┴──────────────────┴────────────┴─────────────┘
column
and it’s of type varchar
.
You’ll also see an empty classifier
block in the output. This is for metadata we’ll attach to columns, but we’ll get to that later.2
Create a New Model
Now, create a new file in the source directory called
main2.sql
with the query:main2.sql
3
Lineage with `sdf lineage`
SDF guarantees rich column level lineage. The command below specifies a particular column, in a particular table that we would like to inspect.
Working set 2 model files, 1 .sdf file
Compiling hello.pub.main (./models/main.sql)
Compiling hello.pub.main2 (./models/main2.sql)
Finished 2 models [2 succeeded] in 0.853 secs
hello.pub.main2.message
│
│ copy
└──────┐
hello.pub.main.message
4
Execution with `sdf run`
Next, let’s execute the query, using SDF as the database. We’ll execute the query with SDF’s integrated execution runtime, right on your machine.Your output should look like:
Working set 2 model files, 1 .sdf file
Running hello.pub.main (./models/main.sql)
Running hello.pub.main2 (./models/main2.sql)
Finished 2 models [2 succeeded] in 0.854 secsTable hello.pub.main
┌──────────────┐
│ message │
╞══════════════╡
│ Hello World! │
└──────────────┘
1 rows.Table hello.pub.main2
┌──────────────┐
│ message │
╞══════════════╡
│ Hello World! │
└──────────────┘
1 rows.
If using VSCode, SDF’s YML schema is available for type and syntax checking via the Red HAT YAML. This will
add auto-fill, type checking, and YML validation directly inline while editing
sdf.yml
files.