Installing SDF

Installing SDF is easy as running the following command:

    curl -LSfs | sh -s

For more information on installation, see our installation guide.

Verify Installation

To verify that SDF is installed correctly, run the following command:

sdf --help

SDF: A fast SQL compiler, local development framework, and in-memory analytical database

Usage: sdf [OPTIONS] <COMMAND>

Commands:   new      Create a new sdf workspace   clean    Remove artifacts that sdf has generated in the past   compile  Compile models   run      Run models   test     Test your models   stats    Statistics for your data   report   Report code quality   check    Check code quality   lineage  Display lineage for a given table and/or column   push     Push a local workspace to the SDF Service   system   System maintenance, install and update   auth     Authenticate CLI to services like SDF, AWS, OpenAI, etc   man      Display reference material, like the CLI, dialect specific functions, schemas for authoring and interchange   dbt      Initialize an sdf workspace from an existing dbt project   exec     Execute custom scripts   help     Print this message or the help of the given subcommand(s)

Options:       —log-level <LOG_LEVEL>           Set log level                      [possible values: trace, debug, info, warn, error]

      —log-file <LOG_FILE>           Creates or replaces the log file

      —log-form <LOG_FORM>           Set the log format; default flat

          Possible values:           - flat:   Writes nd-json, compact, no nested objects           - nested: Writes nd-json, compact, nested objects           - pretty: Writes nd-json, pretty, nested objects

  -h, —help           Print help (see a summary with ‘-h’)

  -V, —version           Print version

Creating a Workspace

To create a new SDF Workspace, run the following command:

sdf new --sample hello && cd hello

After running the command, you will see the following output:

    Created hello/.gitignore     Created hello/models/main.sql     Created hello/workspace.sdf.yml    Finished new in 0.299 secs

You can view and modify the source code for all our examples directly in our GitHub repository.

This will create a new directory with the name of your workspace. In this case, that’s hello. The workspace will contain the following files and folder structure:

. ├── models │   └── main.sql └── workspace.sdf.yml

1 directory, 2 files

The directory will contain a workspace.sdf.yml file, which is the primary configuration file for your project. It contains the following YML:

workspace:   name: hello   edition: “1.3”   description: “A minimal workspace”

  includes:     - path: models

For more on this, see our workspaces guide and for a full reference to our YML schema and more see the reference section.

Lastly, our workspace contains a single model file, main.sql, which contains the following SQL:

select ‘Hello World!’ as message; 

Next, let’s take a deeper look at your project.

SDF uses a cache to fingerprint outputs and accelerate recomputation. This cache is by default located in the sdftarget/ directory. The cache machine specific and should not be checked in to git. An appropriate .gitignore file is created as part of the sdf new command.

We refer to SQL statements in SDF as models. Models are SQL statements that will materialized in your data warehouse, or locally with the SDF DB. They differ from tables as they can be materialized as tables, views, and more based on the configuration. Furthermore, they can be templatized with jinja and SDF SQL variables. SDF recommends specifying one model per file, as each model receives a fully qualified name (database.schema.table) that can correspond nicely to a directory structure. See our indexing documentation for more.

Exploring Your Project

Let’s see just how easy it is to set up SDF and run your first query.


Static Analysis with `sdf compile`

First, we’ll run the core command of SDF: sdf compile.

The --show flag allows you to modulate SDF’s output and the all option indicates that we would like to see all schemas from all models referenced in the workspace.

sdf compile --show all

Your output should look like:

Working set 1 model file, 1 .sdf file   Compiling (./models/main.sql)    Finished 1 model [1 succeeded] in 0.788 secs

Schema ┌─────────────┬──────────────────┬────────────┬─────────────┐ │ column_name ┆ data_type        ┆ classifier ┆ description │ ╞═════════════╪══════════════════╪════════════╪═════════════╡ │ message     ┆ varchar not null ┆            ┆             │ └─────────────┴──────────────────┴────────────┴─────────────┘

As you can see from the output, SDF has statically analyzed the query and determined there’s a single non-nullable column named column and it’s of type varchar. You’ll also see an empty classifier block in the output. This is for metadata we’ll attach to columns, but we’ll get to that later.


Execution with `sdf run`

Next, let’s execute the query, using SDF as the database. We’ll execute the query with SDF’s integrated execution runtime, right on your machine.

sdf run --show all

Your output should look like:

Working set 1 model file, 1 .sdf file     Running (./models/main.sql)    Finished 1 model [1 succeeded] in 0.733 secs

Table ┌──────────────┐ │ message      │ ╞══════════════╡ │ Hello World! │ └──────────────┘ 1 rows.


Create a New Model

Now, create a new file in the source directory called main2.sql with the query:


Lineage with `sdf lineage`

SDF guarantees rich column level lineage. The command below specifies a particular column, in a particular table that we would like to inspect.

sdf lineage main2 --column message

Working set 2 model files, 1 .sdf file   Compiling (./models/main.sql)   Compiling (./models/main2.sql)    Finished 2 models [2 succeeded] in 0.779 secs main2.message │ │ copy └──────┐        main.message

In this guide we showed you just how easy it is to install SDF and run your first query.

If using VSCode, SDF’s YML schema is available for type and syntax checking via the Red HAT YAML. This will add auto-fill, type checking, and YML validation directly inline while editing sdf.yml files.