June 30th, 2024

Better support for local sources, performance improvements, and table & column descriptions propagate to Snowflake

SDF Now Supports Snowflake Stage Path Syntax

Workspace Level Credentials

Documentation

General bug fixes and stability improvements:

Install

Github

Slack

Get a Demo

Database

Cloud

Releases

Reference

Get Started

Support

This document contains release notes from version 0.3.3 of SDF.

V0.3.3

Welcome to SDF

We're glad to see you here. Today we're going to install SDF and create a hello world project. Let's get started.

Getting Started

SDF is a static analysis tool for data warehouses written in Rust. No JVM required

Installation

SDF is a work in progress. The data ecosystem contains many tools, databases, and more to integrate with.

Features Matrix

SDF offers a rich open source library ecosystem, as well as open source components that power the SDF CLI and platform.

Open Source

Explore SDF's series of tutorials and get to know our tool a little better. 

Overview

Creating a model

The part we like the least, yet spend the most time on. SDF is here to help!

Debugging

Deprecating a model

Learn more

SDF Cloud is an integrated, auto-generated data catalog with integrated column level lineage, data classification, and more.

Billing

CLI Authentication

Cloud Deployment

SDF supports signing in with Single Sign-On (SSO) providers through the OAuth2 and OpenID Connect protocols.

SSO Support

Common issues and solutions for SDF Cloud compilation and authentication failures.

Troubleshooting

Oganization roles and permissions with SDF Cloud.

Organization Roles

Integrating with GitHub

Adding Workspace Credentials

Configuring Environment Variables and Secrets

Cloud Reports

Display and IO

In this document, we're going to discuss a critical part of the SDF ecosystem - the workspace.

Workspaces

Integrations are the fundamental connection point between your data warehouse and SDF. Credentials are used by integrations to authenticate to your data warehouse.

Integrations

SDF supports materializing tables in Snowflake and Redshift directly from the command line.

Materialization

SDF Environments were built for teams to help them collaborate, isolate, and maintain development, staging, and production environments.

Environments

This documents aims to layout how SDF's powerful lineage capabilities  creates detailed and accessible visibility into your data warehouse.

Lineage

SDF promotes reasoning on higher level types through a rich data classification & type system. 

Data Classifiers

Integrate SDF into more complex workflows with pre and post hooks

Custom Scripts

Expand SDF Built-in Libraries with custom logic

Custom Libraries

This document aims to outline the concept of indexing within the SDF CLI and outline a specific example to better articulate how to take advantage of this powerful feature.

Workspace Indexing

SDF allows you to author user defined functions using function signatures in sdf.yml blocks. Types are used to statically analyze SQL, provide label propagation and lineage computation. 

User Defined Functions

Understanding Telemetry

SDF provides several options for configuring log output. Furthermore, the SDF DB can be used to query and analyze logs.

Logging

SDF supports three types of integrations: databases (i.e. data warehouses), data source, and metadata sources.

SDF as a best-in-class transformation layer for Snowflake

Getting Started with Snowflake and SDF

Materialize tables and views in Snowflake with SDF.

Basic Materialization

Materialize incremental models in Snowflake to save time and compute.

Incremental Materialization

Materialize snapshot models in Snowflake.

Snapshots

Seeds

SDF as a best-in-class transformation and authoring layer for Dagster Orchestration

Getting Started with Dagster and SDF

SDF as a best-in-class transformation layer for BigQuery

Getting Started with BigQuery and SDF

Materialize tables and views in BigQuery with SDF.

SDF can work alongside an existent DBT project to power column-level lineage, checks, and data classification / governance for DBT models.

Integrating with DBT

SDF is best used as an entire transformation layer in itself, effectively replacing DBT and adding all the benefits of SDF.

Migrating from DBT

SDF is a powerful tool on its own, but its utility is amplified when integrated into CI/CD workflows.

GitHub

Install SDF's Spark Listener in your Databricks Cluster

Databricks Spark Listener

This document contains the help content for the sdf command-line program.

SDF CLI Reference

SDF YML Schema

SDF Information Schema

Submit a Support Request

SDF Error Codes

Account Usage Table Functions

Aggregate Functions

Bitwise Expression Functions

Conditional Expression Functions

Context Functions

Conversion Functions

Data Generation Functions

Date & Time Functions

Encryption Functions

Geospatial Functions

Hash Functions

Information Schema

Metadata Functions

Numeric Functions

Semi-Structured and Structured Data Functions

String & Binary Functions

String Functions

System Functions

Table Functions

Vector Similarity Functions

Window Functions

Math Functions

Other Expressions

Temporal Functions

Array Functions

Binary Functions

Bitwise Functions

Color Functions

Comparison Functions

Conditional Functions

Datetime Functions

Hyperloglog Functions

Json Functions

Lambda Functions

Map Functions

Ml Functions

Mongodb Functions

Qdigest Functions

Regexp Functions

Session Functions

Setdigest Functions

T-digest Functions

Teradata Functions

Url Functions

Uuid Functions

Approximate Aggregate Functions

Bit Functions

Date Functions

Debugging Functions

Geography Functions

Interval Functions

JSON Functions

Security Functions

Statistical Aggregation Functions

Time Functions

Timestamp Functions

Utility Functions

Fast, dependency aware in memory query execution built into the same binary as SDF, with the same SQL syntax as Trino and AWS Athena. Powered by Apache DataFusion.

SDF DB Overview

Use advanced partition capabilities to maximize performance.

Partitioning

Overview of Function Execution

Benchmarks

Latest Release

Workspace edition 1.2 brings new features, improvements to the SDF CLI, and a major cleanup of the workspace configuration file. This guide will help you migrate your workspace from edition 1.1 to 1.2.

1.1 ➡️ 1.2

Workspace edition 1.3 brings a new information schema, improvements to the SDF CLI, and an expansion of what was previously the `provider` configuration. This guide will help you migrate your workspace from edition 1.2 to 1.3.

Releases

​June 30th, 2024

​Better support for local sources, performance improvements, and table & column descriptions propagate to Snowflake

​SDF Now Supports Snowflake Stage Path Syntax

​Workspace Level Credentials

​Documentation