Website

Github

Try SDF

Database

Cloud

Releases

Reference

Get Started

Support

This document contains release notes from version 0.1.192 of SDF.

V0.1.192

Welcome to SDF

We're glad to see you here. Today we're going to install SDF and create a hello world project. Let's get started.

Getting Started

SDF is a work in progress. The data ecosystem contains many tools, databases, and more to integrate with.

Features Matrix

SDF Cloud is an integrated, auto-generated data catalog with integrated column level lineage, data classification, and more.

Overview

Billing

Authentication

Cloud Deployment

Display and IO

In this document, we're going to discuss a critical part of the SDF ecosystem - the workspace.

Workspaces

SDF promotes reasoning on higher level types through a rich data classification & type system. 

Classifiers & SQL Types

This documents aims to layout a very important part of SDF and how it can benefit your data warehouse.

Lineage

Providers are the fundamental connection point between your data warehouse and SDF. Credentials are used by providers to authenticate to your data warehouse.

Providers

SDF supports supports materializing tables in Snowflake and Redshift directly from the command line.

Materialization

SDF Environments were built for teams to help them collaborate, isolate, and maintain development, staging, and production environments.

Environments

SDF supports several macro processing features out of the box that enable compilation of queries containing Jinja and SQL Variables.

Macro Processing Overview

Jinja in SDF

Work with Variables in Jinja to parameterize your SQL and YML files

Jinja Variables

Simple variables and parameterization in SDF

SQL Variables

SDF supports both static (compile time) and dynamic (run time) data quality tools out of the bex.

Data Quality Overview

Checks

Reports

Tests

Stats

This document aims to outline the concept of indexing within the SDF CLI and outline a specific example to better articulate how to take advantage of this powerful feature.

Indexing

SDF allows you to author user defined functions using function signatures in sdf.yml blocks. Types are used to statically analyze SQL, provide label propagation and lineage computation. 

User Defined Functions

Understanding Telemetry

Materialize tables and views in Snowflake with SDF.

Basic Materialization

Materialize incremental models in Snowflake to save time and compute.

Incremental Materialization

SDF can work alongside an existent DBT project to power column-level lineage, checks, and data classification / governance for DBT models.

Integrating with DBT

SDF can replace DBT as your transformation layer of choice. This guide will help you.

Migrating from DBT

SDF is a powerful tool on its own, but its utility is amplified when integrated into CI/CD workflows.

Install SDF's Spark Listener in your Databricks Cluster

Databricks Spark Listener

Use your own OpenAI keypair with SDF to accelerate Data Classification

AI Classification

This document contains the help content for the sdf command-line program.

SDF CLI Reference

SDF YML Schema

SDF Information Schema

All Functions

Aggregate Functions

Math Functions

Other Expressions

String Functions

Temporal Functions

Approximate Aggregate Functions

Array Functions

Bit Functions

Date Functions

Datetime Functions

Debugging Functions

Geography Functions

Hash Functions

Interval Functions

JSON Functions

Security Functions

Statistical Aggregation Functions

Time Functions

Timestamp Functions

Utility Functions

Fast, dependency aware in memory query execution built into the same binary as SDF, with the same SQL syntax as Trino and AWS Athena. Powered by Apache DataFusion.

SDF Database

Latest

This document contains release notes from version 0.2.0 of SDF.

V0.2.0

This document contains release notes from version 0.2.11 of SDF.

V0.2.11

This document contains release notes from version 0.1.184 of SDF.

V0.1.184

This document contains release notes from version 0.1.187 of SDF.

V0.1.187

This document contains release notes from version 0.1.191 of SDF.

V0.1.191

This document contains release notes from version 0.1.136 of SDF.

V0.1.136

This document contains release notes from version 0.1.135 of SDF.

V0.1.135

This document contains release notes from version 0.1.134 of SDF.

V0.1.134

This document contains release notes from version 0.1.90 of SDF.

V0.1.90

This document contains release notes from version 0.1.89 of SDF.

V0.1.89

This document contains release notes from version 0.1.88 of SDF.

V0.1.88

This document contains release notes from version 0.1.87 of SDF.

V0.1.87

This document contains release notes from version 0.1.86 of SDF.

V0.1.86

This document contains release notes from version 0.1.85 of SDF.

V0.1.85

This document contains release notes from version 0.1.84 of SDF.

V0.1.84

This document contains release notes from version 0.1.83 of SDF.

V0.1.83

This document contains release notes from version 0.1.82 of SDF.

V0.1.82

This document contains release notes from version 0.1.81 of SDF.

V0.1.81

This document contains release notes from version 0.1.80 of SDF.

V0.1.80

This document contains release notes from version 0.1.79 of SDF.

V0.1.79

This document contains release notes from version 0.1.78 of SDF.

V0.1.78

This document contains release notes from version 0.1.77 of SDF.

V0.1.77

This document contains release notes from version 0.1.76 of SDF.

V0.1.76

This document contains release notes from version 0.1.75 of SDF.

V0.1.75

This document contains release notes from version 0.1.74 of SDF.

V0.1.74

This document contains release notes from version 0.1.73 of SDF.

V0.1.73

This document contains release notes from version 0.1.72 of SDF.

V0.1.72

This document contains release notes from version 0.1.71 of SDF.

V0.1.71

This document contains release notes from version 0.1.70 of SDF.

V0.1.70

This document contains release notes from version 0.1.69 of SDF.

V0.1.69

This document contains release notes from version 0.1.68 of SDF.

V0.1.68

This document contains release notes from version 0.1.67 of SDF.

V0.1.67

This document contains release notes from version 0.1.66 of SDF.

V0.1.66

This document contains release notes from patched version v0.1.65 of SDF.

V0.1.65

This document contains release notes from patched version v0.1.64 of SDF.

V0.1.64

This document contains release notes from patched version v0.1.63 of SDF.

V0.1.63

This document contains release notes from patched version v0.1.62 of SDF.

V0.1.62

This document contains release notes from patched version v0.1.61 of SDF.

V0.1.61

This document contains release notes from patched version v0.1.60 of SDF.

V0.1.60

This document contains release notes from patched version v0.1.59 of SDF.

V0.1.59

This document contains release notes from patched version v0.1.58 of SDF.

V0.1.58

This document contains release notes from patched version v0.1.57 of SDF.

V0.1.57

This document contains release notes from patched version v0.1.56 of SDF.

V0.1.56

This document contains release notes from patched version v0.1.55 of SDF.

V0.1.55

This document contains release notes from patched version v0.1.54 of SDF.

V0.1.54

This document contains release notes from patched version v0.1.53 of SDF.

V0.1.53

This document contains release notes from patched version v0.1.52 of SDF.

V0.1.52

This document contains release notes from patched version v0.1.51 of SDF.

V0.1.51

This document contains release notes from patched version v0.1.50 of SDF.

V0.1.50

This document contains release notes from patched version v0.1.49 of SDF.

V0.1.49

This document contains release notes from patched version v0.1.48 of SDF.

V0.1.48

This document contains release notes from patched version v0.1.47 of SDF.

V0.1.47

This document contains release notes from patched version v0.1.46 of SDF.

V0.1.46

This document contains release notes from patched version v0.1.45 of SDF.

Architecture	Status	Version	Download
Linux Intel X86-64	✅	0.1.192	Download
Linux Arm ARM-64	✅	0.1.192	Download
Apple Intel X86-64	✅	0.1.192	Download
Apple Arm AARCh-64	✅	0.1.192	Download

Releases

V0.1.192

April 5, 2024

New Macro Processing Docs:

General bug fixes and stability improvements:

Latest Builds

Releases

​April 5, 2024

​New Macro Processing Docs:

​General bug fixes and stability improvements:

​Latest Builds

April 5, 2024

New Macro Processing Docs:

General bug fixes and stability improvements:

Latest Builds