hash_counts

intersection_cardinality

jaccard_index

make_set_digest

merge_set_digest

Install

Github

Slack

Get a Demo

Database

Cloud

Linter

Releases

Reference

Get Started

Support

Setdigest Functions

Welcome to SDF

We're glad to see you here. Today we're going to install SDF and create a hello world project. Let's get started.

Getting Started

SDF is a data development framework for building scalable data warehouses written in Rust. No JVM required

Installation

SDF is a work in progress. The data ecosystem contains many tools, databases, and more to integrate with.

Features Matrix

SDF offers a rich open source library ecosystem, as well as open source components that power the SDF CLI and platform.

Open Source

Introduction

Formatting

Integrating jinja macros into your linting configuration

Macros (jinja)

All SDF Lint Rules, usage, and configuration

Explore SDF's series of tutorials and get to know our tool a little better. 

Overview

Creating a model

The part we like the least, yet spend the most time on. SDF is here to help!

Debugging

Deprecating a model

Learn more

SDF Cloud is an integrated, auto-generated data catalog with integrated column level lineage, data classification, and more.

Billing

CLI Authentication

Cloud Deployment

SDF supports signing in with Single Sign-On (SSO) providers through the OAuth2 and OpenID Connect protocols.

SSO Support

Common issues and solutions for SDF Cloud compilation and authentication failures.

Troubleshooting

Oganization roles and permissions with SDF Cloud.

Organization Roles

Integrating with GitHub

Adding Workspace Credentials

Configuring Environment Variables and Secrets

Cloud Reports

Display and IO

In this document, we're going to discuss a critical part of the SDF ecosystem - the workspace.

Workspaces

Integrations are the fundamental connection point between your data warehouse and SDF. Credentials are used by integrations to authenticate to your data warehouse.

Integrations

SDF supports materializing tables in Snowflake and Redshift directly from the command line.

Materialization

SDF Environments were built for teams to help them collaborate, isolate, and maintain development, staging, and production environments.

Environments

Learn how to run pipelines in production while preserving data quality with SDF build

Build & Deploy

This documents aims to layout how SDF's powerful lineage capabilities  creates detailed and accessible visibility into your data warehouse.

Lineage

SDF promotes reasoning on higher level types through a rich data classification & type system. 

Data Classifiers

Integrate SDF into more complex workflows with pre and post hooks

Custom Scripts

Expand SDF Built-in Libraries with custom logic

Custom Libraries

This document aims to outline the concept of indexing within the SDF CLI and outline a specific example to better articulate how to take advantage of this powerful feature.

Workspace Indexing

SDF allows you to author user defined functions using function signatures in sdf.yml blocks. Types are used to statically analyze SQL, provide label propagation and lineage computation.

User Defined Functions

Understanding Telemetry

SDF provides several options for configuring log output. Furthermore, the SDF DB can be used to query and analyze logs.

Logging

Compile your SDF workspace offline without running any queries against your database.

Local Compilation

SDF supports three types of integrations: databases (i.e. data warehouses), data source, and metadata sources.

SDF as a best-in-class transformation layer for Snowflake

Getting Started with Snowflake and SDF

Materialize tables and views in Snowflake with SDF.

Basic Materialization

Materialize incremental models in Snowflake to save time and compute.

Incremental Materialization

Materialize snapshot models in Snowflake.

Snapshots

Seeds

This guide covers advanced features and configurations for Snowflake integrations in SDF, like role configuration, case-preserving identifiers, and more.

Advanced

Advanced Snowflake Features & Configurations

SDF as a best-in-class transformation and authoring layer for Dagster Orchestration

Getting Started with Dagster and SDF

SDF as a best-in-class transformation layer for BigQuery

Getting Started with BigQuery and SDF

Materialize tables and views in BigQuery with SDF.

Materialize incremental models in BigQuery to save time and compute.

SDF can work alongside an existent DBT project to power column-level lineage, checks, and data classification / governance for DBT models.

Integrating with DBT

SDF is best used as an entire transformation layer in itself, effectively replacing DBT and adding all the benefits of SDF.

Migrating from DBT

SDF is a powerful tool on its own, but its utility is amplified when integrated into CI/CD workflows.

GitHub

Install SDF's Spark Listener in your Databricks Cluster

Databricks Spark Listener

This document contains the help content for the sdf command-line program.

SDF CLI Reference

SDF YML Schema

SDF Information Schema

Submit a Support Request

SDF Error Codes

Account Usage Table Functions

Aggregate Functions

Bitwise Expression Functions

Conditional Expression Functions

Context Functions

Conversion Functions

Data Generation Functions

Date & Time Functions

Encryption Functions

Geospatial Functions

Hash Functions

Information Schema

Metadata Functions

Numeric Functions

Semi-Structured and Structured Data Functions

String & Binary Functions

String Functions

System Functions

Table Functions

Vector Similarity Functions

Window Functions

Math Functions

Other Expressions

Temporal Functions

Array Functions

Binary Functions

Bitwise Functions

Color Functions

Comparison Functions

Conditional Functions

Datetime Functions

Hyperloglog Functions

Json Functions

Lambda Functions

Map Functions

Ml Functions

Mongodb Functions

Qdigest Functions

Regexp Functions

Session Functions

T-digest Functions

Teradata Functions

Url Functions

Uuid Functions

Approximate Aggregate Functions

Bit Functions

Date Functions

Debugging Functions

Geography Functions

Interval Functions

JSON Functions

Security Functions

Statistical Aggregation Functions

Time Functions

Timestamp Functions

Utility Functions

Fast, dependency aware in memory query execution built into the same binary as SDF, with the same SQL syntax as Trino and AWS Athena. Powered by Apache DataFusion.

SDF DB Overview

Use advanced partition capabilities to maximize performance.

Partitioning

Overview of Function Execution

Benchmarks

Latest Release

Workspace edition 1.2 brings new features, improvements to the SDF CLI, and a major cleanup of the workspace configuration file. This guide will help you migrate your workspace from edition 1.1 to 1.2.

1.1 ➡️ 1.2

Workspace edition 1.3 brings a new information schema, improvements to the SDF CLI, and an expansion of what was previously the `provider` configuration. This guide will help you migrate your workspace from edition 1.2 to 1.3.

Reference

SQL Functions

Setdigest Functions

`hash_counts`

`intersection_cardinality`

`jaccard_index`

`make_set_digest`

`merge_set_digest`

Reference

SQL Functions

​hash_counts

​intersection_cardinality

​jaccard_index

​make_set_digest

​merge_set_digest

`hash_counts`

`intersection_cardinality`

`jaccard_index`

`make_set_digest`

`merge_set_digest`