# SDF ## Docs - [CLI Authentication](https://docs.sdf.com/cloud/authentication.md) - [Billing](https://docs.sdf.com/cloud/billing.md) - [Adding Workspace Credentials](https://docs.sdf.com/cloud/credentials.md) - [Cloud Deployment](https://docs.sdf.com/cloud/deployment.md) - [Configuring Environment Variables and Secrets](https://docs.sdf.com/cloud/environments.md) - [Integrating with GitHub](https://docs.sdf.com/cloud/github_integration.md) - [Overview](https://docs.sdf.com/cloud/introduction.md): SDF Cloud is an integrated, auto-generated data catalog with integrated column level lineage, data classification, and more. - [Organization Roles](https://docs.sdf.com/cloud/organization_roles.md): Oganization roles and permissions with SDF Cloud. - [Cloud Reports](https://docs.sdf.com/cloud/reporting.md) - [SSO Support](https://docs.sdf.com/cloud/sso.md): SDF supports signing in with Single Sign-On (SSO) providers through the OAuth2 and OpenID Connect protocols. - [Troubleshooting](https://docs.sdf.com/cloud/troubleshooting.md): Common issues and solutions for SDF Cloud compilation and authentication failures. - [Benchmarks](https://docs.sdf.com/database/benchmarks.md): Monitoring SDF's performance over time - [File formats](https://docs.sdf.com/database/file-formats.md) - [SDF DB Overview](https://docs.sdf.com/database/introduction.md): Fast, dependency aware in memory query execution built into the same binary as SDF, with the same SQL syntax as Trino and AWS Athena. Powered by Apache DataFusion. - [Partitioning](https://docs.sdf.com/database/partitioning.md): Use advanced partition capabilities to maximize performance. - [Overview of Function Execution](https://docs.sdf.com/database/supported-functions/overview.md) - [Aggregate Functions](https://docs.sdf.com/database/supported-functions/trino/aggregate-functions.md) - [Array Functions](https://docs.sdf.com/database/supported-functions/trino/array-functions.md) - [Binary Functions](https://docs.sdf.com/database/supported-functions/trino/binary-functions.md) - [Comparison Functions](https://docs.sdf.com/database/supported-functions/trino/comparison-functions.md) - [Conditional Functions](https://docs.sdf.com/database/supported-functions/trino/conditional-functions.md) - [Datetime Functions](https://docs.sdf.com/database/supported-functions/trino/datetime-functions.md) - [Math Functions](https://docs.sdf.com/database/supported-functions/trino/math-functions.md) - [Regexp Functions](https://docs.sdf.com/database/supported-functions/trino/regexp-functions.md) - [String Functions](https://docs.sdf.com/database/supported-functions/trino/string-functions.md) - [Custom Libraries](https://docs.sdf.com/guide/advanced/custom_libs.md): Expand SDF Built-in Libraries with custom logic - [Custom Scripts](https://docs.sdf.com/guide/advanced/custom_scripts.md): Integrate SDF into more complex workflows with pre and post hooks - [Workspace Indexing](https://docs.sdf.com/guide/advanced/index.md): This document aims to outline the concept of indexing within the SDF CLI and outline a specific example to better articulate how to take advantage of this powerful feature. - [Local Compilation](https://docs.sdf.com/guide/advanced/local_compilation.md): Compile your SDF workspace offline without running any queries against your database. - [Logging](https://docs.sdf.com/guide/advanced/logging.md): SDF provides several options for configuring log output. Furthermore, the SDF DB can be used to query and analyze logs. - [Understanding Telemetry](https://docs.sdf.com/guide/advanced/telemetry.md) - [User Defined Functions](https://docs.sdf.com/guide/advanced/udf.md): SDF allows you to author user defined functions using function signatures in sdf.yml blocks. Types are used to statically analyze SQL, provide label propagation and lineage computation. - [Build & Deploy](https://docs.sdf.com/guide/basics/build_and_deployment.md): Learn how to run pipelines in production while preserving data quality with SDF build - [Data Classifiers](https://docs.sdf.com/guide/basics/classifiers.md): SDF promotes reasoning on higher level types through a rich data classification & type system. - [Lineage](https://docs.sdf.com/guide/basics/lineage_metadata.md): This documents aims to layout how SDF's powerful lineage capabilities creates detailed and accessible visibility into your data warehouse. - [Checks](https://docs.sdf.com/guide/data-quality/checks.md) - [Data Quality Overview](https://docs.sdf.com/guide/data-quality/overview.md): SDF supports both static (compile time) and dynamic (run time) data quality tools out of the box. - [Reports](https://docs.sdf.com/guide/data-quality/reports.md) - [Tests](https://docs.sdf.com/guide/data-quality/tests.md) - [Intro to Jinja](https://docs.sdf.com/guide/macro-processing/intro-to-jinja.md) - [Jinja in SDF](https://docs.sdf.com/guide/macro-processing/jinja.md): Working with Jinja in a Scalable Way - [Jinja Variables](https://docs.sdf.com/guide/macro-processing/jinja-variables.md): Work with Variables in Jinja to parameterize your SQL and YML files - [Macro Processing Overview](https://docs.sdf.com/guide/macro-processing/overview.md): SDF supports several macro processing features out of the box that enable compilation of queries containing Jinja and SQL Variables. - [SQL Variables](https://docs.sdf.com/guide/macro-processing/sdf-variables.md): Simple variables and parameterization in SDF - [Environments](https://docs.sdf.com/guide/setup/environments.md): SDF Environments were built for teams to help them collaborate, isolate, and maintain development, staging, and production environments. - [Integrations](https://docs.sdf.com/guide/setup/integrations.md): Integrations are the fundamental connection point between your data warehouse and SDF. Credentials are used by integrations to authenticate to your data warehouse. - [Display and IO](https://docs.sdf.com/guide/setup/io.md) - [Materialization](https://docs.sdf.com/guide/setup/materialization.md): SDF supports materializing tables in Snowflake and Redshift directly from the command line. - [Workspaces](https://docs.sdf.com/guide/setup/workspaces.md): In this document, we're going to discuss a critical part of the SDF ecosystem - the workspace. - [Getting Started](https://docs.sdf.com/integrations/aws/S3/getting-started.md): SDF supports reading data and metadata from S3, and writing it the results back. - [Executing on Data from S3](https://docs.sdf.com/integrations/aws/S3/s3-example.md): SDF DB can execute queries on data stored in S3 parquet, CSV, JSON, gzip, and more - [Getting Started with Redshift and SDF](https://docs.sdf.com/integrations/aws/redshift/getting-started.md): SDF as a best-in-class transformation layer for Redshift - [Basic Materialization](https://docs.sdf.com/integrations/bigquery/basic-materialization.md): Materialize tables and views in BigQuery with SDF. - [Getting Started with BigQuery and SDF](https://docs.sdf.com/integrations/bigquery/getting-started.md): SDF as a best-in-class transformation layer for BigQuery - [Incremental Materialization](https://docs.sdf.com/integrations/bigquery/incremental-materialization.md): Materialize incremental models in BigQuery to save time and compute. - [Seeds](https://docs.sdf.com/integrations/bigquery/seeds.md): Upload data files as tables - [Snapshots](https://docs.sdf.com/integrations/bigquery/snapshots.md): Materialize snapshot models in BigQuery. - [GitHub](https://docs.sdf.com/integrations/cicd/ci_cd.md): SDF is a powerful tool on its own, but its utility is amplified when integrated into CI/CD workflows. - [Getting Started with Dagster and SDF](https://docs.sdf.com/integrations/dagster/getting-started.md): SDF as a best-in-class transformation and authoring layer for Dagster Orchestration - [Databricks Spark Listener](https://docs.sdf.com/integrations/databricks/databricks.md): Install SDF's Spark Listener in your Databricks Cluster - [Integrating with DBT](https://docs.sdf.com/integrations/dbt/integrating.md): SDF can work alongside an existent DBT project to power column-level lineage, checks, and data classification / governance for DBT models. - [Migrating from DBT](https://docs.sdf.com/integrations/dbt/migrating.md): SDF is best used as an entire transformation layer in itself, effectively replacing DBT and adding all the benefits of SDF. - [Overview](https://docs.sdf.com/integrations/overview.md): SDF supports three types of integrations: databases (i.e. data warehouses), data source, and metadata sources. - [Advanced Snowflake Features & Configurations](https://docs.sdf.com/integrations/snowflake/advanced.md): This guide covers advanced features and configurations for Snowflake integrations in SDF, like role configuration, case-preserving identifiers, and more. - [Basic Materialization](https://docs.sdf.com/integrations/snowflake/basic-materialization.md): Materialize tables and views in Snowflake with SDF. - [Getting Started with Snowflake and SDF](https://docs.sdf.com/integrations/snowflake/getting-started.md): SDF as a best-in-class transformation layer for Snowflake - [Incremental Materialization](https://docs.sdf.com/integrations/snowflake/incremental-materialization.md): Materialize incremental models in Snowflake to save time and compute. - [Seeds](https://docs.sdf.com/integrations/snowflake/seeds.md): Upload data files as tables - [Snapshots](https://docs.sdf.com/integrations/snowflake/snapshots.md): Materialize snapshot models in Snowflake. - [Features Matrix](https://docs.sdf.com/introduction/features.md): SDF is a work in progress. The data ecosystem contains many tools, databases, and more to integrate with. - [Getting Started](https://docs.sdf.com/introduction/getting-started.md): We're glad to see you here. Today we're going to install SDF and create a hello world project. Let's get started. - [Open Source](https://docs.sdf.com/introduction/open-source.md): SDF offers a rich open source library ecosystem, as well as open source components that power the SDF CLI and platform. - [Welcome to SDF](https://docs.sdf.com/introduction/welcome.md) - [Aggregate Functions](https://docs.sdf.com/reference/bigquery/aggregate_functions.md) - [Approximate Aggregate Functions](https://docs.sdf.com/reference/bigquery/approximate_aggregate_functions.md) - [Array Functions](https://docs.sdf.com/reference/bigquery/array_functions.md) - [Bit Functions](https://docs.sdf.com/reference/bigquery/bit_functions.md) - [Date Functions](https://docs.sdf.com/reference/bigquery/date_functions.md) - [Datetime Functions](https://docs.sdf.com/reference/bigquery/datetime_functions.md) - [Debugging Functions](https://docs.sdf.com/reference/bigquery/debugging_functions.md) - [Geography Functions](https://docs.sdf.com/reference/bigquery/geography_functions.md) - [Hash Functions](https://docs.sdf.com/reference/bigquery/hash_functions.md) - [Interval Functions](https://docs.sdf.com/reference/bigquery/interval_functions.md) - [JSON Functions](https://docs.sdf.com/reference/bigquery/json_functions.md) - [Math Functions](https://docs.sdf.com/reference/bigquery/math_functions.md) - [Other Expressions](https://docs.sdf.com/reference/bigquery/other_expressions.md) - [Interval Functions](https://docs.sdf.com/reference/bigquery/search_functions.md) - [Security Functions](https://docs.sdf.com/reference/bigquery/security_functions.md) - [Statistical Aggregation Functions](https://docs.sdf.com/reference/bigquery/statistical_aggregate_functions.md) - [String Functions](https://docs.sdf.com/reference/bigquery/string_functions.md) - [Temporal Functions](https://docs.sdf.com/reference/bigquery/temporal_functions.md) - [Time Functions](https://docs.sdf.com/reference/bigquery/time_functions.md) - [Timestamp Functions](https://docs.sdf.com/reference/bigquery/timestamp_functions.md) - [Utility Functions](https://docs.sdf.com/reference/bigquery/utility_functions.md) - [SDF Error Codes](https://docs.sdf.com/reference/error-codes.md) - [Aggregate Functions](https://docs.sdf.com/reference/redshift/aggregate_functions.md) - [Math Functions](https://docs.sdf.com/reference/redshift/math_functions.md) - [Other Expressions](https://docs.sdf.com/reference/redshift/other_expressions.md) - [String Functions](https://docs.sdf.com/reference/redshift/string_functions.md) - [Temporal Functions](https://docs.sdf.com/reference/redshift/temporal_functions.md) - [SDF CLI Reference](https://docs.sdf.com/reference/sdf-cli.md): This document contains the help content for the sdf command-line program. - [SDF Information Schema](https://docs.sdf.com/reference/sdf-information-schema.md) - [SDF YML Schema](https://docs.sdf.com/reference/sdf-yml.md) - [Account Usage Table Functions](https://docs.sdf.com/reference/snowflake/account_usage_table_functions.md) - [Aggregate Functions](https://docs.sdf.com/reference/snowflake/aggregate_functions.md) - [Bitwise Expression Functions](https://docs.sdf.com/reference/snowflake/bitwise_expression_functions.md) - [Conditional Expression Functions](https://docs.sdf.com/reference/snowflake/conditional_expression_functions.md) - [Context Functions](https://docs.sdf.com/reference/snowflake/context_functions.md) - [Conversion Functions](https://docs.sdf.com/reference/snowflake/conversion_functions.md) - [Data Generation Functions](https://docs.sdf.com/reference/snowflake/data_generation_functions.md) - [Date & Time Functions](https://docs.sdf.com/reference/snowflake/date_and_time_functions.md) - [Encryption Functions](https://docs.sdf.com/reference/snowflake/encryption_functions.md) - [Geospatial Functions](https://docs.sdf.com/reference/snowflake/geospatial_functions.md) - [Hash Functions](https://docs.sdf.com/reference/snowflake/hash_functions.md) - [Information Schema](https://docs.sdf.com/reference/snowflake/information_schema.md) - [Metadata Functions](https://docs.sdf.com/reference/snowflake/metadata_functions.md) - [Numeric Functions](https://docs.sdf.com/reference/snowflake/numeric_functions.md) - [Semi-Structured and Structured Data Functions](https://docs.sdf.com/reference/snowflake/semi-structured_and_structured_data_functions.md) - [String & Binary Functions](https://docs.sdf.com/reference/snowflake/string_and_binary_functions.md) - [String Functions](https://docs.sdf.com/reference/snowflake/string_functions.md) - [System Functions](https://docs.sdf.com/reference/snowflake/system_functions.md) - [Table Functions](https://docs.sdf.com/reference/snowflake/table_functions.md) - [Vector Similarity Functions](https://docs.sdf.com/reference/snowflake/vector_similarity_functions.md) - [Window Functions](https://docs.sdf.com/reference/snowflake/window_functions.md) - [Submit a Support Request](https://docs.sdf.com/reference/support.md): Submitting Support Requests. - [Aggregate Functions](https://docs.sdf.com/reference/trino/aggregate_functions.md) - [Array Functions](https://docs.sdf.com/reference/trino/array_functions.md) - [Binary Functions](https://docs.sdf.com/reference/trino/binary_functions.md) - [Bitwise Functions](https://docs.sdf.com/reference/trino/bitwise_functions.md) - [Color Functions](https://docs.sdf.com/reference/trino/color_functions.md) - [Comparison Functions](https://docs.sdf.com/reference/trino/comparison_functions.md) - [Conditional Functions](https://docs.sdf.com/reference/trino/conditional_functions.md) - [Conversion Functions](https://docs.sdf.com/reference/trino/conversion_functions.md) - [Datetime Functions](https://docs.sdf.com/reference/trino/datetime_functions.md) - [Geospatial Functions](https://docs.sdf.com/reference/trino/geospatial_functions.md) - [Hyperloglog Functions](https://docs.sdf.com/reference/trino/hyperloglog_functions.md) - [Json Functions](https://docs.sdf.com/reference/trino/json_functions.md) - [Lambda Functions](https://docs.sdf.com/reference/trino/lambda_functions.md) - [Map Functions](https://docs.sdf.com/reference/trino/map_functions.md) - [Math Functions](https://docs.sdf.com/reference/trino/math_functions.md) - [Ml Functions](https://docs.sdf.com/reference/trino/ml_functions.md) - [Mongodb Functions](https://docs.sdf.com/reference/trino/mongodb_functions.md) - [Qdigest Functions](https://docs.sdf.com/reference/trino/qdigest_functions.md) - [Regexp Functions](https://docs.sdf.com/reference/trino/regexp_functions.md) - [All Supported Functions](https://docs.sdf.com/reference/trino/sdf_execution_support/all_functions.md) - [Session Functions](https://docs.sdf.com/reference/trino/session_functions.md) - [Setdigest Functions](https://docs.sdf.com/reference/trino/setdigest_functions.md) - [String Functions](https://docs.sdf.com/reference/trino/string_functions.md) - [T-digest Functions](https://docs.sdf.com/reference/trino/t-digest_functions.md) - [Teradata Functions](https://docs.sdf.com/reference/trino/teradata_functions.md) - [Url Functions](https://docs.sdf.com/reference/trino/url_functions.md) - [Uuid Functions](https://docs.sdf.com/reference/trino/uuid_functions.md) - [Window Functions](https://docs.sdf.com/reference/trino/window_functions.md) - [Latest Release](https://docs.sdf.com/releases/latest.md) - [1.1 ➡️ 1.2](https://docs.sdf.com/releases/migrations/11-12.md): Workspace edition 1.2 brings new features, improvements to the SDF CLI, and a major cleanup of the workspace configuration file. This guide will help you migrate your workspace from edition 1.1 to 1.2. - [1.2 ➡️ 1.3](https://docs.sdf.com/releases/migrations/12-13.md): Workspace edition 1.3 brings a new information schema, improvements to the SDF CLI, and an expansion of what was previously the `provider` configuration. This guide will help you migrate your workspace from edition 1.2 to 1.3. - [Creating a model](https://docs.sdf.com/tutorials/creating-a-model.md): Our first model in SDF, that's exciting! - [Debugging](https://docs.sdf.com/tutorials/debugging.md): The part we like the least, yet spend the most time on. SDF is here to help! - [Deprecating a model](https://docs.sdf.com/tutorials/deprecating-a-model.md) - [Enriching your warehouse](https://docs.sdf.com/tutorials/enriching-your-warehouse.md) - [Ensuring data quality](https://docs.sdf.com/tutorials/ensuring-data-quality.md) - [Learn more](https://docs.sdf.com/tutorials/learn-more.md) - [Overview](https://docs.sdf.com/tutorials/tutorials-intro.md): Explore SDF's series of tutorials and get to know our tool a little better. ## Optional - [Github](https://github.com/sdf-labs) - [Slack](https://sdf.com/join) - [Get a Demo](https://sdf.com/book-demo)