Getting Started
Getting started with Snowflake and SDF.
Overview
SDF can hydrate table schemas and execute queries against a Snowflake DB. This guide will walk you through the steps of creating a Snowflake integration. By the end, you will be able to reference tables in your snowflake instance in SDF.
SDF will use remote Snowflake schema information to do type bindings and other static analysis checks on your workspace.
Prerequisites
In order to connect to Snowflake, you will need to have on hand the following:
- A Mac or Linux with a valid installation of the latest SDF version running locally.
- A Snowflake account to connect to.
- Valid credentials with read access to materialize tables in your Snowflake account.
Guide
Collect Required Information
To connect to Snowflake, you need the following materials:
- Account ID - the 8 digit organization identifier
- Username - your Snowflake username
- Password - your Snowflake password
- Role - a role with appropriate permissions
- Warehouse - The warehouse you are connecting to.
Log in to your Snowflake account to find the relevant information.
Run sdf auth login snowflake --help
to see all login options.
Connect SDF to Snowflake
Connect SDF to snowflake. SDF will prompt you for your Snowflake password.
sdf auth login snowflake --account-id <ACCOUNT_ID> --username <USERNAME> --role <ROLE> --warehouse <WAREHOUSE_NAME>
This will create a new credential in a ~/.sdf/
directory in the root of your system. This credential will be used to authenticate with AWS services. By default, the
credential’s name is default. As such, the credential does not need to be explicitly referenced in the provider configuration below.
Add Snowflake Provider in Workspace
Once authenticated, add a table provider block in your workspace.sdf.yml
. This tells SDF to use Snowflake to hydrate missing table schemas.
workspace:
...
providers:
- type: snowflake
sources:
- pattern: <DATABASE>.*.*
Replace <DATABASE>
with the name of the database you want to hydrate. Note this is configurable and can be changed to any database you have access to. For example, if I wanted SDF to pull from two databases, called db1
and db2
, I would write:
workspace:
...
providers:
- type: snowflake
sources:
- pattern: db1.*.*
- pattern: db2.*.*
Providers can also be configured to use specific credentials by referencing the credential name in the provider block. For example, if you wanted to use a credential called snowflake-creds
, you would write:
workspace:
...
providers:
- type: snowflake
credential: snowflake-creds
sources:
- pattern: db1.schema1.mytable
- pattern: db2.schema2.*
For more information on provider configuration, see the provider documentation.
Try it out!
Now that you’re connected, let’s make sure SDF can pull the schema information it needs.
Run sdf compile -q "select * from <DATABASE>.<SCHEMA>.<TABLE>" --show all
If the connection is successful, you will see the schema for the table you selected.
The term ‘DATABASE’ in Snowflake is interchangeable with the term ‘CATALOG’ in SDF.
Next Steps
Now that you have connected to Snowflake, you can start materializing tables in Snowflake. Check out the Snowflake materialization guide to get started.