Workspace edition 1.3 brings a new information schema, improvements to the SDF CLI, and an expansion of what was previously the provider
configuration. This guide will help you migrate your workspace from edition 1.2 to 1.3.
0.3.0
and above, workspace 1.3
is required. As a first step, please update your workspace block to the following:
providers
, then had confusing methods for configuring data lakes and data catalogs.
Now, all of these external relationships are managed in a single, unified integrations
block.
This replaces the providers
block in the workspace configuration.
If you have a workspace with a provider block like this:
type
field is now required for all integrations. This is to differentiate between database, data lake, and data catalog integrations.compute
property was required in the defaults
block to tell SDF where to run the query, i.e. local
or remote
.
This is now inferred from the integrations block, rendering it unnecessary. It should now be removed from the defaults
block.
CONTAINS_ARRAY_VARCHAR
.
For a full reference on the new information schema, see the information schema documentation.
.sdfcache
directory has been renamed to sdftarget
. This is where SDF will store all of its metadata and cache files.
This shouldn’t break anything as is, but if you have any scripts or processes that rely on the .sdfcache
directory, you’ll need to update them to use sdftarget
instead.
Furthermore, .gitignore
files should be updated to ignore sdftarget
instead of .sdfcache
.
trino
syntaxtrino
to be used as an alias for trino
for the workspace’s dialect
property.
To see all accepted dialects, see the dialect documentation.
seeded: true
-> cycle-cut-point: true
seed
models and breaking cycles with the seeded
property, we’d renamed this field to cycle-cut-point
. This now explicitly marks this table as first table to be processed in a cycle.