Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.sdf.com/llms.txt

Use this file to discover all available pages before exploring further.

SDF supports execution of arbitrary scripts which can be executed via the sdf exec command. In this way, you may string multiple commmands together, sequentially. scripts is a reserved name keyword in the config yml block. Each property in scripts will be an independently executable command. To execute any script run: sdf exec [SCRIPT_NAME] Keep in mind:
  • If a subcommand exits with an error code, SDF will propagate that error code, and exit with that same error code
  • SDF pipes both stdout and stderr from subcommands
  • Scripts are run via the sh shell. Color coding will not be preserved.

​
Getting Started

To configure a custom script, add a scripts element to the config block. Each element in the properties array will become a separately executable command. 
---
config:
  name: scripts
  description: My custom scripts
  properties:
    my_script: "echo 'hello world'"
    other_script: "echo $USER"
    macro_script: "echo {{env_var('PWD')}}" # needs preprocessor: jinja
You are now able to execute these scripts as:
  • sdf exec my_script
  • sdf exec other_script
  • sdf exec macro_script

​
Reserved Script Keywords: Pre-Compile, Pre-Run, Post-Compile & Post-Run

In addition to generic scripts, SDF has 4 reserved keywords which will execute scripts natively as part of sdf compile and sdf run commands.
  • pre-compile -> When executing sdf compile or sdf check or sdf report, the script in pre-compile is executed first
  • post-compile -> When executing sdf compile or sdf check or sdf report, the script in post-compile is executed after compile has successfully finished
  • pre-run -> When executing sdf run or sdf test or sdf stats, the script in pre-run is executed first
  • post-run -> When executing sdf run or sdf test or sdf stats, the script in post-run is executed after run has successfully finished

​
Example

The configuration below utilizes all 4 keywords. Let’s add it to the default workspace from sdf new 
---
config:
  name: scripts
  description: Below are the 4 reserved keywords for scripts
  properties:
    pre-compile: >
      echo "RUNNING PRE-COMPILE"
    post-compile: >
      echo "RUNNING POST-COMPILE"
    pre-run: >
      echo "RUNNING PRE-RUN"
    post-run: >
      echo "RUNNING POST-RUN"
When executing sdf compile, we now run the pre-compile script first, and then post-compile resulting in the below output. 
> sdf compile
  Working set 1 model file, 1 .sdf file
         Info Executing pre-hook
  RUNNING PRE-COMPILE
    Compiling cfg.pub.main (./models/main.sql)
         Info Executing post-hook
  RUNNING POST-COMPILE
     Finished 1 model [1 succeeded] in 0.270 secs

​
Use Cases

Scripts allow for organization specific workflows
  • Pre-hooks and Post-hooks -> Run custom logic before or after an SDF command
  • Metadata uploading -> Upload SDF’s information schema to a database for further processing.
  • Post processing -> Parse metadata for post-run statistics
  • Validate results -> Run python, js, shell, or any other process to validate that current changes fit with your current data infrastructure.

​
User & Root User

Scripts inherit the user of the calling SDF process. As a result sudo sdf exec ... will run subcommands as sudo as well. See the below script to validate whether an SDF script is root. 
---
config:
  name: scripts
  properties:
    is_root: >
      if [ "$EUID" -ne 0 ]
        then echo "Not running as root"
      else 
        echo "Running as root"
      fi