SDF outputs valuable context to stdout by default, but rich trace logs are also available for debugging and monitoring.

The following guide will go through the different logging options available in SDF and how to use them. See the reference for details on log formats, levels, and properties.

​
Guide

1

Open an SDF workspace

Let’s use our moms_flower_shop_completed sample workspace as a base for this guide. If you don’t have it, you can create it by running:

sdf new --sample moms_flower_shop_completed && cd moms_flower_shop_completed
2

Explore Log Levels

By default, SDF logs progress, warnings, and errors. We can filter this down by specifying --log-level warn or --log-level error to only see warnings or errors, respectively.

To see more in-depth logs, we can use the --log-level info flag. This will expose logs that capture the internal steps that sdf follows to compile and run a workspace.

Let’s say we want to view info logs along with error and warn logs, we can run:

sdf compile moms_flower_shop.raw.raw_inapp_events --log-level info --show none

{“_ts”:“2024-09-17T23:43:44.583902459+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/main.rs:101”,“ev”:“sdf”}
{“_ts”:“2024-09-17T23:43:44.583953113+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/main.rs:102”,“ev”:“sdf”,“set”:“job_id”,“val_s”:“309ae330-e0fb-4ae0-b16e-b56dba7b8309”}
{“_ts”:“2024-09-17T23:43:44.583967750+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/main.rs:103”,“ev”:“sdf”,“set”:“args”,“val_as”:[“compile”,“moms_flower_shop.raw.raw_inapp_events”,“—log-level”,“info”,“—show”,“none”]}
{“_ts”:“2024-09-17T23:43:44.585182806+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:130”,“ev”:“initialize”}
{“_ts”:“2024-09-17T23:43:44.585345450+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:275”,“ev”:“initialize.load_workspace”}
{“_ts”:“2024-09-17T23:43:44.603414096+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:367”,“ev”:“initialize.load_workspace”,“st_dur_ms”:18}
{“_ts”:“2024-09-17T23:43:44.603436217+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:369”,“ev”:“initialize.load_dependencies”}
{“_ts”:“2024-09-17T23:43:44.603450253+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:501”,“ev”:“initialize.load_dependencies”,“st_dur_ms”:0}
{“_ts”:“2024-09-17T23:43:44.603455493+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:503”,“ev”:“initialize.load_prelude”}
{“_ts”:“2024-09-17T23:43:44.606387400+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/utils/initializing.rs:586”,“ev”:“initialize.load_prelude”,“st_dur_ms”:2}

We used --show none to avoid displaying stdout output. The --show flag controls the standard out, things like “compiling <table>”, and is separate from info, debug, and trace logs.

Nice! We’ve lifted up the hood on SDF and are seeing info logs in action.

debug and trace logs are also available, and are more detailed than info logs. Let’s try trace, the most detailed log level:

sdf compile moms_flower_shop.raw.raw_inapp_events --log-level trace --show none

{“_ts”:“2024-09-17T23:43:46.219914804+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:805”,“ev”:“cmd.prepare.read_makefile”,“set”:“verdict”,“val_s”:“full-rebuild”}
{“_ts”:“2024-09-17T23:43:46.219922098+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:874”,“ev”:“cmd.prepare.read_makefile”,“set”:“analyze_with_sources”,“val_b”:false}
{“_ts”:“2024-09-17T23:43:46.219931526+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:879”,“ev”:“cmd.prepare.read_makefile”,“set”:“resolved_table_names”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.219938198+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:885”,“ev”:“cmd.prepare.read_makefile”,“set”:“seen_these_files”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.219944700+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:891”,“ev”:“cmd.prepare.read_makefile”,“set”:“sources”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.219949128+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:901”,“ev”:“cmd.prepare.read_makefile”,“st_dur_ms”:0}
{“_ts”:“2024-09-17T23:43:46.227984287+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:1904”,“ev”:“cmd.schedule.dag”}
{“_ts”:“2024-09-17T23:43:46.228025754+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:1914”,“ev”:“cmd.schedule.dag”,“set”:“dag.initial”,“val_s”:“{&quot;moms_flower_shop.raw.raw_inapp_events&quot;:[]}”}
{“_ts”:“2024-09-17T23:43:46.228038117+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:1926”,“ev”:“cmd.schedule.dag”,“set”:“dag.cleaned”,“val_s”:“{&quot;moms_flower_shop.raw.raw_inapp_events&quot;:[]}”}
{“_ts”:“2024-09-17T23:43:46.228044529+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:1932”,“ev”:“cmd.schedule.dag”,“set”:“missing_defs”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.228058225+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2010”,“ev”:“cmd.schedule.dag”,“set”:“dag.resolved_targets”,“val_s”:“[&quot;moms_flower_shop.raw.raw_inapp_events&quot;]”}
{“_ts”:“2024-09-17T23:43:46.228071770+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2021”,“ev”:“cmd.schedule.dag”,“set”:“dag.targeted”,“val_s”:“{&quot;moms_flower_shop.raw.raw_inapp_events&quot;:[]}”}
{“_ts”:“2024-09-17T23:43:46.228109530+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2036”,“ev”:“cmd.schedule.dag”,“set”:“dag.acyclic”,“val_s”:“{&quot;moms_flower_shop.raw.raw_inapp_events&quot;:[]}”}
{“_ts”:“2024-09-17T23:43:46.228119459+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2042”,“ev”:“cmd.schedule.dag”,“set”:“cycles”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.228125781+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2048”,“ev”:“cmd.schedule.dag”,“set”:“cut_points”,“val_s”:”[]”}
{“_ts”:“2024-09-17T23:43:46.228134016+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2054”,“ev”:“cmd.schedule.dag”,“set”:“order.initial”,“val_s”:“[&quot;moms_flower_shop.raw.raw_inapp_events&quot;]”}
{“_ts”:“2024-09-17T23:43:46.228172979+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2497”,“ev”:“cmd.schedule.dag”,“set”:“order.without_providers”,“val_s”:“[&quot;moms_flower_shop.raw.raw_inapp_events&quot;]”}
{“_ts”:“2024-09-17T23:43:46.228182376+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2523”,“ev”:“cmd.schedule.dag”,“st_dur_ms”:0}
{“_ts”:“2024-09-17T23:43:46.228192665+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2530”,“ev”:“cmd.do”}
{“_ts”:“2024-09-17T23:43:46.228205339+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2556”,“ev”:“cmd.do”,“set”:“order.without_providers_or_cached”,“val_s”:“[&quot;moms_flower_shop.raw.raw_inapp_events&quot;]”}
{“_ts”:“2024-09-17T23:43:46.228213424+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2573”,“ev”:“cmd.do.sources”}
{“_ts”:“2024-09-17T23:43:46.229783599+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:3536”,“ev”:“cmd.do.derived.wave_1.mat”,“ev_tb”:“moms_flower_shop.raw.raw_inapp_events”,“ev_tb_catalog”:“moms_flower_shop”,“ev_tb_schema”:“raw”,“ev_tb_table”:“raw_inapp_events”}
{“_ts”:“2024-09-17T23:43:46.230655405+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/describe.rs:289”,“ev”:“cmd.do.derived.wave_1.meta_analysis”,“ev_tb”:“moms_flower_shop.raw.raw_inapp_events”,“ev_tb_catalog”:“moms_flower_shop”,“ev_tb_schema”:“raw”,“ev_tb_table”:“raw_inapp_events”}
{“_ts”:“2024-09-17T23:43:46.230710087+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/subcommands/describe.rs:327”,“ev”:“cmd.do.derived.wave_1.meta_analysis”,“ev_tb”:“moms_flower_shop.raw.raw_inapp_events”,“ev_tb_catalog”:“moms_flower_shop”,“ev_tb_schema”:“raw”,“ev_tb_table”:“raw_inapp_events”,“st_dur_ms”:0}
{“_ts”:“2024-09-17T23:43:46.230738500+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:3638”,“ev”:“cmd.do.derived.wave_1.mat”,“ev_tb”:“moms_flower_shop.raw.raw_inapp_events”,“ev_tb_catalog”:“moms_flower_shop”,“ev_tb_purpose”:“model”,“ev_tb_schema”:“raw”,“ev_tb_table”:“raw_inapp_events”,“st_code”:“succeeded”,“st_done”:“executed”,“st_dur_ms”:0}
{“_ts”:“2024-09-17T23:43:46.231519369+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:2998”,“ev”:“cmd.do.gen_makefile”}
{“_ts”:“2024-09-17T23:43:46.231731034+00:00”,“_ll”:“TRACE”,“_fn”:“crates/sdf-cli/src/utils/dep_cache.rs:2150”,“ev”:“written_makefile”,“set”:“makefile”,“val_m”:{“contents”:{“env_vars”:“d751713988987e9331980363e24189ce”,“cli_vars”:“99914b932bd37a50b983c5e7c90ae93b”,“credentials”:“99914b932bd37a50b983c5e7c90ae93b”},“prelude”:[{“file”:“workspace.sdf.yml”,“mtime”:“2024-09-17T23:43:44.265181933Z”},{“file”:“metadata/staging/inapp_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/raw/raw_addresses.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/raw/raw_marketing_campaign_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/staging/app_installs.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/raw/raw_customers.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/analytics/dim_marketing_campaigns.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/analytics/agg_installs_and_campaigns.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/raw/raw_inapp_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“classifications/column_classifiers.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“classifications/table_classifiers.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/staging/inapp_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/raw/raw_addresses.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/raw/raw_marketing_campaign_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/staging/app_installs.sdf.yml”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“metadata/raw/raw_customers.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/analytics/dim_marketing_campaigns.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/analytics/agg_installs_and_campaigns.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“metadata/raw/raw_inapp_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“sdftarget/dbg/.lib/sdf_test/workspace.sdf.yml”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/workspace.sdf.yml”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_mat/workspace.sdf.yml”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_test/macros/test.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/macros/generate_integer_values.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/macros/group_by.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/macros/generate_surrogate_key.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/macros/date_spine.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_utils/macros/generate_date_values.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”},{“file”:“sdftarget/dbg/.lib/sdf_mat/macros/dialect/trino/materialize.jinja”,“mtime”:“2024-09-17T23:43:45.743182522Z”}],“deps”:[{“target”:“moms_flower_shop.raw.raw_inapp_events”,“deps”:[]}],“source_files”:{“values”:{“moms_flower_shop.raw.raw_inapp_events”:[{“file”:“metadata/raw/raw_inapp_events.sdf.yml”,“mtime”:“2024-09-17T23:43:44.263181932Z”},{“file”:“models/raw/raw_inapp_events.sql”,“mtime”:“2024-09-17T23:43:44.264181933Z”},{“file”:“seeds/parquet/inapp_events.parquet”,“mtime”:“2024-09-17T23:43:44.265181933Z”}]}},“incomplete_source_files”:{“values”:[]},“target_status”:{“values”:{“moms_flower_shop.raw.raw_inapp_events”:{“purpose”:“model”,“status”:“succeeded”}}}}}
{“_ts”:“2024-09-17T23:43:46.231791587+00:00”,“_ll”:“DEBUG”,“_fn”:“crates/sdf-cli/src/utils/dep_cache.rs:2150”,“ev”:“written_makefile”,“message”:“Exceeds limit; use log-level trace”,“set”:“makefile”}
{“_ts”:“2024-09-17T23:43:46.231880543+00:00”,“_ll”:“INFO”,“_fn”:“crates/sdf-cli/src/subcommands/build.rs:3012”,“ev”:“cmd.do.gen_makefile”,“st_dur_ms”:0}

Notice that the trace logs include info, debug, and trace logs. Furthermore, the trace log is very very long. Trace logs are typically used for internal use cases, such as debugging SDF itself. As such, we recommend using info or debug logs for most use cases.

3

Writing Logs to Files

By default, SDF logs to stdout. However, we can write logs to a file by specifying the --log-file flag.

Let’s write the logs to a file named sdf.log:

sdf compile moms_flower_shop.raw.raw_inapp_events --log-level info --log-file sdf.log --show none

This command will produce no output, but if we inspect the new file sdf.log, we’ll see the logs output to the file.

The logs outputted to the file respect the --log-level flag you’ve set.

4

Bonus: Analyze the Logs with SDF DB

As mentioned earlier, SDF’s in-memory database can be used to query and analyze logs, right from the terminal on your machine.

Let’s start by gathering up all the info logs from a fresh compile.

First, let’s clean the workspace cache:

sdf clean

Now, let’s compile the workspace and output the logs to a file called logs/compile_info.json:

sdf compile moms_flower_shop.raw.raw_inapp_events --log-level info --log-file logs/compile_info.json --show none

Now that we have the logs written to a file, let’s include them in our SDF workspace by adding the following to the includes block in your workspace.sdf.yml:

workspace:
...
  includes:
    - path: logs
      type: seed

Finally, let’s compile our new logs model to ensure SDF can process it:

sdf compile moms_flower_shop.pub.compile_info

Working set 12 model files, 1 seed file, 19 .sdf files
   Finished 1 model [1 succeeded] in 0.826 secs
Schema moms_flower_shop.pub.compile_info
┌───────────────┬────────────────┬────────────┬─────────────┐
│ column_name   ┆ data_type      ┆ classifier ┆ description │
╞═══════════════╪════════════════╪════════════╪═════════════╡
│ _ts           ┆ varchar        ┆            ┆             │
│ _ll           ┆ varchar        ┆            ┆             │
│ _fn           ┆ varchar        ┆            ┆             │
│ ev            ┆ varchar        ┆            ┆             │
│ set           ┆ varchar        ┆            ┆             │
│ val_s         ┆ varchar        ┆            ┆             │
│ val_as        ┆ array<varchar> ┆            ┆             │
│ st_dur_ms     ┆ bigint         ┆            ┆             │
│ ev_fn         ┆ varchar        ┆            ┆             │
│ st_code       ┆ varchar        ┆            ┆             │
│ ev_tb         ┆ varchar        ┆            ┆             │
│ ev_tb_catalog ┆ varchar        ┆            ┆             │
│ ev_tb_schema  ┆ varchar        ┆            ┆             │
│ ev_tb_table   ┆ varchar        ┆            ┆             │
│ ev_tb_purpose ┆ varchar        ┆            ┆             │
│ st_done       ┆ varchar        ┆            ┆             │
│ err_msg       ┆ varchar        ┆            ┆             │
│ exit_code     ┆ bigint         ┆            ┆             │
└───────────────┴────────────────┴────────────┴─────────────┘

Awesome! SDF can process the logs model. Let’s run a quick inline query to select some properties from the logs:

sdf run -q 'select "_ts" as timestamp, "ev" as event, "st_dur_ms" as duration_in_ms from compile_info'

Working set 13 model files, 1 seed file, 19 .sdf files
    Running moms_flower_shop.pub.query (./sdftarget/dbg/query.sql)
    Warning moms_flower_shop.pub.query attempting to execute Trino, local execution is currently experimental
   Finished 2 models [2 succeeded] in 0.807 secs
Table moms_flower_shop.pub.query
┌─────────────────────────────────────┬──────────────────────────────┬────────────────┐
│ timestamp                           ┆ event                        ┆ duration_in_ms │
╞═════════════════════════════════════╪══════════════════════════════╪════════════════╡
│ 2024-09-17T23:43:47.704589039+00:00 ┆ sdf                          ┆                │
│ 2024-09-17T23:43:47.704717248+00:00 ┆ sdf                          ┆                │
│ 2024-09-17T23:43:47.704804040+00:00 ┆ sdf                          ┆                │
│ 2024-09-17T23:43:47.705794824+00:00 ┆ initialize                   ┆                │
│ 2024-09-17T23:43:47.706018461+00:00 ┆ initialize.load_workspace    ┆                │
│ 2024-09-17T23:43:47.723702213+00:00 ┆ initialize.load_workspace    ┆ 17             │
│ 2024-09-17T23:43:47.723757036+00:00 ┆ initialize.load_dependencies ┆                │
│ 2024-09-17T23:43:47.723795948+00:00 ┆ initialize.load_dependencies ┆ 0              │
│ 2024-09-17T23:43:47.723834410+00:00 ┆ initialize.load_prelude      ┆                │
│ 2024-09-17T23:43:47.726823314+00:00 ┆ initialize.load_prelude      ┆ 2              │
└─────────────────────────────────────┴──────────────────────────────┴────────────────┘
58 rows, showing only 10 rows.
  Run with —limit 0 to show all rows.

Nice! We’ve successfully queried the logs using the SDF DB. Any models you create from the logs can be used to analyze and query the logs further. To learn more about the SDF DB, visit our SDF DB Guide.

​
Reference

As a quick reference, let’s review the different log levels, format, and properties within a log:

​
Log Levels

  • info - general information about the execution of the command
  • debug - more detailed information about the execution of the command
  • trace - very detailed information about the execution of the command

​
Log Properties:

  • _ts - the timestamp of the log
  • _lvl - the log level (info, debug, trace, etc.)
  • id - the ID of the variable being logged
  • val_s - the value of a variable being logged if it’s a string
  • val_a* - the value of a variable being logged if it’s an array, the * represents the value type of the array. I.e. as for an array of strings
  • val_m* - the value of a variable being logged if it’s a map, the * represents the value type of the map. I.e. mss for a map of string to string
  • val_b - the value of a variable being logged if it’s a boolean
  • val_i - the value of a variable being logged if it’s an integer
  • ith - the index of the wave being logged
  • num_files - the number of files being processed by an operation
  • file - the filename being processed
  • num_errors - the number of errors encountered during an operation
  • num_statements - the number of statements being processed by an operation
  • num_tables - the number of tables being processed by an operation
  • table - the table being processed by an operation
  • message - a message associated with the log
  • error_code - the error code associated with the log (0 if no error)
  • _src - the source of the log in the Rust codebase
  • ev - the event name corresponding to this log. Used to map open and close events
  • ev_type - the type of the event (see reference below)
  • ev_dur_ms - the duration of the event in milliseconds
  • status - the status of the event (see reference below)

Why all the val_ properties? SDF logs are designed to be queryable by engines that support structured types. As such, we map the Rust variable types to structured types like so:

Rust types
h - hashmap
z - hashset
a - array
v - vector
o - option
t - table_name
s - string
n - number
b - bool

Json (also SQL) Types
m - map
a - array
n - number
b - bool
s - string

So, msas is map of string to array of string

​
Event Types:

  • open - the start of an event
  • close - the end of an event. Use the ev property to map this to the corresponding open event
  • state - a summary of state at a given point in time for an event
  • wave - represents a batch operation operating on a subset of a dag or process. Waves have state, duration, and status properties

​
Status:

  • succeeded - the event was successful
  • failed - the event failed
  • skipped - the event was skipped
Understanding TelemetryOverview