rsrc_static attribute clarification

[kafkaguru]

Hello,

I've been reading the documentation for the rsrc_static attribute and I'm a bit confused.

I understand its purpose, but the explanation to how it's used is unclear to me. The relevant paragraphs are:

Sometimes the record source column inside one staging area is not always the same, but has some dynamic parts in it. Sometimes the rsrc column includes the ldts of each load and could look something like this: ‘SALESFORCE/Partners/2022-01-01T07:00:00’.

Obviously the timestamp part inside that rsrc would change from load to load, and we now need to identify parts of it that will be static over all loads. In the shown example it would be ‘SALESFORCE/Partners’. That expression now needs to be enriched with wildcard expressions to catch all occurrences of the static part of the record source. In BigQuery the wildcard would be ‘’ and therefore the rsrc_static would be ‘/SALESFORCE/Partners/*’.

Should the value be a static string? A column name reference? A path for the value within a column?
I think a full example would be helpful to describe the different scenarios referenced under special cases.

Here's an example to help:

source_sys_name is the rsrc_alias. Value is source_catalog.source_schema.source_table_name'
load_timestamp is the ldts_alias. Value is the time data was written to our database.

sat_sales_order_header

column	datatype
sales_order_hashkey	binary
sales_order_hashdiff	binary
source_sys_name	string
load_timestamp	timestamp

sat_sales_order_status

column	datatype
sales_order_hashkey	binary
sales_order_hashdiff	binary
source_sys_name	string
load_timestamp	timestamp

hub_sales_order

column	datatype
sales_order_hashkey	binary
order_number	int

To configure the hub would I do the following?

{%- set yaml_metadata -%}
hashkey: 'sales_order_hashkey'
business_keys:
  - 'order_number'
source_models:
  - name: stg_sales_order_header
    rsrc_static: 'source_catalog.source_schema.sales_order_header'
  - name: stg_sales_order_status
    rsrc_static: 'source_catalog.source_schema.sales_order_status'
{%- endset -%}

{{ datavault4dbt.hub(yaml_metadata=yaml_metadata) }}

Or would I do:

{%- set yaml_metadata -%}
hashkey: 'sales_order_hashkey'
business_keys:
  - 'order_number'
source_models:
  - name: stg_sales_order_header
    rsrc_static: 'source_sys_name'
  - name: stg_sales_order_status
    rsrc_static: 'source_sys_name'
{%- endset -%}

{{ datavault4dbt.hub(yaml_metadata=yaml_metadata) }}

Or finally:

{%- set yaml_metadata -%}
hashkey: 'sales_order_hashkey'
business_keys:
  - 'order_number'
source_models:
  - name: stg_sales_order_header
    rsrc_static: 'load_timestamp'
  - name: stg_sales_order_status
    rsrc_static: 'load_timestamp'
{%- endset -%}

{{ datavault4dbt.hub(yaml_metadata=yaml_metadata) }}

[tkirschke]

Hi @kafkaguru and thanks for reaching out!

Should the value be a static string? A column name reference? A path for the value within a column?

The value should be a static string that matches the content of your 'source_sys_name' column.

You define the content of your record_source column, or in your case the source_sys_name column inside the stage model.
In here you basically either set your own static string, e.g. SALESFORCE.CUSTOMER, or you use an existing column inside the source data as a record source that holds more specific information. This could e.g. be a column lake_path which holds the explicit path to the json file inside your data lake.

Depending on the actual content of your source_sys_name column, you have to configure the rsrc_static to find results inside a SQL LIKE statement.

So, to help you better, could you provide me sample values that you find inside stg_sales_order_header and stg_sales_order_status in the source_sys_name column?

[kafkaguru]

Sorry, forgot to follow up. Your explanation cleared things up! Thanks!
I'd only suggest that maybe the documentation be a little more clear on how it works exactly. I had to look at the generated sql to figure it out :)