11## Table templates
22######(macros/tables)
33
4- These macros form the core of the package and can be called in your models to build the tables for your Data Vault.
4+ These macros form the core of the package and can be called in your models to build the different types of tables needed
5+ for your Data Vault.
56
7+ ### Metadata notes
68#### Using a source reference for the target metadata
79
8- As of release 0.3, you may now use a reference as a target metadata value, to streamline metadata entry.
10+ !!! note
11+ As of release 0.3, you may now use a source reference as a target metadata value, to streamline metadata entry.
12+ Read below!
913
10- In the usage examples for the table template macros in this section, you will see ``` source ``` provided as the values for some
11- of the target metadata variables. ``` source ``` has been declared as a variable at the top of the models, and holds a
12- reference to the source table we are loading from. This is shorthand for retaining the name and data types of the columns as they were
13- provided in the ``` src ``` variables. You may wish to alias the columns or change their data types in specific
14- circumstances, which is possible by providing a triple in the form of a list.
14+ In the usage examples for the table template macros in this section, you will see ``` source ``` provided as the values
15+ for some of the target metadata variables. ``` source ``` has been declared as a variable at the top of the models,
16+ and holds a reference to the source table we are loading from. This is shorthand for retaining the name and data types
17+ of the columns as they are provided in the ``` src ``` variables. You may wish to alias the columns or change their data
18+ types in specific circumstances, which is possible by providing an additional parameter as a list of triples:
19+ ``` (source column name, data type to cast to, target column name) ``` .
1520
1621Both approaches are shown in the snippet below:
1722
1823``` mysql
24+ {%- set src_pk = ' CUSTOMER_NATION_PK' - %}
25+ {%- set src_fk = [' CUSTOMER_PK' ' NATION_PK' - %}
26+ {%- ...other src metadata... - %}
27+
1928{%- set tgt_pk = source - %}
2029{%- set tgt_fk = [[' CUSTOMER_PK' ' BINARY(16)' ' CUSTOMER_FK'
2130 [' NATION_PK' ' BINARY(16)' ' NATION_FK' - %}
2231```
2332
33+ Here, we are keeping the ``` tgt_pk ``` (the target table's primary key) the same as the primary key identified in the
34+ source (``` src_pk ``` ).
35+ Behind the scenes, the macro will get the datatype of the column provided in the ``` src_pk ``` variable and generate a
36+ mapping for us. If the ``` src_pk ``` column does not exist, an appropriate exception will be raised.
37+
38+ Alternatively we have provided a manual mapping for the ``` tgt_fk ``` (the target table's foreign key).
39+
40+ * For further details and examples on both methods, refer to the usage examples
41+ and snippets in the table template documentation below (both Single-Source and Union).*
2442
2543!!! note
2644 If only aliasing and ** not** changing data types, we suggest using the [ add_columns] ( #add_columns ) macro.
3048
3149### hub_template
3250
33- Creates a hub with provided metadata.
51+ Generates sql to build a hub table using the provided metadata.
3452
3553``` mysql
3654dbtvault .hub_template (src_pk, src_nk, src_ldts, src_source,
152170
153171### link_template
154172
155- Creates a link with provided metadata.
173+ Generates sql to build a link table using the provided metadata.
156174
157175``` mysql
158176dbtvault .link_template (src_pk, src_fk, src_ldts, src_source,
@@ -228,7 +246,6 @@ dbtvault.link_template(src_pk, src_fk, src_ldts, src_source,
228246 source) }}
229247```
230248
231-
232249#### Output
233250
234251``` mysql tab="Single-Source"
280297
281298### sat_template
282299
283- Creates a satellite with provided metadata.
300+ Generates sql to build a satellite table using the provided metadata.
284301
285302``` mysql
286303dbtvault .sat_template (src_pk, src_hashdiff, src_payload,
398415 [ Read More] ( https://www.md5online.org/blog/why-md5-is-not-safe/ )
399416
400417!!! seealso "See Also"
401- [ hash] ( #hash )
418+ - [ hash] ( #hash )
419+ - [ Hashing best practises and why we hash] ( bestpractices.md#hashing )
402420
403- A macro for generating multiple lines of hashing SQL for columns:
421+ This macro will generate SQL hashing sequences for one or more columns as below :
404422``` sql
405423CAST(MD5_BINARY(UPPER (TRIM (CAST(column1 AS VARCHAR )))) AS BINARY(16 )) AS alias1,
406424CAST(MD5_BINARY(UPPER (TRIM (CAST(column2 AS VARCHAR )))) AS BINARY(16 )) AS alias2
@@ -437,9 +455,9 @@ CAST(MD5_BINARY(CONCAT(
437455```
438456
439457!!! success "Column sorting"
440- You do not need to worry about providing the columns in any particular order, as long as you set the
441- ``` sort ``` flag to true when creating hashdiffs.
442-
458+ If you wish to sort columns in alphabetical order as per [ best practices ] ( bestpractices.md#hashing ) ,
459+ you do not need to worry about doing this manually, just set the
460+ ``` sort ``` flag to true when creating hashdiffs as per the above example.
443461___
444462
445463### add_columns
@@ -483,16 +501,21 @@ OLD_CUSTOMER_PK AS CUSTOMER_PK
483501The ``` add_columns ``` macro will automatically select all columns from the optional ``` source_table ``` reference,
484502if provided.
485503
486- ##### Overring source columns
504+ ##### Overriding source columns
487505
488506You may wish to override some of the source columns with different values. To replace the ``` SOURCE ```
489507or ``` LOADDATE ``` column value, for example, then you must provide the column name
490508that you wish to override as the alias in the pair.
491509
510+ !!! note
511+ The macro will not actually override (delete or replace) any of the source columns, but simply add new columns
512+ using the provided column as a basis.
513+
492514##### Functions
493515
494- Database functions may be used, for example ``` CURRENT_DATE() ``` , to set the current date as the value of a column, as on
495- ``` line 2 ``` of the usage example.
516+ Database functions may be used, for example ``` CURRENT_DATE() ``` to set the current date as the value of a column, as on
517+ ``` line 2 ``` of the usage example. Any function supported by the database is valid, for example ``` LPAD() ``` , which pads
518+ a column with leading zeroes.
496519
497520##### Adding constants
498521With the ``` add_columns ``` macro, you may provide constants.
@@ -502,9 +525,11 @@ and the macro will do the rest. See ```line 3``` of the usage example above, and
502525
503526##### Aliasing columns
504527
505- As of release 0.3, columns must now be aliased prior to loading, in the staging layer. This can be done by providing the
528+ As of release 0.3, columns should now be aliased in the staging layer prior to loading . This can be achieved by providing the
506529column name you wish to alias as the first argument in a pair, and providing the alias for that column as the second argument.
507- This process can be observed on ``` line 4 ``` of the usage example above.
530+ This can be observed on ``` line 4 ``` of the usage example above. Aliasing can still be carried out using a
531+ manual mapping (shown in the [ table template] ( #table-templates ) section examples) but this is less concise for aliasing
532+ purposes.
508533
509534___
510535
@@ -517,7 +542,7 @@ FROM MYDATABASE.MYSCHEMA.MYTABLE
517542```
518543
519544!!! info
520- Sources need to be set up in dbt. [ Read More] ( https://docs.getdbt.com/docs/using-sources )
545+ Sources need to be set up in dbt to ensure this works . [ Read More] ( https://docs.getdbt.com/docs/using-sources )
521546
522547#### Parameters
523548
@@ -604,14 +629,19 @@ ___
604629 The intended use is for creating checksum-like fields only, so that a record change can be detected.
605630
606631 [ Read More] ( https://www.md5online.org/blog/why-md5-is-not-safe/ )
632+
633+ !!! seealso "See Also"
634+ - [ multi-hash] ( #multi_hash )
635+ - [ Hashing best practises and why we hash] ( bestpractices.md#hashing )
607636
608637A macro for generating hashing SQL for columns:
609638``` sql
610639CAST(MD5_BINARY(UPPER (TRIM (CAST(column AS VARCHAR )))) AS BINARY(16 )) AS alias
611640```
612641
613642- Can provide multiple columns as a list to create a concatenated hash
614- - Hashdiffs should be alpha sorted using the ``` sort ``` flag.
643+ - Columns are sorted alphabetically (by alias) if you set the ``` sort ``` flag to true.
644+ - Generally, you should alpha sort hashdiffs using the ``` sort ``` flag.
615645- Casts a column as ``` VARCHAR ``` , transforms to ``` UPPER ``` case and trims whitespace
616646- ``` '^^' ``` Accounts for null values with a double caret
617647- ``` '||' ``` Concatenates with a double pipe
0 commit comments