Skip to content

Commit 6a929dc

Browse files
DVAlexHiggsDVAlexHiggs
committed
Release 0.11.0
2 parents f3d29cd + de32631 commit 6a929dc

File tree

119 files changed

+727
-1005
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

119 files changed

+727
-1005
lines changed

‎NOTICE

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
AutomateDV (f.k.a dbtvault)
2-
Copyright 2019-2023 Business Thinking Ltd (Trading as Datavault).
2+
Copyright 2019-2024 Business Thinking Ltd (Trading as Datavault).
33

44
Licensed under the Apache License, Version 2.0 (the "License");
55
you may not use this file except in compliance with the License.

‎dbt_project.yml

+8-8
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
name: automate_dv
2-
version: 0.10.2
3-
require-dbt-version: [">=1.0.0", "<2.0.0"]
2+
version: 0.11.0
3+
require-dbt-version: [ ">=1.0.0", "<2.0.0" ]
44
config-version: 2
55

6-
model-paths: ["models"]
7-
analysis-paths: ["analyses"]
8-
test-paths: ["tests"]
9-
seed-paths: ["seeds"]
10-
macro-paths: ["macros"]
11-
docs-paths: ["docs"]
6+
model-paths: [ "models" ]
7+
analysis-paths: [ "analyses" ]
8+
test-paths: [ "tests" ]
9+
seed-paths: [ "seeds" ]
10+
macro-paths: [ "macros" ]
11+
docs-paths: [ "docs" ]
1212

1313
target-path: "target"
1414
clean-targets:

‎macros/internal/helpers/dateadd.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/is_checks.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/materialisations/error_messages.sql ‎macros/internal/helpers/logging/error_messages.sql

+32-42
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,9 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

66

7-
{% macro wrap_warning(warning_message) %}
8-
9-
{%- set new_message = [] -%}
10-
{%- set length_list = [] -%}
11-
12-
{%- for ln in warning_message.split('\n') -%}
13-
{%- do new_message.append((ln | trim)) -%}
14-
{%- do length_list.append((ln | length)) -%}
15-
{%- endfor -%}
16-
17-
{%- set max_line_length = length_list | max -%}
18-
{%- set padding_length = (max_line_length - 7) // 2 -%}
19-
20-
{%- set border = modules.itertools.repeat('=', padding_length) | join ('') ~ 'WARNING' ~ modules.itertools.repeat('=', padding_length) | join ('') -%}
21-
22-
{%- set wrapped_message = '\n' ~ border ~ '\n' ~ new_message | join('\n') ~ '\n' ~ border -%}
23-
24-
{%- do return(wrapped_message) -%}
25-
26-
{% endmacro %}
27-
28-
297
{%- macro datepart_too_small_error(period) -%}
308

319
{%- set message -%}
@@ -35,10 +13,7 @@
3513
Please see: https://automate-dv.readthedocs.io/en/latest/materialisations/
3614
{%- endset -%}
3715

38-
{%- if execute -%}
39-
{{- exceptions.raise_compiler_error(message) -}}
40-
{%- endif -%}
41-
16+
{%- do automate_dv.log_error(message) -%}
4217
{%- endmacro -%}
4318

4419

@@ -51,10 +26,7 @@
5126
Please see: https://automate-dv.readthedocs.io/en/latest/materialisations/
5227
{%- endset -%}
5328

54-
{%- if execute -%}
55-
{{- exceptions.warn(automate_dv.wrap_warning(message)) -}}
56-
{%- endif -%}
57-
29+
{%- do automate_dv.log_warning(message) -%}
5830
{%- endmacro -%}
5931

6032

@@ -67,10 +39,7 @@
6739
Please see: https://automate-dv.readthedocs.io/en/latest/materialisations/
6840
{%- endset %}
6941

70-
{%- if execute -%}
71-
{{- exceptions.raise_compiler_error(message) -}}
72-
{%- endif -%}
73-
42+
{%- do automate_dv.log_warning(message) -%}
7443
{%- endmacro -%}
7544

7645

@@ -82,10 +51,7 @@
8251
Using this functionality in a live production setting may result in unpredictable outcomes, data loss, or system instability.
8352
{%- endset -%}
8453

85-
{%- if execute -%}
86-
{{- exceptions.warn(automate_dv.wrap_warning(message)) -}}
87-
{%- endif -%}
88-
54+
{%- do automate_dv.log_warning(message) -%}
8955
{%- endmacro -%}
9056

9157

@@ -99,8 +65,32 @@
9965
We are actively working to get this fixed. Thank you for your understanding.
10066
{%- endset -%}
10167

102-
{%- if execute -%}
103-
{{- exceptions.raise_compiler_error(automate_dv.wrap_warning(message)) -}}
104-
{%- endif -%}
68+
{%- do automate_dv.log_error(message) -%}
69+
{%- endmacro -%}
70+
71+
72+
{%- macro materialisation_deprecation_warning() -%}
73+
74+
{%- set message -%}
75+
DEPRECATED: Since AutomateDV v0.11.0, vault_insert_by_x materialisations are now deprecated.
76+
These materialisation were initially designed to provide an option for rapid iterative development of
77+
incremental loading patterns in local environments for development and testing, allowing users to bypass
78+
the need for a comprehensive PSA or delta-loading solution. They are being deprecated to encourage the use
79+
of more robust solutions.
80+
{%- endset -%}
81+
82+
{%- do automate_dv.log_warning(message) -%}
83+
{%- endmacro -%}
84+
85+
86+
{%- macro pit_bridge_deprecation_warning() -%}
87+
88+
{%- set message -%}
89+
DEPRECATED: Since AutomateDV v0.11.0, the pit() and bridge() macros are now deprecated.
90+
This is because they are not currently fit-for-purpose and need significant usability
91+
and peformance improvements, as well as a design overhaul.
92+
Improved implementations will be released in a future version of AutomateDV.
93+
{%- endset -%}
10594

95+
{%- do automate_dv.log_warning(message) -%}
10696
{%- endmacro -%}

‎macros/internal/helpers/logging/log_relation_sources.sql

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

@@ -9,7 +9,7 @@
99

1010
{% macro default__log_relation_sources(relation, source_count) %}
1111

12-
{%- if 'docs' not in invocation_args_dict['rpc_method'] and execute -%}
12+
{%- if execute and automate_dv.is_something(invocation_args_dict.get('which')) and invocation_args_dict.get('which') != 'docs' -%}
1313

1414
{%- do dbt_utils.log_info('Loading {} from {} source(s)'.format("{}.{}.{}".format(relation.database, relation.schema, relation.identifier),
1515
source_count)) -%}
@@ -18,7 +18,7 @@
1818

1919
{% macro databricks__log_relation_sources(relation, source_count) %}
2020

21-
{%- if 'docs' not in invocation_args_dict['rpc_method'] and execute -%}
21+
{%- if execute and automate_dv.is_something(invocation_args_dict.get('which')) and invocation_args_dict.get('which') != 'docs' -%}
2222

2323
{%- do dbt_utils.log_info('Loading {} from {} source(s)'.format("{}.{}".format(relation.schema, relation.identifier),
2424
source_count)) -%}

‎macros/internal/helpers/logging/logging_helpers.sql

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
/*
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
3+
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
4+
*/
5+
6+
7+
{% macro wrap_message(message, type='WARNING') %}
8+
9+
{%- set new_message = [] -%}
10+
{%- set length_list = [] -%}
11+
12+
{%- for ln in message.split('\n') -%}
13+
{%- do new_message.append((ln | trim)) -%}
14+
{%- do length_list.append((ln | length)) -%}
15+
{%- endfor -%}
16+
17+
{%- set max_line_length = length_list | max -%}
18+
{%- set padding_length = (max_line_length - 7) // 2 -%}
19+
20+
{%- set border = modules.itertools.repeat('=', padding_length) | join ('') ~ (type | upper) ~ modules.itertools.repeat('=', padding_length) | join ('') -%}
21+
22+
{%- set wrapped_message = '\n' ~ border ~ '\n' ~ new_message | join('\n') ~ '\n' ~ border -%}
23+
24+
{%- do return(wrapped_message) -%}
25+
26+
{% endmacro %}
27+
28+
29+
{%- macro log_warning(message) -%}
30+
31+
{%- set message = automate_dv.wrap_message(message) -%}
32+
33+
{%- if execute and automate_dv.is_something(invocation_args_dict.get('which')) and invocation_args_dict.get('which') != 'compile' -%}
34+
{%- do exceptions.warn(message) -%}
35+
{%- endif -%}
36+
37+
{%- endmacro -%}
38+
39+
40+
{%- macro log_error(message) -%}
41+
42+
{%- set message = automate_dv.wrap_message(message, type='ERROR') -%}
43+
44+
{%- if execute and automate_dv.is_something(invocation_args_dict.get('which')) and invocation_args_dict.get('which') != 'compile' -%}
45+
{%- do exceptions.raise_compiler_error(message) -%}
46+
{%- endif -%}
47+
48+
{%- endmacro -%}

‎macros/internal/helpers/prepend_generated_by.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/extract_column_names.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/extract_null_column_names.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/print_list.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/process_columns_to_escape.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/process_columns_to_select.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/stage_processing_macros/process_hash_column_excludes.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/helpers/timestamp_add.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/alias.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/alias_all.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/as_constant.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/check_required_parameters.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/concat_ws.sql

+19-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

@@ -11,6 +11,16 @@
1111

1212
{%- macro default__concat_ws(string_list, separator="||") -%}
1313

14+
CONCAT_WS('{{ separator }}',
15+
{%- for str_obj in string_list %}
16+
{{ str_obj }}{% if not loop.last %},{% endif %}
17+
{%- endfor %}
18+
)
19+
20+
{%- endmacro -%}
21+
22+
{%- macro bigquery__concat_ws(string_list, separator="||") -%}
23+
1424
CONCAT(
1525
{%- for str in string_list %}
1626
{{ str }}
@@ -20,13 +30,19 @@ CONCAT(
2030

2131
{%- endmacro -%}
2232

23-
{%- macro bigquery__concat_ws(string_list, separator="||") -%}
33+
{%- macro sqlserver__concat_ws(string_list, separator="||") -%}
2434

2535
{{ automate_dv.default__concat_ws(string_list=string_list, separator=separator) }}
2636

2737
{%- endmacro -%}
2838

29-
{%- macro sqlserver__concat_ws(string_list, separator="||") -%}
39+
{%- macro postgres__concat_ws(string_list, separator="||") -%}
40+
41+
{{ automate_dv.default__concat_ws(string_list=string_list, separator=separator) }}
42+
43+
{%- endmacro -%}
44+
45+
{%- macro databricks__concat_ws(string_list, separator="||") -%}
3046

3147
{{ automate_dv.default__concat_ws(string_list=string_list, separator=separator) }}
3248

‎macros/internal/metadata_processing/escape_column_name.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/escape_column_names.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/expand_column_list.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/get_escape_characters.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

‎macros/internal/metadata_processing/multikey.sql

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) Business Thinking Ltd. 2019-2023
2+
* Copyright (c) Business Thinking Ltd. 2019-2024
33
* This software includes code developed by the AutomateDV (f.k.a dbtvault) Team at Business Thinking Ltd. Trading as Datavault
44
*/
55

0 commit comments

Comments
 (0)