Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

possibility to use templated properties to apply by default on file update #38

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,42 @@ dbt-invoke properties.delete <options>
- `<options>` uses the same arguments as for creating/updating property
files, except for `--threads`.


### Applying default properties using `dbt_invoke_template.yml`

You can pass custom default properties to be added to property files on update.
To do so, add a `dbt_invoke_template.yml` at the root of your dbt project,
using the following convention:


```
model:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be nitpicking a bit here, but I think we should change model, seed, snapshot, and analysis to their plural versions; models, seeds, snapshots, and analyses so that dbt_invoke_template.yml matches the convention used in property files. Thoughts?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you! I think it's better to align to the used convention, I'll update properties.py to use plural versions instead.

meta:
owner: "@default"
robastel marked this conversation as resolved.
Show resolved Hide resolved
columns:
foo: "bar"
robastel marked this conversation as resolved.
Show resolved Hide resolved

seed:
meta:
owner: "@default"
columns:
foo: "bar"

snapshot:
meta:
owner: "@default"
columns:
foo: "bar"

analysis:
meta:
owner: "@default"
columns:
foo: "bar"
```

Note: only `model`, `seed`, `snapshot` and `analysis` are supported.

### Help

- To view the list of available commands and their short descriptions, run:
Expand Down
6 changes: 6 additions & 0 deletions dbt_invoke/internal/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,12 @@ def get_project_info(ctx, project_dir=None):
Path(project_path, macro_path)
for macro_path in project_yml.get('macro-paths', ['macros'])
]
# retrieves template information if provided
template_path = Path(project_path, 'dbt_invoke_template.yml')
if template_path.exists():
ctx.config['template_yml'] = parse_yaml(template_path)
david-whimsical marked this conversation as resolved.
Show resolved Hide resolved
else:
ctx.config['template_yml'] = None
# Set context config key-value pairs
ctx.config['project_path'] = project_path
ctx.config['project_name'] = project_name
Expand Down
35 changes: 34 additions & 1 deletion dbt_invoke/properties.py
Original file line number Diff line number Diff line change
Expand Up @@ -715,6 +715,7 @@ def _create_property_file(
property_path,
resource_dict,
columns,
ctx.config['template_yml']
)
_utils.write_yaml(
property_path,
Expand Down Expand Up @@ -786,7 +787,12 @@ def _get_columns(ctx, resource_location, resource_dict, **kwargs):
return columns


def _structure_property_file_dict(location, resource_dict, columns_list):
def _structure_property_file_dict(
location,
resource_dict,
columns_list,
template_yml
):
"""
Structure a dictionary that will be used to create a property file

Expand All @@ -813,6 +819,12 @@ def _structure_property_file_dict(location, resource_dict, columns_list):
property_file_dict = _get_property_header(resource_name, resource_type)
# Get the sub-dictionaries of each existing column
resource_type_plural = _SUPPORTED_RESOURCE_TYPES[resource_type]
# If dbt_invoke_template.yml exists, adds templated properties to header
# when not already present
if template_yml and resource_type in template_yml:
_apply_template(
property_file_dict[resource_type_plural][0], template_yml[resource_type]
)
existing_columns_dict = {
item['name']: item
for item in property_file_dict[resource_type_plural][0]['columns']
Expand All @@ -825,12 +837,33 @@ def _structure_property_file_dict(location, resource_dict, columns_list):
column_dict = existing_columns_dict.get(
column, _get_property_column(column)
)
# If dbt_invoke_template.yml exists, adds templated properties to column
# when not already present
if template_yml and 'columns' in template_yml.get(resource_type, dict()):
_apply_template(
column_dict, template_yml[resource_type]['columns']
)
property_file_dict[resource_type_plural][0]['columns'].append(
column_dict
)
return property_file_dict


def _apply_template(property_dict, template):
"""
Updates a dictionary with template's yml default properties
if said dictionary does not already contain the properties.

:param property_dict: the dictionary to be updated
:param template: a dictionary representing the templated properties
to be applied on property_dict
:return: None
"""
for key in template:
if key not in property_dict:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the key is already in the property_dict, but we had altered dbt_invoke_template.yml between the previous run and the current run?

Copy link
Author

@david-whimsical david-whimsical Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would keep the value already in the file – else I think it could lead to frustrating behaviors, where dbt-invoke overwrites things.

In my opinion, The template should only add new lines, and not update any existing ones, if that makes sense!

property_dict[key] = template[key]


def _get_property_header(resource, resource_type, properties=None):
"""
Create a dictionary representing resources properties
Expand Down