dbt Core 1.0.0rc2

A second release candidate for dbt-core v1.0.0. Changes since rc1:

Breaking changes

• Restrict secret env vars (prefixed DBT_ENV_SECRET_) to profiles.yml + packages.yml only. Raise an exception if a secret env var is used elsewhere (#4310, #4311)
• Reorder arguments to config.get() so that default is second (#4273, #4297)

Features

• Avoid error when missing column in YAML description (#4151, #4285)
• Allow --defer flag to dbt snapshot (#4110, #4296)
• Install prerelease packages when version explicitly references a prerelease version, regardless of install-prerelease status (#4243, #4295)
• Add data attributes to json log messages (#4301)
• Add event codes to all log events (#4319)

Fixes

• Fix serialization error with missing quotes in metrics model ref (#4252, #4287)
• Correct definition of 'created_at' in ParsedMetric nodes (#4298, #4299)

Under the hood

• Add --indirect-selection parameter to profiles.yml and builtin DBT_ env vars; stringified parameter to enable multi-modal use (#3997, #4270)
• Fix filesystem searcher test failure on Python 3.9 (#3689, #4271)
• Clean up deprecation warnings shown for dbt_project.yml config renames (#4276, #4291)
• Fix metrics count in compiled project stats (#4290, #4292)
• First pass at supporting more dbt tasks via python lib (#4200)

Contributors

• @kadero (#4285, #4296)
• @joellabes (#4295)

dbt 0.21.1rc2

A second release candidate for v0.21.1.

Fixes

• Add get_where_subquery to test macro namespace, fixing custom generic tests that rely on introspecting the model arg at parse time (#4195, #4197)

dbt Core 1.0.0rc1

A first release candidate for dbt-core v1.0.0. Includes everything in b1 + b2, plus:

Breaking changes

• Replace greedy flag/property for test selection with indirect_selection: eager/cautious flag/property. Set to eager by default. Note: This reverts test selection to its pre-v0.20 behavior by default. dbt test -s my_model will select multi-parent tests, such as relationships, that depend on unselected resources. To achieve the behavior change in v0.20 + v0.21, set --indirect-selection=cautious on the CLI or indirect_selection: cautious in yaml selectors. (#4082, #4104)
• In v1.0.0, pip install dbt will raise an explicit error. Instead, please use pip install dbt-<adapter> (to use dbt with that database adapter), or pip install dbt-core (for core functionality). For parity with the previous behavior of pip install dbt, you can use: pip install dbt-core dbt-postgres dbt-redshift dbt-snowflake dbt-bigquery (#4100, #4133)
• Reorganize the global_project (macros) into smaller files with clearer names. Remove unused global macros: column_list, column_list_for_create_table, incremental_upsert (#4154)
• Introduce structured event interface, and begin conversion of all legacy logging (#3359, #4055)
  • This is a breaking change for adapter plugins, requiring a very simple migration. See events module README for details.
  • If you maintain another kind of dbt-core plugin that makes heavy use of legacy logging, and you need time to cut over to the new event interface, you can re-enable the legacy logger via an environment variable shim, DBT_ENABLE_LEGACY_LOGGER=True. Be advised that we will remove this capability in a future version of dbt-core.

Features

• Allow nullable error_after in source freshness (#3874, #3955)
• Add metrics nodes (#4071, #4235)
• Add support for dbt init <project_name>, and support for skip_profile_setup argument (dbt init -s) (#4156, #4249)

Fixes

• Changes unit tests using assertRaisesRegexp to assertRaisesRegex (#4136, #4136)
• Allow retries when the answer from a dbt deps is None (#4178, #4225)

Docs

• Fix non-alphabetical sort of Source Tables in source overview page (docs#81, docs#218)
• Add title tag to node elements in tree (docs#202, docs#203)
• Account for test rename: schema → generic, data → singular. Use test_metadata instead of schema/data tags to differentiate (docs#216, docs#222)
• Add metrics (core#216, docs#223)

Under the hood

• Bump artifact schema versions for 1.0.0: manifest v4, run results v4, sources v3. Notable changes: added metrics nodes; schema test + data test nodes are renamed to generic test + singular test nodes; freshness threshold default values (#4191)
• Speed up node selection by skipping incorporate_indirect_nodes if not needed (#4213, #4214)
• When on_schema_change is set, pass common columns as dest_columns in incremental merge macros (#4144, #4170)
• Clear adapters before registering in lib module config generation (#4218)
• Remove official support for python 3.6, which is reaching end of life on December 23, 2021 (#4134, #4223)

Contributors

• @kadero (#3955, #4249)
• @frankcash (#4136)
• @Kayrnt (#4136)
• @VersusFacit (#4104)
• @joellabes (#4104)
• @b-per (#4225)
• @salmonsd (docs#218)
• @miike (docs#203)

dbt 0.21.1rc1

An initial release candidate for v0.21.1.

Fixes

• Performance: Use child_map to find tests for nodes in resolve_graph (#4012, #4057)
• Switch unique_field from abstractproperty to optional property. Add docstring (#4025, #4028)
• Fix multiple partial parsing errors (#3996, #4020)
• Include only relational nodes in database_schema_set (#4063, #4077)
• Fixed bug with error_if test option (#4070)
• Added support for tests on databases that lack real boolean types. (#4084)
• Prefer macros defined in the project over the ones in a package by default (#4106, #4114)
• Scrub secrets coming from CommandErrors so they don't get exposed in logs. (#4138)
• Syntax fix in alter_relation_add_remove_columns if only removing columns in on_schema_change: sync_all_columns (#4147)
• Increase performance of graph subset selection (#4135,#4155)
• Add downstream test edges for build task only. Restore previous graph construction, compilation performance, and node selection behavior (test+) for all other tasks (#4135, #4143)
• Don't require a strict/proper subset when adding testing edges to specialized graph for build (#4158, #4158)
• Capping google-api-core to version 1.31.3 due to protobuf dependency conflict (#4192)

Contributors:

• @ljhopkins2 (#4077)
• @JCZuurmond (#4114)

dbt Core 1.0.0b2

An second beta release for v1.0.0.

Breaking changes

• Enable on-run-start and on-run-end hooks for dbt test. Add flags.WHICH to execution context, representing current task (#3463, #4004)

Features

• Normalize global CLI arguments/flags (#2990, #3839)
• Turns on the static parser by default and adds the flag --no-static-parser to disable it. (#3377, #3939)
• Generic test FQNs have changed to include the relative path, resource, and column (if applicable) where they are defined. This makes it easier to configure them from the tests block in dbt_project.yml (#3259, #3880
• Turn on partial parsing by default (#3867, #3989)
• Add result:<status> selectors to automatically rerun failed tests and erroneous models. This makes it easier to rerun failed dbt jobs with a simple selector flag instead of restarting from the beginning or manually running the dbt models in scope. (#3859, #4017)
• dbt init is now interactive, generating profiles.yml when run inside existing project (#3625)

Under the hood

• Fix intermittent errors in partial parsing tests (#4060, #4068)
• Make finding disabled nodes more consistent (#4069, #4073)
• Remove connection from render_with_context during parsing, thereby removing misleading log message (#3137, #4062)
• Wait for postgres docker container to be ready in setup_db.sh. (#3876, #3908)
• Prefer macros defined in the project over the ones in a package by default (#4106, #4114)
• Dependency updates (#4079), (#3532

Contributors:

• @sungchun12 (#4017)
• @matt-winkler (#4017)
• @NiallRees (#3625)
• @rvacaru (#3908)
• @JCZuurmond (#4114)
• @ljhopkins2

dbt Core 1.0.0b1

An initial beta release for v1.0.0.

Breaking changes

• The two type of test definitions are now "singular" and "generic" (instead of "data" and "schema", respectively). The test_type: selection method accepts test_type:singular and test_type:generic. (It will also accept test_type:schema and test_type:data for backwards compatibility) (#3234, #3880). Not backwards compatible: The --data and --schema flags to dbt test are no longer supported, and tests no longer have the tags 'data' and 'schema' automatically applied.
• Deprecated the use of the packages arg adapter.dispatch in favor of the macro_namespace arg. (#3895)

Features

• Normalize global CLI arguments/flags (#2990, #3839)
• Turns on the static parser by default and adds the flag --no-static-parser to disable it. (#3377, #3939)
• Generic test FQNs have changed to include the relative path, resource, and column (if applicable) where they are defined. This makes it easier to configure them from the tests block in dbt_project.yml (#3259, #3880
• Turn on partial parsing by default (#3867, #3989)

Fixes

• Add generic tests defined on sources to the manifest once, not twice (#3347, #3880)
• Skip partial parsing if certain macros have changed (#3810, #3982)
• Enable cataloging of unlogged Postgres tables (3961, #3993)
• Fix multiple disabled nodes (#4013, #4018)
• Fix multiple partial parsing errors (#3996, #4020)
• Return an error instead of a warning when runing with --warn-error and no models are selected (#4006, #4019)

Under the hood

• Enact deprecation for materialization-return and replace deprecation warning with an exception. (#3896)
• Build catalog for only relational, non-ephemeral nodes in the graph (#3920)
• Enact deprecation to remove the release arg from the execute_macro method. (#3900)
• Enact deprecation for default quoting to be True. Override for the dbt-snowflake adapter so it stays False. (#3898)
• Enact deprecation for object used as dictionaries when they should be dataclasses. Replace deprecation warning with an exception for the dunder methods of __iter__ and __len__ for all superclasses of FakeAPIObject. (#3897)
• Enact deprecation for adapter-macro and replace deprecation warning with an exception. (#3901)
• Add warning when trying to put a node under the wrong key. ie. A seed under models in a schema.yml file. (#3899)
• Plugins for redshift, snowflake, and bigquery have moved to separate repos: dbt-redshift, dbt-snowflake, dbt-bigquery
• Change the default dbt packages installation directory to dbt_packages from dbt_modules. Also rename module-path to packages-install-path to allow default overrides of package install directory. Deprecation warning added for projects using the old dbt_modules name without specifying a packages-install-path. (#3523)
• Update the default project paths to be analysis-paths = ['analyses'] and test-paths = ['tests]. Also have starter project set analysis-paths: ['analyses'] from now on. (#2659)
• Define the data type of sources as an array of arrays of string in the manifest artifacts. (#3966, #3967)

Contributors:

• @dave-connors-3 (#3920)
• @kadero (#3952)
• @samlader (#3993)
• @yu-iskw (#3967)
• @laxjesse (#4019)

dbt 0.21.0

dbt Core 0.21.0 - Louis Kahn (October 4, 2021)

• 💬 Discourse: v0.21.0
• 📖 New & changed documentation

Breaking changes

• Rename source freshness command, add full node selection, and align selection syntax with other tasks (#2987, #3554)
  • dbt source snapshot-freshness -> dbt source freshness (backwards compatible)
  • dbt source freshness --select source_name --> dbt source freshness --select source:source_name (not backwards compatible)
• dbt-snowflake: Disable most transactional logic. See note below for details.

Features

• Add dbt build command to run models, tests, seeds, and snapshots in DAG order. (#2743, #3490, #3608)
  • Skip downstream nodes during the build task when a test fails. (#3597, #3792)
  • Update cli and rpc flags for the build task to align with other commands (--resource-type, --store-failures) (#3596, #3884)
• Introduce on_schema_change config to detect and handle schema changes on incremental models (#1132, #3387)
• Capture changes to macros in state:modified. Introduce new state: sub-selectors: modified.body, modified.configs, modified.persisted_descriptions, modified.relation, modified.macros (#2704, #3278, #3559)
• Enable setting configs in schema files for models, seeds, snapshots, analyses, tests (#2401, #3616)
• Make --models and --select synonyms, except for ls (to preserve existing behavior) (#3210, #3791)
• Customize ls task JSON output by adding new flag --output-keys (#3778, #3395)
• Added default field in the selectors.yml to allow user to define default selector (#3448, #3875, #3892)
• Log tests that are not indirectly selected. Add --greedy flag to test, list, build and greedy property in yaml selectors (#3723, #3833)
• Scrub environment secrets prefixed with DBT_ENV_SECRET_ from logs and console output (#3440, #3617)

Artifacts

• Added timing and thread information to sources.json artifact (#3804, #3894)
• Bump artifact schema versions for 0.21.0 (#3945)

Fixes

• Fix type coercion issues when fetching query result sets (#2984, #3499)
• Handle whitespace after a plus sign on the project config (#3526)
• Fix table and view materialization issue when switching from one to the other (#2161), #3547)
• Avoid caching schemas for tests when store_failures is not enabled (#3715, #3716)
• dbt debug shows a summary of whether all checks passed or not (#3831, #3832)
• Fix bug with initializing a dataclass that inherits from typing.Protocol, specifically for dbt.config.profile.Profile (#3843, #3855)
• Introduce a macro, get_where_subquery, for tests that use where config. Alias filtering subquery as dbt_subquery instead of resource identifier (#3857, #3859)
• Use group by column_name in accepted_values test for compatibility with most database engines (#3905, #3906)
• Avoid infinite recursion in state:modified.macros check (#3904, #3957)
• Cast log messages to strings before scrubbing of prefixed env vars (#3971, #3972)

Under the hood

• Improve default view and table materialization performance by checking relational cache before attempting to drop temp relations (#3112, #3468)
• Move the example project used by dbt init into dbt repository, to avoid cloning an external repo (#3005, #3474, #3536)
• Better interaction between dbt init and adapters. Avoid raising errors while initializing a project (#2814, #3483)
• Update create_adapter_plugins script to include latest accessories, and stay up to date with latest dbt-core version (#3002, #3509)
• Allow the default seed macro's SQL parameter, %s, to be replaced by dispatching a new macro, get_binding_char(). This enables adapters with parameter marker characters such as ? to not have to override basic_load_csv_rows. (#3622, #3623)
• Add adapter_unique_id to invocation context in anonymous usage tracking, to better understand dbt adoption (#3713, #3796)
• Specify macro_namespace = 'dbt' for all dispatched macros in the global project, making it possible to dispatch to macro implementations defined in packages. Dispatch generate_schema_name and generate_alias_name (#3456, #3851)

Packages

• Better dbt hub registry packages version logging that prompts the user for upgrades to relevant packages (#3560, #3763, #3759)
• Alert users on package rename (hub.getdbt.com#180, #3825)
• Retry transient GitHub failures during download (#3546, #3729)

Performance

• Experimental parser now detects macro overrides of ref, source, and config builtins. (#3581, #3582)
• Add performance regression testing #3602
• Don't reload and validate schema files if they haven't changed (#3563, #3888)

Dependencies

• Require werkzeug>=1 (#3590)

Plugins

dbt-postgres

• Add optional sslcert, sslkey, and sslrootcert profile arguments to the Postgres connector. (#3472, #3473)
• Add connect_timeout profile configuration for Postgres and child adapters. (#3581, #3582)

dbt-redshift

• Introduce ra3 profile configuration on Redshift to support cross-db sources (#3236, #3408)

dbt-snowflake

• Breaking: Turn off transactions and turn on autocommit by default. Within dbt materializations, wrap DML statements in explicit begin and commit. Note that it is not recommended to run DML statements outside of dbt materialization logic. If you do this, despite our recommendation, you will need to wrap those statements in explicit begin and commit. Note also that this may affect user-space code that depends on transactions, such as pre-hooks and post-hooks that specify transaction: true or transaction: false. We recommend removing those references to transactions. (#2748, #3480, #3510)
• Fix batching for large seeds on Snowflake (#3941, [#39...

dbt 0.21.0rc2

A second release candidate for v0.21.0.

Fixes

• Fix batching for large seeds on Snowflake (#3941, #3942)
• Avoid infinite recursion in state:modified.macros check (#3904, #3957)

Under the hood

• Bump artifact schema versions for 0.21.0 (#3945)

dbt 0.21.0rc1

An initial release candidate for v0.21.0.

Features

• Experimental parser now detects macro overrides of ref, source, and config builtins. (#3581, #3582)
• Add connect_timeout profile configuration for Postgres and Redshift adapters. (#3581, #3582)
• Enhance BigQuery copy materialization (#3570, #3606):
  • to simplify config (default usage of copy_materialization='table' if is is not found in global or local config)
  • to let copy several source tables into single target table at a time. (Google doc reference)
• Customize ls task JSON output by adding new flag --output-keys (#3778, #3395)
• Add support for execution project on BigQuery through profile configuration (#3707, #3708)
• Skip downstream nodes during the build task when a test fails. (#3597, #3792)
• Added default field in the selectors.yml to allow user to define default selector (#3448, #3875, #3892)
• Added timing and thread information to sources.json artifact (#3804, #3894)
• Update cli and rpc flags for the build task to align with other commands (--resource-type, --store-failures) (#3596, #3884)

Fixes

• Support BigQuery-specific aliases target_dataset and target_project in snapshot configs (#3694, #3834)
• dbt debug shows a summary of whether all checks passed or not (#3831, #3832)
• Fix issue when running the deps task after the list task in the RPC server (#3846, #3848, #3850)
• Fix bug with initializing a dataclass that inherits from typing.Protocol, specifically for dbt.config.profile.Profile (#3843, #3855)
• Introduce a macro, get_where_subquery, for tests that use where config. Alias filtering subquery as dbt_subquery instead of resource identifier (#3857, #3859)
• Use group by column_name in accepted_values test for compatibility with most database engines (#3905, #3906)
• Separated table vs view configuration for BigQuery since some configuration is not possible to set for tables vs views. (#3682, #3691)

Under the hood

• Use GitHub Actions for CI (#3688, #3669)
• Better dbt hub registry packages version logging that prompts the user for upgrades to relevant packages (#3560, #3763, #3759)
• Allow the default seed macro's SQL parameter, %s, to be replaced by dispatching a new macro, get_binding_char(). This enables adapters with parameter marker characters such as ? to not have to override basic_load_csv_rows. (#3622, #3623)
• Alert users on package rename (hub.getdbt.com#180, #3825)
• Add adapter_unique_id to invocation context in anonymous usage tracking, to better understand dbt adoption (#3713, #3796)
• Specify macro_namespace = 'dbt' for all dispatched macros in the global project, making it possible to dispatch to macro implementations defined in packages. Dispatch generate_schema_name and generate_alias_name (#3456, #3851)
• Retry transient GitHub failures during download (#3729)

Contributors:

• @xemuliam (#3606)
• @sungchun12 (#3759)
• @dbrtly (#3834)
• @swanderz #3623
• @JasonGluck (#3582)
• @joellabes (#3669)
• @juma-adoreme (#3838)
• @annafil (#3825)
• @AndreasTA-AW (#3691)
• @Kayrnt (3707)
• @TeddyCr (#3448)
• @sdebruyn (#3906)

dbt 0.20.2

Fixes

• Fix issue when running the deps task after the list task in the RPC server (#3846, #3848)
• Fix bug with initializing a dataclass that inherits from typing.Protocol, specifically for dbt.config.profile.Profile (#3843, #3855)
• Avoid caching schemas for tests when store_failures is not enabled (#3715, #3716)
• Fix bug in finding analysis nodes when applying analysis patch (#3764, #3767)
• Rewrite built-in generic tests to support column_name expressions (#3790, #3811)
• Check for existence of test node when removing. (#3711, #3750)

Under the hood

• Get more information on partial parsing version mismatches (#3757, #3758)
• Use GitHub Actions for CI (#3688, #3669)
• Switch to full reparse on partial parsing exceptions. Log and report exception information. (#3725, #3733)
• Check for existence of test node when removing. (#3711, #3750)
• Better error handling for BigQuery job labels that are too long. #3703

Contributors:

• @sungchun12 (#3703)