Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] generic data test on source is running after model #11309

Open
2 tasks done
dchimeno opened this issue Feb 14, 2025 · 0 comments
Open
2 tasks done

[Bug] generic data test on source is running after model #11309

dchimeno opened this issue Feb 14, 2025 · 0 comments
Labels
bug Something isn't working triage

Comments

@dchimeno
Copy link

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I wrote a generic test like under tests/generic/column_distinct_count_greather_than_threshold

{% test column_distinct_count_greater_than_threshold(model, column_name, thresholds_table, thresholds_column) %}

WITH t AS (
    SELECT *
    FROM {{ thresholds_table }}
    WHERE prov_code = {{ var('prov_code') }}
    LIMIT 1
),
d AS (
    SELECT 
        {{ var('prov_code')}}::INTEGER AS prov_code,
        COUNT(DISTINCT {{ column_name }}) AS distinct_count
    FROM {{ model }}
)
SELECT * from t
RIGHT JOIN d USING (prov_code)
WHERE coalesce(t.min_threshold,99999999999) > d.distinct_count

{% endtest %}

its then referenced in a source column like:

          - name: the_column_name
            description: ""
            type: string
            data_tests:
              - column_distinct_count_greater_than_threshold:
                  thresholds_table: source('raw.devo_by_prov', 'thresholds')
                  thresholds_column: "min_threshold"

the model is specifying the correct source like:

select *  FROM  {{ source('raw.devo_by_prov', 'xxxx' ) }}

When I run:

dbt build --select @namespace.model

The test is running, failing or passing accordingly, but the model is always executed, so rows are inserted event when failing.

Expected Behavior

source test should run first in order to skip model generation.

Steps To Reproduce

see above.

Relevant log output

08:09:28  1 of 3 START sql table model bdia_devo.stg_devo_prov__nokia_4g_unpivot ......... [RUN]
08:09:29  1 of 3 OK created sql table model bdia_devo.stg_devo_prov__nokia_4g_unpivot .... [OK in 0.99s]
08:09:29  2 of 3 START test source_column_distinct_count_greater_than_threshold_raw.devo_by_prov_nokia_4g_ECGI__min_threshold__source_raw_devo_by_prov_thresholds_  [RUN]
08:09:30  2 of 3 FAIL 1 source_column_distinct_count_greater_than_threshold_raw.devo_by_prov_nokia_4g_ECGI__min_threshold__source_raw_devo_by_prov_thresholds_  [FAIL 1 in 0.99s]
08:09:30  3 of 3 START sql external model bdia.devo_events_prov_nokia_4g ................. [RUN]

Environment

- OS: python:3.12-slim
- Python:3.12.7
- dbt:

Core:
  - installed: 1.9.2
  - latest:    1.9.2 - Up to date!

Plugins:
  - duckdb: 1.9.2 - Up to date!

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

duckdb adapter

@dchimeno dchimeno added bug Something isn't working triage labels Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

1 participant