Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DATA-1750 Add sensors to staging models for int models #72

Closed
wants to merge 70 commits into from
Closed

DATA-1750 Add sensors to staging models for int models #72

wants to merge 70 commits into from

Conversation

kiranvasudev
Copy link
Contributor

DATA-1750

The changes in this PR make sure to get the parents of an intermediate table and add it as an input to the dbt task config. By doing this, we can make sure that data flows through int models as they are updated in the staging layer of DBT(which is connected to the core layer)

siklosid and others added 30 commits September 30, 2023 15:45
@siklosid
Turning follow_external_dependency into bool or dict property
ea36004
@siklosid
Handling the new dict format of follow_external_dependency
5113752
@siklosid
Changing test case to see if it handles sensor parameters properly
fc2a2e2
@siklosid
Merge pull request #23 from chocoapp/feature/sensor_parameters
d0780dd 
Feature/sensor parameters
upgrade version of tenacity
db6dee2
@claudiazi
Merge pull request #24 from chocoapp/feature/DATA-1562-upgrade-tenacity
bd5adcf 
upgrade version of tenacity
@kiranvasudev
added new dbt config parser module
d463438
@kiranvasudev
added class with constructor
a91a58f
@kiranvasudev
added method to parse dbt model inputs
102620e
@kiranvasudev
added functions to generate dagger input and outputs for dbt models
c61e656
@kiranvasudev
added fn to generate io for dbt task
29858a9
@kiranvasudev
black format
1a93f05
@kiranvasudev
use new module in fn to generate configs
847f622
@kiranvasudev
changed output bucket
104013a
@kiranvasudev
fixed import
de82c14
@kiranvasudev
renamed functions and variables
dcb6854
@kiranvasudev
added type hints and docstrings
fb6f8f6
@kiranvasudev
changed how models are selected when config is generated
06dc86b
@kiranvasudev
black
19a13bc
@kiranvasudev
fix project dir to load the correct profile
5f83797
@kiranvasudev
get external_location from config instead of unrendered config
209efd6
@kiranvasudev
renamed variables for better understanding
3c4e4c3
@kiranvasudev
added doctest for fn
1bd525c
@kiranvasudev
added files for fixtures and tests
4b651ad
@kiranvasudev
added fixture for manifest and profiles
7123cbc
@kiranvasudev
setUp test class
77fb2b2
@kiranvasudev
added test for get_dbt_model_parents
47c0638
@kiranvasudev
added tests for generate_dagger_inputs
4eced6b
@kiranvasudev
added test for generate_dagger_outputs
aa767e8
@kiranvasudev
fixed type hint
8cecbe6
@kiranvasudev
modified tests for model containing dbt seed as a dependency
876010c
@kiranvasudev
refactor
9cd0a51 
* deduplicate list of input dictionaries
* created functions that generate the seed input and athena and s3 tasks
* removed the follow_external_dependency as true as default for all athena inputs
@kiranvasudev
updated tests and fixtures
2e6abf6
@kiranvasudev
changed name of seed task generating fn
c652d56
@kiranvasudev
removed unused line
ac5f48f
@kiranvasudev
added docstrings
46a833d
@kiranvasudev
removed unused test parameter
cdd0fea
@kiranvasudev
changed name of function for better understanding
829ac94
@kiranvasudev
added test to check for de-duplication of inputs
eaf4286
@kiranvasudev
refactored getting model location function
ab9a2c8 
this was done because the bucket name in the main module config can be different for how the manifest file is compiled
@kiranvasudev
refactor dummy task generation
8ef7050
@kiranvasudev
generate inputs for intermediate models and updated tests
65a07d7
@kiranvasudev
uncomment skipping local test
e5afcfa
@kiranvasudev
refactor generate_dagger_tasks fn to make recursive
e28a078
@kiranvasudev
updated fixtures and tests
ab48229
@kiranvasudev
Merge pull request #26 from chocoapp/feature/dbt-task-split
3c8c92c 
Added functionality for DBT seed and intermediate model input generation
@kiranvasudev
bugfix
d653977
@kiranvasudev
only return dummy when stg model
48a102f
@kiranvasudev
adapted tests
f6c7226
@kiranvasudev
Merge pull request #27 from chocoapp/fix/external-dag-sensors
32fdd1f 
fix external sensor bug
@kiranvasudev
Merge pull request #28 from chocoapp/feature/dbt-task-split
4ceca95 
Feature/dbt task split
@kiranvasudev
initialize dbt module only when its a dbt pipeline config
146846e
@kiranvasudev
format
fd2bd0f
@kiranvasudev
made logic to check for dbt task easier
fb0c2e9
@kiranvasudev
Merge pull request #29 from chocoapp/feature/dbt-task-split
bd8e44c 
Feature/dbt task split
@kiranvasudev
fix: follow external dependency for staging models
a2b3fdd
@kiranvasudev
Merge pull request #30 from chocoapp/feature/dbt-task-split
4f1783b 
fix: follow external dependency for staging models
@kiranvasudev
added logic to get parents of int models
3653646 
this is done to keep track of dependencies of int models that are ephemeral
@kiranvasudev
updated tests and fixtures
7dec65a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kiranvasudev @siklosid @claudiazi