You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This dbt package transforms data from Fivetran's Aws Cloud Cost connector into analytics-ready tables.
20
20
21
-
This package models AWS Cloud Cost data from [Fivetran's AWS Cost Report connector](https://fivetran.com/docs/connectors/applications/aws-cost-report). It uses data in the format described by the [AWS Cost & Usage Report](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2.html).
21
+
## Resources
22
22
23
-
The main focus of the package is to transform the core object tables into analytics-ready models, including:
24
-
- Materializes [AWS Cloud Cost staging tables](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.stg_aws_cloud_cost__report) which leverage data in the format described by the [AWS Cost & Usage Report](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2.html). These staging tables clean, test, and prepare your AWS Cloud Cost data from [Fivetran's connector](https://fivetran.com/docs/connectors/applications/aws-cost-report) for analysis by doing the following:
25
-
- Names columns for consistency across all packages and for easier analysis:
26
-
- Column names are shortened for convenience and to avoid redundancy.
27
-
- Takes the latest export of each account's billing month report.
28
-
- Creates analytics-ready end models for monitoring and investigating cost & usage of different AWS services across your organizations.
29
-
- Generates a comprehensive data dictionary of both your AWS Cloud Cost source and modeled data through the [dbt docs site](https://fivetran.github.io/dbt_aws_cloud_cost/).
This package enables you to monitor and investigate cost & usage of different AWS services across your organizations. It creates enriched models with metrics focused on billing, pricing, line item buckets, products, reservations, and savings plans.
| [aws_cloud_cost__daily_overview](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_overview) | Daily aggregation of the [Standard](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html) Cost & Usage Report (2.0) exported from AWS. Includes slew of commonly analyzed dimensions related to [billing](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-bill.html), [pricing](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-pricing.html), [line item](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-line-item.html) buckets, and [products](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-product.html). Contains both high-level cost and usage metrics, along with all metrics related to [reservations](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-reservation.html) and [savings plans](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-savings-plan.html). Also includes financial reporting fieds relating to invoices and billing periods. |
40
-
|[aws_cloud_cost__daily_product_report](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_product_report)| Daily view of each account's use of and associated costs from individual AWS products for each billing period. Built off of the daily overview model. |
41
-
|[aws_cloud_cost__daily_instance_report](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_instance_report)| Daily view of each account's use of and associated costs from different Amazon Elastic Compute Cloud (EC2) instances for each billing period. Built off of the daily overview model. |
37
+
### Output schema
38
+
Final output tables are generated in the following target schema:
42
39
43
-
### Materialized Models
44
-
Each Quickstart transformation job run materializes 5 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
By default, this package materializes the following final tables:
47
+
48
+
| Table | Description |
49
+
| :---- | :---- |
50
+
| [aws_cloud_cost__daily_overview](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_overview) | Provides a comprehensive daily view of AWS costs and usage across all services, accounts, and billing dimensions from the [Standard](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html) Cost & Usage Report (2.0) to monitor spending patterns, optimize costs, and track [reservations](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-reservation.html) and [savings plans](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-savings-plan.html). <br></br>**Example Analytics Questions:**<ul><li>What are our daily AWS costs by [billing](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-bill.html) account, [product](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-product.html), or region?</li><li>How much are we saving through reservations and savings plans compared to on-demand pricing?</li><li>Which [line item](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-line-item.html) buckets or [pricing](https://docs.aws.amazon.com/cur/latest/userguide/table-dictionary-cur2-pricing.html) categories are driving the biggest spending increases?</li></ul>|
51
+
|[aws_cloud_cost__daily_product_report](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_product_report)| Breaks down daily AWS spending by individual product and service for each account to identify which services consume the most budget and where optimization opportunities exist. <br></br>**Example Analytics Questions:**<ul><li>Which AWS products or services drive the highest costs by account?</li><li>How are product-specific costs trending day-over-day or month-over-month?</li><li>Where should we focus cost optimization efforts based on product spending patterns?</li></ul>|
52
+
|[aws_cloud_cost__daily_instance_report](https://fivetran.github.io/dbt_aws_cloud_cost/#!/model/model.aws_cloud_cost.aws_cloud_cost__daily_instance_report)| Analyzes daily costs and usage for EC2 instances across all accounts to optimize compute spending and identify underutilized or oversized instances. <br></br>**Example Analytics Questions:**<ul><li>Which EC2 instance types generate the highest daily costs by account?</li><li>How does EC2 instance usage and spending vary across different accounts or environments?</li><li>Are there opportunities to right-size instances based on usage patterns?</li></ul>|
46
53
47
-
### How do I use the dbt package?
54
+
¹ Each Quickstart transformation job run materializes these models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
48
55
49
-
> DISCLAIMER: This package transforms source data of potentially very high volumes. Please be aware of the size of your dataset(s) and take this into consideration when configuring the frequency with which you will orchestrate the package models. See [Step 4](https://github.com/fivetran/dbt_aws_cloud_cost?tab=readme-ov-file#optional-step-4-additional-configurations) for tools to mitigate compute and storage costs.
56
+
---
50
57
51
-
#### Step 1: Prerequisites
58
+
## Prerequisites
52
59
To use this dbt package, you must have the following:
53
60
54
61
- At least one Fivetran AWS Cloud Cost connection syncing data into your destination.
55
62
- A **BigQuery**, **Snowflake**, **Redshift**, **Databricks**, or **PostgreSQL** destination.
56
63
57
-
##### Databricks dispatch configuration
58
-
If you are using a Databricks destination with this package, you must add the following (or a variation of the following) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.
59
-
```yml
60
-
dispatch:
61
-
- macro_namespace: dbt_utils
62
-
search_order: ['spark_utils', 'dbt_utils']
63
-
```
64
+
## How do I use the dbt package?
65
+
You can either add this dbt package in the Fivetran dashboard or import it into your dbt project:
64
66
65
-
#### Step 2: Install the package
67
+
- To add the package in the Fivetran dashboard, follow our [Quickstart guide](https://fivetran.com/docs/transformations/data-models/quickstart-management#quickstartmanagement).
68
+
- To add the package to your dbt project, follow the setup instructions in the dbt package's [README file](https://github.com/fivetran/dbt_aws_cloud_cost/blob/main/README.md#how-do-i-use-the-dbt-package) to use this package.
69
+
70
+
> DISCLAIMER: This package transforms source data of potentially very high volumes. Please be aware of the size of your dataset(s) and take this into consideration when configuring the frequency with which you will orchestrate the package models. See [Additional configurations](https://github.com/fivetran/dbt_aws_cloud_cost?tab=readme-ov-file#optional-additional-configurations) for tools to mitigate compute and storage costs.
71
+
72
+
<!--section-end-->
73
+
74
+
### Install the package
66
75
Include the following AWS Cloud Cost package version in your `packages.yml` file:
67
76
> TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
68
77
```yml
69
78
packages:
70
79
- package: fivetran/aws_cloud_cost
71
-
version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
80
+
version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
81
+
```
82
+
83
+
##### Databricks dispatch configuration
84
+
If you are using a Databricks destination with this package, you must add the following (or a variation of the following) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.
85
+
```yml
86
+
dispatch:
87
+
- macro_namespace: dbt_utils
88
+
search_order: ['spark_utils', 'dbt_utils']
72
89
```
73
90
74
-
#### Step 3: Define database, schema, and table name variables
91
+
### Define database, schema, and table name variables
75
92
76
93
##### Option A: Single connection
77
94
By default, this package assumes your AWS Cost & Usage Report data lives in the following location:
@@ -372,7 +389,7 @@ vars:
372
389
has_defined_sources: true
373
390
```
374
391
375
-
#### (Optional) Step 4: Additional configurations
392
+
### (Optional) Additional configurations
376
393
377
394
##### Limit Date Range
378
395
Although the package transforms the latest version of each report, your AWS Cost & Usage Report data may still be quite large. In order to avoid unnecessary compute and storage costs, we have added a minimum (INCLUSIVE) **start date** variable that can be used to limit the data's date range.
@@ -425,11 +442,11 @@ models:
425
442
```
426
443
</details>
427
444
428
-
#### (Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Core™
445
+
### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
429
446
<details><summary>Expand for details</summary>
430
447
<br>
431
448
432
-
Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core setup guides](https://fivetran.com/docs/transformations/dbt#setupguide).
449
+
Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core setup guides](https://fivetran.com/docs/transformations/dbt/setup-guide#transformationsfordbtcoresetupguide).
433
450
</details>
434
451
435
452
### Does this package have dependencies?
@@ -445,15 +462,19 @@ packages:
445
462
version: [">=1.0.0", "<2.0.0"]
446
463
```
447
464
448
-
### How is this package maintained and can I contribute?
449
-
#### Package Maintenance
450
-
The Fivetran team maintaining this package _only_ maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/aws_cloud_cost/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_aws_cloud_cost/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.
465
+
<!--section="aws-cloud-cost_maintenance"-->
466
+
## How is this package maintained and can I contribute?
467
+
468
+
### Package Maintenance
469
+
The Fivetran team maintaining this package only maintains the [latest version](https://hub.getdbt.com/fivetran/aws_cloud_cost/latest/) of the package. We highly recommend you stay consistent with the latest version of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_aws_cloud_cost/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.
451
470
452
-
#### Contributions
471
+
### Contributions
453
472
A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions.
454
473
455
-
We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package.
474
+
We highly encourage and welcome contributions to this package. Learn how to contribute to a package in dbt's [Contributing to an external dbt package article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657).
475
+
476
+
#<!--section-end-->
456
477
457
-
### Are there any resources available?
478
+
## Are there any resources available?
458
479
- If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_aws_cloud_cost/issues/new/choose) section to find the right avenue of support for you.
459
-
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
480
+
- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4&size=40){kind=avatar}
0 commit comments