Feature/retained earnings to sub calendar

.quickstart/quickstart.yml

@@ -1,7 +1,7 @@
 database_key: netsuite_database
 schema_key: netsuite_schema
 
-dbt_versions: ">=1.3.0 <2.0.0"
+dbt_versions: ">=1.3.0 <3.0.0"
 
 table_variables:
   # Feature flags

CHANGELOG.md

@@ -1,3 +1,26 @@
+# dbt_netsuite v1.3.0
+
+[PR #187](https://github.com/fivetran/dbt_netsuite/pull/187) includes the following updates:
+
+## Schema/Data Change
+**2 total changes • 2 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| [netsuite2__balance_sheet](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netsuite2__balance_sheet)<br>[netsuite2__income_statement](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netsuite2__income_statement)<br>[netsuite2__transaction_details](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.netstuite2__transaction_details)<br>[stg_netsuite2__accounting_period_fiscal_cal](https://fivetran.github.io/dbt_netsuite/#!/model/model.netsuite.stg_netsuite2__accounting_period_fiscal_cal)  | New column | | `accounting_period_full_name` | Adds the full name field from the accounting period fiscal calendar source table, providing descriptive period names like "FY2023 : Q1 2023". |
+
+## Feature Update
+- When `netsuite2__using_to_subsidiary` is enabled, `netsuite2__balance_sheet` applies each transaction’s `to_subsidiary` fiscal calendar. If `to_subsidiary` is `null`, the model falls back to the fiscal calendar of the transaction’s `subsidiary_id`.
+- Increases the required dbt version upper limit to v3.0.0
+
+## Under the Hood
+- Adds `full_name` column to the `get_accountingperiodfiscalcalendars_columns` macro to support the new staging model field.
+- Updates integration test seed data to include sample `full_name` values for testing the new functionality.
+- In `netsuite2__balance_sheet`, combines variables `netsuite2__using_to_subsidiary` and `netsuite2__using_exchange_rate` into `using_to_subsidiary_and_exchange_rate` to simplify configuration.
+
+## Contributors
+- [@jmongerlyra](https://github.com/jmongerlyra) ([PR #183](https://github.com/fivetran/dbt_netsuite/pull/183))
+
 # dbt_netsuite v1.2.1
 [PR #186](https://github.com/fivetran/dbt_netsuite/pull/186) includes the following updates:
 

README.md

@@ -5,7 +5,7 @@
         href="https://github.com/fivetran/dbt_netsuite/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
     <a alt="dbt-core">
-        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_,<2.0.0-orange.svg" /></a>
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0,_<3.0.0-orange.svg" /></a>
     <a alt="Maintained?">
         <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
     <a alt="PRs">
@@ -120,7 +120,7 @@ Include the following netsuite package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/netsuite
-    version: [">=1.2.0", "<1.3.0"]
+    version: [">=1.3.0", "<1.4.0"]
 ```
 ### Step 3: Define Netsuite.com or Netsuite2 Source
 As of April 2022 Fivetran released a new Netsuite connector version which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package.

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'netsuite'
-version: '1.2.1'
-require-dbt-version: [">=1.3.0", "<2.0.0"]
+version: '1.3.0'
+require-dbt-version: [">=1.3.0", "<3.0.0"]
 
 models:
   netsuite:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'netsuite_integration_tests'
-version: '1.2.1'
+version: '1.3.0'
 profile: 'integration_tests'
 config-version: 2
 

integration_tests/seeds/netsuite2_accounting_period_fiscal_cal_data.csv

@@ -1,7 +1,7 @@
-_fivetran_id,_fivetran_deleted,_fivetran_synced,accountingperiod,date_deleted,fiscalcalendar,parent
-t6+LTClexXfChh/pMhoj1F9SpSs=,FALSE,2022-07-10 22:17:10,280,,1,244
-hbwug8y6jPTTembEaYVZlsljZuQ=,FALSE,2022-07-10 22:17:10,330,,1,239
-hxkIkIIeDjsG8OgPAlvQVe9a6TE=,FALSE,2022-07-10 22:17:10,348,,2,274
-H00Ert0t6kQHovZX4pHHC4N4d0Y=,FALSE,2022-07-10 22:17:10,6,,2,1
-KI+pra1uPLFPsBHFuvT/tXCaWdg=,FALSE,2022-07-10 22:17:10,21,,3,171
-JIpeyH5xsYzCF0YbPKr7obXrkhA=,FALSE,2022-07-10 22:17:10,20,,3,205
+_fivetran_id,_fivetran_deleted,_fivetran_synced,accountingperiod,date_deleted,fiscalcalendar,fullname,parent
+t6+LTClexXfChh/pMhoj1F9SpSs=,FALSE,2022-07-10 22:17:10,280,,1,FY2023 : Q1 2023,244
+hbwug8y6jPTTembEaYVZlsljZuQ=,FALSE,2022-07-10 22:17:10,330,,1,FY2023 : Q4 2023,239
+hxkIkIIeDjsG8OgPAlvQVe9a6TE=,FALSE,2022-07-10 22:17:10,348,,2,FY2024 : Q2 2024,274
+H00Ert0t6kQHovZX4pHHC4N4d0Y=,FALSE,2022-07-10 22:17:10,6,,2,FY2022 : Jan 2022,1
+KI+pra1uPLFPsBHFuvT/tXCaWdg=,FALSE,2022-07-10 22:17:10,21,,3,FY2022 : Q2 2022,171
+JIpeyH5xsYzCF0YbPKr7obXrkhA=,FALSE,2022-07-10 22:17:10,20,,3,FY2022 : Q1 2022,205

integration_tests/tests/consistency/consistency_balance_sheet_amounts.sql

@@ -7,7 +7,6 @@ with prod as (
     select 
         case when account_id is null then -999999 else account_id end as account_id,
         date_trunc(accounting_period_ending, month) as prod_account_period_month,
-        count(*) as prod_row_count,
         sum(converted_amount) as prod_converted_amount 
     from {{ target.schema }}_netsuite_prod.netsuite2__balance_sheet
     where date_trunc(accounting_period_ending, month) < date_trunc(current_date(), month) - 1 
@@ -18,7 +17,6 @@ dev as (
     select 
         case when account_id is null then -999999 else account_id end as account_id,
         date_trunc(accounting_period_ending, month) as dev_account_period_month,
-        count(*) as dev_row_count,
         sum(converted_amount) as dev_converted_amount 
     from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
     where date_trunc(accounting_period_ending, month) < date_trunc(current_date(), month) - 1 
@@ -30,8 +28,6 @@ final as (
         prod.account_id,
         prod.prod_account_period_month,
         dev.dev_account_period_month,
-        prod.prod_row_count,
-        dev.dev_row_count,
         prod.prod_converted_amount,
         dev.dev_converted_amount
     from prod
@@ -43,5 +39,4 @@ final as (
 
 select *
 from final 
-where abs(prod_converted_amount - dev_converted_amount) >= 0.01
-or abs(prod_row_count - dev_row_count) >= 0.01
+where abs(prod_converted_amount - dev_converted_amount) >= 0.01

integration_tests/tests/consistency/consistency_balance_sheet_count.sql

@@ -5,27 +5,28 @@
 
 with prod as (
     select 
-        1 as join_key,
+        case when account_id is null then -999999 else account_id end as account_id,
         count(*) as total_balance_sheet_prod_rows
     from {{ target.schema }}_netsuite_prod.netsuite2__balance_sheet
     group by 1
 ),
 
 dev as (
     select 
-        1 as join_key,
+        case when account_id is null then -999999 else account_id end as account_id,
         count(*) as total_balance_sheet_dev_rows
     from {{ target.schema }}_netsuite_dev.netsuite2__balance_sheet
     group by 1
 ),
 
 final as (
     select
-        total_balance_sheet_prod_rows,
-        total_balance_sheet_dev_rows
+        prod.account_id,
+        prod.total_balance_sheet_prod_rows,
+        dev.total_balance_sheet_dev_rows
     from prod
     full outer join dev
-        on dev.join_key = prod.join_key
+        on dev.account_id = prod.account_id
 )
 
 select *

macros/staging/get_accountingperiodfiscalcalendars_columns.sql

@@ -22,7 +22,8 @@
     {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
     {"name": "accountingperiod", "datatype": dbt.type_int()},
     {"name": "fiscalcalendar", "datatype": dbt.type_int()},
-    {"name": "parent", "datatype": dbt.type_int()}
+    {"name": "parent", "datatype": dbt.type_int()},
+    {"name": "fullname", "datatype": dbt.type_string()}
 ] %}
 
 {{ return(columns) }}

models/docs.md

@@ -133,4 +133,4 @@
 
 {% docs accounting_book_id %} Unique identifier of the accounting book. {% enddocs %}
 
-{% docs account_type_id %} Unique identifier of the account type. {% enddocs %}
+{% docs account_type_id %} Unique identifier of the account type. {% enddocs %}

models/netsuite2.yml

@@ -67,6 +67,9 @@ models:
       description: End date of the accounting period
     - name: accounting_period_name
       description: Name of the accounting period.
+    - name: accounting_period_full_name
+      description: |
+        The full name or display name of the accounting period within the fiscal calendar, providing descriptive period names like 'FY2023 : Q1 2023'.
     - name: is_accounting_period_adjustment
       description: Boolean field, indicating whether or not the selecting accounting period is an adjustment period.
     - name: is_accounting_period_closed
@@ -143,6 +146,9 @@ models:
       description: End date of the accounting period.
     - name: accounting_period_name
       description: Name of the accounting period.
+    - name: accounting_period_full_name
+      description: |
+        The full name or display name of the accounting period within the fiscal calendar, providing descriptive period names like 'FY2023 : Q1 2023'.
     - name: is_accounting_period_adjustment
       description: Boolean field, indicating whether or not the selected accounting period is an adjustment period.
     - name: is_accounting_period_closed
@@ -252,6 +258,9 @@ models:
       description: The unique identifier of the accounting period.
     - name: accounting_period_name
       description: Name of the accounting period.
+    - name: accounting_period_full_name
+      description: |
+        The full name or display name of the accounting period within the fiscal calendar, providing descriptive period names like 'FY2023 : Q1 2023'.
     - name: is_accounting_period_adjustment
       description: Boolean field, indicating whether or not the selecting accounting period is an adjustment period.
     - name: is_accounting_period_closed

models/netsuite2/intermediate/base/int_netsuite2__accounting_periods.sql

@@ -22,6 +22,7 @@ joined as (
     select
         accounting_periods.*,
         accounting_period_fiscal_calendars.fiscal_calendar_id,
+        accounting_period_fiscal_calendars.accounting_period_full_name,
         fiscal_calendar.fiscal_month
     from accounting_periods
 
@@ -65,6 +66,7 @@ final as (
     select
         accounting_periods.*,
         accounting_period_fiscal_calendars.fiscal_calendar_id,
+        accounting_period_fiscal_calendars.accounting_period_full_name,
         cast({{ dbt.date_trunc('year', 'accounting_periods.starting_at') }} as date) as fiscal_year_trunc
     from accounting_periods
 

models/netsuite2/netsuite2__balance_sheet.sql

@@ -1,6 +1,5 @@
 {%- set multibook_accounting_enabled = var('netsuite2__multibook_accounting_enabled', false) -%}
-{%- set using_to_subsidiary = var('netsuite2__using_to_subsidiary', false) -%}
-{%- set using_exchange_rate = var('netsuite2__using_exchange_rate', true) -%}
+{%- set using_to_subsidiary_and_exchange_rate = (var('netsuite2__using_to_subsidiary', false) and var('netsuite2__using_exchange_rate', true)) -%}
 {%- set balance_sheet_transaction_detail_columns = var('balance_sheet_transaction_detail_columns', []) -%}
 {%- set accounts_pass_through_columns = var('accounts_pass_through_columns', []) -%}
 {%- set lookback_window = var('lookback_window', 3) -%}
@@ -54,14 +53,6 @@ currencies as (
     select *
     from {{ ref('stg_netsuite2__currencies') }}
 ),
-
-primary_subsidiary_calendar as (
-    select 
-        fiscal_calendar_id, 
-        source_relation 
-    from subsidiaries 
-    where parent_id is null
-),
 
 balance_sheet as ( 
     select
@@ -79,14 +70,15 @@ balance_sheet as (
         transactions_with_converted_amounts.accounting_book_name,
         {% endif %}
 
-        {% if using_to_subsidiary and using_exchange_rate %}
+        {% if using_to_subsidiary_and_exchange_rate %}
         transactions_with_converted_amounts.to_subsidiary_id,
         transactions_with_converted_amounts.to_subsidiary_name,
         transactions_with_converted_amounts.to_subsidiary_currency_symbol,
         {% endif %}
 
         reporting_accounting_periods.accounting_period_id as accounting_period_id,
         reporting_accounting_periods.ending_at as accounting_period_ending,
+        reporting_accounting_periods.accounting_period_full_name,
         reporting_accounting_periods.name as accounting_period_name,
         reporting_accounting_periods.is_adjustment as is_accounting_period_adjustment,
         reporting_accounting_periods.is_closed as is_accounting_period_closed,
@@ -202,7 +194,7 @@ balance_sheet as (
         and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id
         {% endif %}
 
-        {% if using_to_subsidiary and using_exchange_rate %}
+        {% if using_to_subsidiary_and_exchange_rate %}
         and transaction_details.to_subsidiary_id = transactions_with_converted_amounts.to_subsidiary_id
         {% endif %}
     {% endif %}
@@ -211,31 +203,33 @@ balance_sheet as (
         on accounts.account_id = transactions_with_converted_amounts.account_id
         and accounts.source_relation = transactions_with_converted_amounts.source_relation
 
-    left join accounting_periods as reporting_accounting_periods
+    left join subsidiaries
+        on subsidiaries.subsidiary_id = transactions_with_converted_amounts.subsidiary_id
+        and subsidiaries.source_relation = transactions_with_converted_amounts.source_relation
+
+    {% if using_to_subsidiary_and_exchange_rate %}
+    left join subsidiaries as to_subsidiaries
+        on to_subsidiaries.subsidiary_id = coalesce(transactions_with_converted_amounts.to_subsidiary_id, subsidiaries.subsidiary_id)
+        and to_subsidiaries.source_relation = transactions_with_converted_amounts.source_relation
+    {% endif %}
+
+    left join accounting_periods as reporting_accounting_periods 
         on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id
         and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation
+        and reporting_accounting_periods.fiscal_calendar_id = {{ 'to_subsidiaries' if using_to_subsidiary_and_exchange_rate else 'subsidiaries' }}.fiscal_calendar_id
 
     left join accounting_periods as transaction_accounting_periods
         on transaction_accounting_periods.accounting_period_id = transactions_with_converted_amounts.transaction_accounting_period_id
         and transaction_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation
-
-    left join subsidiaries
-        on subsidiaries.subsidiary_id = transactions_with_converted_amounts.subsidiary_id
-        and subsidiaries.source_relation = transactions_with_converted_amounts.source_relation
+        and transaction_accounting_periods.fiscal_calendar_id = {{ 'to_subsidiaries' if using_to_subsidiary_and_exchange_rate else 'subsidiaries' }}.fiscal_calendar_id
 
     left join currencies subsidiaries_currencies
         on subsidiaries_currencies.currency_id = subsidiaries.currency_id
         and subsidiaries_currencies.source_relation = subsidiaries.source_relation
 
-    join primary_subsidiary_calendar 
-        on reporting_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id
-        and reporting_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation
-
-        and transaction_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id
-        and transaction_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation
-
-    where accounts.is_balancesheet 
-        or transactions_with_converted_amounts.is_income_statement
+    where (accounts.is_balancesheet 
+        or transactions_with_converted_amounts.is_income_statement)
+        and transactions_with_converted_amounts.reporting_accounting_period_id is not null
 
     union all
 
@@ -254,14 +248,15 @@ balance_sheet as (
         transactions_with_converted_amounts.accounting_book_name,
         {% endif %}
 
-        {% if using_to_subsidiary and using_exchange_rate %}
+        {% if using_to_subsidiary_and_exchange_rate %}
         transactions_with_converted_amounts.to_subsidiary_id,
         transactions_with_converted_amounts.to_subsidiary_name,
         transactions_with_converted_amounts.to_subsidiary_currency_symbol,
         {% endif %}
 
         reporting_accounting_periods.accounting_period_id as accounting_period_id,
         reporting_accounting_periods.ending_at as accounting_period_ending,
+        reporting_accounting_periods.accounting_period_full_name,
         reporting_accounting_periods.name as accounting_period_name,
         reporting_accounting_periods.is_adjustment as is_accounting_period_adjustment,
         reporting_accounting_periods.is_closed as is_accounting_period_closed,
@@ -310,7 +305,7 @@ balance_sheet as (
         and transaction_details.accounting_book_id = transactions_with_converted_amounts.accounting_book_id
         {% endif %}
 
-        {% if using_to_subsidiary and using_exchange_rate %}
+        {% if using_to_subsidiary_and_exchange_rate %}
         and transaction_details.to_subsidiary_id = transactions_with_converted_amounts.to_subsidiary_id
         {% endif %}
     {% endif %}
@@ -319,29 +314,33 @@ balance_sheet as (
         on accounts.account_id = transactions_with_converted_amounts.account_id
         and accounts.source_relation = transactions_with_converted_amounts.source_relation
 
-    left join accounting_periods as reporting_accounting_periods
-        on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id
-        and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation
-
     left join subsidiaries
         on subsidiaries.subsidiary_id = transactions_with_converted_amounts.subsidiary_id
         and subsidiaries.source_relation = transactions_with_converted_amounts.source_relation
 
+    {% if using_to_subsidiary_and_exchange_rate %}
+    left join subsidiaries as to_subsidiaries
+        on to_subsidiaries.subsidiary_id = coalesce(transactions_with_converted_amounts.to_subsidiary_id, subsidiaries.subsidiary_id)
+        and to_subsidiaries.source_relation = transactions_with_converted_amounts.source_relation
+    {% endif %}
+
+    left join accounting_periods as reporting_accounting_periods 
+        on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id
+        and reporting_accounting_periods.source_relation = transactions_with_converted_amounts.source_relation
+        and reporting_accounting_periods.fiscal_calendar_id = {{ 'to_subsidiaries' if using_to_subsidiary_and_exchange_rate else 'subsidiaries' }}.fiscal_calendar_id
+
     left join currencies subsidiaries_currencies
         on subsidiaries_currencies.currency_id = subsidiaries.currency_id
         and subsidiaries_currencies.source_relation = subsidiaries.source_relation
 
-    join primary_subsidiary_calendar 
-        on reporting_accounting_periods.fiscal_calendar_id = primary_subsidiary_calendar.fiscal_calendar_id
-        and reporting_accounting_periods.source_relation = primary_subsidiary_calendar.source_relation
-
-    where accounts.is_balancesheet
-        or transactions_with_converted_amounts.is_income_statement
+    where (accounts.is_balancesheet
+        or transactions_with_converted_amounts.is_income_statement)
+        and transactions_with_converted_amounts.reporting_accounting_period_id is not null
     ),
 
     surrogate_key as ( 
     {% set surrogate_key_fields = ['source_relation', 'transaction_line_id', 'transaction_id', 'accounting_period_id', 'account_name', 'account_id'] %}
-    {% do surrogate_key_fields.append('to_subsidiary_id') if using_to_subsidiary and using_exchange_rate %}
+    {% do surrogate_key_fields.append('to_subsidiary_id') if using_to_subsidiary_and_exchange_rate %}
     {% do surrogate_key_fields.append('accounting_book_id') if multibook_accounting_enabled %}
 
     select 
@@ -352,4 +351,4 @@ balance_sheet as (
 )
 
 select *
-from surrogate_key
+from surrogate_key