-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Showing
6 changed files
with
146 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
Configuration | ||
############# | ||
|
||
The plugin runs in the context of a dbt project. | ||
|
||
Project directory | ||
************************ | ||
When you run `pytest` from the root of your project, you do **not** need to set | ||
the project directory. If you want to run `pytest` from another location, you | ||
point the `---dbt-project-dir` | ||
`option <https://docs.pytest.org/en/6.2.x/usage.html#getting-help-on-version-option-names-environment-variables>`_ | ||
to the root of your project. | ||
|
||
Profiles directory | ||
********************** | ||
If you want to change dbt's profiles directory, use the `DBT_PROFILES_DIR` environment `variable <https://docs.getdbt.com/dbt-cli/configure-your-profile/#advanced-customizing-a-profile-directory>`_. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
dbt-spark | ||
######### | ||
|
||
`dbt-spark` users are recommend to use the Spark connection method when testing. | ||
Together with the `pytest Spark plugin <https://github.com/malexer/pytest-spark>`_, | ||
a on-the-fly Spark session removes the need for hosting Spark. | ||
|
||
Installation | ||
************ | ||
|
||
Install `dbt-spark`, `pytest-dbt-core` and `pytest-spark` via pip with | ||
|
||
.. code-block:: bash | ||
python -m pip install dbt-spark pytest-dbt-core pytest-spark | ||
Configuration | ||
************* | ||
|
||
Configure `pytest-spark` via `pytest configuration <https://docs.pytest.org/en/6.2.x/customize.html#configuration>`_. | ||
|
||
.. code-block:: cfg | ||
# setup.cfg | ||
[tool:pytest] | ||
spark_options = | ||
spark.executor.instances: 1 | ||
spark.sql.catalogImplementation: in-memory | ||
Usage | ||
***** | ||
|
||
Use the `spark_session` fixture to set-up the unit test for your macro: | ||
|
||
.. code-block:: python | ||
from __future__ import annotations | ||
import pytest | ||
from dbt.clients.jinja import MacroGenerator | ||
from pyspark.sql import SparkSession | ||
@pytest.mark.parametrize( | ||
"macro_generator", ["macro.spark_utils.get_tables"], indirect=True | ||
) | ||
def test_create_table( | ||
spark_session: SparkSession, macro_generator: MacroGenerator | ||
) -> None: | ||
expected_table = "default.example" | ||
spark_session.sql(f"CREATE TABLE {expected_table} (id int) USING parquet") | ||
tables = macro_generator() | ||
assert tables == [expected_table] | ||
Test | ||
**** | ||
|
||
Run the Pytest via your preferred interface. | ||
|
||
.. code-block:: bash | ||
pytest |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
Projects | ||
############# | ||
|
||
The following projects use the `pytest-dbt-core` plugin: | ||
|
||
* `spark-utils <https://github.com/dbt-labs/spark-utils>`_ |