Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Fabric notebook for data transform step #837

Closed
Tracked by #779
sreedhar-guda opened this issue Nov 14, 2024 · 3 comments · Fixed by #959
Closed
Tracked by #779

Create Fabric notebook for data transform step #837

sreedhar-guda opened this issue Nov 14, 2024 · 3 comments · Fixed by #959
Assignees
Labels
e2e: fabric Related with E2E Fabric Sample

Comments

@sreedhar-guda
Copy link
Contributor

sreedhar-guda commented Nov 14, 2024

The steps for data transformation currently use Databricks(30_transform.py). This logic needs to be converted for Microsoft Fabric notebook.

DoD:

A Fabric notebook is created with existing data transformation logic.

NOTES:

  • Template file (nb-030-transform.ipynb) is created as part of Create initial version of notebooks #862
    • If the Lakehouse tables ddls needs to be modified- then these can be done in lakehouse_ddls.yaml followed by execution of nb-010-setup.ipynb(Lake schema and tables creation) and nb-020-standardize.ipynb(prepare data for transform).
@sreedhar-guda sreedhar-guda added the e2e: fabric Related with E2E Fabric Sample label Nov 14, 2024
@sreedhar-guda sreedhar-guda self-assigned this Nov 14, 2024
@sreedhar-guda sreedhar-guda removed their assignment Nov 20, 2024
@maye-msft
Copy link
Contributor

maye-msft commented Dec 16, 2024

@yuna-s
Please merge the spike backlog item #924 with this one

@yuna-s yuna-s self-assigned this Dec 16, 2024
@yuna-s
Copy link

yuna-s commented Dec 16, 2024

  • Implement existing data transformation logic and save the data into the table
  • Modify lakehouse_ddls.yaml to fit the Lakehouse tables with table data
  • Manually upload the ddo-transformation wheel file without modifying and repackaging its python code
  • Integrate setup and standardize notebook with this notebook in a pipeline

@promisinganuj
Copy link
Contributor

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
e2e: fabric Related with E2E Fabric Sample
Projects
None yet
5 participants