Create initial version of notebooks

[sreedhar-guda]

Type of PR

• Documentation changes
• Code changes
• Test changes
• CI-CD changes

Purpose

The changes are to create the first version of the setup book notebook which creates database schemas and tables needed for Parking sensor sample. The changes include:

• creation of the notebook without any default lakehouse attached
• A script to capture telemetry using Open Telemetry
• A config to store all the DDLs (specific for this notebook)
• A common config file to store common information like workspace id, lakehouse id etc.
• Additional changes to Ci/CD linter hooks are done based on the errors encountered in the DEV container, and the changes were made with help and guidance of Naga.

What it includes:

• A notebook for setup activity. This is a full working example with all the steps: inputs params, mounting, OTEL logging and fully implemented code functions to create schemas and tables. Default lakehouse only attached during runtime using user input parameters.
• Notebooks for Standardization and Transform steps. These are similar to Setup notebook - but the code functions are right now dummy, while rest of it like params, OTEL implementation is all set. Once ready, developers can fill that portion and rest of the portion/s need not be updated.
• A data pipeline with all notebooks, The pipeline is coded to support high concurrency executions if the feature is enabled at workspace level for data pipelines. This would be equiavalent to using %run_multiple to call notebooks using another notebook. All notebooks are also re-written to support high concurrency executions.
• Configuration files for all notebooks
• Configuration file to store DDLs for the lakehouse
• Python script to capture telemetry using Open Telemetry

Note that, the config and OTEL script files are named and in locations (configs and otel script) as per discussion (References #840).

Does this introduce a breaking change? If yes, details on what can break

No

Author pre-publish checklist

• Added test to prove my fix is effective or new feature works
• No PII in logs
• Made corresponding changes to the documentation

Validation steps

• ...

Issues Closed or Referenced

• Closes Create Fabric notebook for setup steps #843

• References

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ydaponte]

@sreedhar-guda, great job - just minor comments and the main thing for me is the ReadMe really needs to be updated before merging this, otherwise there is no connection to what to do with the content for the user.

[promisinganuj]

@ydaponte , I have created a separate item for updating the documentation
#917

[yunishimura0716]

Hi @sreedhar-guda, it's tiny thing but let me ask.
Do you have any intention here why not setting tracer_provider globally like trace.set_tracer_provider(tracer_provider).
The reason why I'm asking this is because this code will have a following error (but it's not blocker) during exporting.

ERROR:azure.monitor.opentelemetry.exporter.export.trace._exporter:Failed to derive Resource from Tracer Provider: 'ProxyTracerProvider' object has no attribute 'resource'

(Reference: azure monitor otel github looks globally setting tracer_provider

[promisinganuj]

@maniSbindra , do you know the above details?

[naga-nandyala]

Tried deploying and running notebooks. All looks good.