Upload spark libraries to Fabric Environment

[promisinganuj]

As part of the initial setup, we need to upload the python OTEL library and environment.yml files to Fabric Environment.

Reference: https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/feat/e2e-fabric-dataops-sample/e2e_samples/fabric_dataops_sample/scripts/setup_fabric_environment.py

DoD

• Upload (Stage) the environment file "otel_monitor_invoker.py" to the Fabric Environment.
• Upload (Stage) the "environment.yml" to the Fabric Environment.
• Publish the libraries.
• Do it before enabling the Fabric GIT integration so that the environment gets synced to the repo.

Out of Scope

The same process would also be performed as part of CI/CD process, but that it out of scope of this item.

Update The REST APIs are not working ATM. So, we might have to consider about making this step manual.

Depends-on #946

[sreedhar-guda]

@promisinganuj
The manual step here to still make these files available as common modules (outside if Environment) could be:

• Upload the files to ADLS (may be main/scripts)
• Create shortcut (sc-adls-main)
• The Create initial version of notebooks #862 implementation already mounts this file system as /local_data
• Users can perform:

  import sys
  sys.path.append('/local_data/scripts')

• Use the code module by importing it as needed

[promisinganuj]

As the APIs are now working, this is unblocked. We have already implemented the library upload as part of initial IaC deployment.