[BUG] Issue with OneLake authentication in Notebook

[promisinganuj]

As part of setup, we create a OneLake shortcut to ADLS Gen2 storage account. This shortcut uses "Workspace Identity" for authentication.

Now, in the "setup" notebook, we add a default lakehouse and mount it locally.

When trying to access a config file stored within a folder of this shortcut, it fails initially. However, after 1-2 minutes, the file becomes accessible.

I have created an IcM for investigating this issue: https://portal.microsofticm.com/imp/v5/incidents/details/572273196/summary

Relates to #862

DoD

• Find out the root cause of the problem.
• Add workaround/fix in the "setup" notebook.

[promisinganuj]

FYI @devlace

[promisinganuj]

I had a call with the engineer today. We were able to reproduce the error during our call. The team is going to investigate this issue tomorrow.

[promisinganuj]

For now, we have added a workaround in the notebook itself.

## Temporary workaround. Remove when issue with OneLake authentication in Notebook is resolved.
file_path = f"{local_data_mount_path}/{config_file_path}"
os.listdir(f"{local_data_mount_path}/")
os.listdir(f"{local_data_mount_path}/sc-adls-main")
os.listdir(f"{local_data_mount_path}/sc-adls-main/config")
with open(file_path, "r") as file:
    file_content = file.read()

print(file_content)

[promisinganuj]

There is no update on the IcM.

[promisinganuj]

Engineering team has responded to our IcM https://portal.microsofticm.com/imp/v5/incidents/details/572273196/summary and acknowledged that it's a bug. There is already a PR out for the fix. Meanwhile, they also attested the workaround that we are using. Here is an excerpt from the IcM:

---

Hey Anuj Parashar , I sincerely apologize for the delay in the response.
I was able to investigate the issue and looks like whenever HEAD call goes out at the container level, we are missing a // in the call being made which is why its returning a 404 in the case of firewall bypass enabled adls accounts (it returns a 403 in non firewall bypass enabled accounts).
This is a small known bug for which we have a work item as well as a fix PR:

Bug 1548911 Head call to OneLake for External ADLS is failing at container level
Here is the PR: Bug 1548911 Head call to OneLake for External ADLS is failing at container level

As a workaround, could you please make a list call first (os.listdir() which you are already using) and then read the file? The list call should succeed as the issue is primarily with the head call - internally I suspect that Spark validates whether the container exists before reading the file but when a list is performed, it may cache this info due to which read may not perform a head call - I will confirm the behavior from the Spark team. In the meantime, I hope this workaround helps you while we fix the bug. Please let me know.

---

[promisinganuj]

Workaround incorporated, and issue acknowledged by PG.