[BUG] Unable to list public blobs (with no credentials)

[tooptoop4]

https://azuremlexamples.blob.core.windows.net/datasets/iris.csv is public so I want to list blobs without passing in credential.

I tried the below

          BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint("https://azuremlexamples.blob.core.windows.net").credential(new DefaultAzureCredentialBuilder().build()).buildClient();
          String containerName = "datasets";
          BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(containerName);
          for (BlobItem blobItem : containerClient.listBlobs()) {
              System.out.println("\t" + blobItem.getName());
          }

with

      <dependency>
  <groupId>com.azure</groupId>
  <artifactId>azure-core</artifactId>
  <version>1.0.0</version>
</dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-storage-blob</artifactId>
            <version>12.0.0</version>
            <!--https://github.com/Azure/azure-sdk-for-java/issues/15215#issuecomment-735940668 need to upgrade -->
            <!--https://stackoverflow.com/questions/60589689/the-exception-java-lang-nosuchmethoderror-thrown-when-invoking-azure-storage-rel/60589738#60589738 mentions could be netty conflict -->
        </dependency>
        <dependency>
          <groupId>com.azure</groupId>
          <artifactId>azure-identity</artifactId>
          <version>1.0.3</version>
        </dependency>
        <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-security-keyvault-secrets</artifactId>
    <version>4.1.0</version>
  </dependency>

I get

	at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:59)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:50)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:33)
	at com.azure.identity.DefaultAzureCredential.<init>(DefaultAzureCredential.java:37)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:18)

If I try with different versions of dependencies in pom (including latest versions) I get:

java.lang.NoSuchMethodError: com.azure.core.http.policy.HttpLogOptions.getApplicationId()Ljava/lang/String;
	at com.azure.storage.blob.implementation.util.BuilderHelper.getUserAgentPolicy(BuilderHelper.java:165)
	at com.azure.storage.blob.implementation.util.BuilderHelper.buildPipeline(BuilderHelper.java:73)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildAsyncClient(BlobServiceClientBuilder.java:107)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildClient(BlobServiceClientBuilder.java:84)

OR

Caused by: java.lang.NoClassDefFoundError: com/azure/core/implementation/serializer/jackson/JacksonAdapter
	at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:60)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:50)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:33)
	at com.azure.identity.DefaultAzureCredential.<init>(DefaultAzureCredential.java:37)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:18)

Is there any plan to provide a single 'working' bundle jar?

I also tried but hit issue in #17970 / https://stackoverflow.com/questions/65135299/azure-java-sdk-for-blob-a-request-was-made-to-load-the-default-httpclient-prov/65138749#65138749:

          BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint("https://azuremlexamples.blob.core.windows.net").buildClient();
          String containerName = "datasets";
          BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(containerName);
          for (BlobItem blobItem : containerClient.listBlobs()) {
              System.out.println("\t" + blobItem.getName());
          }

[rickle-msft]

Hi, @tooptoop4. Thank you for your question. Correct me if I'm wrong, but it seems like your underlying issue is not with listing public blobs so much as dependency management. NoSuchMethodError and NoClassDefFoundError are both indications of dependency mismatches.

As to your question about a single working bundle jar, we have shipped this bom, which I think should answer your question. I'm not sure why it hasn't released since the summer, but I'm looking into that now. In any case, that bom should still work.

The other alternative I would recommend trying if the bom does not suit you is to upgrade your identity, core, keyvault, and blob dependencies to latest as I am quite confident there will be no conflicts if everything is on the most recent version. If there's a particular reason why you feel you can't upgrade to those versions yet, I'd still like to at least run some tests with those versions to see if it resolves the exceptions you're seeing and then we can figure out how to successfully target versions that work for you

[rickle-msft]

Also, the first stack trace you shared seems to be cut off if you could please share the entire thing?

[tooptoop4]

@rickle-msft
when using the bom:

Caused by: java.lang.IllegalStateException: Cannot find any HttpClient provider on the classpath - unable to create a default HttpClient instance
	at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:35)
	at com.azure.core.http.HttpClient.createDefault(HttpClient.java:27)
	at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:60)
	at com.azure.storage.blob.implementation.util.BuilderHelper.buildPipeline(BuilderHelper.java:108)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildAsyncClient(BlobServiceClientBuilder.java:105)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildClient(BlobServiceClientBuilder.java:82)

using either of these code gets same error:

          BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint("https://azuremlexamples.blob.core.windows.net").buildClient();
          String containerName = "datasets";
          BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(containerName);
          for (BlobItem blobItem : containerClient.listBlobs()) {
              System.out.println("\t" + blobItem.getName());
          }

          BlobServiceClient blobServiceClient = new BlobServiceClientBuilder().endpoint("https://azuremlexamples.blob.core.windows.net").credential(new DefaultAzureCredentialBuilder().build()).buildClient();
          String containerName = "datasets";
          BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(containerName);
          for (BlobItem blobItem : containerClient.listBlobs()) {
              System.out.println("\t" + blobItem.getName());
          }

imports are:

import com.azure.storage.blob.*;
import com.azure.storage.blob.models.*;
import com.azure.core.http.HttpClient;
import com.azure.core.http.HttpClientProvider;
import com.azure.identity.DefaultAzureCredentialBuilder;
import java.io.*;
import java.net.*;

note that I am running this from personal PC, not on an azure vm.

[rickle-msft]

Thanks, @tooptoop4 for the information. Can you also try specifying the latest versions of the each of the libraries?
blob: 12.9.0
core: 1.11.0
security-keyvault-secrets: 4.3.0-beta.2
identity: 1.2.0

And can you please share the rest of this stack trace from your original post?

at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:59)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:50)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:33)
	at com.azure.identity.DefaultAzureCredential.<init>(DefaultAzureCredential.java:37)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:18)

[tooptoop4]

@rickle-msft

original post:

java.lang.NoClassDefFoundError: com/azure/core/util/serializer/JacksonAdapter
	at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:59)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:50)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:33)
	at com.azure.identity.DefaultAzureCredential.<init>(DefaultAzureCredential.java:37)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:18)
	at org.ebyhr.presto.flex.FlexRecordCursor.<init>(FlexRecordCursor.java:528)
	at org.ebyhr.presto.flex.FlexRecordSet.cursor(FlexRecordSet.java:55)
	at io.prestosql.spi.connector.RecordPageSource.<init>(RecordPageSource.java:37)
	at io.prestosql.split.RecordPageSourceProvider.createPageSource(RecordPageSourceProvider.java:50)
	at io.prestosql.spi.connector.ConnectorPageSourceProvider.createPageSource(ConnectorPageSourceProvider.java:68)
	at io.prestosql.split.PageSourceManager.createPageSource(PageSourceManager.java:64)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:298)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_344_41_g3758d9f_dirty____20201204_101810_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Stack trace using
blob: 12.9.0
core: 1.11.0
security-keyvault-secrets: 4.3.0-beta.2
identity: 1.2.0

java.lang.IllegalStateException: A request was made to load the default HttpClient provider but one could not be found on the classpath. If you are using a dependency manager, consider including a dependency on azure-core-http-netty or azure-core-http-okhttp. Depending on your existing dependencies, you have the choice of Netty or OkHttp implementations. Additionally, refer to https://aka.ms/azsdk/java/docs/custom-httpclient to learn about writing your own implementation.
	at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:37)
	at com.azure.core.http.HttpClient.createDefault(HttpClient.java:39)
	at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:60)
	at com.azure.storage.blob.implementation.util.BuilderHelper.buildPipeline(BuilderHelper.java:110)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildAsyncClient(BlobServiceClientBuilder.java:107)
	at com.azure.storage.blob.BlobServiceClientBuilder.buildClient(BlobServiceClientBuilder.java:84)
	at org.ebyhr.presto.flex.FlexRecordCursor.<init>(FlexRecordCursor.java:529)
	at org.ebyhr.presto.flex.FlexRecordSet.cursor(FlexRecordSet.java:55)
	at io.prestosql.spi.connector.RecordPageSource.<init>(RecordPageSource.java:37)
	at io.prestosql.split.RecordPageSourceProvider.createPageSource(RecordPageSourceProvider.java:50)
	at io.prestosql.spi.connector.ConnectorPageSourceProvider.createPageSource(ConnectorPageSourceProvider.java:68)
	at io.prestosql.split.PageSourceManager.createPageSource(PageSourceManager.java:64)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:298)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_344_41_g3758d9f_dirty____20201204_211715_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[tooptoop4]

https://stackoverflow.com/a/65179461/8874837 works so must be an issue in my app's classloader/conflict versions

[rickle-msft]

@tooptoop4 Sorry for the delay and glad to hear it's working! If your question has been resolved, then please close the issue. Otherwise please let me know if there are other areas in which you need support