Updated to include Managed Identity support for On-demand cluster #127766

KrishnakumarRukmangathan · 2025-10-23T12:40:27Z

No description provided.

prmerger-automator · 2025-10-23T12:40:39Z

@KrishnakumarRukmangathan : Thanks for your contribution! The author(s) and reviewer(s) have been notified to review your proposed change.

learn-build-service-prod · 2025-10-23T12:47:24Z

Learn Build status updates of commit d07c19f:

✅ Validation status: passed

File	Status	Preview URL	Details
articles/data-factory/compute-linked-services.md	✅Succeeded

For more details, please refer to the build report.

KrishnakumarRukmangathan · 2025-10-23T12:51:59Z

#sign-off

prmerger-automator · 2025-10-23T12:52:08Z

Invalid command: '#sign-off'. Only the assigned author of one or more file in this PR can sign off. @nabhishek

Copilot

Pull Request Overview

This PR adds support for Managed Identity authentication methods (System Assigned and User Assigned) for on-demand HDInsight clusters in Azure Data Factory, expanding beyond the existing Service Principal Key authentication.

Key Changes:

Added two new authentication examples demonstrating System Assigned and User Assigned Managed Identity configurations
Introduced new properties clusterResourceGroupAuthType and credential for managing authentication types
Added comprehensive documentation explaining Managed Identity authentication requirements and limitations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-23T14:43:06Z

articles/data-factory/compute-linked-services.md

 | clusterSize                  | Number of worker/data nodes in the cluster. The HDInsight cluster is created with 2 head nodes along with the number of worker nodes you specify for this property. The nodes are of size Standard_D3 that has 4 cores, so a 4 worker node cluster takes 24 cores (4\*4 = 16 cores for worker nodes, plus 2\*4 = 8 cores for head nodes). See [Set up clusters in HDInsight with Hadoop, Spark, Kafka, and more](../hdinsight/hdinsight-hadoop-provision-linux-clusters.md) for details. | Yes      |
 | linkedServiceName            | Azure Storage linked service to be used by the on-demand cluster for storing and processing data. The HDInsight cluster is created in the same region as this Azure Storage account. Azure HDInsight has limitation on the total number of cores you can use in each Azure region it supports. Make sure you have enough core quotas in that Azure region to meet the required clusterSize. For details, refer to [Set up clusters in HDInsight with Hadoop, Spark, Kafka, and more](../hdinsight/hdinsight-hadoop-provision-linux-clusters.md)<p>Currently, you cannot create an on-demand HDInsight cluster that uses an Azure Data Lake Storage (Gen 2) as the storage. If you want to store the result data from HDInsight processing in an Azure Data Lake Storage (Gen 2), use a Copy Activity to copy the data from the Azure Blob Storage to the Azure Data Lake Storage (Gen 2). </p> | Yes      |
 | clusterResourceGroup         | The HDInsight cluster is created in this resource group. | Yes      |
+| clusterResourceGroupAuthType | Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". | Required for using Managed Identity authentication. If field is not there, will default to ServicePrincipalKey     |


The description in the 'Required?' column is unclear. It states 'Required for using Managed Identity authentication' but then says 'will default to ServicePrincipalKey' if not present, which contradicts the 'required' statement. Consider clarifying to: 'No (defaults to ServicePrincipalKey). Required when using Managed Identity authentication.'

Suggested change

| clusterResourceGroupAuthType | Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". | Required for using Managed Identity authentication. If field is not there, will default to ServicePrincipalKey |

| clusterResourceGroupAuthType | Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". | No (defaults to ServicePrincipalKey). Required when using Managed Identity authentication. |

Copilot · 2025-10-23T14:43:06Z

articles/data-factory/compute-linked-services.md

 | linkedServiceName            | Azure Storage linked service to be used by the on-demand cluster for storing and processing data. The HDInsight cluster is created in the same region as this Azure Storage account. Azure HDInsight has limitation on the total number of cores you can use in each Azure region it supports. Make sure you have enough core quotas in that Azure region to meet the required clusterSize. For details, refer to [Set up clusters in HDInsight with Hadoop, Spark, Kafka, and more](../hdinsight/hdinsight-hadoop-provision-linux-clusters.md)<p>Currently, you cannot create an on-demand HDInsight cluster that uses an Azure Data Lake Storage (Gen 2) as the storage. If you want to store the result data from HDInsight processing in an Azure Data Lake Storage (Gen 2), use a Copy Activity to copy the data from the Azure Blob Storage to the Azure Data Lake Storage (Gen 2). </p> | Yes      |
 | clusterResourceGroup         | The HDInsight cluster is created in this resource group. | Yes      |
+| clusterResourceGroupAuthType | Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". | Required for using Managed Identity authentication. If field is not there, will default to ServicePrincipalKey     |
+| credential                   | Specify the credential reference containing Managed Identity object which has access over the resource group.  | Only required for "UserAssignedManagedIdentity" authentication.     |


The description states 'containing Managed Identity object which has access over the resource group' but should be 'that has access to the resource group' for grammatical correctness.

Suggested change

| credential | Specify the credential reference containing Managed Identity object which has access over the resource group. | Only required for "UserAssignedManagedIdentity" authentication. |

| credential | Specify the credential reference that has access to the resource group. | Only required for "UserAssignedManagedIdentity" authentication. |

Copilot · 2025-10-23T14:43:07Z

articles/data-factory/compute-linked-services.md

+      "credential": {
+            "referenceName": "CredentialName",
+            "type": "CredentialReference"
+       },


Inconsistent indentation within the credential object. Lines 155-156 have 6 spaces of indentation, while line 157 has only 1 space. Should maintain consistent indentation (either 6 or 8 spaces for inner properties) to match the formatting of other properties in the JSON example.

Suggested change

},

},

v-dirichards · 2025-10-23T14:44:26Z

Hi @KrishnakumarRukmangathan - This pull request was opened in the public repo. PMs should work in the private repo, per the Microsoft Docs contributor guide. We can keep this PR open for review and merge, but would you make future content updates in the private repo? Thank you!

@nabhishek

Can you review the proposed changes?

Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

prmerger-automator · 2025-10-23T17:59:15Z

Invalid command: '#sign-off'. Only the assigned author of one or more file in this PR can sign off. @nabhishek

KrishnakumarRukmangathan · 2025-10-23T18:02:46Z

Hi @KrishnakumarRukmangathan - This pull request was opened in the public repo. PMs should work in the private repo, per the Microsoft Docs contributor guide. We can keep this PR open for review and merge, but would you make future content updates in the private repo? Thank you!

@nabhishek

Can you review the proposed changes?

Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.

#label:"aq-pr-triaged" @MicrosoftDocs/public-repo-pr-review-team

Since when has this change come in effect. Just 2 weeks back I had modified this same article, made necessary changes and merged the PR myself. Can you please let me know what are the new changes in effect?

KrishnakumarRukmangathan · 2025-10-23T18:06:03Z

@v-dirichards Can you please proceed to merge these changes. The author of this article doesn't work on this feature.
Should I change the author to myself?

v-dirichards · 2025-10-23T20:10:11Z

Since when has this change come in effect. Just 2 weeks back I had modified this same article, made necessary changes and merged the PR myself. Can you please let me know what are the new changes in effect?

@KrishnakumarRukmangathan When you made the changes two weeks ago, you made them in the private repo (azure-docs-pr).

This current PR was opened in the public repo (azure-docs), and the author must sign off on it before we can merge.

You can still pull these changes into the private repo, review the articles on staging, fix any validation issues, and then you can add the sign-off comment to let PR reviewers know that the updates are ready to be merged.

nabhishek · 2025-10-24T06:07:06Z

#sign-off

KrishnakumarRukmangathan · 2025-10-24T06:58:30Z

#sign-off

prmerger-automator · 2025-10-24T06:58:35Z

Invalid command: '#sign-off'. Only the assigned author of one or more file in this PR can sign off. @nabhishek

Updated to include Managed Identity support for On-demand cluster

d07c19f

prmerger-automator bot added the do-not-merge label Oct 23, 2025

prmerger-automator bot assigned nabhishek Oct 23, 2025

prmerger-automator bot requested a review from nabhishek October 23, 2025 12:40

prmerger-automator bot added azure-data-factory/svc Change sent to author orchestration/subsvc labels Oct 23, 2025

v-dirichards requested a review from Copilot October 23, 2025 14:42

Copilot AI reviewed Oct 23, 2025

View reviewed changes

prmerger-automator bot added the aq-pr-triaged tracking label for the PR review team label Oct 23, 2025

nabhishek approved these changes Oct 24, 2025

View reviewed changes

prmerger-automator bot added ready-to-merge and removed do-not-merge labels Oct 24, 2025

v-dirichards merged commit 2df9f9e into MicrosoftDocs:main Oct 24, 2025
2 checks passed

	\| clusterResourceGroupAuthType \| Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". \| Required for using Managed Identity authentication. If field is not there, will default to ServicePrincipalKey \|
	\| clusterResourceGroupAuthType \| Specify the HDInsight On-demand cluster resource group authentication type. Supported auth types are "ServicePrincipalKey", "SystemAssignedManagedIdentity", "UserAssignedManagedIdentity". \| No (defaults to ServicePrincipalKey). Required when using Managed Identity authentication. \|

	\| credential \| Specify the credential reference containing Managed Identity object which has access over the resource group. \| Only required for "UserAssignedManagedIdentity" authentication. \|
	\| credential \| Specify the credential reference that has access to the resource group. \| Only required for "UserAssignedManagedIdentity" authentication. \|

Uh oh!

Updated to include Managed Identity support for On-demand cluster #127766

Updated to include Managed Identity support for On-demand cluster #127766

Conversation

KrishnakumarRukmangathan commented Oct 23, 2025

Uh oh!

prmerger-automator bot commented Oct 23, 2025

Uh oh!

learn-build-service-prod bot commented Oct 23, 2025

✅ Validation status: passed

Uh oh!

KrishnakumarRukmangathan commented Oct 23, 2025

Uh oh!

prmerger-automator bot commented Oct 23, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

v-dirichards commented Oct 23, 2025

Uh oh!

prmerger-automator bot commented Oct 23, 2025

Uh oh!

KrishnakumarRukmangathan commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KrishnakumarRukmangathan commented Oct 23, 2025

Uh oh!

v-dirichards commented Oct 23, 2025

Uh oh!

nabhishek commented Oct 24, 2025

Uh oh!

KrishnakumarRukmangathan commented Oct 24, 2025

Uh oh!

prmerger-automator bot commented Oct 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KrishnakumarRukmangathan commented Oct 23, 2025 •

edited

Loading