From 9d3310b0a00db15375d6d3b3cc7dd2c0e9b6f1d2 Mon Sep 17 00:00:00 2001 From: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Date: Thu, 15 May 2025 16:53:12 +0100 Subject: [PATCH 1/4] docs(Azure): Document Azure environment variables The Azure environment variables were not documented. This PR adds them to the documentation pages. Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> --- docs/azure.md | 42 ++++++++++++++++++++++++++++++++++++++++ docs/reference/config.md | 14 +++++++------- 2 files changed, 49 insertions(+), 7 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index 9cb93d957a..b319eba8a5 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -34,6 +34,14 @@ The value of `sasToken` is the token stripped by the character `?` from the begi Once the Blob Storage credentials are set, you can access the files in the blob container like local files by prepending the file path with `az://` followed by the container name. For example, a blob container named `my-data` with a file named `foo.txt` can be specified in your Nextflow script as `az://my-data/foo.txt`. +:::{tip} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_STORAGE_ACCOUNT_NAME` - The name of your Azure Storage account +- `AZURE_STORAGE_ACCOUNT_KEY` - The access key for your Azure Storage account +- `AZURE_STORAGE_SAS_TOKEN` - A shared access signature (SAS) token for Azure Storage access +::: + ## Azure File Shares *New in `nf-azure` version `0.11.0`* @@ -155,6 +163,13 @@ Replacing `` with a pipeline name e.g. `nextflow-io/rnaseq-nf` an See the [Batch documentation](https://docs.microsoft.com/en-us/azure/batch/quick-create-portal) for further details about the configuration for Azure Batch. +:::{tip} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_BATCH_ACCOUNT_NAME` - The name of your Azure Batch account +- `AZURE_BATCH_ACCOUNT_KEY` - The access key for your Azure Batch account +::: + ### Autopools When using the `autoPoolMode` option, Nextflow automatically creates a `pool` of compute nodes appropriate for your pipeline. @@ -387,6 +402,13 @@ The private registry is an addition, not a replacement, to the existing configur When using containers hosted in a private registry, the registry name must also be provided in the container name specified via the {ref}`container ` directive using the format: `[server]/[your-organization]/[your-image]:[tag]`. Read more about fully qualified image names in the [Docker documentation](https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry). ::: +:::{tip} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_REGISTRY_USER_NAME` - The username for Azure Container Registry authentication +- `AZURE_REGISTRY_PASSWORD` - The password for Azure Container Registry authentication +::: + ### Virtual Network :::{versionadded} 23.03.0-edge @@ -526,6 +548,12 @@ azure { } ``` +:::{note} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_MANAGED_IDENTITY_SYSTEM` - When set to "true", enables system-assigned managed identity +::: + #### User Assigned Managed Identity A system-assigned managed identity is essentially 'anonymous' and is tied to a single resource. By comparison, a user-assigned managed identity is created by the user and can be assigned to multiple resources, furthermore the lifecycle of a user-assigned managed identity is not tied to the resource. See [the Azure Documentation](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identity-best-practice-recommendations#choosing-system-or-user-assigned-managed-identities) for further details. @@ -556,6 +584,12 @@ azure { } ``` +:::{note} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_MANAGED_IDENTITY_USER` - The client ID for a user-assigned managed identity +::: + (azure-service-principal)= ### Service Principals @@ -586,6 +620,14 @@ azure { } ``` +:::{note} +Nextflow will read the following environment variables if a config item is not provided: + +- `AZURE_CLIENT_ID` - The service principal client ID (also known as application ID) +- `AZURE_CLIENT_SECRET` - The service principal secret key +- `AZURE_TENANT_ID` - The Azure Active Directory tenant ID +::: + ## Advanced configuration Read the {ref}`Azure configuration` section to learn more about advanced configuration options. diff --git a/docs/reference/config.md b/docs/reference/config.md index bbac6272d3..d5107de365 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -294,13 +294,13 @@ Read the {ref}`azure-page` page for more information. The following settings are available: `azure.activeDirectory.servicePrincipalId` -: The service principal client ID +: The service principal client ID (defaults to environment variable `AZURE_CLIENT_ID`) `azure.activeDirectory.servicePrincipalSecret` -: The service principal client secret +: The service principal client secret (defaults to environment variable `AZURE_CLIENT_SECRET`) `azure.activeDirectory.tenantId` -: The Azure tenant ID +: The Azure tenant ID (defaults to environment variable `AZURE_TENANT_ID`) `azure.azcopy.blobTier` : The blob [access tier](https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview) used by `azcopy` to upload files to Azure Blob Storage. Valid options are `None`, `Hot`, or `Cool` (default: `None`). @@ -309,10 +309,10 @@ The following settings are available: : The block size (in MB) used by `azcopy` to transfer files between Azure Blob Storage and compute nodes (default: 4). `azure.batch.accountName` -: The batch service account name. +: The batch service account name. Defaults to environment variable `AZURE_BATCH_ACCOUNT_NAME`. `azure.batch.accountKey` -: The batch service account key. +: The batch service account key. Defaults to environment variable `AZURE_BATCH_ACCOUNT_KEY`. `azure.batch.allowPoolCreation` : Enable the automatic creation of batch pools specified in the Nextflow configuration file (default: `false`). @@ -415,10 +415,10 @@ The following settings are available: : Specify the virtual machine type used by the pool identified with ``. `azure.managedIdentity.clientId` -: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. +: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. (defaults to environment variable `AZURE_MANAGED_IDENTITY_USER`) `azure.managedIdentity.system` -: When `true`, use the system-assigned [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) to authenticate Azure resources. See {ref}`azure-managed-identities` for more details. +: When `true`, use the system-assigned [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) to authenticate Azure resources. See {ref}`azure-managed-identities` for more details. (defaults to environment variable `AZURE_MANAGED_IDENTITY_SYSTEM`) `azure.registry.server` : Specify the container registry from which to pull the Docker images (default: `docker.io`). From f0b7573971cbb718d5dc66d0d23b78db38f978bb Mon Sep 17 00:00:00 2001 From: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Date: Fri, 16 May 2025 09:33:05 +0100 Subject: [PATCH 2/4] Apply style suggestions from review Co-authored-by: Chris Hakkaart Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> --- docs/azure.md | 20 ++++++++++---------- docs/reference/config.md | 10 +++++----- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index b319eba8a5..0b35b85de1 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -37,9 +37,9 @@ Once the Blob Storage credentials are set, you can access the files in the blob :::{tip} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_STORAGE_ACCOUNT_NAME` - The name of your Azure Storage account -- `AZURE_STORAGE_ACCOUNT_KEY` - The access key for your Azure Storage account -- `AZURE_STORAGE_SAS_TOKEN` - A shared access signature (SAS) token for Azure Storage access +- `AZURE_STORAGE_ACCOUNT_NAME`: The name of your Azure Storage account. +- `AZURE_STORAGE_ACCOUNT_KEY`: The access key for your Azure Storage account. +- `AZURE_STORAGE_SAS_TOKEN`: A shared access signature (SAS) token for Azure Storage access. ::: ## Azure File Shares @@ -166,8 +166,8 @@ See the [Batch documentation](https://docs.microsoft.com/en-us/azure/batch/quick :::{tip} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_BATCH_ACCOUNT_NAME` - The name of your Azure Batch account -- `AZURE_BATCH_ACCOUNT_KEY` - The access key for your Azure Batch account +- `AZURE_BATCH_ACCOUNT_NAME`: The name of your Azure Batch account. +- `AZURE_BATCH_ACCOUNT_KEY`: The access key for your Azure Batch account. ::: ### Autopools @@ -551,7 +551,7 @@ azure { :::{note} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_MANAGED_IDENTITY_SYSTEM` - When set to "true", enables system-assigned managed identity +- `AZURE_MANAGED_IDENTITY_SYSTEM`: When set to `true`, enables system-assigned managed identity. ::: #### User Assigned Managed Identity @@ -587,7 +587,7 @@ azure { :::{note} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_MANAGED_IDENTITY_USER` - The client ID for a user-assigned managed identity +- `AZURE_MANAGED_IDENTITY_USER`: The client ID for a user-assigned managed identity. ::: (azure-service-principal)= @@ -623,9 +623,9 @@ azure { :::{note} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_CLIENT_ID` - The service principal client ID (also known as application ID) -- `AZURE_CLIENT_SECRET` - The service principal secret key -- `AZURE_TENANT_ID` - The Azure Active Directory tenant ID +- `AZURE_CLIENT_ID`: The service principal client ID (also known as application ID). +- `AZURE_CLIENT_SECRET`: The service principal secret key. +- `AZURE_TENANT_ID`: The Azure Active Directory tenant ID. ::: ## Advanced configuration diff --git a/docs/reference/config.md b/docs/reference/config.md index d5107de365..bf6c1c2177 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -294,13 +294,13 @@ Read the {ref}`azure-page` page for more information. The following settings are available: `azure.activeDirectory.servicePrincipalId` -: The service principal client ID (defaults to environment variable `AZURE_CLIENT_ID`) +: The service principal client ID. Defaults to environment variable `AZURE_CLIENT_ID`. `azure.activeDirectory.servicePrincipalSecret` -: The service principal client secret (defaults to environment variable `AZURE_CLIENT_SECRET`) +: The service principal client secret. Defaults to environment variable `AZURE_CLIENT_SECRET`. `azure.activeDirectory.tenantId` -: The Azure tenant ID (defaults to environment variable `AZURE_TENANT_ID`) +: The Azure tenant ID. Defaults to environment variable `AZURE_TENANT_ID`. `azure.azcopy.blobTier` : The blob [access tier](https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview) used by `azcopy` to upload files to Azure Blob Storage. Valid options are `None`, `Hot`, or `Cool` (default: `None`). @@ -415,10 +415,10 @@ The following settings are available: : Specify the virtual machine type used by the pool identified with ``. `azure.managedIdentity.clientId` -: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. (defaults to environment variable `AZURE_MANAGED_IDENTITY_USER`) +: Specify the client ID for an Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). See {ref}`azure-managed-identities` for more details. Defaults to environment variable `AZURE_MANAGED_IDENTITY_USER`. `azure.managedIdentity.system` -: When `true`, use the system-assigned [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) to authenticate Azure resources. See {ref}`azure-managed-identities` for more details. (defaults to environment variable `AZURE_MANAGED_IDENTITY_SYSTEM`) +: When `true`, uses the system-assigned [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) to authenticate Azure resources. See {ref}`azure-managed-identities` for more details. Defaults to environment variable `AZURE_MANAGED_IDENTITY_SYSTEM`. `azure.registry.server` : Specify the container registry from which to pull the Docker images (default: `docker.io`). From c62c28e520c6ea025cee7b1fe47aa7a8261b4473 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 21 May 2025 11:20:36 +0200 Subject: [PATCH 3/4] Update docs/azure.md [ci skip] Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: Paolo Di Tommaso --- docs/azure.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index 0b35b85de1..6bd6554bd8 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -405,8 +405,8 @@ When using containers hosted in a private registry, the registry name must also :::{tip} Nextflow will read the following environment variables if a config item is not provided: -- `AZURE_REGISTRY_USER_NAME` - The username for Azure Container Registry authentication -- `AZURE_REGISTRY_PASSWORD` - The password for Azure Container Registry authentication +- `AZURE_REGISTRY_USER_NAME`: The username for Azure Container Registry authentication +- `AZURE_REGISTRY_PASSWORD`: The password for Azure Container Registry authentication ::: ### Virtual Network From c605f4afd9e243f45e628279e2bc65cfe50096a1 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Thu, 22 May 2025 08:47:50 +0200 Subject: [PATCH 4/4] Minor improvements [ci skip] Signed-off-by: Paolo Di Tommaso --- docs/azure.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index 6bd6554bd8..93bf7c5176 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -35,7 +35,7 @@ The value of `sasToken` is the token stripped by the character `?` from the begi Once the Blob Storage credentials are set, you can access the files in the blob container like local files by prepending the file path with `az://` followed by the container name. For example, a blob container named `my-data` with a file named `foo.txt` can be specified in your Nextflow script as `az://my-data/foo.txt`. :::{tip} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variables if storage settings are not provided in the Nextflow config file: - `AZURE_STORAGE_ACCOUNT_NAME`: The name of your Azure Storage account. - `AZURE_STORAGE_ACCOUNT_KEY`: The access key for your Azure Storage account. @@ -164,7 +164,7 @@ Replacing `` with a pipeline name e.g. `nextflow-io/rnaseq-nf` an See the [Batch documentation](https://docs.microsoft.com/en-us/azure/batch/quick-create-portal) for further details about the configuration for Azure Batch. :::{tip} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variables if the Batch settings are not provided in the Nextflow config file: - `AZURE_BATCH_ACCOUNT_NAME`: The name of your Azure Batch account. - `AZURE_BATCH_ACCOUNT_KEY`: The access key for your Azure Batch account. @@ -403,7 +403,7 @@ When using containers hosted in a private registry, the registry name must also ::: :::{tip} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variables if the registry credentials are not provided in the Nextflow config file: - `AZURE_REGISTRY_USER_NAME`: The username for Azure Container Registry authentication - `AZURE_REGISTRY_PASSWORD`: The password for Azure Container Registry authentication @@ -549,7 +549,7 @@ azure { ``` :::{note} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variable if the managed identity setting is not provided in the Nextflow config file: - `AZURE_MANAGED_IDENTITY_SYSTEM`: When set to `true`, enables system-assigned managed identity. ::: @@ -585,7 +585,7 @@ azure { ``` :::{note} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variable if the managed identity client ID is not provided in the Nextflow config file: - `AZURE_MANAGED_IDENTITY_USER`: The client ID for a user-assigned managed identity. ::: @@ -621,7 +621,7 @@ azure { ``` :::{note} -Nextflow will read the following environment variables if a config item is not provided: +Nextflow will use the following environment variables if the service principal settings are not provided in the Nextflow config file: - `AZURE_CLIENT_ID`: The service principal client ID (also known as application ID). - `AZURE_CLIENT_SECRET`: The service principal secret key.