Can Mimir read secrets from Volume mounts

[wonderphil]

Describe the bug

When using secrets from volume mount mimir applications not starting

Values file:

alertmanager:
  extraVolumes:
    - name: secret-grafana-mimir
      secret:
        secretName: secret-grafana-mimir
  # Extra volume mounts that will be added to the alertmanager container
  extraVolumeMounts:
    - name: secret-grafana-mimir
      mountPath: "/etc/secret"
      readOnly: true

mimir:
  structuredConfig:
    
    alertmanager_storage:
      backend: azure
      azure:
        ## Recplaced the ${} var with the file path
        account_name: /etc/secret/MIMIR_STORAGE_ACCOUNT_NAME
        account_key: /etc/secret/MIMIR_STORAGE_ACCOUNT_KEY
        container_name: mimir-alaertmanager
    compactor:
      data_dir: "/data"
...

if I bash into a container and look at secret files I can see the correct content but I get the following errors for all the services

Error from Mimir Alert manager:

ts=2023-09-28T06:03:38.93765098Z caller=main.go:225 level=info msg="Starting application" version="(version=2.10.0, branch=HEAD, revision=77906f7)"
ts=2023-09-28T06:03:39.337442912Z caller=server.go:335 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2023-09-28T06:03:39.537634469Z caller=multitenant.go:156 level=warn msg="The configured Alertmanager HTTP prefix '/alertmanager' is different than the path specified in the external URL 'http://prometheus-alertmanager.observability:9093': the Alertmanager UI and API may not work as expected unless you have a reverse proxy exposing the Alertmanager endpoints under '/alertmanager' prefix"
ts=2023-09-28T06:03:39.538016272Z caller=log.go:87 level=error msg="error running application" err="decode account key: illegal base64 data at input byte 17\nerror initialising module: alertmanager\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

Error from Mimir Ingestor

ts=2023-09-28T06:02:59.75854223Z caller=main.go:225 level=info msg="Starting application" version="(version=2.10.0, branch=HEAD, revision=77906f7)"
ts=2023-09-28T06:02:59.762267683Z caller=log.go:87 level=error msg="error running application" err="decode account key: illegal base64 data at input byte 17\ncreate usage-stats bucket client\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).initUsageStats\n\t/__w/mimir/mimir/pkg/mimir/modules.go:905\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nerror initialising module: usage-stats\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/__w/mimir/mimir/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:800\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

All other services have similar errors (Alert manager, compactor, ingester, querier, querier-frontend, ruler, storage-gateway)

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (SHA or version) - version="(version=2.10.0, branch=HEAD, revision=77906f7)"
2. add the above values
3. deploy to K8 cluster and wait for pods to deploy (K8 version v1.26.6, hosted on Azure)

Expected behavior

Expect each service to read the secrets file and use the contents

Environment

• Infrastructure: Azure Kubernetes version 1.26.6
• Deployment tool: argocd and helm charts

Additional Context

Logs are above along with config file