Azure Volume Populator

Kubernetes Volume Populator for Azure Blob Storage.

This application supports two modes:

• controller: Run as a Kubernetes controller that watches a AzureVolumePopulator custom resource and schedules a pod to populate volumes using this same binary in populate mode.
• populate: Run as a standalone tool, started by the mentioned Kubernetes controller, to actually populate a PersistentVolume with blobs from Azure Blob Storage.

It supports downloading all blobs under a given prefix (container/path).

Build

Using Docker:

docker build -t pdok/azure-volume-populator .

Using Go:

go mod download
go generate ./...
go build -o azure-volume-populator cmd/main.go

Run

In controller mode (example in bash, but you would usually do this in a Kubernetes Deployment):

export MODE=controller
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=...;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"

./azure-volume-populator \
  --mode=controller \
  --azure-storage-connection-string="$AZURE_STORAGE_CONNECTION_STRING" \
  --image-name="docker.io/pdok/azure-volume-populator:latest" \
  --namespace="default" \
  --http-endpoint=":8080" \
  --metrics-path="/metrics"

In populate mode (example in bash, but usually this will be done by the above Kubernetes controller):

export MODE=populate
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=...;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"

./azure-volume-populator \
  --mode=populate \
  --azure-storage-connection-string="$AZURE_STORAGE_CONNECTION_STRING" \
  --blob-prefix="mycontainer/folder/subfolder" \
  --volume-path="/data" \
  --blob-block-size=4194304 \
  --blob-concurrency=10

Misc

How to Contribute

Make a pull request...

Contact

Contacting the maintainers can be done through the issue tracker.