diff --git a/README.md b/README.md index 2cd42d059..fb7ac6e16 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,28 @@ pre-allocated disks by detecting and creating PVs for each local disk on the host, and cleaning up the disks when released. It does not support dynamic provisioning. +## Table of Contents + +- [Overview](#overview) +- [User Guide](#user-guide) + * [Getting started](#getting-started) + * [Managing your local volumes](#managing-your-local-volumes) + * [FAQs](#faqs) + * [Best Practices](#best-practices) +- [Version Compatibility](#version-compatibility) +- [K8s Feature Status](#k8s-feature-status) + * [1.14: GA](#114-ga) + * [1.12: Beta](#112-beta) + * [1.10: Beta](#110-beta) + * [1.9: Alpha](#19-alpha) + * [1.7: Alpha](#17-alpha) + * [Future features](#future-features) +- [E2E Tests](#e2e-tests) + * [Running](#running) + * [View CI Results](#view-ci-results) +- [Community, discussion, contribution, and support](#community-discussion-contribution-and-support) + * [Code of conduct](#code-of-conduct) + ## Overview Local persistent volumes allows users to access local storage through the @@ -12,10 +34,10 @@ standard PVC interface in a simple and portable way. The PV contains node affinity information that the system uses to schedule pods to the correct nodes. -An external static provisioner is available to help simplify local storage -management once the local volumes are configured. Note that the local -storage provisioner is different from most provisioners and does -not support dynamic provisioning yet. Instead, it requires that administrators +An [external static provisioner](docs/provisioner.md) is provided here to help +simplify local storage management once the local volumes are configured. Note +that the local storage provisioner is different from most provisioners and does +not support dynamic provisioning. Instead, it requires that administrators preconfigure the local volumes on each node and if volumes are supposed to be 1. Filesystem volumeMode (default) PVs - mount them under discovery directories. @@ -25,13 +47,32 @@ preconfigure the local volumes on each node and if volumes are supposed to be The provisioner will manage the volumes under the discovery directories by creating and cleaning up PersistentVolumes for each volume. -## Configuration Requirements +## User Guide + +### Getting started + +To get started with local static provisioning, you can follow our [getting +started guide](docs/getting-started.md) to bring up a Kubernetes cluster with +some local disks, deploy local-volume-provisioner to provision local volumes +and use PVC in your pod to request a local PV. + +### Managing your local volumes + +See our [operations](docs/operations.md) documentation which contains of +preparing, setting up and cleaning up local volumes on the nodes. + +### Deploy provisioner with helm + +See our [helm](helm/README.md) documentation for how to deploy and configure +local-volume-provisioner in Kubernetes cluster. + +### FAQs -* The local-volume plugin expects paths to be stable, including across - reboots and when disks are added or removed. -* The static provisioner only discovers either mount points (for Filesystem mode volumes) - or symbolic links (for Block mode volumes). For directory-based local volumes, they - must be bind-mounted into the discovery directories. +See [FAQs](docs/faqs.md). + +### Best Practices + +See [Best Practices](docs/best-practices.md). ## Version Compatibility @@ -86,200 +127,6 @@ Also see [known issues](KNOWN_ISSUES.md) and [CHANGELOG](CHANGELOG.md). * Local PV health monitoring, taints and tolerations * Inline PV (use dedicated local disk as ephemeral storage) -## User Guide - -These instructions reflect the latest version of the codebase. For instructions -on older versions, please see version links under -[Version Compatibility](#version-compatibility). - -### Step 1: Bringing up a cluster with local disks - -#### Enabling the alpha feature gates - -##### 1.10-1.12 - -If raw local block feature is needed, -``` -$ export KUBE_FEATURE_GATES="BlockVolume=true" -``` - -Note: Kubernetes versions prior to 1.10 require [several additional -feature-gates](https://github.com/kubernetes-incubator/external-storage/tree/local-volume-provisioner-v2.0.0/local-volume#enabling-the-alpha-feature-gates) -be enabled on all Kubernetes components, because the persistent local volumes and other features were in alpha. - -#### Option 1: GCE - -GCE clusters brought up with kube-up.sh will automatically format and mount the -requested Local SSDs, so you can deploy the provisioner with the pre-generated -deployment spec and skip to [step 4](#step-4-create-local-persistent-volume-claim), -unless you want to customize the provisioner spec or storage classes. - -``` console -$ NODE_LOCAL_SSDS_EXT=,,fs cluster/kube-up.sh -$ kubectl create -f helm/generated-examples/gce.yaml -``` - -#### Option 2: GKE - -GKE clusters will automatically format and mount the -requested Local SSDs. Please see -[GKE -documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) -for instructions for how to create a cluster with Local SSDs. - -Then skip to [step 4](#step-4-create-local-persistent-volume-claim). - -**Note:** The raw block feature is only supported on GKE Kubernetes alpha clusters. - -#### Option 3: Baremetal environments - -1. Partition and format the disks on each node according to your application's - requirements. -2. Mount all the filesystems under one directory per StorageClass. The directories - are specified in a configmap, see below. -3. Configure the Kubernetes API Server, controller-manager, scheduler, and all kubelets - with `KUBE_FEATURE_GATES` as described [above](#enabling-the-alpha-feature-gates). -4. If not using the default Kubernetes scheduler policy, the following - predicates must be enabled: - * Pre-1.9: `NoVolumeBindConflict` - * 1.9+: `VolumeBindingChecker` - -#### Option 4: Local test cluster - -1. Create `/mnt/disks` directory and mount several volumes into its subdirectories. - The example below uses three ram disks to simulate real local volumes: -```console -$ mkdir /mnt/disks -$ for vol in vol1 vol2 vol3; do - mkdir /mnt/disks/$vol - mount -t tmpfs $vol /mnt/disks/$vol -done -``` - -2. Run the local cluster. - -```console -$ ALLOW_PRIVILEGED=true LOG_LEVEL=5 FEATURE_GATES=$KUBE_FEATURE_GATES hack/local-up-cluster.sh -``` - -### Step 2: Creating a StorageClass (1.9+) - -To delay volume binding until pod scheduling and to handle multiple local PVs in -a single pod, a StorageClass must to be created with `volumeBindingMode` set to -`WaitForFirstConsumer`. - -```console -$ kubectl create -f provisioner/deployment/kubernetes/example/default_example_storageclass.yaml -``` - -### Step 3: Creating local persistent volumes - -#### Option 1: Using the local volume static provisioner - -1. Generate Provisioner's ServiceAccount, Roles, DaemonSet, and ConfigMap spec, and customize it. - - This step uses helm templates to generate the specs. See the [helm README](helm) for setup instructions. - To generate the provisioner's specs using the [default values](helm/provisioner/values.yaml), run: - - ``` console - helm template ./helm/provisioner > ./provisioner/deployment/kubernetes/provisioner_generated.yaml - ``` - - You can also provide a custom values file instead: - - ``` console - helm template ./helm/provisioner --values custom-values.yaml > ./provisioner/deployment/kubernetes/provisioner_generated.yaml - ``` - -2. Deploy Provisioner - - Once a user is satisfied with the content of Provisioner's yaml file, **kubectl** can be used - to create Provisioner's DaemonSet and ConfigMap. - - ``` console - $ kubectl create -f ./provisioner/deployment/kubernetes/provisioner_generated.yaml - ``` - -3. Check discovered local volumes - - Once launched, the external static provisioner will discover and create local-volume PVs. - - For example, if the directory `/mnt/disks/` contained one directory `/mnt/disks/vol1` then the following - local-volume PV would be created by the static provisioner: - - ``` - $ kubectl get pv - NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE - local-pv-ce05be60 1024220Ki RWO Delete Available local-storage 26s - - $ kubectl describe pv local-pv-ce05be60 - Name: local-pv-ce05be60 - Labels: - Annotations: pv.kubernetes.io/provisioned-by=local-volume-provisioner-minikube-18f57fb2-a186-11e7-b543-080027d51893 - StorageClass: local-fast - Status: Available - Claim: - Reclaim Policy: Delete - Access Modes: RWO - Capacity: 1024220Ki - NodeAffinity: - Required Terms: - Term 0: kubernetes.io/hostname in [my-node] - Message: - Source: - Type: LocalVolume (a persistent volume backed by local storage on a node) - Path: /mnt/disks/vol1 - Events: - ``` - - The PV described above can be claimed and bound to a PVC by referencing the `local-fast` storageClassName. - -#### Option 2: Manually create local persistent volume - -See [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volumes/#local) -for an example PersistentVolume spec. - -### Step 4: Create local persistent volume claim - -``` yaml -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: example-local-claim -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 5Gi - storageClassName: local-storage -``` -Please replace the following elements to reflect your configuration: - - * "5Gi" with required size of storage volume - * "local-storage" with the name of storage class associated with the - local PVs that should be used for satisfying this PVC - -For "Block" volumeMode PVC, which tries to claim a "Block" PV, the following -example can be used: - -``` yaml -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: example-local-claim -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 5Gi - volumeMode: Block - storageClassName: local-storage -``` -Note that the only additional field of interest here is volumeMode, which has been set -to "Block". - ## E2E Tests ### Running @@ -290,38 +137,6 @@ Run `./hack/e2e.sh -h` to view help. Check testgrid [sig-storage-local-static-provisioner](https://testgrid.k8s.io/sig-storage-local-static-provisioner) dashboard. -## Best Practices - -* For IO isolation, a whole disk per volume is recommended -* For capacity isolation, separate partitions per volume is recommended -* Avoid recreating nodes with the same node name while there are still old PVs - with that node's affinity specified. Otherwise, the system could think that - the new node contains the old PVs. -* For volumes with a filesystem, it's recommended to utilize their UUID (e.g. - the output from `ls -l /dev/disk/by-uuid`) both in fstab entries - and in the directory name of that mount point. This practice ensures - that the wrong local volume is not mistakenly mounted, even if its device path - changes (e.g. if /dev/sda1 becomes /dev/sdb1 when a new disk is added). - Additionally, this practice will ensure that if another node with the - same name is created, that any volumes on that node are unique and not - mistaken for a volume on another node with the same name. -* For raw block volumes without a filesystem, use a unique ID as the symlink - name. Depending on your environment, the volume's ID in `/dev/disk/by-id/` - may contain a unique hardware serial number. Otherwise, a unique ID should be - generated. The uniqueness of the symlink name will ensure that if another - node with the same name is created, that any volumes on that node are - unique and not mistaken for a volume on another node with the same name. - - -### Deleting/removing the underlying volume - -When you want to decommission the local volume, here is a possible workflow. -1. Stop the pods that are using the volume -2. Remove the local volume from the node (ie unmounting, pulling out the disk, etc) -3. Delete the PVC -4. The provisioner will try to cleanup the volume, but will fail since the volume no longer exists -5. Manually delete the PV object - ## Community, discussion, contribution, and support Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/). diff --git a/TODO.md b/TODO.md index 96167b01e..b9942d570 100644 --- a/TODO.md +++ b/TODO.md @@ -6,7 +6,6 @@ ## P1 ## P2 -* Dynamic provisioning using lvm * Partitioning, formatting, and mount extensions * Refactor to just use informer's cache (and need to stub out API calls for unit testing) diff --git a/docs/best-practices.md b/docs/best-practices.md new file mode 100644 index 000000000..ae386ea56 --- /dev/null +++ b/docs/best-practices.md @@ -0,0 +1,21 @@ +# Best Practices + +* For IO isolation, a whole disk per volume is recommended +* For capacity isolation, separate partitions per volume is recommended +* Avoid recreating nodes with the same node name while there are still old PVs + with that node's affinity specified. Otherwise, the system could think that + the new node contains the old PVs. +* For volumes with a filesystem, it's recommended to utilize their UUID (e.g. + the output from `ls -l /dev/disk/by-uuid`) both in fstab entries + and in the directory name of that mount point. This practice ensures + that the wrong local volume is not mistakenly mounted, even if its device path + changes (e.g. if /dev/sda1 becomes /dev/sdb1 when a new disk is added). + Additionally, this practice will ensure that if another node with the + same name is created, that any volumes on that node are unique and not + mistaken for a volume on another node with the same name. +* For raw block volumes without a filesystem, use a unique ID as the symlink + name. Depending on your environment, the volume's ID in `/dev/disk/by-id/` + may contain a unique hardware serial number. Otherwise, a unique ID should be + generated. The uniqueness of the symlink name will ensure that if another + node with the same name is created, that any volumes on that node are + unique and not mistaken for a volume on another node with the same name. diff --git a/docs/faqs.md b/docs/faqs.md new file mode 100644 index 000000000..d7d1fd15e --- /dev/null +++ b/docs/faqs.md @@ -0,0 +1,51 @@ +# FAQs + +## Table of Contents + +- [I updated provisioner configuration but volumes are not discovered](#i-updated-provisioner-configuration-but-volumes-are-not-discovered) +- [I bind mounted a directory into sub-directory of discovery directory, but no PVs created](#i-bind-mounted-a-directory-into-sub-directory-of-discovery-directory-but-no-pvs-created) +- [Failed to start when docker --init flag is enabled.](#failed-to-start-when-docker---init-flag-is-enabled) + +## I updated provisioner configuration but volumes are not discovered + +Currently, provisioner will not reload configuration automatically. You need to restart them. + + +Check your local volume provisioners: + +``` +kubectl -n get pods -l app=local-volume-provisioner +``` + +Delete them: + +``` +kubectl -n delete pods -l app=local-volume-provisioner +``` + +Check new pods are created by daemon set, also please make sure they are running in the last. + +``` +kubectl -n get pods -l app=local-volume-provisioner +``` + +## I bind mounted a directory into sub-directory of discovery directory, but no PVs created + +Provisioner only creates local PVs for mounted directories in the first level +of discovery directory. This is because there is no way to detect whether the +sub-directory is created by system admin for discovering as local PVs or by +user applications. + +## Failed to start when docker --init flag is enabled. + +We need to mount `/dev` into container for device discovery, it conflicts when +`docker --init` flag is enabled because docker will mount [`init` process](https://docs.docker.com/engine/reference/run/#specify-an-init-process) at `/dev/init`. + +This has been fixed in +[moby/moby#37665](https://github.com/moby/moby/pull/37665). + +Workarounds before the fix is released: + +- do not use docker `--init`, packs [tini](https://github.com/krallin/tini) in your docker image +- do not use docker `--init`, [share process namespace](https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/) in your pod and use [pause:3.1+](https://github.com/kubernetes/kubernetes/blob/master/build/pause/CHANGELOG.md) to clean up orphaned zombie processes +- do not mount `/dev`, provisioner will discover current mounted devices, but it cannot discovery the newly mounted (see [why](https://github.com/kubernetes-incubator/external-storage/issues/783#issuecomment-395013458)) diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 000000000..f7ef16b08 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,209 @@ +## Getting started + +These instructions reflect the latest version of the codebase. For instructions +kn older versions, please see version links under +[Version Compatibility](../README.md#version-compatibility). + +## Table of Contents + +- [Step 1: Bringing up a cluster with local disks](#step-1-bringing-up-a-cluster-with-local-disks) + * [Enabling the alpha feature gates](#enabling-the-alpha-feature-gates) + + [1.10-1.12](#110-112) + * [Option 1: GCE](#option-1-gce) + * [Option 2: GKE](#option-2-gke) + * [Option 3: Baremetal environments](#option-3-baremetal-environments) + * [Option 4: Local test cluster](#option-4-local-test-cluster) +- [Step 2: Creating a StorageClass (1.9+)](#step-2-creating-a-storageclass-19) +- [Step 3: Creating local persistent volumes](#step-3-creating-local-persistent-volumes) + * [Option 1: Using the local volume static provisioner](#option-1-using-the-local-volume-static-provisioner) + * [Option 2: Manually create local persistent volume](#option-2-manually-create-local-persistent-volume) +- [Step 4: Create local persistent volume claim](#step-4-create-local-persistent-volume-claim) + +### Step 1: Bringing up a cluster with local disks + +#### Enabling the alpha feature gates + +##### 1.10-1.12 + +If raw local block feature is needed, +``` +$ export KUBE_FEATURE_GATES="BlockVolume=true" +``` + +Note: Kubernetes versions prior to 1.10 require [several additional +feature-gates](https://github.com/kubernetes-incubator/external-storage/tree/local-volume-provisioner-v2.0.0/local-volume#enabling-the-alpha-feature-gates) +be enabled on all Kubernetes components, because the persistent local volumes and other features were in alpha. + +#### Option 1: GCE + +GCE clusters brought up with kube-up.sh will automatically format and mount the +requested Local SSDs, so you can deploy the provisioner with the pre-generated +deployment spec and skip to [step 4](#step-4-create-local-persistent-volume-claim), +unless you want to customize the provisioner spec or storage classes. + +``` console +$ NODE_LOCAL_SSDS_EXT=,,fs cluster/kube-up.sh +$ kubectl create -f helm/generated-examples/gce.yaml +``` + +#### Option 2: GKE + +GKE clusters will automatically format and mount the +requested Local SSDs. Please see +[GKE +documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) +for instructions for how to create a cluster with Local SSDs. + +Then skip to [step 4](#step-4-create-local-persistent-volume-claim). + +**Note:** The raw block feature is only supported on GKE Kubernetes alpha clusters. + +#### Option 3: Baremetal environments + +1. Partition and format the disks on each node according to your application's + requirements. +2. Mount all the filesystems under one directory per StorageClass. The directories + are specified in a configmap, see below. +3. Configure the Kubernetes API Server, controller-manager, scheduler, and all kubelets + with `KUBE_FEATURE_GATES` as described [above](#enabling-the-alpha-feature-gates). +4. If not using the default Kubernetes scheduler policy, the following + predicates must be enabled: + * Pre-1.9: `NoVolumeBindConflict` + * 1.9+: `VolumeBindingChecker` + +#### Option 4: Local test cluster + +1. Create `/mnt/disks` directory and mount several volumes into its subdirectories. + The example below uses three ram disks to simulate real local volumes: +```console +$ mkdir /mnt/disks +$ for vol in vol1 vol2 vol3; do + mkdir /mnt/disks/$vol + mount -t tmpfs $vol /mnt/disks/$vol +done +``` + +2. Run the local cluster. + +```console +$ ALLOW_PRIVILEGED=true LOG_LEVEL=5 FEATURE_GATES=$KUBE_FEATURE_GATES hack/local-up-cluster.sh +``` + +### Step 2: Creating a StorageClass (1.9+) + +To delay volume binding until pod scheduling and to handle multiple local PVs in +a single pod, a StorageClass must to be created with `volumeBindingMode` set to +`WaitForFirstConsumer`. + +```console +$ kubectl create -f deployment/kubernetes/example/default_example_storageclass.yaml +``` + +### Step 3: Creating local persistent volumes + +#### Option 1: Using the local volume static provisioner + +1. Generate Provisioner's ServiceAccount, Roles, DaemonSet, and ConfigMap spec, and customize it. + + This step uses helm templates to generate the specs. See the [helm README](helm) for setup instructions. + To generate the provisioner's specs using the [default values](../helm/provisioner/values.yaml), run: + + ``` console + helm template ./helm/provisioner > deployment/kubernetes/provisioner_generated.yaml + ``` + + You can also provide a custom values file instead: + + ``` console + helm template ./helm/provisioner --values custom-values.yaml > deployment/kubernetes/provisioner_generated.yaml + ``` + +2. Deploy Provisioner + + Once a user is satisfied with the content of Provisioner's yaml file, **kubectl** can be used + to create Provisioner's DaemonSet and ConfigMap. + + ``` console + $ kubectl create -f deployment/kubernetes/provisioner_generated.yaml + ``` + +3. Check discovered local volumes + + Once launched, the external static provisioner will discover and create local-volume PVs. + + For example, if the directory `/mnt/disks/` contained one directory `/mnt/disks/vol1` then the following + local-volume PV would be created by the static provisioner: + + ``` + $ kubectl get pv + NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE + local-pv-ce05be60 1024220Ki RWO Delete Available local-storage 26s + + $ kubectl describe pv local-pv-ce05be60 + Name: local-pv-ce05be60 + Labels: + Annotations: pv.kubernetes.io/provisioned-by=local-volume-provisioner-minikube-18f57fb2-a186-11e7-b543-080027d51893 + StorageClass: local-fast + Status: Available + Claim: + Reclaim Policy: Delete + Access Modes: RWO + Capacity: 1024220Ki + NodeAffinity: + Required Terms: + Term 0: kubernetes.io/hostname in [my-node] + Message: + Source: + Type: LocalVolume (a persistent volume backed by local storage on a node) + Path: /mnt/disks/vol1 + Events: + ``` + + The PV described above can be claimed and bound to a PVC by referencing the `local-fast` storageClassName. + +#### Option 2: Manually create local persistent volume + +See [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volumes/#local) +for an example PersistentVolume spec. + +### Step 4: Create local persistent volume claim + +``` yaml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: example-local-claim +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + storageClassName: local-storage +``` +Please replace the following elements to reflect your configuration: + + * "5Gi" with required size of storage volume + * "local-storage" with the name of storage class associated with the + local PVs that should be used for satisfying this PVC + +For "Block" volumeMode PVC, which tries to claim a "Block" PV, the following +example can be used: + +``` yaml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: example-local-claim +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + volumeMode: Block + storageClassName: local-storage +``` +Note that the only additional field of interest here is volumeMode, which has been set +to "Block". + diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 000000000..42b120250 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,190 @@ +# Operations + +This document provides guides on how to manage local volumes in your Kubernetes +cluster. Before managing local volumes on your cluster nodes, here are some +configuration requirements you must know: + +* The local-volume plugin expects paths to be stable, including across + reboots and when disks are added or removed. +* The static provisioner only discovers either mount points (for Filesystem mode volumes) + or symbolic links (for Block mode volumes). For directory-based local volumes, they + must be bind-mounted into the discovery directories. + +Glossary: + +- _discovery directory_: Directory on the host from which provisioner will + discover both filesystem and block local PVs. +- _provisioner_: In our documents, provisioner indicates local-volume-provisioner program. +- *local PV*: Kubernetes local persistent volume. +- *filesystem local PV*: Local PV with Filesystem mode +- *block local PV*: Local PV with Block mode + +## Table of Contents + +- [Create a directory for provisioner discovering](#create-a-directory-for-provisioner-discovering) +- [Prepare and set up local volumes in discovery directory](#prepare-and-set-up-local-volumes-in-discovery-directory) + * [Use a whole disk as a filesystem PV](#use-a-whole-disk-as-a-filesystem-pv) + * [Sharing a disk filesystem by multiple filesystem PVs](#sharing-a-disk-filesystem-by-multiple-filesystem-pvs) + * [Link devices into directory to be discovered as block PVs](#link-devices-into-directory-to-be-discovered-as-block-pvs) + * [Separate disk into multiple partitions](#separate-disk-into-multiple-partitions) +- [Deleting/removing the underlying volume](#deletingremoving-the-underlying-volume) + +## Create a directory for provisioner discovering + +``` +$ sudo mkdir -p /mnt/disks +``` + +NOTE: + +- We use `/mnt/disks` as an example, but you can use any directory +- This directory is configured in `hostDir` field in provisioner configuration + and can only be configured for one storage class +- If you want to configure more than one local storage class, create one + directory for each storage class + +## Prepare and set up local volumes in discovery directory + +After you prepared discovery directory, you can set up local +volumes to be discovered by provisioner. + +### Use a whole disk as a filesystem PV + +If you attached a disk onto your machine (e.g `/dev/path/to/disk`). You +can format and mount it into discovery directory with the following commands: + +1) Format and mount + +``` +$ sudo mkfs.ext4 /dev/path/to/disk +$ DISK_UUID=$(blkid -s UUID -o value /dev/path/to/disk) +$ sudo mkdir /mnt/disks/$DISK_UUID +$ sudo mount -t ext4 /dev/path/to/disk /mnt/disks/$DISK_UUID +``` + +2) Persistent mount entry into /etc/fstab + +``` +$ echo UUID=`sudo blkid -s UUID -o value /dev/path/to/disk` /mnt/disks/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab +``` + +NOTE: + +- We use `/dev/path/to/disk` as a disk example, change it to real path of your + device. +- You can also adjust filesystem and mount options as you wish +- It's best practice to use UUID both in fstab entries and the directory name + of mount point, see our [best practices](best-practices.md). +- Using a whole disk is best practice if you need IO isolation +- On some cloud platforms, they may provide mechanism to format and mount local + disks automatically which is recommended. Please refer to cloud platform + documentation. + +### Sharing a disk filesystem by multiple filesystem PVs + +Instead of mount root of disk filesystem into discovery directory, you can +create multiple directory in disk, and bind mount them into discovery +directory. By doing this, a disk can be shared by multiple local filesystem +PVs. Here is an example: + +1) Format and mount + +``` +$ sudo mkfs.ext4 /dev/path/to/disk +$ DISK_UUID=$(blkid -s UUID -o value /dev/path/to/disk) +$ sudo mkdir /mnt/$DISK_UUID +$ sudo mount -t ext4 /dev/path/to/disk /mnt/$DISK_UUID +``` + +NOTE: we should not mount disk into discovery directory. + +2) Persistent mount entry into /etc/fstab + +``` +$ echo UUID=`sudo blkid -s UUID -o value /dev/path/to/disk` /mnt/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab +``` + +3) Create multiple directories and bind mount them into discovery directory + +``` +for i in $(seq 1 10); do + sudo mkdir -p /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} + sudo mount --bind /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} +done +``` + +4) Persistent bind mount entries into /etc/fstab + +``` +for i in $(seq 1 10); do + echo /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} none bind 0 0 | sudo tee -a /etc/fstab +done +``` + +NOTE: + +- Local PVs sharing one disk filesystem will have same capacity and will have + no capacity isolation. If you want to separate a disk into multiple PVs with + capacity isolation. You can separate disk into multiple + partitions](#separate-disk-into-multiple-partitions) first. + +### Link devices into directory to be discovered as block PVs + +If you want to use block devices directly, you can simply link them into +discovery directory. + +For safety, you must use the unique path of device. + +Find unique path of device: + +``` +$ ls -l /dev/disk/by-id/ +lrwxrwxrwx 1 root root 9 Apr 18 14:26 lvm-pv-uuid-kdWgMJ-OOfq-ox5N-ie4E-NU2h-8zPJ-edX1Og -> ../../sde +lrwxrwxrwx 1 root root 9 Apr 18 14:26 lvm-pv-uuid-VqD1G2-upe2-Xnek-PdXD-mkOT-LhSv-rUV2is -> ../../sdc +lrwxrwxrwx 1 root root 9 Apr 18 14:26 lvm-pv-uuid-yyTnct-TpUS-U93g-JoFs-6seh-Yy29-Dn6Irf -> ../../sdb +``` + +For example, if you want to use `/dev/sdb`, you must link +`/dev/disk/by-id/lvm-pv-uuid-yyTnct-TpUS-U93g-JoFs-6seh-Yy29-Dn6Irf` not +`/dev/sdb`. + +Link it into discovery directory: + +``` +$ sudo ln -s /dev/disk/by-id/lvm-pv-uuid-yyTnct-TpUS-U93g-JoFs-6seh-Yy29-Dn6Irf /mnt/disks +``` + +### Separate disk into multiple partitions + +You can use [parted](https://www.gnu.org/s/parted/manual/parted.html) or other +tools to separate your disk into multiple partitions. This helps to isolate +capacity. Here is an example: + +``` +sudo parted --script /dev/path/to/disk \ + mklabel gpt \ + mkpart primary 1MiB 1000MiB \ + mkpart primary 1000MiB 2000MiB \ + mkpart primary 2000MiB 3000MiB \ + mkpart primary 3000MiB 4000MiB + +sudo parted /dev/path/to/disk print +``` + +NOTE: + +- Partition disk is dangerous, please check your command carefully, use at your own risk +- Adjust arguments according to your needs, refer to [parted manual](https://www.gnu.org/s/parted/manual/parted.html) + +After disks are partitioned, for each partition, you can follow above operation +guide to format and discover as filesystem volume or use as block device. + +## Deleting/removing the underlying volume + +When you want to decommission the local volume, here is a possible workflow. + +1. Stop the pods that are using the volume +2. Remove the local volume from the node (ie unmounting, pulling out the disk, etc) +3. Delete the PVC +4. The provisioner will try to cleanup the volume, but will fail since the volume no longer exists +5. Manually delete the PV object diff --git a/docs/provisioner.md b/docs/provisioner.md index 4774e6f1b..d2f0469d4 100644 --- a/docs/provisioner.md +++ b/docs/provisioner.md @@ -1,9 +1,7 @@ # local-volume-provisioner -`quay.io/external-storage/local-volume-provisioner:1.0.0` - -local-volume-provisioner is an out-of-tree static provisioner for the local volume -plugin, which is a 1.7 & 1.8 alpha feature. +local-volume-provisioner is an out-of-tree static provisioner for the +Kubernetes [local volume](https://kubernetes.io/docs/concepts/storage/volumes/#local), which is GA feature since 1.14. It runs on each node in the cluster and monitors specified directories to look for new local file-based volumes. The volumes can be a mount point or a directory in @@ -11,19 +9,15 @@ a shared filesystem. It then statically creates a Local PersistentVolume for ea local volume. It also monitors when the PersistentVolumes have been released, and will clean up the volume, and recreate the PV. -## [Changelog](CHANGELOG.md) +## Table of Contents -## Development +- [Changelog](#changelog) +- [Design](#design) +- [Prometheus Metrics](#prometheus-metrics) -Compile the provisioner -``` console -make -``` +## Changelog -Make the container image and push to the registry -``` console -make push -``` +See [CHANGELOG.md](../CHANGELOG.md). ## Design diff --git a/helm/README.md b/helm/README.md index 35fd24c92..67162fa97 100644 --- a/helm/README.md +++ b/helm/README.md @@ -53,12 +53,13 @@ provisioner chart and their default values. | --- | --- | --- | --- | | common.rbac | Generating RBAC (Role Based Access Control) objects. | bool | `true` | | common.namespace | Namespace where provisioner runs. | str | `default` | -| common.createNamespace | Whether to create namespace for provisioner. | bool | `false` | +| common.createNamespace | Whether to create namespace for provisioner. | bool | `false` | | common.useAlphaAPI | If running against pre-1.10 k8s version, the `useAlphaAPI` flag must be enabled. | bool | `false` | | common.useJobForCleaning | If set to true, provisioner will use jobs-based block cleaning. | bool | `false` | | common.useNodeNameOnly | If set to true, provisioner name will only use Node.Name and not Node.UID. | bool | `false` | | common.minResyncPeriod | Resync period in reflectors will be random between `minResyncPeriod` and `2*minResyncPeriod`. | str | `5m0s` | | common.configMapName | Provisioner ConfigMap name. | str | `local-provisioner-config` | +| common.podSecurityPolicy | Whether to create pod security policy or not. | bool | `false` | | classes.[n].name | StorageClass name. | str | `-` | | classes.[n].hostDir | Path on the host where local volumes of this storage class are mounted under. | str | `-` | | classes.[n].mountDir | Optionally specify mount path of local volumes. By default, we use same path as hostDir in container. | str | `-` | @@ -81,6 +82,7 @@ provisioner chart and their default values. | prometheus.operator.serviceMonitor.interval | Interval at which Prometheus scrapes the provisioner | str | `10s` | | prometheus.operator.serviceMonitor.namespace | The namespace Prometheus is installed in | str | `monitoring` | | prometheus.operator.serviceMonitor.selector | The Prometheus selector label | map | `prometheus: kube-prometheus` | + Note: `classes` is a list of objects, you can specify one or more classes. ## Examples