Lift the 8GB etcd limit to 100GB

https://www.cncf.io/blog/2019/05/09/performance-optimization-of-etcd-in-web-scale-data-scenario/

Signed-off-by: Ronald Ngounou <[email protected]>

Author: ronaldngounou

content/en/blog/2023/how_to_debug_large_db_size_issue.md

@@ -8,7 +8,7 @@ draft: false
 # Background
 Users can configure the quota of the backend db size using flag `--quota-backend-bytes`. It's the max number of bytes
 the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the
-suggested max value is 8GB.
+suggested max value is 100GB.
 
 2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message `etcdserver: mvcc: database space exceeded`
 when trying to write more data, and see alarm "NOSPACE" (see example below) when checking the endpoint status or health state. It would be better to figure out whether it's expected. It's exactly the reason why I provide this guide.
@@ -248,8 +248,8 @@ to do the similar analysis(see example below). Note that you can build the binar
 If the behavior (db size exceeds the quota) is expected, then you can set a bigger value for `--quota-backend-bytes`.
 You need to make sure your cloud provider supports this, otherwise the manual update might not survive across cluster
 upgrading or VM recreating. **Note that etcd (actually boltDB) maps the db file into memory directly, so a larger value
-also means more memory usage**. Just I mentioned in the beginning of this post, the suggested max value is 8GB. Of course,
-If your VM has big memory (e.g. 64GB), it's OK to set a value > 8GB.
+also means more memory usage**. Just I mentioned in the beginning of this post, the suggested max value is 100GB. Of course,
+If your VM has big memory (e.g. 64GB), it's OK to set a value > 100GB.
 
 The other solution is to set per-resource etcd servers overrides using apiserver flag `--etcd-servers-overrides`.
 In the following example, there are two etcd clusters; one for the normal usage, and the other dedicated to events.

content/en/docs/v3.1/dev-guide/limit.md

@@ -8,5 +8,5 @@ etcd is designed to handle small key value pairs typical for metadata. Larger re
 
 ## Storage size limit
 
-The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag; supports up to 8GB.
+The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag; supports up to 100GB.
 

content/en/docs/v3.1/faq.md

@@ -20,7 +20,7 @@ title: Frequently Asked Questions (FAQ)
 
 #### System requirements
 
-Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a 2GB default storage size quota, configurable up to 8GB. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
+Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a 2GB default storage size quota, configurable up to 100GB. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
 
 Most stable production environment is Linux operating system with amd64 architecture; see [supported platform][supported-platform] for more.
 

content/en/docs/v3.1/op-guide/hardware.md

@@ -12,7 +12,7 @@ Heavily loaded etcd deployments, serving thousands of clients or tens of thousan
 
 ## Memory
 
-etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 8GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
+etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 100GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
 
 
 ## Disks

content/en/docs/v3.2/dev-guide/limit.md

@@ -8,5 +8,5 @@ etcd is designed to handle small key value pairs typical for metadata. Larger re
 
 ## Storage size limit
 
-The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag; supports up to 8GB.
+The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag; supports up to 100GB.
 

content/en/docs/v3.2/faq.md

@@ -20,7 +20,7 @@ title: Frequently Asked Questions (FAQ)
 
 #### System requirements
 
-Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a 2GB default storage size quota, configurable up to 8GB. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
+Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a 2GB default storage size quota, configurable up to 100GB. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
 
 Most stable production environment is Linux operating system with amd64 architecture; see [supported platform][supported-platform] for more.
 

content/en/docs/v3.2/op-guide/hardware.md

@@ -12,7 +12,7 @@ Heavily loaded etcd deployments, serving thousands of clients or tens of thousan
 
 ## Memory
 
-etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 8GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
+etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 100GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
 
 
 ## Disks

content/en/docs/v3.3/dev-guide/limit.md

@@ -8,4 +8,4 @@ etcd is designed to handle small key value pairs typical for metadata. Larger re
 
 ## Storage size limit
 
-The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.
+The default storage size limit is 2GB, configurable with `--quota-backend-bytes` flag. 100GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

content/en/docs/v3.3/faq.md

@@ -24,7 +24,7 @@ A member's advertised peer URLs come from `--initial-advertise-peer-urls` on ini
 
 ### System requirements
 
-Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a configurable storage size quota set to 2GB by default. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
+Since etcd writes data to disk, SSD is highly recommended. To prevent performance degradation or unintentionally overloading the key-value store, etcd enforces a configurable storage size quota set to 2GB by default. To avoid swapping or running out of memory, the machine should have at least as much RAM to cover the quota. 100GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it. At CoreOS, an etcd cluster is usually deployed on dedicated CoreOS Container Linux machines with dual-core processors, 2GB of RAM, and 80GB of SSD *at the very least*. **Note that performance is intrinsically workload dependent; please test before production deployment**. See [hardware][hardware-setup] for more recommendations.
 
 Most stable production environment is Linux operating system with amd64 architecture; see [supported platform][supported-platform] for more.
 

content/en/docs/v3.3/op-guide/hardware.md

@@ -12,7 +12,7 @@ Heavily loaded etcd deployments, serving thousands of clients or tens of thousan
 
 ## Memory
 
-etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 8GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
+etcd has a relatively small memory footprint but its performance still depends on having enough memory. An etcd server will aggressively cache key-value data and spends most of the rest of its memory tracking watchers. Typically 100GB is enough. For heavy deployments with thousands of watchers and millions of keys, allocate 16GB to 64GB memory accordingly.
 
 
 ## Disks