Update readmes (#18)

Author: efritz

LICENSE

@@ -1,5 +1,5 @@
 The Sourcegraph Enterprise license (the “Enterprise License”)
-Copyright (c) 2018 Sourcegraph Inc.
+Copyright (c) 2021 Sourcegraph Inc.
 
 With regard to the Sourcegraph Software:
 

README.md

@@ -1,42 +1,24 @@
 # Terraform module for Sourcegraph executors (AWS)
 
-This repository provides a [Terraform module](https://learn.hashicorp.com/tutorials/terraform/module-use?in=terraform/modules) to provision Sourcegraph executor compute resources on AWS.
+This repository provides a [Terraform module](https://learn.hashicorp.com/tutorials/terraform/module-use?in=terraform/modules) to provision [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors) compute resources on AWS. If you are installing executors for the first time, [follow our complete setup guide](https://docs.sourcegraph.com/admin/deploy_executors).
 
 ![Infrastructure overview](https://raw.githubusercontent.com/sourcegraph/terraform-aws-executors/master/images/infrastructure.png)
 
-## Usage examples
+This repository provides four submodules:
 
-There are several examples in the [`examples`](./examples) directory.
+1. The [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/executors) provisions compute resources for executors.
+1. The [docker-mirror module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/docker-mirror) provisions a Docker registry pull-through cache.
+1. The [networking module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/networking) provisions a network to be shared by the executor and Docker registry resources.
+1. The [credentials module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/credentials) provisions credentials required by the Sourcegraph instance to enable observability and auto-scaling of executors.
 
-#### [`examples/single-executor`](./examples/single-executor)
+The [multiple-executors example](https://github.com/sourcegraph/terraform-aws-executors/blob/master/examples/multiple-executors) uses the submodule directly to provision multiple executor resource groups performing different types of work. Follow this example if you are:
 
-This example uses the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest) that provisions a network, a Docker registry mirror, and a set of resources to run _one_ type of executor. To provision more than one type of executor (multiple queues or multiple environments), see the following `multiple-executors` example.
+1. Provisioning executors for use with multiple features (e.g., both [auto-indexing](https://docs.sourcegraph.com/code_intelligence/explanations/auto_indexing) and [server-side batch changes](https://docs.sourcegraph.com/batch_changes/explanations/server_side)), or
+1. Provisioning resources for multiple Sourcegraph instances (e.g., test, prod)
 
-The following variables must be supplied:
+This repository also provides a [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest) combining the executors, network, and docker-mirror resources into an easier to use package.
 
-**Note**: these variables must be set in tandem with changes to the target [Sourcegraph deployment](https://docs.sourcegraph.com/admin/deploy_executors).
-
-- `executor_sourcegraph_external_url`: The URL from which the target Sourcegraph instance is accessible from the executor instances.
-- `executor_sourcegraph_executor_proxy_password`: The shared executor password defined in Sourcegraph. The value must be the same as the `executors.accessToken` site setting described in [Configuring executors and instance communication](https://docs.sourcegraph.com/admin/deploy_executors#configuring-executors-and-instance-communication).
-- `executor_queue_name`: The name of the target queue to process (e.g., `codeintel`, `batches`).
-- `executor_metrics_environment_label`: The name of the target environment (e.g., `staging`, `prod`). This value must be the same as the `EXECUTOR_METRIC_ENVIRONMENT_LABEL` environment variable as described in [Configuring auto scaling](https://docs.sourcegraph.com/admin/deploy_executors#aws).
-- `executor_instance_tag`: Compute instances are tagged by this value by the key `executor_tag`. We recommend this value take the form `{executor_queue_name}-{executor_metrics_environment_label}`. This value must be the same as `INSTANCE_TAG` as described in [Configuring observability](https://docs.sourcegraph.com/admin/deploy_executors#aws-1).
-
-All of this module's variables are defined in [variables.tf](./variables.tf).
-
-#### [`examples/multiple-executors`](./examples/multiple-executors)
-
-This example uses [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/networking), [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/docker-mirror), and [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/executors) submodules that provision a network, a Docker registry mirror, and sets of resources running one or more types of executors.
-
-The following variables must be supplied:
-
-- `sourcegraph_external_url`, `sourcegraph_executor_proxy_password`, `queue_name`, `metrics_environment_label`, and `instance_tag`: Analogous to the `executor_*` variables in the `single-executor` example.
-- `resource_prefix`: A prefix unique to each set of compute resources. This prevents collisions between two uses of the `executors` module. We recommend this value be constructed the same way `instance_tag` is constructed.
-- `docker_registry_mirror`: This variable is given the value `"http://${module.docker-mirror.ip_address}:5000"`, which converts the raw external IP address to an address resolvable by the executor instances.
-
-If your deployment environment already has a Docker registry that can be used, only the `executor` submodule must be used (and references to the `networking` and `docker-mirror` modules can be dropped). The Docker registry mirror address can be supplied along with its containing vcp and subnet as pre-existing identifier literals.
-
-All of these module's variables are defined in [modules/networking/variables.tf](./modules/networking/variables.tf), [modules/docker-mirror/variables.tf](./modules/docker-mirror/variables.tf), and [modules/executors/variables.tf](./modules/executors/variables.tf).
+The [single-executor example](https://github.com/sourcegraph/terraform-aws-executors/blob/master/examples/single-executor) uses the root module to provision a single executor type. Follow this example if you are deploying to a single Sourcegraph instance and using a single executors-backed feature.
 
 ## Requirements
 

examples/multiple-executors/README.md

@@ -0,0 +1,13 @@
+# Multiple executor example
+
+This example uses [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/networking), [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/docker-mirror), and [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/executors) submodules that provision a network, a Docker registry mirror, and sets of resources running one or more types of executors.
+
+The following variables must be supplied:
+
+- `sourcegraph_external_url`, `sourcegraph_executor_proxy_password`, `queue_name`, `metrics_environment_label`, and `instance_tag`: Analogous to the `executor_*` variables in the `single-executor` example.
+- `resource_prefix`: A prefix unique to each set of compute resources. This prevents collisions between two uses of the `executors` module. We recommend this value be constructed the same way `instance_tag` is constructed.
+- `docker_registry_mirror`: This variable is given the value `"http://${module.docker-mirror.ip_address}:5000"`, which converts the raw external IP address to an address resolvable by the executor instances.
+
+If your deployment environment already has a Docker registry that can be used, only the `executor` submodule must be used (and references to the `networking` and `docker-mirror` modules can be dropped). The Docker registry mirror address can be supplied along with its containing VPC and subnet as pre-existing identifier literals.
+
+All of these module's variables are defined in [modules/networking/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/master/modules/networking/variables.tf), [modules/docker-mirror/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/master/modules/docker-mirror/variables.tf), and [modules/executors/variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/master/modules/executors/variables.tf).

examples/single-executor/README.md

@@ -0,0 +1,15 @@
+# Single executor example
+
+This example uses the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest) that provisions a network, a Docker registry mirror, and a set of resources to run _one_ type of executor. To provision more than one type of executor (multiple queues or multiple environments), see the following `multiple-executors` example.
+
+The following variables must be supplied:
+
+**Note**: these variables must be set in tandem with changes to the target [Sourcegraph deployment](https://docs.sourcegraph.com/admin/deploy_executors).
+
+- `executor_sourcegraph_external_url`: The URL from which the target Sourcegraph instance is accessible from the executor instances.
+- `executor_sourcegraph_executor_proxy_password`: The shared executor password defined in Sourcegraph. The value must be the same as the `executors.accessToken` site setting described in [Configuring executors and instance communication](https://docs.sourcegraph.com/admin/deploy_executors#configuring-executors-and-instance-communication).
+- `executor_queue_name`: The name of the target queue to process (e.g., `codeintel`, `batches`).
+- `executor_metrics_environment_label`: The name of the target environment (e.g., `staging`, `prod`). This value must be the same as the `EXECUTOR_METRIC_ENVIRONMENT_LABEL` environment variable as described in [Configuring auto scaling](https://docs.sourcegraph.com/admin/deploy_executors#aws).
+- `executor_instance_tag`: Compute instances are tagged by this value by the key `executor_tag`. We recommend this value take the form `{executor_queue_name}-{executor_metrics_environment_label}`. This value must be the same as `INSTANCE_TAG` as described in [Configuring observability](https://docs.sourcegraph.com/admin/deploy_executors#aws-1).
+
+All of this module's variables are defined in [variables.tf](https://github.com/sourcegraph/terraform-aws-executors/blob/master/variables.tf).

main.tf

@@ -45,5 +45,5 @@ module "aws-executor" {
   max_replicas                        = var.executor_max_replicas
   jobs_per_instance_scaling           = var.executor_jobs_per_instance_scaling
   metrics_environment_label           = var.executor_metrics_environment_label
-  docker_registry_mirror              = var.executor_docker_registry_mirror
+  docker_registry_mirror              = "http://${var.docker_mirror_static_ip}:5000"
 }

modules/credentials/README.md

@@ -0,0 +1,7 @@
+# Credentials module
+
+This module can be optionally used to create the IAM user policies required to configure auto-scaling and observability of [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors) in AWS.
+
+Auto-scaling requires that the executor compute instances have permissions to emit CloudWatch metrics. As outlined in [how to configure auto scaling](https://docs.sourcegraph.com/admin/deploy_executors#aws), the Sourcegraph `worker` service must set the `EXECUTOR_METRIC_AWS_ACCESS_KEY_ID` and `EXECUTOR_METRIC_AWS_SECRET_ACCESS_KEY` environment variables to be the same as the `metric_writer_access_key_id` and `metric_writer_secret_key` values provided by running this module.
+
+Observability of executor compute resources require that the target Sourcegraph instance's Prometheus have permissions to scrape the executor compute resources. As outlined in [how to configure observability](https://docs.sourcegraph.com/admin/deploy_executors#aws-1), the `instance_scraper_access_key_id` and `instance_scraper_access_secret_key` values provided by running this module must be supplied to the Sourcegraph Prometheus instance.

modules/docker-mirror/README.md

@@ -0,0 +1,5 @@
+# Docker mirror module
+
+This module provides a hosted Docker registry pull-through cache to be used by [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors). It is strongly recommended to deploy a Docker mirror as a cache to reduce rate limiting by the public [Docker Hub registry](https://hub.docker.com/). We have also seen deploying a Docker mirror in the same physical zone as the executors significantly decreased latencies during image pulls.
+
+When using the sibling [executors module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/executors), the `vpc_id` and `subnet_id` values must match and the executor module `docker_registry_mirror` value should match `"http://${static_ip}:5000"`.

modules/executors/README.md

@@ -0,0 +1,13 @@
+# Executors module
+
+This module provides the resources to provision [Sourcegraph executor](https://docs.sourcegraph.com/admin/executors) compute resources on AWS. For a high-level overview of the resources deployed by this module, see the [root module](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest). This module includes the following resources:
+
+- AWS launch template
+- AWS autoscaler and autoscaling policy
+- CloudWatch log group
+- CloudWatch metric alarms
+- Security groups and IAM role policy attachments
+
+This module does **not** automatically create networking or Docker mirror resources. The `vpc_id`, `subnet_id`, and `docker_registry_mirror` variables must be supplied explicitly with resources that have been previously created.
+
+This module is often used with the sibling modules that create [networking](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/networking) and [Docker mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/docker-mirror) resources which can be shared by multiple instances of the executor module (listening to different queues or being deployed in a different environment).

modules/networking/README.md

@@ -0,0 +1,5 @@
+# Networking module
+
+This module provides the networking glue between the sibling [executors](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/executors) and [docker-mirror](https://registry.terraform.io/modules/sourcegraph/executors/aws/latest/submodules/docker-mirror) modules.
+
+_(There's very little to talk about in here.)_

variables.tf

@@ -164,9 +164,3 @@ variable "executor_metrics_environment_label" {
   type        = string
   description = "The value for environment by which to filter the custom metrics."
 }
-
-variable "executor_docker_registry_mirror" {
-  type        = string
-  default     = ""
-  description = "A URL to a docker registry mirror to use (falling back to docker.io)."
-}