MWI: Documentation for Bound Keypair static keys (#59854)

* MWI: Documentation for Bound Keypair static keys

This adds documentation and a guide for bound keypair static keys,
added in #59620.

* Fill in remaining sections

* Tweak warning message

* Address review comments

* Fix excessive whitespace

* Address review feedback

* Further tweak private key storage wording

* Fix broken link to bound keypair guide

Author: timothyb89

docs/pages/machine-workload-identity/machine-id/deployment/deployment.mdx

@@ -42,7 +42,7 @@ authentication methods:
 
 ## Deployment guides
 
-The guides in this section show you how to deploy Machine ID and join it to your cluster. 
+The guides in this section show you how to deploy Machine ID and join it to your cluster.
 Choose a guide based on the platform where you intend to run Machine ID.
 
 If a specific guide does not exist for your platform, the [Linux
@@ -114,9 +114,9 @@ and [Architecture](../../../reference/architecture/machine-id-architecture.mdx)
     },
     {
       icon: <Icon name="lock" size="xl" />,
-      to: "../../reference/machine-workload-identity/machine-id/bound-keypair/getting-started",
+      to: "../../../reference/machine-workload-identity/machine-id/bound-keypair/getting-started",
       name: "Bound Keypair Joining",
-    }  
+    }
   ]}
 />
 
@@ -165,6 +165,18 @@ Read the following guides for how to deploy Machine ID on a continuous integrati
       icon: <Icon name="terraform" size="xl" />,
       to: "../../zero-trust-access/infrastructure-as-code/terraform-provider/terraform-cloud",
       name: "Terraform Cloud",
+    },
+    {
+      icon: <Icon name="lock" size="xl" />,
+      to: "../../../reference/machine-workload-identity/machine-id/bound-keypair/static-keys",
+      name: "Bound Keypair static keys (Generic)",
     }
   ]}
 />
+
+<Admonition type="tip" title="Unsupported Provider?">
+If your CI/CD provider does not have a dedicated join method listed above,
+consider using [Bound Keypair static keys][bound-keypair-static] as a fallback.
+</Admonition>
+
+[bound-keypair-static]: ../../../reference/machine-workload-identity/machine-id/bound-keypair/static-keys.mdx

docs/pages/reference/cli/tbot.mdx

@@ -288,7 +288,9 @@ specific section for details when using a YAML config file or legacy output.
 | `--[no-]oneshot`        | If set, quit after the first renewal. |
 | `--diag-addr`           | If set and the bot is in debug mode, a diagnostics service will listen on specified address. |
 | `--storage`             | A destination URI for tbot's internal storage, e.g. `file:///foo/bar`. See [Destination URIs](#destination-uris) for more info. |
-| `--registration-secret` | An optional joining secret to use on first join with the `bound_keypair` join method. |
+| `--registration-secret` | An optional joining secret to use on first join with the `bound_keypair` join method. This can also be provided via the `TBOT_REGISTRATION_SECRET` environment variable. |
+| `--registration-secret-path` | An optional path to a file containing a joining secret to use on first join with the `bound_keypair` join method. |
+| `--static-key-path`     | An optional path to a file containing a static private key for use with the `bound_keypair` join method. A base64-encoded key can also be provided via the `TBOT_BOUND_KEYPAIR_STATIC_KEY` environment variable. |
 
 ## tbot start legacy
 
@@ -727,9 +729,12 @@ new keypair.
 
 | Flag | Description |
 |------|-------------|
-| `--storage`      | A destination URI to be used for bot internal storage. Required. |
-| `--proxy-server` | A Teleport Proxy Service address. Required. |
-| `--overwrite`    | If set, always generate a new key. If unset, the existing public key will be printed if one already exists in the destination specified by `--storage` |
+| `--storage`         | A destination URI to be used for bot internal storage. Required. |
+| `--proxy-server`    | A Teleport Proxy Service address. Required. |
+| `--overwrite`       | If set, always generate a new key. If unset, the existing public key will be printed if one already exists in the destination specified by `--storage`. |
+| `--static`          | If set, generates a static keypair. For more information, see [the static key guide](../machine-workload-identity/machine-id/bound-keypair/static-keys.mdx). |
+| `--static-key-path` | If set with `--static`, writes the static keypair to a file. |
+| `--format`          | Override the output format, supported values: `text`, `json`. |
 
 ### Examples
 

docs/pages/reference/machine-workload-identity/machine-id/bound-keypair/admin-guide.mdx

@@ -274,10 +274,65 @@ its previous value.
 
 If [bot keypairs were rotated](#requesting-a-keypair-rotation) between the
 snapshot and restore of the Teleport cluster, note that bots only keep a record
-of the previous 10 keypairs. This means server-side recovery may impossible if
-the keypair expected by the restored Teleport cluster has been rotated out of
+of the previous 10 keypairs. This means server-side recovery may be impossible
+if the keypair expected by the restored Teleport cluster has been rotated out of
 the client-side history, or if the client-side history has been lost or deleted.
 
+## Manually rotating static keys
+
+Static keys prevent automatic key rotation as the `tbot` client cannot update
+keys in an arbitrary keystore. However, it may still possible to automate
+rotation if your environment or secret store allows you to update secrets
+through an API.
+
+The specific steps needed to automate these will vary based on your environment,
+but the general steps are:
+
+1. Generate a new keypair on any node using the tbot client:
+
+   ```code
+   $ tbot keypair create --proxy-server example.teleport.sh:443 --static --format json
+   ```
+
+2. Parse the `.public_key` and `.private_key` values using your tool of choice,
+   like `jq` or any other JSON parser.
+
+3. Replace the token in Teleport to trust the new public key, using the value in
+   the `.public_key` field:
+   ```code
+   $ cat my-token.yaml
+   version: v2
+   kind: token
+   metadata:
+     name: my-token
+   spec:
+     bot_name: example-bot
+     bound_keypair:
+       onboarding:
+         initial_public_key: <insert .public_key value here>
+       recovery:
+         mode: insecure
+     join_method: bound_keypair
+     roles:
+     - Bot
+   $ tctl create -f my-token.yaml
+   ```
+
+3. Insert the new private key into your keystore. This will vary depending on
+   which keystore or provider you are using.
+   * If passing the private key via an environment variable, copy the value directly
+   * If passing the private key via file, decode the base64-encoded private key first:
+     ```code
+     $ tbot keypair create --proxy-server example.teleport.sh:443 --static --format json | jq -r .private_key | base64 -d
+     ```
+     ...and store the result as needed.
+
+4. Future jobs should now use the new keypair.
+
+Frequently rotating static keys can help to mitigate the security tradeoffs of
+`insecure` recovery. See the [concepts page](concepts.mdx#static-keys) for more
+information about static keys.
+
 [rbac]: ../../../access-controls/roles.mdx
 [ephemeral]: ../../../architecture/machine-id-architecture.mdx#ephemeral-token
 [delegated]: ../../../deployment/join-methods.mdx#delegated-join-methods

docs/pages/reference/machine-workload-identity/machine-id/bound-keypair/bound-keypair.mdx

@@ -45,10 +45,10 @@ a bot initially. When ready to deploy to production, it's trivial to adjust
 onboarding and recovery settings to select your desired balance between
 resiliency and security.
 
-Additionally, with [`insecure` recovery](admin-guide.mdx#disabling-join-state-verification)
-and in situations that can accommodate the security complications, Bound Keypair
-Joining can be used to join bots in otherwise unsupported CI/CD providers by
-persisting the bot's keypair in the platform keystore.
+Additionally, with [static keys](static-keys.mdx) and in situations that can
+accommodate the security complications, Bound Keypair Joining can be used to
+join bots in otherwise unsupported CI/CD providers by persisting the bot's
+keypair in a platform keystore.
 
 ## Limitations
 
@@ -75,6 +75,9 @@ You can read step-by-step guides on using Bound Keypair Joining with Machine ID:
 
 - [Using Machine ID with Bound Keypair Joining](./getting-started.mdx):
   How to install and configure Machine ID with Bound Keypair Joining
+- [Using Machine ID with Bound Keypair static keys](./static-keys.mdx): How to
+  use Bound Keypair static keys with stateless hosts, like otherwise
+  unsupported CI/CD providers
 - [Bound Keypair Joining Concepts](./concepts.mdx): Learn more about the
   components and architecture of Bound Keypair Joining
 - [Bound Keypair Joining Admin Guide](./admin-guide.mdx): Learn how to deploy

docs/pages/reference/machine-workload-identity/machine-id/bound-keypair/concepts.mdx

@@ -118,6 +118,10 @@ values:
   [See the admin guide](admin-guide.mdx#disabling-join-state-verification) for
   more information on using `insecure` mode in practice.
 
+  Static key joining (`tbot keypair create --static ...`) requires use of
+  `insecure` mode as it disables client-side credential storage, meaning bots
+  cannot complete [join state verification](#join-state-verification).
+
 ## Join state verification
 
 A **join state document** is an additional piece of information provided to
@@ -145,5 +149,42 @@ credentials in use by the attacker.
 Note that join state verification is disabled when the token's
 `spec.bound_keypair.recovery.mode` is set to `insecure`.
 
+## Static Keys
+
+Static keys trade some security guarantees for flexibility, and can help enable
+use of `tbot` in environments where bots otherwise could not authenticate, such
+as:
+- CI/CD providers for which Teleport has no dedicated join method
+- Ephemeral bare-metal nodes without TPMs
+- Any other environment where persistent storage is not available
+- Any environment where an arbitrary number of instances may join, like
+  ephemeral CI runners.
+
+However, static keys have disadvantages:
+- `insecure` mode means any client with the private key can join without
+  restrictions. If the key is stolen, the attacker will have access to Teleport
+  and any resources the bot is allowed to access.
+- `insecure` mode additionally means you cannot limit the number of uses of the
+  bound keypair token (`spec.bound_keypair.recovery.limit`).
+- Keypair rotation is not supported, as client storage is not writable, and
+  attempting to require a rotation will prevent bots from joining.
+
+When using static keys, a keypair is generated ahead of time and [preregistered
+with Teleport](#onboarding). You can then provide the keypair to the `tbot`
+client using a file or the `TBOT_BOUND_KEYPAIR_STATIC_KEY` environment variable.
+
+The `insecure` recovery mode is required when using static keys (with
+`tbot keypair create --static ...`) as this mode disables client-side storage
+for join state documents. With no previous state, bots will not be able to
+complete join state verification, and will be unable to recover after their
+first authentication attempt; as such, join state verification must be disabled.
+
+With this context in mind, when using static keys, be aware that any client with
+knowledge of the private key can authenticate to Teleport with no additional
+restrictions. Before deploying a bot with static keys, take additional care to
+fully understand your environment's threat model and security needs, and reduce
+access using [Teleport's RBAC](../../../access-controls/roles.mdx) to ensure the
+minimum possible blast radius in the event the keypair is compromised.
+
 [renewable]: ../../../deployment/join-methods.mdx#renewable-vs-non-renewable
 [token]: ../../../deployment/join-methods.mdx#ephemeral-tokens