From 6eecde0dfd397cfb562c566dcd64db0e2050cf04 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Mon, 14 Aug 2023 13:37:23 +0100 Subject: [PATCH 01/14] Move existing How-To guides down in sidebar. --- docs/how-to-guides/creating_environment.md | 2 +- docs/how-to-guides/updating_environment.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-to-guides/creating_environment.md b/docs/how-to-guides/creating_environment.md index 85aebeb..b52393b 100644 --- a/docs/how-to-guides/creating_environment.md +++ b/docs/how-to-guides/creating_environment.md @@ -1,5 +1,5 @@ --- -sidebar_position: 1 +sidebar_position: 2 --- # Creating an environment diff --git a/docs/how-to-guides/updating_environment.md b/docs/how-to-guides/updating_environment.md index cbb268b..56ee982 100644 --- a/docs/how-to-guides/updating_environment.md +++ b/docs/how-to-guides/updating_environment.md @@ -1,5 +1,5 @@ --- -sidebar_position: 2 +sidebar_position: 3 --- # Updating an environment From 96aed49d142f537b9b3ed853a2fc5621473c9b10 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Mon, 14 Aug 2023 13:37:40 +0100 Subject: [PATCH 02/14] Add deployment How-To. --- docs/how-to-guides/deploy.md | 228 +++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 docs/how-to-guides/deploy.md diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md new file mode 100644 index 0000000..3e99b94 --- /dev/null +++ b/docs/how-to-guides/deploy.md @@ -0,0 +1,228 @@ +--- +sidebar_position: 1 +--- + +# Deployment + +As Softpack is currently under active development, these instructions are likely +to change over time to match the current state of the project. + +## Overview + +These instructions are intended as a step-by-step guide to get the latest +development version of SoftPack up-and-running so that it can be used to build +images. + +It is recommended that those following these instructions are comfortable with +using a unix-like terminal interface, and have some experience administering a +linux environment. + +## The Steps + +### Clone Infrastructure + +The first step is to clone the +[Infrastructure](https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/) +repo: + +```git clone https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/``` + +…then after entering the newly creating directory (```cd infrastructure```), +checkout the development branch: + +```git checkout develop``` + +### Terraform Configuration + +The next step is to modify several of the configuration files in order to create +a personal instance: + +In `terraform/terraform.tfvars`, add the following lines to the end of the file: + +``` +ARTIFACTS_ROOT = "./" +ANSIBLE_INVENTORY = "/path/to/softpack/infrastructure/terraform/ansible/hosts.yml" +ENVIRONMENT = "development" +SSH_KEY = "~/.ssh/id_rsa" +SSH_PUBLIC_KEY = "~/.ssh/id_rsa.pub" +``` + +You may need to replace the values of SSH_KEY and SSH_PUBLIC with the paths to +your own keys, and make sure the `ANSIBLE_INVENTORY` accurately reflects where +you cloned the repo (the hosts.yml file will not currently exist). + +NB: If you do not already have SSH keys created, you can do so with the +`ssh-keygen` command. + +In `terraform/backend.tf`, replace the `"http"` value with `"local"`. + +In `terraform/builder.tf`, in the `openstack_compute_instance_v2` resource, +you'll need to replace the key_pair line with the following: + +``` +key_pair = openstack_compute_keypair_v2.keypair.name +``` + +…and add the following section at the end: + +``` +resource "openstack_compute_keypair_v2" "keypair" { + name = "mercury-USERNAME-keypair" + public_key = file(var.SSH_PUBLIC_KEY) +} +``` + +The name needs to be unique, so you may need to change this if it conflicts with +an existing keypair setup. + +In `terraform/frontend.tf`, we need to make a similar change to the last, in the +`openstack_compute_instance_v2` resource, you'll need to replace the key_pair +line with the following: + +``` +key_pair = openstack_compute_keypair_v2.keypair.name +``` + +In `terraform/openstack.tf`, you'll need to remove the following key_pair line: + +``` + keypair = join("-", [var.name, var.ENVIRONMENT]) +``` + +Lastly, in the `terraform/variables.tf` file, you'll need to make sure that the +subdomain default value is unique, and will need to add the following lines to +the end of the file: + +``` +variable "SSH_PUBLIC_KEY" { + type = string +} +``` + +## Building the VM + +For this stage, you'll need to have both [Terraform](https://www.terraform.io/) +and [Ansible](https://www.ansible.com/) installed. It is recommended to install +`ansible` using `pip`: + +``` +pip install ansible +``` + +With the tools installed, you'll need to export some environmental variables +that terraform requires: + +``` +export VAULT_ADDR="https://vault.internal.sanger.ac.uk" +export VAULT_TOKEN="TOKEN_HERE" +``` + +…replacing the `VAULT_TOKEN` value with your own. + +NB: You can find your token by logging in to [The +Vault](https://vault.internal.sanger.ac.uk/) and selecting `Copy token` from the +top-right menu. If you have not previously generated a token, you will need to +do so now. + +Once done, make sure that you're in the `infrastructure/terraform` directory, +and run the following two commands to provision the VM for SoftPack: + +``` +terraform init +terraform apply +``` + +…following the onscreen instructions to confirm the actions that will take +place. + +## Installing the Software + +While still in the `infrastructure/terraform` directory, you can run the +following command to set-up the newly provisioned VM with the necessary software +and configuration: + +``` +ansible-playbook --inventory ansible/hosts.yml --limit builder ansible/main.yml +``` + +NB: This will most likely take a few minutes + +## After Installation Modification + +Once the initial software configuration is done, you'll need to log in to the +machine to make some changes: + +``` +ssh ubuntu@IP_ADDRESS +``` + +The IP Address can either be found in the `terraform` output, or in the +`terraform/ansible/hosts.yml` file, in the `builder` section. + +Once logged in, you'll need to modify two files. The first in the softpack +builder configuration file, located at +`/opt/softpack/builder/softpack_builder/config/conf/config.yml` and requires the +following changes: + +In the `vault` section, You'll need to fill in the details for the URL, Path, +and Token for the vault, either with real values, or dummy ones such as the +following: + +``` +vault: + url: https://something.com + path: /something + token: something +``` + +In the `spack->manifest->spack->container->images` section, remove the following +two lines: + +``` + build: spack/ubuntu-focal:v0.20.1 + final: ubuntu:20.04 +``` + +Lastly, in the `/opt/softpack/builder/softpack_builder/singularity.py` file, +find and delete the following line: + +``` + file.writelines(commands) +``` + +## Restart the SoftPack Service + +With the changes made, you should now be able to start the SoftPack Builder +Service, with the following command: + +``` +sudo systemctl restart softpack-builder +``` + +## Building Software + +You should now be able to build software. + +First, you need to activate the correct environment, which you can do with the +following command: + +``` +cd /opt/softpack/builder/ && source .venv/bin/activate +``` + +Once run, you should be able to install software with the following command: + +``` +softpack-builder environment build --name NAME_OF_INSTALL /path/to/softpack/file.yaml +``` + +NB: You must execute that command from the `/opt/softpack/builder/` directory. + + +The progress of the build can be viewed by accessing the IP address of your VM +in a browser, which will bring up the Prefect Flow interface, which also allows +you to view the logs. + +If you previously entered dummy vault information in the softpack configuration file, the +image will be built, but will not be uploaded anywhere; the location of the +image will be in the log. From 85e3e2864369900ea6600fa37448bec395fc6be1 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Thu, 17 Aug 2023 16:43:28 +0100 Subject: [PATCH 03/14] Update deploy docs to reflect current state of infrastructure repo. --- docs/how-to-guides/deploy.md | 154 +++++++++++++++++++++-------------- 1 file changed, 93 insertions(+), 61 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 3e99b94..c750110 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -17,8 +17,24 @@ It is recommended that those following these instructions are comfortable with using a unix-like terminal interface, and have some experience administering a linux environment. +The current automated build process is not intended to be used to create an +individual instance, and as such will require some modification. + ## The Steps +### Generate Application Credentials + +To build and set-up OpenStack VMs you'll need to generate some credntials: + +https://theta.internal.sanger.ac.uk/identity/application_credentials/ + +Select `Create Application Credential`, give it a name, and click `Create Application Credential`. + +Make note of the ID and Secret values, as you will need them later. + +NB: The site will not be able to retrieve the secret for you if you lose it. If +that happens you'll need to generate a new one. + ### Clone Infrastructure The first step is to clone the @@ -27,7 +43,7 @@ repo: ```git clone https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/``` -…then after entering the newly creating directory (```cd infrastructure```), +…then after entering the newly creating directory (`cd infrastructure`), checkout the development branch: ```git checkout develop``` @@ -37,66 +53,97 @@ checkout the development branch: The next step is to modify several of the configuration files in order to create a personal instance: -In `terraform/terraform.tfvars`, add the following lines to the end of the file: - -``` -ARTIFACTS_ROOT = "./" -ANSIBLE_INVENTORY = "/path/to/softpack/infrastructure/terraform/ansible/hosts.yml" -ENVIRONMENT = "development" -SSH_KEY = "~/.ssh/id_rsa" -SSH_PUBLIC_KEY = "~/.ssh/id_rsa.pub" -``` - -You may need to replace the values of SSH_KEY and SSH_PUBLIC with the paths to -your own keys, and make sure the `ANSIBLE_INVENTORY` accurately reflects where -you cloned the repo (the hosts.yml file will not currently exist). - -NB: If you do not already have SSH keys created, you can do so with the -`ssh-keygen` command. - In `terraform/backend.tf`, replace the `"http"` value with `"local"`. -In `terraform/builder.tf`, in the `openstack_compute_instance_v2` resource, -you'll need to replace the key_pair line with the following: +In `terraform/openstack.tf`, replace the locals section with the following: ``` -key_pair = openstack_compute_keypair_v2.keypair.name +locals { + openstack = { + keypair = join("-", [var.name, var.ENVIRONMENT]) + auth_url = var.openstack_auth_url + application_credential = { + id = var.openstack_application_credential_id + secret = var.openstack_application_credential_secret + } + } +} ``` -…and add the following section at the end: +…and add the following to the bottom of the file: ``` resource "openstack_compute_keypair_v2" "keypair" { - name = "mercury-USERNAME-keypair" + name = local.openstack.keypair public_key = file(var.SSH_PUBLIC_KEY) } ``` -The name needs to be unique, so you may need to change this if it conflicts with -an existing keypair setup. +In `terraform/variables.tf`, you'll want to make the following changes: -In `terraform/frontend.tf`, we need to make a similar change to the last, in the -`openstack_compute_instance_v2` resource, you'll need to replace the key_pair -line with the following: +Change the default value of the `name` variable from "softpack" to +"softpack-USERNAME" (where USERNAME is your username). -``` -key_pair = openstack_compute_keypair_v2.keypair.name -``` +Change the default value of `subdomain` to match that of `name`. -In `terraform/openstack.tf`, you'll need to remove the following key_pair line: +Add a default value for `ANSIBLE_INVENTORY` set to `"./ansible/hosts.yml"`. -``` - keypair = join("-", [var.name, var.ENVIRONMENT]) -``` +Add a default value for `SSH_KEY` set to `"~/.ssh/id_rsa"`. + +NB: You may need to replace the default value of SSH_KEY with the path to your +own keys if it is in a different location, or is of a different key type. If you +do not already have SSH keys created, you can do so with the `ssh-keygen` +command. + +Add a default value for `ENVIRONMENT` set to `"development"`. -Lastly, in the `terraform/variables.tf` file, you'll need to make sure that the -subdomain default value is unique, and will need to add the following lines to -the end of the file: +Add the following to the bottom of the file: ``` variable "SSH_PUBLIC_KEY" { type = string + default = "~/.ssh/id_rsa.pub" } + +variable "openstack_auth_url" { + type = string + default = "https://theta.internal.sanger.ac.uk:5000" +} + +variable "openstack_application_credential_id" { + type = string + default = "YOUR_CREDENTIAL_SECRET" +} + +variable "openstack_application_credential_secret" { + type = string + default = "YOUR_CREDENTIAL_SECRET" +} +``` + +…filling in your credentials that you generated earlier, and making sure the +`SSH_PUBLIC_KEY` path matches that of the accompanying public key to the +`SSH_KEY` you supplied earlier. + +Lastly, in the `terraform/ansible/roles/softpack_builder/tasks/main.yml` file, +**remove** the following lines: + +``` +- name: Import GPG key + vars: + key: /tmp/softpack.pgp + block: + - name: Copy GPG key + ansible.builtin.copy: + src: "{{ lookup('ansible.builtin.env', 'SPACK_KEY') }}" + dest: "{{ key }}" + mode: u=rw,g=-,o=- + + - name: Import GPG key + ansible.builtin.shell: | + SPACK_GNUPGHOME={{ builder.gpg }} {{ spack_path }}/bin/spack gpg trust {{ key }} + args: + creates: "{{ builder.gpg }}/trustdb.gpg" ``` ## Building the VM @@ -109,21 +156,6 @@ and [Ansible](https://www.ansible.com/) installed. It is recommended to install pip install ansible ``` -With the tools installed, you'll need to export some environmental variables -that terraform requires: - -``` -export VAULT_ADDR="https://vault.internal.sanger.ac.uk" -export VAULT_TOKEN="TOKEN_HERE" -``` - -…replacing the `VAULT_TOKEN` value with your own. - -NB: You can find your token by logging in to [The -Vault](https://vault.internal.sanger.ac.uk/) and selecting `Copy token` from the -top-right menu. If you have not previously generated a token, you will need to -do so now. - Once done, make sure that you're in the `infrastructure/terraform` directory, and run the following two commands to provision the VM for SoftPack: @@ -138,14 +170,17 @@ place. ## Installing the Software While still in the `infrastructure/terraform` directory, you can run the -following command to set-up the newly provisioned VM with the necessary software -and configuration: +following command to set-up the newly provisioned VMs with the necessary +software and configuration: ``` -ansible-playbook --inventory ansible/hosts.yml --limit builder ansible/main.yml +ansible-playbook --inventory ansible/hosts.yml ansible/main.yml ``` -NB: This will most likely take a few minutes +You will need to response 'yes' to each of the three SSH authenticity checks, +but should otherwise not be required to interact with the process. + +NB: This will most likely take a few minutes. ## After Installation Modification @@ -153,12 +188,9 @@ Once the initial software configuration is done, you'll need to log in to the machine to make some changes: ``` -ssh ubuntu@IP_ADDRESS +ssh ubuntu@builder.softpack-USERNAME.hgi-dev.sanger.ac.uk ``` -The IP Address can either be found in the `terraform` output, or in the -`terraform/ansible/hosts.yml` file, in the `builder` section. - Once logged in, you'll need to modify two files. The first in the softpack builder configuration file, located at `/opt/softpack/builder/softpack_builder/config/conf/config.yml` and requires the From 7a2a928e0fec4e42a7457b28b30641df23179732 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 18 Aug 2023 11:19:32 +0100 Subject: [PATCH 04/14] Add note about an optional change to the builder flavour. --- docs/how-to-guides/deploy.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index c750110..c8845a4 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -79,6 +79,10 @@ resource "openstack_compute_keypair_v2" "keypair" { } ``` +An optional change would be in `terraform/terraform.tfvars`, in which you may +wish to replace the builder flavour (`m4.medium`) with `m2.medium` as there are +generally more available. + In `terraform/variables.tf`, you'll want to make the following changes: Change the default value of the `name` variable from "softpack" to From 2de1c0d315da8384987cb7c8a81afe1f002a2917 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Thu, 24 Aug 2023 11:42:15 +0100 Subject: [PATCH 05/14] Corrected instructions for setting up correct configuration. --- docs/how-to-guides/deploy.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index c8845a4..b2a6d6a 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -197,12 +197,9 @@ ssh ubuntu@builder.softpack-USERNAME.hgi-dev.sanger.ac.uk Once logged in, you'll need to modify two files. The first in the softpack builder configuration file, located at -`/opt/softpack/builder/softpack_builder/config/conf/config.yml` and requires the -following changes: - -In the `vault` section, You'll need to fill in the details for the URL, Path, -and Token for the vault, either with real values, or dummy ones such as the -following: +`/opt/softpack/builder/softpack_builder/config/conf/config.yml`. In the `vault` +section, You'll need to fill in the details for the URL, Path, and Token for the +vault, either with real values, or dummy ones such as the following: ``` vault: @@ -211,12 +208,14 @@ vault: token: something ``` -In the `spack->manifest->spack->container->images` section, remove the following -two lines: +For the file `~/.softpack/builder/config.yml`, in the +`spack->manifest->spack->container->images` section, remove the following three +lines: ``` - build: spack/ubuntu-focal:v0.20.1 - final: ubuntu:20.04 + images: + os: ubuntu:20.04 + spack: v0.20.0 ``` Lastly, in the `/opt/softpack/builder/softpack_builder/singularity.py` file, From 53c5a35094de9b446f649ab1f3b52b3315e9c612 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Thu, 24 Aug 2023 13:57:30 +0100 Subject: [PATCH 06/14] Add instructions to correct PATH environmental variable for prefect agent service. --- docs/how-to-guides/deploy.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index b2a6d6a..2f45392 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -218,6 +218,13 @@ lines: spack: v0.20.0 ``` +In the file `/lib/systemd/system/prefect-agent.service`, you'll need to append +the spack path to the `Environment` line, like so: + +``` +Environment=PATH=/opt/softpack/builder/.venv/bin:/usr/local/bin:/usr/bin:/opt/spack/v0.20.1/bin +``` + Lastly, in the `/opt/softpack/builder/softpack_builder/singularity.py` file, find and delete the following line: @@ -225,12 +232,14 @@ find and delete the following line: file.writelines(commands) ``` -## Restart the SoftPack Service +## Restart the SoftPack Services -With the changes made, you should now be able to start the SoftPack Builder -Service, with the following command: +With the changes made, you should now be able to restart both the Prefect Agent +and SoftPack Builder services, with the following commands: ``` +sudo systemctl daemon-reload +sudo systemctl restart prefect-agent sudo systemctl restart softpack-builder ``` From 92be3f991ef6448cb0371cce2cfbdf3f10beab0d Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Thu, 24 Aug 2023 14:38:13 +0100 Subject: [PATCH 07/14] Add instruction to create missing 'gpg' directory. --- docs/how-to-guides/deploy.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 2f45392..41bad7a 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -195,8 +195,15 @@ machine to make some changes: ssh ubuntu@builder.softpack-USERNAME.hgi-dev.sanger.ac.uk ``` -Once logged in, you'll need to modify two files. The first in the softpack -builder configuration file, located at +Once logged in, you'll need to create a directory and modify some files. + +Create the needed directory with the following command: + +``` +mkdir /opt/softpack/gpg/ +``` + +The first file to edit is in the softpack builder configuration file, located at `/opt/softpack/builder/softpack_builder/config/conf/config.yml`. In the `vault` section, You'll need to fill in the details for the URL, Path, and Token for the vault, either with real values, or dummy ones such as the following: From 3f0ce75357833700fc3758b83fbb8abf97c4dd90 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 25 Aug 2023 09:25:32 +0100 Subject: [PATCH 08/14] Modify instructions to make it clearer what directory the user should be in. --- docs/how-to-guides/deploy.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 41bad7a..5cfcd16 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -160,7 +160,7 @@ and [Ansible](https://www.ansible.com/) installed. It is recommended to install pip install ansible ``` -Once done, make sure that you're in the `infrastructure/terraform` directory, +Once done, make sure that you're in the `terraform` sub-directory, and run the following two commands to provision the VM for SoftPack: ``` @@ -173,7 +173,7 @@ place. ## Installing the Software -While still in the `infrastructure/terraform` directory, you can run the +While still in the `terraform` sub-directory, you can run the following command to set-up the newly provisioned VMs with the necessary software and configuration: From 8c0d798094f1a0c107eff917a3248837f3254ec5 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 25 Aug 2023 09:34:56 +0100 Subject: [PATCH 09/14] Add instructions for setting up vault token environmental variable. --- docs/how-to-guides/deploy.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 5cfcd16..50cbb47 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -160,6 +160,16 @@ and [Ansible](https://www.ansible.com/) installed. It is recommended to install pip install ansible ``` +Next, you'll need to set up the environmental variables for your vault token. +Login to the [vault](https://vault.internal.sanger.ac.uk/) and you can copy your +vault token from the top-right drop-down menu. Then, in your terminal, run the +following: + +``` +export VAULT_ADDR="https://vault.internal.sanger.ac.uk" +export VAULT_TOKEN="YOUR_TOKEN_HERE" +``` + Once done, make sure that you're in the `terraform` sub-directory, and run the following two commands to provision the VM for SoftPack: From 17e09135769a1a43bba2468979eae56e1feb879c Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 25 Aug 2023 10:00:54 +0100 Subject: [PATCH 10/14] Add note about 'hacky' nature of source code change. --- docs/how-to-guides/deploy.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 50cbb47..5411969 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -249,6 +249,11 @@ find and delete the following line: file.writelines(commands) ``` +NB: This is a temporary work-around, as the builder is currently in development. +This should not be required in the future, and this line may not exist if this +document hasn't been updated to reflect the latest state of the builder +development. + ## Restart the SoftPack Services With the changes made, you should now be able to restart both the Prefect Agent @@ -279,7 +284,6 @@ softpack-builder environment build --name NAME_OF_INSTALL /path/to/softpack/file NB: You must execute that command from the `/opt/softpack/builder/` directory. - The progress of the build can be viewed by accessing the IP address of your VM in a browser, which will bring up the Prefect Flow interface, which also allows you to view the logs. From 6641401165b278172649d4cbb90ccf234f845f4a Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 25 Aug 2023 11:17:41 +0100 Subject: [PATCH 11/14] Update progress/log viewing instructions to include ability to use hostname. --- docs/how-to-guides/deploy.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 5411969..5650382 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -284,9 +284,9 @@ softpack-builder environment build --name NAME_OF_INSTALL /path/to/softpack/file NB: You must execute that command from the `/opt/softpack/builder/` directory. -The progress of the build can be viewed by accessing the IP address of your VM -in a browser, which will bring up the Prefect Flow interface, which also allows -you to view the logs. +The progress of the build can be viewed by accessing the IP/host address of your +builder VM in a browser, which will bring up the Prefect Flow interface, which +also allows you to view the logs. If you previously entered dummy vault information in the softpack configuration file, the image will be built, but will not be uploaded anywhere; the location of the From 706b9248cbfbe61873c64ead308ed2e4afdb55d7 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Fri, 25 Aug 2023 11:54:43 +0100 Subject: [PATCH 12/14] Added note about why we're removing configuration entries. --- docs/how-to-guides/deploy.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index 5650382..feafea0 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -235,6 +235,10 @@ lines: spack: v0.20.0 ``` +NB: These lines are default spack configuration that conflicts with our custom +configuration in SoftPack. This may be fixed in the infrastructure repo at some +point, and so may not need doing. + In the file `/lib/systemd/system/prefect-agent.service`, you'll need to append the spack path to the `Environment` line, like so: From 06d323fc4f1fabf12b1fc35e457d232575f5de52 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Tue, 29 Aug 2023 09:32:53 +0100 Subject: [PATCH 13/14] Add information about how building may or may not have external side effects. --- docs/how-to-guides/deploy.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md index feafea0..d1d1866 100644 --- a/docs/how-to-guides/deploy.md +++ b/docs/how-to-guides/deploy.md @@ -235,9 +235,9 @@ lines: spack: v0.20.0 ``` -NB: These lines are default spack configuration that conflicts with our custom -configuration in SoftPack. This may be fixed in the infrastructure repo at some -point, and so may not need doing. +NB: These lines are default spack configuration entries that conflicts with our +custom configuration in SoftPack. This may be fixed in the infrastructure repo +at some point, and so may not need doing. In the file `/lib/systemd/system/prefect-agent.service`, you'll need to append the spack path to the `Environment` line, like so: @@ -271,7 +271,12 @@ sudo systemctl restart softpack-builder ## Building Software -You should now be able to build software. +You should now be able to build software. If you previously entered dummy +information for the vault configuration, then you are safe to test and build +without affecting anything outside of your instance. This means that the +Singularity images built will not be pushed to a container registry (the +location of the built image can be found at the end of the log, look for the +line containing the text `Build complete`). First, you need to activate the correct environment, which you can do with the following command: @@ -290,8 +295,4 @@ NB: You must execute that command from the `/opt/softpack/builder/` directory. The progress of the build can be viewed by accessing the IP/host address of your builder VM in a browser, which will bring up the Prefect Flow interface, which -also allows you to view the logs. - -If you previously entered dummy vault information in the softpack configuration file, the -image will be built, but will not be uploaded anywhere; the location of the -image will be in the log. +also allows you to view the logs. \ No newline at end of file From 066554e5f0a6e3c6f6f32fd6b57bd6cfc8d83894 Mon Sep 17 00:00:00 2001 From: Michael Woolnough Date: Mon, 4 Sep 2023 11:42:28 +0100 Subject: [PATCH 14/14] WIP: Changes following review. --- docs/how-to-guides/creating_environment.md | 2 +- docs/how-to-guides/deploy.md | 298 --------------------- docs/how-to-guides/updating_environment.md | 2 +- docs/overview/deployment.md | 199 +++++++++++++- 4 files changed, 197 insertions(+), 304 deletions(-) delete mode 100644 docs/how-to-guides/deploy.md diff --git a/docs/how-to-guides/creating_environment.md b/docs/how-to-guides/creating_environment.md index b52393b..85aebeb 100644 --- a/docs/how-to-guides/creating_environment.md +++ b/docs/how-to-guides/creating_environment.md @@ -1,5 +1,5 @@ --- -sidebar_position: 2 +sidebar_position: 1 --- # Creating an environment diff --git a/docs/how-to-guides/deploy.md b/docs/how-to-guides/deploy.md deleted file mode 100644 index d1d1866..0000000 --- a/docs/how-to-guides/deploy.md +++ /dev/null @@ -1,298 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Deployment - -As Softpack is currently under active development, these instructions are likely -to change over time to match the current state of the project. - -## Overview - -These instructions are intended as a step-by-step guide to get the latest -development version of SoftPack up-and-running so that it can be used to build -images. - -It is recommended that those following these instructions are comfortable with -using a unix-like terminal interface, and have some experience administering a -linux environment. - -The current automated build process is not intended to be used to create an -individual instance, and as such will require some modification. - -## The Steps - -### Generate Application Credentials - -To build and set-up OpenStack VMs you'll need to generate some credntials: - -https://theta.internal.sanger.ac.uk/identity/application_credentials/ - -Select `Create Application Credential`, give it a name, and click `Create Application Credential`. - -Make note of the ID and Secret values, as you will need them later. - -NB: The site will not be able to retrieve the secret for you if you lose it. If -that happens you'll need to generate a new one. - -### Clone Infrastructure - -The first step is to clone the -[Infrastructure](https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/) -repo: - -```git clone https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/``` - -…then after entering the newly creating directory (`cd infrastructure`), -checkout the development branch: - -```git checkout develop``` - -### Terraform Configuration - -The next step is to modify several of the configuration files in order to create -a personal instance: - -In `terraform/backend.tf`, replace the `"http"` value with `"local"`. - -In `terraform/openstack.tf`, replace the locals section with the following: - -``` -locals { - openstack = { - keypair = join("-", [var.name, var.ENVIRONMENT]) - auth_url = var.openstack_auth_url - application_credential = { - id = var.openstack_application_credential_id - secret = var.openstack_application_credential_secret - } - } -} -``` - -…and add the following to the bottom of the file: - -``` -resource "openstack_compute_keypair_v2" "keypair" { - name = local.openstack.keypair - public_key = file(var.SSH_PUBLIC_KEY) -} -``` - -An optional change would be in `terraform/terraform.tfvars`, in which you may -wish to replace the builder flavour (`m4.medium`) with `m2.medium` as there are -generally more available. - -In `terraform/variables.tf`, you'll want to make the following changes: - -Change the default value of the `name` variable from "softpack" to -"softpack-USERNAME" (where USERNAME is your username). - -Change the default value of `subdomain` to match that of `name`. - -Add a default value for `ANSIBLE_INVENTORY` set to `"./ansible/hosts.yml"`. - -Add a default value for `SSH_KEY` set to `"~/.ssh/id_rsa"`. - -NB: You may need to replace the default value of SSH_KEY with the path to your -own keys if it is in a different location, or is of a different key type. If you -do not already have SSH keys created, you can do so with the `ssh-keygen` -command. - -Add a default value for `ENVIRONMENT` set to `"development"`. - -Add the following to the bottom of the file: - -``` -variable "SSH_PUBLIC_KEY" { - type = string - default = "~/.ssh/id_rsa.pub" -} - -variable "openstack_auth_url" { - type = string - default = "https://theta.internal.sanger.ac.uk:5000" -} - -variable "openstack_application_credential_id" { - type = string - default = "YOUR_CREDENTIAL_SECRET" -} - -variable "openstack_application_credential_secret" { - type = string - default = "YOUR_CREDENTIAL_SECRET" -} -``` - -…filling in your credentials that you generated earlier, and making sure the -`SSH_PUBLIC_KEY` path matches that of the accompanying public key to the -`SSH_KEY` you supplied earlier. - -Lastly, in the `terraform/ansible/roles/softpack_builder/tasks/main.yml` file, -**remove** the following lines: - -``` -- name: Import GPG key - vars: - key: /tmp/softpack.pgp - block: - - name: Copy GPG key - ansible.builtin.copy: - src: "{{ lookup('ansible.builtin.env', 'SPACK_KEY') }}" - dest: "{{ key }}" - mode: u=rw,g=-,o=- - - - name: Import GPG key - ansible.builtin.shell: | - SPACK_GNUPGHOME={{ builder.gpg }} {{ spack_path }}/bin/spack gpg trust {{ key }} - args: - creates: "{{ builder.gpg }}/trustdb.gpg" -``` - -## Building the VM - -For this stage, you'll need to have both [Terraform](https://www.terraform.io/) -and [Ansible](https://www.ansible.com/) installed. It is recommended to install -`ansible` using `pip`: - -``` -pip install ansible -``` - -Next, you'll need to set up the environmental variables for your vault token. -Login to the [vault](https://vault.internal.sanger.ac.uk/) and you can copy your -vault token from the top-right drop-down menu. Then, in your terminal, run the -following: - -``` -export VAULT_ADDR="https://vault.internal.sanger.ac.uk" -export VAULT_TOKEN="YOUR_TOKEN_HERE" -``` - -Once done, make sure that you're in the `terraform` sub-directory, -and run the following two commands to provision the VM for SoftPack: - -``` -terraform init -terraform apply -``` - -…following the onscreen instructions to confirm the actions that will take -place. - -## Installing the Software - -While still in the `terraform` sub-directory, you can run the -following command to set-up the newly provisioned VMs with the necessary -software and configuration: - -``` -ansible-playbook --inventory ansible/hosts.yml ansible/main.yml -``` - -You will need to response 'yes' to each of the three SSH authenticity checks, -but should otherwise not be required to interact with the process. - -NB: This will most likely take a few minutes. - -## After Installation Modification - -Once the initial software configuration is done, you'll need to log in to the -machine to make some changes: - -``` -ssh ubuntu@builder.softpack-USERNAME.hgi-dev.sanger.ac.uk -``` - -Once logged in, you'll need to create a directory and modify some files. - -Create the needed directory with the following command: - -``` -mkdir /opt/softpack/gpg/ -``` - -The first file to edit is in the softpack builder configuration file, located at -`/opt/softpack/builder/softpack_builder/config/conf/config.yml`. In the `vault` -section, You'll need to fill in the details for the URL, Path, and Token for the -vault, either with real values, or dummy ones such as the following: - -``` -vault: - url: https://something.com - path: /something - token: something -``` - -For the file `~/.softpack/builder/config.yml`, in the -`spack->manifest->spack->container->images` section, remove the following three -lines: - -``` - images: - os: ubuntu:20.04 - spack: v0.20.0 -``` - -NB: These lines are default spack configuration entries that conflicts with our -custom configuration in SoftPack. This may be fixed in the infrastructure repo -at some point, and so may not need doing. - -In the file `/lib/systemd/system/prefect-agent.service`, you'll need to append -the spack path to the `Environment` line, like so: - -``` -Environment=PATH=/opt/softpack/builder/.venv/bin:/usr/local/bin:/usr/bin:/opt/spack/v0.20.1/bin -``` - -Lastly, in the `/opt/softpack/builder/softpack_builder/singularity.py` file, -find and delete the following line: - -``` - file.writelines(commands) -``` - -NB: This is a temporary work-around, as the builder is currently in development. -This should not be required in the future, and this line may not exist if this -document hasn't been updated to reflect the latest state of the builder -development. - -## Restart the SoftPack Services - -With the changes made, you should now be able to restart both the Prefect Agent -and SoftPack Builder services, with the following commands: - -``` -sudo systemctl daemon-reload -sudo systemctl restart prefect-agent -sudo systemctl restart softpack-builder -``` - -## Building Software - -You should now be able to build software. If you previously entered dummy -information for the vault configuration, then you are safe to test and build -without affecting anything outside of your instance. This means that the -Singularity images built will not be pushed to a container registry (the -location of the built image can be found at the end of the log, look for the -line containing the text `Build complete`). - -First, you need to activate the correct environment, which you can do with the -following command: - -``` -cd /opt/softpack/builder/ && source .venv/bin/activate -``` - -Once run, you should be able to install software with the following command: - -``` -softpack-builder environment build --name NAME_OF_INSTALL /path/to/softpack/file.yaml -``` - -NB: You must execute that command from the `/opt/softpack/builder/` directory. - -The progress of the build can be viewed by accessing the IP/host address of your -builder VM in a browser, which will bring up the Prefect Flow interface, which -also allows you to view the logs. \ No newline at end of file diff --git a/docs/how-to-guides/updating_environment.md b/docs/how-to-guides/updating_environment.md index 56ee982..cbb268b 100644 --- a/docs/how-to-guides/updating_environment.md +++ b/docs/how-to-guides/updating_environment.md @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 2 --- # Updating an environment diff --git a/docs/overview/deployment.md b/docs/overview/deployment.md index 3df6df7..a3d26a1 100644 --- a/docs/overview/deployment.md +++ b/docs/overview/deployment.md @@ -4,8 +4,199 @@ sidebar_position: 6 # Deployment -:::caution -This documentation is under development and may be incomplete. -::: +As Softpack is currently under active development, these instructions are likely +to change over time to match the current state of the project. -Deployment details will be available once [Milestone 2](/docs/reference/development_roadmap/#milestone-2) is scheduled. \ No newline at end of file +## Overview + +These instructions are intended as a step-by-step guide to get SoftPack +up-and-running so that it can be used to build images. + +The current automated build process is not intended to be used to create an +individual instance, and as such will require some modification. + +## The Steps + +### Generate Application Credentials + +To build and set-up OpenStack VMs you'll need to generate some credentials: + +https://theta.internal.sanger.ac.uk/identity/application_credentials/ + +Select `Create Application Credential`, give it a name, and click `Create +Application Credential`. + +Make note of the ID and Secret values, as you will need them later. + +NB: The site will not be able to retrieve the secret for you if you lose it. If +that happens you'll need to generate a new one. + +### Keypair creation + +If you do not already have an OpenStack keypair created, you will need to do so +now. + +https://theta.internal.sanger.ac.uk/project/key_pairs + +Go to the above URL and click `Create Key Pair`, fill in a unique `Key Pair +Name` and select the `SSH Key` Key Type. + +This will generate a public/private keypair and download the private key to your +local machine. + +You will need to move the downloaded key to your SSH directory (usually ~/.ssh) +and set the correct permissions on it (0600). + +### Clone Infrastructure + +The first step is to clone the +[Infrastructure](https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/) +repo: + +```git clone https://gitlab.internal.sanger.ac.uk/hgi-projects/softpack/infrastructure/``` + +…then after entering the newly creating directory (`cd infrastructure`), +checkout the development branch: + +```git checkout develop``` + +### Terraform Configuration + +The next step is to modify several of the configuration files in order to create +a personal instance: + +In `terraform/backend.tf`, replace the `"http"` value with `"local"`. + +In `terraform/openstack.tf`, replace the locals section with the following: + +# TODO: Change to reflect recent changes. +# TF_VAR_OPENSTACK_APPLICATION_ID +# TF_VAR_OPENSTACK_APPLICATION_SECRET +# TF_VAR_OPENSTACK_AUTH_URL +``` +locals { + openstack = { + keypair = join("-", [var.name, var.ENVIRONMENT]) + auth_url = var.openstack_auth_url + application_credential = { + id = var.openstack_application_credential_id + secret = var.openstack_application_credential_secret + } + } +} +``` + +An optional change would be in `terraform/terraform.tfvars`, in which you may +wish to replace the builder flavour (`m4.medium`) with `m2.medium` as there are +generally more available. + +``` +name = "USERNAME-softpack" +subdomain = "USERNAME-softpack" +ANSIBLE_INVENTORY = "./ansible/hosts.yml" +SSH_KEY = "~/path/to/downloaded/private.pem" +ENVIRONMENT = "development" +``` + +Where `USERNAME` is your username. + +# TODO: Add instruction about setting up unique DNS zone. `Ask service desk to create user specific zone in infoblox` + +# TODO: Create GPG key and set SPACK_KEY env var to the path of the key. + +## Building the VM + +For this stage, you'll need to have both [Terraform](https://www.terraform.io/) +and [Ansible](https://www.ansible.com/) installed. It is recommended to install +`ansible` using `pip`: + +``` +pip install ansible +``` + +# TODO: Modify to get read-only token + +Next, you'll need to set up the environmental variables for your vault token. +Login to the [vault](https://vault.internal.sanger.ac.uk/) and you can copy your +vault token from the top-right drop-down menu. Then, in your terminal, run the +following: + + +``` +export VAULT_ADDR="https://vault.internal.sanger.ac.uk" +export VAULT_TOKEN="YOUR_TOKEN_HERE" +``` + +Once done, make sure that you're in the `terraform` sub-directory, +and run the following two commands to provision the VM for SoftPack: + +``` +terraform init +terraform plan +``` + +# TODO: Expand +Check that the planned tasks make sense. + +Then run: + +``` +terraform apply +``` + +…following the onscreen instructions to confirm the actions that will take +place. + +## Installing the Software + +While still in the `terraform` sub-directory, you can run the +following command to set-up the newly provisioned VMs with the necessary +software and configuration: + +``` +ansible-playbook --private-key ~/path/to/downloaded/private.pem --inventory ansible/hosts.yml ansible/main.yml +``` + +You will need to response 'yes' to each of the three SSH authenticity checks, +but should otherwise not be required to interact with the process. + +NB: This will most likely take a few minutes. + +## Building Software + +Once Ansible is done, you'll need to log in to the builder: + +``` +ssh -i ~/path/to/downloaded/private.pem ubuntu@builder.$USER-softpack.hgi-dev.sanger.ac.uk +``` + +# TODO: Confirm dummy inforation is required/still works. + +You should now be able to build software. If you previously entered dummy +information for the vault configuration, then you are safe to test and build +without affecting anything outside of your instance. This means that the +Singularity images built will not be pushed to a container registry (the +location of the built image can be found at the end of the log, look for the +line containing the text `Build complete`). + +First, you need to activate the correct environment, which you can do with the +following commands: + +``` +cd /opt/softpack/builder/ +source .venv/bin/activate +``` + +Try a test build with the following command. + +``` +softpack-builder environment build --name test_run test/data/specs/test-env.yml +``` + +# TODO: Add example output, and note flow ID and /op/softpack/envs/ID/ artifacts dir. + +NB: You must execute that command from the `/opt/softpack/builder/` directory. + +The progress of the build can be viewed by accessing +`http://builder.USERNAME-softpack.hgi-dev.sanger.ac.uk` in a browser, which will +bring up the Prefect Flow interface, which also allows you to view the logs. \ No newline at end of file