Skip to content

Commit ee5a1fd

Browse files
zeidlitzzeidlitz
committed
feat: render job manifest from template
- added new command to bpctl called render, this will consume from config.yaml and produce a job.yaml based on those values. - added two new config fields, DB_SECRET_NAME & CERT_SECRET_NAME and are used to point out what secrets to consume sensetive values from. They replace the old DB_ prefixed values in config.yaml making configuration less verbose locally. - added a helper script that can be used to provision secrets - other general housekeeping and fixes
1 parent a33574e commit ee5a1fd

8 files changed

Lines changed: 282 additions & 56 deletions

File tree

‎README.md‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# sda-bpctl
22

3+
34
A tool that can be used to deal with administrative workflows for the big picture project. It supports three primary functions, making data ingestion, assigning accession ids to each ingested file, and creating a dataset for all files ingested with a accession id.
45

56
It can be used either locally as a cli tool or be packaged and run as a job in kubernetes.
@@ -48,19 +49,45 @@ running mail notification with the dry-run flag
4849

4950
#### kubernetes job
5051

51-
The `job` command is meant to try and run all the steps of the dataset submission process in order; ingest -> accession -> dataset. It will need environment variables specified as found in `job.yaml.example` and will need to be adjusted for the environment to run in. The `job` command also needs a input argument that represents the ammount of files that is expected to be included in the finalized dataset. If at some point during the process this number does not match the job will fail and the user have to take over the process from that point.
52+
The `job` command is meant to try and run all the steps of the dataset submission process in order; ingest -> accession -> dataset. It will need environment variables to configure and will need to be adjusted for the environment to run in. The `job` command also needs a input argument that represents the ammount of files that is expected to be included in the finalized dataset. If at some point during the process this number does not match the job will fail and the user have to take over the process from that point.
53+
54+
specify your maniifest, for example in a `job.yaml` or you can render a templated manifest based on your `config.yaml` using the `render` command
55+
56+
```bash
57+
./bpctl render -o job.yaml
58+
```
5259

53-
specify your maniifest, for example in a `job.yaml` and it can be started using `kubectl`
60+
and apply it using `kubectl`:
5461

5562
```bash
5663
kubectl apply -f job.yaml
5764
```
5865

66+
67+
will render a job.yaml manifest for you based on the configuration values you have supplied
68+
69+
5970
### configuration
6071

6172
bpctl can consume configuration from either `config.yaml` or from environment variables. If both are supplied then the environment variables will take priority. If using config.yaml it is expected to be located in the root directory of the project. It can also be supplied by using the `--config` flag if located elsewhere.
6273

63-
see the `config.yaml.example` or `job.yaml.example` for a base template with what fields to fill
74+
see the `config.yaml.example` for a base template with what fields to fill
75+
| Name | Example | Description | requiered for |
76+
| --------------- | --------------- | --------------- | --------------- |
77+
| USER_ID | "user-1234" | The user ID for the uploader, acts as identifier for the uploaded data | `ingest`, `accession`, `dataset`, `job`, `render` |
78+
| DATASET_ID | "aa-Dataset-abc" | The ID that will be set for the finalized dataset, will be used during the `dataset` command | `ingest`, `accession`, `dataset`, `job`, `mail`, `render` |
79+
| DATASET_FOLDER | "DATASET_ABC" | The folder where the uploaded data resides in s3inbox | `ingest`, `accession`, `dataset`, `job`, `mail`, `render` |
80+
| JOB_EXPECTED_NR_FILES | 0 | The expected number of files to be part of the finalized dataset, set this when using `render` to include it in the rendered job.yaml | `job`, `render` |
81+
| CLIENT_API_HOST | "https://api.example.com" | The hostname for the SDA API to communicate with | `ingest`, `accession`, `dataset`, `job` |
82+
| CLIENT_ACCESS_TOKEN | "youraccesstoken" | The access token to authenticate towards the client api host | Yes | `ingest`, `accession`, `dataset`, `job` |
83+
| MAIL_ADDRESS | "myemail@example.com" | Used for the `mail` command, this will be the email address the outgoing emails will be sent from | `mail`, `render` |
84+
| MAIL_PASSWORD | "mypasswordemail" | Password associated with mail address | `mail`, `render` |
85+
| MAIL_UPLOADER | "jane@example.com" | Mail address to the uploader, this is the address the outgoing email will be sent to | `mail`, `render` |
86+
| MAIL_UPLOADER_NAME | "Jane Doe" | Name of the uploader | `mail`, `render` |
87+
| MAIL_SMTP_HOST | "smtp.example.com" | Hostname to a mail server to relay mails through | `mail`, `render` |
88+
| MAIL_SMTP_PORT | 587 | Port for the mail server | `mail`, `render` |
89+
| DB_SECRET_NAME | "db-secret" | The name of the kubernetes secret that holds connection details for the sda database | `job` |
90+
| CERT_SECRET_NAME | "cert-secret" | The name of the kubernetes secret that holds a tls certificate to use | `job` |
6491

6592
### testing
6693

‎config.yaml.example‎

Lines changed: 9 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,36 +2,28 @@
22
# Application Configuration Example. Use this to populate your own config
33
# ===============================
44

5-
# shared configs
5+
# shared configurations
66
USER_ID: "user-1234"
77
DATASET_ID: "aa-Dataset-abc"
88
DATASET_FOLDER: "DATASET_ABC"
9-
SSL_CA_CERT: "/etc/ssl/certs/ca.crt"
109

11-
# job.go
10+
# job specific configurations
1211
JOB_TIMEOUT: 3
1312
JOB_POLL_RATE: 2
13+
JOB_EXPECTED_NR_FILES: 0
1414

15-
# client.go
15+
# client specific configurations
1616
CLIENT_API_HOST: "https://api.example.com"
1717
CLIENT_ACCESS_TOKEN: "youraccesstoken"
1818

19-
# mail.go
19+
# secrets to consume sensetive data from
20+
DB_SECRET_NAME: "db-secret"
21+
CERT_SECRET_NAME: "cert-secret"
22+
23+
# mail specific configurations
2024
MAIL_ADDRESS: "myemail@example.com"
2125
MAIL_PASSWORD: "mypasswordemail"
2226
MAIL_UPLOADER: "jane@example.com"
2327
MAIL_UPLOADER_NAME: "Jane Doe"
2428
MAIL_SMTP_HOST: "smtp.example.com"
2529
MAIL_SMTP_PORT: 587
26-
27-
# database.go
28-
DB_HOST: "localhost"
29-
DB_PORT: "15432"
30-
DB_USER: "api"
31-
DB_PASSWORD: "api"
32-
DB_NAME: "postgres"
33-
DB_SCHEMA: "sda"
34-
DB_SSL_MODE: "disable"
35-
DB_CA_CERT: ""
36-
DB_CLIENT_CERT: ""
37-
DB_CLIENT_KEY: ""

‎helpers/helpers.go‎

100644100755
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,137 @@
11
package helpers
22

33
import (
4+
"bytes"
5+
"embed"
46
"fmt"
7+
"log/slog"
8+
"os"
9+
"strconv"
10+
"strings"
11+
"text/template"
12+
13+
"github.com/NBISweden/sda-bpctl/cmd"
14+
"github.com/NBISweden/sda-bpctl/internal/config"
15+
"github.com/spf13/cobra"
516
)
617

18+
//go:embed templates/*.yaml
19+
var templateFS embed.FS
20+
var configPath string
21+
var output string
22+
23+
type TemplateData struct {
24+
JobName string
25+
JobReleaseLabel string
26+
JobArgs string
27+
UserID string
28+
DatasetID string
29+
DatasetFolder string
30+
SslCaCert string
31+
ClientApiHost string
32+
ClientAccesToken string
33+
MailUploaderName string
34+
MailUploader string
35+
MailAddress string
36+
MailPassword string
37+
MailSmptHost string
38+
MailSmptPort string
39+
DbSecretName string
40+
DbCaCert string
41+
DbClientCert string
42+
DbClientKey string
43+
CertSecretName string
44+
}
45+
46+
var renderCmd = &cobra.Command{
47+
Use: "render [flags]",
48+
Short: "Render job manifest",
49+
Long: "Render job manifest",
50+
Args: func(cmd *cobra.Command, args []string) error {
51+
return nil
52+
},
53+
RunE: func(cmd *cobra.Command, args []string) error {
54+
cfg, err := config.NewConfig(configPath)
55+
if err != nil {
56+
return err
57+
}
58+
59+
templateData, err := createTemplateData(cfg)
60+
if err != nil {
61+
return err
62+
}
63+
64+
jobManifest, err := renderTemplate(templateData)
65+
if err != nil {
66+
return err
67+
}
68+
69+
err = writeManifest(jobManifest, output)
70+
if err != nil {
71+
return err
72+
}
73+
74+
return nil
75+
},
76+
}
77+
78+
func init() {
79+
cmd.AddCommand(renderCmd)
80+
renderCmd.Flags().StringVarP(&configPath, "config", "c", "config.yaml", "Path to configuration file")
81+
renderCmd.Flags().StringVarP(&output, "output", "o", "job.yaml", "Path to write the rendered file to")
82+
}
83+
84+
func createTemplateData(cfg *config.Config) (TemplateData, error) {
85+
templateData := &TemplateData{
86+
JobName: strings.ToLower(strings.ReplaceAll(cfg.DatasetFolder, "_", "-")),
87+
JobReleaseLabel: "sda",
88+
JobArgs: fmt.Sprintf("[\"job\", \"%d\"]", cfg.ExpectedNrFiles),
89+
UserID: cfg.UserID,
90+
DatasetID: cfg.DatasetID,
91+
DatasetFolder: cfg.DatasetFolder,
92+
SslCaCert: "/.secrets/tls/ca.crt",
93+
ClientApiHost: cfg.ClientApiHost,
94+
ClientAccesToken: cfg.ClientAccessToken,
95+
MailUploaderName: cfg.MailUploaderName,
96+
MailUploader: cfg.MailUploader,
97+
MailAddress: cfg.MailAddress,
98+
MailPassword: cfg.MailPassword,
99+
MailSmptHost: cfg.MailSmtpHost,
100+
MailSmptPort: strconv.Itoa(cfg.MailSmtpPort),
101+
CertSecretName: cfg.CertSecretName,
102+
DbSecretName: cfg.DbSecretName,
103+
}
104+
return *templateData, nil
105+
}
106+
107+
func writeManifest(jobManifest string, output string) error {
108+
file, err := os.Create(output)
109+
if err != nil {
110+
return err
111+
}
112+
defer file.Close()
113+
114+
_, err = file.WriteString(jobManifest)
115+
if err != nil {
116+
return err
117+
}
118+
slog.Info("writing", "output", output)
119+
return nil
120+
}
121+
122+
func renderTemplate(data TemplateData) (string, error) {
123+
tmpl, err := template.ParseFS(templateFS, "templates/job.template.yaml")
124+
if err != nil {
125+
return "", err
126+
}
127+
128+
var buf bytes.Buffer
129+
if err := tmpl.Execute(&buf, data); err != nil {
130+
return "", err
131+
}
132+
return buf.String(), nil
133+
}
134+
7135
func GetFileIDsPath(dataDirectory string, datasetFolder string) string {
8136
return fmt.Sprintf("%s/%s-fileIDs.txt", dataDirectory, datasetFolder)
9137
}

job.yaml.example renamed to helpers/templates/job.template.yaml

Lines changed: 27 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,15 @@
1-
# ===============================
2-
# Job Configuration Example. Use this as a starting point to populate your own config
3-
# ===============================
41
apiVersion: batch/v1
52
kind: Job
63
metadata:
7-
name: aa-dataset-abc
8-
labels:
9-
release: sda
4+
name: {{ .JobName }}
5+
labels:
6+
release: {{ .JobReleaseLabel }}
107
spec:
118
backoffLimit: 0
129
template:
1310
metadata:
1411
labels:
15-
release: sda
12+
release: {{ .JobReleaseLabel }}
1613
spec:
1714
securityContext:
1815
fsGroup: 65534
@@ -23,73 +20,73 @@ spec:
2320
- name: bpctl
2421
image: harbor.nbis.se/sda/bpctl:latest
2522
imagePullPolicy: Always
26-
args: ["job", "0"] # The second arg represents the expected number of files to be in the final dataset
23+
args: {{ .JobArgs }}
2724
env:
2825
- name: USER_ID
29-
value: johndoe@lifescience-ri.eu
26+
value: {{ .UserID }}
3027
- name: DATASET_ID
31-
value: aa-Dataset-abc
28+
value: {{ .DatasetID }}
3229
- name: DATASET_FOLDER
33-
value: DATASET_ABC
30+
value: {{ .DatasetFolder }}
3431
- name: SSL_CA_CERT
35-
value: "/.secrets/tls/ca.crt"
32+
value: "{{ .SslCaCert }}"
3633
- name: CLIENT_API_HOST
37-
value: https://api:8080
34+
value: {{ .ClientApiHost }}
3835
- name: CLIENT_ACCESS_TOKEN
39-
value:
36+
value: {{ .ClientAccesToken }}
4037
- name: MAIL_UPLOADER_NAME
41-
value: "John Doe"
38+
value: "{{ .MailUploaderName }}"
4239
- name: MAIL_UPLOADER
43-
value: uploader@email.com
40+
value: {{ .MailUploader }}
4441
- name: MAIL_ADDRESS
45-
value: yourmail@yourmailservice.com
42+
value: {{ .MailAddress }}
4643
- name: MAIL_PASSWORD
47-
value: yourpassword
44+
value: {{ .MailPassword }}
4845
- name: MAIL_SMTP_HOST
49-
value: yourmailservice.com
46+
value: {{ .MailSmptHost }}
5047
- name: MAIL_SMTP_PORT
51-
value: "587"
48+
value: "{{ .MailSmptPort }}"
5249
- name: DB_HOST
5350
valueFrom:
5451
secretKeyRef:
55-
name: sda-sda-svc-bpctl
52+
name: {{ .DbSecretName }}
5653
key: DB_HOST
5754
- name: DB_PORT
5855
valueFrom:
5956
secretKeyRef:
60-
name: sda-sda-svc-bpctl
57+
name: {{ .DbSecretName }}
6158
key: DB_PORT
6259
- name: DB_USER
6360
valueFrom:
6461
secretKeyRef:
65-
name: sda-sda-svc-bpctl
62+
name: {{ .DbSecretName }}
6663
key: DB_USER
6764
- name: DB_PASSWORD
6865
valueFrom:
6966
secretKeyRef:
70-
name: sda-sda-svc-bpctl
67+
name: {{ .DbSecretName }}
7168
key: DB_PASSWORD
7269
- name: DB_NAME
7370
valueFrom:
7471
secretKeyRef:
75-
name: sda-sda-svc-bpctl
72+
name: {{ .DbSecretName }}
7673
key: DB_NAME
7774
- name: DB_SCHEMA
7875
valueFrom:
7976
secretKeyRef:
80-
name: sda-sda-svc-bpctl
77+
name: {{ .DbSecretName }}
8178
key: DB_SCHEMA
8279
- name: DB_SSL_MODE
8380
valueFrom:
8481
secretKeyRef:
85-
name: sda-sda-svc-bpctl
82+
name: {{ .DbSecretName }}
8683
key: DB_SSL_MODE
8784
- name: DB_CA_CERT
8885
value: "/.secrets/tls/ca.crt"
8986
- name: DB_CLIENT_CERT
90-
value: /.secrets/tls/tls.crt
87+
value: "/.secrets/tls/tls.crt"
9188
- name: DB_CLIENT_KEY
92-
value: /.secrets/tls/tls.key
89+
value: "/.secrets/tls/tls.key"
9390
securityContext:
9491
allowPrivilegeEscalation: false
9592
runAsNonRoot: true
@@ -110,4 +107,4 @@ spec:
110107
- name: tls
111108
secret:
112109
defaultMode: 288
113-
secretName: sda-sda-svc-api-certs
110+
secretName: {{ .CertSecretName }}

‎internal/config/config.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,11 @@ type Config struct {
1414
SslCaCert string `mapstructure:"SSL_CA_CERT"`
1515
Timeout int `mapstructure:"JOB_TIMEOUT"`
1616
PollRate int `mapstructure:"JOB_POLL_RATE"`
17+
ExpectedNrFiles int `mapstructure:"JOB_EXPECTED_NR_FILES"`
1718
ClientApiHost string `mapstructure:"CLIENT_API_HOST"`
1819
ClientAccessToken string `mapstructure:"CLIENT_ACCESS_TOKEN"`
20+
CertSecretName string `mapstructure:"CERT_SECRET_NAME"`
21+
DbSecretName string `mapstructure:"DB_SECRET_NAME"`
1922
DbHost string `mapstructure:"DB_HOST"`
2023
DbPort int `mapstructure:"DB_PORT"`
2124
DbUser string `mapstructure:"DB_USER"`
@@ -42,6 +45,7 @@ func NewConfig(configPath string) (*Config, error) {
4245

4346
v.SetDefault("JOB_TIMEOUT", 4320)
4447
v.SetDefault("JOB_POLL_RATE", 180)
48+
v.SetDefault("JOB_EXPECTED_NR_FILES", 0)
4549

4650
if err := v.ReadInConfig(); err != nil {
4751
if _, ok := err.(viper.ConfigFileNotFoundError); ok {
@@ -70,6 +74,7 @@ func bindKeys(v *viper.Viper) {
7074
v.BindEnv("SSL_CA_CERT")
7175
v.BindEnv("JOB_TIMEOUT")
7276
v.BindEnv("JOB_POLL_RATE")
77+
v.BindEnv("JOB_EXPECTED_NR_FILES")
7378
v.BindEnv("CLIENT_API_HOST")
7479
v.BindEnv("CLIENT_ACCESS_TOKEN")
7580
v.BindEnv("DB_HOST")

0 commit comments

Comments
 (0)