-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathminio.yaml
318 lines (281 loc) · 12.9 KB
/
minio.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
blocks:
- name: deploy_minio_webui_domain_blk
content:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "myminio-ui-ingress"
namespace: minio
annotations:
cert-manager.io/cluster-issuer: cluster-issuer
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
external-dns.alpha.kubernetes.io/hostname: "{{ $cndi.get_prompt_response(minio_webui_domain) }}"
$cndi.comment(nginx-ingress-annotation): Increase the maximum body size to 50TiB
$cndi.comment(nginx-ingress-annotation-cont): to align with Minio max file size
nginx.ingress.kubernetes.io/proxy-body-size: 50000g
spec:
tls:
- hosts:
- "{{ $cndi.get_prompt_response(minio_webui_domain) }}"
secretName: cluster-issuer-private-key-minio-ui
rules:
- host: "{{ $cndi.get_prompt_response(minio_webui_domain) }}"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: "myminio-console"
port:
name: https-console
prompts:
- $cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/core-prompts.yaml):
{}
- $cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/{{ $cndi.get_prompt_response(deployment_target_provider) }}/core-prompts.yaml):
{}
- name: server_domain_name
default: myminiotenant.example.com
message: "What hostname should the MinIO tenant server be accessible at"
type: Input
validators:
- hostname # FQDN
- name: servers
default: 4
message: "Specify the number of tenant servers to deploy:"
type: Number
- name: volumes_per_server
default: 1
message: "Specify the number of volumes per server:"
type: Number
- name: deploy_minio_webui_domain
default: true
message: >-
Do you want to expose your minio console ui to the web?
type: Confirm
- name: minio_webui_domain
default: myminioconsole.example.com
message: "What hostname should the MinIO console ui be accessible at"
type: Input
validators:
- hostname # FQDN
- min_length: 3
condition:
- "{{ $cndi.get_prompt_response(deploy_minio_webui_domain) }}"
- ==
- true
- name: access_key
default: "{{ $cndi.get_random_string(12) }}"
message: 'Enter the "Username/Accesskey" for the MinIO tenant user:'
type: Secret
validators:
- min_length: 3
- name: secret_key
default: "{{ $cndi.get_random_string(18) }}"
message: 'Enter the "Password/SecretKey" for your MinIO tenant user:'
type: Secret
validators:
- min_length: 8
- name: bucket_name
default: myminiobucket
message: "Enter the name of the bucket to create in your MinIO tenant account:"
type: Input
outputs:
cndi_config:
cndi_version: v2
project_name: "{{ $cndi.get_prompt_response(project_name) }}"
provider: "{{ $cndi.get_prompt_response(deployment_target_provider) }}"
distribution: "{{ $cndi.get_prompt_response(deployment_target_distribution) }}"
infrastructure:
cndi:
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/observability-config.yaml):
condition:
- "{{ $cndi.get_prompt_response(deploy_grafana_ingress) }}"
- ==
- true
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/argocd-config.yaml):
condition:
- "{{ $cndi.get_prompt_response(deploy_argocd_ingress) }}"
- ==
- true
cert_manager:
email: "{{ $cndi.get_prompt_response(cert_manager_email) }}"
external_dns:
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/external-dns/config/{{ $cndi.get_prompt_response(dns_provider) }}.yaml):
condition:
- "{{ $cndi.get_prompt_response(enable_external_dns) }}"
- ==
- true
nodes:
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/{{ $cndi.get_prompt_response(deployment_target_provider) }}/basic-node-pool.yaml):
{}
cluster_manifests:
minio-server-ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minio-server-ingress
namespace: minio
annotations:
cert-manager.io/cluster-issuer: cluster-issuer
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
external-dns.alpha.kubernetes.io/hostname: "{{ $cndi.get_prompt_response(server_domain_name) }}"
$cndi.comment(nginx-ingress-annotation-2): Increase the maximum body size to 50TiB
$cndi.comment(nginx-ingress-annotation-2-cont): to align with Minio max file size
nginx.ingress.kubernetes.io/proxy-body-size: 50000g
spec:
tls:
- hosts:
- "{{ $cndi.get_prompt_response(server_domain_name) }}"
secretName: cluster-issuer-private-key-minio
rules:
- host: "{{ $cndi.get_prompt_response(server_domain_name) }}"
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: minio
port:
name: https-minio
minio-tenant-root-creds-secret:
apiVersion: v1
kind: Secret
metadata:
name: minio-tenant-root-creds-secret
namespace: minio
annotations:
sealedsecrets.bitnami.com/cluster-wide: "true"
labels:
argocd.argoproj.io/secret-type: cluster
app.kubernetes.io/instance: tenant
type: Opaque
stringData:
config.env: $cndi_on_ow.seal_secret_from_env_var(MINIO_TENANT_ROOT_CREDENTIALS)
minio-tenant-account:
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
annotations:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
name: "myminio"
namespace: minio
labels:
app: minio
spec:
features:
bucketDNS: false
users:
- name: admin-user
buckets:
- name: "{{ $cndi.get_prompt_response(bucket_name) }}"
mountPath: /data
configuration:
name: minio-tenant-root-creds-secret
pools:
- servers: "{{ $cndi.get_prompt_response(servers) }}"
name: pool-0
volumesPerServer: "{{ $cndi.get_prompt_response(volumes_per_server) }}"
volumeClaimTemplate:
apiVersion: v1
kind: persistentvolumeclaims
metadata: {}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: rwo
admin-user-secret:
apiVersion: v1
kind: Secret
metadata:
name: admin-user
namespace: minio
type: Opaque
stringData:
CONSOLE_ACCESS_KEY: $cndi_on_ow.seal_secret_from_env_var(MINIO_TENANT_ACCESS_KEY)
CONSOLE_SECRET_KEY: $cndi_on_ow.seal_secret_from_env_var(MINIO_TENANT_SECRET_KEY)
minio-webui-ingress:
$cndi.get_block(deploy_minio_webui_domain_blk):
condition:
- "{{ $cndi.get_prompt_response(deploy_minio_webui_domain) }}"
- ==
- true
external-dns-secret:
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/external-dns/secret/{{ $cndi.get_prompt_response(dns_provider) }}.yaml):
condition:
- "{{ $cndi.get_prompt_response(enable_external_dns) }}"
- ==
- true
airbyte-ns:
apiVersion: v1
kind: Namespace
metadata:
name: minio
applications:
minio:
chart: operator
destinationNamespace: minio
repoURL: https://operator.min.io/
targetRevision: 6.0.3
env:
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/git-credentials-{{ $cndi.get_prompt_response(git_credentials_mode) }}-env.yaml):
{}
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/{{ $cndi.get_prompt_response(deployment_target_provider) }}/env.yaml):
{}
$cndi.get_block(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/env.yaml):
{}
$cndi.comment(minio-heading): Minio Connection Parameters
MINIO_TENANT_ACCESS_KEY: "{{ $cndi.get_prompt_response(access_key) }}"
MINIO_TENANT_SECRET_KEY: "{{ $cndi.get_prompt_response(secret_key) }}"
MINIO_TENANT_ROOT_CREDENTIALS: |
'export MINIO_ROOT_USER="{{ $cndi.get_random_string(18) }}"
export MINIO_ROOT_PASSWORD="{{ $cndi.get_random_string(24) }}"'
MINIO_TENANT_SERVER_DOMAIN_NAME: "{{ $cndi.get_prompt_response(server_domain_name) }}"
MINIO_TENANT_NAME: "myminio"
MINIO_TENANT_BUCKET_NAME: "{{ $cndi.get_prompt_response(bucket_name) }}"
MINIO_CLIENT_COMMAND: mc alias set myminio https://{{ $cndi.get_prompt_response(server_domain_name) }} {{ $cndi.get_prompt_response(access_key) }} {{ $cndi.get_prompt_response(secret_key) }}
readme:
project_name: "# {{ $cndi.get_prompt_response(project_name) }}"
$cndi.get_string(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/common/cluster/core-readme.md):
{}
$cndi.get_string(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/{{ $cndi.get_prompt_response(deployment_target_provider) }}/core.md):
{}
$cndi.get_string(https://raw.githubusercontent.com/polyseam/cndi/main/blocks/{{ $cndi.get_prompt_response(deployment_target_provider) }}/{{ $cndi.get_prompt_response(deployment_target_distribution) }}.md):
{}
minio: |
## Minio Deployment Guide
This template deploys a Standalone production ready
[minio operator and tenant](https://github.com/minio/operator/tree/master)
MinIO is a high-performance, distributed object storage system designed to support cloud-native applications with S3-compatible storage for unstructured, semi-structured and structured data.
Some benefits of combining MinIO include:
High Performance: MinIO writes as fast as they come in. MinIO is used to build high performance infrastructure for machine learning, analytics and application data workloads.
Scalability: MinIO handles large amounts of data and scales horizontally across multiple nodes, This allows organizations to store and process massive amounts of data in real-time, making it suitable for big data and high-velocity data streaming use cases.
Durability: MinIO provides durable storage, allowing organizations to retain data for long periods of time, such as for historical analysis, compliance requirements, or data recovery purposes.
Fault Tolerance: MinIO erasure codes data across multiple nodes, providing fault tolerance and ensuring data durability.
Easy Integration: MinIO is easily integrated with like aws s3, a built-in framework for connecting with external systems. This makes it straightforward to stream data to MinIO for storage, and vice versa for data retrieval.
## Testing Your Minio Connection
### Using MinIO Console
When you first open the MinIO console web interface, you'll be prompted to enter your access and secret keys. The credentials are in the ENV file:
Username: <MINIO_TENANT_ACCESS_KEY>
Password: <MINIO_TENANT_SECRET_KEY>
Once logged in, you will see the MinIO dashboard, which provides an easy-to-use interface for managing buckets and objects.
You can create new buckets, upload files, and manage your stored data directly through this interface.
### Using MinIO `mc` Command Line Tool
To test the connection you can install the [MinIO CLient](https://min.io/docs/minio/linux/reference/minio-mc.html).
# Set up MinIO alias
mc alias set myminio https://<MINIO_TENANT_SERVER_DOMAIN_NAME> <MINIO_TENANT_ACCESS_KEY> <MINIO_TENANT_SECRET_KEY>
# Create a new bucket if it doesn't exist
mc mb myminio/myminiobucket || echo "Bucket already exists";
# Create a file to upload
echo "Hello, MinIO!" > /tmp/hello.txt;
# Copy the file to the MinIO bucket
mc cp /tmp/hello.txt myminio/myminiobucket;
# List objects in the bucket to verify the upload
mc ls myminio/myminiobucket;
You can find the MINIO_TENANT_ACCESS_KEY MINIO_TENANT_SECRET_KEY and MINIO_TENANT_SERVER_DOMAIN_NAME in the .env file.