Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NETOBSERV-101 R&D: Kube enricher write path for downstream operator #22

Merged
merged 3 commits into from
Oct 20, 2022
+54 −66
Merged

NETOBSERV-101 R&D: Kube enricher write path for downstream operator #22

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
13b379b
loki operator cluster role + binding
jpinsonneau Jul 20, 2022
8611185
loki operator gateway draft
jpinsonneau Jul 21, 2022
796dc04
openshift-network mode
jpinsonneau Aug 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions examples/loki-role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: lokistack-netobserv-tenant-logs
rules:
- apiGroups:
- 'loki.grafana.com'
resources:
- application
- infrastructure
- audit
resourceNames:
- logs
verbs:
- 'get'
- 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: lokistack-netobserv-tenant-logs
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: lokistack-netobserv-tenant-logs
subjects:
- kind: ServiceAccount
name: flowlogs-pipeline
namespace: network-observability
- kind: ServiceAccount
name: network-observability-plugin
namespace: network-observability
89 changes: 23 additions & 66 deletions hack_loki.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,91 +54,48 @@ kubectl apply -f hack/lokistack_dev.yaml

This will create `distributor`, `compactor`, `ingester`, `querier` and `query-frontend` components.

## Loki Operator on Openshift (WIP, gateway is [still unstable](https://github.com/ViaQ/loki-operator/pull/100/) and configuration may change)
## Loki Operator on Openshift with gateway
Loki operator on Openshift will allow you to configure [gateway](https://github.com/observatorium/api) for loki multi-tenancy & authentication

Check [Docs](https://github.com/ViaQ/loki-operator/tree/master/docs)
Check [Docs](https://loki-operator.dev/docs/prologue/introduction.md/)

### Requirements
- Install oc CLI for communicating with the cluster.
- Running Openshift cluster
- [Configured DEX](./hack_dex.md)
- A container registry that you and your openshift cluster can reach.

### Setup
Since loki-operator is not already available on operator hub, you will need to build it from sources for now.

Clone loki-operator repository and deploy
```bash
git clone https://github.com/ViaQ/loki-operator.git
cd loki-operator
make olm-deploy REGISTRY_ORG=$YOUR_QUAY_ORG VERSION=$VERSION
```
Install loki operator using Operator Hub. Open Openshift Console and move to
Administrator view -> Operators -> OperatorHub
Search for `loki`. You should find `Loki Operator` in `Red Hat` catalog.

[Create DEX instance](https://github.com/netobserv/documents/blob/main/hack_dex.md#create-dex-instance) in the `openshift-logging` namespace
Install the operator with default configuration.

Create aws bucket and secret. You can check [deploy-example-secret.sh](https://github.com/ViaQ/loki-operator/blob/master/hack/deploy-example-secret.sh) for infos.
Example with us-east-1 region for netobserv-loki bucket:
```bash
aws s3api create-bucket --bucket netobserv-loki --region us-east-1
oc -n openshift-logging create secret generic test --from-literal=endpoint="https://s3.us-east-1.amazonaws.com" --from-literal=region="us-east-1" --from-literal=bucketnames="netobserv-loki" --from-literal=access_key_id="XXXXXXXXXXXXXXXXXXXX" --from-literal=access_key_secret="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
Create a namespace called `openshift-logging`:
```

If you want to use internal HTTP urls, remove `--with-cert-signing-service`, `--with-service-monitors` and `--with-tls-service-monitors` flags in `config/overlays/openshift/manager_run_flags_patch.yaml`.
Your container spec should look like this :
```yaml
containers:
- name: manager
args:
- "--with-lokistack-gateway"
- "--with-lokistack-gateway-route"
kubectl create ns openshift-logging
```
Else you will have to create reencrypt routes to access services.

Create tenant secret with cliendID, clientSecret and ca according to your dex configuration:
```bash
oc create -n openshift-logging secret generic tenant-a --from-literal=clientID="tenant-a" --from-literal=clientSecret="password" --from-literal=issuerCAPath=""
```
`issuerCAPath` can be left empty if you want to use server default API CA file. Else use relative path in gateway pod.
Then create a `LokiStack` in `openshift-logging` namespace from:
Administrator view -> Operators -> Installed Operators -> Loki Operator -> LokiStack -> Create LokiStack
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way to do it from CLI?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can put a yaml in our examples but we will need to maintain it on their api changes

- ensure the name is `lokistack`
- set `Object Storage` -> `Secret`. Check [documentation](https://loki-operator.dev/docs/object_storage.md/).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any secret in particular? The command for creating the secret has been removed from the previous section

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The object storage secret from documentation examples. Since it's different for each storage, I prefer to be generic here

- ensure `Tenants Configuration` -> `Mode` is set to `openshift-logging`

Update `oidc secret name`, `issuerURL` and `redirectURL` routes in `hack/lokistack_gateway_dev.yaml`:
```yaml
secret:
name: tenant-a
issuerURL: https://dex-openshift-logging.apps.<MY_CLUSTER_URL>/dex/
redirectURL: http://gateway-openshift-logging.apps.<MY_CLUSTER_URL>/oidc/tenant-a/callback
```
You can check `examples/lokistack_gateway.yaml` in this repository for a compatible configuration with static users created in `examples/dex.yaml`.
`usernameClaim` will take dex email and `groupClaim` is empty since DEX staticPasswords doesn't support groups.
`subjects` users are taken from Openshift users matching with identities.
This will create `gateway`, `distributor`, `compactor`, `ingester`, `querier` and `query-frontend` components.

Create LokiStack instance with static mode:
```bash
oc -n openshift-logging apply -f hack/lokistack_gateway_dev.yaml
To allow `flowlogs-pipeline` to write to the gateway and `network-observability-plugin` to read from the gateway, you will need to create related `ClusterRole` and `ClusterRoleBinding` using:
```

OR

Open your Openshift Administrator Console and go to:
Installed Operators => Openshift Loki Operator (in openshift-logging namespace)
Click on `Create instance` in LokiStacks card
Copy / Paste `hack/lokistack_gateway_dev.yaml` content from sources to YAML tab

This will create `distributor`, `compactor`, `ingester`, `querier`, `query-frontend` and `lokistack-gateway` components.

Create gateway and gateway-status routes:
```bash
oc -n openshift-logging apply -f examples/gateway_routes.yaml
oc apply -f examples/loki-role.yaml
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off-topic: Should this be created automatically by the operator?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be in followup https://issues.redhat.com/browse/NETOBSERV-473

```

Gateway status will be available at:
`http://gateway-status-openshift-logging.apps.<MY_CLUSTER_URL>`

Loki will now be exposed at `api/logs/v1/tenant-a`. You can now open a private navigation and try the following url in your browser:
`http://gateway-openshift-logging.apps.<MY_CLUSTER_URL>/api/logs/v1/tenant-a/loki/api/v1/labels`
You will be redirected to DEX login before accessing this resource. It should return `status "success"`

Check all available routes in [api/logs/v1/http.go](https://github.com/observatorium/api/blob/main/api/logs/v1/http.go#L132)
Then you will be able to set the following configuration in `FlowCollector` for `application` tenant:
```yaml
loki:
tenantID: application
sendAuthToken: true
url: 'https://lokistack-gateway-http.openshift-logging.svc.cluster.local:8080/api/logs/v1/application/'
```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add these fields into the config/samples/flows_v1alpha1_flowcollector.yaml file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we will automatically deploy loki-operator from our operator, these will be forced to use the loki stack instance so the user will not have to set them.
If the user manage it's instance manually, I guess it's better to keep this separate (at least in another yaml file).

Note: I'm still looking at tenantID to add a network tenant in loki-operator.

### Troubleshooting
- Logs are by default `--log.level=warn`.
Expand Down