Skip to content

installation.md

Latest commit

 

History

History
285 lines (181 loc) · 7 KB

installation.md

File metadata and controls

285 lines (181 loc) · 7 KB
Raw

Installation & Bootstrap Guide

This document describes the end-to-end bootstrap flow for deploying the observability stack using Argo CD and GitOps.

It assumes a pre-existing Kubernetes cluster and focuses exclusively on the GitOps control plane and platform reconciliation.

Prerequisites

  • Kubernetes cluster (tested with AWS EKS)
  • kubectl configured with cluster-admin access
  • Git installed
  • (Optional) argocd CLI for local interaction

Infrastructure prerequisite (mandatory)

This repository assumes that the Kubernetes cluster and all infrastructure-level dependencies already exist and are provisioned outside of this GitOps repository.

A reference, production-aligned implementation is provided here:

https://github.com/LaurisNeimanis/aws-eks-platform

That repository is responsible for provisioning:

  • AWS EKS cluster
  • Worker node groups (capacity baseline: 2× t3.medium)
  • VPC, subnets, routing, and security groups
  • IAM roles and workload identity bindings
  • StorageClasses (e.g. gp3)
  • S3 buckets and access policies required by Loki

This GitOps repository does not create or modify infrastructure resources. All infrastructure must be in place and functional before proceeding.

1. Clone the Repository

Clone the repository locally:

git clone https://github.com/LaurisNeimanis/gitops-observability-stack.git
cd gitops-observability-stack

This repository is the single source of truth for the observability platform state.

2. Install Argo CD (Bootstrap Layer)

Argo CD is installed using a Kustomize-based bootstrap located under gitops/bootstrap.

Apply it once:

kubectl apply -k gitops/bootstrap/argocd

Verify that Argo CD pods are running:

kubectl get pods -n argocd

Wait until all components are in Running state before proceeding.

3. (Optional) Access Argo CD UI & CLI

This step is optional and intended for local inspection and demo purposes.

Port-forward Argo CD UI

kubectl port-forward svc/argocd-server -n argocd 8080:443

Access the UI at:

https://localhost:8080

Login Credentials

Username

admin

Password

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d

(Optional) CLI Login

argocd login localhost:8080

4. Install Argo CD Projects (Required)

Before deploying any applications, the Argo CD Projects must be created.

These define:

  • allowed source repositories
  • allowed destination namespaces
  • cluster-level permissions (CRDs, RBAC)

Apply the projects manifest:

kubectl apply -k gitops/argo/projects

This step is mandatory. Without it, subsequent applications will fail authorization and sync.

5. Bootstrap the Platform (GitOps Entry Point)

Apply the root application, which uses the App-of-Apps pattern:

kubectl apply -f gitops/argo/root-application.yaml

From this point onward, Argo CD takes full control.

It will reconcile, in order:

  • Prometheus Operator CRDs (sync-wave -1)
  • Observability platform components
  • Custom alerting rules

No further manual kubectl apply commands are required.

6. Reconciliation Behavior

Once bootstrapped, Argo CD enforces:

  • Automated sync
  • Self-healing
  • Controlled ordering via sync-wave
  • Explicit CRD lifecycle management

This ensures deterministic and reproducible platform state.

7. Post-bootstrap Resource Sanity Check (Required)

After Argo CD has finished reconciling all components, verify node-level resource usage.

Run:

kubectl top nodes

On the documented baseline (2× t3.medium), steady-state usage should remain below approximately 70% CPU and memory per node.

Sustained usage above this threshold indicates that the cluster does not have sufficient capacity for this observability stack.

This check ensures:

  • Prometheus is not memory-constrained
  • Loki ingestion and compaction are stable
  • No immediate scaling or OOM risk exists

Do not proceed with further configuration until node-level resource usage is within acceptable limits.

8. Configure Grafana Admin Credentials (Required)

Grafana is configured to read admin credentials from an externally managed Kubernetes Secret.

The secret must exist before Grafana can be accessed:

kubectl -n observability create secret generic grafana-admin --from-literal=admin-user=admin --from-literal=admin-password='CHANGE_ME'

Grafana access will not be available until this secret exists.

9. Configure Alertmanager Slack Webhooks (Required for live notifications)

Alertmanager needs valid Slack Incoming Webhook URLs before it can deliver notifications.

The committed values files intentionally contain placeholder webhook URLs only:

  • gitops/apps/platform/observability/kube-prometheus-stack/values/values.yaml
  • gitops/apps/platform/observability/kube-prometheus-stack/values/values-dev.yaml

Do not commit real Slack webhook URLs to this repository. For a real environment, provide those values through a private overlay or a secret-management workflow such as SOPS, External Secrets, or your platform's standard secret injection mechanism.

The current public/demo repository documents the required integration point, but it does not implement secret material management.

10. (Optional) Access Platform Components

These steps are not required for installation, but useful for validation and demos.

Grafana

Grafana uses the externally provided admin credentials created earlier.

  • Username: admin
  • Password: value defined in the grafana-admin Secret

Port-forward Grafana:

kubectl -n observability port-forward svc/kube-prometheus-stack-grafana 3000:80

Access:

http://localhost:3000

Alertmanager

kubectl -n observability port-forward svc/kube-prometheus-stack-alertmanager 9093:9093

Access:

http://localhost:9093

Notes & Design Considerations

  • Argo CD Projects are a hard prerequisite for application sync
  • Grafana admin credentials are provided externally by design
  • Alertmanager requires explicit notification configuration
  • CRDs are managed explicitly and not pruned
  • All applications use automated sync with self-healing enabled
  • Namespace creation is handled automatically by Argo CD
  • Infrastructure dependencies (S3, IAM, StorageClasses) are out of scope

Summary

After completing these steps:

  • Argo CD becomes the authoritative control plane
  • The observability stack is fully GitOps-managed
  • Dashboards become available once Grafana credentials are provided
  • Alert delivery becomes available only after webhook values are supplied through an external secret-management path
  • All future changes are performed via Git commits

This bootstrap flow is designed to be:

  • minimal
  • deterministic
  • production-aligned

For full platform context and infrastructure details, review the infrastructure repository.