Skip to content

kentik/kagent-helm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Kagent Helm Chart

A Helm chart for deploying Kentik Universal Agent (kagent) with support for multiple deployment patterns: StatefulSet and DaemonSet.

Prerequisites

  • Kubernetes 1.20+
  • Helm 3.x
  • Kentik account with API access

Quick Start

1. Install the Chart

# Clone the repository
git clone https://github.com/kentik/kagent-helm.git
cd kagent-helm

# Install with StatefulSet pattern (persistent storage)
helm install kagent . \
  --set-string kagent.companyId=YOUR_COMPANY_ID

Alternatively, install directly from the GitHub repository without cloning:

# Install with StatefulSet pattern
helm install kagent oci://ghcr.io/kentik/kagent-helm \
  --set-string kagent.companyId=YOUR_COMPANY_ID

# Or from a specific version/tag
helm install kagent https://github.com/kentik/kagent-helm/archive/refs/heads/main.tar.gz \
  --set-string kagent.companyId=YOUR_COMPANY_ID

2. Verify Installation

# Check deployment status
kubectl get pods -l app.kubernetes.io/name=kagent

# View logs
kubectl logs -l app.kubernetes.io/name=kagent --tail=100

3. Authorize agent via UI

  • Go to Kentik Portal → Settings → Universal Agents
  • Find your new agent in the "Pending Authorization" section
  • Click "Authorize" to approve the agent

Deployment Patterns

StatefulSet (Persistent Storage)

Use Case: Standard deployments requiring persistent storage for flow data buffers or agent state.

Characteristics:

  • Ordered pod naming (kagent-0, kagent-1, ...)
  • Persistent volumes for data and keypair
  • Stable identity across pod restarts
  • Ordered rolling updates

Important: The agent's ed25519 keypair is stored in /opt/ua/keys/ and MUST persist across pod restarts as it's the agent's unique identity.

Data Persistence: The StatefulSet creates two persistent volumes per pod:

  • /data (10Gi default): Application data, buffers, logs, temp files
  • /opt/ua/keys (100Mi): ed25519 keypair (agent identity)

Scaling:

Before scaling, you need to ensure that the secret with agent's identity includes keypairs for each additional agent (if scaling up). Then you can run a regular scale command:

# Scale StatefulSet
kubectl scale statefulset kagent --replicas=3

Cleanup: PVCs are NOT deleted automatically on uninstall:

# Delete the release
helm uninstall kagent

# Manually clean up PVCs
kubectl delete pvc -l app.kubernetes.io/name=kagent

DaemonSet (Node-Level)

Use Case: Node-level network monitoring, SNMP polling of node interfaces, or collecting node-specific metrics.

Characteristics:

  • Exactly one pod per Kubernetes node
  • Automatic pod scheduling on node add/remove
  • Tolerations for control-plane nodes included
  • Lower resource requests per pod

Storage: DaemonSet deployments only support hostPath for persistent storage (not PVC). Set persistence.type: hostPath when using DaemonSet.

Check examples/daemon-set.yaml to get started.

hostPath Permissions: The hostPath directories on each node must be readable and writable by user ID 500 (the default runAsUser). Create and set permissions before deploying:

# On each node, create directories with correct ownership
sudo mkdir -p /var/lib/kagent/data /var/lib/kagent/keys
sudo chown -R 500:500 /var/lib/kagent

Node Coverage:

# Verify one pod per node
kubectl get nodes --no-headers | wc -l
kubectl get pods -l app.kubernetes.io/name=kagent --no-headers | wc -l
# These numbers should match

# View pods with node assignment
kubectl get pods -l app.kubernetes.io/name=kagent -o wide

Configuration

Kagent Settings

Parameter Description Default
kagent.provisioningToken Provisioning token from CreateAgent API ""
kagent.companyId Company ID for agent registration 1013
kagent.agentId Agent ID for Terraform tracking ""
kagent.releaseChannel Release channel (stable, beta, dev) stable
kagent.diskReservation.enabled Enable disk space reservation true
kagent.diskReservation.initialSize Initial reserved disk space 200MB
kagent.logDest Log destination (stdout, stderr, discard, filename) stdout
kagent.logLevel Log level (debug, info, warn, error) info
kagent.apiEndpoint Kentik API endpoint grpc.api.kentik.com:443

Deployment Configuration

Parameter Description Default
deploymentType Pattern: statefulset, daemonset statefulset
replicaCount Number of replicas (StatefulSet only) 1
image.repository Kagent container image kentik/kagent
image.tag Image tag v5.0.1
image.pullPolicy Image pull policy IfNotPresent

Persistence

Parameter Description Default
persistence.enabled Enable persistent storage true
persistence.type Volume type: pvc, hostPath, emptyDir pvc
persistence.pvc.storageClass Storage class (empty = cluster default) ""
persistence.pvc.size Data volume size 10Gi
persistence.pvc.accessModes PVC access modes [ReadWriteOnce]
persistence.hostPath.path Host path for data (DaemonSet) /var/lib/kagent/data
persistence.hostPath.type HostPath volume type DirectoryOrCreate
persistence.keypair.enabled Separate volume for keypair true
persistence.keypair.type Keypair storage: secret, pvc, hostPath, emptyDir secret
persistence.keypair.pvc.storageClass Storage class for keypair PVC ""
persistence.keypair.pvc.size Keypair volume size (when using PVC) 100Mi
persistence.keypair.pvc.accessModes Keypair PVC access modes [ReadWriteOnce]
persistence.keypair.hostPath.path Host path for keypair (DaemonSet) /var/lib/kagent/keys
persistence.keypair.hostPath.type Keypair hostPath volume type DirectoryOrCreate
persistence.keypair.secret.name Secret name pattern (empty = auto-generated) ""

Resources

Parameter Description Default
resources.requests.cpu CPU request 100m (50m for DaemonSet)
resources.requests.memory Memory request 128Mi (64Mi for DaemonSet)
resources.limits.cpu CPU limit 500m (200m for DaemonSet)
resources.limits.memory Memory limit 512Mi (256Mi for DaemonSet)

Security

Parameter Description Default
podSecurityContext.runAsNonRoot Run as non-root user true
podSecurityContext.runAsUser User ID 500
podSecurityContext.fsGroup File system group 500
serviceAccount.create Create service account true
rbac.create Create RBAC resources false
networkPolicy.enabled Enable network policy false

Cloud-Specific Examples

AWS (EKS)

# statefulset-aws-gp3.yaml
deploymentType: statefulset
persistence:
  pvc:
    storageClass: "gp3"
  keypair:
    pvc:
      storageClass: "gp3"

Azure (AKS)

# statefulset-azure-premium.yaml
deploymentType: statefulset
persistence:
  pvc:
    storageClass: "managed-premium"
  keypair:
    pvc:
      storageClass: "managed-premium"

Google Cloud (GKE)

# statefulset-gcp-standard.yaml
deploymentType: statefulset
persistence:
  pvc:
    storageClass: "standard-rwo"
  keypair:
    pvc:
      storageClass: "standard-rwo"

Migration Between Patterns

IMPORTANT: Switching between deployment patterns (e.g., DaemonSet → StatefulSet) requires a clean uninstall and fresh install. In-place upgrades between patterns are NOT supported due to Kubernetes resource type immutability.

# Uninstall existing release
helm uninstall kagent

# Clean up PVCs if switching from StatefulSet
kubectl delete pvc -l app.kubernetes.io/name=kagent

# Install with new pattern
helm install kagent . --set deploymentType=statefulset

Troubleshooting

Agent Not Connecting

  1. Check provisioning token:
kubectl get secret kagent-secret -o jsonpath='{.data.K_REGISTER_PROVISIONING_TOKEN}' | base64 -d
  1. View agent logs:
kubectl logs -l app.kubernetes.io/name=kagent --tail=100
  1. Verify network connectivity:
kubectl exec -it <pod-name> -- ping grpc.api.kentik.com

StatefulSet PVC Issues

  1. Check PVC status:
kubectl get pvc -l app.kubernetes.io/name=kagent
  1. Verify storage class exists:
kubectl get storageclass
  1. Check pod volume mounts:
kubectl exec -it kagent-0 -- ls -la /data
kubectl exec -it kagent-0 -- ls -la /opt/ua/keys

Keypair Persistence Issues

Problem: Agent creates new identity after pod restart, or pod fails to start with keypair errors.

Using Secret-Based Storage

  1. Verify secrets exist and have correct names:
# List all kagent secrets
kubectl get secrets | grep kagent

# Should show: kagent-0-secret, kagent-1-secret, etc.
# Verify secret content
kubectl get secret kagent-0-secret -o yaml

# Check that it has the required keys
kubectl get secret kagent-0-secret -o jsonpath='{.data}' | jq 'keys'
# Should output: ["private_key.pem", "public_key.pem"]
  1. Verify secret content is valid:
# Decode and check private key format
kubectl get secret kagent-0-secret -o jsonpath='{.data.private_key\.pem}' | base64 -d
# Should start with: -----BEGIN PRIVATE KEY-----

# Decode and check public key format
kubectl get secret kagent-0-secret -o jsonpath='{.data.public_key\.pem}' | base64 -d
# Should start with: -----BEGIN PUBLIC KEY-----
  1. Check pod init container logs (when using secret type):
# View init container logs
kubectl logs kagent-0 -c setup-keypair

# Should show: "Keypair for pod-0 copied successfully"
# If you see "Warning: No keypair found", the secret name is incorrect
  1. Verify keypair was mounted correctly:
# Check if keys exist in the container
kubectl exec -it kagent-0 -- ls -la /opt/ua/keys/
# Should show: private_key.pem, public_key.pem

# Verify permissions
kubectl exec -it kagent-0 -- stat /opt/ua/keys/private_key.pem
# Should show mode: 0400 (read-only)

Using PVC-Based Storage

  1. Check PVC status:
# List keypair PVCs
kubectl get pvc | grep keys

# Verify PVC is bound
kubectl get pvc keys-kagent-0
# Status should be "Bound"
  1. Verify mount inside pod:
kubectl exec -it kagent-0 -- ls -la /opt/ua/keys/
# Should show: agent.key (private key), agent.pub (public key)
  1. Check if keys persist across restarts:
# Note the key fingerprint
kubectl exec -it kagent-0 -- cat /opt/ua/keys/agent.pub

# Delete the pod (StatefulSet will recreate it)
kubectl delete pod kagent-0

# Wait for pod to restart
kubectl wait --for=condition=Ready pod/kagent-0 --timeout=60s

# Verify the same key exists
kubectl exec -it kagent-0 -- cat /opt/ua/keys/agent.pub
# Should match the previous fingerprint

Common Issues

Issue: Error: secret "kagent-0-secret" not found

  • Cause: Secrets not created before helm install
  • Solution: Create secrets first (see Generating Keypairs)

Issue: Warning: No keypair found for pod-X

  • Cause: Secret name doesn't match expected pattern
  • Solution: Ensure secret name is {{ .Release.Name }}-{{ replica-index }}-secret

Issue: Pod stuck in Init:Error state

  • Cause: Invalid secret content or missing keys
  • Solution: Verify secret has both private_key.pem and public_key.pem keys with valid PEM content

Issue: Agent shows "Keypair validation failed"

  • Cause: Corrupted or invalid keypair format
  • Solution: Regenerate keypair using generate-secrets.sh or verify external secret manager integration

DaemonSet Not on All Nodes

  1. Check node taints:
kubectl get nodes -o json | jq '.items[].spec.taints'
  1. Verify tolerations are applied:
kubectl get daemonset kagent -o yaml | grep -A 10 tolerations
  1. Check node selectors:
# If you have node selectors, ensure nodes have matching labels
kubectl get nodes --show-labels

Advanced Configuration

Keypair Management

The agent's ed25519 keypair serves as its unique identity in the Kentik platform. This keypair MUST persist across pod restarts to maintain the agent's identity and prevent orphaned agents.

Storage Options

The chart supports four keypair storage types via persistence.keypair.type:

  1. secret (Default, Recommended): Kubernetes Secrets - best for production, GitOps, and external secret managers
  2. pvc: PersistentVolumeClaim - traditional persistent storage approach
  3. hostPath: Node-local storage - useful for DaemonSet deployments
  4. emptyDir: Temporary storage - NOT recommended for production (keypair lost on pod restart)

Secret-Based Keypair Storage (Recommended)

Secret-based storage (persistence.keypair.type: secret) provides several advantages:

  • Production-ready: Integrates with external secret managers (Vault, AWS Secrets Manager, etc.)
  • GitOps-friendly: Works with Sealed Secrets, SOPS, or other encryption tools
  • Easy key rotation: Update secrets centrally and restart pods to pick up new keys
  • Stateless-compatible: Works with all deployment patterns (StatefulSet, DaemonSet, Deployment)
Secret Naming Convention

Each replica requires its own secret following this naming pattern:

{{ .Release.Name }}-{{ replica-index }}-secret

Examples:

  • Replica 0: kagent-0-secret
  • Replica 1: kagent-1-secret
  • Replica 2: kagent-2-secret

Each secret must contain exactly two keys:

  • private_key.pem: Ed25519 private key (PEM format)
  • public_key.pem: Ed25519 public key (PEM format)
Option 1: Generating Keypairs with Helper Script

Use the provided generate-secrets.sh script to quickly create ed25519 keypairs:

# Generate secrets for 3 replicas
./generate-secrets.sh 3

# This creates:
# - generated_secrets/generated_secrets.yaml (Kubernetes secret manifest)
# - generated_secrets/private_key_*.pem (individual keypair files for backup)
# - generated_secrets/public_key_*.pem (individual public keys for backup)

# Apply the secrets before installing the chart
kubectl apply -f generated_secrets/generated_secrets.yaml

# Install kagent (it will automatically use the pre-created secrets)
helm install kagent . \
  --set deploymentType=statefulset \
  --set replicaCount=3

# For custom release name
RELEASE_NAME=my-kagent ./generate-secrets.sh 3
kubectl apply -f generated_secrets/generated_secrets.yaml
helm install my-kagent . --set replicaCount=3

Important: Keep the generated PEM files in generated_secrets/ directory as backups. Store them securely (e.g., password-protected archive, secret management system).

Using PVC for Keypair Storage

If you prefer PersistentVolumeClaims for keypairs instead of secrets:

# StatefulSet with PVC-based keypair storage
helm install kagent . \
  --set deploymentType=statefulset \
  --set persistence.keypair.type=pvc

Using hostPath for Keypair Storage (DaemonSet)

For DaemonSet deployments with node-local keypair storage:

# DaemonSet pattern with hostPath keypairs
helm install kagent . \
  --set deploymentType=daemonset \
  --set persistence.keypair.enabled=true \
  --set persistence.keypair.type=hostPath \
  --set persistence.keypair.hostPath.path=/var/lib/kagent/keys
Option 2: Integration with External Secret Managers

For production environments, integrate with external secret managers to avoid storing sensitive keypairs in Git:

External Secrets Operator (ESO)

External Secrets Operator syncs secrets from external providers to Kubernetes Secrets.

Prerequisites:

  1. Install External Secrets Operator in your cluster
  2. Configure a SecretStore for your provider

AWS Secrets Manager Example:

# 1. Store keypairs in AWS Secrets Manager (do this for each replica)
aws secretsmanager create-secret \
  --name kagent/keypairs/replica-0 \
  --description "Kagent replica 0 ed25519 keypair" \
  --secret-string "{\"private_key\":\"$(cat generated_secrets/private_key_0.pem)\",\"public_key\":\"$(cat generated_secrets/public_key_0.pem)\"}"

# Repeat for replica-1, replica-2, etc.

Create ExternalSecret resources:

# external-secrets.yaml
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kagent-0-keypair
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: kagent-0-secret
    creationPolicy: Owner
  data:
  - secretKey: private_key.pem
    remoteRef:
      key: kagent/keypairs/replica-0
      property: private_key
  - secretKey: public_key.pem
    remoteRef:
      key: kagent/keypairs/replica-0
      property: public_key
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kagent-1-keypair
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: kagent-1-secret
    creationPolicy: Owner
  data:
  - secretKey: private_key.pem
    remoteRef:
      key: kagent/keypairs/replica-1
      property: private_key
  - secretKey: public_key.pem
    remoteRef:
      key: kagent/keypairs/replica-1
      property: public_key
---
# Repeat for each replica...

Deploy and install:

# Apply ExternalSecret resources (creates the kubernetes secrets)
kubectl apply -f external-secrets.yaml

# Wait for secrets to be created
kubectl wait --for=condition=Ready externalsecret/kagent-0-keypair --timeout=60s

# Install kagent
helm install kagent . \
  --set deploymentType=statefulset \
  --set replicaCount=2 \
  --set persistence.keypair.type=secret

Google Cloud Secret Manager Example:

# 1. Store in GCP Secret Manager
gcloud secrets create kagent-replica-0-private \
  --data-file=generated_secrets/private_key_0.pem

gcloud secrets create kagent-replica-0-public \
  --data-file=generated_secrets/public_key_0.pem

# 2. Create ExternalSecret
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kagent-0-keypair
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcpsm-secret-store
    kind: SecretStore
  target:
    name: kagent-0-secret
  data:
  - secretKey: private_key.pem
    remoteRef:
      key: kagent-replica-0-private
  - secretKey: public_key.pem
    remoteRef:
      key: kagent-replica-0-public
EOF

Azure Key Vault Example:

# 1. Store in Azure Key Vault
az keyvault secret set \
  --vault-name my-keyvault \
  --name kagent-replica-0-private \
  --file generated_secrets/private_key_0.pem

az keyvault secret set \
  --vault-name my-keyvault \
  --name kagent-replica-0-public \
  --file generated_secrets/public_key_0.pem

# 2. Create ExternalSecret
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kagent-0-keypair
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: SecretStore
  target:
    name: kagent-0-secret
  data:
  - secretKey: private_key.pem
    remoteRef:
      key: kagent-replica-0-private
  - secretKey: public_key.pem
    remoteRef:
      key: kagent-replica-0-public
EOF
HashiCorp Vault

Use the Vault Secrets Operator or vault-csi-provider:

Vault KV Secrets Example:

# 1. Store keypairs in Vault KV store
vault kv put secret/kagent/replica-0 \
  private_key="$(cat generated_secrets/private_key_0.pem)" \
  public_key="$(cat generated_secrets/public_key_0.pem)"

# 2. Create VaultStaticSecret (using Vault Secrets Operator)
cat <<EOF | kubectl apply -f -
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: kagent-0-keypair
spec:
  vaultAuthRef: vault-auth
  mount: secret
  path: kagent/replica-0
  refreshAfter: 1h
  destination:
    name: kagent-0-secret
    create: true
    transformation:
      templates:
        private_key.pem:
          text: "{{ .Secrets.private_key }}"
        public_key.pem:
          text: "{{ .Secrets.public_key }}"
EOF
Sealed Secrets (GitOps)

Use Sealed Secrets to encrypt secrets for safe Git storage:

# 1. Generate keypairs using the helper script
./generate-secrets.sh 3

# 2. Create individual secrets and seal them
for i in 0 1 2; do
  kubectl create secret generic kagent-${i}-secret \
    --from-file=private_key.pem=generated_secrets/private_key_${i}.pem \
    --from-file=public_key.pem=generated_secrets/public_key_${i}.pem \
    --dry-run=client -o yaml | \
  kubeseal -o yaml > kagent-${i}-sealed-secret.yaml
done

# 3. Commit sealed secrets to Git
git add kagent-*-sealed-secret.yaml
git commit -m "Add kagent sealed secrets"

# 4. Apply sealed secrets (via GitOps or manually)
kubectl apply -f kagent-0-sealed-secret.yaml
kubectl apply -f kagent-1-sealed-secret.yaml
kubectl apply -f kagent-2-sealed-secret.yaml

# 5. Install kagent (sealed-secrets controller will create the actual secrets)
helm install kagent . \
  --set deploymentType=statefulset \
  --set replicaCount=3 \
  --set persistence.keypair.type=secret
SOPS (Encrypted Files in Git)

Use SOPS with age, GPG, or cloud KMS:

# 1. Generate keypairs
./generate-secrets.sh 3

# 2. Encrypt the generated secret manifest
sops --encrypt --age <your-age-public-key> \
  generated_secrets/generated_secrets.yaml > \
  generated_secrets/generated_secrets.enc.yaml

# 3. Commit encrypted file to Git
git add generated_secrets/generated_secrets.enc.yaml
git commit -m "Add encrypted kagent secrets"

# 4. Decrypt and apply (in CI/CD or locally)
sops --decrypt generated_secrets/generated_secrets.enc.yaml | kubectl apply -f -

# 5. Install kagent
helm install kagent . --set replicaCount=3

Important Notes:

  • When using persistence.keypair.type: secret, secrets must exist before installing the chart
  • Each secret must contain both private_key.pem and public_key.pem keys
  • Secret names must follow the pattern: {{ .Release.Name }}-{{ replica-index }}-secret
  • For custom release names, ensure secret names match (e.g., my-agent-0-secret, my-agent-1-secret)
  • Never commit unencrypted keypairs to Git - always use encryption (Sealed Secrets, SOPS) or external secret managers

Using PVC for Keypair Storage

If you prefer PersistentVolumeClaims for keypairs instead of secrets:

# StatefulSet with PVC-based keypair storage
helm install kagent . \
  --set deploymentType=statefulset \
  --set persistence.keypair.type=pvc

Using hostPath for Keypair Storage (DaemonSet)

For DaemonSet deployments with node-local keypair storage:

# DaemonSet pattern with hostPath keypairs
helm install kagent . \
  --set deploymentType=daemonset \
  --set persistence.keypair.enabled=true \
  --set persistence.keypair.type=hostPath \
  --set persistence.keypair.hostPath.path=/var/lib/kagent/keys

Network Policies

Enable network policies to restrict egress traffic:

networkPolicy:
  enabled: true
  egress:
    # DNS
    - to:
      - namespaceSelector: {}
      ports:
      - protocol: UDP
        port: 53
    # Kentik API
    - to:
      - ipBlock:
          cidr: 0.0.0.0/0
      ports:
      - protocol: TCP
        port: 443
    # SNMP (adjust CIDR for your network)
    - to:
      - ipBlock:
          cidr: 10.0.0.0/8
      ports:
      - protocol: UDP
        port: 161

Custom ConfigMap

For advanced configuration overrides:

configmap:
  enabled: true
  data:
    K_CUSTOM_SETTING: "value"
    K_ANOTHER_SETTING: "value2"

Note: Environment variables take precedence over ConfigMap values.

Upgrade Strategy

Within Same Pattern

Upgrades within the same deployment pattern are supported:

# Upgrade StatefulSet pattern
helm upgrade kagent . --set replicaCount=3

# Upgrade with new image version
helm upgrade kagent . --set image.tag=v4.2.0

Changing Configuration

# Update log level
helm upgrade kagent . --set kagent.logLevel=debug

# Update release channel
helm upgrade kagent . --set kagent.releaseChannel=beta

Values Schema

For complete field definitions and validation, see contracts/values.schema.json.

Development

Linting

helm lint .

Template Rendering

# Render StatefulSet pattern
helm template kagent . --set deploymentType=statefulset

# Render DaemonSet pattern
helm template kagent . --set deploymentType=daemonset

Testing

# Dry run install
helm install kagent . --dry-run --debug

Support

License

Copyright © 2025 Kentik

About

Kentik Universal Agent Helm chart

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •