What is Piglet?
Piglet is a light-weight Prometheus (https://prometheus.io/) exporter (python) that scrapes /proc/<splunk_pid>/stat for the utime /cpu seconds data available by Splunk pid, instance and team (optional) labels and sends to a metrics cache running a Promtheus+PushGateway url (https://github.com/prometheus/pushgateway).
Piglet uses a simple text/plain exposition format (https://prometheus.io/docs/instrumenting/exposition_formats/) and requires only the default installed Splunk Python modules to run. This is intended to limit the need for any additional clients or python modules to be deployed to clients making integration easier.
Installation
- Piglet installs as Splunk app in etc/apps/piglet and comes with node-exporter (https://github.com/prometheus/node_exporter) as pre-packaged binary running in daemon mode *legal might not like this
- The default scrape interval is 15 seconds & can be changed in the inputs.conf
- Piglet uses the following python modules which are installed by default with the Splunk packaged python binary.
#!Modules
import traceback
import sys
import subprocess
import re
import requests
import configparser
import logging
import os
Why Piglet?
- Splunk cannot measure its own activity in CPU seconds.
- CPU seconds is an industry-accepted metric that allows for an objective workload measurements across platforms and providers.
- CPU seconds can provide a statistically measurable basis for calculating Splunk SVCs + Workload pricing strategies across all platforms (SplunkCloud, BYOL, on-premise)
- Allows for building a capacity managment program that can be aligned to both Search & Ingest & Workload managment.
Project Goals
Piglet is the client piece for a larger framework called PigPen which will handle the backend aggregation, collection & visualization of the client data. See Splunk internal POC https://docs.google.com/document/d/18e-CMUECM0XP-LfqQ7lNK-GkOQkfxd_b6-z-J5-60M8/edit?usp=sharing for details.
Requirements
The Piglet client requires:
-
node-exporter (https://github.com/prometheus/node_exporter)
-
Splunk 8.2+
-
a Prometheus + PushGateway endpoint/url
v1.0.3 - RC build
- removed wrapper.sh script now uses default Splunk python bin
- local/settings.conf will now take precedence over the default configuration per Splunk hierarchy.
v1.0.2
Modules added
+ requests for posting to Push-Gateway
+ configparser for reading a standard conf file
+ logging for logging
-- added to log/piglet_client.log to keep track of errors/debug activity
| default/settings.conf | [default] |
|---|---|
| command to pass to the OS and return an list of Splunk pid | splunk_pids_cmd = "ps -e| grep splunkd| awk '{print $1}'" |
| comand to pass to the OS and return the default hostname. | uname_cmd = "hostname" |
| default headers for reqeusts POSt | headers = {'X-Requested-With': 'Python requests', 'Content-type': 'text/plain'} |
| Push-Gateway URL/endpoint | url = "http://10.202.1.127:9091/metrics/job" |
| Optional free label | team_name = "avengers" |