Skip to main content

Prometheus

Time-series metrics database. Collects numeric measurements and stores them for querying and alerting.

What's configured

This stack runs Prometheus alongside a Pushgateway, which is how Decree automations push metrics rather than being scraped directly.

ServicePortRole
prometheus49090Stores and queries metrics
pushgateway9091 (internal)Receives pushed metrics from short-lived jobs

Prometheus scrapes Pushgateway every 15s and retains data for 90 days.

How metrics get in

The afterEach hook in automations/hooks/afterEach.sh fires after every Decree run and pushes three gauges to Pushgateway:

MetricLabels
decree_run_successtrigger_type, instance=<routine>
decree_run_duration_secondssame
decree_run_attemptssame

Prometheus pulls these into its time series on the next scrape interval.

Adding more scrapers

Edit hosting/prometheus/prometheus.yml and add a new entry under scrape_configs. Any container on the exist network is reachable by container name.

Viewing metrics

Raw metric explorer at http://localhost:49090. For dashboards, use Grafana.