Prometheus natively supports pull model, but there’s a long discussion for Prometheus between pull and push, e.g. Pull doesn’t scale - or does it?. In this post it gives detailed pros and cons. Considering that we prefer not to expose an extra HTTP endpoint for each worker node, we would like to take some walk around to implement a push based monitoring with node_exporter, pushgateway and prometheus.

Main Structure

We have multiple service nodes, each of them hosts multiple services, we use node_exporter to collect system metrics.

img

What is different here is we add a forwarder between node_exporter and pushgateway, which helps us to switch the direction of data pipeline. Basically, node_exporter still listens on a specific port to expose metrics, but it only accepts internal requests. It helps us to keep the node safe from external malicious visits. Forwarder gets the metrics and send it to pushgateway, prometheus scraps data from pushgateway, instead of node_exporter.

Configuration

I. node_exporter

It does not make any sense if we launch node_exporter in a separate container, because it needs to be able to access the host status directly by scraping system info files, not the container status. So in order to make life easy, we create a exporter.service to manage its lifetime.

By default the port 9014 will be listened by node_exporter, we should setup the inbound rules of network security group and make sure that it can not be accessed from outside to keep the node safe.

II. Forwarder

Instead of pulling data from node_exporter, we create an internal cron task to pull the metrics constantly from node_exporter and send it to pushgateway passively. The script is simple enough with no more than two curl calls. Of course it would be better if we check the exporter status before local pulling, this helps to eliminate empty push actions.

1
2
# Send metrics to pushgateway
curl -s http://$EXPORTER_ADDR/$EXPORTER_METRIC | curl --data-binary @- http://$PGW_ADDR/metrics/job/$PGW_JOB/instance/$PGW_INSTANCE

The cron task can be added as below. However, it is not recommended to do that because it has the risk of adding duplicated tasks. A better way of doing this is to create PID file and use it as a lock. More details can be found here, Preventing duplicate cron job executions

1
echo "*/1 * * * * /root/exporter/cron.sh > /dev/null" | crontab

III. Pushgateway

Launching pushgateway is easy, simply follow the official document here README.md. I’d suggest to run pushgateway in a separate container by drafting a local Dockerfile.

1
2
3
4
5
6
7
8
9
# This is an official Dockerfile
FROM quay.io/prometheus/busybox:latest
MAINTAINER The Prometheus Authors <prometheus-developers@googlegroups.com>
COPY pushgateway /bin/pushgateway
EXPOSE 9091
WORKDIR /pushgateway
ENTRYPOINT [ "/bin/pushgateway" ]

You can add more parameters by appending CMD commands after ENTRYPOINT. Details of its parameters can be found by using -h option.

1
./pushgateway -h

IV. Prometheus

I’d recommend to launch prometheus by using container. You can follow the documentation here to setup a prometheus container within 5 min, Install Prometheus using Docker.

Summary

This post gives a brief instructions on how to setup a push based monitoring service with node_exporter, pushgateway and prometheus. The key of replacing pull with push is simply adding a cron task that pulls local metrics and send it to pushgateway. It helps to hide the port 9014 from outside and keep the system safe.