Installation and Architecture

Terminologies

Architecture

Metrics

Notes

Unique advantage of Prometheus is the pull system because most monitoring tools like AWS’ Cloud Monitoring is a push based system which can flood the network
To make sure an exporter is running, visit the machine’s IP with the exporter’s port and it should display the exporter’s page
Its not good to restart Prometheus every time whenever there’s a new config or a small change, it can impact negatively on production monitoring and affect the metrics (ex: reset the counter metrics and gauge metrics)
- There are two ways to reload Prometheus, either by sending a SIGHUP (hangup signal) to the process, or using their API (its disabled by default for security reasons)
  - kill -HUP <prometheus_process_id>
    - ps -ef | grep -i prometheus
  - curl -X POST http://<prometheus_url>:<port>/-/reload
    - Be sure to start Prometheus with the API enabled first nohup /path/to/prometheus --web.enable-lifecycle > prometheus.log 2>&1 &
    - Prometheus API doc

You can delete metric history for a specific job/instance using Prometheus’ API

curl -X POST -g '<http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="<target_name>:<port>"}>'

Grafana

Grafana is an open source interactive data-visualization platform which allows users to see their data via charts and graphs that are unified into one dashboard. Using Grafana for Prometheus is pretty straightforward you can just check this article or explore all on your own

You can even import an already created dashboard as a template by picking any of the free visualizations here

Dashboards