Real-Time Monitoring & Alerting Dashboard using a Time-Series Database (Feestdagen website)
Launch date: October 2014.
Discover the endless possibilities of monitoring your Servers, Applications, Web Analytics and Internet Of Things in real-time.
This project explains how these features are being used to monitor in real-time the high traffic website Feestdagen Belgie, and alert in case of anomalies.
The products InfluxDB, Grafana and Collectd have been selected, after a careful evaluation of the available products in the market, as the pluggable platform for this solution.
- Upgraded InfluxDB to V1.2, upgraded the Go Client SDK for InfluxDB to V2 (breaking changes; new feature: Batch Points), upgraded the Grafana Dashboard to V4.3.
- Made some preparations to interweave a Kafka Message Bus pipeline between the data collector components and the data persistence components. the plan is to not only stream the data to the InFluxDB time-series database but also to a Cassandra key-value store for further analysis.
- The dashboard now monitors a cluster of cloud servers and cloud services.
- The InfluxDB time-series database engine has been upgraded from V0.8 to V1.1. This meant a complete rewrite of the data pipelines because the data API’s have been changed (alpha -> production release).
- The Grafana visualization solution has been upgraded from V1.9 to V2.0 and then to V4.0. The V4.0 release is faster and more stable (it now contains a Golang backend), more feature rich (more panel types such as gauge meters and tables). All dashboards have been redefined.
- The Collectd data collector has been upgraded from to V4 to V5.7. The newer version captures more interesting metrics and can be customized very easily. The collector declarations have been refined.
- The Dashboard is not publicly available. Please request a demo or watch the videos (#1 #2).
- The website @ http://www.feestdagen-belgie.be/
- InfluxDB is the new home for all of your metrics, events, and analytics.
- Grafana, An open source, feature rich metrics dashboard and graph editor.
- Wikipedia: Time series database.
Many other products have been evaluated for this project but they did not make it to the finish line. Please find below a subset of those:
- Heka (Mozilla).
- Kibana (ElasticSearch).
- Statd (Etsy).
- Apache Cassandra.
- Brussels Airport Flights Monitor using a Real-Time Dashboard.
- Custom Dashboards using the Google Analytics Embed API.
- Varnish Cache & Cloud Server migration (Feestdagen website).