Yesterday, the Oath network automation team open sourced Panoptes, a distributed system for collecting, enriching and distributing network telemetry.
This pluggable, distributed and high-performance data collection system supports multiple polling formats, including SNMP and vendor-specific APIs. It also supports emerging streaming telemetry standards including gNMI. Panoptes is written primarily in Python. It leverages multiple open-source technologies to provide the most value for the least development effort.
Panoptes Architecture
Source: Yahoo Developers
The architecture is designed to enable easy data distribution and integration with other systems. The plugin to push metrics into InfluxDB allows Panoptes to evolve with industry standards. Teams can quickly set up a fully-featured monitoring environment because of the combination of Grafana and the InfluxData ecosystem.
There were multiple issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems. Panoptes aims to overcome all these issues.
Check scheduling is accomplished using Celery, which is a horizontally scalable, open-source scheduler that utilizes a Redis data store.
Panoptes ships with a simple, CSV-based discovery system. It can be integrated with a CMDB. From there, Panoptes will manage the task of scheduling polling for the desired devices. Users can also develop custom discovery plugins to integrate with their CMDB and other device inventory data sources.
Vendors are moving towards a more streamlined model of telemetry. Panoptes’ flexible architecture will minimize the effort required to adopt these new protocols.
The metric bus at the center of the model is implemented on Kafka. All data plane transactions flow across this bus. Discovery plugins publish devices to the bus and polling plugins publish metrics to the bus. Similarly, numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems.
The team at Oath has deployed Panoptes in a tiered, federated model. They have developed numerous custom applications on the platform, including a load balancer monitor, a BGP session monitor, and a topology discovery application. All this was done at a reduced cost, thanks to Panoptes. This open-source release is packaged for easy deployment into any Linux-based environment and available on Github.
You can head over to Yahoo Developer Network for deeper insights into this news.
Read Next
Performing Vehicle Telemetry job analysis with Azure Stream Analytics tools
Anaconda 5.3.0 released, takes advantage of Python’s Speed and feature improvements
Arm releases free Cortex-M processor cores for FPGAs, includes measures to combat FOSSi threat