Federator.ai®Integration With Data Monitoring – Datadog

Federator.ai^®& Datadog Integration

Datadog provides monitoring for servers, applications, and services. With Datadog, enterprise customers are able to monitor their application workloads and get visibility into Kubernetes clusters of any scale. Federator.ai integration with Datadog aggregates metrics and events and provides resource prediction/recommendation for Kubernetes deployments and application-aware acceleration and optimization for Kafka. The following diagram illustrates the Federator.ai/Datadog integration workflow.

1.

Datadog Agent posts workload metrices to Datadog Services

2.

Data-Adapter queries workload metrics from Datadog Services

3.

Data-Adapter posts the Prediction/Recommendation created by Federator.ai to Datadog Services

4.

Datadog Cluster Agent gets Prediction/Recommendation from Datadog Services

5.

WPA applies Recommendation to applications

6.

Datadog Dashboard displays workload metrics and Prediction/Recommendation by Federator.ai

Using Kafka as an example, the Datadog Agent monitors and collects many different types of Kafka metrics, including resources (e.g., CPU, memory) utilization and Kafka workload and performance metrics (e.g., message queue offset/length, consumer lags). With integration of ProphetStor Federator.ai, Kafka message production/consumption rate is continuously collected from Datadog by Federator.ai’s data-adapter and analyzed by the AI engine of Federator.ai. Federator.ai uses advanced machine learning algorithms to predict the Kafka message production rate. Based on the prediction of message production rate, Federator.ai works with WPA to automatically scale Kafka consumer replicas to handle the increased or decreased workload.