ProphetStor Logo

Federator.ai® & Kafka Integration

ProphetStor Logo

Apache Kafka, an open-source distributed event streaming platform, is primarily used to build a real-time streaming platform that handles trillions of constant influx of data a day and processes the data pipelines and applications that adapt to the fluctuation of data streams. By using Machine Learning technologies for predictions and analysis, Federator.ai achieves much better performance (reduced latency) with much fewer resources (reduced number of Kafka consumers) and makes implementing autoscaling of Kafka consumers simple and straightforward.

DiagramThe Federator.ai/Kafka integration workflow

1.

Workload indicators based on message production rate and consumer lag from Kafka.

2.

Modeling and Machine-Learning-based predictions for workloads.

3.

Apply Federator.ai recommendations for the right numbers of consumer replicas.

Benefits from Federator.ai®

Effective workload predictions

Federator.ai uses message production rate of a Kafka topic and target KPI metrics such as the desired latency as the key metrics for autoscaling Kafka consumers. Predictions of message production rate give a more accurate indication of real workloads for Kafka consumers.

Cost-effective application deployments

Federator.ai integrates the workload metrics, workload predictions, and application KPI in deciding the right number of replicas and achieves more cost-effective application deployments.

Achieving desired performance

Without guessing or experimenting on what metric threshold to set in Kubernetes native HPA, Federator.ai achieves better use of resources for desired performance automatically.

DashboardFederator.ai dashboard for Kafka Consumers

VideoA presentation in Kafka Summit on why intelligent autoscaling is better than Kubernetes native HPA

VideoHow to configure an application on Federator.ai for autoscaling Kafka consumer

Start Federator.ai for free today and benefit from the values of machine-learning for your Kafka application.

Read More

Please select the software you would like a demo of:

Federator.ai GPU Booster ®

Maximizing GPU utilization for AI workloads and doubling your server’s training capacity

Federator.ai ®

Simplifying complexity and continuously optimizing cloud costs and performance