Skip to content

Pravega Connectors

Connectors allow integrating Pravega with different data sources and sinks.

Supported connectors

Currently, Pravega offers the following connectors:

  • Flink Connector: The Flink Connector enables building end-to-end stream processing pipelines with Pravega in Apache Flink. This also allows reading and writing data to external data sources and sinks via Flink Connector.

  • Spark Connector: Connector to read and write Pravega Streams with Apache Spark, a high-performance analytics engine for batch and streaming data. The connector can be used to build end-to-end stream processing pipelines that use Pravega as the stream storage and message bus, and Apache Spark for computation over the streams.

  • Hadoop Connector: Implements both the input and the output format interfaces for Apache Hadoop. It leverages Pravega batch client to read existing events in parallel; and uses write API to write events to Pravega streams.

  • Presto Connector: Presto is a distributed SQL query engine for big data. Presto uses connectors to query storage from different storage sources. This connector allows Presto to query storage from Pravega streams.

  • Boomi Connector: A Pravega connector for the Boomi Atomsphere.

  • Nifi Connector: Connector to read and write Pravega streams with Apache NiFi.

Third-party contributions

In addition to the connectors provided by the Pravega organization, open-source contributors have also created connectors for external projects:

  • Alpakka connector: The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams, and has been designed from the ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for back-pressure. This Alpakka Pravega connector lets you connect Pravega to Akka Streams.

  • Pravega Spring Cloud Stream Binder: Spring Cloud Stream is a framework for building highly scalable event-driven microservices connected with shared messaging systems. The Pravega Binder connects Pravega to Spring Cloud Data Flow, which provides tools to create complex topologies for streaming and batch data pipelines. The data pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks.

  • Memstate Connector: Memstate is an in-memory event-sourced ACID-transactional replicated object graph engine. The Pravega connector allows Memstate to use it as storage provider for persistence and global message ordering.