Skip to content

Getting Started

Note: You can skip this step if you have a streaming project set up already.

Please use the following project templates and setup guidelines, to set up a stream processing project with Apache Flink using Connectors

Once after the set up, please follow the below instructions to add the Flink Pravega connectors to the project.

Add the Connector Dependencies

To add the Pravega connector dependencies to your project, add the following entry to your project file: (For example, pom.xml for Maven)

<!-- Before Pravega 0.6 -->
<dependency>
  <groupId>io.pravega</groupId>
  <artifactId>pravega-connectors-flink_2.12</artifactId>
  <version>0.5.1</version>
</dependency>

<!-- Pravega 0.6 and After -->
<dependency>
  <groupId>io.pravega</groupId>
  <artifactId>pravega-connectors-flink-1.9_2.12</artifactId>
  <version>0.6.0</version>
</dependency>

Use appropriate version as necessary. 1.9 is the Flink Major-Minor version. 2.12 is the Scala version. 0.6.0 is the Pravega version. The snapshot versions are published to jcenter repository and the release artifacts are available in Maven Central repository.

Alternatively, we could build and publish the connector artifacts to local maven repository by executing the following command and make use of that version as your application dependency.

./gradlew clean install

Running / Deploying the Application

From Flink's perspective, the connector to Pravega is part of the streaming application (not part of Flink's core runtime), so the connector code must be part of the application's code artifact (JAR file). Typically, a Flink application is bundled as a fat-jar (also known as an uber-jar) , such that all its dependencies are embedded.

  • The project set up should have been a success, if you have used the above linked templates/guides.

  • If you set up a application's project and dependencies manually, you need to make sure that it builds a jar with dependencies, to include both the application and the connector classes.

  • The Flink connector has embedded and shaded the pravega-client dependency of the same version. Please DO NOT include both pravega-client and pravega-flink-connector dependency into the jar file, otherwise there will be dependency conflict issues.

  • If user application uses Table API and SQL, please DO NOT compile flink-table-planner or flink-table-planner-blink into the jar file as they are provided by the Flink cluster.