[Datastream to BigQuery] NullPointerException in DataStreamIO when eventType is null

[gauravjain6633]

Related Template(s)

DataStream To BigQuery template

Template Version

datastream to bigquery template

What happened?

Summary

The "Datastream to BigQuery" Dataflow template crashes with a NullPointerException when processing Pub/Sub notification messages that lack required attributes (specifically eventType or objectId).

Stack Trace

java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "eventType" is null
    com.google.cloud.teleport.v2.datastream.sources.DataStreamIO$ExtractGcsFile.process(DataStreamIO.java:337)

Problem Description

In DataStreamIO$ExtractGcsFile.java (lines 333-337):

String eventType = message.getAttribute("eventType");
String bucketId = message.getAttribute("bucketId");
String objectId = message.getAttribute("objectId");

if (eventType.equals("OBJECT_FINALIZE") && !objectId.endsWith("/")) {

• If eventType is null, invoking eventType.equals(...) throws a NullPointerException.
• If objectId is null, invoking objectId.endsWith(...) will also throw a NullPointerException.

Impact

Because this exception occurs in the initial source stage (ExtractGcsFile) before the pipeline's DLQ handling is reached, Dataflow continuously nacks the message. Pub/Sub redelivers the message repeatedly, causing worker errors and stalling processing for up to the 7-day Pub/Sub message retention limit.

Proposed Fix

1. Make attribute comparisons null-safe:

   if ("OBJECT_FINALIZE".equals(eventType) && objectId != null && !objectId.endsWith("/"))

2. Handle malformed or unexpected Pub/Sub messages gracefully by logging a warning or routing to a DLQ rather than throwing an uncaught runtime exception.
3. Open a Pull Request against the upstream repository: https://github.com/GoogleCloudPlatform/DataflowTemplates

Relevant log output

[gauravjain6633]

Working on the fix