CDAP Dataproc Response Code Metrics For Operators

CDAP had a small activity window, but the useful change is concrete for data engineering operators who run pipelines on Google Dataproc. The main backend commit adds response code detail to Dataproc operation metrics, while a separate UI submodule update keeps pipeline screens moving with the rest of develop.

The important backend change is Collecting response code for Dataproc operations. It touches seven Dataproc runtime files and adds most of the 477 inserted lines in the window. This is not a broad runtime rewrite. It is an observability change for cluster create and delete paths.

The data model change starts in DataprocMetric.java. The metric object now carries optional method and statusCode fields. That gives callers a way to report an explicit Google API status code rather than relying only on an exception attached to the metric object.

DataprocUtils.java then applies the new shape. It still emits the existing reg and sc tags, and it still adds imgVer when image version data is present. The new part is the optional method tag, plus code selection that prefers a supplied status code, falls back to an ApiException cause, and then falls back to INTERNAL for other exception shapes. A clean metric still resolves to OK.

For operators, this matters because cluster status alone is too coarse. A failed provision can land as FAILED, but the status code tells you whether the backend saw invalid input, quota pressure, service unavailability, or another API class. That is the difference between opening an app issue and raising a platform capacity incident.

The provisioner side change is in DataprocProvisioner.java. getClusterStatus now keeps the previous cluster status before asking Dataproc for the current status. When the cluster moves from a non terminal state into RUNNING, FAILED, or NOT_EXISTS, the provisioner attempts to emit an operation response metric.

That timing is sensible. A long running operation is only useful for this metric after it has reached a final state. The code also maps CREATING to CREATE and DELETING to DELETE, which keeps the method tag bounded. It does not attempt to classify every status transition. It focuses on the two lifecycle operations where Dataproc LRO metadata can explain the outcome.

The emitted metric name is provisioner.operation.response.count. On a finished operation, the builder adds region, image version, method, and an error derived status code when the LRO contains an error. If the operation is absent or not done, the provisioner returns without emitting the metric. If the metadata call fails, the code logs a warning and keeps returning the actual cluster status. That last behavior is important. Observability enrichment should not make status polling fail.

DataprocClient.java gets a small helper named getLatestOperation. It builds the operations collection path from project and region, then filters by clusterName and operationType. The result is the first operation in the first page, wrapped as an optional value.

This keeps the API call out of DataprocProvisioner, which is the right boundary. The provisioner decides when an LRO result is worth sampling. The client owns the Dataproc operations API shape and wraps ApiException through the existing CDAP error handling path.

There is one assumption to keep in view. The helper name says latest operation, but the implementation takes the first item returned by listOperations. If Dataproc ordering is stable and newest first, the behavior matches the name. If that ordering changes or is not guaranteed, the metric can attach the wrong operation to a terminal transition. That is the main sharp edge in this change.

The test updates are not decoration. DataprocClientTest.java adds cases for one operation, no operations, API failure, and multiple operations. The multiple operation case asserts that the first item is returned, which documents the ordering assumption above.

DataprocProvisionerTest.java then checks the operator visible metric behavior. It covers create to running, delete to not exists, create to failed with an operation error, operation not done, and metadata lookup failure. It also adds direct coverage for the exception branches in DataprocUtils.emitMetric.

The support change in MockProvisionerContext.java is small but useful. The mock context now records metric counts in a concurrent map. That turns metric emission into something tests can assert, rather than a side effect that disappears inside an empty mock.

The other commit, Update cdap ui submodule to latest develop, changes only the cdap-ui pointer in the main repository. GitHub reports the submodule update as 80 files on the UI side, including pipeline error detail, pipeline triggers, log viewer, and source control management files.

For this review, the important point is separation. The Dataproc response code work sits in the runtime extension and test tree. The UI bump may affect how pipeline operators inspect runs, errors, and triggers, but the staged core diff does not show a matching backend contract change for those screens. Treat it as a surface update, not as part of the Dataproc metric feature.

  • Check dashboard cardinality after rollout. method is bounded to CREATE and DELETE, and status code is bounded by GAX values, but imgVer can still split series by image version.
  • Validate the operations API ordering assumption. If the first returned operation is not the newest one, getLatestOperation needs an explicit sort or a stronger filter.
  • Look for follow up work that surfaces provisioner.operation.response.count in default alerts. The metric is useful only if operators can separate invalid config, quota, and transient service failures without digging through Dataproc logs.