HDDS-16398. Fix intermittent AlreadyClosedException in TestCommitWatcher

[yandrey321]

What changes were proposed in this pull request?

TestCommitWatcher fails intermittently in CI, in both test methods:

AlreadyClosedException: SlidingWindow$Client:client-B2BDCC4ECA19->RAFT is closed.
Caused by: RaftRetryFailureException: Failed Watch(0), seq=1*, ... for 3 attempts with ...[6x5s]
Caused by: NotLeaderException: Server aab42b7f... is not the leader ... suggested leader is 155c2039...

Root cause

1. SCM's leader hint is reported-once, not current. PipelineReportHandler.setPipelineLeaderId only
   ever sets leaderId, from a datanode report riding the heartbeat — 10s here. It is never absent
   (RATIS OPEN requires Pipeline.isHealthy() ⇒ leaderId != null) but can name a datanode that has
   stopped leading, and XceiverClientRatis.connect() feeds it to setLeaderId(...) unvalidated.
2. Ratis puts a dummy watch at the head of the ordered window. The first async() call instantiates
   OrderedAsync, which sends Watch(0) "to establish the connection" as seq=1 of the shared
   ->RAFT window; the test's own first request becomes seq=2, parked, never reaching the wire.
3. The watch budget cannot survive the wrong hint. Watch(0) gets NotLeaderException, and WATCH
   retries are capped by watchRequestTimeout, lowered 30s → 10s here — the stack's "3 attempts".
4. Exhaustion fails everything behind the dummy. SlidingWindow.Client.fail(1, e) fails the parked
   request with AlreadyClosedException and latches the window closed.

A harness problem, not a product defect: production keeps watchRequestTimeout at 30s, and on this
failure the Ozone client discards the pipeline and retries elsewhere.

Fix

A new allocateContainerWithElectedLeader() helper re-reads the container from SCM until the pipeline
record names a leader and that datanode reports itself as the current Raft leader; the client is then
acquired on that refreshed pipeline.

GenericTestUtils.waitFor(() -> {
  ContainerWithPipeline current = storageContainerLocationClient.getContainerWithPipeline(containerId);
  Pipeline pipeline = current.getPipeline();
  return pipeline.getLeaderId() != null
      && RatisTestHelper.isRatisLeader(cluster.getHddsDatanode(pipeline.getLeaderNode()), pipeline);
}, 500, 60000);

This addresses step 1, the only step the test controls: given a current hint the dummy watch reaches the
real leader and steps 2-4 never fire. isRatisLeader is the load-bearing half, distinguishing current
from reported at some point; getLeaderId() != null only keeps getLeaderNode() off its closest-node
fallback; re-reading is needed because allocateContainer's ContainerWithPipeline is a snapshot whose
leaderId never updates. (waitForPipelineTobeReady is no substitute — it waits for some pipeline to
reach OPEN.) Exposure narrows from the ~10s heartbeat window to the gap between check and first request;
closing it entirely would mean weakening the timeouts the test exists to exercise.

Also here, unrelated to the flake: each loop iteration now uses a distinct BlockID instead of
writing two different chunks to one blockID and offset (CHUNK_FILE_INCONSISTENCY, tolerated under
HDDS-11239). Hygiene only.

Generated-by: Claude Code (Claude Opus 5)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16398

How was this patch tested?

CI:

[chihsuan]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

[yandrey321]

Thanks for looking into this flaky test! @yandrey321 Could you share the CI log of the failure you investigated? The Jira stack trace is truncated.

Would it also be worth running the intermittent-test-check workflow on your branch to confirm it's stable?

Small nit: the code snippets in the description could use ``` fences so they render properly. Thanks!

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

I saw multiple CI runs with the same signature.

[chihsuan]

here is the link to the failed CI: https://github.com/yandrey321/ozone/actions/runs/34395742534/job/102619847831

Thanks for the link! @yandrey321 I ran the intermittent-test-check on your branch and it still failed 3 times: https://github.com/chihsuan/ozone/actions/runs/34612123189

The failures have different signatures, so I suspect the root cause is different and there may be multiple issues. Could you take a look?

[yandrey321]

The failures have different signatures, so I suspect the root cause is different and there may be multiple issues. Could you take a look?
it should be fixed now.

[chihsuan]

Thanks! @yandrey321 I checked the datanode logs from my earlier intermittent run, and I'm not sure the current fix addresses the actual failure. Please see inline comments. Would it be worth running intermittent-test-check on the new head as well?

[chihsuan]

I wonder if leader election is really the cause here. In the DN logs from my run, the second WriteChunk failed with CHUNK_FILE_INCONSISTENCY after the first PutBlock closed the block file. The container was then marked UNHEALTHY, followed by pipeline closure. Would using a different blockID per iteration avoid this?

[chihsuan]

Do we still need this wait? I noticed allocateContainer only picks pipelines already in OPEN state, and OPEN already implies a reported leader, so this may not add any guarantee.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerManagerImpl.java

Lines 182 to 186 in f9f6184

	pipelines = pipelineManager
	.getPipelines(replicationConfig, Pipeline.PipelineState.OPEN);
	if (!pipelines.isEmpty()) {
	pipeline = pipelines.get(random.nextInt(pipelines.size()));
	containerInfo = createContainer(pipeline, owner);

[chihsuan]

nit: Could we drop createPipelineOnDatanode here? The groups should already exist for an OPEN pipeline, so these calls normally just hit duplicate-group errors that the helper swallows. Same for line 246.

[chihsuan]

nit: Could we trim this comment and keep the explanation in one place? The same explanation appears here, at line 241, and in init(), and the exception chain is already in the PR description.

[chihsuan]

Hi @yandrey321, the new head looks much better, with intermittent-test-check going from 3 failures to 1 of 100. https://github.com/chihsuan/ozone/actions/runs/36010552048/job/107671428630

The remaining one failed at the new CreateContainer call, with a LeaderNotReadyException right after leader election.

14:27:06,273 155c2039@group-C61D84748144: change Leader from null to 155c2039
14:27:06,404 LeaderNotReadyException: 155c2039@group-C61D84748144 is in LEADER state but not ready yet.
14:27:20,214 NotLeaderException: Server aab42b7f@group-C61D84748144 is not the leader, suggested leader is: 155c2039
             RaftRetryFailureException: ... for 3 attempts
TestCommitWatcher.testReleaseBuffersOnException:241 » AlreadyClosedException: ... RAFT is closed.

Could we wait for the leader to be ready before the first write?