Skip to content

Bound MPP sink notifications by the query deadline - #19344

Open
JackieTien97 wants to merge 6 commits into
masterfrom
ty/mpp-notification-deadline
Open

JackieTien97 wants to merge 6 commits into
masterfrom
ty/mpp-notification-deadline

Conversation

@JackieTien97

@JackieTien97 JackieTien97 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

When a downstream DataNode stops responding, a sink can keep retrying onNewDataBlockEvent or onEndOfDataBlockEvent after the query times out. With a 60-second node RPC timeout and three attempts, a notification for a 4-second query can occupy a shared exchange worker for about three minutes. Other queries then queue behind it to deliver their results.

Changes

  • Send the remaining query duration immediately before each fragment dispatch RPC, including retries. The receiver anchors it to its own clock at RPC receipt and uses the earlier of the available exchange and fragment deadlines. Older requests fall back to the local fragment deadline. Expired reads now return QUERY_TIMEOUT before dispatch. With an unspecified coordinator timeout (zero or negative), a transport failure can take the usual reconnect retry and the fragment's own limit still applies.
  • Recompute the remaining time when a notification leaves the queue, borrows a connection, connects, waits for a response, or retries. An expired queued notification sends zero RPCs. A normal final ACK stops new attempts and lets outstanding responses return their sockets to the pool until the original deadline. Cancellation and expiry interrupt outstanding notifications. The one-shot deadline alarm uses the JDK timer independently of fragment cleanup and coordinator retries. Timeout reports retain the triggering RPC error, and terminal success and failure callbacks are mutually exclusive.
  • Set the socket timeout on each deadline-aware checkout and restore the pool configuration on return. Failed connection setup closes the socket and preserves the original exception. A locally interrupted socket is discarded individually, while peer disconnects clear the endpoint's cached connections. Source handles and driver-side device-entry fetches share these interruptible sockets. An interrupted driver exits its fetch or cleanup retry loop.
  • New connections resolve hostnames on the borrower thread. IP literals take a direct path with or without TLS. The MPP client caches the SSL socket factory and reloads it when the configured protocol, passwords, or store identity, modification time or size changes. Failed loads are retried. JVM hostname resolution and local store I/O can exceed the query budget. The client checks the remaining time again before socket I/O.
  • The MPP service follows the DataNode process lifecycle. During shutdown, the executor termination hook closes the client pool after queued and running exchange tasks finish. The public synchronous MPP client constructor, client factory and pool factory remain available to downstream callers. Direct server-side wiring of MPPDataExchangeManager uses the deadline-aware MPPDataExchangeClientManager.

TLS compatibility

The repository's pinned libthrift 0.25.0 sets HTTPS endpoint identification in the original internal client. Loopback tests use the same trusted certificate with a DNS SAN for localhost. Both transports accept that hostname and reject the IP literal. Thrift 0.25.0 client implementation

The general server thread pool guidance is tracked separately in #19354.

Example

A 4-second query that has already used 3.2 seconds has 800 ms left for result exchange. In a loopback test, waiting for the pool took 310 ms. The borrowed socket received a 475 ms read timeout. With a silent peer, the call returned a timeout 820 ms after it started waiting for the pool, including exception handling.

Verification

  • Self-reviewed the notification, cancellation, and connection reuse paths.
  • Ran 76 targeted unit tests covering exchange, client pooling, TLS, driver interruption, and fragment dispatch. Three regressions reproduced lost retry diagnostics and premature socket interrupts after a normal finish on the previous revision, then passed in three consecutive runs on this revision.
  • Ran the table-model IoTDBMPPDataExchangeLargeTsBlock128KIT and tree-model IoTDBShuffleSink1IT on clusters with one ConfigNode and three DataNodes.
  • Verified mutual TLS with both standard and Snappy framing. Additional tests cover concurrent TLS factory reuse, certificate-store rotation, recovery after an invalid store, and interruption of a TLS read while another pooled client stays usable.
  • Packaged all 50 Maven reactor modules in both the default locale and with-zh-locale.

Key changed classes

SinkChannel, MPPDataExchangeClientManager, SyncDataNodeMPPDataExchangeServiceClient, and FragmentInstanceDispatcherImpl.

@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.79580% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.48%. Comparing base (dd450f7) to head (fbfa79d).

Files with missing lines Patch % Lines
...sync/SyncDataNodeMPPDataExchangeServiceClient.java 93.45% 7 Missing ⚠️
...eryengine/execution/exchange/sink/SinkChannel.java 97.76% 3 Missing ⚠️
...mons/client/sync/MPPDataExchangeClientManager.java 90.00% 2 Missing ⚠️
...ol/thrift/impl/DataNodeInternalRPCServiceImpl.java 66.66% 1 Missing ⚠️
...plan/scheduler/FragmentInstanceDispatcherImpl.java 95.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19344      +/-   ##
============================================
+ Coverage     49.44%   49.48%   +0.04%     
  Complexity      718      718              
============================================
  Files          5656     5657       +1     
  Lines        418222   418452     +230     
  Branches      55296    55336      +40     
============================================
+ Hits         206769   207067     +298     
+ Misses       211453   211385      -68     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the inline findings before merging:

  1. Cross-node clock skew (FragmentInstanceDispatcherImpl / datanode.thrift / FragmentInstanceContext): queryDeadlineMs sends a coordinator System.currentTimeMillis() timestamp and compares it directly against the receiver DataNode's wall clock, and getExchangeDeadlineMs() drops exchangeDeadlineMs when outerQueryDeadlineMs is -1L.
  2. SonarCloud blocker & exception masking (SyncDataNodeMPPDataExchangeServiceClient): SocketChannel.open() triggers BLOCKER java:S2095 (failing the SonarCloud gate), and socket.close() in connect() can mask QueryTimeoutException / Error.
  3. Interrupt hygiene & callback races (SinkChannel): Self-interrupts from deadlineAlarm / stopNotifications() remain set on shared MPP-Data-Exchange-Task-Executors workers across sinkListener callbacks and ThreadPoolExecutor#getTask, and concurrent SendEndOfDataBlockEventTask / SendNewDataBlockEventTask completion can invoke onFinish / onEndOfBlocks after onFailure.
  4. False Broken pipe logs & pool invalidation (SyncDataNodeMPPDataExchangeServiceClient / ThriftClient): ClosedByInterruptException matches ClosedChannelException in ThriftClient.isConnectionBroken, logging misleading BROKEN_PIPE_CLEAR_CONNECTIONS INFO messages on every cancelled or timed-out query.
  5. Timeout/cancellation exception consistency & test timing (MPPDataExchangeClientManager, ClientManager, tests): Align QueryTimeoutException vs ClientManagerException in borrowClient, handle CancellationException in prepareConnection, clamp negative wait durations, and tighten wall-clock vs monotonic-clock waits in the new unit tests.

Comment on lines +512 to +515
if (queryContext.getTimeOut() > 0) {
instance.setQueryDeadlineMs(
saturatedAdd(queryContext.getStartTime(), queryContext.getTimeOut()));
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

queryContext.getStartTime() is an epoch timestamp from the coordinator's System.currentTimeMillis(). Sending queryDeadlineMs in TSendFragmentInstanceReq and comparing it directly against the receiver DataNode's System.currentTimeMillis() (SinkChannel#deadlineReached and MPPDataExchangeClientManager#remainingMillis) makes exchange deadlines sensitive to cross-node wall-clock skew:

  • If a receiver's clock is ahead of the coordinator's by Δ >= timeout (e.g., 5s skew on a 4s query), DriverTask still runs (because FragmentInstance#timeOut is relative), and SinkChannel then fails the first notification immediately at 0ms with QueryTimeoutException.
  • If a receiver's clock is behind by Δ, Math.min(outerQueryDeadlineMs, exchangeDeadlineMs) falls back to the planning-time outerQueryDeadlineMs and ignores time spent between planning and dispatch/retry on the coordinator.

Note how prepareAdmission in this same class (lines 865-867) avoids comparing coordinator and receiver clocks by sending a relative duration. Please carry the remaining relative budget in milliseconds computed right before each sendFragmentInstance attempt (deadline - System.currentTimeMillis(), failing fast on the coordinator if already <= 0) and convert it into a local deadline on the receiver.

Comment on lines +670 to +675
public long getExchangeDeadlineMs() {
// Legacy coordinators only send the remaining FI timeout. Also honor an earlier local FI limit.
return exchangeDeadlineMs < 0
? outerQueryDeadlineMs
: Math.min(outerQueryDeadlineMs, exchangeDeadlineMs);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several FragmentInstanceContext constructors leave outerQueryDeadlineMs at its default -1L. When outerQueryDeadlineMs < 0 and exchangeDeadlineMs >= 0, Math.min(outerQueryDeadlineMs, exchangeDeadlineMs) returns -1L, which SinkChannel then converts to Long.MAX_VALUE, silently dropping exchangeDeadlineMs. Please guard outerQueryDeadlineMs < 0 as well and cover that case in FragmentInstanceContextTest.

Comment on lines +77 to +88
final void prepare(long deadlineMs) throws TTransportException, IOException {
if (iprot_ == null) {
// A channel-backed socket makes blocking connects, writes and reads interruptible,
// including reads during the TLS handshake.
TTransport transport = connect(SocketChannel.open().socket(), deadlineMs);
try {
iprot_ = oprot_ = property.getProtocolFactory().getProtocol(transport);
} catch (RuntimeException | Error e) {
transport.close();
throw e;
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 81 (SocketChannel.open().socket()) triggers a BLOCKER SonarCloud issue (java:S2095: Use try-with-resources or close this "SocketChannel" in a "finally" clause), which is failing the SonarCloud Quality Gate with an E Reliability Rating on New Code. Please structure the SocketChannel lifecycle so cleanup is explicit (or suppress java:S2095 with a clear ownership comment once connect() cleanup is hardened).

Comment on lines +123 to +126
} catch (IOException | TTransportException | RuntimeException | Error e) {
socket.close();
throw e;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If e is QueryTimeoutException (for example thrown by rpcTimeout(deadlineMs) on line 121 after socket has been wrapped in SSLSocket) or an Error, and socket.close() throws an IOException, that IOException replaces and masks e. Please wrap socket.close() in try { socket.close(); } catch (IOException closeEx) { e.addSuppressed(closeEx); } (and ensure the underlying SocketChannel is closed if SSLSocket#close fails). Also guard property.getConnectionTimeoutMs() < 0 in rpcTimeout so a negative config cannot pass < 0 to Socket#connect / setSoTimeout.

Comment on lines 179 to 184
public void invalidateAll() {
clientManager.clear(endpoint);
// An interrupt closes just this channel. Other clients to the same peer remain usable.
if (!Thread.currentThread().isInterrupted()) {
clientManager.clear(endpoint);
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interrupting I/O on a SocketChannel-backed socket throws ClosedByInterruptException (a subclass of ClosedChannelException). Before calling invalidateAll(), ThriftClient.resolveException checks ThriftClient.isConnectionBroken(rootCause)—which returns true for any ClosedChannelException—and logs BROKEN_PIPE_CLEAR_CONNECTIONS ("Broken pipe error happened in sending RPC, we need to clear all previous cached connection...") at INFO level on every cancelled or timed-out query.

Conversely, if a real Broken pipe / Connection reset happens at the same time deadlineAlarm interrupts the worker, gating on Thread.currentThread().isInterrupted() skips clearing stale pooled connections. Excluding ClosedByInterruptException directly in ThriftClient.isConnectionBroken addresses both the misleading INFO log and pool invalidation.

Comment on lines +76 to +88
public SyncDataNodeMPPDataExchangeServiceClient borrowClient(TEndPoint endpoint, long deadlineMs)
throws ClientManagerException {
SyncDataNodeMPPDataExchangeServiceClient client =
borrowClientWithTimeout(endpoint, remainingMillis(deadlineMs));
boolean ready = false;
try {
client.prepare(deadlineMs);
ready = true;
return client;
} catch (Exception e) {
throw new ClientManagerException(e);
} finally {
if (!ready) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

borrowClient(endpoint, deadlineMs) throws unchecked QueryTimeoutException when deadlineMs expires on line 79 (remainingMillis), whereas if it expires 1ms later inside client.prepare(deadlineMs) (rpcTimeout), catch (Exception e) wraps QueryTimeoutException in ClientManagerException. That makes the exception type timing-dependent (and can make MPPDataExchangeClientManagerTest#exhaustedPoolHonorsBothQueryAndPoolWaitLimits fail if the test thread pauses >= 150ms before entering borrowClient). Please handle QueryTimeoutException consistently across both stages.

Comment on lines +98 to +111
<T> T prepareConnection(Callable<T> prepare, int timeoutMs) throws TTransportException {
FutureTask<T> task = new FutureTask<>(prepare);
connectionSetup.execute(task);
try {
return timeoutMs == 0 ? task.get() : task.get(timeoutMs, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new TTransportException(e);
} catch (TimeoutException e) {
throw new TTransportException(TTransportException.TIMED_OUT, e);
} catch (ExecutionException e) {
throw new TTransportException(e.getCause());
} finally {
if (task.cancel(true)) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

close() (lines 119-121) cancels queued FutureTasks via cancel(false), so a concurrent borrower blocked in task.get(...) will receive an unchecked CancellationException bypassing throws TTransportException. Please catch CancellationException here as well and wrap it in TTransportException.

Comment on lines +63 to +64
long poolWaitMs = pool.getMaxWaitDuration().toMillis();
return pool.borrowObject(node, poolWaitMs < 0 ? maxWaitMs : Math.min(poolWaitMs, maxWaitMs));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commons Pool2 treats a negative borrowMaxWaitMillis in pool.borrowObject(node, borrowMaxWaitMillis) as an unbounded takeFirst() wait. Since borrowClientWithTimeout is protected, clamping Math.max(0L, maxWaitMs) avoids turning a negative wait argument from future callers into an infinite block.

Comment on lines +118 to +121
assertFalse(dequeue.await(Math.max(0, deadline - System.currentTimeMillis()), MILLISECONDS));
dequeue.countDown();
awaitWorker();
verify(clients, never()).borrowClient(any(), anyLong());

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CountDownLatch#await measures monotonic System.nanoTime(), while SinkChannel#deadlineReached checks System.currentTimeMillis() >= queryDeadlineMs. Due to sub-millisecond truncation or slight wall-clock slewing, dequeue.await(...) can return false when System.currentTimeMillis() is still deadline - 1, letting the worker run before deadline and invoke borrowClient. Looping until System.currentTimeMillis() >= deadline before dequeue.countDown() avoids flaky failures.

Comment on lines +92 to +96
try (SyncDataNodeMPPDataExchangeServiceClient shortQuery =
manager.borrowClient(endpoint, System.currentTimeMillis() + 800)) {
assertSame(first, shortQuery);
assertTrue(shortQuery.getTimeout() > 0 && shortQuery.getTimeout() <= 800);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reusedConnectionUsesEachBorrowersTimeout only checks getTimeout() without sleeping for expiry, so using e.g. 5_000 ms instead of 800 ms avoids spurious QueryTimeoutExceptions on paused CI runners with zero runtime cost (and similarly check System.currentTimeMillis() >= deadline - 1 vs nanoTime jitter on line 127). Please also consider adding a loopback TLS test case here to cover sslSocketFactory() and SSLSocket wrapping in SyncDataNodeMPPDataExchangeServiceClient.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough work. The overall design looks right:

  • The coordinator sends a relative remaining budget, and the receiver anchors it to its own clock, which avoids cross-node clock skew.
  • An expired queued notification sends zero RPCs.
  • claimFinish and claimFailure make the finish and failure callbacks mutually exclusive.
  • The channel clears only the interrupts it issued, so none leak to shared workers.
  • A locally interrupted socket is discarded individually instead of clearing the whole endpoint.

Inline comments are ordered roughly by priority. The two [Major] items (TLS hostname verification, and caching the SSLSocketFactory so the connection setup pool can be removed) should be addressed before merging.

Items without a specific line:

  • API change for downstream code: this removes ClientPoolFactory.SyncDataNodeMPPDataExchangeServiceClientPoolFactory, makes SyncDataNodeMPPDataExchangeServiceClient.Factory package-private, and makes the constructor protected. Downstream code such as TimechoDB that references them needs a matching change.
  • Interrupt cleanup edge case: if an external interrupt (for example, from shutdownNow) arrives after the channel's own interrupt, Thread.interrupted() in sendNotification clears both. The probability is negligible, so this is acceptable.
  • SonarCloud: the failing reliability condition (rating C) comes from files outside this PR. The remaining findings in this PR are code smells: java:S3776 (cognitive complexity 16 in sendWithRetry) and java:S1181 (catching Error in SyncDataNodeMPPDataExchangeServiceClient).

peer.sslFactory().createSocket(socket, endpoint.getIp(), endpoint.getPort(), true);
socket = ssl;
SSLParameters params = ssl.getSSLParameters();
params.setEndpointIdentificationAlgorithm("HTTPS");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] This newly enables TLS hostname verification, for MPP data exchange only.

The previous path (TSSLTransportFactory.getClientSocket) never set an endpoint identification algorithm, and this is the only place in the repository that sets one. Certificates must now carry a SAN for every DataNode internal address. The test stores pass because they contain IP:127.0.0.1 (SAN=ip:127.0.0.1 here, and the IT test-keystore). A common production setup, one cluster-wide certificate with only a CN, would now fail the handshake for MPP data exchange while every other internal RPC keeps working. That partial failure is hard to diagnose.

This changes security semantics and is outside the scope of this PR. Please remove this line to keep parity with the other internal clients. If hostname verification is wanted, it should be a separate change that covers all internal clients behind a configuration switch, and it should be verified with TLCP/Kona.


// Native name resolution and TLS store loading can ignore interrupts. Keep these cold-path
// operations off exchange workers. Each waiting borrower removes its task on cancellation.
private final WrappedThreadPoolExecutor connectionSetup;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Major] Cache the SSLSocketFactory and remove connectionSetup.

With internal SSL enabled, every new connection now queues on this 4-thread pool, reloads the keystore and truststore from disk, and re-initializes the key and trust managers. After a peer restart, invalidateAll clears the endpoint and all borrowers rebuild connections at once. They serialize behind 4 threads and can only wait until rpcTimeout.

Suggestion:

  1. Build the SSLSocketFactory once, lazily or eagerly in the constructor, and cache it. Do not cache a failed load. Loading local stores on the caller thread is a one-off that takes milliseconds.
  2. Resolve host names on the caller thread. getaddrinfo cannot be interrupted or timed out, so bounding it would require another thread. However, a DNS stall is a different failure mode from the unresponsive peer this PR targets, and every other internal client already resolves on the caller thread. Only host-name endpoints with a JVM DNS cache miss are affected; by default, successful lookups are cached for 30 s and failed lookups for 10 s. Please document this as a known limitation, for example: "Host name resolution is not bounded by the query deadline, consistent with other internal clients."
  3. Then connectionSetup, prepareConnection, ThreadName.MPP_DATA_EXCHANGE_CONNECTION_SETUP, and the shutdown logic in close() can be removed. This also avoids the INFO TASK_CANCELLED_IN_POOL log that WrappedThreadPoolExecutor.afterExecute writes for every cancelled setup task; disableErrorLog() does not suppress it.

Certificate rotation: the old code re-read the store files for every new connection, so replacing the files on disk took effect for new connections without a restart. To keep that behavior, key the cache by (path, lastModified, size). Otherwise, please mention the change in the PR description.

String host = endpoint.getIp();
int port = endpoint.getPort();
boolean ssl = commonConfig.isEnableInternalSSL();
if (!ssl && InetAddresses.isInetAddress(host)) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the factory is cached, this fast path should not depend on ssl. An IP literal can skip resolution regardless of TLS, but today every TLS connection goes through the setup pool, even to an IP literal. For example:

InetSocketAddress address =
    InetAddresses.isInetAddress(host)
        ? new InetSocketAddress(InetAddresses.forString(host), port)
        // Caller-thread DNS; see the comment on MPPDataExchangeClientManager.
        : new InetSocketAddress(host, port);
SSLSocketFactory factory = ssl ? cachedSslSocketFactory() : null;

private void openProtocol(long deadlineMs) throws IOException, TTransportException {
SocketChannel channel = SocketChannel.open();
try {
// A channel-backed socket makes blocking connects, writes and reads interruptible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pool is also used by SourceHandle and by DeviceEntryRpcSegmentFetcher, which runs on driver threads. With channel-backed sockets, any interrupt now closes the connection in the middle of an RPC. For example, Driver.interruptCurrentOwner() makes an in-flight DeviceEntryRpcSegmentFetcher call fail with ClosedByInterruptException, whereas a classic socket ignored the interrupt. The result looks acceptable because the driver is being torn down, but this behavior change reaches beyond the sink path. Please mention it in the PR description and consider a test for the driver-interrupt case.

checkNotificationActive();
return;
} catch (Exception e) {
checkNotificationActive();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the deadline has passed when this catch block runs, checkNotificationActive() throws QueryTimeoutException, and e is neither logged nor attached. Similarly, line 722 reports new QueryTimeoutException() in place of a real TApplicationException or network failure that happened just before the deadline. Please keep the original cause for diagnosis, for example with timeout.addSuppressed(e), or at least log it at debug level.

// Attempt metadata is absent for a legacy coordinator. Do not change the binary FI body.
3: optional i32 queryAttemptId
4: optional TQueryAdmissionToken admissionToken
// Field 5 carried an absolute deadline in earlier development builds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Field 5 (queryDeadlineMs) only existed in the first commit of this PR and was never on master. After a squash merge, this comment refers to history that no longer exists. If field 5 was never released, please use 5 and remove the comment. If a TimechoDB build already shipped it, please name that version in the comment.

if (queryContext.getTimeOut() > 0) {
instance.setQueryDeadlineFromTimeout(queryContext.getStartTime(), queryContext.getTimeOut());
}
remainingQueryTimeMs(instance);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces two behavior changes that the PR description should mention:

  1. A query that has already timed out now fails with QUERY_TIMEOUT before dispatch. Previously the FI was dispatched and timed out in the driver.
  2. When queryContext.getTimeOut() <= 0 (MPPQueryContext defaults to 0), the old retry check currentTime - startTime >= timeOut was always true, so the reconnect retry never ran. Now the deadline is -1, so the retry runs. This looks like a fix, but it changes behavior and should be documented, ideally with a test.

public void stop() {
super.stop();
executorService.shutdown();
mppDataExchangeManager.getMppDataExchangeServiceClientManager().close();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: close() runs right after executorService.shutdown() without waiting for running exchange tasks, which then fail on the closed pool. Also, this singleton creates its client manager only in the constructor, so the service cannot start again after stop(). This is fine if stop() is only called on process exit; please confirm.

Comment thread AGENTS.md Outdated
- **Checkstyle** is also configured (see `checkstyle.xml` at project root).
- Java source/target level is 17 (compiled with `maven.compiler.release=17` on JDK 17+).

### Java server thread pools

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These thread pool conventions are unrelated to the MPP deadline change. Please move them to a separate PR. If the setup pool is removed as suggested on MPPDataExchangeClientManager, this PR no longer depends on them.

"-dname",
"CN=localhost",
"-ext",
"SAN=ip:127.0.0.1",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a case with a certificate whose SAN does not contain the peer IP, to pin down the hostname-verification behavior (see the comment on SyncDataNodeMPPDataExchangeServiceClient). A case asserting that an interrupted TLS read does not trigger invalidateAll would also be useful.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant