Repository navigation
Bound MPP sink notifications by the query deadline - #19344
JackieTien97 wants to merge 6 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19344 +/- ##
============================================
+ Coverage 49.44% 49.48% +0.04%
Complexity 718 718
============================================
Files 5656 5657 +1
Lines 418222 418452 +230
Branches 55296 55336 +40
============================================
+ Hits 206769 207067 +298
+ Misses 211453 211385 -68 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JackieTien97
left a comment
There was a problem hiding this comment.
Please address the inline findings before merging:
- Cross-node clock skew (
FragmentInstanceDispatcherImpl/datanode.thrift/FragmentInstanceContext):queryDeadlineMssends a coordinatorSystem.currentTimeMillis()timestamp and compares it directly against the receiver DataNode's wall clock, andgetExchangeDeadlineMs()dropsexchangeDeadlineMswhenouterQueryDeadlineMsis-1L. - SonarCloud blocker & exception masking (
SyncDataNodeMPPDataExchangeServiceClient):SocketChannel.open()triggers BLOCKERjava:S2095(failing the SonarCloud gate), andsocket.close()inconnect()can maskQueryTimeoutException/Error. - Interrupt hygiene & callback races (
SinkChannel): Self-interrupts fromdeadlineAlarm/stopNotifications()remain set on sharedMPP-Data-Exchange-Task-Executorsworkers acrosssinkListenercallbacks andThreadPoolExecutor#getTask, and concurrentSendEndOfDataBlockEventTask/SendNewDataBlockEventTaskcompletion can invokeonFinish/onEndOfBlocksafteronFailure. - False
Broken pipelogs & pool invalidation (SyncDataNodeMPPDataExchangeServiceClient/ThriftClient):ClosedByInterruptExceptionmatchesClosedChannelExceptioninThriftClient.isConnectionBroken, logging misleadingBROKEN_PIPE_CLEAR_CONNECTIONSINFOmessages on every cancelled or timed-out query. - Timeout/cancellation exception consistency & test timing (
MPPDataExchangeClientManager,ClientManager, tests): AlignQueryTimeoutExceptionvsClientManagerExceptioninborrowClient, handleCancellationExceptioninprepareConnection, clamp negative wait durations, and tighten wall-clock vs monotonic-clock waits in the new unit tests.
| if (queryContext.getTimeOut() > 0) { | ||
| instance.setQueryDeadlineMs( | ||
| saturatedAdd(queryContext.getStartTime(), queryContext.getTimeOut())); | ||
| } |
There was a problem hiding this comment.
queryContext.getStartTime() is an epoch timestamp from the coordinator's System.currentTimeMillis(). Sending queryDeadlineMs in TSendFragmentInstanceReq and comparing it directly against the receiver DataNode's System.currentTimeMillis() (SinkChannel#deadlineReached and MPPDataExchangeClientManager#remainingMillis) makes exchange deadlines sensitive to cross-node wall-clock skew:
- If a receiver's clock is ahead of the coordinator's by
Δ >= timeout(e.g., 5s skew on a 4s query),DriverTaskstill runs (becauseFragmentInstance#timeOutis relative), andSinkChannelthen fails the first notification immediately at 0ms withQueryTimeoutException. - If a receiver's clock is behind by
Δ,Math.min(outerQueryDeadlineMs, exchangeDeadlineMs)falls back to the planning-timeouterQueryDeadlineMsand ignores time spent between planning and dispatch/retry on the coordinator.
Note how prepareAdmission in this same class (lines 865-867) avoids comparing coordinator and receiver clocks by sending a relative duration. Please carry the remaining relative budget in milliseconds computed right before each sendFragmentInstance attempt (deadline - System.currentTimeMillis(), failing fast on the coordinator if already <= 0) and convert it into a local deadline on the receiver.
| public long getExchangeDeadlineMs() { | ||
| // Legacy coordinators only send the remaining FI timeout. Also honor an earlier local FI limit. | ||
| return exchangeDeadlineMs < 0 | ||
| ? outerQueryDeadlineMs | ||
| : Math.min(outerQueryDeadlineMs, exchangeDeadlineMs); | ||
| } |
There was a problem hiding this comment.
Several FragmentInstanceContext constructors leave outerQueryDeadlineMs at its default -1L. When outerQueryDeadlineMs < 0 and exchangeDeadlineMs >= 0, Math.min(outerQueryDeadlineMs, exchangeDeadlineMs) returns -1L, which SinkChannel then converts to Long.MAX_VALUE, silently dropping exchangeDeadlineMs. Please guard outerQueryDeadlineMs < 0 as well and cover that case in FragmentInstanceContextTest.
| final void prepare(long deadlineMs) throws TTransportException, IOException { | ||
| if (iprot_ == null) { | ||
| // A channel-backed socket makes blocking connects, writes and reads interruptible, | ||
| // including reads during the TLS handshake. | ||
| TTransport transport = connect(SocketChannel.open().socket(), deadlineMs); | ||
| try { | ||
| iprot_ = oprot_ = property.getProtocolFactory().getProtocol(transport); | ||
| } catch (RuntimeException | Error e) { | ||
| transport.close(); | ||
| throw e; | ||
| } | ||
| } |
There was a problem hiding this comment.
Line 81 (SocketChannel.open().socket()) triggers a BLOCKER SonarCloud issue (java:S2095: Use try-with-resources or close this "SocketChannel" in a "finally" clause), which is failing the SonarCloud Quality Gate with an E Reliability Rating on New Code. Please structure the SocketChannel lifecycle so cleanup is explicit (or suppress java:S2095 with a clear ownership comment once connect() cleanup is hardened).
| } catch (IOException | TTransportException | RuntimeException | Error e) { | ||
| socket.close(); | ||
| throw e; | ||
| } |
There was a problem hiding this comment.
If e is QueryTimeoutException (for example thrown by rpcTimeout(deadlineMs) on line 121 after socket has been wrapped in SSLSocket) or an Error, and socket.close() throws an IOException, that IOException replaces and masks e. Please wrap socket.close() in try { socket.close(); } catch (IOException closeEx) { e.addSuppressed(closeEx); } (and ensure the underlying SocketChannel is closed if SSLSocket#close fails). Also guard property.getConnectionTimeoutMs() < 0 in rpcTimeout so a negative config cannot pass < 0 to Socket#connect / setSoTimeout.
| public void invalidateAll() { | ||
| clientManager.clear(endpoint); | ||
| // An interrupt closes just this channel. Other clients to the same peer remain usable. | ||
| if (!Thread.currentThread().isInterrupted()) { | ||
| clientManager.clear(endpoint); | ||
| } | ||
| } |
There was a problem hiding this comment.
Interrupting I/O on a SocketChannel-backed socket throws ClosedByInterruptException (a subclass of ClosedChannelException). Before calling invalidateAll(), ThriftClient.resolveException checks ThriftClient.isConnectionBroken(rootCause)—which returns true for any ClosedChannelException—and logs BROKEN_PIPE_CLEAR_CONNECTIONS ("Broken pipe error happened in sending RPC, we need to clear all previous cached connection...") at INFO level on every cancelled or timed-out query.
Conversely, if a real Broken pipe / Connection reset happens at the same time deadlineAlarm interrupts the worker, gating on Thread.currentThread().isInterrupted() skips clearing stale pooled connections. Excluding ClosedByInterruptException directly in ThriftClient.isConnectionBroken addresses both the misleading INFO log and pool invalidation.
| public SyncDataNodeMPPDataExchangeServiceClient borrowClient(TEndPoint endpoint, long deadlineMs) | ||
| throws ClientManagerException { | ||
| SyncDataNodeMPPDataExchangeServiceClient client = | ||
| borrowClientWithTimeout(endpoint, remainingMillis(deadlineMs)); | ||
| boolean ready = false; | ||
| try { | ||
| client.prepare(deadlineMs); | ||
| ready = true; | ||
| return client; | ||
| } catch (Exception e) { | ||
| throw new ClientManagerException(e); | ||
| } finally { | ||
| if (!ready) { |
There was a problem hiding this comment.
borrowClient(endpoint, deadlineMs) throws unchecked QueryTimeoutException when deadlineMs expires on line 79 (remainingMillis), whereas if it expires 1ms later inside client.prepare(deadlineMs) (rpcTimeout), catch (Exception e) wraps QueryTimeoutException in ClientManagerException. That makes the exception type timing-dependent (and can make MPPDataExchangeClientManagerTest#exhaustedPoolHonorsBothQueryAndPoolWaitLimits fail if the test thread pauses >= 150ms before entering borrowClient). Please handle QueryTimeoutException consistently across both stages.
| <T> T prepareConnection(Callable<T> prepare, int timeoutMs) throws TTransportException { | ||
| FutureTask<T> task = new FutureTask<>(prepare); | ||
| connectionSetup.execute(task); | ||
| try { | ||
| return timeoutMs == 0 ? task.get() : task.get(timeoutMs, TimeUnit.MILLISECONDS); | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| throw new TTransportException(e); | ||
| } catch (TimeoutException e) { | ||
| throw new TTransportException(TTransportException.TIMED_OUT, e); | ||
| } catch (ExecutionException e) { | ||
| throw new TTransportException(e.getCause()); | ||
| } finally { | ||
| if (task.cancel(true)) { |
There was a problem hiding this comment.
close() (lines 119-121) cancels queued FutureTasks via cancel(false), so a concurrent borrower blocked in task.get(...) will receive an unchecked CancellationException bypassing throws TTransportException. Please catch CancellationException here as well and wrap it in TTransportException.
| long poolWaitMs = pool.getMaxWaitDuration().toMillis(); | ||
| return pool.borrowObject(node, poolWaitMs < 0 ? maxWaitMs : Math.min(poolWaitMs, maxWaitMs)); |
There was a problem hiding this comment.
Commons Pool2 treats a negative borrowMaxWaitMillis in pool.borrowObject(node, borrowMaxWaitMillis) as an unbounded takeFirst() wait. Since borrowClientWithTimeout is protected, clamping Math.max(0L, maxWaitMs) avoids turning a negative wait argument from future callers into an infinite block.
| assertFalse(dequeue.await(Math.max(0, deadline - System.currentTimeMillis()), MILLISECONDS)); | ||
| dequeue.countDown(); | ||
| awaitWorker(); | ||
| verify(clients, never()).borrowClient(any(), anyLong()); |
There was a problem hiding this comment.
CountDownLatch#await measures monotonic System.nanoTime(), while SinkChannel#deadlineReached checks System.currentTimeMillis() >= queryDeadlineMs. Due to sub-millisecond truncation or slight wall-clock slewing, dequeue.await(...) can return false when System.currentTimeMillis() is still deadline - 1, letting the worker run before deadline and invoke borrowClient. Looping until System.currentTimeMillis() >= deadline before dequeue.countDown() avoids flaky failures.
| try (SyncDataNodeMPPDataExchangeServiceClient shortQuery = | ||
| manager.borrowClient(endpoint, System.currentTimeMillis() + 800)) { | ||
| assertSame(first, shortQuery); | ||
| assertTrue(shortQuery.getTimeout() > 0 && shortQuery.getTimeout() <= 800); | ||
| } |
There was a problem hiding this comment.
reusedConnectionUsesEachBorrowersTimeout only checks getTimeout() without sleeping for expiry, so using e.g. 5_000 ms instead of 800 ms avoids spurious QueryTimeoutExceptions on paused CI runners with zero runtime cost (and similarly check System.currentTimeMillis() >= deadline - 1 vs nanoTime jitter on line 127). Please also consider adding a loopback TLS test case here to cover sslSocketFactory() and SSLSocket wrapping in SyncDataNodeMPPDataExchangeServiceClient.
JackieTien97
left a comment
There was a problem hiding this comment.
Thanks for the thorough work. The overall design looks right:
- The coordinator sends a relative remaining budget, and the receiver anchors it to its own clock, which avoids cross-node clock skew.
- An expired queued notification sends zero RPCs.
claimFinishandclaimFailuremake the finish and failure callbacks mutually exclusive.- The channel clears only the interrupts it issued, so none leak to shared workers.
- A locally interrupted socket is discarded individually instead of clearing the whole endpoint.
Inline comments are ordered roughly by priority. The two [Major] items (TLS hostname verification, and caching the SSLSocketFactory so the connection setup pool can be removed) should be addressed before merging.
Items without a specific line:
- API change for downstream code: this removes
ClientPoolFactory.SyncDataNodeMPPDataExchangeServiceClientPoolFactory, makesSyncDataNodeMPPDataExchangeServiceClient.Factorypackage-private, and makes the constructorprotected. Downstream code such as TimechoDB that references them needs a matching change. - Interrupt cleanup edge case: if an external interrupt (for example, from
shutdownNow) arrives after the channel's own interrupt,Thread.interrupted()insendNotificationclears both. The probability is negligible, so this is acceptable. - SonarCloud: the failing reliability condition (rating C) comes from files outside this PR. The remaining findings in this PR are code smells:
java:S3776(cognitive complexity 16 insendWithRetry) andjava:S1181(catchingErrorinSyncDataNodeMPPDataExchangeServiceClient).
| peer.sslFactory().createSocket(socket, endpoint.getIp(), endpoint.getPort(), true); | ||
| socket = ssl; | ||
| SSLParameters params = ssl.getSSLParameters(); | ||
| params.setEndpointIdentificationAlgorithm("HTTPS"); |
There was a problem hiding this comment.
[Major] This newly enables TLS hostname verification, for MPP data exchange only.
The previous path (TSSLTransportFactory.getClientSocket) never set an endpoint identification algorithm, and this is the only place in the repository that sets one. Certificates must now carry a SAN for every DataNode internal address. The test stores pass because they contain IP:127.0.0.1 (SAN=ip:127.0.0.1 here, and the IT test-keystore). A common production setup, one cluster-wide certificate with only a CN, would now fail the handshake for MPP data exchange while every other internal RPC keeps working. That partial failure is hard to diagnose.
This changes security semantics and is outside the scope of this PR. Please remove this line to keep parity with the other internal clients. If hostname verification is wanted, it should be a separate change that covers all internal clients behind a configuration switch, and it should be verified with TLCP/Kona.
|
|
||
| // Native name resolution and TLS store loading can ignore interrupts. Keep these cold-path | ||
| // operations off exchange workers. Each waiting borrower removes its task on cancellation. | ||
| private final WrappedThreadPoolExecutor connectionSetup; |
There was a problem hiding this comment.
[Major] Cache the SSLSocketFactory and remove connectionSetup.
With internal SSL enabled, every new connection now queues on this 4-thread pool, reloads the keystore and truststore from disk, and re-initializes the key and trust managers. After a peer restart, invalidateAll clears the endpoint and all borrowers rebuild connections at once. They serialize behind 4 threads and can only wait until rpcTimeout.
Suggestion:
- Build the
SSLSocketFactoryonce, lazily or eagerly in the constructor, and cache it. Do not cache a failed load. Loading local stores on the caller thread is a one-off that takes milliseconds. - Resolve host names on the caller thread.
getaddrinfocannot be interrupted or timed out, so bounding it would require another thread. However, a DNS stall is a different failure mode from the unresponsive peer this PR targets, and every other internal client already resolves on the caller thread. Only host-name endpoints with a JVM DNS cache miss are affected; by default, successful lookups are cached for 30 s and failed lookups for 10 s. Please document this as a known limitation, for example: "Host name resolution is not bounded by the query deadline, consistent with other internal clients." - Then
connectionSetup,prepareConnection,ThreadName.MPP_DATA_EXCHANGE_CONNECTION_SETUP, and the shutdown logic inclose()can be removed. This also avoids the INFOTASK_CANCELLED_IN_POOLlog thatWrappedThreadPoolExecutor.afterExecutewrites for every cancelled setup task;disableErrorLog()does not suppress it.
Certificate rotation: the old code re-read the store files for every new connection, so replacing the files on disk took effect for new connections without a restart. To keep that behavior, key the cache by (path, lastModified, size). Otherwise, please mention the change in the PR description.
| String host = endpoint.getIp(); | ||
| int port = endpoint.getPort(); | ||
| boolean ssl = commonConfig.isEnableInternalSSL(); | ||
| if (!ssl && InetAddresses.isInetAddress(host)) { |
There was a problem hiding this comment.
Once the factory is cached, this fast path should not depend on ssl. An IP literal can skip resolution regardless of TLS, but today every TLS connection goes through the setup pool, even to an IP literal. For example:
InetSocketAddress address =
InetAddresses.isInetAddress(host)
? new InetSocketAddress(InetAddresses.forString(host), port)
// Caller-thread DNS; see the comment on MPPDataExchangeClientManager.
: new InetSocketAddress(host, port);
SSLSocketFactory factory = ssl ? cachedSslSocketFactory() : null;| private void openProtocol(long deadlineMs) throws IOException, TTransportException { | ||
| SocketChannel channel = SocketChannel.open(); | ||
| try { | ||
| // A channel-backed socket makes blocking connects, writes and reads interruptible. |
There was a problem hiding this comment.
This pool is also used by SourceHandle and by DeviceEntryRpcSegmentFetcher, which runs on driver threads. With channel-backed sockets, any interrupt now closes the connection in the middle of an RPC. For example, Driver.interruptCurrentOwner() makes an in-flight DeviceEntryRpcSegmentFetcher call fail with ClosedByInterruptException, whereas a classic socket ignored the interrupt. The result looks acceptable because the driver is being torn down, but this behavior change reaches beyond the sink path. Please mention it in the PR description and consider a test for the driver-interrupt case.
| checkNotificationActive(); | ||
| return; | ||
| } catch (Exception e) { | ||
| checkNotificationActive(); |
There was a problem hiding this comment.
If the deadline has passed when this catch block runs, checkNotificationActive() throws QueryTimeoutException, and e is neither logged nor attached. Similarly, line 722 reports new QueryTimeoutException() in place of a real TApplicationException or network failure that happened just before the deadline. Please keep the original cause for diagnosis, for example with timeout.addSuppressed(e), or at least log it at debug level.
| // Attempt metadata is absent for a legacy coordinator. Do not change the binary FI body. | ||
| 3: optional i32 queryAttemptId | ||
| 4: optional TQueryAdmissionToken admissionToken | ||
| // Field 5 carried an absolute deadline in earlier development builds. |
There was a problem hiding this comment.
Field 5 (queryDeadlineMs) only existed in the first commit of this PR and was never on master. After a squash merge, this comment refers to history that no longer exists. If field 5 was never released, please use 5 and remove the comment. If a TimechoDB build already shipped it, please name that version in the comment.
| if (queryContext.getTimeOut() > 0) { | ||
| instance.setQueryDeadlineFromTimeout(queryContext.getStartTime(), queryContext.getTimeOut()); | ||
| } | ||
| remainingQueryTimeMs(instance); |
There was a problem hiding this comment.
This introduces two behavior changes that the PR description should mention:
- A query that has already timed out now fails with
QUERY_TIMEOUTbefore dispatch. Previously the FI was dispatched and timed out in the driver. - When
queryContext.getTimeOut() <= 0(MPPQueryContextdefaults to0), the old retry checkcurrentTime - startTime >= timeOutwas always true, so the reconnect retry never ran. Now the deadline is-1, so the retry runs. This looks like a fix, but it changes behavior and should be documented, ideally with a test.
| public void stop() { | ||
| super.stop(); | ||
| executorService.shutdown(); | ||
| mppDataExchangeManager.getMppDataExchangeServiceClientManager().close(); |
There was a problem hiding this comment.
Minor: close() runs right after executorService.shutdown() without waiting for running exchange tasks, which then fail on the closed pool. Also, this singleton creates its client manager only in the constructor, so the service cannot start again after stop(). This is fine if stop() is only called on process exit; please confirm.
| - **Checkstyle** is also configured (see `checkstyle.xml` at project root). | ||
| - Java source/target level is 17 (compiled with `maven.compiler.release=17` on JDK 17+). | ||
|
|
||
| ### Java server thread pools |
There was a problem hiding this comment.
These thread pool conventions are unrelated to the MPP deadline change. Please move them to a separate PR. If the setup pool is removed as suggested on MPPDataExchangeClientManager, this PR no longer depends on them.
| "-dname", | ||
| "CN=localhost", | ||
| "-ext", | ||
| "SAN=ip:127.0.0.1", |
There was a problem hiding this comment.
Please add a case with a certificate whose SAN does not contain the peer IP, to pin down the hostname-verification behavior (see the comment on SyncDataNodeMPPDataExchangeServiceClient). A case asserting that an interrupted TLS read does not trigger invalidateAll would also be useful.
This reverts commit d4fa00c.
|




Description
When a downstream DataNode stops responding, a sink can keep retrying
onNewDataBlockEventoronEndOfDataBlockEventafter the query times out. With a 60-second node RPC timeout and three attempts, a notification for a 4-second query can occupy a shared exchange worker for about three minutes. Other queries then queue behind it to deliver their results.Changes
QUERY_TIMEOUTbefore dispatch. With an unspecified coordinator timeout (zero or negative), a transport failure can take the usual reconnect retry and the fragment's own limit still applies.MPPDataExchangeManageruses the deadline-awareMPPDataExchangeClientManager.TLS compatibility
The repository's pinned libthrift 0.25.0 sets HTTPS endpoint identification in the original internal client. Loopback tests use the same trusted certificate with a DNS SAN for
localhost. Both transports accept that hostname and reject the IP literal. Thrift 0.25.0 client implementationThe general server thread pool guidance is tracked separately in #19354.
Example
A 4-second query that has already used 3.2 seconds has 800 ms left for result exchange. In a loopback test, waiting for the pool took 310 ms. The borrowed socket received a 475 ms read timeout. With a silent peer, the call returned a timeout 820 ms after it started waiting for the pool, including exception handling.
Verification
IoTDBMPPDataExchangeLargeTsBlock128KITand tree-modelIoTDBShuffleSink1ITon clusters with one ConfigNode and three DataNodes.with-zh-locale.Key changed classes
SinkChannel,MPPDataExchangeClientManager,SyncDataNodeMPPDataExchangeServiceClient, andFragmentInstanceDispatcherImpl.