Repository navigation
Fix heap accounting for sealed TsFile resources - #19347
JackieTien97 wants to merge 2 commits into
Conversation
Estimate resource metadata and device indexes separately. Track the charge for each registered file so metadata changes, degradation, and removal update the time-index budget consistently.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19347 +/- ##
============================================
+ Coverage 49.44% 49.45% +0.01%
Complexity 718 718
============================================
Files 5656 5656
Lines 418222 418433 +211
Branches 55296 55351 +55
============================================
+ Hits 206769 206954 +185
- Misses 211453 211479 +26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JackieTien97
left a comment
There was a problem hiding this comment.
Request Changes (submitted as COMMENT because GitHub does not allow REQUEST_CHANGES from the PR author account)
Overall, separating resident metadata from the degradable ITimeIndex and tracking per-registration charges as deltas in TsFileResourceManager is a solid design fix. However, there are a few correctness, concurrency, performance, and Sonar Quality Gate issues that should be addressed before merging:
- [High] File-identity lookup vs. degradation priority in
TsFileResourceManager:registerSealedTsFileResourceandremoveTsFileResource(whenmemoryRegistration == null) rely oncomparePriority((fileIndex, startTime, path)) to find an existingRegistrationfor the same file, which misses entries whenever the recoveryTsFileResourcehas a differentfileIndex(e.g., after degradation) orstartTime(e.g., an undeserializednew TsFileResource(file)in compaction recovery). - [Medium] Concurrent index swap vs. metadata-only
refresh(entry, false):refresh(entry, false)updatesentry.fileIndextotruewithout refreshingentry.indexBytes, which can leave the old device-index size charged while movingentryout of the degradable head and cause unnecessary degradation of another file. - [Medium] Unreleased
exclusiveModFileFutureafter mod resolution/replacement & field visibility:exclusiveModFileFutureis never cleared onceexclusiveModFileis resolved or overwritten (retaining an unaccountedModificationFileduring LOAD V1-mod upgrades), and several fields read cross-thread incalculateMetadataRamSize()are written withoutvolatile. - [Medium] Compaction selection
O(devices)rescan on resident indexes & unusedgetDeviceTimeIndexRamSize(): DroppinggetDeviceTimeIndexRamSize()inCompactionScheduleContextleavesTsFileResource#getDeviceTimeIndexRamSize()unused in production and rescans every residentArrayDeviceTimeIndexon each compaction selection pass instead of reusing the registeredindexBytes. - [CI] SonarCloud Quality Gate failure:
java:S3077on the newvolatilefields inTsFileResource(plusjava:S3776inLoadTsFileandjava:S1210inTsFileResourceManager).
See inline comments for details.
| Registration entry = new Registration(resource, ++nextRegistrationId); | ||
| Registration previous = sealedTsFileResources.floor(entry); | ||
| if (previous == null || entry.comparePriority(previous) != 0) { | ||
| sealedTsFileResources.add(entry); | ||
| if (entry.fileIndex) { | ||
| degradedTimeIndexNum++; | ||
| } | ||
| // Publish before sampling so a concurrent metadata change cannot miss its charge. | ||
| resource.setMemoryRegistration(entry); | ||
| try { | ||
| refresh(entry, true); | ||
| } catch (RuntimeException | Error e) { | ||
| removeTsFileResource(resource); | ||
| throw e; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /** delete the TsFileResource in PriorityQueue when the source file is deleted */ | ||
| public synchronized void removeTsFileResource(TsFileResource tsFileResource) { | ||
| if (sealedTsFileResources.contains(tsFileResource)) { | ||
| sealedTsFileResources.remove(tsFileResource); | ||
| if (TimeIndexLevel.valueOf(tsFileResource.getTimeIndexType()) | ||
| == TimeIndexLevel.FILE_TIME_INDEX) { | ||
| memoryBlock.release(tsFileResource.calculateRamSize()); | ||
| public synchronized void removeTsFileResource(TsFileResource resource) { | ||
| Registration entry = resource.getMemoryRegistration(); | ||
| if (entry == null) { | ||
| // Recovery can supply another resource describing the same file. | ||
| if (sealedTsFileResources.isEmpty()) { | ||
| return; | ||
| } | ||
| Registration key = new Registration(resource, Long.MIN_VALUE); | ||
| entry = sealedTsFileResources.ceiling(key); | ||
| if (entry == null || entry.comparePriority(key) != 0) { | ||
| return; | ||
| } | ||
| } |
There was a problem hiding this comment.
[High] comparePriority ((fileIndex, startTime, path)) cannot reliably match another TsFileResource describing the same physical file:
- Silent leak in
removeTsFileResourcewhenentry == null: Compaction recovery/rollback paths (InnerSpaceCompactionTask#recoverTargetResource,InsertionCrossSpaceCompactionTask#recoverTaskInfoFromLogFile,CrossSpaceCompactionTask) constructnew TsFileResource(targetFile)without callingdeserialize()and pass it throughreplaceTsFileInMemory -> tsFileManager.remove -> removeTsFileResource. That instance has an emptyArrayDeviceTimeIndex(startTime = Long.MAX_VALUE,fileIndex = false), whereas the registeredRegistrationhas the actualminStartTime(andfileIndex = trueif already degraded). Even if the caller deserialized.resource(fileIndex = false), if the registered entry had already degraded (entry.fileIndex = true),ceiling(key)searches the wrong partition ofsealedTsFileResourcesandentry.comparePriority(key) != 0—soremoveTsFileResourcereturns without removing the registration, releasingentry.bytes, or decrementingdegradedTimeIndexNum. - Duplicate registration in
registerSealedTsFileResource: Conversely, if a registered resource has degraded (previous.fileIndex = true) and anotherTsFileResourcefor the same file is passed toregisterSealedTsFileResourcewith anArrayDeviceTimeIndex(entry.fileIndex = false) or a differentstartTime,floor(entry)will not matchprevious, inserting a secondRegistrationfor the same file and double-charging its metadata.
Consider maintaining a secondary Map<String, Registration> keyed by file.getAbsolutePath() (kept in sync in refresh when entry.file != resource.getTsFile(), with its HashMap entry overhead included in REGISTRATION_RAM_BYTES) so both deduplication on register and fallback lookup on remove match by file identity in O(1) regardless of fileIndex or startTime.
| ITimeIndex index = resource.getTimeIndex(); | ||
| boolean fileIndex = index.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE; | ||
| if (entry.fileIndex != fileIndex | ||
| || entry.startTime != index.getMinStartTime() | ||
| || entry.file != resource.getTsFile()) { | ||
| sealedTsFileResources.remove(entry); | ||
| if (entry.fileIndex != fileIndex) { | ||
| degradedTimeIndexNum += fileIndex ? 1 : -1; | ||
| } | ||
| long memoryReduce = resource.degradeTimeIndex(); | ||
| degradedTimeIndexNum++; | ||
| releaseTimeIndexMemCost(memoryReduce); | ||
| sealedTsFileResources.add(resource); | ||
| entry.updatePriority(index); | ||
| sealedTsFileResources.add(entry); | ||
| } | ||
| if (indexChanged) { | ||
| entry.indexBytes = index.calculateRamSize(); | ||
| } |
There was a problem hiding this comment.
[Medium] refresh(entry, false) updates entry.fileIndex without updating entry.indexBytes when racing with replaceDeviceTimeIndexWithFileTimeIndex():
replaceDeviceTimeIndexWithFileTimeIndex() swaps timeIndex via TIME_INDEX_UPDATER.compareAndSet outside the TsFileResourceManager monitor and then calls updateMemoryCostIfRegistered(true). If Thread A swaps r1.timeIndex from a large ArrayDeviceTimeIndex to FileTimeIndex, and Thread B performs a metadata-only update on r1 (e.g., getExclusiveModFile() or updateProgressIndex(...)) and enters refresh(entry, false) first:
- Lines 167–177 observe the new
FileTimeIndex, incrementdegradedTimeIndexNum, setentry.fileIndex = true, and moveentryto the degraded tail ofsealedTsFileResources. - Because
indexChanged == false, line 179 skipsentry.indexBytes = index.calculateRamSize(), leavingentry.indexBytesat the old device-index size. - If
delta > 0pushesusedMemoryInBytesabovetotalMemorySizeInBytes,chooseTsFileResourceToDegrade()runs immediately; sincer1already hasentry.fileIndex == true(bypassing lines 215–218), it will degrade anotherTsFileResourceunnecessarily.
If Registration records the ITimeIndex reference last measured (e.g., if (indexChanged || entry.index != index) and updating entry.index = index, cleared on removal/clear), any observed index swap will update entry.indexBytes in the same refresh call that updates entry.fileIndex.
| public static final class Registration implements Comparable<Registration> { | ||
| private final TsFileResource resource; | ||
| private final long id; | ||
| private long indexBytes; | ||
| private File file; | ||
| private long startTime; | ||
| private boolean fileIndex; | ||
| private long bytes; | ||
|
|
||
| private Registration(TsFileResource resource, long id) { | ||
| this.resource = resource; | ||
| this.id = id; | ||
| updatePriority(resource.getTimeIndex()); | ||
| } | ||
|
|
||
| private void updatePriority(ITimeIndex index) { | ||
| file = resource.getTsFile(); | ||
| startTime = index.getMinStartTime(); | ||
| fileIndex = index.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE; | ||
| } | ||
|
|
||
| private int comparePriority(Registration other) { | ||
| int cmp = Boolean.compare(fileIndex, other.fileIndex); | ||
| if (cmp == 0) { | ||
| cmp = Long.compare(startTime, other.startTime); | ||
| } | ||
| return cmp != 0 ? cmp : file.getAbsolutePath().compareTo(other.file.getAbsolutePath()); | ||
| } | ||
|
|
||
| @Override | ||
| public int compareTo(Registration other) { | ||
| int cmp = comparePriority(other); | ||
| // Two previously distinct entries can acquire the same priority after degradation. | ||
| return cmp == 0 ? Long.compare(id, other.id) : cmp; | ||
| } |
There was a problem hiding this comment.
[Minor / Sonar java:S1210]: Implementing Comparable<Registration> without overriding equals triggers Sonar java:S1210. Passing a Comparator<Registration> to new TreeSet<>(Registration::compare) instead of implementing Comparable preserves identity equality and avoids the Sonar warning cleanly.
| final Future<ModificationFile> upgrade = exclusiveModFileFuture; | ||
| final CompletableFuture<String> recovery = sharedModFilePathFuture; | ||
| if (upgrade != null) { | ||
| size += RamUsageEstimator.shallowSizeOf(upgrade); | ||
| } | ||
| if (recovery != null) { | ||
| size += RamUsageEstimator.shallowSizeOf(recovery); | ||
| } |
There was a problem hiding this comment.
[Medium] exclusiveModFileFuture is retained after exclusiveModFile is resolved/replaced, and non-volatile fields are read cross-thread here:
- In
DataRegion#loadModFile, upgrading a V1.modsfile callstsFileResource.upgradeModFile(null, false), settingexclusiveModFileFuture = CompletableFuture.completedFuture(newMFile1)(newMFile1~320 B). Right after publishing (DataRegion.java:4941),tsFileResource.setExclusiveModFile(ModificationFile.getExclusiveMods(tsFileResource))assigns a secondModificationFiletoexclusiveModFile. Because neithergetExclusiveModFile(),setExclusiveModFile(...), norresetModFile()clearsexclusiveModFileFuture = null,exclusiveModFileFuturepinsnewMFile1in heap for the lifetime of theTsFileResourcewhilecalculateMetadataRamSize()only countsshallowSizeOf(upgrade)(32 B). exclusiveModFileFuture,sharedModFilePathFuture, andfileare written withoutvolatileor monitor synchronization (upgradeModFile,deserialize,setSharedModFilePathFuture,setFile), butcalculateMetadataRamSize()/TsFileResourceManager#refreshcan read them from any thread.
Please clear exclusiveModFileFuture = null inside synchronized (this) in getExclusiveModFile(), setExclusiveModFile(...), and resetModFile(), and make the cross-thread reference fields volatile.
| private TsFileID tsFileID; | ||
|
|
||
| private long deviceTimeIndexRamSize; | ||
| private volatile Registration memoryRegistration; |
There was a problem hiding this comment.
[CI / Sonar java:S3077]: Both private volatile Registration memoryRegistration; (line 188) and private volatile Map<IDeviceID, List<Pair<String, TimeValuePair>>> lastValues; (line 227) are flagged as BUG (java:S3077) by SonarCloud, causing the PR Quality Gate to fail with a C Reliability Rating on New Code. Please add @SuppressWarnings("java:S3077") (matching exclusiveModFile and compactionModFile above).
| public ITimeIndex replaceDeviceTimeIndexWithFileTimeIndex() { | ||
| ITimeIndex current = timeIndex; | ||
| while (current.getTimeIndexType() != ITimeIndex.FILE_TIME_INDEX_TYPE) { | ||
| FileTimeIndex replacement = |
There was a problem hiding this comment.
[Nit / Null safety]: Both calculateRamSize() and getDeviceTimeIndexRamSize() guard timeIndex == null, whereas current.getTimeIndexType() here will throw NullPointerException if timeIndex == null (e.g., on @TestOnly new TsFileResource() or setTimeIndex(null)). Guarding while (current != null && current.getTimeIndexType() != ITimeIndex.FILE_TIME_INDEX_TYPE) keeps it consistent.
| public void addResourceDeviceTimeIndex( | ||
| TsFileResource tsFileResource, ArrayDeviceTimeIndex deviceTimeIndex) { | ||
| partitionFileDeviceInfoCache.put(tsFileResource, deviceTimeIndex); | ||
| ArrayDeviceTimeIndex previous = | ||
| partitionFileDeviceInfoCache.put(tsFileResource, deviceTimeIndex); | ||
| if (previous == deviceTimeIndex) { | ||
| return; | ||
| } | ||
| long deviceTimeIndexSize = | ||
| tsFileResource.getDeviceTimeIndexRamSize().orElse(deviceTimeIndex.calculateRamSize()); | ||
| deviceTimeIndex.calculateRamSize() - (previous == null ? 0 : previous.calculateRamSize()); | ||
| cachedDeviceInfoSize += deviceTimeIndexSize; | ||
| CompactionMetrics.getInstance().addSelectionCachedDeviceTimeIndexSize(deviceTimeIndexSize); |
There was a problem hiding this comment.
[Medium] O(devices) rescan of resident ArrayDeviceTimeIndex on every compaction selection cycle, and unused TsFileResource#getDeviceTimeIndexRamSize():
addResourceDeviceTimeIndexwas previously the only production caller ofTsFileResource#getDeviceTimeIndexRamSize(), sogetDeviceTimeIndexRamSize()inTsFileResourcenow has zero callers insrc/main.- When memory is within budget, sealed
TsFileResources keep theirArrayDeviceTimeIndexresident. On every periodic compaction selection scan,TsFileResourceCandidate#prepareDeviceInfos()passes(ArrayDeviceTimeIndex) resource.getTimeIndex()here. Previously, reading the cached size wasO(1); callingdeviceTimeIndex.calculateRamSize()unconditionally now walksdeviceToIndex.keySet()and computesdevice.ramBytesUsed()for every device of every non-degraded file on each selection pass, even thoughRegistrationalready computedindexBytes.
If Registration tracks the ITimeIndex reference alongside indexBytes, tsFileResource can return registration.indexBytes in O(1) when registration.index == deviceTimeIndex, falling back to deviceTimeIndex.calculateRamSize() only for unregistered resources or rebuilt indexes.
| if (tsFiles != null) { | ||
| size += RamUsageEstimator.shallowSizeOf(tsFiles); | ||
| for (int i = 0; i < tsFiles.size(); i++) { | ||
| File file = tsFiles.get(i); | ||
| // Analysis appends one resource (possibly null) for each processed file, in order. | ||
| TsFileResource resource = | ||
| resources != null && i < resources.size() ? resources.get(i) : null; | ||
| if (resource == null || resource.getTsFile() != file) { | ||
| size += MemUtils.getFileRamBytes(file); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
[Minor / Sonar java:S3776]: Adding this nested loop/branch pushes ramBytesUsed()'s Cognitive Complexity to 17 (limit 15, flagged as a CRITICAL code smell by SonarCloud). Extracting a small private helper (e.g., estimateUnsharedTsFilesSize()) will keep ramBytesUsed() below the threshold.
| public static long getTsPrimitiveTypeRamBytes(TsPrimitiveType value) { | ||
| long bytes = PRIMITIVE_INSTANCE_SIZE.get(value.getClass()); | ||
| if (value instanceof TsPrimitiveType.TsBinary && value.getBinary() != null) { |
There was a problem hiding this comment.
[Nit / Defensive check]: getFileRamBytes(File) returns 0 on null, whereas public static getTsPrimitiveTypeRamBytes(TsPrimitiveType) will throw NPE on null. Adding if (value == null) { return 0; } (and guarding series == null in TsFileResource#estimateLastValuesSize, which already guards pair == null) makes both helpers null-safe.
Use the absolute file path for registration ownership and match each measured size to its time index. Reuse resident measurements during compaction selection and release consumed or replaced modification upgrade futures.
|




Description
The DataNode uses the estimated heap of sealed TsFile resources to decide when to degrade device indexes. The previous estimate omitted the file path, per-resource locks and atomic objects, and lazily created modification-file handles. It also cached the first device-index estimate across changes. On removal, the manager recalculated the charge, so a resource whose progress index had grown could release bytes belonging to another file.
TsFileResourcenow measures resident metadata separately from the current time index.ArrayDeviceTimeIndexincludes the device map's buckets and entries alongside the device IDs and time arrays.TsFileResourceManagerrecords each registration's charge, applies changes as deltas, and releases the remaining balance on removal. Metadata-only changes reuse the registered index size. Compaction accounts for the device index it actually caches, and LOAD accounts for a shared file handle once.Example
For a sealed resource with a 54-character file path on JDK 17.0.5 with compressed references, creating the exclusive mods handle previously left the estimate at 272 bytes.
The measurements count each owned object once. The budget uses estimates for strings and hash-table capacity.
Verification
with-zh-localeprofile on JDK 17.