Skip to content

Fix heap accounting for sealed TsFile resources - #19347

Open
JackieTien97 wants to merge 2 commits into
masterfrom
ty/tsfile-resource-memory
Open

JackieTien97 wants to merge 2 commits into
masterfrom
ty/tsfile-resource-memory

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Description

The DataNode uses the estimated heap of sealed TsFile resources to decide when to degrade device indexes. The previous estimate omitted the file path, per-resource locks and atomic objects, and lazily created modification-file handles. It also cached the first device-index estimate across changes. On removal, the manager recalculated the charge, so a resource whose progress index had grown could release bytes belonging to another file.

TsFileResource now measures resident metadata separately from the current time index. ArrayDeviceTimeIndex includes the device map's buckets and entries alongside the device IDs and time arrays. TsFileResourceManager records each registration's charge, applies changes as deltas, and releases the remaining balance on removal. Metadata-only changes reuse the registered index size. Compaction accounts for the device index it actually caches, and LOAD accounts for a shared file handle once.

Example

For a sealed resource with a 54-character file path on JDK 17.0.5 with compressed references, creating the exclusive mods handle previously left the estimate at 272 bytes.

Resource state Previous estimate Updated estimate Measured owned heap after the change
Degraded to a file-level index 272 B 528 B 472 B
After creating the exclusive mods handle 272 B 848 B 768 B

The measurements count each owned object once. The budget uses estimates for strings and hash-table capacity.

Verification

  • Self-reviewed the registration, degradation, and removal paths.
  • Added regression coverage for index growth, degradation before the first estimate, metadata changes after registration, and release of LOAD last values.
  • Ran 92 tests across six DataNode test classes covering resources, progress indexes, compaction, and LOAD.
  • Packaged all 50 Maven reactor modules with both the default English messages and the with-zh-locale profile on JDK 17.
  • Ran 32 concurrent degradation, metadata-update, and removal checks in an isolated JVM. Each removal returned the memory balance to its starting value.

Estimate resource metadata and device indexes separately. Track the charge for each registered file so metadata changes, degradation, and removal update the time-index budget consistently.
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.34884% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.45%. Comparing base (dd450f7) to head (afb3799).

Files with missing lines Patch % Lines
...torageengine/dataregion/tsfile/TsFileResource.java 91.05% 11 Missing ⚠️
...ageengine/rescon/memory/TsFileResourceManager.java 98.27% 2 Missing ⚠️
...ueryengine/plan/relational/sql/ast/LoadTsFile.java 90.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19347      +/-   ##
============================================
+ Coverage     49.44%   49.45%   +0.01%     
  Complexity      718      718              
============================================
  Files          5656     5656              
  Lines        418222   418433     +211     
  Branches      55296    55351      +55     
============================================
+ Hits         206769   206954     +185     
- Misses       211453   211479      +26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request Changes (submitted as COMMENT because GitHub does not allow REQUEST_CHANGES from the PR author account)

Overall, separating resident metadata from the degradable ITimeIndex and tracking per-registration charges as deltas in TsFileResourceManager is a solid design fix. However, there are a few correctness, concurrency, performance, and Sonar Quality Gate issues that should be addressed before merging:

  1. [High] File-identity lookup vs. degradation priority in TsFileResourceManager: registerSealedTsFileResource and removeTsFileResource (when memoryRegistration == null) rely on comparePriority ((fileIndex, startTime, path)) to find an existing Registration for the same file, which misses entries whenever the recovery TsFileResource has a different fileIndex (e.g., after degradation) or startTime (e.g., an undeserialized new TsFileResource(file) in compaction recovery).
  2. [Medium] Concurrent index swap vs. metadata-only refresh(entry, false): refresh(entry, false) updates entry.fileIndex to true without refreshing entry.indexBytes, which can leave the old device-index size charged while moving entry out of the degradable head and cause unnecessary degradation of another file.
  3. [Medium] Unreleased exclusiveModFileFuture after mod resolution/replacement & field visibility: exclusiveModFileFuture is never cleared once exclusiveModFile is resolved or overwritten (retaining an unaccounted ModificationFile during LOAD V1-mod upgrades), and several fields read cross-thread in calculateMetadataRamSize() are written without volatile.
  4. [Medium] Compaction selection O(devices) rescan on resident indexes & unused getDeviceTimeIndexRamSize(): Dropping getDeviceTimeIndexRamSize() in CompactionScheduleContext leaves TsFileResource#getDeviceTimeIndexRamSize() unused in production and rescans every resident ArrayDeviceTimeIndex on each compaction selection pass instead of reusing the registered indexBytes.
  5. [CI] SonarCloud Quality Gate failure: java:S3077 on the new volatile fields in TsFileResource (plus java:S3776 in LoadTsFile and java:S1210 in TsFileResourceManager).

See inline comments for details.

Comment on lines +122 to +152
Registration entry = new Registration(resource, ++nextRegistrationId);
Registration previous = sealedTsFileResources.floor(entry);
if (previous == null || entry.comparePriority(previous) != 0) {
sealedTsFileResources.add(entry);
if (entry.fileIndex) {
degradedTimeIndexNum++;
}
// Publish before sampling so a concurrent metadata change cannot miss its charge.
resource.setMemoryRegistration(entry);
try {
refresh(entry, true);
} catch (RuntimeException | Error e) {
removeTsFileResource(resource);
throw e;
}
}
}

/** delete the TsFileResource in PriorityQueue when the source file is deleted */
public synchronized void removeTsFileResource(TsFileResource tsFileResource) {
if (sealedTsFileResources.contains(tsFileResource)) {
sealedTsFileResources.remove(tsFileResource);
if (TimeIndexLevel.valueOf(tsFileResource.getTimeIndexType())
== TimeIndexLevel.FILE_TIME_INDEX) {
memoryBlock.release(tsFileResource.calculateRamSize());
public synchronized void removeTsFileResource(TsFileResource resource) {
Registration entry = resource.getMemoryRegistration();
if (entry == null) {
// Recovery can supply another resource describing the same file.
if (sealedTsFileResources.isEmpty()) {
return;
}
Registration key = new Registration(resource, Long.MIN_VALUE);
entry = sealedTsFileResources.ceiling(key);
if (entry == null || entry.comparePriority(key) != 0) {
return;
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[High] comparePriority ((fileIndex, startTime, path)) cannot reliably match another TsFileResource describing the same physical file:

  1. Silent leak in removeTsFileResource when entry == null: Compaction recovery/rollback paths (InnerSpaceCompactionTask#recoverTargetResource, InsertionCrossSpaceCompactionTask#recoverTaskInfoFromLogFile, CrossSpaceCompactionTask) construct new TsFileResource(targetFile) without calling deserialize() and pass it through replaceTsFileInMemory -> tsFileManager.remove -> removeTsFileResource. That instance has an empty ArrayDeviceTimeIndex (startTime = Long.MAX_VALUE, fileIndex = false), whereas the registered Registration has the actual minStartTime (and fileIndex = true if already degraded). Even if the caller deserialized .resource (fileIndex = false), if the registered entry had already degraded (entry.fileIndex = true), ceiling(key) searches the wrong partition of sealedTsFileResources and entry.comparePriority(key) != 0—so removeTsFileResource returns without removing the registration, releasing entry.bytes, or decrementing degradedTimeIndexNum.
  2. Duplicate registration in registerSealedTsFileResource: Conversely, if a registered resource has degraded (previous.fileIndex = true) and another TsFileResource for the same file is passed to registerSealedTsFileResource with an ArrayDeviceTimeIndex (entry.fileIndex = false) or a different startTime, floor(entry) will not match previous, inserting a second Registration for the same file and double-charging its metadata.

Consider maintaining a secondary Map<String, Registration> keyed by file.getAbsolutePath() (kept in sync in refresh when entry.file != resource.getTsFile(), with its HashMap entry overhead included in REGISTRATION_RAM_BYTES) so both deduplication on register and fallback lookup on remove match by file identity in O(1) regardless of fileIndex or startTime.

Comment on lines +167 to +181
ITimeIndex index = resource.getTimeIndex();
boolean fileIndex = index.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE;
if (entry.fileIndex != fileIndex
|| entry.startTime != index.getMinStartTime()
|| entry.file != resource.getTsFile()) {
sealedTsFileResources.remove(entry);
if (entry.fileIndex != fileIndex) {
degradedTimeIndexNum += fileIndex ? 1 : -1;
}
long memoryReduce = resource.degradeTimeIndex();
degradedTimeIndexNum++;
releaseTimeIndexMemCost(memoryReduce);
sealedTsFileResources.add(resource);
entry.updatePriority(index);
sealedTsFileResources.add(entry);
}
if (indexChanged) {
entry.indexBytes = index.calculateRamSize();
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] refresh(entry, false) updates entry.fileIndex without updating entry.indexBytes when racing with replaceDeviceTimeIndexWithFileTimeIndex():

replaceDeviceTimeIndexWithFileTimeIndex() swaps timeIndex via TIME_INDEX_UPDATER.compareAndSet outside the TsFileResourceManager monitor and then calls updateMemoryCostIfRegistered(true). If Thread A swaps r1.timeIndex from a large ArrayDeviceTimeIndex to FileTimeIndex, and Thread B performs a metadata-only update on r1 (e.g., getExclusiveModFile() or updateProgressIndex(...)) and enters refresh(entry, false) first:

  1. Lines 167–177 observe the new FileTimeIndex, increment degradedTimeIndexNum, set entry.fileIndex = true, and move entry to the degraded tail of sealedTsFileResources.
  2. Because indexChanged == false, line 179 skips entry.indexBytes = index.calculateRamSize(), leaving entry.indexBytes at the old device-index size.
  3. If delta > 0 pushes usedMemoryInBytes above totalMemorySizeInBytes, chooseTsFileResourceToDegrade() runs immediately; since r1 already has entry.fileIndex == true (bypassing lines 215–218), it will degrade another TsFileResource unnecessarily.

If Registration records the ITimeIndex reference last measured (e.g., if (indexChanged || entry.index != index) and updating entry.index = index, cleared on removal/clear), any observed index swap will update entry.indexBytes in the same refresh call that updates entry.fileIndex.

Comment on lines +61 to +95
public static final class Registration implements Comparable<Registration> {
private final TsFileResource resource;
private final long id;
private long indexBytes;
private File file;
private long startTime;
private boolean fileIndex;
private long bytes;

private Registration(TsFileResource resource, long id) {
this.resource = resource;
this.id = id;
updatePriority(resource.getTimeIndex());
}

private void updatePriority(ITimeIndex index) {
file = resource.getTsFile();
startTime = index.getMinStartTime();
fileIndex = index.getTimeIndexType() == ITimeIndex.FILE_TIME_INDEX_TYPE;
}

private int comparePriority(Registration other) {
int cmp = Boolean.compare(fileIndex, other.fileIndex);
if (cmp == 0) {
cmp = Long.compare(startTime, other.startTime);
}
return cmp != 0 ? cmp : file.getAbsolutePath().compareTo(other.file.getAbsolutePath());
}

@Override
public int compareTo(Registration other) {
int cmp = comparePriority(other);
// Two previously distinct entries can acquire the same priority after degradation.
return cmp == 0 ? Long.compare(id, other.id) : cmp;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor / Sonar java:S1210]: Implementing Comparable<Registration> without overriding equals triggers Sonar java:S1210. Passing a Comparator<Registration> to new TreeSet<>(Registration::compare) instead of implementing Comparable preserves identity equality and avoids the Sonar warning cleanly.

Comment on lines +1214 to 1221
final Future<ModificationFile> upgrade = exclusiveModFileFuture;
final CompletableFuture<String> recovery = sharedModFilePathFuture;
if (upgrade != null) {
size += RamUsageEstimator.shallowSizeOf(upgrade);
}
if (recovery != null) {
size += RamUsageEstimator.shallowSizeOf(recovery);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] exclusiveModFileFuture is retained after exclusiveModFile is resolved/replaced, and non-volatile fields are read cross-thread here:

  1. In DataRegion#loadModFile, upgrading a V1 .mods file calls tsFileResource.upgradeModFile(null, false), setting exclusiveModFileFuture = CompletableFuture.completedFuture(newMFile1) (newMFile1 ~320 B). Right after publishing (DataRegion.java:4941), tsFileResource.setExclusiveModFile(ModificationFile.getExclusiveMods(tsFileResource)) assigns a second ModificationFile to exclusiveModFile. Because neither getExclusiveModFile(), setExclusiveModFile(...), nor resetModFile() clears exclusiveModFileFuture = null, exclusiveModFileFuture pins newMFile1 in heap for the lifetime of the TsFileResource while calculateMetadataRamSize() only counts shallowSizeOf(upgrade) (32 B).
  2. exclusiveModFileFuture, sharedModFilePathFuture, and file are written without volatile or monitor synchronization (upgradeModFile, deserialize, setSharedModFilePathFuture, setFile), but calculateMetadataRamSize() / TsFileResourceManager#refresh can read them from any thread.

Please clear exclusiveModFileFuture = null inside synchronized (this) in getExclusiveModFile(), setExclusiveModFile(...), and resetModFile(), and make the cross-thread reference fields volatile.

private TsFileID tsFileID;

private long deviceTimeIndexRamSize;
private volatile Registration memoryRegistration;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CI / Sonar java:S3077]: Both private volatile Registration memoryRegistration; (line 188) and private volatile Map<IDeviceID, List<Pair<String, TimeValuePair>>> lastValues; (line 227) are flagged as BUG (java:S3077) by SonarCloud, causing the PR Quality Gate to fail with a C Reliability Rating on New Code. Please add @SuppressWarnings("java:S3077") (matching exclusiveModFile and compactionModFile above).

Comment on lines +1508 to +1511
public ITimeIndex replaceDeviceTimeIndexWithFileTimeIndex() {
ITimeIndex current = timeIndex;
while (current.getTimeIndexType() != ITimeIndex.FILE_TIME_INDEX_TYPE) {
FileTimeIndex replacement =

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit / Null safety]: Both calculateRamSize() and getDeviceTimeIndexRamSize() guard timeIndex == null, whereas current.getTimeIndexType() here will throw NullPointerException if timeIndex == null (e.g., on @TestOnly new TsFileResource() or setTimeIndex(null)). Guarding while (current != null && current.getTimeIndexType() != ITimeIndex.FILE_TIME_INDEX_TYPE) keeps it consistent.

Comment on lines 90 to 100
public void addResourceDeviceTimeIndex(
TsFileResource tsFileResource, ArrayDeviceTimeIndex deviceTimeIndex) {
partitionFileDeviceInfoCache.put(tsFileResource, deviceTimeIndex);
ArrayDeviceTimeIndex previous =
partitionFileDeviceInfoCache.put(tsFileResource, deviceTimeIndex);
if (previous == deviceTimeIndex) {
return;
}
long deviceTimeIndexSize =
tsFileResource.getDeviceTimeIndexRamSize().orElse(deviceTimeIndex.calculateRamSize());
deviceTimeIndex.calculateRamSize() - (previous == null ? 0 : previous.calculateRamSize());
cachedDeviceInfoSize += deviceTimeIndexSize;
CompactionMetrics.getInstance().addSelectionCachedDeviceTimeIndexSize(deviceTimeIndexSize);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] O(devices) rescan of resident ArrayDeviceTimeIndex on every compaction selection cycle, and unused TsFileResource#getDeviceTimeIndexRamSize():

  1. addResourceDeviceTimeIndex was previously the only production caller of TsFileResource#getDeviceTimeIndexRamSize(), so getDeviceTimeIndexRamSize() in TsFileResource now has zero callers in src/main.
  2. When memory is within budget, sealed TsFileResources keep their ArrayDeviceTimeIndex resident. On every periodic compaction selection scan, TsFileResourceCandidate#prepareDeviceInfos() passes (ArrayDeviceTimeIndex) resource.getTimeIndex() here. Previously, reading the cached size was O(1); calling deviceTimeIndex.calculateRamSize() unconditionally now walks deviceToIndex.keySet() and computes device.ramBytesUsed() for every device of every non-degraded file on each selection pass, even though Registration already computed indexBytes.

If Registration tracks the ITimeIndex reference alongside indexBytes, tsFileResource can return registration.indexBytes in O(1) when registration.index == deviceTimeIndex, falling back to deviceTimeIndex.calculateRamSize() only for unregistered resources or rebuilt indexes.

Comment on lines +398 to +409
if (tsFiles != null) {
size += RamUsageEstimator.shallowSizeOf(tsFiles);
for (int i = 0; i < tsFiles.size(); i++) {
File file = tsFiles.get(i);
// Analysis appends one resource (possibly null) for each processed file, in order.
TsFileResource resource =
resources != null && i < resources.size() ? resources.get(i) : null;
if (resource == null || resource.getTsFile() != file) {
size += MemUtils.getFileRamBytes(file);
}
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor / Sonar java:S3776]: Adding this nested loop/branch pushes ramBytesUsed()'s Cognitive Complexity to 17 (limit 15, flagged as a CRITICAL code smell by SonarCloud). Extracting a small private helper (e.g., estimateUnsharedTsFilesSize()) will keep ramBytesUsed() below the threshold.

Comment on lines +62 to +64
public static long getTsPrimitiveTypeRamBytes(TsPrimitiveType value) {
long bytes = PRIMITIVE_INSTANCE_SIZE.get(value.getClass());
if (value instanceof TsPrimitiveType.TsBinary && value.getBinary() != null) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit / Defensive check]: getFileRamBytes(File) returns 0 on null, whereas public static getTsPrimitiveTypeRamBytes(TsPrimitiveType) will throw NPE on null. Adding if (value == null) { return 0; } (and guarding series == null in TsFileResource#estimateLastValuesSize, which already guards pair == null) makes both helpers null-safe.

Use the absolute file path for registration ownership and match each measured size to its time index. Reuse resident measurements during compaction selection and release consumed or replaced modification upgrade futures.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant