Skip to content

Load: Fix active load reliability issues#18224

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/load-active-reliability
Open

Load: Fix active load reliability issues#18224
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:fix/load-active-reliability

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Active load handoff

  • Transfer each TsFile and its resource/mods files through a unique directory, publish the TsFile last, and clean partial output on failure.
  • Keep source files until the whole handoff succeeds and isolate same-name file groups from one another.

Scanner and loader lifecycle

  • Deduplicate scanner candidates derived from TsFiles and companion files.
  • Prevent internal attribute/transfer directories from being inferred as table databases.
  • Clear stale pending state across loader restarts and clean empty transfer directories after processing.

Load resource cleanup

  • Release cached LOAD memory when split/routing exits early.
  • Retry deletion of the TsFile and each companion independently so one failure does not skip the remaining cleanup.

Test coverage

  • Datanode load unit-test suite: 41 tests passed.
  • Final focused regression suite: 15 tests passed.
  • Spotless check, Checkstyle, and git diff --check passed.

This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • LoadSingleTsFileNode and LoadTsFileScheduler
  • ActiveLoadDirScanner, ActiveLoadPendingQueue, and ActiveLoadTsFileLoader
  • LoadUtil

});
}
} catch (final IOException | RuntimeException e) {
org.apache.iotdb.commons.utils.FileUtils.deleteFileOrDirectory(transferDir, true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind this

Comment on lines +272 to 274
private static boolean isTsFile(final File file) {
return file.getAbsolutePath().equals(getTsFilePath(file.getAbsolutePath()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use endsWith?

Comment on lines +61 to +83
final List<File> sourceFiles = createTsFileAndCompanions();
for (final File sourceFile : sourceFiles) {
Files.write(
new File(targetDir, sourceFile.getName()).toPath(),
"existing".getBytes(StandardCharsets.UTF_8));
}

LoadUtil.transferFilesToActiveDir(targetDir, sourceFiles, true);

final File[] transferDirs = targetDir.listFiles(File::isDirectory);
Assert.assertNotNull(transferDirs);
Assert.assertEquals(1, transferDirs.length);
for (final File sourceFile : sourceFiles) {
Assert.assertFalse(sourceFile.exists());
final File transferredFile = new File(transferDirs[0], sourceFile.getName());
Assert.assertTrue(transferredFile.exists());
Assert.assertArrayEquals(
sourceFile.getName().getBytes(StandardCharsets.UTF_8),
Files.readAllBytes(transferredFile.toPath()));
Assert.assertArrayEquals(
"existing".getBytes(StandardCharsets.UTF_8),
Files.readAllBytes(new File(targetDir, sourceFile.getName()).toPath()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of the "existing" file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants