[common] Keep the shared _temporary directory while other writers stage there

[sundapeng]

Problem

RenamingTwoPhaseOutputStream stages into <target-dir>/_temporary/.tmp.<uuid> — the same directory name Hadoop's FileOutputCommitter uses, and with it Hive, Spark and MapReduce writing to the same location. clean() removed that directory recursively:

public void clean(FileIO fileIO) {
    fileIO.deleteDirectoryQuietly(tempPath.getParent());   // <target-dir>/_temporary
}

clean() runs after every successful commit, so a Paimon write into a directory where another job is mid-write deleted that job's staged files, and the job then failed to commit its own output.

Fix

Delete only the file this committer staged, then ask whether the directory can go by attempting a non-recursive delete: it removes the directory when nothing is staged there any more, and refuses while a concurrent writer still has files in it. Removing it stays best-effort — a writer that stages here next recreates it — so a failure is logged at debug and the commit is unaffected.

Scope

paimon-common only, and in practice this is a format-table path: the sole production caller of FileIO.newTwoPhaseOutputStream is FormatTableSingleFileWriter, so managed tables never construct this stream. OSSFileIO, S3FileIO and JindoFileIO override it with multipart upload and do not stage under _temporary at all.

This is one of three PRs splitting up the original change; the other two are the format-table read path (#8861) and the INSERT OVERWRITE delete path, both in paimon-core and both independent of this one.

Tests

RenamingTwoPhaseOutputStreamTest:

• testCleanKeepsTheSharedStagingDirectory — a foreign file staged in the same _temporary survives clean(); fails against the previous recursive delete
• testCleanRemovesTheStagingDirectoryOnceEmpty — nothing else staged, so the directory does go, and a caller listing the target directory does not trip over a leftover _temporary

[JingsongLi]

When clean() non-recursively deletes the empty _temporary directory, another writer may have just completed mkdirs(_temporary) but not yet created the temporary file; in this case, the directory will be deleted, and subsequently, when that writer attempts to open the temporary file, it will immediately throw an IOException. It is recommended that clean() delete only its own tempPath and not delete the shared _temporary directory.

[sundapeng]

Thanks for the review, you are right.

LocalFileIO.newOutputStream does mkdirs(parent) first and then opens the file, they are two steps. So an empty _temporary does not mean nobody is using it, another writer may be in the middle of these two steps.

I updated the PR. Now clean() only deletes its own tempPath, and does not touch the _temporary directory anymore. The directory will stay there, and the writers who use it can remove it. Tests are updated too.