[Harness Bug] Output overwrite conflict when running multiple versions of the same executor on the same ICU version

[sffc-bot]

Describe the Bug

The test harness currently overwrites test outputs when multiple versions of the same executor (e.g., rust for ICU4X) are configured to run on the same ICU version.

Details

In run_config.json, it is possible to configure multiple versions of the same executor to run on the same ICU version. For example:

• rust2.1 (ICU4X 2.1) targeting icu78
• rust2.2 (ICU4X 2.2) targeting icu78

Both of these entries use "exec": "rust" and "icu_version": "icu78".

During the execution of generateDataAndRun.sh (or genData100.sh), the harness runs these sequentially:

1. rust2.1 runs and writes its output to TEMP_DATA/testOutput/rust/icu78/.
2. rust2.2 runs immediately after and writes its output to the exact same directory TEMP_DATA/testOutput/rust/icu78/, completely overwriting the outputs of rust2.1.

When the verifier runs, it only finds the outputs of rust2.2. The results for rust2.1 are completely lost and do not appear on the dashboard.

This also affected rust2.0 and rust2.0-beta2 on icu77.

Root Cause

In testdriver/testplan.py, the outputFilePath is determined without incorporating the platformVersion (which is commented out):

                self.outputFilePath = os.path.join(self.options.file_base,
                                                   self.options.output_path,
                                                   self.test_lang,
                                                   self.options.icu_version,
                                                   # self.platformVersion,
                                                   self.testData.testDataFilename)

Uncommenting self.platformVersion would resolve the overwrite, but it requires refactoring verifier/verifier.py to recursively scan for version directories instead of assuming they map directly to ICU versions.

Temporary Workaround

We have temporarily removed the older conflicting versions (rust2.1 and rust2.0-beta2) from run_config.json to avoid redundant runs and data loss.