File params-sim-run.json

[chenyifanthu]

How to generate file params-sim-run.json?

[limjcst]

params-sim-run.json should be generated in the following steps.

1. Execute python -m circa.experiment generate to generate the simulation data.
2. Execute python -m circa.experiment tune to explore parameter combinations.
   • You can insert --max-workers 16 before tune to enable multi-processing. 16 can be replaced with other positive integers, e.g., the number of CPU cores.
   • --model-params params-sim-tune.json stated in README.md is also optional, which is used to filtered out some parameter combinations.
3. Execute Rscript img/draw.sim.R to produce summaries under img/output.
   • This command may end with a file-not-found error, as other simulation experiments have not been conducted.
   • Only the content in img/output/best-sim-tuning.tex is in need now.
4. Execute python -m circa.experiment params > params-sim-run.json to create a parameter template.
5. Edit params-sim-run.json (generated in the Step 4) according to img/output/best-sim-tuning.tex (generated in the Step 3).
   • For each Method in img/output/best-sim-tuning.tex, choose the row with the highest AC.5. The related parameters can be found in the name column.
   • For example, in my previous run, name of the SPOT instance with the highest AC.5 is Empty-SPOT\_p0.01. Hence, in params-sim-run.json, only 0.01 in the spot.risk list is reserved and other options (0.001 and 0.0001) are removed.
   • All the graph parameters are neglected in the simulation study. As for a real-world dataset, the leading component in name before the first dash (-) is for the graph. For example, PC\_gauss\_a0.5 means,
     • only pc_gauss is used,
     • choose 0.5 for alpha, and
     • max_conds_dim is unlimited with null in the json file.

Our experiment results can be found in figshare.
In circa.zip, you can find all four param-*.json we used.