Implement --powerset-skip-first and --powerset-num-tests

[vi]

No description provided.

[taiki-e]

Thanks. I think adding support for the --randomize-poerset option to the --partition option (taiki-e#253) would be easier to use than adding these individual flags.

[vi]

Thanks. I think adding support for the --randomize-poerset option to the --partition option (taiki-e#253) would be easier to use than adding these individual flags.

Do you mean rebasing the --randomize-powerset pull requests against the (already merged) --partition merge request?

Or implementing --powerset-num-tests similarly to --partition implementation (i.e. not specific to powersets)?

---

Main use case of --randomize-powerset is to non-exhaustively test crates with a lot of features, where powerset can be considered effectively infinite. Partitioning can be cumbersome there, as each new feature would still double the number of tests.

---

P.S. I didn't know about partitioning is already supported.

[taiki-e]

Do you mean rebasing the --randomize-powerset pull requests against the (already merged) --partition merge request?

Or implementing --powerset-num-tests similarly to --partition implementation (i.e. not specific to powersets)?

I thought it might be possible to reuse the (API) design of existing partitioning feature and it would be more user-friendly than adding a new feature with a completely new design for a similar purpose.

Main use case of --randomize-powerset is to non-exhaustively test crates with a lot of features, where powerset can be considered effectively infinite. Partitioning can be cumbersome there, as each new feature would still double the number of tests.

Hmm, if there are too many combinations of features to make an exhaustive check, then I think the taiki-e#255 's approach would be more efficient. (An approach that calculates powerset in advance can result in OOM.)

[vi]

Why is taiki-e#255 unmerged?

(If I looked in unmerged pull requests prior to implementing, neither this nor taiki-e#266 would have been needed)

The only thing taiki-e#255 + taiki-e#266 does not cover is both parallel and randomized runs (without unnecessary repetition).

[taiki-e]

Why is taiki-e#255 unmerged?

There are some TODOs and I don't have the bandwidth to complete them.

From the description of taiki-e#255:

There are a few things missing as I have just roughly implemented the idea:

// TODO: 
// - If duplicates are found, they should be de-duplicated and regenerated. 
//   - Same for filtered case. 
// - If the total number of possible combinations is less than num_samples, 
//   then we should use normal powerset().