Feat/fuzzing

[FloydZ]

Description

This PR contains

• the possibility to run fuzzing tests
• fixes for found issues with the fuzzer

Check

source .venv/bin/activate
pip install atheris
cd tests/fuzzing

python fuzz.py --bytes 1 --sd
python fuzz.py --bytes 1 --mq
python fuzz.py --bytes 1 --sdfq
python fuzz.py --bytes 1 --regsd
python fuzz.py --bytes 1 --pk
python fuzz.py --bytes 1 --lw
python fuzz.py --bytes 1 --pe
python fuzz.py --bytes 1 --mr
python fuzz.py --bytes 1 --uov
python fuzz.py --bytes 1 --mayo

python fuzz.py --sd
python fuzz.py --mq
python fuzz.py --sdfq
python fuzz.py --regsd
python fuzz.py --pk
python fuzz.py --lw
python fuzz.py --pe
python fuzz.py --mr
python fuzz.py --uov
python fuzz.py --mayo

Docker Setup:

make docker-fuzzing
docker run --rm cf-fuzzer ./tests/fuzzing/fuzz.py --sd

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Dioprz]

(I changed this to a draft to avoid accidental merging)

After some reads about Atheris and fuzzy testing, this sounds very interesting. Happy to learn about its existence.

Let me know if I can help with something here, @FloydZ.

[FloydZ]

Hi @Dioprz ,
currently I'm not really 100% happy with the solution. There are two main issues:

• atheris only allows for a single function to be fuzzed at each time. Thats why I have code like:

#atheris.Setup(sys.argv, SDFuzz)
...
atheris.Setup(sys.argv, RankSDFuzz)
...
#atheris.Setup(sys.argv, MAYOFuzz)

And you need enable/disable the estimator by hand. If you know how to circumvent that, without adding a bash script, which calls the python code multiple times. let me know.
The second problem is the exception handling. At first I thought catching ValueError is enough to separate between "real problematic" errors in the code, and errors which are already caught by the class constructor. But turns out errors like log2(0) are also ValueErrors, hence I have this ugly:

if type(e) != ValueError or str(e) == "math domain error":

check to difference between those two. If you have a better solution for this, I would really appreciate it. Plus Im not 100% that this is actually really correct.

[FloydZ]

Apparently, sonar cloud thinks that:

if w <= 0 or k <= 0:

always evaluates to false, for what ever reason.

[Dioprz]

Apparently, sonar cloud thinks that:

if w <= 0 or k <= 0:

always evaluates to false, for what ever reason.

The full code snippet failing (well, one of them) is this one:

    def __init__(self, n: int, k: int, w: int, **kwargs):
        super().__init__(**kwargs)
        if k <= 0 or n <= 0 or w <= 0:
            raise ValueError("n, k, w must be positive integers")
        if k > n:
            raise ValueError("k must be smaller or equal to n")
        if w > n - k:
            raise ValueError("w must be smaller or equal to n-k")
        if w <= 0 or k <= 0:
            raise ValueError("w and k must be at least 1")

And looking at it... I think it makes sense because you will never be able to reach the last if branch, as the same conditions will trigger the first if branch. Makes sense, or I'm overlooking something?

[FloydZ]

updated PR. Can be reviewed

[Memphisd]

I think this atheris library should be preinstalled in the dev-docker. It seems rather elaborate to install it, on Mac at least.

[Javierverbel]

Second command doesn't work for me. I get

Unable to find image 'cf-fuzzer:latest' locally
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "./tests/fuzzing/fuzz.py": permission denied: unknown.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud