test examples

[ElLorans]

fixes #2470

[ElLorans]

Locally the new tests are much faster, not sure how to tackle this

[samuelhwilliams]

6 hours 😂 😭 yeah something doesn't feel quite right ha. Not sure when I can help out with investigating this but can try to 👍

[ElLorans]

6 hours 😂 😭 yeah something doesn't feel quite right ha. Not sure when I can help out with investigating this but can try to 👍

locally most tests pass (I still have to verify that they do not if I add an error to the examples), probably subprocess.Popen and requests.get are not supported by github actions. I need to find a new way, I probably just overnengineered it and I can simply build a dynamic test matrix with tox.

[samialfattani]

I have tried an approach like this but i could not make it to succesfful run, have a look it might be an idea. Assuming that all examples should have create_app() function that returns the Flask app object.

@pytest.mark.parametrize("example_name,example_path", get_example_directories())
def test_example_runs(example_name: str, example_path: Path):
    """
    Test that the example can be started and that the /admin endpoint returns HTTP 200.

    Args:
        example_name: Name of the example.
        example_path: Path to the example directory.
    """

    def run_tester(app):
        try:
            # Wait for the server to start
            time.sleep(2)

            Check if the /admin endpoint is reachable
            response = requests.get("http://localhost:5000/admin")
            assert (
                response.status_code == 200
            ), f"/admin endpoint returned {response.status_code}"

        finally:
            app.sutdown()
            print(f"Example '{example_name}' ran successfully.")

    sys.path.append(str(example_path))
    from main import create_app

    app = create_app()

    import threading

    thread = threading.Thread(target=run_tester, args=(app), daemon=True)
    thread.start()

    # app.run(debug=True)

[ElLorans]

Unfortunately, this does not work because each example needs its own env

[hasansezertasan]

Unfortunately, this does not work because each example needs its own env

WDYT about #2470 (comment)?