add /test endpoint

[JosefAssadERST]

This job type already gives us free prometheus metrics and free swagger. How about free test running?

I'm thinking that calling an endpoint like /test triggers a test runner, and the return to the client is the result of the test run.

Surely this is fairly doable?

[iszulcdeepsense]

What kind of things would you like to test, your model? Would you like to feed it with bunch of inputs and check the expected outcomes?

Out of curiosity, is this something that can change in time, so you need an endpoint to call it more than once? If not, I suggest to run unit tests before deploying. However I see that maybe you don't trust your local environment (vs the one deployed on Racetrack) and that's why you want to test it in the wild.

Nevertheless, I suppose it may be done by means of auxiliary endpoints, though we can consider a dedicated feature to make it easier.

[iszulcdeepsense]

What about running end-to-end test suite, doing requests from your computer to the deployed job? I believe this would be the most trustworthy method, cause passing /test endpoint won't guarantee you didn't mess up something in the /perform endpoint.

[JosefAssadERST]

I think the initial idea is something simpler, literally just invoking something like here and serving the output on an endpoint.

e2e is a lot more complex and cannot really be encapsulated inside a job by its jobtype.

[LookACastle]

Reminder that this issue was raised due to concerns from developers using racetrack (that is, not infrastructure guys, nor people putting code into this repo. The ones running jobs on racetrack) that their environments didn't match the one on racetrack. As far as I understood, this was based on actual experience and not just people being worrywarts.

A /test endpoint would solve this issue for classic unit tests and such, in a way e2e via /perform would not. After all, a unit test suite might cover erroneous conditions that aren't reproducible through an e2e test - like, some 3rd party server erroring out.

[JosefAssadERST]

Not a bad point @LookACastle . You're thinking it allows people to run a test suite in production conditions which they might not be able to (easily) replicate off-production?