So that part of it runs „during syntax phase“, part of it „during semantics phase“.
Could be done via two separate test suites, or even just giving TestSuite a number argument, where „TestSuite 3“ would mean that if one of the first three entries from the test suite list fail, that’s considered a syntax error, if a later one fails, it’s considered a semantics error.
One application could be to prevent students „passing“ the syntax phase by simply submitting the original template with all „undefined“ untouched (which usually will compile). The test suites could start with tests that catch those undefineds (as used to be the case for CodeWorld tasks). Possibly there are also other applications of „considering some tests early, others later“. One that comes to mind is to check those tests early which also appear in the student-visible test suite inside the task template.
So that part of it runs „during syntax phase“, part of it „during semantics phase“.
Could be done via two separate test suites, or even just giving TestSuite a number argument, where „TestSuite 3“ would mean that if one of the first three entries from the test suite list fail, that’s considered a syntax error, if a later one fails, it’s considered a semantics error.
One application could be to prevent students „passing“ the syntax phase by simply submitting the original template with all „undefined“ untouched (which usually will compile). The test suites could start with tests that catch those undefineds (as used to be the case for CodeWorld tasks). Possibly there are also other applications of „considering some tests early, others later“. One that comes to mind is to check those tests early which also appear in the student-visible test suite inside the task template.