[3.1.0] Plugin fails on failed test cases even configured with success

[bicschneider]

Done Status As: Success
Write keyframes of passed flows: false
Schedule xclsc1702[7c6f8dd2-5b24-4e7b-836e-0b4ae73c3958] successfully detected!
//////////////////////////////////////////////////////////////////////////////////////
Schedule xclsc1702[7c6f8dd2-5b24-4e7b-836e-0b4ae73c3958] Launched Successfully!
The schedule status is already 'Executing' - wait a minute...
----------------------------------------------------------------------------------------
........
Schedule: xclsc1702[7c6f8dd2-5b24-4e7b-836e-0b4ae73c3958]
Passed testcases: 0
Failed testcases: 1
Error testcases: 0
|---------------------------------------------------------------
| Total passed testcases: 0
| Total failed testcases: 1
| Total error testcases: 0
Creating report file by path: reports.xml
FAILURE
Leapwork for Jenkins  plugin  successfully finished!

This is really "crucial" for our pipeline that "flow" failures are treated as non-intrusive to the build execution

           pluginHandler.createJUnitReport(run,workspace,getLeapworkReport(),listener,buildResult);

            if (buildResult.getErrors() > 0 ) {
                listener.getLogger().println("[ERROR] There were detected case(s) with status 'Error', 'Inconclusive', 'Timeout' or 'Cancelled'. Please check the report or console output for details. Set the build status to FAILURE as the results of the cases are not deterministic..");
                run.setResult(Result.FAILURE);
                listener.getLogger().println("");
            }
            if ( buildResult.getFailedTests() > 0 ) {
                if ( "Success".equals(this.leapworkDoneStatusAs) ){
                    listener.getLogger().println("There were test cases that had failures/issues, but the plugin has been configured to return: 'Success' in this case");
                    run.setResult(Result.SUCCESS);
                } else if ( "Unstable".equals(this.leapworkDoneStatusAs) ) {
                    listener.getLogger().println("There were test cases that had failures/issues, but the plugin has been configured to return: 'Unstable' in this case");
                    run.setResult(Result.UNSTABLE);
                }
            } else {
                listener.getLogger().println("No issues detected");
            }
            listener.getLogger().println(Messages.PLUGIN_SUCCESSFUL_FINISH);

[romanchetto]

Hello @bicschneider
I've just tested it, but failed to reproduce it. Could you provide more information?

[bicschneider]

@romanchetto : Thanks for the response.. I agree that it works in a positive scenario(passed), but in case testcases fail the problem occurs: Total failed testcases: 1. Please have a look at my log on the reported..

[romanchetto]

Hello @bicschneider . I'm a little confused. If you set default as "Success" the plugin considers the case to be passed, if NOT "Success" it is considered to be failed. It was designed in such way.
This is parts of code, which are responsible for that:

leaptest-plugin/src/main/java/com/customatics/leaptest_plugin/PluginHandler.java

Lines 118 to 121 in d57543c

	public boolean isDoneStatusAsSuccess(String doneStatusAs)
	{
	return doneStatusAs.contentEquals("Success");
	}

And

leaptest-plugin/src/main/java/com/customatics/leaptest_plugin/PluginHandler.java

Lines 647 to 662 in d57543c

	if(flowStatus.contentEquals("Initializing") ||
	flowStatus.contentEquals("Connecting") ||
	flowStatus.contentEquals("Connected") ||
	flowStatus.contentEquals("Running") ||
	flowStatus.contentEquals("NoStatus") ||
	(flowStatus.contentEquals("Passed") && !writePassedKeyframes) ||
	(flowStatus.contentEquals("Done") && doneStatusAsSuccess && !writePassedKeyframes))
	{
	return runItem;
	}
	else
	{
	Failure keyframes = getRunItemKeyFrames(client,controllerApiHttpAddress,accessKey,runItemId,runItem,scheduleTitle,environmentTitle,listener);
	runItem.failure = keyframes;
	return runItem;
	}

So dou you want to add new statuses for "Done" ?

[bicschneider]

@romanchetto
I know it is the way, but I find it a problem.. A "FAILURE" in Jenkins is considered an execution failure. ( eg . We could not execute what we expect execute) which is an undetermined state. The consequence is that any further execution in build-steps are stopped. (consider this syntax error and identical to throw an exception from the main program including stacktrace)
A test case that failed is not an execution error. The test ran to an determined state. (consider this a semantic issue ). These issues are recorded in the report and it is up to Junit plugin to determine if the report is acceptable or not. (UNSTABLE and SUCCESS the execution will continue)
Also.. If Jenkins first are in 'FAILURE'-state it cannot be made good again to UNSTABLE or SUCCESS if it considered ok by analyzing the test-cases in question.
eg. A build cannot be made "better" again

• Acceptable: SUCCESS -> UNSTABLE -> FAILURE is ok,
• Non-acceptable: FAILURE -> SUCCESS || FAILURE -> UNSTABLE || UNSTABLE -> SUCCESS.

[bicschneider]

@romanchetto: gentle ping

[romanchetto]

Hello @bicschneider . So in this situation, do you suggest changing the plugin build result to UNSTABLE for all "failed" (not "errored") cases in this part of code?:

leaptest-plugin/src/main/java/com/customatics/leaptest_plugin/LeaptestJenkinsBridgeBuilder.java

Lines 185 to 190 in d57543c

	if (buildResult.getErrors() > 0 || buildResult.getFailedTests() > 0 || invalidSchedules.size() > 0) {
	if(buildResult.getErrors() > 0)
	listener.getLogger().println(Messages.ERROR_NOTIFICATION);
	listener.getLogger().println("FAILURE");
	build.setResult(Result.FAILURE);
	}

[bicschneider]

@romanchetto : In my fork I have made the configure option to set it according to your need/preference. It is hard to predict what the right status in all situation. If you do not want the configuration then I would leave it as SUCCESS if no errors (eg failed testcases does not have influence on the result). As CI/CD engineer you can interpret the report and set the status accordingly ( f.ex via JUNIT plugin). Here are my the snips:
config.xml

        <f:entry title="Set status if test cases fail:">
            <f:select field="leapworkDoneStatusAs"/>
        </f:entry>

help-leapworkDoneStatusAs.html

<div>
    Choose how plugin should interpret defects in test cases. It can be set to "Failed", "Unstable or as "Success".
    It should be seen in relation Jenkins JUnit parser,reporter and evaluator, which can be figured to set the job status
    more fine grained. It is then recommended to set this to "Success".
    NOTE: Any execution errors( network etc. and invalid test case ) will always result in FAILURE and throw
    abort exception.
</div>

LeaptestJenkinsBridgeBuilder.java

        public ListBoxModel doFillLeapworkDoneStatusAsItems() {
            return new ListBoxModel(
                    new ListBoxModel.Option("Success", "Success" ),
                    new ListBoxModel.Option("Unstable", "Unstable" ),
                    new ListBoxModel.Option("Failed", "Failed" )
            );
        }

LeaptestJenkinsBridgeBuilder.java

            if (buildResult.getErrors() > 0 ) {
                listener.getLogger().println("[ERROR] There were detected case(s) with status 'Error', 'Inconclusive', 'Timeout' or 'Cancelled'. Please check the report or console output for details. Set the build status to FAILURE as the results of the cases are not deterministic..");
                run.setResult(Result.FAILURE);
                listener.getLogger().println("");
            }
            if ( buildResult.getFailedTests() > 0 ) {
                if ( "Success".equals(this.leapworkDoneStatusAs) ){
                    listener.getLogger().println("There were test cases that had failures/issues, but the plugin has been configured to return: 'Success' in this case");
                    run.setResult(Result.SUCCESS);
                } else if ( "Unstable".equals(this.leapworkDoneStatusAs) ) {
                    listener.getLogger().println("There were test cases that had failures/issues, but the plugin has been configured to return: 'Unstable' in this case");
                    run.setResult(Result.UNSTABLE);
                }
            } else {
                listener.getLogger().println("No issues detected");
            }
            listener.getLogger().println(Messages.PLUGIN_SUCCESSFUL_FINISH);