[Spark] Parse Spark version from JAR/RELEASE before spark-submit fallback

[shangeyao]

Problem

SparkVersion currently relies solely on spark-submit --version, which requires the Spark installation's JDK to be compatible with the Console JVM. This blocks registering Spark 4.x environments when Console runs on JDK 8.

Proposed solution

Adopt jar-first version detection (similar to Flink env registration improvements):

• Parse version from spark-core_*.jar or RELEASE file under SPARK_HOME
• Fall back to spark-submit --version only when jar parsing fails
• Propagate Spark JAVA_HOME from spark-env.sh for job submission

Scope

• SparkVersion.scala, SparkEnvUtils.scala
• YarnClient.scala (setJavaHome)
• Tests and documentation

[shangeyao]

Closing because related PR #4383 was closed without merge. Work is not being pursued on this branch at the moment.