Conversation
|
💔 -1 overall
This message was automatically generated. |
|
rerun test please |
|
Ping |
ajfabbri
left a comment
ajfabbri
left a comment
There was a problem hiding this comment.
Thanks for the contribution. I don't know enough about this stuff yet to approve but asked a question.
| <!-- OWASP's dependency-check plugin will scan the third party | ||
| dependencies of this project for known CVEs (security | ||
| vulnerabilities against them). It will produce a report | ||
| in target/dependency-check-report.html. To invoke, run |
There was a problem hiding this comment.
I'm not familiar with this stuff. Does this dependency-check-report.html still get generated with the same filename? Do you know what consumes it?
There was a problem hiding this comment.
Its a tool you run on the command line . Just run it you will see! Our checks all the pom dependencies vs the CVE database and tells you which software has what problems.
There was a problem hiding this comment.
I added the plugin to livy here:
https://issues.apache.org/jira/browse/LIVY-1032?filter=-2
You can see that the dependency tree of Hadoop is gruesome with CVE
There was a problem hiding this comment.
Looks like this:
CVE-2023-26031, CVE-2024-23454
hadoop-client-api-3.3.4.jar (pkg:maven/org.apache.hadoop/hadoop-client-api@3.3.4, pkg:maven/org.apache.hadoop/hadoop-client@3.3.4, cpe:2.3:a:apache:hadoop:3.3.4:*:*:*:*:*:*:*) : CVE-2023-26031, CVE-2024-23454
hadoop-client-api-3.3.4.jar: jquery-ui-1.13.1.custom.min.js (pkg:javascript/jquery-ui@1.13.1) : CVE-2022-31160
hadoop-client-api-3.3.4.jar: jquery.dataTables.min.js (pkg:javascript/jquery.datatables@1.10.18) : CVE-2020-28458, CVE-2021-23445, prototype pollution, possible XSS
hadoop-client-runtime-3.3.4.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml (pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.12.7, cpe:2.3:a:fasterxml:jackson-databind:2.12.7:*:*:*:*:*:*:*, cpe:2.3:a:fasterxml:jackson-modules-java8:2.12.7:*:*:*:*:*:*:*) : CVE-2022-42003, CVE-2022-42004, CVE-2023-35116
hadoop-client-runtime-3.3.4.jar/META-INF/maven/com.fasterxml.woodstox/woodstox-core/pom.xml (pkg:maven/com.fasterxml.woodstox/woodstox-core@5.3.0, cpe:2.3:a:fasterxml:woodstox:5.3.0:*:*:*:*:*:*:*) : CVE-2022-40152
hadoop-client-runtime-3.3.4.jar/META-INF/maven/com.google.guava/guava/pom.xml (pkg:maven/com.google.guava/guava@30.1.1-jre, cpe:2.3:a:google:guava:30.1.1:*:*:*:*:*:*:*) : CVE-2023-2976, CVE-2020-8908
hadoop-client-runtime-3.3.4.jar/META-INF/maven/com.google.protobuf/protobuf-java/pom.xml (pkg:maven/com.google.protobuf/protobuf-java@3.7.1, cpe:2.3:a:google:protobuf-java:3.7.1:*:*:*:*:*:*:*) : CVE-2024-7254, CVE-2022-3171, CVE-2021-22569
hadoop-client-runtime-3.3.4.jar/META-INF/maven/com.nimbusds/nimbus-jose-jwt/pom.xml (pkg:maven/com.nimbusds/nimbus-jose-jwt@9.8.1, cpe:2.3:a:connect2id:nimbus_jose\+jwt:9.8.1:*:*:*:*:*:*:*) : CVE-2023-52428
hadoop-client-runtime-3.3.4.jar/META-INF/maven/commons-beanutils/commons-beanutils/pom.xml (pkg:maven/commons-beanutils/commons-beanutils@1.9.4, cpe:2.3:a:apache:commons_beanutils:1.9.4:*:*:*:*:*:*:*) : CVE-2025-48734
hadoop-client-runtime-3.3.4.jar/META-INF/maven/commons-io/commons-io/pom.xml (pkg:maven/commons-io/commons-io@2.8.0, cpe:2.3:a:apache:commons_io:2.8.0:*:*:*:*:*:*:*) : CVE-2024-47554
hadoop-client-runtime-3.3.4.jar/META-INF/maven/commons-net/commons-net/pom.xml (pkg:maven/commons-net/commons-net@3.6, cpe:2.3:a:apache:commons_net:3.6:*:*:*:*:*:*:*) : CVE-2021-37533
hadoop-client-runtime-3.3.4.jar/META-INF/maven/net.minidev/json-smart/pom.xml (pkg:maven/net.minidev/json-smart@1.3.2, cpe:2.3:a:json-smart_project:json-smart:1.3.2:*:*:*:*:*:*:*, cpe:2.3:a:json-smart_project:json-smart-v1:1.3.2:*:*:*:*:*:*:*) : CVE-2021-31684, CVE-2023-1370
hadoop-client-runtime-3.3.4.jar/META-INF/maven/org.apache.commons/commons-compress/pom.xml (pkg:maven/org.apache.commons/commons-compress@1.21, cpe:2.3:a:apache:commons_compress:1.21:*:*:*:*:*:*:*) : CVE-2024-25710, CVE-2024-26308
hadoop-client-runtime-3.3.4.jar/META-INF/maven/org.apache.commons/commons-configuration2/pom.xml (pkg:maven/org.apache.commons/commons-configuration2@2.1.1, cpe:2.3:a:apache:commons_configuration:2.1.1:*:*:*:*:*:*:*) : CVE-2024-29131, CVE-2024-29133
hadoop-client-runtime-3.3.4.jar/META-INF/maven/org.apache.commons/commons-lang3/pom.xml (pkg:maven/org.apache.commons/commons-lang3@3.12.0, cpe:2.3:a:apache:commons_lang:3.12.0:*:*:*:*:*:*:*) : CVE-2025-48924
hadoop-client-runtime-3.3.4.jar/META-INF/maven/org.eclipse.jetty/jetty-io/pom.xml (pkg:maven/org.eclipse.jetty/jetty-io@9.4.43.v20210629, cpe:2.3:a:eclipse:jetty:9.4.43:20210629:*:*:*:*:*:*, cpe:2.3:a:jetty:jetty:9.4.43:20210629:*:*:*:*:*:*) : CVE-2022-2048, CVE-2023-36478, CVE-2023-44487, CVE-2024-22201, CVE-2024-9823, CVE-2024-13009, CVE-2024-8184, CVE-2023-26048, CVE-2023-26049, CVE-2023-40167, CVE-2024-6763, CVE-2023-41900, CVE-2023-36479, CVE-2022-2047
hadoop-client-runtime-3.3.4.jar/META-INF/maven/org.jline/jline/pom.xml (pkg:maven/org.jline/jline@3.9.0, cpe:2.3:a:jline:jline:3.9.0:*:*:*:*:*:*:*) : CVE-2023-50572
hive-exec-2.3.9-core.jar (pkg:maven/org.apache.hive/hive-exec@2.3.9, cpe:2.3:a:apache:hive:2.3.9:*:*:*:*:*:*:*) : CVE-2020-13949, CVE-2021-34538, CVE-2024-23953, CVE-2024-23945, CVE-2024-29869
hive-storage-api-2.8.1.jar (pkg:maven/org.apache.hive/hive-storage-api@2.8.1, cpe:2.3:a:apache:hive:2.8.1:*:*:*:*:*:*:*) : CVE-2021-34538, CVE-2024-23953, CVE-2024-23945, CVE-2024-29869
ivy-2.5.1.jar (pkg:maven/org.apache.ivy/ivy@2.5.1, cpe:2.3:a:apache:ivy:2.5.1:*:*:*:*:*:*:*) : CVE-2022-46751
jackson-databind-2.15.2.jar (pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.15.2, cpe:2.3:a:fasterxml:jackson-databind:2.15.2:*:*:*:*:*:*:*, cpe:2.3:a:fasterxml:jackson-modules-java8:2.15.2:*:*:*:*:*:*:*) : CVE-2023-35116
jackson-mapper-asl-1.9.13.jar (pkg:maven/org.codehaus.jackson/jackson-mapper-asl@1.9.13, cpe:2.3:a:fasterxml:jackson-mapper-asl:1.9.13:*:
There was a problem hiding this comment.
Does this dependency-check-report.html still get generated with the same filename? Do you know what consumes it?
There was a problem hiding this comment.
Nothing consumes it. It is only a tool to provide an adhoc audit
|
I confirmed the existing command is broken. With this patch I get a very slow download and a warning that we should use an NVD API key to speed it up. @edwardcapriolo do you know if this is worthwhile? I can help do it as a separate PR if so. |
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @edwardcapriolo for the patch.
-
The bug description seems wrong: "The owasp plugin doesn't specify a version and it does not run.", since it does specify one:
Lines 525 to 529 in 2ecd622
Line 121 in 2ecd622
-
The PR description mentions the command to run the plugin, which is great. To make review easier, it would be nice to also include an excerpt of the output before and after this change, like:
Before:
$ mvn org.owasp:dependency-check-maven:check ... [INFO] --- dependency-check:7.1.1:check (default-cli) @ hadoop-main --- [INFO] Checking for updates [ERROR] Error retrieving https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta; received response code 403; ForbiddenAfter:
$ mvn org.owasp:dependency-check-maven:check ... [INFO] --- dependency-check:12.1.9:check (default-cli) @ hadoop-main --- [INFO] Checking for updates [WARNING] An NVD API Key was not provided - it is highly recommended to use an NVD API key as the update can take a VERY long time without an API Key [INFO] NVD API has 341,897 records in this update [INFO] Downloaded 10,000/341,897 (3%) ...I haven't waited for it to complete, but I guess we can say that the situation has improved.
-
There are a few changes (see inline comments) which are not clearly required for the fix. It would be nice to explain why those changes are being made.
pom.xml
Outdated
| <cyclonedx.version>2.9.1</cyclonedx.version> | ||
| <docker-maven-plugin.version>0.29.0</docker-maven-plugin.version> | ||
|
|
||
| <owasp-maven-plugin.version>12.1.9</owasp-maven-plugin.version> |
There was a problem hiding this comment.
Instead of adding a new property, please update the existing one: dependency-check-maven.version. Not only does this reduce change, it may better match existing naming convention.
On a related note, a newer version of the plugin is already available: https://github.com/dependency-check/DependencyCheck/releases/tag/v12.2.0
| <configuration> | ||
| <assemblyAnalyzerEnabled>false</assemblyAnalyzerEnabled> | ||
| <ossindexAnalyzerEnabled>false</ossindexAnalyzerEnabled> | ||
| </configuration> |
There was a problem hiding this comment.
Can you please explain
- why these are being disabled,
- whether this change is necessary for the fix,
- what about dozens of other analyzers?
assemblyAnalyzerEnabled is related to .NET. The plugin's doc says:
specific analyzers will automatically disable themselves if no file types that they support are detected
so it seems to be unnecessary to explicitly set it.
There was a problem hiding this comment.
ossindexAnalyzerEnabled this one requires an extra subscription/login to a sontype service. If you dont disable it your get error messages.
There was a problem hiding this comment.
{quote}
so it seems to be unnecessary to explicitly set it.
{quote}
It isnt exactly true if it finds a file with a .net extension it may be temped to scan it and look for .net stuff and its better to explicitly disable things.
| <groupId>org.owasp</groupId> | ||
| <artifactId>dependency-check-maven</artifactId> | ||
| <version>${dependency-check-maven.version}</version> | ||
| </plugin> |
There was a problem hiding this comment.
Can you please explain why this is being removed?
There was a problem hiding this comment.
Its moved. Not removed
ajfabbri
left a comment
ajfabbri
left a comment
There was a problem hiding this comment.
Two major issues here:
- Command fails for me. After just over an hour of execution time:
:~/hadoop$ mvn org.owasp:dependency-check-maven:check
...
[ERROR] Failed to execute goal org.owasp:dependency-check-maven:12.1.9:check (default-cli) on project hadoop-yarn-applications-catalog-webapp: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR] InitializationException: Unable to read yarn audit output.
[ERROR] caused by IOException: Cannot run program "yarn": error=2, No such file or directory
[ERROR] caused by IOException: error=2, No such file or directory
[ERROR] -> [Help 1]
- Diff can be minimized. As @adoroszlai mentioned, please minimize the number of lines of change to help with backports, etc.
Do you know if this command is only run manually, or if it is part of CI / branch testing? I'd be concerned about the glacial execution time if it is part of CI. As you know, that needs to be faster, not slower.
|
@ajfabbri the plugin takes a long time the first time
You got the YARN error because it does try to analyalize NODEjs and you do not have node js (YARN) in your path. Not the hadoop YARN :) This isn't part of the build process. it is just a tool people used to see OSS issues. As you can see there are many and they arent false positives. So the tool gives you a laundry list of things to update |
|
...
@edwardcapriolo thanks for the background. Mind if I push a commit here which minimizes the diff? Worth doing for cleaner backports. |
|
New pr here: #8404 |
Thanks for minimizing the patch. Please update this PR instead of opening a new one, to keep discussion and iterations in one place. |
edwardcapriolo
force-pushed
the
HADOOP-19786
branch
from
0170ea0 to
76390f6
Compare
|
Done again |
adoroszlai
left a comment
adoroszlai
left a comment
There was a problem hiding this comment.
Thanks @edwardcapriolo for updating the patch, LGTM.
4 participants
Description of PR
Owasp plugin doesn't work
How was this patch tested?
:~/hadoop$ mvn org.owasp:dependency-check-maven:check
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
No ai was used
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html