Cloudstack is not FIPS compliant

[rwdj]

ISSUE TYPE

• Bug Report

COMPONENT NAME

UI, more?

CLOUDSTACK VERSION

4.16

CONFIGURATION

OS / ENVIRONMENT

RHEL 8 FIPS compliant system

SUMMARY

Cloudstack uses multiple instances of non-FIPS-compliant features. This makes it impossible to work with in a FIPS compliant environment.

STEPS TO REPRODUCE

On a FIPS compliant system, build as instructed in https://docs.cloudstack.apache.org/en/latest/installguide/building_from_source.html:

$ mvn -P deps

On a FIPS compliant RHEL8 system, install and run cloudstack-management.service from https://download.cloudstack.org/centos/8/4.16/ as instructed in https://docs.cloudstack.apache.org/en/latest/installguide/overview/index.html.
See #6232 for why I do the chown and chmod steps.

[root@rwdj ~]# systemctl stop firewalld nftables
[root@rwdj ~]# systemctl start iptables
[root@rwdj ~]# dnf install cloudstack-management mysql-connector-java
[root@rwdj ~]# cloudstack-setup-databases cloud:<omitted> --deploy-as=root:<omitted> -i 127.0.0.1
[root@rwdj ~]# chown :cloud /etc/cloudstack/management/key
[root@rwdj ~]# chmod 0640 /etc/cloudstack/management/key
[root@rwdj ~]# cloudstack-setup-management

EXPECTED RESULTS

Able to build or run cloudstack-management.

ACTUAL RESULTS

Build

$ mvn -P deps

[INFO] Running com.cloud.utils.testcase.NioTest                                                                                                                     
2022-04-11 13:55:28,671 INFO  [utils.testcase.NioTest] (main:) Setting up Benchmark Test                                                                            
2022-04-11 13:55:28,695 INFO  [utils.nio.NioServer] (main:) NioServer started and listening on /0:0:0:0:0:0:0:0:41907                                               
2022-04-11 13:55:28,714 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:28,720 INFO  [utils.testcase.NioTest] (MaliciousNioClientHandler-2:) Connecting to 127.0.0.1:41907                                                 
2022-04-11 13:55:28,720 INFO  [utils.nio.NioClient] (NioClientHandler-1:) Connecting to 127.0.0.1:41907                                                             
2022-04-11 13:55:28,720 INFO  [utils.nio.NioClient] (NioClientHandler-2:) Connecting to 127.0.0.1:41907                                                             
2022-04-11 13:55:28,720 INFO  [utils.testcase.NioTest] (MaliciousNioClientHandler-1:) Connecting to 127.0.0.1:41907                                                 
2022-04-11 13:55:29,715 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:30,715 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:31,716 DEBUG [utils.testcase.NioTest] (Time-limited test:) 0/2 tests done. Waiting for completion                                                  
2022-04-11 13:55:32,003 WARN  [utils.nio.Link] (NioClientHandler-2:) Failed to load keystore, using trust all manager                                               
2022-04-11 13:55:32,003 WARN  [utils.nio.Link] (NioClientHandler-1:) Failed to load keystore, using trust all manager                                               
2022-04-11 13:55:32,041 ERROR [utils.nio.Link] (NioTestServer-NioConnectionHandler-1:) CA service is not configured, by-passing CA manager to create SSL engine     
2022-04-11 13:55:32,043 ERROR [utils.nio.NioConnection] (NioClientHandler-1:) Unable to initialize the threads.                                                     
java.io.IOException: Failed to initialise security                                                                                                                  
        at com.cloud.utils.nio.NioClient.init(NioClient.java:82)                                                                                                    
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)                                                                                           
        at com.cloud.utils.testcase.NioTest$ThreadedNioClient.run(NioTest.java:172)                                                                                 
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)                                                                        
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)                                                                                       
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)                                                                
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)                                                                
        at java.base/java.lang.Thread.run(Thread.java:829)                                                                                                          
Caused by: java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                  
        at java.base/sun.security.ssl.SSLContextImpl.chooseTrustManager(SSLContextImpl.java:133)                                                                    
        at java.base/sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:95)                                                                             
        at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:297)                                                                                             
        ... 9 more
[...]
[INFO] Running com.cloud.utils.rest.HttpClientHelperTest                                                                                                            
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.116 s <<< FAILURE! - in com.cloud.utils.rest.HttpClientHelperTest                         
[ERROR] testCreateClient(com.cloud.utils.rest.HttpClientHelperTest)  Time elapsed: 0.115 s  <<< ERROR!                                                              
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                             
        at com.cloud.utils.rest.HttpClientHelperTest.testCreateClient(HttpClientHelperTest.java:33)                                                                 
                                                                                                                                                                    
[INFO] Running com.cloud.utils.rest.BasicRestClientTest                                                                                                             
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.087 s <<< FAILURE! - in com.cloud.utils.rest.BasicRestClientTest                          
[ERROR] com.cloud.utils.rest.BasicRestClientTest  Time elapsed: 0.087 s  <<< ERROR!                                                                                 
java.security.KeyManagementException: FIPS mode: only SunJSSE TrustManagers may be used                                                                             
        at com.cloud.utils.rest.BasicRestClientTest.setupClass(BasicRestClientTest.java:62)

Run.

INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:null) (logid:) Loading module context [system] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstac
k-4.16.1.0.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]                                                                                
WARNING: An illegal reflective access operation has occurred                                                                                                           
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$1 (file:/usr/share/cloudstack-management/lib/cloudstack-4.16.1.0.jar) to method java.lang.ClassLoa
der.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)                                                                                        
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$1                                                                         
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                  
WARNING: All illegal access operations will be denied in a future release                                                                                              
INFO  [c.c.u.d.T.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : false                                                                
WARN  [c.c.u.d.T.Transaction] (main:null) (logid:) Unable to load db configuration, using defaults with 5 connections. Falling back on assumed datasource on localhost:
3306 using username:password=cloud:<omitted>. Please check your configuration                                                                                              
org.jasypt.exceptions.EncryptionInitializationException: java.security.NoSuchAlgorithmException: PBEWithMD5AndDES SecretKeyFactory not available                       
        at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.initialize(StandardPBEByteEncryptor.java:773)                                                            
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.initialize(StandardPBEStringEncryptor.java:566)                                                        
        at org.jasypt.encryption.pbe.StandardPBEStringEncryptor.decrypt(StandardPBEStringEncryptor.java:718)                                                           
        at org.jasypt.properties.PropertyValueEncryptionUtils.decrypt(PropertyValueEncryptionUtils.java:72)                                                            
        at org.jasypt.properties.EncryptableProperties.decode(EncryptableProperties.java:230)                                                                          
        at org.jasypt.properties.EncryptableProperties.getProperty(EncryptableProperties.java:172)                                                                     
        at com.cloud.utils.db.TransactionLegacy.initDataSource(TransactionLegacy.java:1034)                                                                            
        at com.cloud.utils.db.TransactionLegacy.<clinit>(TransactionLegacy.java:1008)                                                                                  
        at com.cloud.utils.db.Merovingian2.<init>(Merovingian2.java:68)                                                                                                
        at com.cloud.utils.db.Merovingian2.createLockController(Merovingian2.java:88)                                                                                  
        at com.cloud.server.LockControllerListener.<init>(LockControllerListener.java:33)
        ... more
Caused by: com.mysql.cj.exceptions.SSLParamsException: Cannot open file:NONE [NONE (No such file or directory)]                                                       
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)                                                                   
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)                                            
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                                    
        ... 90 more
Caused by: java.io.FileNotFoundException: NONE (No such file or directory)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        ... 90 more

Upon checking, I found utils/src/main/java/com/cloud/utils/crypt/DBEncryptionUtil.java#L81, where PBEWithMD5AndDES is not FIPS compliant. I'm concerned there are also other spots to check that I can't check because it doesn't build which brings me to the extent of my knowledge.

FIPS compliance can be a stickler, because the core documentation isn't very clear from what I can find. Regarding PBEWithMD5AndDES, after a few searches, the most clear documentation on approved algorithms I could find was this (which is out of date with FIPS 140-3 existing, but probably still correct enough): https://csrc.nist.gov/csrc/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2261.pdf
And additional resources that may prove useful:
https://github.com/jasypt/jasypt/blob/master/jasypt/src/main/java/org/jasypt/util/text/AES256TextEncryptor.java
https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#cipher-algorithm-names
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_installing-a-rhel-8-system-with-fips-mode-enabled_security-hardening

[weizhouapache]

@rwdj
is this a update of #6232 ?
it looks the db configuration file /etc/cloudstack/management/db.properties is missing.

[rwdj]

@weizhouapache, no, #6232 is mostly unrelated.

/etc/cloudstack/management/db.properties isn't missing. Although I noticed those lines in the error as well. But it also doesn't compile.

[root@rwdj ~]# su - cloud -c 'ls -l /etc/cloudstack/management/db.properties'
-rw-r-----. 1 root cloud 3248 Apr 12 09:53 /etc/cloudstack/management/db.properties

It was generated appropriately from:

[root@rwdj ~]# cloudstack-setup-databases cloud:<omitted> --deploy-as=root:<omitted> -i 127.0.0.1
Mysql user name:cloud                                                           [ OK ]                                                                   
Mysql user password:******                                                      [ OK ]                                                                   
Mysql server ip:localhost                                                       [ OK ]                                                                   
Mysql server port:3306                                                          [ OK ]                                                                   
Mysql root user name:root                                                       [ OK ]                                                                   
Mysql root user password:******                                                 [ OK ]                                                                   
Using specified cluster management server node IP 127.0.0.1                     [ OK ]                                                                   
Checking Cloud database files ...                                               [ OK ]                                                                   
Checking local machine hostname ...                                             [ OK ]                                                                   
Checking SELinux setup ...                                                      [ OK ]                                                                   
Preparing /etc/cloudstack/management/db.properties                              [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-database.sql             [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-schema.sql               [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-database-premium.sql     [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/create-schema-premium.sql       [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/server-setup.sql                [ OK ]                                                                   
Applying /usr/share/cloudstack-management/setup/templates.sql                   [ OK ]                                                                   
Processing encryption ...                                                       [ OK ]                                                                   
Finalizing setup ...                                                            [ OK ]

[rwdj]

A general rule of thumb: if the word "MD5" is ever mentioned at any point, then you'll run into a FIPS issue. FIPS machines aren't able to use MD5 at all. Based on the stacktrace, Java looks to me like it's trying to find files relating to MD5 and they aren't found because Java is running in FIPS-mode. The missing file NONE should be the mentioned "SecretKeyFactory" for PBEWithMD5AndDES, but I'm unaware of the technical details of Java's cipher algorithms.

[rwdj@rwdj ~]$ openssl md5
Error setting digest
140235974117184:error:060800C8:digital envelope routines:EVP_DigestInit_ex:disabled for FIPS:crypto/evp/digest.c:135:

And then there's the issue where it won't compile, flat out refusing on the tests due to FIPS being enabled.

FIPS is a Category 1 (almost no exceptions given) specification of DISA's Security Technical Implementation Guides: https://www.stigviewer.com/stig/red_hat_enterprise_linux_8/2021-12-03/finding/V-230223 or https://stigviewer.com/stig/canonical_ubuntu_20.04_lts/2021-11-19/finding/V-238363

[yadvr]

Thanks for reporting @rwdj, can you advise how to setup a FIPS complaint EL8 mgmt/KVM host? That is what additional steps were taken before you deployed CloudStack on it, maybe there's a common document that describe this process?

[rwdj]

Regarding additional steps, it would be most of the Red Hat Enterprise Linux 8 Security Technical Implementation Guide, but that's probably too much. Checking full STIG compliance for a system takes way too long for this purpose.

The document https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_installing-a-rhel-8-system-with-fips-mode-enabled_security-hardening describes enabling FIPS. In general, it's just making sure the system has fips=1 in the kernel parameters during installation/initialization. And then you can use fips-mode-setup --check to verify it worked (which is not the same way it was done in RHEL7, by the way).

Alternatively, an existing system can be made FIPS compliant with as described here, but it's not recommended by Red Hat. I believe there are a number of things done during initialization that are less complex to do during initialization, but that's only my guess.

[rwdj]

Alternatively, here's documentation for an Ubuntu FIPS environment: https://ubuntu.com/security/certifications/docs/fips-enablement

[rwdj]

Found a page on automated DISA STIG full compliance for Ubuntu Pro (including FIPS): https://ubuntu.com/security/disa-stig