Anomalous 'enc' parameter (value 2) passed to Libcrypto EVP_CipherInit_ex in 1.27.4 image

[abwesend890]

Bug Overview

Description:
Dynamic tracing reveals that the official Nginx docker image is passing an anomalous parameter to the OpenSSL EVP_CipherInit_ex function. The enc parameter is being set to 2.

Expected Behavior

According to the official OpenSSL manual, the enc parameter should only accept 1 (encrypt), 0 (decrypt), or -1 (leave unchanged). While OpenSSL currently tolerates this due to permissive internal checks, this is undocumented behavior and could break in future cryptographic updates.

Steps to Reproduce the Bug

Reproduction Environment:

• Nginx Docker Tag: 1.27.4
• Image ID: 97662d24417b

Environment Details

> uname -a
Linux x1-c 6.8.0-111-generic #111~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 14 17:13:45 UTC  x86_64 x86_64 x86_64 GNU/Linux
> docker compose version
Docker Compose version v5.1.3

Additional Context

Additional Context:
This behavior appears specifically in pre-built configurations like the Docker image mentioned above. When OpenSSL and Nginx are built manually from source, this anomalous parameter passing disappears. This suggests the issue might be tied to how the libraries are compiled or linked in the official image environment.

Could the maintainers investigate what is causing Nginx (or its included modules) to pass 2 to the cipher initialization context in this specific build?

Related Issue

openssl/openssl#31357

[oxpa]

First of all, I doubt this is a docker image problem. We install prebuilt packages into images and if there is a problem it's in nginx itself, not docker. And a rebuild should do nothing for a problem like this.

Secondly, I glanced at the code: nginx itself calls EVP_CipherInit_ex three ways:

1. with enc == 0
2. with enc == 1
3. and with enc coming from OpenSSL itself (as a function can be used as a callback).

It would be nice, if possible, to have a stacktrace or similar information to figure out how a call is made. Or a better description how to reproduce the issue. I can patch nginx to log the value of enc and do stack traces but I have no idea what configuration and client to use.

[abwesend890]

hey,

also not sure, that is why i also opened here openssl/openssl#31357

still, here is the configuration on which I can observe the behavior.

user  nginx;
worker_processes 1;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    
    server {
        listen 443 ssl;

        server_name docker.compose.local;
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }


        ssl_certificate /etc/nginx/certs/webserver.pem;
        ssl_certificate_key /etc/nginx/certs/webserver-key.pem;
	    ssl_protocols TLSv1.3;

        ssl_ecdh_curve secp521r1:secp384r1:prime256v1;
        ssl_dhparam /etc/ssl/dhparams.pem;
	    ssl_session_tickets off;
    
    }


}

Retrieving a stacktrace is non-trivial unfortunately (Frida instrumentation reported this issue to me);
Thus, using log statements is likely the least effort to get around this issue.

As client, I just used curl:

curl --connect-timeout 3600 -v https://localhost:4443/hello_world.html

[thresheek]

Does it also happen with more recent images, e.g. nginx:1.31.1 ?

[abwesend890]

no, with nginx:1.31.1, I can not observe this issue anymore...

guess it can be closed then?

[thresheek]

I guess. It's up to you to figure out if that was a quirk on that particular nginx/openssl verison combo (or rather even openssl version). Not sure we can be of help here.