Fix GnuTLS renegotiation failure on GNUTLS_E_GOT_APPLICATION_DATA

[nanangizz]

Summary

• Handle GNUTLS_E_GOT_APPLICATION_DATA (-38) as non-fatal in ssl_do_handshake(), returning PJ_EPENDING so renegotiation retries after draining buffered application data
• Add write_mutex around gnutls_rehandshake() for session access consistency with ssl_do_handshake() and ssl_read()

Background

During server-initiated TLS renegotiation under load, the client's gnutls_handshake() may encounter application data records (echoed by the server) instead of the expected handshake records, returning GNUTLS_E_GOT_APPLICATION_DATA (-38). On some GnuTLS builds (notably macOS Homebrew), gnutls_error_is_fatal() classifies this as fatal, causing ssl_do_handshake() to return PJ_EINVAL and abort the renegotiation.

The fix treats this as non-fatal (PJ_EPENDING). The buffered application data is consumed by subsequent gnutls_record_recv() calls in the normal read path, and the handshake retries incrementally until completion.

CI failures: This has been failing consistently on the GnuTLS / pjlib,util,pjmedia,pjnath job across multiple PRs since the ssl_sock_stress_test renegotiation subtest was introduced:

• https://github.com/pjsip/pjproject/actions/runs/24388112149/job/71226856744
• https://github.com/pjsip/pjproject/actions/runs/24391009862/job/71236772876
• https://github.com/pjsip/pjproject/actions/runs/24392970976/job/71243572307

Test plan

• Reproduced locally with GnuTLS 3.8.3 on Linux (ssl_sock_stress_test -w 2): fails on "send load + server renegotiation test" with identical error
• Applied fix: ssl_sock_stress_test passes 5/5 runs
• ssl_sock_test passes (no regression)
• CI: GnuTLS job should now pass the renegotiation subtest

Co-Authored-By: Claude Code

[Copilot]

Pull request overview

Fixes a GnuTLS renegotiation edge case where gnutls_handshake() can return GNUTLS_E_GOT_APPLICATION_DATA during server-initiated renegotiation, and ensures session access is synchronized consistently during renegotiation initiation.

Changes:

• Treat GNUTLS_E_GOT_APPLICATION_DATA from gnutls_handshake() as non-fatal by returning PJ_EPENDING to allow renegotiation to continue after buffered app data is drained.
• Add write_mutex protection around gnutls_rehandshake() to align session access synchronization with existing handshake/read paths.