Skip to content

[fix] match BCJSSE key types when choosing the TLS credential - #384

Closed
toabctl wants to merge 2 commits into
jruby:masterfrom
toabctl:ec-keytype-chooseAlias
Closed

toabctl wants to merge 2 commits into
jruby:masterfrom
toabctl:ec-keytype-chooseAlias

Conversation

@toabctl

@toabctl toabctl commented Oct 8, 2026

Copy link
Copy Markdown

Fixes #383.

With an explicit TLS version (SSLContext.new(:TLSv1_2) / ssl_version=), and always in the FIPS variant, SSLContext uses BCJSSE, which asks the key manager for ECDHE_ECDSA / ECDHE_RSA / DHE_DSS (TLS 1.2), EC/<group> (TLS 1.3) or Ed25519 / Ed448. chooseAlias compared those against the plain key type, so EC and Ed25519 server keys never got a credential. This maps the requested key types to the credential's key type and, for EC/<group>, checks the key's curve.

Adds test_server_key_types_with_explicit_tls_version (P-256, P-384, Ed25519 on TLS 1.2 and 1.3): it fails on master and passes with the change; test/ssl/test_ssl.rb passes otherwise unchanged (JRuby 9.4.15.0, OpenJDK 21).

🤖 Generated with Claude Code

With an explicit TLS version (e.g. SSLContext.new(:TLSv1_2)), and
always in the FIPS variant, SSLContext uses BCJSSE. BCJSSE asks the key
manager for "ECDHE_ECDSA" / "ECDHE_RSA" / "DHE_DSS" (TLS 1.2),
"EC/<group>" (TLS 1.3) or "Ed25519" / "Ed448", but chooseAlias compared
against the plain key type ("EC", "RSA", "EdDSA"). EC and Ed25519
server keys then got no credential and every handshake failed with
"found no selectable cipher suite".

Map these key types to the credential's key type, and for "EC/<group>"
also check that the key is on that curve.

Fixes jruby#383

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
BCJSSE in FIPS mode does not offer the ed25519 signature scheme, so an
Ed25519 server key cannot be selected there regardless of key-type
matching.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@kares

kares commented Oct 27, 2026

Copy link
Copy Markdown
Member

Thanks Thomas, we've built upon this approach (translating key-type names) and landed #394; differences:

  • Curve check: the curve comes from the certificate's public key, not the private key.
    Opaque private keys (PKCS#11, some FIPS setups) aren't ECKeys, so this PR would not match those.
  • Ed25519 vs Ed448: matched on the certificate's key algorithm. This approach accepts any key whose private key reports "EdDSA" for either request, which can't tell Ed25519 and Ed448 apart.
  • SunJSSE: not implemented here - the Ed25519 test fails when the suite runs on SunJSSE and plain SSLContext.new stays broken for Ed25519.
  • Coverage: additional tests for add_certificate as well as cert=/key=, RSA vs EC selection by cipher, EC client certificates over TLS 1.3, and Ed25519 on a default context (server and client cert, checking cert/peer_cert public keys).

Also tried handing selection over to the provider's own key manager (an in-memory KeyStore + KeyManagerFactory("PKIX")), which avoids translating names. Something worth reconsidering later.

@kares kares closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TLS server with an EC key fails on 0.19.x (chooseAlias doesn't match BCJSSE key types)

2 participants