Fix DTLS 1.2 signature hash mismatch for P-384 keys. Issue #96

[tyhik]

The bug manifests only on 1.2 and only on the server side. It is exposed when providing to dimpl an ECDSA P-384 cert/key . Here's what happens:

1. Server has a P-384 key
2. select_ske_signature_algorithm() negotiates SHA-256 + ECDSA
3. Server writes {hash: SHA256, sig: ECDSA} into the signed header of the ServerKeyExchange message
4. Server calls sign(data) on its P-384 key using SHA-384 instead of the negotiated SHA-256 [BUG]
5. Client reads the header, sees "SHA-256 + ECDSA", so it computes SHA-256(data), but verification of course fails.

Looks like there are two possible solutions: either sign with the negotiated SHA-256 in step 4 or negotiate the hash based on the provided cert, SHA-384 as per P-384, steps 2,3. The PR implements the latter.

[algesten]

@HMBSbige does this look correct?

[xnorpx]

@tyhik "cargo fmt --all" to fix the lint issue.

[HMBSbige]

From reviewing the PR, this changes the selection preference in SKE negotiation, but it does not address the signing path itself. It should improve the common case, but it still looks incomplete to me. I also do not see a regression test for the actual failing scenario here, namely a DTLS 1.2 server using a P-384 cert/key. As written, this reads more like a mitigation than a complete fix.

More importantly, I think the protocol model here is different from what TLS/DTLS 1.2 requires. In TLS/DTLS 1.2, a P-384 ECDSA key is not inherently tied to SHA-384. What matters is that the SignatureAndHashAlgorithm advertised on the wire matches the hash actually used for signing. So the real issue is not that “P-384 should negotiate SHA-384”, but that negotiation and signing can diverge.

If we want to handle this correctly end to end, backend support is the real constraint. RustCrypto can do that, since it can sign with a P-384 key over a SHA-256 prehash. aws-lc-rs currently cannot: its ECDSA signing API is tied to the keypair’s configured digest, and it does not expose a P-384/SHA-256 signing algorithm.

[HMBSbige]

At a higher level, I think this should be capability-aware negotiation.

Different crypto backends expose different signing combinations for a given key, so the server should choose the DTLS 1.2 SignatureAndHashAlgorithm from the intersection of (1) the client's advertised list and (2) what the configured backend can actually sign with the loaded key.

If there is no overlap, the handshake should fail locally before emitting ServerKeyExchange, rather than negotiating one pair and producing a signature with another.

[tyhik]

Reworked to the best of my understanding. Added the test with P-384 certs that without the fix fails for both rust-crypto and aws-lc-rs backends.