DSA Signature sign/verify using JCE/BouncyCastle/libgcrypt

In my previous update I described various interoperability problems in the signing/verification process.

- Problem 1: signing with standard JCE and verifying with BouncyCastle (and vica versa)

The answer to this problem is that... the BouncyCastle DSA code does not perform the SHA1 hash of the message; they expect you to do that and pass the 20 byte hash in rather than the message. The JCE code automatically does the hashing for you; you just pass it the message you want to get signed, as ghstark and Werner mention here and here accordingly.

- Problem 2: signing with BouncyCastle and verifying with libgcrypt (and vica versa)

This problem is slightly more complicated and in order to understand what's going on I had to debug SIP Communicator/otr4j/BC & Pidgin/libotr/libgcrypt simultaneously. Fortunately we got to an end.

The BC DSA signature algorithm implements the latest standard according to FIPS-186-3 while libgcrypt adheres to FIPS-186-1, FIPS-186-2.

According to FIPS-186-3/p. DSA Signature Generation, (r,s) should be generated as follows:

Let N be the bit length of q. Let min(N, outlen) denote the minimum of the positive integers N and outlen, where outlen is the bit length of the hash function output block. The signature of a message M consists of the pair of numbers r and s that is computed according to the following equations:

r = (g^k mod p) mod q.
z = the leftmost min(N, outlen) bits of Hash(M).
s = (k^-1 (z + xr)) mod q.

while according to FIPS-186-1, FIPS-186-2
z = Hash(M)

Full topic discussion is available here.

Finally, Ian Goldberg in this thread provided a solution to this problem to pass to the signature generator (Hash(M) mod q) rather than truncating the Hash(M) value.

Note that this solution will not work with standard JCE because it requires a DSA signer implementation that doesn't hash it's input...