OpenSSL uses DER when encoding DSA signatures. DER is a standardized encoding of structured data that is widely implemented. Integers are among the primitives that DER specifics how to encode. In particular, DER specifies that encoded integers are signed. However, when OpenSSL decodes signatures, it always decodes integers as if they are unsigned. In particular, OpenSSL will decode negative numbers as positive numbers.
OpenSSL claims this is not a problem by noting that only improperly encoded signatures will use negative numbers.
/* Custom primitive type for BIGNUM handling. This reads in an ASN1_INTEGER as a * BIGNUM directly. Currently it ignores the sign which isn't a problem since all * BIGNUMs used are non negative and anything that looks negative is normally due * to an encoding error. */— openssl-1.0.0e/crypto/asn1/x_bignum.c
I am skeptical of their claim that this is not a problem. This means a user, with access to a properly encoded signature, can create an improper DER encoding of a signature using negative numbers that is still a proper DER encoding of a pair of integers and that OpenSSL will validate as a proper signature. This could wreak havoc on any system with two verification engines; one using OpenSSL and another using different software that, correctly, rejects such a signature for using a negative number.
Unfortunately people, even in security, are reactive instead of proactive; I presume that the OpenSSL people will not change anything unless there is an exploit. Your task, should you chose to accept it, is to use this knowledge to find an exploit that uses negative numbers in DSA signatures to exploit or create a denial of service on some system using OpenSSL. Oh, and then tell the OpenSSL people about it.
Good luck.