valid input encoded as windows-1252 is detected with nil encoding

[coledot]

I have an input file encoded in windows-1252 with the following (anonymized) contents:
1234567890,ASDF,JKL,123 WHEREVER AVE,SOMEWHERE TOWN,UBERLâNDIA,[email protected]
The case has been preserved; note the lowercased accented 'â'.
When running the contents of this file through CharDet.detect, I get back a nil encoding with 0 confidence. Running the same input through uchardet and chardetect (command line wrappers for the C++ and Python implementations of chardet, respectively) both report windows-1252.
I understand mixed case like this might be considered "weird" input, but getting a nil back as the encoding doesn't sound right to me. At the very least, rchardet appears to be doing something inconsistent with the other implementations.

[coledot]

Aha! The problem is here, this line in latin1prober.rb:
confidence = (@freqCounter[3] / total) - (@freqCounter[1] * 20.0 / total)
@freqCounter[3] and total are both Fixnum and will result in an integer division. It's supposed to be computing a float value. @freqCounter[3].to_f should fix that up nicely.
Compare this behavior to uchardet; the equivalent calculation there uses a cast to float: mFreqCounter[3]*1.0f / total

[coledot]

Fixed in this pull request: #18 (and added a spec)

[coledot]

@grosser would love to see this merged in, if possible :)

[grosser]

taking discussion to the PR