Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings

Question

Weirdest error, who’s up for a challenge to try and help me? Spent hours yesterday on it and it’s like magic. I can't believe this is happening and it's driving me crazy.

Btw, using:

• Rails 4.2.3
• Ruby 2.2.3
• pg 0.18
• Encoding.default_internal = Encoding.default_external = UTF-8
• config.encoding = 'utf-8'

It has to do with Postgres, ActiveRecord and Encodings! Turns out that we’ve been getting these errors whenever we create new Users on the database with special characters:

Encoding::UndefinedConversionError: "\xC3" from ASCII-8BIT to UTF-8

\xC3 may vary depending on the characters. The weird part is that we’ve set since the beginning EVERYTHING to UTF-8. So it really makes no sense and I digged into it, and used this script:

User.all.each do
  |user| user.attributes.each do
    |name, value| if value.is_a? String
      puts user.email + name.encoding.to_s + value.encoding.to_s
    end
  end
end

And this is the output of one of the users, but all users get the same:

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

Turns out that only 3 fields get the ASCII-8BIT encoding. Which absolutely makes no sense to me!

(Those fields btw, first_name, middle_name, last_name, are exactly the same in the database as the other text fields that got listed on the output, they don't get any special treatment).

Another funny thing is that if I don't fill any of those fields, the encoding is UTF-8 for the value. But once I fill it, even if I don't use special characters it will get converted to ASCII-8BIT.

Any suggestions? I've tried every single thing.

Thank you.

Answer 1

Okay, so in my case it had to do with the gem crypt_keeper. If you are using special Unicode characters such as ´, you have to explicitly tell the gem to use UTF-8 to encode, otherwise it will store the values on your database as ASCII-8BIT.

Wish I got a better error message to know it was caused because of that, hopefully someone with the same problem will see this someday!