Is it possible to check string encoding?

Question

In my application I import some text into database from files that users upload on site. Database SQL Server 2005, text is stored nvarchar column, I use EF and L2SQL.

Users should make their files with UTF-8 but unfortunately some of them apparently used different encoding. In result some characters are invalid.

I'd like to find which records are valid. I use utf8checker. It works fine with original files, but when text is from database IsUtf8 method always returns true.

Answer 1

I think SQL server will always store Unicode as UCS-2. So, you need to ensure that the data has the correct encoding at insert time rather than read time. Otherwise SQL server will garble it for you and I don't think there is a way to determine the original encoding after the data has been inserted - unless maybe you have the encoding definition in the record itself, like another column or the first few characters of your data element. Eg. - XML does it this way.

Hope this helps.