How to print C++ wstring UTF-8 characters to Mac OS or Unix terminal?

Question

How can I print a std::wstring using std::wcout?

I tried the following, which was recommended here, but it works only for printing this ¡Hola! but not this 日本:

#include <iostream>
#include <clocale>

int main(int argc, char* argv[])
{
  char* locale = setlocale(LC_ALL, ""); 
  std::cout << "locale: " << locale << std::endl; // "C" for me
  std::locale lollocale(locale);
  setlocale(LC_ALL, locale); 
  std::wcout.imbue(lollocale);
  std::wcout << L"¡Hola!" << std::endl; // ok
  std::wcout << L"日本" << std::endl;    // empty :(
  return 0;
}

Also the following (which was recommended here) does not print the Japanese characters at all:

#include <stdio.h>
#include <string>
#include <locale>
#include <iostream>

using namespace std;

int main()
{

        std::locale::global(std::locale(""));
        wstring japan = L"日本";
        wstring message = L"Welcome! Japan is ";

        message += japan;

        wprintf(message.c_str());
        wcout << message << endl;
}

All this is on Mac OS 10.6.8. using g++ 4.2.1, using Terminal 2.1.2.

The terminal can display the characters just fine in general, e.g., when I cat the source code. Also, this command works fine cout << "日本" << std::endl;, but I do need to print wstring.

My $LANG is this:

$ echo $LANG 
en_US.UTF-8

Answer 1

The way you print wstring is by converting it to a UTF-8 char based string. Seriously wchar_t is pointless outside of Windows or one of the various other platform libraries that unfortunately adopted use of wchar_t before it became clear what a bad idea it is.

// move to clang and libc++ then
#include <codecvt>
#include <locale>

int main(){
    std::wstring_convert<std::codecvt_utf8<wchar_t>,wchar_t> convert; // converts between UTF-8 and UCS-4 (given sizeof(wchar_t)==4)
    std:wstring s = L"日本";
    std::cout << convert.to_bytes(s);
}

---

And just to explain what's going wrong in the code you show;

char* locale = setlocale(LC_ALL, ""); 
std::cout << "locale: " << locale << std::endl; // "C" for me

The locale string here is the locale name after applying changes. Since you say you get "C" it means you're using the "C" locale. Normally one would get a name like "en_US.UTF-8" but for whatever reason your environment isn't set up correctly for that. You show that $LANG is set correctly but perhaps one of the other locale environment variables is set differently.

In any case you're using the "C" locale, which is only required to support the basic character set. I believe on OS X the behavior you'll get is that any char will directly convert to the same wchar_t value, and only wchar_t values in the range supported by char will convert back. That's effectively the same as using an ISO 8859-1 based locale, so Japanese characters will not work.

---

If you really insist on getting this locale based stuff to work then you need to get an appropriate locale, one that uses UTF-8. You can either figure out what's wrong with your environment or you can use a non-portable, explicit locale name.

std::wcout.imbue(std::locale("en_US.UTF-8"));
std::wcout << L"¡Hola!\n";
std::wcout << L"日本\n";

Also, if you're using libstdc++ you should know that it doesn't support locales properly on OS X. You'll have to use libc++ in order for OS X's locale names (e.g., "en_US.UTF-8") to work.