Unicode handling in `to_latex`. Needs encoding?

[jseabold]

I can't seem to get this one to work and to_latex doesn't allow a user-specified encoding. I think this might need a look.

I have it in unicode, so try that way.

pd.DataFrame([[u'au\xdfgangen']]).to_latex('test.tex')

Nope. Ok, so let's encode it as a utf-8 string

pd.DataFrame([[u'au\xdfgangen']]).apply(lambda x : x.str.encode('utf-8')).to_latex('test.tex')

Nope. It looks like it's getting coerced back to unicode in formatter._to_str_columns() then tries to write it as ASCII...

[jseabold]

Can pass a StringIO instance to buf, then encode and write this yourself as a workaround.

[TomAugspurger]

Works correctly in python 3 as well.

I've got a fix that seems to work for python 2. Changing

            with open(self.buf, 'w') as f:
                write(f, frame, column_format, strcols, longtable)

to

            import codecs
            with codecs.open(self.buf, 'wb', encoding=encoding) as f:
                write(f, frame, column_format, strcols, longtable)

along with adding an encoding kwarg to to_latex (default to utf-8?). I haven't done much with unicode, so I'm still reading about it. Let me know if this seems wrong to you, or if it needs to be done elsewhere.

[jseabold]

Yes, I'm slowly trying to move to python 3 partially for this reason.

That seems reasonable to me. I assumed that the other functions just encoded the unicode/string according to the given encoding, but I'm not sure.

The default in to_csv and friends is encoding=None. I assume it falls back to the default encoding for the locale, but I'm not positive on that. I started to check but I'm under a deadline right now.

[nbonnotte]

I just encountered the same problem with pandas 0.17, so I guess the fix has not been included?

@TomAugspurger do you intent on making a PR?

[TomAugspurger]

I never got around to submitting a pull request. Feel free to do so if you want! My fix above might work (would need to be tested), but it might be better to tie this in with how to_csv handles encodings (not sure, haven't looked).