Re: UTF-16 support?

[Kai Willadsen <kai willadsen gmail com>]

On 11 April 2013 01:00, Nick <nospam codesniffer com> wrote:
> On Sun, 2013-04-07 at 00:56 -0400, Nick wrote:
> > > I'd
> > > certainly be interested to know whether you can reproduce with the
> > > current git version (it's easy to try! clone and run from the
> > > directory).
> >
> > I'll look into installing git and giving it a try.  I can also try
> > applying my patch to another repo of mine to verify it works as
> > intended.
>
> So I applied my patch to another repo and it didn't repro because the
> resulting state was not the same (BOM & content was not preserved in the
> files).
>
> I created a script to re-create the environment (attached).
>
> The good news is that I tested with the latest version in Git and it
> works fine!
>
>
> > > I'll try not to, though I'd still like to know why it's working. Could
> > > I ask you to dump whatever you currently have to hand in that bugzilla
> > > bug, just for future reference?
> >
> > Absolutely.  What would you like me to put in there?  This patch?
> > Files?  Anything else that would help?
>
> Since this is not an issue in the latest Git, do you still want me to
> add content to the bug?  If so, give me a list of the things that will
> be helpful.

Apologies for leaving this so long. No, that's all good. Thanks very
much for the test case, as it let me actually figure out what was
happening here. I can also confirm that this works for me, so that's
cool.

The short version of why it works is that by the time we get to our
null-byte check, we've already done our UTF-16 decode and no longer
have any such garbage in the output.

We should still probably include an "Open it anyway" prompt for binary
files, but at least UTF-16 is supportable.

cheers,
Kai