utf-8 issues in gyb and line-directive

ahti · 2016-01-12T18:12:33.000Z

I’ve tried to compile Swift in a clean Arch Linux chroot. In this environment, `sys.getdefaultencoding()` does not return `utf-8`, which causes gyb and line-directive to fail with non-ascii characters in input files.

One example of such a file is Arrays.swift.gyb, which contains a ≤-character [here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).

In gyb, `argparse.FileType` is used to open input files, and line-directive uses plain old `open(…)`. Sadly, both of those don’t provide functionality to specify the file encoding in Python 2.7.10, which is what OSX ships with.

I’m not that experienced in Python, so I don’t know what the pythonic way to solve this would be, so if anyone with more Python experience would help solve this, that would be great.

— Lukas

rlovelett · 2016-01-12T19:31:06.000Z

I'm thought this was resolved in master already. Are you running the
latest version of master?

See

github.com/apple/swift

[gyb] Ignore filesystem locale and enforce UTF-8

committed 01:39AM - 06 Jan 16 UTC

RLovelett

+2 -2

···

On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:

I’ve tried to compile Swift in a clean Arch Linux chroot. In this
environment, `sys.getdefaultencoding()` does not return `utf-8`, which
causes gyb and line-directive to fail with non-ascii characters in input
files.

One example of such a file is Arrays.swift.gyb, which contains a
≤-character
[here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).

In gyb, `argparse.FileType` is used to open input files, and
line-directive uses plain old `open(…)`. Sadly, both of those don’t
provide functionality to specify the file encoding in Python 2.7.10,
which is what OSX ships with.

I’m not that experienced in Python, so I don’t know what the pythonic way
to solve this would be, so if anyone with more Python experience would
help solve this, that would be great.

— Lukas
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev
Email had 1 attachment:
+ signature.asc
1k (application/pgp-signature)

ahti · 2016-01-12T20:04:18.000Z

This is a separate issue. Here are the backtraces for the two issues:

This is the issue in gyb (note that this occurs here because file is opened lazily. The real issue is a few lines before, where the argparse.FileType is instantiated)

   FAILED: cd /build/swiftc/src/swift/stdlib/public/core && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8 && /build/swiftc/src/swift/utils/gyb --test -DunicodeGraphemeBreakPropertyFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakProperty.txt -DunicodeGraphemeBreakTestFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakTest.txt -DCMAKE_SIZEOF_VOID_P=8 -o /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp Arrays.swift.gyb && /usr/bin/cmake -E copy_if_different /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift && /usr/bin/cmake -E remove /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/gyb", line 3, in <module>
       gyb.main()
     File "/build/swiftc/src/swift/utils/gyb.py", line 1064, in main
       ast = parseTemplate(args.file.name, args.file.read())
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13116: ordinal not in range(128)

After fixing this (in a way incompatible with OSX Python), I get this error in line-directive:

   FAILED: cd /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/linux/x86_64 && /usr/bin/cmake -E make_directory /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/./lib/swift/linux/x86_64 && /build/swiftc/src/swift/utils/line-directive /build/swiftc/src/swift/stdlib/public/core/Algorithm.swift [… more files omitted for brevity …] /build/swiftc/src/swift/stdlib/public/core/VarArgs.swift /build/swiftc/src/swift/stdlib/public/core/Zip.swift /build/swiftc/src/swift/stdlib/public/core/Prespecialized.swift
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/line-directive", line 104, in <module>
       run()
     File "/build/swiftc/src/swift/utils/line-directive", line 92, in run
       file, line_num = map_line(m.group(1), int(m.group(2)))
     File "/build/swiftc/src/swift/utils/line-directive", line 60, in map_line
       map = fline_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 54, in fline_map
       map = _make_line_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 43, in _make_line_map
       for i, l in enumerate(input.readlines()):
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6851: ordinal not in range(128)

— Lukas

···

On 12 Jan 2016, at 20:31, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

I'm thought this was resolved in master already. Are you running the
latest version of master?

See
[gyb] Ignore filesystem locale and enforce UTF-8 · apple/swift@97b98b1 · GitHub

On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:

I’ve tried to compile Swift in a clean Arch Linux chroot. In this
environment, `sys.getdefaultencoding()` does not return `utf-8`, which
causes gyb and line-directive to fail with non-ascii characters in input
files.

One example of such a file is Arrays.swift.gyb, which contains a
≤-character
[here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).

In gyb, `argparse.FileType` is used to open input files, and
line-directive uses plain old `open(…)`. Sadly, both of those don’t
provide functionality to specify the file encoding in Python 2.7.10,
which is what OSX ships with.

I’m not that experienced in Python, so I don’t know what the pythonic way
to solve this would be, so if anyone with more Python experience would
help solve this, that would be great.

— Lukas
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev
Email had 1 attachment:
+ signature.asc
1k (application/pgp-signature)

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

rlovelett · 2016-01-12T23:26:06.000Z

Ok I was able to reproduce this. Basically it comes down to you not
having your locale set.

I've almost got a patch that works for Python 2 and 3.

···

On Tue, Jan 12, 2016, at 03:04 PM, Lukas Stabe wrote:

This is a separate issue. Here are the backtraces for the two issues:

This is the issue in gyb (note that this occurs here because file is
opened lazily. The real issue is a few lines before, where the
argparse.FileType is instantiated)

   FAILED: cd /build/swiftc/src/swift/stdlib/public/core &&
   /usr/bin/cmake -E make_directory
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8
   && /build/swiftc/src/swift/utils/gyb --test
   -DunicodeGraphemeBreakPropertyFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakProperty.txt
   -DunicodeGraphemeBreakTestFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakTest.txt
   -DCMAKE_SIZEOF_VOID_P=8 -o
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
   Arrays.swift.gyb && /usr/bin/cmake -E copy_if_different
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift
   && /usr/bin/cmake -E remove
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/gyb", line 3, in <module>
       gyb.main()
     File "/build/swiftc/src/swift/utils/gyb.py", line 1064, in main
       ast = parseTemplate(args.file.name, args.file.read())
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
   13116: ordinal not in range(128)

After fixing this (in a way incompatible with OSX Python), I get this
error in line-directive:

   FAILED: cd
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core
   && /usr/bin/cmake -E make_directory
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/linux/x86_64
   && /usr/bin/cmake -E make_directory
   /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/./lib/swift/linux/x86_64
   && /build/swiftc/src/swift/utils/line-directive
   /build/swiftc/src/swift/stdlib/public/core/Algorithm.swift [… more
   files omitted for brevity …]
   /build/swiftc/src/swift/stdlib/public/core/VarArgs.swift
   /build/swiftc/src/swift/stdlib/public/core/Zip.swift
   /build/swiftc/src/swift/stdlib/public/core/Prespecialized.swift
   Traceback (most recent call last):
     File "/build/swiftc/src/swift/utils/line-directive", line 104, in
     <module>
       run()
     File "/build/swiftc/src/swift/utils/line-directive", line 92, in run
       file, line_num = map_line(m.group(1), int(m.group(2)))
     File "/build/swiftc/src/swift/utils/line-directive", line 60, in
     map_line
       map = fline_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 54, in
     fline_map
       map = _make_line_map(filename)
     File "/build/swiftc/src/swift/utils/line-directive", line 43, in
     _make_line_map
       for i, l in enumerate(input.readlines()):
     File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
       return codecs.ascii_decode(input, self.errors)[0]
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
   6851: ordinal not in range(128)

— Lukas

> On 12 Jan 2016, at 20:31, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:
>
> I'm thought this was resolved in master already. Are you running the
> latest version of master?
>
> See
> [gyb] Ignore filesystem locale and enforce UTF-8 · apple/swift@97b98b1 · GitHub
>
> On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:
>> I’ve tried to compile Swift in a clean Arch Linux chroot. In this
>> environment, `sys.getdefaultencoding()` does not return `utf-8`, which
>> causes gyb and line-directive to fail with non-ascii characters in input
>> files.
>>
>> One example of such a file is Arrays.swift.gyb, which contains a
>> ≤-character
>> [here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).
>>
>> In gyb, `argparse.FileType` is used to open input files, and
>> line-directive uses plain old `open(…)`. Sadly, both of those don’t
>> provide functionality to specify the file encoding in Python 2.7.10,
>> which is what OSX ships with.
>>
>> I’m not that experienced in Python, so I don’t know what the pythonic way
>> to solve this would be, so if anyone with more Python experience would
>> help solve this, that would be great.
>>
>> — Lukas
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
>> Email had 1 attachment:
>> + signature.asc
>> 1k (application/pgp-signature)
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

Email had 1 attachment:
+ signature.asc
  1k (application/pgp-signature)

rlovelett · 2016-01-13T02:06:52.000Z

Attached is patch
`0001-gyb-Force-UTF-8-encoding-when-parsing-templates-on-L.patch` that
should fix this issue for both Python 2 and 3. I’ve tested it on OS X
and Arch as well.

I’ve submitted the patch as a MR [1].

[1] [gyb] Force UTF-8 encoding when parsing templates on Linux by RLovelett · Pull Request #950 · apple/swift · GitHub

0001-gyb-Force-UTF-8-encoding-when-parsing-templates-on-L.patch (4.65 KB)

···

On Tue, Jan 12, 2016, at 06:26 PM, Ryan Lovelett via swift-dev wrote:

Ok I was able to reproduce this. Basically it comes down to you not
having your locale set.

I've almost got a patch that works for Python 2 and 3.

On Tue, Jan 12, 2016, at 03:04 PM, Lukas Stabe wrote:
> This is a separate issue. Here are the backtraces for the two issues:
>
> This is the issue in gyb (note that this occurs here because file is
> opened lazily. The real issue is a few lines before, where the
> argparse.FileType is instantiated)
>
> FAILED: cd /build/swiftc/src/swift/stdlib/public/core &&
> /usr/bin/cmake -E make_directory
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8
> && /build/swiftc/src/swift/utils/gyb --test
> -DunicodeGraphemeBreakPropertyFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakProperty.txt
> -DunicodeGraphemeBreakTestFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakTest.txt
> -DCMAKE_SIZEOF_VOID_P=8 -o
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
> Arrays.swift.gyb && /usr/bin/cmake -E copy_if_different
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift
> && /usr/bin/cmake -E remove
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
> Traceback (most recent call last):
> File "/build/swiftc/src/swift/utils/gyb", line 3, in <module>
> gyb.main()
> File "/build/swiftc/src/swift/utils/gyb.py", line 1064, in main
> ast = parseTemplate(args.file.name, args.file.read())
> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 13116: ordinal not in range(128)
>
> After fixing this (in a way incompatible with OSX Python), I get this
> error in line-directive:
>
> FAILED: cd
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core
> && /usr/bin/cmake -E make_directory
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/linux/x86_64
> && /usr/bin/cmake -E make_directory
> /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/./lib/swift/linux/x86_64
> && /build/swiftc/src/swift/utils/line-directive
> /build/swiftc/src/swift/stdlib/public/core/Algorithm.swift [… more
> files omitted for brevity …]
> /build/swiftc/src/swift/stdlib/public/core/VarArgs.swift
> /build/swiftc/src/swift/stdlib/public/core/Zip.swift
> /build/swiftc/src/swift/stdlib/public/core/Prespecialized.swift
> Traceback (most recent call last):
> File "/build/swiftc/src/swift/utils/line-directive", line 104, in
> <module>
> run()
> File "/build/swiftc/src/swift/utils/line-directive", line 92, in run
> file, line_num = map_line(m.group(1), int(m.group(2)))
> File "/build/swiftc/src/swift/utils/line-directive", line 60, in
> map_line
> map = fline_map(filename)
> File "/build/swiftc/src/swift/utils/line-directive", line 54, in
> fline_map
> map = _make_line_map(filename)
> File "/build/swiftc/src/swift/utils/line-directive", line 43, in
> _make_line_map
> for i, l in enumerate(input.readlines()):
> File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
> 6851: ordinal not in range(128)
>
> — Lukas
>
> > On 12 Jan 2016, at 20:31, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:
> >
> > I'm thought this was resolved in master already. Are you running the
> > latest version of master?
> >
> > See
> > [gyb] Ignore filesystem locale and enforce UTF-8 · apple/swift@97b98b1 · GitHub
> >
> > On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:
> >> I’ve tried to compile Swift in a clean Arch Linux chroot. In this
> >> environment, `sys.getdefaultencoding()` does not return `utf-8`, which
> >> causes gyb and line-directive to fail with non-ascii characters in input
> >> files.
> >>
> >> One example of such a file is Arrays.swift.gyb, which contains a
> >> ≤-character
> >> [here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).
> >>
> >> In gyb, `argparse.FileType` is used to open input files, and
> >> line-directive uses plain old `open(…)`. Sadly, both of those don’t
> >> provide functionality to specify the file encoding in Python 2.7.10,
> >> which is what OSX ships with.
> >>
> >> I’m not that experienced in Python, so I don’t know what the pythonic way
> >> to solve this would be, so if anyone with more Python experience would
> >> help solve this, that would be great.
> >>
> >> — Lukas
> >> _______________________________________________
> >> swift-dev mailing list
> >> swift-dev@swift.org
> >> https://lists.swift.org/mailman/listinfo/swift-dev
> >> Email had 1 attachment:
> >> + signature.asc
> >> 1k (application/pgp-signature)
> > _______________________________________________
> > swift-dev mailing list
> > swift-dev@swift.org
> > https://lists.swift.org/mailman/listinfo/swift-dev
>
> Email had 1 attachment:
> + signature.asc
> 1k (application/pgp-signature)
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

tkremenek · 2016-01-13T05:56:36.000Z

Merged!

···

On Jan 12, 2016, at 6:06 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

Attached is patch
`0001-gyb-Force-UTF-8-encoding-when-parsing-templates-on-L.patch` that
should fix this issue for both Python 2 and 3. I’ve tested it on OS X
and Arch as well.

I’ve submitted the patch as a MR [1].

[1] [gyb] Force UTF-8 encoding when parsing templates on Linux by RLovelett · Pull Request #950 · apple/swift · GitHub

On Tue, Jan 12, 2016, at 06:26 PM, Ryan Lovelett via swift-dev wrote:
Ok I was able to reproduce this. Basically it comes down to you not
having your locale set.

I've almost got a patch that works for Python 2 and 3.

On Tue, Jan 12, 2016, at 03:04 PM, Lukas Stabe wrote:
This is a separate issue. Here are the backtraces for the two issues:

This is the issue in gyb (note that this occurs here because file is
opened lazily. The real issue is a few lines before, where the
argparse.FileType is instantiated)

  FAILED: cd /build/swiftc/src/swift/stdlib/public/core &&
  /usr/bin/cmake -E make_directory
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8
  && /build/swiftc/src/swift/utils/gyb --test
  -DunicodeGraphemeBreakPropertyFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakProperty.txt
  -DunicodeGraphemeBreakTestFile=/build/swiftc/src/swift/utils/UnicodeData/GraphemeBreakTest.txt
  -DCMAKE_SIZEOF_VOID_P=8 -o
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
  Arrays.swift.gyb && /usr/bin/cmake -E copy_if_different
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift
  && /usr/bin/cmake -E remove
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/8/Arrays.swift.tmp
  Traceback (most recent call last):
    File "/build/swiftc/src/swift/utils/gyb", line 3, in <module>
      gyb.main()
    File "/build/swiftc/src/swift/utils/gyb.py", line 1064, in main
      ast = parseTemplate(args.file.name, args.file.read())
    File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
  13116: ordinal not in range(128)

After fixing this (in a way incompatible with OSX Python), I get this
error in line-directive:

  FAILED: cd
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core
  && /usr/bin/cmake -E make_directory
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/stdlib/public/core/linux/x86_64
  && /usr/bin/cmake -E make_directory
  /build/swiftc/src/build/buildbot_linux/swift-linux-x86_64/./lib/swift/linux/x86_64
  && /build/swiftc/src/swift/utils/line-directive
  /build/swiftc/src/swift/stdlib/public/core/Algorithm.swift [… more
  files omitted for brevity …]
  /build/swiftc/src/swift/stdlib/public/core/VarArgs.swift
  /build/swiftc/src/swift/stdlib/public/core/Zip.swift
  /build/swiftc/src/swift/stdlib/public/core/Prespecialized.swift
  Traceback (most recent call last):
    File "/build/swiftc/src/swift/utils/line-directive", line 104, in
    <module>
      run()
    File "/build/swiftc/src/swift/utils/line-directive", line 92, in run
      file, line_num = map_line(m.group(1), int(m.group(2)))
    File "/build/swiftc/src/swift/utils/line-directive", line 60, in
    map_line
      map = fline_map(filename)
    File "/build/swiftc/src/swift/utils/line-directive", line 54, in
    fline_map
      map = _make_line_map(filename)
    File "/build/swiftc/src/swift/utils/line-directive", line 43, in
    _make_line_map
      for i, l in enumerate(input.readlines()):
    File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
  6851: ordinal not in range(128)

— Lukas

On 12 Jan 2016, at 20:31, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

I'm thought this was resolved in master already. Are you running the
latest version of master?

See
[gyb] Ignore filesystem locale and enforce UTF-8 · apple/swift@97b98b1 · GitHub

On Tue, Jan 12, 2016, at 01:12 PM, Lukas Stabe via swift-dev wrote:
I’ve tried to compile Swift in a clean Arch Linux chroot. In this
environment, `sys.getdefaultencoding()` does not return `utf-8`, which
causes gyb and line-directive to fail with non-ascii characters in input
files.

One example of such a file is Arrays.swift.gyb, which contains a
≤-character
[here](https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L358\).

In gyb, `argparse.FileType` is used to open input files, and
line-directive uses plain old `open(…)`. Sadly, both of those don’t
provide functionality to specify the file encoding in Python 2.7.10,
which is what OSX ships with.

I’m not that experienced in Python, so I don’t know what the pythonic way
to solve this would be, so if anyone with more Python experience would
help solve this, that would be great.

— Lukas
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev
Email had 1 attachment:
+ signature.asc
1k (application/pgp-signature)

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Email had 1 attachment:
+ signature.asc
1k (application/pgp-signature)

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

<0001-gyb-Force-UTF-8-encoding-when-parsing-templates-on-L.patch>
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev