Testing fails in GYBUnicodeDataUtils.py


(Tom Gall) #1

Building with: ./swift/utils/build-script -R -t --foundation

on Linux (gentoo amd64) fails with

+ /usr/bin/cmake --build
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
SwiftUnitTests

[6/29] Generating UnicodeGraphemeBreakTest.cpp from
UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8

FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
-E make_directory
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
&& /home/tgall/swift/swift/utils/gyb --test
-DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
-DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
-DCMAKE_SIZEOF_VOID_P=8 -o
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
copy_if_different
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
&& /usr/bin/cmake -E remove
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp

Traceback (most recent call last):

  File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
    gyb.main()
  File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
    args.target.write(executeTemplate(ast, args.line_directive, **bindings))
  File "/home/tgall/swift/swift/utils/gyb.py", line 974, in executeTemplate
    ast.execute(executionContext)
  File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
    x.execute(context)
  File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
    result = eval(self.code, context.localBindings)
  File "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
line 23, in <module>
    get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
  File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
553, in get_grapheme_cluster_break_tests_as_UTF8
    for line in f:
  File "/usr/lib64/python2.7/codecs.py", line 687, in next
    return self.reader.next()
  File "/usr/lib64/python2.7/codecs.py", line 618, in next
    line = self.readline()
  File "/usr/lib64/python2.7/codecs.py", line 533, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib64/python2.7/codecs.py", line 480, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
0: ordinal not in range(128)
[6/29] Building CXX object
unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
ninja: build stopped: subcommand failed.

Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:

in swift/utils/GYBUnicodeDataUtils.py there is:

with codecs.open(grapheme_break_test_file_name,
encoding=sys.getfilesystemencoding(), errors='strict') as f:

It appears to be our offending bit of python code. Now my unicode &
python foo isn't the strongest, but if I change what is passed as
encoding to : encoding='utf-8', the swift testcases seem to run quite
a bit better and end up reporting :

Testing Time: 65.82s
  Expected Passes : 1748
  Expected Failures : 83
  Unsupported Tests : 585
-- check-swift-linux-x86_64 finished --
--- Finished tests for swift ---

Question is, is that little fix the 'right thing' (TM) ? If so happy
to submit this as my first 'lame' patch.

Thanks

···

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall


(Jordan Rose) #2

Hi, Tom. That seems right to me: these files should always be UTF-8. And getting the build working on more platforms definitely isn't "lame". :slight_smile: Please send us a pull request on GitHub!

Jordan

···

On Jan 4, 2016, at 12:40, Tom Gall via swift-dev <swift-dev@swift.org> wrote:

Building with: ./swift/utils/build-script -R -t --foundation

on Linux (gentoo amd64) fails with

+ /usr/bin/cmake --build
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
SwiftUnitTests

[6/29] Generating UnicodeGraphemeBreakTest.cpp from
UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8

FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
-E make_directory
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
&& /home/tgall/swift/swift/utils/gyb --test
-DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
-DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
-DCMAKE_SIZEOF_VOID_P=8 -o
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
copy_if_different
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
&& /usr/bin/cmake -E remove
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp

Traceback (most recent call last):

File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
   gyb.main()
File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
   args.target.write(executeTemplate(ast, args.line_directive, **bindings))
File "/home/tgall/swift/swift/utils/gyb.py", line 974, in executeTemplate
   ast.execute(executionContext)
File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
   x.execute(context)
File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
   result = eval(self.code, context.localBindings)
File "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
line 23, in <module>
   get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
553, in get_grapheme_cluster_break_tests_as_UTF8
   for line in f:
File "/usr/lib64/python2.7/codecs.py", line 687, in next
   return self.reader.next()
File "/usr/lib64/python2.7/codecs.py", line 618, in next
   line = self.readline()
File "/usr/lib64/python2.7/codecs.py", line 533, in readline
   data = self.read(readsize, firstline=True)
File "/usr/lib64/python2.7/codecs.py", line 480, in read
   newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
0: ordinal not in range(128)
[6/29] Building CXX object
unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
ninja: build stopped: subcommand failed.

Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:

in swift/utils/GYBUnicodeDataUtils.py there is:

with codecs.open(grapheme_break_test_file_name,
encoding=sys.getfilesystemencoding(), errors='strict') as f:

It appears to be our offending bit of python code. Now my unicode &
python foo isn't the strongest, but if I change what is passed as
encoding to : encoding='utf-8', the swift testcases seem to run quite
a bit better and end up reporting :

Testing Time: 65.82s
Expected Passes : 1748
Expected Failures : 83
Unsupported Tests : 585
-- check-swift-linux-x86_64 finished --
--- Finished tests for swift ---

Question is, is that little fix the 'right thing' (TM) ? If so happy
to submit this as my first 'lame' patch.


(Ryan Lovelett) #3

Building with: ./swift/utils/build-script -R -t --foundation

on Linux (gentoo amd64) fails with

+ /usr/bin/cmake --build
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
SwiftUnitTests

[6/29] Generating UnicodeGraphemeBreakTest.cpp from
UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8

FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
-E make_directory
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
&& /home/tgall/swift/swift/utils/gyb --test
-DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
-DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
-DCMAKE_SIZEOF_VOID_P=8 -o
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
copy_if_different
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
&& /usr/bin/cmake -E remove
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp

Traceback (most recent call last):

  File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
    gyb.main()
  File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
    args.target.write(executeTemplate(ast, args.line_directive,
    **bindings))
  File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
  executeTemplate
    ast.execute(executionContext)
  File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
    x.execute(context)
  File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
    result = eval(self.code, context.localBindings)
  File
  "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
line 23, in <module>
    get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
  File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
553, in get_grapheme_cluster_break_tests_as_UTF8
    for line in f:
  File "/usr/lib64/python2.7/codecs.py", line 687, in next
    return self.reader.next()
  File "/usr/lib64/python2.7/codecs.py", line 618, in next
    line = self.readline()
  File "/usr/lib64/python2.7/codecs.py", line 533, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib64/python2.7/codecs.py", line 480, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
0: ordinal not in range(128)
[6/29] Building CXX object
unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
ninja: build stopped: subcommand failed.

Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:

in swift/utils/GYBUnicodeDataUtils.py there is:

with codecs.open(grapheme_break_test_file_name,
encoding=sys.getfilesystemencoding(), errors='strict') as f:

I wrote that code and patch (see:
https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
The change was in service of adding Python 3 support to GYB. So first of
all let me say: I'm sorry. :smirk:

Open up your python interpreter and figure out what your filesystem is
reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
OS X and my copy of Arch linux it reports `'utf-8'` which is why it
doesn't have an issue. Worst case scenario we can just force it to be
`with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
errors='strict') as f:` but I went with the filesystem encoding because
hopefully it is always UTF-8.

···

On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:

It appears to be our offending bit of python code. Now my unicode &
python foo isn't the strongest, but if I change what is passed as
encoding to : encoding='utf-8', the swift testcases seem to run quite
a bit better and end up reporting :

Testing Time: 65.82s
  Expected Passes : 1748
  Expected Failures : 83
  Unsupported Tests : 585
-- check-swift-linux-x86_64 finished --
--- Finished tests for swift ---

Question is, is that little fix the 'right thing' (TM) ? If so happy
to submit this as my first 'lame' patch.

Thanks

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Tom Gall) #4

Hi Ryan,

In my case I'm on Python 2.7. Your comment is interesting as I was
just tracking down why python-config --libs and python-config
--includes doesn't seem to be used to determine what is the system
python install. I was just starting to trace through the build tool to
figure out how build-script works it's initial magic.

In my case I have both python 3.4 and 2.7 installed but python 2.7 is
the system default. This ends up causing some interesting brand of
hurt, when trying to build swift's lldb. (Test Case
'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
that)

To answer some of your questions:

tgall@mars ~/swift $ locale

LANG=en_US
LC_CTYPE=C
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=

import sys
x=sys.getfilesystemencoding()
print x

ANSI_X3.4-1968

That explains some things :slight_smile:

···

On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:

Building with: ./swift/utils/build-script -R -t --foundation

on Linux (gentoo amd64) fails with

+ /usr/bin/cmake --build
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
SwiftUnitTests

[6/29] Generating UnicodeGraphemeBreakTest.cpp from
UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8

FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
-E make_directory
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
&& /home/tgall/swift/swift/utils/gyb --test
-DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
-DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
-DCMAKE_SIZEOF_VOID_P=8 -o
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
copy_if_different
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
&& /usr/bin/cmake -E remove
/home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp

Traceback (most recent call last):

  File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
    gyb.main()
  File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
    args.target.write(executeTemplate(ast, args.line_directive,
    **bindings))
  File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
  executeTemplate
    ast.execute(executionContext)
  File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
    x.execute(context)
  File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
    result = eval(self.code, context.localBindings)
  File
  "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
line 23, in <module>
    get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
  File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
553, in get_grapheme_cluster_break_tests_as_UTF8
    for line in f:
  File "/usr/lib64/python2.7/codecs.py", line 687, in next
    return self.reader.next()
  File "/usr/lib64/python2.7/codecs.py", line 618, in next
    line = self.readline()
  File "/usr/lib64/python2.7/codecs.py", line 533, in readline
    data = self.read(readsize, firstline=True)
  File "/usr/lib64/python2.7/codecs.py", line 480, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
0: ordinal not in range(128)
[6/29] Building CXX object
unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
ninja: build stopped: subcommand failed.

Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:

in swift/utils/GYBUnicodeDataUtils.py there is:

with codecs.open(grapheme_break_test_file_name,
encoding=sys.getfilesystemencoding(), errors='strict') as f:

I wrote that code and patch (see:
https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
The change was in service of adding Python 3 support to GYB. So first of
all let me say: I'm sorry. :smirk:

Open up your python interpreter and figure out what your filesystem is
reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
OS X and my copy of Arch linux it reports `'utf-8'` which is why it
doesn't have an issue. Worst case scenario we can just force it to be
`with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
errors='strict') as f:` but I went with the filesystem encoding because
hopefully it is always UTF-8.

It appears to be our offending bit of python code. Now my unicode &
python foo isn't the strongest, but if I change what is passed as
encoding to : encoding='utf-8', the swift testcases seem to run quite
a bit better and end up reporting :

Testing Time: 65.82s
  Expected Passes : 1748
  Expected Failures : 83
  Unsupported Tests : 585
-- check-swift-linux-x86_64 finished --
--- Finished tests for swift ---

Question is, is that little fix the 'right thing' (TM) ? If so happy
to submit this as my first 'lame' patch.

Thanks

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall


(Ryan Lovelett) #5

I wonder what the value of LC_ALL, LC_CTYPE, LANG are set to in your
environment? On my system LC_CTYPE=en_US.UTF-8 and LANG=en_US.UTF-8. My
understand of Python on Linux is that it reads these environment
variables to set `sys.getfilesystemencoding()`. This has to do with
configuring Python to consistently read filenames and such with the way
the OS is presenting them.

https://docs.python.org/2/library/sys.html#sys.getfilesystemencoding

···

On Mon, Jan 4, 2016, at 05:12 PM, Ryan Lovelett via swift-dev wrote:

On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
> Building with: ./swift/utils/build-script -R -t --foundation
>
> on Linux (gentoo amd64) fails with
>
> + /usr/bin/cmake --build
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
> SwiftUnitTests
>
> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
>
> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
> -E make_directory
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
> && /home/tgall/swift/swift/utils/gyb --test
> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
> -DCMAKE_SIZEOF_VOID_P=8 -o
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
> copy_if_different
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
> && /usr/bin/cmake -E remove
> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>
> Traceback (most recent call last):
>
> File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
> gyb.main()
> File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
> args.target.write(executeTemplate(ast, args.line_directive,
> **bindings))
> File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
> executeTemplate
> ast.execute(executionContext)
> File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
> x.execute(context)
> File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
> result = eval(self.code, context.localBindings)
> File
> "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
> line 23, in <module>
> get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
> File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
> 553, in get_grapheme_cluster_break_tests_as_UTF8
> for line in f:
> File "/usr/lib64/python2.7/codecs.py", line 687, in next
> return self.reader.next()
> File "/usr/lib64/python2.7/codecs.py", line 618, in next
> line = self.readline()
> File "/usr/lib64/python2.7/codecs.py", line 533, in readline
> data = self.read(readsize, firstline=True)
> File "/usr/lib64/python2.7/codecs.py", line 480, in read
> newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> 0: ordinal not in range(128)
> [6/29] Building CXX object
> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
> ninja: build stopped: subcommand failed.
>
> Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:
>
> in swift/utils/GYBUnicodeDataUtils.py there is:
>
> with codecs.open(grapheme_break_test_file_name,
> encoding=sys.getfilesystemencoding(), errors='strict') as f:
>

I wrote that code and patch (see:
https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
The change was in service of adding Python 3 support to GYB. So first of
all let me say: I'm sorry. :smirk:

Open up your python interpreter and figure out what your filesystem is
reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
OS X and my copy of Arch linux it reports `'utf-8'` which is why it
doesn't have an issue. Worst case scenario we can just force it to be
`with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
errors='strict') as f:` but I went with the filesystem encoding because
hopefully it is always UTF-8.

> It appears to be our offending bit of python code. Now my unicode &
> python foo isn't the strongest, but if I change what is passed as
> encoding to : encoding='utf-8', the swift testcases seem to run quite
> a bit better and end up reporting :
>
> Testing Time: 65.82s
> Expected Passes : 1748
> Expected Failures : 83
> Unsupported Tests : 585
> -- check-swift-linux-x86_64 finished --
> --- Finished tests for swift ---
>
> Question is, is that little fix the 'right thing' (TM) ? If so happy
> to submit this as my first 'lame' patch.
>
> Thanks
>
> --
> Regards,
> Tom
>
> "Where's the kaboom!? There was supposed to be an earth-shattering
> kaboom!" Marvin Martian
> Director, Linaro Mobile Group
> Tech Lead, GPGPU
> Linaro.org │ Open source software for ARM SoCs
> irc: tgall_foo | skype : tom_gall
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev


(Ryan Lovelett) #6

I would suggest setting the environment variables and re-running but
that is just my opinion.

I went through many of the same issues on Arch as well. Arch uses Python
2.7.11 and Python 3.5.1; with 3.5.1 being default.

I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1]
which links to the upstream LLDB Bug 25744 [2]. That report seems to
indicate there is a known bug compiling on Gentoo. They've even created
a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing
it in Gentoo.

Unfortunately, it seems that the experimental patch submitted there
(which I use to build on Arch) seems to only work on Arch. However,
maybe your just the person to fix that bug! Anyways food for thought.
Happy compiling. :beers:

[1] https://bugs.swift.org/browse/SR-14
[2] https://llvm.org/bugs/show_bug.cgi?id=25744
[3] https://llvm.org/bugs/show_bug.cgi?id=25866

···

On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote:

Hi Ryan,

In my case I'm on Python 2.7. Your comment is interesting as I was
just tracking down why python-config --libs and python-config
--includes doesn't seem to be used to determine what is the system
python install. I was just starting to trace through the build tool to
figure out how build-script works it's initial magic.

In my case I have both python 3.4 and 2.7 installed but python 2.7 is
the system default. This ends up causing some interesting brand of
hurt, when trying to build swift's lldb. (Test Case
'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
that)

To answer some of your questions:

tgall@mars ~/swift $ locale

LANG=en_US
LC_CTYPE=C
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=

>>> import sys
>>> x=sys.getfilesystemencoding()
>>> print x
ANSI_X3.4-1968

That explains some things :slight_smile:

On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev > <swift-dev@swift.org> wrote:
> On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
>> Building with: ./swift/utils/build-script -R -t --foundation
>>
>> on Linux (gentoo amd64) fails with
>>
>> + /usr/bin/cmake --build
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
>> SwiftUnitTests
>>
>> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
>> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
>>
>> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
>> -E make_directory
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
>> && /home/tgall/swift/swift/utils/gyb --test
>> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
>> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
>> -DCMAKE_SIZEOF_VOID_P=8 -o
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
>> copy_if_different
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
>> && /usr/bin/cmake -E remove
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>>
>> Traceback (most recent call last):
>>
>> File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
>> gyb.main()
>> File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
>> args.target.write(executeTemplate(ast, args.line_directive,
>> **bindings))
>> File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
>> executeTemplate
>> ast.execute(executionContext)
>> File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
>> x.execute(context)
>> File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
>> result = eval(self.code, context.localBindings)
>> File
>> "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
>> line 23, in <module>
>> get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
>> File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
>> 553, in get_grapheme_cluster_break_tests_as_UTF8
>> for line in f:
>> File "/usr/lib64/python2.7/codecs.py", line 687, in next
>> return self.reader.next()
>> File "/usr/lib64/python2.7/codecs.py", line 618, in next
>> line = self.readline()
>> File "/usr/lib64/python2.7/codecs.py", line 533, in readline
>> data = self.read(readsize, firstline=True)
>> File "/usr/lib64/python2.7/codecs.py", line 480, in read
>> newchars, decodedbytes = self.decode(data, self.errors)
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
>> 0: ordinal not in range(128)
>> [6/29] Building CXX object
>> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
>> ninja: build stopped: subcommand failed.
>>
>> Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:
>>
>> in swift/utils/GYBUnicodeDataUtils.py there is:
>>
>> with codecs.open(grapheme_break_test_file_name,
>> encoding=sys.getfilesystemencoding(), errors='strict') as f:
>>
>
> I wrote that code and patch (see:
> https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
> The change was in service of adding Python 3 support to GYB. So first of
> all let me say: I'm sorry. :smirk:
>
> Open up your python interpreter and figure out what your filesystem is
> reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
> OS X and my copy of Arch linux it reports `'utf-8'` which is why it
> doesn't have an issue. Worst case scenario we can just force it to be
> `with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
> errors='strict') as f:` but I went with the filesystem encoding because
> hopefully it is always UTF-8.
>
>> It appears to be our offending bit of python code. Now my unicode &
>> python foo isn't the strongest, but if I change what is passed as
>> encoding to : encoding='utf-8', the swift testcases seem to run quite
>> a bit better and end up reporting :
>>
>> Testing Time: 65.82s
>> Expected Passes : 1748
>> Expected Failures : 83
>> Unsupported Tests : 585
>> -- check-swift-linux-x86_64 finished --
>> --- Finished tests for swift ---
>>
>> Question is, is that little fix the 'right thing' (TM) ? If so happy
>> to submit this as my first 'lame' patch.
>>
>> Thanks
>>
>> --
>> Regards,
>> Tom
>>
>> "Where's the kaboom!? There was supposed to be an earth-shattering
>> kaboom!" Marvin Martian
>> Director, Linaro Mobile Group
>> Tech Lead, GPGPU
>> Linaro.org │ Open source software for ARM SoCs
>> irc: tgall_foo | skype : tom_gall
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall


(Tom Gall) #7

I would suggest setting the environment variables and re-running but
that is just my opinion.

I went through many of the same issues on Arch as well. Arch uses Python
2.7.11 and Python 3.5.1; with 3.5.1 being default.

Well point I was suggesting is that with the use of python-config the
version of python, include and libs location is easily determined.[4]
Looking at CMakeCache.txt in
build/Ninja-ReleaseAssert/swift-linux-x86_64, it's pretty obvious the
build environment isn't using it but seems to respect the PYTHON env
variable. I find this curious.

I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1]
which links to the upstream LLDB Bug 25744 [2]. That report seems to
indicate there is a known bug compiling on Gentoo. They've even created
a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing
it in Gentoo.

Fixing CMakeCache.txt by hand to use the system's python2.7 default
lldb builds successfully on gentoo, no changes necessary.

Still a good data point would be to switch my default python over to
3.x and see what kind of trouble i can find. Easy enough to do.

Unfortunately, it seems that the experimental patch submitted there
(which I use to build on Arch) seems to only work on Arch. However,
maybe your just the person to fix that bug! Anyways food for thought.
Happy compiling. :beers:

[1] https://bugs.swift.org/browse/SR-14
[2] https://llvm.org/bugs/show_bug.cgi?id=25744
[3] https://llvm.org/bugs/show_bug.cgi?id=25866

[4] https://www.python.org/dev/peps/pep-0394/

Back to the original issue. Do you want to require unicode locales?

Cheers!

···

On Mon, Jan 4, 2016 at 4:44 PM, Ryan Lovelett <swift-dev@ryan.lovelett.me> wrote:

On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote:

Hi Ryan,

In my case I'm on Python 2.7. Your comment is interesting as I was
just tracking down why python-config --libs and python-config
--includes doesn't seem to be used to determine what is the system
python install. I was just starting to trace through the build tool to
figure out how build-script works it's initial magic.

In my case I have both python 3.4 and 2.7 installed but python 2.7 is
the system default. This ends up causing some interesting brand of
hurt, when trying to build swift's lldb. (Test Case
'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
that)

To answer some of your questions:

tgall@mars ~/swift $ locale

LANG=en_US
LC_CTYPE=C
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=

>>> import sys
>>> x=sys.getfilesystemencoding()
>>> print x
ANSI_X3.4-1968

That explains some things :slight_smile:

On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev >> <swift-dev@swift.org> wrote:
> On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
>> Building with: ./swift/utils/build-script -R -t --foundation
>>
>> on Linux (gentoo amd64) fails with
>>
>> + /usr/bin/cmake --build
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
>> SwiftUnitTests
>>
>> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
>> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
>>
>> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
>> -E make_directory
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
>> && /home/tgall/swift/swift/utils/gyb --test
>> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
>> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
>> -DCMAKE_SIZEOF_VOID_P=8 -o
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
>> copy_if_different
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
>> && /usr/bin/cmake -E remove
>> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>>
>> Traceback (most recent call last):
>>
>> File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
>> gyb.main()
>> File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
>> args.target.write(executeTemplate(ast, args.line_directive,
>> **bindings))
>> File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
>> executeTemplate
>> ast.execute(executionContext)
>> File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
>> x.execute(context)
>> File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
>> result = eval(self.code, context.localBindings)
>> File
>> "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
>> line 23, in <module>
>> get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
>> File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
>> 553, in get_grapheme_cluster_break_tests_as_UTF8
>> for line in f:
>> File "/usr/lib64/python2.7/codecs.py", line 687, in next
>> return self.reader.next()
>> File "/usr/lib64/python2.7/codecs.py", line 618, in next
>> line = self.readline()
>> File "/usr/lib64/python2.7/codecs.py", line 533, in readline
>> data = self.read(readsize, firstline=True)
>> File "/usr/lib64/python2.7/codecs.py", line 480, in read
>> newchars, decodedbytes = self.decode(data, self.errors)
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
>> 0: ordinal not in range(128)
>> [6/29] Building CXX object
>> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
>> ninja: build stopped: subcommand failed.
>>
>> Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:
>>
>> in swift/utils/GYBUnicodeDataUtils.py there is:
>>
>> with codecs.open(grapheme_break_test_file_name,
>> encoding=sys.getfilesystemencoding(), errors='strict') as f:
>>
>
> I wrote that code and patch (see:
> https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
> The change was in service of adding Python 3 support to GYB. So first of
> all let me say: I'm sorry. :smirk:
>
> Open up your python interpreter and figure out what your filesystem is
> reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
> OS X and my copy of Arch linux it reports `'utf-8'` which is why it
> doesn't have an issue. Worst case scenario we can just force it to be
> `with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
> errors='strict') as f:` but I went with the filesystem encoding because
> hopefully it is always UTF-8.
>
>> It appears to be our offending bit of python code. Now my unicode &
>> python foo isn't the strongest, but if I change what is passed as
>> encoding to : encoding='utf-8', the swift testcases seem to run quite
>> a bit better and end up reporting :
>>
>> Testing Time: 65.82s
>> Expected Passes : 1748
>> Expected Failures : 83
>> Unsupported Tests : 585
>> -- check-swift-linux-x86_64 finished --
>> --- Finished tests for swift ---
>>
>> Question is, is that little fix the 'right thing' (TM) ? If so happy
>> to submit this as my first 'lame' patch.
>>
>> Thanks
>>
>> --
>> Regards,
>> Tom
>>
>> "Where's the kaboom!? There was supposed to be an earth-shattering
>> kaboom!" Marvin Martian
>> Director, Linaro Mobile Group
>> Tech Lead, GPGPU
>> Linaro.org │ Open source software for ARM SoCs
>> irc: tgall_foo | skype : tom_gall
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall


(Ryan Lovelett) #8

> I would suggest setting the environment variables and re-running but
> that is just my opinion.
>
> I went through many of the same issues on Arch as well. Arch uses Python
> 2.7.11 and Python 3.5.1; with 3.5.1 being default.

Well point I was suggesting is that with the use of python-config the
version of python, include and libs location is easily determined.[4]
Looking at CMakeCache.txt in
build/Ninja-ReleaseAssert/swift-linux-x86_64, it's pretty obvious the
build environment isn't using it but seems to respect the PYTHON env
variable. I find this curious.

To me that sounds exactly what those bugs I talked about are trying to
solve; the system selecting the wrong Python from the environment.

https://bugs.swift.org/browse/SR-14?focusedCommentId=10432&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-10432

Of course, you are right I digress.

> I can _maybe_ give you a clue on the lldb issue. Check out SR-14 [1]
> which links to the upstream LLDB Bug 25744 [2]. That report seems to
> indicate there is a known bug compiling on Gentoo. They've even created
> a specific bug Bug 25866 [3] which Bug 25744 depends on to track fixing
> it in Gentoo.

Fixing CMakeCache.txt by hand to use the system's python2.7 default
lldb builds successfully on gentoo, no changes necessary.

Still a good data point would be to switch my default python over to
3.x and see what kind of trouble i can find. Easy enough to do.

> Unfortunately, it seems that the experimental patch submitted there
> (which I use to build on Arch) seems to only work on Arch. However,
> maybe your just the person to fix that bug! Anyways food for thought.
> Happy compiling. :beers:

> [1] https://bugs.swift.org/browse/SR-14
> [2] https://llvm.org/bugs/show_bug.cgi?id=25744
> [3] https://llvm.org/bugs/show_bug.cgi?id=25866

[4] https://www.python.org/dev/peps/pep-0394/

Back to the original issue. Do you want to require unicode locales?

tl;dr If you hard coded it I certainly would not object and I doubt it
would break anything. If you think its necessary than I say: go for it!

In my opinion you've already illustrated that your locale setup is in
need of tweaking and that Python is behaving as designed. Therefore, to
me, hard-coding the encoding to be UTF-8 is a work around for an
improperly configured system. But I'm not in charge of anything.

Like Jordan said those specific files will always be UTF-8 (because they
are testing unicode encoding in Swift) so it should not break anything
to force the encoding as UTF-8.

···

On Mon, Jan 4, 2016, at 06:09 PM, Tom Gall wrote:

On Mon, Jan 4, 2016 at 4:44 PM, Ryan Lovelett > <swift-dev@ryan.lovelett.me> wrote:

Cheers!

> On Mon, Jan 4, 2016, at 05:29 PM, Tom Gall wrote:
>> Hi Ryan,
>>
>> In my case I'm on Python 2.7. Your comment is interesting as I was
>> just tracking down why python-config --libs and python-config
>> --includes doesn't seem to be used to determine what is the system
>> python install. I was just starting to trace through the build tool to
>> figure out how build-script works it's initial magic.
>>
>> In my case I have both python 3.4 and 2.7 installed but python 2.7 is
>> the system default. This ends up causing some interesting brand of
>> hurt, when trying to build swift's lldb. (Test Case
>> 'TestNSTimer.test_timerTickOnce' is freezing so was looking to debug
>> that)
>>
>> To answer some of your questions:
>>
>> tgall@mars ~/swift $ locale
>>
>> LANG=en_US
>> LC_CTYPE=C
>> LC_NUMERIC="en_US"
>> LC_TIME="en_US"
>> LC_COLLATE="en_US"
>> LC_MONETARY="en_US"
>> LC_MESSAGES="en_US"
>> LC_PAPER="en_US"
>> LC_NAME="en_US"
>> LC_ADDRESS="en_US"
>> LC_TELEPHONE="en_US"
>> LC_MEASUREMENT="en_US"
>> LC_IDENTIFICATION="en_US"
>> LC_ALL=
>>
>> >>> import sys
>> >>> x=sys.getfilesystemencoding()
>> >>> print x
>> ANSI_X3.4-1968
>>
>> That explains some things :slight_smile:
>>
>> On Mon, Jan 4, 2016 at 4:12 PM, Ryan Lovelett via swift-dev > >> <swift-dev@swift.org> wrote:
>> > On Mon, Jan 4, 2016, at 03:40 PM, Tom Gall via swift-dev wrote:
>> >> Building with: ./swift/utils/build-script -R -t --foundation
>> >>
>> >> on Linux (gentoo amd64) fails with
>> >>
>> >> + /usr/bin/cmake --build
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64 -- -j4
>> >> SwiftUnitTests
>> >>
>> >> [6/29] Generating UnicodeGraphemeBreakTest.cpp from
>> >> UnicodeGraphemeBreakTest.cpp.gyb with ptr size = 8
>> >>
>> >> FAILED: cd /home/tgall/swift/swift/unittests/Basic && /usr/bin/cmake
>> >> -E make_directory
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8
>> >> && /home/tgall/swift/swift/utils/gyb --test
>> >> -DunicodeGraphemeBreakPropertyFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakProperty.txt
>> >> -DunicodeGraphemeBreakTestFile=/home/tgall/swift/swift/utils/UnicodeData/GraphemeBreakTest.txt
>> >> -DCMAKE_SIZEOF_VOID_P=8 -o
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> >> UnicodeGraphemeBreakTest.cpp.gyb && /usr/bin/cmake -E
>> >> copy_if_different
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp
>> >> && /usr/bin/cmake -E remove
>> >> /home/tgall/swift/build/Ninja-ReleaseAssert/swift-linux-x86_64/unittests/Basic/8/UnicodeGraphemeBreakTest.cpp.tmp
>> >>
>> >> Traceback (most recent call last):
>> >>
>> >> File "/home/tgall/swift/swift/utils/gyb", line 3, in <module>
>> >> gyb.main()
>> >> File "/home/tgall/swift/swift/utils/gyb.py", line 1071, in main
>> >> args.target.write(executeTemplate(ast, args.line_directive,
>> >> **bindings))
>> >> File "/home/tgall/swift/swift/utils/gyb.py", line 974, in
>> >> executeTemplate
>> >> ast.execute(executionContext)
>> >> File "/home/tgall/swift/swift/utils/gyb.py", line 591, in execute
>> >> x.execute(context)
>> >> File "/home/tgall/swift/swift/utils/gyb.py", line 667, in execute
>> >> result = eval(self.code, context.localBindings)
>> >> File
>> >> "/home/tgall/swift/swift/unittests/Basic/UnicodeGraphemeBreakTest.cpp.gyb",
>> >> line 23, in <module>
>> >> get_grapheme_cluster_break_tests_as_UTF8(unicodeGraphemeBreakTestFile)
>> >> File "/home/tgall/swift/swift/utils/GYBUnicodeDataUtils.py", line
>> >> 553, in get_grapheme_cluster_break_tests_as_UTF8
>> >> for line in f:
>> >> File "/usr/lib64/python2.7/codecs.py", line 687, in next
>> >> return self.reader.next()
>> >> File "/usr/lib64/python2.7/codecs.py", line 618, in next
>> >> line = self.readline()
>> >> File "/usr/lib64/python2.7/codecs.py", line 533, in readline
>> >> data = self.read(readsize, firstline=True)
>> >> File "/usr/lib64/python2.7/codecs.py", line 480, in read
>> >> newchars, decodedbytes = self.decode(data, self.errors)
>> >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
>> >> 0: ordinal not in range(128)
>> >> [6/29] Building CXX object
>> >> unittests/Parse/CMakeFiles/SwiftParseTests.dir/LexerTests.cpp.o
>> >> ninja: build stopped: subcommand failed.
>> >>
>> >> Ah yes ... the joys of python stack dumps... anyway, tracing this a bit:
>> >>
>> >> in swift/utils/GYBUnicodeDataUtils.py there is:
>> >>
>> >> with codecs.open(grapheme_break_test_file_name,
>> >> encoding=sys.getfilesystemencoding(), errors='strict') as f:
>> >>
>> >
>> > I wrote that code and patch (see:
>> > https://github.com/apple/swift/commit/7dbb4127f55022bca7b191d448652b5decf8626e).
>> > The change was in service of adding Python 3 support to GYB. So first of
>> > all let me say: I'm sorry. :smirk:
>> >
>> > Open up your python interpreter and figure out what your filesystem is
>> > reporting its encoding to be (e.g., `sys.getfilesystemencoding()`). On
>> > OS X and my copy of Arch linux it reports `'utf-8'` which is why it
>> > doesn't have an issue. Worst case scenario we can just force it to be
>> > `with codecs.open(grapheme_break_test_file_name, encoding='utf-8',
>> > errors='strict') as f:` but I went with the filesystem encoding because
>> > hopefully it is always UTF-8.
>> >
>> >> It appears to be our offending bit of python code. Now my unicode &
>> >> python foo isn't the strongest, but if I change what is passed as
>> >> encoding to : encoding='utf-8', the swift testcases seem to run quite
>> >> a bit better and end up reporting :
>> >>
>> >> Testing Time: 65.82s
>> >> Expected Passes : 1748
>> >> Expected Failures : 83
>> >> Unsupported Tests : 585
>> >> -- check-swift-linux-x86_64 finished --
>> >> --- Finished tests for swift ---
>> >>
>> >> Question is, is that little fix the 'right thing' (TM) ? If so happy
>> >> to submit this as my first 'lame' patch.
>> >>
>> >> Thanks
>> >>
>> >> --
>> >> Regards,
>> >> Tom
>> >>
>> >> "Where's the kaboom!? There was supposed to be an earth-shattering
>> >> kaboom!" Marvin Martian
>> >> Director, Linaro Mobile Group
>> >> Tech Lead, GPGPU
>> >> Linaro.org │ Open source software for ARM SoCs
>> >> irc: tgall_foo | skype : tom_gall
>> >> _______________________________________________
>> >> swift-dev mailing list
>> >> swift-dev@swift.org
>> >> https://lists.swift.org/mailman/listinfo/swift-dev
>> > _______________________________________________
>> > swift-dev mailing list
>> > swift-dev@swift.org
>> > https://lists.swift.org/mailman/listinfo/swift-dev
>>
>>
>>
>> --
>> Regards,
>> Tom
>>
>> "Where's the kaboom!? There was supposed to be an earth-shattering
>> kaboom!" Marvin Martian
>> Director, Linaro Mobile Group
>> Tech Lead, GPGPU
>> Linaro.org │ Open source software for ARM SoCs
>> irc: tgall_foo | skype : tom_gall

--
Regards,
Tom

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian
Director, Linaro Mobile Group
Tech Lead, GPGPU
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | skype : tom_gall