New to building Swift toolchain

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to
get into Swift development one day, at least a tiny bit, at least to
understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to
understand how lit works, how to walk the AST and write changes back to the
editor. I've got code and tests for new refactoring kind and new
refactoring action, so I was like "yay, time to build the keychain and just
try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest *master* branch and ran *build-toolchain *and all of that
just to be told after almost an hour or so that there are 3 failing tests...

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is
limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I
don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?
Should I build using Xcode beta rather then GM?
Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?
Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a
lot of information for newbies like me trying to get into Swift compiler
development. Well, let's just say I couldn't find much :) I've just
realized that this mail list is better option that StackOverflow about an
hour ago :) Though, of course, I have read Brian Gesiak's posts and the
swift.org blog post on refactoring.

Just to sum up all of the above, how to I get to build local Xcode
toolchain which I can plug in to Xcode and test?

Thank you!

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to get into Swift development one day, at least a tiny bit, at least to understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to understand how lit works, how to walk the AST and write changes back to the editor. I've got code and tests for new refactoring kind and new refactoring action, so I was like "yay, time to build the keychain and just try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest master branch and ran build-toolchain and all of that just to be told after almost an hour or so that there are 3 failing tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was published on swift.org <http://swift.org/&gt;, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the README.md under System Requirements for macOS [and don't worry I run into this issue sometimes as well = )]. It is also documented on the CI: https://ci.swift.org <https://ci.swift.org/&gt; with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the amount of work performed, build-toolchain command is intended to be used for producing snapshots and thus it has to be complete and include code for all platforms/etc. We could potentially have a preset that builds a toolchain only for 1 platform. I am not 100% sure if such a toolchain would work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a lot of information for newbies like me trying to get into Swift compiler development. Well, let's just say I couldn't find much :) I've just realized that this mail list is better option that StackOverflow about an hour ago :) Though, of course, I have read Brian Gesiak's posts and the swift.org <http://swift.org/&gt; blog post on refactoring.

There are no bad questions or too many questions. Welcome to the dev-list = ).

Just to sum up all of the above, how to I get to build local Xcode toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there are more starter tasks than have been documented. If you are interested in a specific area of the compiler and can not find starter tasks in this area, please feel free to send a nice email to this list. I am sure we can find /something/ ; ).

Michael

···

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev <swift-dev@swift.org> wrote:

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hey Maksym. Your email was too big b/c of the attachment. Can you put the attachment into a pastebin or something like that?

···

On Sep 19, 2017, at 12:35 AM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

Hi Michael

Thanks for the feedback!

I gave it another go and this time made sure I have a proper setup.

./swift/utils/update-checkout --clone
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a

I've double checked all the submodules directories to make sure the tag was properly checked out.

Next I've configured Xcode with this command:

sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer

The xcodebuild -version output is:

Xcode 9.0
Build version 9M214v

And the Host OS is 10.12.6, which matches requirements on CI: https://ci.swift.org/\.

With all that configured I finally kick off the build:

./swift/utils/build-toolchain swift.local | tee build.log

After about 2+ hours it ends up with same 3 failures:

Failing Tests (3):

    Swift(iphonesimulator-i386) :: stdlib/Character.swift
    Swift(iphonesimulator-i386) :: stdlib/CodableTests.swift
    Swift(iphonesimulator-i386) :: stdlib/MapKit.swift

  Expected Passes : 9957
  Expected Failures : 27
  Unsupported Tests : 446
  Unexpected Failures: 3

Here's log for one of the failing tests:

--

********************

Testing: 0

FAIL: Swift(iphonesimulator-i386) :: stdlib/MapKit.swift (566 of 10433)

******************** TEST 'Swift(iphonesimulator-i386) :: stdlib/MapKit.swift' FAILED ********************

Script:

--

rm -rf /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && mkdir -p /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/bin/swiftc -target i386-apple-ios7.0 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' -F /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk/../../../Developer/Library/Frameworks -swift-version 3 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' /Users/grebenma/Projects/oss/swift/swift/test/stdlib/MapKit.swift -o /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out -module-name main && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk simctl spawn 'iPhone 5' /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out

--

Exit Code: 134

Command Output (stdout):

--

[ RUN ] MapKit.NSValue bridging

>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130

>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

[ FAIL ] MapKit.NSValue bridging

MapKit: Some tests failed, aborting

UXPASS:

FAIL: ["NSValue bridging"]

SKIP:

To debug, run:

$ /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out --stdlib-unittest-in-process --stdlib-unittest-filter "NSValue bridging"

--

Command Output (stderr):

--

Child process terminated with signal 6: Abort trap

--

The logs don't make much sense to me, especially since actual and expected values look the same :shrugs:.

I'm not sure if mailing lists accept attachment, but I attached full build log just in case.

I've also downloaded full console log from build #303 (which builds same snapshot as I have) and I find that that build log is somewhat different but overall has same contents.

So far this is where I got stuck with building toolchain.
There wasn't much issues with running stuff like
swift/utils/build-script --release-debuginfo --debug-swift
and then running incremental builds, e.g.
ninja -C build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64 swift-refactor

But when it comes to building the toolchain, this is what I arrived at.

Thanks again for your help!

Max

On Tue, Sep 19, 2017 at 4:58 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to get into Swift development one day, at least a tiny bit, at least to understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to understand how lit works, how to walk the AST and write changes back to the editor. I've got code and tests for new refactoring kind and new refactoring action, so I was like "yay, time to build the keychain and just try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest master branch and ran build-toolchain and all of that just to be told after almost an hour or so that there are 3 failing tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was published on swift.org <http://swift.org/&gt;, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the README.md under System Requirements for macOS [and don't worry I run into this issue sometimes as well = )]. It is also documented on the CI: https://ci.swift.org <https://ci.swift.org/&gt; with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the amount of work performed, build-toolchain command is intended to be used for producing snapshots and thus it has to be complete and include code for all platforms/etc. We could potentially have a preset that builds a toolchain only for 1 platform. I am not 100% sure if such a toolchain would work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a lot of information for newbies like me trying to get into Swift compiler development. Well, let's just say I couldn't find much :) I've just realized that this mail list is better option that StackOverflow about an hour ago :) Though, of course, I have read Brian Gesiak's posts and the swift.org <http://swift.org/&gt; blog post on refactoring.

There are no bad questions or too many questions. Welcome to the dev-list = ).

Just to sum up all of the above, how to I get to build local Xcode toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there are more starter tasks than have been documented. If you are interested in a specific area of the compiler and can not find starter tasks in this area, please feel free to send a nice email to this list. I am sure we can find /something/ ; ).

Michael

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

<build.log.zip>

Sure, I've put it in a gist here:

Cheers.

···

On Wed, Sep 20, 2017 at 5:46 AM, Michael Gottesman <mgottesman@apple.com> wrote:

Hey Maksym. Your email was too big b/c of the attachment. Can you put the
attachment into a pastebin or something like that?

On Sep 19, 2017, at 12:35 AM, Maksym Grebenets <mgrebenets@gmail.com> > wrote:

Hi Michael

Thanks for the feedback!

I gave it another go and this time made sure I have a proper setup.

*./swift/utils/update-checkout --clone./swift/utils/update-checkout --tag
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a*

I've double checked all the submodules directories to make sure the tag
was properly checked out.

Next I've configured Xcode with this command:

*sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer*

The *xcodebuild -version *output is:

*Xcode 9.0Build version 9M214v*

And the Host OS is *10.12.6*, which matches requirements on CI:
https://ci.swift.org/\.

With all that configured I finally kick off the build:

*./swift/utils/build-toolchain swift.local | tee build.log*

After about 2+ hours it ends up with same 3 failures:

*Failing Tests (3):*

* Swift(iphonesimulator-i386) :: stdlib/Character.swift
Swift(iphonesimulator-i386) :: stdlib/CodableTests.swift
Swift(iphonesimulator-i386) :: stdlib/MapKit.swift*

* Expected Passes : 9957 Expected Failures : 27 Unsupported Tests
: 446 Unexpected Failures: 3*

Here's log for one of the failing tests:

--

********************

Testing: 0

FAIL: Swift(iphonesimulator-i386) :: stdlib/MapKit.swift (566 of 10433)

******************** TEST 'Swift(iphonesimulator-i386) ::
stdlib/MapKit.swift' FAILED ********************

Script:

--

rm -rf /Users/grebenma/Projects/oss/swift/build/buildbot_osx/
swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp
&& mkdir -p /Users/grebenma/Projects/oss/swift/build/buildbot_osx/
swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp
&& xcrun --toolchain default --sdk /Applications/Xcode-beta.app/
Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/bin/swiftc
-target i386-apple-ios7.0 -module-cache-path '/var/folders/vs/50_
hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs'
-F /Applications/Xcode-beta.app/Contents/Developer/Platforms/
iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk/../../
../Developer/Library/Frameworks -swift-version 3 -module-cache-path
'/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/
swift-testsuite-clang-module-cachefYcCzs' /Users/grebenma/Projects/oss/
swift/swift/test/stdlib/MapKit.swift -o /Users/grebenma/Projects/oss/
swift/build/buildbot_osx/swift-macosx-x86_64/test-
iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out -module-name
main && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/
Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk
simctl spawn 'iPhone 5' /Users/grebenma/Projects/oss/
swift/build/buildbot_osx/swift-macosx-x86_64/test-
iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out

--

Exit Code: 134

Command Output (stdout):

--

[ RUN ] MapKit.NSValue bridging

>>> check failed at /Users/grebenma/Projects/oss/
swift/swift/stdlib/private/StdlibUnittestFoundationExtras/
StdlibUnittestFoundationExtras.swift, line 130

>>> expected: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

>>> actual: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

[ FAIL ] MapKit.NSValue bridging

MapKit: Some tests failed, aborting

UXPASS:

FAIL: ["NSValue bridging"]

SKIP:

To debug, run:

$ /Users/grebenma/Projects/oss/swift/build/buildbot_osx/
swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out
--stdlib-unittest-in-process --stdlib-unittest-filter "NSValue bridging"

--

Command Output (stderr):

--

Child process terminated with signal 6: Abort trap

--

The logs don't make much sense to me, especially since *actual* and
*expected* values look the same :shrugs:.

I'm not sure if mailing lists accept attachment, but I attached full build
log just in case.

I've also downloaded full console log from build #303 (which builds same
snapshot as I have) and I find that that build log is somewhat different
but overall has same contents.

So far this is where I got stuck with building toolchain.
There wasn't much issues with running stuff like
*swift/utils/build-script --release-debuginfo --debug-swift *
and then running incremental builds, e.g.
*ninja -C
build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64
swift-refactor*

But when it comes to building the toolchain, this is what I arrived at.

Thanks again for your help!

Max

On Tue, Sep 19, 2017 at 4:58 AM, Michael Gottesman <mgottesman@apple.com> > wrote:

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev < >> swift-dev@swift.org> wrote:

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to
get into Swift development one day, at least a tiny bit, at least to
understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to
understand how lit works, how to walk the AST and write changes back to the
editor. I've got code and tests for new refactoring kind and new
refactoring action, so I was like "yay, time to build the keychain and just
try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest *master* branch and ran *build-toolchain *and all of
that just to be told after almost an hour or so that there are 3 failing
tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSH
OT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that
also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is
limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I
don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was
published on swift.org, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the
README.md under System Requirements for macOS [and don't worry I run into
this issue sometimes as well = )]. It is also documented on the CI:
https://ci.swift.org with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the
amount of work performed, build-toolchain command is intended to be used
for producing snapshots and thus it has to be complete and include code for
all platforms/etc. We could potentially have a preset that builds a
toolchain only for 1 platform. I am not 100% sure if such a toolchain would
work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a
lot of information for newbies like me trying to get into Swift compiler
development. Well, let's just say I couldn't find much :) I've just
realized that this mail list is better option that StackOverflow about an
hour ago :) Though, of course, I have read Brian Gesiak's posts and the
swift.org blog post on refactoring.

There are no bad questions or too many questions. Welcome to the dev-list
= ).

Just to sum up all of the above, how to I get to build local Xcode
toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there
are more starter tasks than have been documented. If you are interested in
a specific area of the compiler and can not find starter tasks in this
area, please feel free to send a nice email to this list. I am sure we can
find /something/ ; ).

Michael

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

<build.log.zip>

Can you do a new clean build and post the log?

···

On Sep 19, 2017, at 4:11 PM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

Sure, I've put it in a gist here: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Build Log · GitHub

Cheers.

On Wed, Sep 20, 2017 at 5:46 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:
Hey Maksym. Your email was too big b/c of the attachment. Can you put the attachment into a pastebin or something like that?

On Sep 19, 2017, at 12:35 AM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

Hi Michael

Thanks for the feedback!

I gave it another go and this time made sure I have a proper setup.

./swift/utils/update-checkout --clone
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a

I've double checked all the submodules directories to make sure the tag was properly checked out.

Next I've configured Xcode with this command:

sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer

The xcodebuild -version output is:

Xcode 9.0
Build version 9M214v

And the Host OS is 10.12.6, which matches requirements on CI: https://ci.swift.org/\.

With all that configured I finally kick off the build:

./swift/utils/build-toolchain swift.local | tee build.log

After about 2+ hours it ends up with same 3 failures:

Failing Tests (3):

    Swift(iphonesimulator-i386) :: stdlib/Character.swift
    Swift(iphonesimulator-i386) :: stdlib/CodableTests.swift
    Swift(iphonesimulator-i386) :: stdlib/MapKit.swift

  Expected Passes : 9957
  Expected Failures : 27
  Unsupported Tests : 446
  Unexpected Failures: 3

Here's log for one of the failing tests:

--

********************

Testing: 0

FAIL: Swift(iphonesimulator-i386) :: stdlib/MapKit.swift (566 of 10433)

******************** TEST 'Swift(iphonesimulator-i386) :: stdlib/MapKit.swift' FAILED ********************

Script:

--

rm -rf /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && mkdir -p /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/bin/swiftc -target i386-apple-ios7.0 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' -F /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk/../../../Developer/Library/Frameworks -swift-version 3 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' /Users/grebenma/Projects/oss/swift/swift/test/stdlib/MapKit.swift -o /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out -module-name main && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk simctl spawn 'iPhone 5' /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out

--

Exit Code: 134

Command Output (stdout):

--

[ RUN ] MapKit.NSValue bridging

>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130

>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

[ FAIL ] MapKit.NSValue bridging

MapKit: Some tests failed, aborting

UXPASS:

FAIL: ["NSValue bridging"]

SKIP:

To debug, run:

$ /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out --stdlib-unittest-in-process --stdlib-unittest-filter "NSValue bridging"

--

Command Output (stderr):

--

Child process terminated with signal 6: Abort trap

--

The logs don't make much sense to me, especially since actual and expected values look the same :shrugs:.

I'm not sure if mailing lists accept attachment, but I attached full build log just in case.

I've also downloaded full console log from build #303 (which builds same snapshot as I have) and I find that that build log is somewhat different but overall has same contents.

So far this is where I got stuck with building toolchain.
There wasn't much issues with running stuff like
swift/utils/build-script --release-debuginfo --debug-swift
and then running incremental builds, e.g.
ninja -C build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64 swift-refactor

But when it comes to building the toolchain, this is what I arrived at.

Thanks again for your help!

Max

On Tue, Sep 19, 2017 at 4:58 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to get into Swift development one day, at least a tiny bit, at least to understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to understand how lit works, how to walk the AST and write changes back to the editor. I've got code and tests for new refactoring kind and new refactoring action, so I was like "yay, time to build the keychain and just try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest master branch and ran build-toolchain and all of that just to be told after almost an hour or so that there are 3 failing tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was published on swift.org <http://swift.org/&gt;, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the README.md under System Requirements for macOS [and don't worry I run into this issue sometimes as well = )]. It is also documented on the CI: https://ci.swift.org <https://ci.swift.org/&gt; with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the amount of work performed, build-toolchain command is intended to be used for producing snapshots and thus it has to be complete and include code for all platforms/etc. We could potentially have a preset that builds a toolchain only for 1 platform. I am not 100% sure if such a toolchain would work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a lot of information for newbies like me trying to get into Swift compiler development. Well, let's just say I couldn't find much :) I've just realized that this mail list is better option that StackOverflow about an hour ago :) Though, of course, I have read Brian Gesiak's posts and the swift.org <http://swift.org/&gt; blog post on refactoring.

There are no bad questions or too many questions. Welcome to the dev-list = ).

Just to sum up all of the above, how to I get to build local Xcode toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there are more starter tasks than have been documented. If you are interested in a specific area of the compiler and can not find starter tasks in this area, please feel free to send a nice email to this list. I am sure we can find /something/ ; ).

Michael

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

<build.log.zip>

The reason why I know it wasn't clean is that you didn't rebuild LLVM.

···

On Sep 19, 2017, at 5:11 PM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

Hm. But this one was pretty much a clean build. I’ve made sure I removed all build directories before running it.

Is there a flag to pass to build-toolchain to ensure clean build? Would —clean work?
On Wed, 20 Sep 2017 at 9:48 am, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:
Can you do a new clean build and post the log?

On Sep 19, 2017, at 4:11 PM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

Sure, I've put it in a gist here: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Build Log · GitHub

Cheers.

On Wed, Sep 20, 2017 at 5:46 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:
Hey Maksym. Your email was too big b/c of the attachment. Can you put the attachment into a pastebin or something like that?

On Sep 19, 2017, at 12:35 AM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

Hi Michael

Thanks for the feedback!

I gave it another go and this time made sure I have a proper setup.

./swift/utils/update-checkout --clone
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a

I've double checked all the submodules directories to make sure the tag was properly checked out.

Next I've configured Xcode with this command:

sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer

The xcodebuild -version output is:

Xcode 9.0
Build version 9M214v

And the Host OS is 10.12.6, which matches requirements on CI: https://ci.swift.org/\.

With all that configured I finally kick off the build:

./swift/utils/build-toolchain swift.local | tee build.log

After about 2+ hours it ends up with same 3 failures:

Failing Tests (3):

    Swift(iphonesimulator-i386) :: stdlib/Character.swift
    Swift(iphonesimulator-i386) :: stdlib/CodableTests.swift
    Swift(iphonesimulator-i386) :: stdlib/MapKit.swift

  Expected Passes : 9957
  Expected Failures : 27
  Unsupported Tests : 446
  Unexpected Failures: 3

Here's log for one of the failing tests:

--

********************

Testing: 0

FAIL: Swift(iphonesimulator-i386) :: stdlib/MapKit.swift (566 of 10433)

******************** TEST 'Swift(iphonesimulator-i386) :: stdlib/MapKit.swift' FAILED ********************

Script:

--

rm -rf /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && mkdir -p /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/bin/swiftc -target i386-apple-ios7.0 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' -F /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk/../../../Developer/Library/Frameworks -swift-version 3 -module-cache-path '/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs' /Users/grebenma/Projects/oss/swift/swift/test/stdlib/MapKit.swift -o /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out -module-name main && xcrun --toolchain default --sdk /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk simctl spawn 'iPhone 5' /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out

--

Exit Code: 134

Command Output (stdout):

--

[ RUN ] MapKit.NSValue bridging

>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130

>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

[ FAIL ] MapKit.NSValue bridging

MapKit: Some tests failed, aborting

UXPASS:

FAIL: ["NSValue bridging"]

SKIP:

To debug, run:

$ /Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out --stdlib-unittest-in-process --stdlib-unittest-filter "NSValue bridging"

--

Command Output (stderr):

--

Child process terminated with signal 6: Abort trap

--

The logs don't make much sense to me, especially since actual and expected values look the same :shrugs:.

I'm not sure if mailing lists accept attachment, but I attached full build log just in case.

I've also downloaded full console log from build #303 (which builds same snapshot as I have) and I find that that build log is somewhat different but overall has same contents.

So far this is where I got stuck with building toolchain.
There wasn't much issues with running stuff like
swift/utils/build-script --release-debuginfo --debug-swift
and then running incremental builds, e.g.
ninja -C build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64 swift-refactor

But when it comes to building the toolchain, this is what I arrived at.

Thanks again for your help!

Max

On Tue, Sep 19, 2017 at 4:58 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all

I guess like many other iOS devs with c/c++ background I really wanted to get into Swift development one day, at least a tiny bit, at least to understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to understand how lit works, how to walk the AST and write changes back to the editor. I've got code and tests for new refactoring kind and new refactoring action, so I was like "yay, time to build the keychain and just try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest master branch and ran build-toolchain and all of that just to be told after almost an hour or so that there are 3 failing tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout --tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was published on swift.org <http://swift.org/&gt;, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the README.md under System Requirements for macOS [and don't worry I run into this issue sometimes as well = )]. It is also documented on the CI: https://ci.swift.org <https://ci.swift.org/&gt; with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the amount of work performed, build-toolchain command is intended to be used for producing snapshots and thus it has to be complete and include code for all platforms/etc. We could potentially have a preset that builds a toolchain only for 1 platform. I am not 100% sure if such a toolchain would work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't a lot of information for newbies like me trying to get into Swift compiler development. Well, let's just say I couldn't find much :) I've just realized that this mail list is better option that StackOverflow about an hour ago :) Though, of course, I have read Brian Gesiak's posts and the swift.org <http://swift.org/&gt; blog post on refactoring.

There are no bad questions or too many questions. Welcome to the dev-list = ).

Just to sum up all of the above, how to I get to build local Xcode toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there are more starter tasks than have been documented. If you are interested in a specific area of the compiler and can not find starter tasks in this area, please feel free to send a nice email to this list. I am sure we can find /something/ ; ).

Michael

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

<build.log.zip>

Hm. But this one was pretty much a clean build. I’ve made sure I removed
all build directories before running it.

Is there a flag to pass to build-toolchain to ensure clean build? Would
—clean work?

···

On Wed, 20 Sep 2017 at 9:48 am, Michael Gottesman <mgottesman@apple.com> wrote:

Can you do a new clean build and post the log?

On Sep 19, 2017, at 4:11 PM, Maksym Grebenets <mgrebenets@gmail.com> > wrote:

Sure, I've put it in a gist here:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Build Log · GitHub

Cheers.

On Wed, Sep 20, 2017 at 5:46 AM, Michael Gottesman <mgottesman@apple.com> > wrote:

Hey Maksym. Your email was too big b/c of the attachment. Can you put the
attachment into a pastebin or something like that?

On Sep 19, 2017, at 12:35 AM, Maksym Grebenets <mgrebenets@gmail.com> >> wrote:

Hi Michael

Thanks for the feedback!

I gave it another go and this time made sure I have a proper setup.

*./swift/utils/update-checkout --clone./swift/utils/update-checkout --tag
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a*

I've double checked all the submodules directories to make sure the tag
was properly checked out.

Next I've configured Xcode with this command:

*sudo xcode-select -s /Applications/Xcode-beta.app/Contents/Developer*

The *xcodebuild -version *output is:

*Xcode 9.0Build version 9M214v*

And the Host OS is *10.12.6*, which matches requirements on CI:
https://ci.swift.org/\.

With all that configured I finally kick off the build:

*./swift/utils/build-toolchain swift.local | tee build.log*

After about 2+ hours it ends up with same 3 failures:

*Failing Tests (3):*

* Swift(iphonesimulator-i386) :: stdlib/Character.swift
Swift(iphonesimulator-i386) :: stdlib/CodableTests.swift
Swift(iphonesimulator-i386) :: stdlib/MapKit.swift*

* Expected Passes : 9957 Expected Failures : 27 Unsupported Tests
: 446 Unexpected Failures: 3*

Here's log for one of the failing tests:

--

********************

Testing: 0

FAIL: Swift(iphonesimulator-i386) :: stdlib/MapKit.swift (566 of 10433)

******************** TEST 'Swift(iphonesimulator-i386) ::
stdlib/MapKit.swift' FAILED ********************

Script:

--

rm -rf
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp
&& mkdir -p
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp
&& xcrun --toolchain default --sdk
/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/bin/swiftc
-target i386-apple-ios7.0 -module-cache-path
'/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs'
-F
/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk/../../../Developer/Library/Frameworks
-swift-version 3 -module-cache-path
'/var/folders/vs/50_hfsmx4m1cpwbn7j3x6r6c0000gp/T/swift-testsuite-clang-module-cachefYcCzs'
/Users/grebenma/Projects/oss/swift/swift/test/stdlib/MapKit.swift -o
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out
-module-name main && xcrun --toolchain default --sdk
/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.0.sdk
simctl spawn 'iPhone 5'
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out

--

Exit Code: 134

Command Output (stdout):

--

[ RUN ] MapKit.NSValue bridging

>>> check failed at
/Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift,
line 130

>>> expected: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

>>> actual: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

[ FAIL ] MapKit.NSValue bridging

MapKit: Some tests failed, aborting

UXPASS:

FAIL: ["NSValue bridging"]

SKIP:

To debug, run:

$
/Users/grebenma/Projects/oss/swift/build/buildbot_osx/swift-macosx-x86_64/test-iphonesimulator-i386/stdlib/Output/MapKit.swift.tmp/a.out
--stdlib-unittest-in-process --stdlib-unittest-filter "NSValue bridging"

--

Command Output (stderr):

--

Child process terminated with signal 6: Abort trap

--

The logs don't make much sense to me, especially since *actual* and
*expected* values look the same :shrugs:.

I'm not sure if mailing lists accept attachment, but I attached full
build log just in case.

I've also downloaded full console log from build #303 (which builds same
snapshot as I have) and I find that that build log is somewhat different
but overall has same contents.

So far this is where I got stuck with building toolchain.
There wasn't much issues with running stuff like
*swift/utils/build-script --release-debuginfo --debug-swift *
and then running incremental builds, e.g.
*ninja -C
build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64
swift-refactor*

But when it comes to building the toolchain, this is what I arrived at.

Thanks again for your help!

Max

On Tue, Sep 19, 2017 at 4:58 AM, Michael Gottesman <mgottesman@apple.com> >> wrote:

On Sep 18, 2017, at 8:54 AM, Maksym Grebenets via swift-dev < >>> swift-dev@swift.org> wrote:

Hi all

I guess like many other iOS devs with c/c++ background I really wanted
to get into Swift development one day, at least a tiny bit, at least to
understand how things work.

So new refactoring tools and starter tasks finally pulled me in.

With a lot of struggle and a lot of help from community I was able to
understand how lit works, how to walk the AST and write changes back to the
editor. I've got code and tests for new refactoring kind and new
refactoring action, so I was like "yay, time to build the keychain and just
try it in Xcode!!!".

That's where I was thrown back to the "what do I do?" state again...

I pulled latest *master* branch and ran *build-toolchain *and all of
that just to be told after almost an hour or so that there are 3 failing
tests...

What were the failing tests?

So I thought:

cd swift
git checkout swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a
cd ..
./swift/utils/update-checkout
--tag swift-DEVELOPMENT-SNAPSHOT-2017-09-15-a

then build and fail again :(...

So you are saying that you took the last "known good snapshot" and that
also failed. How did it fail? On the same 3 tests?

OK then, so I just disable the tests and give it another go:

# build-presets.ini
#test
#validation-test
#long-test

Yay! I got a toolchain!
I copy it over to /Library/Developer/Toolchains/
Select it in Xcode (latest Xcode 9 GM) and... no luck
The source code highlighting disappears,
I get "An internal error occurred. Source editor functionality is
limited..." error.
All of the refactoring actions are disabled.

I guess those 3 failing tests did matter in the end...

I may be frustrated, because I don't understand what's going on, but I
don't feel like giving up.

I'd appreciate any help or advice I can get!

Should I choose an older tag from before to get all tests passing?

Do you mean before the last known good snapshot? If a snapshot was
published on swift.org, it /should/ pass all of the tests.

Should I build using Xcode beta rather then GM?

You should be using the Xcode beta. This can be seen either via the
README.md under System Requirements for macOS [and don't worry I run into
this issue sometimes as well = )]. It is also documented on the CI:
https://ci.swift.org with a nice link.

Is there any way to build/re-build the OS X toolchain any faster?
Why does it take so long even if I just re-run same command again?

When you rerun is it not incremental? (I forgot TBH). In terms of the
amount of work performed, build-toolchain command is intended to be used
for producing snapshots and thus it has to be complete and include code for
all platforms/etc. We could potentially have a preset that builds a
toolchain only for 1 platform. I am not 100% sure if such a toolchain would
work, but it should take significantly less time to build. Would that help?

Where did I get it wrong so the toolchain doesn't really work with Xcode?
Can I connect lldb to process that runs my lit tests?

I'm sorry for having so many questions, but you can imagine there isn't
a lot of information for newbies like me trying to get into Swift compiler
development. Well, let's just say I couldn't find much :) I've just
realized that this mail list is better option that StackOverflow about an
hour ago :) Though, of course, I have read Brian Gesiak's posts and the
swift.org blog post on refactoring.

There are no bad questions or too many questions. Welcome to the
dev-list = ).

Just to sum up all of the above, how to I get to build local Xcode
toolchain which I can plug in to Xcode and test?

Answer my questions above... I'll help you get across the finish line.

On another note, I noticed you mention starter tasks: keep in mind there
are more starter tasks than have been documented. If you are interested in
a specific area of the compiler and can not find starter tasks in this
area, please feel free to send a nice email to this list. I am sure we can
find /something/ ; ).

Michael

Thank you!
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

<build.log.zip>

I see.

I gave it another go:

Looking at the build logs, I can definitely see llvm source code being
built.
I've looked up a bunch of llvm .cpp files in the build log,
e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using
/Applications/CMake.app, while on my machine it is
*/usr/local/Cellar/cmake/3.9.1/bin/cmake.*
So I've tried to upgrade to 3.9.2 and run same test command manually - same
failures.
Same for using CMake.app downloaded from cmake.org (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on
Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this
could be related.

Looking at the failing tests, it seems like all of them are related to
encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my
machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error
like:
Decoded URLComponents <//0.0.0.0> not equal to original <//0.0.0.0>
I can't really spot the difference, unless there's some invisible escape
character...

Last failing test is for NSValue bridging:

check failed at

/Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift,
line 130

expected: <00000000 00003140 00000000 00004340> (of type

NSConcreteValue)

actual: <00000000 00003140 00000000 00004340> (of type

NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came
from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured
stdout: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too
many files open" and somehow still using cmake 3.9.2 from Homebrew
installation:

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:

Regards,
Max

···

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

I see.

I gave it another go: mgrebenets’s gists · GitHub
e2c981951586910c679df17d377c9e69

Looking at the build logs, I can definitely see llvm source code being
built.
I've looked up a bunch of llvm .cpp files in the build log,
e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using
/Applications/CMake.app, while on my machine it is
*/usr/local/Cellar/cmake/3.9.1/bin/cmake.*
So I've tried to upgrade to 3.9.2 and run same test command manually -
same failures.
Same for using CMake.app downloaded from cmake.org (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on
Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this
could be related.

Looking at the failing tests, it seems like all of them are related to
encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my
machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error
like:
Decoded URLComponents <//0.0.0.0> not equal to original <//0.0.0.0>
I can't really spot the difference, unless there's some invisible escape
character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/
swift/swift/stdlib/private/StdlibUnittestFoundationExtras/
StdlibUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

FYI, I haven't forgotten you! I just need to land some code tonight ; ). I want to try to reproduce exactly what you are doing.

Michael

···

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured stdout: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too many files open" and somehow still using cmake 3.9.2 from Homebrew installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:
I see.

I gave it another go: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Clean Build · GitHub

Looking at the build logs, I can definitely see llvm source code being built.
I've looked up a bunch of llvm .cpp files in the build log, e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using /Applications/CMake.app, while on my machine it is /usr/local/Cellar/cmake/3.9.1/bin/cmake.
So I've tried to upgrade to 3.9.2 and run same test command manually - same failures.
Same for using CMake.app downloaded from cmake.org <http://cmake.org/&gt; (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this could be related.

Looking at the failing tests, it seems like all of them are related to encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error like:
Decoded URLComponents <//0.0.0.0 <http://0.0.0.0/&gt;&gt; not equal to original <//0.0.0.0 <http://0.0.0.0/&gt;&gt;
I can't really spot the difference, unless there's some invisible escape character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

Ok. I am looking at this now on my side machine.

Sorry for the delay, I had to land code and then I had vacation = (.

Michael

···

On Sep 20, 2017, at 10:05 PM, Michael Gottesman via swift-dev <swift-dev@swift.org> wrote:

FYI, I haven't forgotten you! I just need to land some code tonight ; ). I want to try to reproduce exactly what you are doing.

Michael

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured stdout: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too many files open" and somehow still using cmake 3.9.2 from Homebrew installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:
I see.

I gave it another go: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Clean Build · GitHub

Looking at the build logs, I can definitely see llvm source code being built.
I've looked up a bunch of llvm .cpp files in the build log, e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using /Applications/CMake.app, while on my machine it is /usr/local/Cellar/cmake/3.9.1/bin/cmake.
So I've tried to upgrade to 3.9.2 and run same test command manually - same failures.
Same for using CMake.app downloaded from cmake.org <http://cmake.org/&gt; (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this could be related.

Looking at the failing tests, it seems like all of them are related to encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error like:
Decoded URLComponents <//0.0.0.0 <http://0.0.0.0/&gt;&gt; not equal to original <//0.0.0.0 <http://0.0.0.0/&gt;&gt;
I can't really spot the difference, unless there's some invisible escape character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Ok. My side machine info:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G26
$ xcodebuild -version
Xcode 9.0
Build version 9M214v
$ cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$

So I think I should be able to reproduce since we have the same xcode/os version. I am starting a build now.

···

On Sep 26, 2017, at 9:44 AM, Michael Gottesman <mgottesman@apple.com> wrote:

Ok. I am looking at this now on my side machine.

Sorry for the delay, I had to land code and then I had vacation = (.

Michael

On Sep 20, 2017, at 10:05 PM, Michael Gottesman via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

FYI, I haven't forgotten you! I just need to land some code tonight ; ). I want to try to reproduce exactly what you are doing.

Michael

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured stdout: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too many files open" and somehow still using cmake 3.9.2 from Homebrew installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:
I see.

I gave it another go: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Clean Build · GitHub

Looking at the build logs, I can definitely see llvm source code being built.
I've looked up a bunch of llvm .cpp files in the build log, e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using /Applications/CMake.app, while on my machine it is /usr/local/Cellar/cmake/3.9.1/bin/cmake.
So I've tried to upgrade to 3.9.2 and run same test command manually - same failures.
Same for using CMake.app downloaded from cmake.org <http://cmake.org/&gt; (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this could be related.

Looking at the failing tests, it seems like all of them are related to encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error like:
Decoded URLComponents <//0.0.0.0 <http://0.0.0.0/&gt;&gt; not equal to original <//0.0.0.0 <http://0.0.0.0/&gt;&gt;
I can't really spot the difference, unless there's some invisible escape character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

Hi Michael

Thanks for following up!

I've found some extra time too to give it few more tries.
As a matter of fact, I've tried quite a few things.
Spoiler alert - none of them worked :)
Here's the list of things anyways:
- A completely clean checkout of sift repo and all submodules (still same
laptop, but to be 100% there's no left overs of any kind)
- Change LANG env var from en_GB.UTF-8 to en_US.UTF-8
- Use exactly the same version of CMake as on CI: 3.6.3
- Remove all extra software which is not on CI: python v3, Z3, cmake from
Homebrew
- Make sure python 2.x version is almost the same as on CI (Jenkins has
2.7.10, while I have 2.7.13, recently bumped to 2.7.14)

- Checkout 100% identical source code as it is used by Jenkins
For this one I checked build #354
https://ci.swift.org/job/oss-swift-package-osx/354/parameters/ and made
sure all submodules have identical commits.
This is where I noticed that I was using latest ninja version 1.8.2, while
Jenkins uses 1.7.2.
I also noticed that llvm is build from 1 commit above the
swift-DEVELOPMENT-SNAPSHOT-2017-09-25-a tag somehow... (418d91a243e)

This one wasn't intentional, but I ended up trying to build 2017-09-25-a
snapshot using 2017-09-25-a toolchain downloaded from swift.org.

All of that lands me with the same 3 failing tests for iPhone 5 simulator.

I know that I can work on code base without running the tests.

But I really wanted to be able to build the toolchain and try it out in
Xcode, especially since blog post on refactoring just says "Use
build-toolchain script to build your own toolchain"...

Regards,
Max

···

On Wed, Sep 27, 2017 at 3:11 AM, Michael Gottesman <mgottesman@apple.com> wrote:

Ok. My side machine info:

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G26
$ xcodebuild -version
Xcode 9.0
Build version 9M214v
$ cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$

So I think I should be able to reproduce since we have the same xcode/os
version. I am starting a build now.

On Sep 26, 2017, at 9:44 AM, Michael Gottesman <mgottesman@apple.com> > wrote:

Ok. I am looking at this now on my side machine.

Sorry for the delay, I had to land code and then I had vacation = (.

Michael

On Sep 20, 2017, at 10:05 PM, Michael Gottesman via swift-dev < > swift-dev@swift.org> wrote:

FYI, I haven't forgotten you! I just need to land some code tonight ; ). I
want to try to reproduce exactly what you are doing.

Michael

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com> > wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/
FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured
stdout: https://gist.github.com/mgrebenets/bf3a54cb90141c17c715f5ddfddf6c
04

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too
many files open" and somehow still using cmake 3.9.2 from Homebrew
installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com> > wrote:

I see.

I gave it another go: mg’s gists · GitHub
rebenets/e2c981951586910c679df17d377c9e69

Looking at the build logs, I can definitely see llvm source code being
built.
I've looked up a bunch of llvm .cpp files in the build log,
e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using
/Applications/CMake.app, while on my machine it is
*/usr/local/Cellar/cmake/3.9.1/bin/cmake.*
So I've tried to upgrade to 3.9.2 and run same test command manually -
same failures.
Same for using CMake.app downloaded from cmake.org (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on
Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this
could be related.

Looking at the failing tests, it seems like all of them are related to
encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my
machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error
like:
Decoded URLComponents <//0.0.0.0> not equal to original <//0.0.0.0>
I can't really spot the difference, unless there's some invisible escape
character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/s
wift/swift/stdlib/private/StdlibUnittestFoundationExtras/Std
libUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Hi Michael

Thanks for following up!

I've found some extra time too to give it few more tries.
As a matter of fact, I've tried quite a few things.
Spoiler alert - none of them worked :)
Here's the list of things anyways:
- A completely clean checkout of sift repo and all submodules (still same laptop, but to be 100% there's no left overs of any kind)
- Change LANG env var from en_GB.UTF-8 to en_US.UTF-8
- Use exactly the same version of CMake as on CI: 3.6.3
- Remove all extra software which is not on CI: python v3, Z3, cmake from Homebrew
- Make sure python 2.x version is almost the same as on CI (Jenkins has 2.7.10, while I have 2.7.13, recently bumped to 2.7.14)

- Checkout 100% identical source code as it is used by Jenkins
For this one I checked build #354 https://ci.swift.org/job/oss-swift-package-osx/354/parameters/ and made sure all submodules have identical commits.
This is where I noticed that I was using latest ninja version 1.8.2, while Jenkins uses 1.7.2.
I also noticed that llvm is build from 1 commit above the swift-DEVELOPMENT-SNAPSHOT-2017-09-25-a tag somehow... (418d91a243e)

This one wasn't intentional, but I ended up trying to build 2017-09-25-a snapshot using 2017-09-25-a toolchain downloaded from swift.org <http://swift.org/&gt;\.

All of that lands me with the same 3 failing tests for iPhone 5 simulator.

I know that I can work on code base without running the tests.

But I really wanted to be able to build the toolchain and try it out in Xcode, especially since blog post on refactoring just says "Use build-toolchain script to build your own toolchain"...

Quick update: I just checked my side machine. I was able to build everything successfully, pass tests, and build the package/toolchain for tag 'swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a'.

I am going to swing back and read the rest of this email later today. I need to finish a few things today.

Michael

···

On Sep 26, 2017, at 3:57 PM, Maksym Grebenets <mgrebenets@gmail.com> wrote:

Regards,
Max

On Wed, Sep 27, 2017 at 3:11 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:
Ok. My side machine info:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G26
$ xcodebuild -version
Xcode 9.0
Build version 9M214v
$ cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake <http://kitware.com/cmake>).
$

So I think I should be able to reproduce since we have the same xcode/os version. I am starting a build now.

On Sep 26, 2017, at 9:44 AM, Michael Gottesman <mgottesman@apple.com <mailto:mgottesman@apple.com>> wrote:

Ok. I am looking at this now on my side machine.

Sorry for the delay, I had to land code and then I had vacation = (.

Michael

On Sep 20, 2017, at 10:05 PM, Michael Gottesman via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

FYI, I haven't forgotten you! I just need to land some code tonight ; ). I want to try to reproduce exactly what you are doing.

Michael

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured stdout: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too many files open" and somehow still using cmake 3.9.2 from Homebrew installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com <mailto:mgrebenets@gmail.com>> wrote:
I see.

I gave it another go: swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Clean Build · GitHub

Looking at the build logs, I can definitely see llvm source code being built.
I've looked up a bunch of llvm .cpp files in the build log, e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using /Applications/CMake.app, while on my machine it is /usr/local/Cellar/cmake/3.9.1/bin/cmake.
So I've tried to upgrade to 3.9.2 and run same test command manually - same failures.
Same for using CMake.app downloaded from cmake.org <http://cmake.org/&gt; (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this could be related.

Looking at the failing tests, it seems like all of them are related to encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my machine it's 5.
Other failures are in CodableTests.swift, but all of them report an error like:
Decoded URLComponents <//0.0.0.0 <http://0.0.0.0/&gt;&gt; not equal to original <//0.0.0.0 <http://0.0.0.0/&gt;&gt;
I can't really spot the difference, unless there's some invisible escape character...

Last failing test is for NSValue bridging:
>>> check failed at /Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift, line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

So it must be something not related to the build tools being used. At least
I went though tons of variations on those.

It could be the particular version of simulator being used for running
tests on iPhone 5. Maybe the fact that I have multiple Xcode versions
contributes to that problem... I have 8.3.3, 9 and latest beta of 9.

If only I could get something like docker image to roll into my laptop to
get “vanilla” build environment....

I’ll try to see if somethings wrong with simulator setup and try to get my
hands on some Mac mini with clean Xcode 9 setup.

Cheers
Max

···

On Thu, 28 Sep 2017 at 4:27 am, Michael Gottesman <mgottesman@apple.com> wrote:

On Sep 26, 2017, at 3:57 PM, Maksym Grebenets <mgrebenets@gmail.com> > wrote:

Hi Michael

Thanks for following up!

I've found some extra time too to give it few more tries.
As a matter of fact, I've tried quite a few things.
Spoiler alert - none of them worked :)
Here's the list of things anyways:
- A completely clean checkout of sift repo and all submodules (still same
laptop, but to be 100% there's no left overs of any kind)
- Change LANG env var from en_GB.UTF-8 to en_US.UTF-8
- Use exactly the same version of CMake as on CI: 3.6.3
- Remove all extra software which is not on CI: python v3, Z3, cmake from
Homebrew
- Make sure python 2.x version is almost the same as on CI (Jenkins has
2.7.10, while I have 2.7.13, recently bumped to 2.7.14)

- Checkout 100% identical source code as it is used by Jenkins
For this one I checked build #354
https://ci.swift.org/job/oss-swift-package-osx/354/parameters/ and made
sure all submodules have identical commits.
This is where I noticed that I was using latest ninja version 1.8.2, while
Jenkins uses 1.7.2.
I also noticed that llvm is build from 1 commit above the
swift-DEVELOPMENT-SNAPSHOT-2017-09-25-a tag somehow... (418d91a243e)

This one wasn't intentional, but I ended up trying to build 2017-09-25-a
snapshot using 2017-09-25-a toolchain downloaded from swift.org.

All of that lands me with the same 3 failing tests for iPhone 5 simulator.

I know that I can work on code base without running the tests.

But I really wanted to be able to build the toolchain and try it out in
Xcode, especially since blog post on refactoring just says "Use
build-toolchain script to build your own toolchain"...

Quick update: I just checked my side machine. I was able to build
everything successfully, pass tests, and build the package/toolchain for
tag 'swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a'.

I am going to swing back and read the rest of this email later today. I
need to finish a few things today.

Michael

Regards,
Max

On Wed, Sep 27, 2017 at 3:11 AM, Michael Gottesman <mgottesman@apple.com> > wrote:

Ok. My side machine info:

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G26
$ xcodebuild -version
Xcode 9.0
Build version 9M214v
$ cmake --version
cmake version 3.8.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
$

So I think I should be able to reproduce since we have the same xcode/os
version. I am starting a build now.

On Sep 26, 2017, at 9:44 AM, Michael Gottesman <mgottesman@apple.com> >> wrote:

Ok. I am looking at this now on my side machine.

Sorry for the delay, I had to land code and then I had vacation = (.

Michael

On Sep 20, 2017, at 10:05 PM, Michael Gottesman via swift-dev < >> swift-dev@swift.org> wrote:

FYI, I haven't forgotten you! I just need to land some code tonight ; ).
I want to try to reproduce exactly what you are doing.

Michael

On Sep 20, 2017, at 6:08 AM, Maksym Grebenets <mgrebenets@gmail.com> >> wrote:

I tried few more options.

I tried switching my shell to bash and giving it a go.
This time I got some unusual configuration error.
The error came
from /usr/local/Cellar/cmake/3.9.2/share/cmake/Modules/FindPythonInterp.cmake.
Unfortunately the error message was in stderr, while I only captured
stdout:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a using /bin/bash and Homebrew cmake 3.9.2 · GitHub

So then I added CMake.app bin to the path
export PATH=/Applications/CMake.app/Contents/bin:PATH
and ran the build again.
This time it didn't fail with configuration error, but failed with "too
many files open" and somehow still using cmake 3.9.2 from Homebrew
installation:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a with CMake.app 3.9.2 in the path and Homebrew cmake 3.9.2 still installed · GitHub

So finally, I decided to use CMake.app only:
brew unlink cmake
Run again, and same 3 tests fail on same i386 simulator:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a CMake.app only · GitHub

Regards,
Max

On Wed, Sep 20, 2017 at 2:59 PM, Maksym Grebenets <mgrebenets@gmail.com> >> wrote:

I see.

I gave it another go:
swift-DEVELOPMENT-SNAPSHOT-2017-09-18-a Clean Build · GitHub

Looking at the build logs, I can definitely see llvm source code being
built.
I've looked up a bunch of llvm .cpp files in the build log,
e.g. LLVMTargetMachine.cpp.
Unless I'm not looking in the right place.

Still same 3 failures in the end.

I have noticed that in certain places Jenkings builds are using
/Applications/CMake.app, while on my machine it is
*/usr/local/Cellar/cmake/3.9.1/bin/cmake.*
So I've tried to upgrade to 3.9.2 and run same test command manually -
same failures.
Same for using CMake.app downloaded from cmake.org (3.9.2) version.

I've double checked python version that I have, it's 2.7.10, same as on
Jenkins instances.

I do have python3 installed in /usr/local/bin/python3, not sure if this
could be related.

Looking at the failing tests, it seems like all of them are related to
encoding.
The first one is related to Unicode 9 graphemes.
For a string ":us::canada::denmark::rainbow_flag:" the expected count is 4, while on my
machine it's 5.
Other failures are in CodableTests.swift, but all of them report an
error like:
Decoded URLComponents <//0.0.0.0> not equal to original <//0.0.0.0>
I can't really spot the difference, unless there's some invisible escape
character...

Last failing test is for NSValue bridging:
>>> check failed at
/Users/grebenma/Projects/oss/swift/swift/stdlib/private/StdlibUnittestFoundationExtras/StdlibUnittestFoundationExtras.swift,
line 130
>>> expected: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)
>>> actual: <00000000 00003140 00000000 00004340> (of type
NSConcreteValue)

It appears that all failures are for iPhone Simulator i386.
So it must be related to 32-bit platforms only.

I don't have much ideas on what it could be.
Is it something set in my bash (zsh) profile?
E.g. something in the path, or some 3rd party tool installed.
Are there any hints in the log that may point me in the right direction?

Thanks.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev