I’ve been thinking about the NSURLSession implementation and maybe we should include CFNetwork in the project ? We can also use it for NSStream implementation (CFStream dynamically link with CFNetwork).
It can save us a lot of time, but it comes with other dependencies (Security, SystemConfiguration …) and I don’t think the sources have been updated recently (http://www.opensource.apple.com/source/CFNetwork/\).
Antoine
Getting GCD up and rolling is definitely a bottleneck for a few things (not just NSURLSession).
There are a few things that we need as checkpoints:
A set of build instructions tailored for the swift project
GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15
We need to move Availability.h and friends down underneath both GCD and Foundation since they both use them
We need to move the blocks runtime out of Foundation and underneath or inside of GCD.
The master build script in swift/utils/build-script needs to be able to build GCD
The master build script in swift/utils/build-script to ensure GCD and all required components are built before Foundation and the appropriate linker paths are passed to Foundation (see how XCTest is handled)
On Jan 13, 2016, at 10:39 AM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
> From: Robert Stephen Thompson via swift-corelibs-dev <swift-
> corelibs-dev@swift.org>
>
> I might be wrong (I obviously don’t work for Apple or anything), but
> when I looked at it, the main stumbling block at this point is the
> lack of libdispatch. I mean, all the basic network stuff could be
> implemented with posix APIs no problem, but basically all of what
> makes NSURLSession et al actually cool comes from GCD. Which does
> make me wonder what the status of libdispatch is now… I did try
> building it on Linux and it appears to now build “out of the box”
> (after installing some additional dependencies), but tests hang.Hi,
I'm hoping you are seeing as a hang is just an unfortunate current ordering of test cases. For me, the tests run and 29 of 32 pass. It is currently running dispatch_select as the first test, which is perhaps a bad choice. The sub test "kevent read 1" of dispatch_select takes about 90 seconds with no output for me, so the test may appear to be hanging. Be patient ;)
> How
> will we go about integrating it with the corelibs-foundation project
> once it’s ready to try doing that?
>This is a great question (but for someone who knows more about corelibs-foundation than me....). I think libdispatch on Linux is getting close to being ready for this next step.
--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
Splitting this off to it’s own thread since this is not per-se networking related
>
> Getting GCD up and rolling is definitely a bottleneck for a few
> things (not just NSURLSession).Thanks Philippe! I was looking for things to do :)
>
> There are a few things that we need as checkpoints:
> A set of build instructions tailored for the swift project
> GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15I can take a look at Ubuntu 14. I know that one of the issues was the old version of libpthread_workqueue (under discussion for other reasons; we need fixes that aren't in the Ubuntu 15 version). I'll see if there is anything else standing in the way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c -fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an incomplete type 'struct dispatch_root_queue_context_s '
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com> wrote:
phausler@apple.com wrote on 01/13/2016 01:43:17 PM:
> We need to move Availability.h and friends down underneath both GCD
> and Foundation since they both use them
> We need to move the blocks runtime out of Foundation and underneath
> or inside of GCD.
> The master build script in swift/utils/build-script needs to be able
> to build GCD
> The master build script in swift/utils/build-script to ensure GCD
> and all required components are built before Foundation and the
> appropriate linker paths are passed to Foundation (see how XCTest is handled)
>--dave
it needs pthread wq changes that are still in review (the pull request is tagged as such).
-Pierre
On Jan 13, 2016, at 11:11 AM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
Splitting this off to it’s own thread since this is not per-se networking related
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 01:43:17 PM:
>
> Getting GCD up and rolling is definitely a bottleneck for a few
> things (not just NSURLSession).Thanks Philippe! I was looking for things to do :)
>
> There are a few things that we need as checkpoints:
> A set of build instructions tailored for the swift project
> GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15I can take a look at Ubuntu 14. I know that one of the issues was the old version of libpthread_workqueue (under discussion for other reasons; we need fixes that aren't in the Ubuntu 15 version). I'll see if there is anything else standing in the way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c -fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an incomplete type 'struct dispatch_root_queue_context_s '> We need to move Availability.h and friends down underneath both GCD
> and Foundation since they both use them
> We need to move the blocks runtime out of Foundation and underneath
> or inside of GCD.
> The master build script in swift/utils/build-script needs to be able
> to build GCD
> The master build script in swift/utils/build-script to ensure GCD
> and all required components are built before Foundation and the
> appropriate linker paths are passed to Foundation (see how XCTest is handled)
>--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
Almost all of the Foundation overlay stuff we have had to integrate directly into Foundation itself (which has made some of the build process a bit funky). The one to watch out for is @_silgen_name since that cannot be used outside of the stdlib.
On Jan 13, 2016, at 12:35 PM, Chris Bailey <BAILEYC@uk.ibm.com> wrote:
I believe another task for the list of libdispatch integration items is the implementation of the Dispatch overlay for Linux. I've been taking a stab at that based on changes made (and then reverted) by Slava in November - hopefully I'll have something worth looking at for it tomorrow.
From: Pierre Habouzit via swift-corelibs-dev <swift-corelibs-dev@swift.org>
To: Philippe Hausler <phausler@apple.com>
Cc: Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 13/01/2016 20:04
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: swift-corelibs-dev-bounces@swift.orgit needs pthread wq changes that are still in review (the pull request is tagged as such).
-Pierre
On Jan 13, 2016, at 11:11 AM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:
Splitting this off to it’s own thread since this is not per-se networking related
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 01:43:17 PM:
>
> Getting GCD up and rolling is definitely a bottleneck for a few
> things (not just NSURLSession).Thanks Philippe! I was looking for things to do :)
>
> There are a few things that we need as checkpoints:
> A set of build instructions tailored for the swift project
> GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15I can take a look at Ubuntu 14. I know that one of the issues was the old version of libpthread_workqueue (under discussion for other reasons; we need fixes that aren't in the Ubuntu 15 version). I'll see if there is anything else standing in the way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c -fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an incomplete type 'struct dispatch_root_queue_context_s '> We need to move Availability.h and friends down underneath both GCD
> and Foundation since they both use them
> We need to move the blocks runtime out of Foundation and underneath
> or inside of GCD.
> The master build script in swift/utils/build-script needs to be able
> to build GCD
> The master build script in swift/utils/build-script to ensure GCD
> and all required components are built before Foundation and the
> appropriate linker paths are passed to Foundation (see how XCTest is handled)
>--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
I might be wrong (I obviously don’t work for Apple or anything), but when I looked at it, the main stumbling block at this point is the lack of libdispatch. I mean, all the basic network stuff could be implemented with posix APIs no problem, but basically all of what makes NSURLSession et al actually cool comes from GCD. Which does make me wonder what the status of libdispatch is now… I did try building it on Linux and it appears to now build “out of the box” (after installing some additional dependencies), but tests hang. How will we go about integrating it with the corelibs-foundation project once it’s ready to try doing that?
Thanks,
Robert
On Jan 13, 2016, at 1:02 PM, Antoine Palazzolo via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
I’ve been thinking about the NSURLSession implementation and maybe we should include CFNetwork in the project ? We can also use it for NSStream implementation (CFStream dynamically link with CFNetwork).
It can save us a lot of time, but it comes with other dependencies (Security, SystemConfiguration …) and I don’t think the sources have been updated recently (http://www.opensource.apple.com/source/CFNetwork/\).Antoine
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
From: Robert Stephen Thompson via swift-corelibs-dev <swift-
corelibs-dev@swift.org>I might be wrong (I obviously don’t work for Apple or anything), but
when I looked at it, the main stumbling block at this point is the
lack of libdispatch. I mean, all the basic network stuff could be
implemented with posix APIs no problem, but basically all of what
makes NSURLSession et al actually cool comes from GCD. Which does
make me wonder what the status of libdispatch is now… I did try
building it on Linux and it appears to now build “out of the box”
(after installing some additional dependencies), but tests hang.
Hi,
I'm hoping you are seeing as a hang is just an unfortunate current
ordering of test cases. For me, the tests run and 29 of 32 pass. It is
currently running dispatch_select as the first test, which is perhaps a bad
choice. The sub test "kevent read 1" of dispatch_select takes about 90
seconds with no output for me, so the test may appear to be hanging. Be
patient ;)
How
will we go about integrating it with the corelibs-foundation project
once it’s ready to try doing that?
This is a great question (but for someone who knows more about
corelibs-foundation than me....). I think libdispatch on Linux is getting
close to being ready for this next step.
--dave
Getting GCD up and rolling is definitely a bottleneck for a few
things (not just NSURLSession).
Thanks Philippe! I was looking for things to do :)
There are a few things that we need as checkpoints:
A set of build instructions tailored for the swift project
GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15
I can take a look at Ubuntu 14. I know that one of the issues was the old
version of libpthread_workqueue (under discussion for other reasons; we
need fixes that aren't in the Ubuntu 15 version). I'll see if there is
anything else standing in the way for Ubuntu 14.
We need to move Availability.h and friends down underneath both GCD
and Foundation since they both use them
We need to move the blocks runtime out of Foundation and underneath
or inside of GCD.
The master build script in swift/utils/build-script needs to be able
to build GCD
The master build script in swift/utils/build-script to ensure GCD
and all required components are built before Foundation and the
appropriate linker paths are passed to Foundation (see how XCTest is
handled)
--dave
phausler@apple.com wrote on 01/13/2016 01:43:17 PM:
So hey, you’re right! What’s more, it turned out that Ubuntu doesn’t necessarily have a /usr/share/dict/words file installed by default. Once I put something there (I’m sure anything would do, but I actually installed the wamerican-large package because why not), I actually had all 32 tests pass! So yeah, looking pretty good heh.
Thanks,
Robert Thompson
On Jan 13, 2016, at 1:39 PM, David P Grove <groved@us.ibm.com> wrote:
Hi,
I'm hoping you are seeing as a hang is just an unfortunate current ordering of test cases. For me, the tests run and 29 of 32 pass. It is currently running dispatch_select as the first test, which is perhaps a bad choice. The sub test "kevent read 1" of dispatch_select takes about 90 seconds with no output for me, so the test may appear to be hanging. Be patient ;)
Tony brought up an important point about the prep for integration this morning: the blocks runtime from libblocksruntime-dev will be incompatible with the layout of blocks referenced from swift since the object size there is 2 uint32’s bigger to handle the RR (we don’t have objc). Without modifying clang and the runtime we won’t have a way to properly handoff blocks back and forth w/o objc.
I am currently taking a look at this to see what we can do to add an option to the clang code-gen to properly emit this structural difference. This isn’t a big issue for Foundation to CF since we don’t have many block APIs but dispatch is mostly blocks and that might pose an issue.
On Jan 13, 2016, at 2:55 PM, David P Grove <groved@us.ibm.com> wrote:
On the branch that the pthread wq pull request comes from, libdispatch builds on Ubuntu 14 and tests run.
So, no more problems lurking behind that one.
--dave
<graycol.gif>Pierre Habouzit ---01/13/2016 02:53:53 PM---it needs pthread wq changes that are still in review (the pull request is tagged as such). -Pierre
From: Pierre Habouzit <phabouzit@apple.com>
To: Philippe Hausler <phausler@apple.com>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 01/13/2016 02:53 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: phabouzit@apple.comit needs pthread wq changes that are still in review (the pull request is tagged as such).
-Pierre
On Jan 13, 2016, at 11:11 AM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:Splitting this off to it’s own thread since this is not per-se networking related
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 01:43:17 PM:
>
> Getting GCD up and rolling is definitely a bottleneck for a few
> things (not just NSURLSession).Thanks Philippe! I was looking for things to do :)
>
> There are a few things that we need as checkpoints:
> A set of build instructions tailored for the swift project
> GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15I can take a look at Ubuntu 14. I know that one of the issues was the old version of libpthread_workqueue (under discussion for other reasons; we need fixes that aren't in the Ubuntu 15 version). I'll see if there is anything else standing in the way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c -fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an incomplete type 'struct dispatch_root_queue_context_s '> We need to move Availability.h and friends down underneath both GCD
> and Foundation since they both use them
> We need to move the blocks runtime out of Foundation and underneath
> or inside of GCD.
> The master build script in swift/utils/build-script needs to be able
> to build GCD
> The master build script in swift/utils/build-script to ensure GCD
> and all required components are built before Foundation and the
> appropriate linker paths are passed to Foundation (see how XCTest is handled)
>--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
On the branch that the pthread wq pull request comes from, libdispatch
builds on Ubuntu 14 and tests run.
So, no more problems lurking behind that one.
--dave
From: Pierre Habouzit <phabouzit@apple.com>
To: Philippe Hausler <phausler@apple.com>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs
<swift-corelibs-dev@swift.org>
Date: 01/13/2016 02:53 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to
the rest of swift-corelibs
Sent by: phabouzit@apple.com
it needs pthread wq changes that are still in review (the pull request is
tagged as such).
-Pierre
On Jan 13, 2016, at 11:11 AM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
Splitting this off to it’s own thread since this is not per-se
networking related
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com> wrote:
phausler@apple.com wrote on 01/13/2016 01:43:17 PM:
>
> Getting GCD up and rolling is definitely a bottleneck for a
few
> things (not just NSURLSession).
Thanks Philippe! I was looking for things to do :)
>
> There are a few things that we need as checkpoints:
> A set of build instructions tailored for the swift project
> GCD needs to be able to build on Ubuntu 14 as well as Ubuntu
15
I can take a look at Ubuntu 14. I know that one of the issues
was the old version of libpthread_workqueue (under discussion
for other reasons; we need fixes that aren't in the Ubuntu 15
version). I'll see if there is anything else standing in the
way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H
-I. -I../config -I.. -I.. -I../private -I../os
-DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden
-momit-leaf-frame-pointer -I/usr/include/kqueue
-isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o
queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I..
-I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall
-fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue
-isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c
-fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier
'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier
'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an
incomplete type 'struct dispatch_root_queue_context_s '
> We need to move Availability.h and friends down underneath
both GCD
> and Foundation since they both use them
> We need to move the blocks runtime out of Foundation and
underneath
> or inside of GCD.
> The master build script in swift/utils/build-script needs to
be able
> to build GCD
> The master build script in swift/utils/build-script to ensure
GCD
> and all required components are built before Foundation and
the
> appropriate linker paths are passed to Foundation (see how
XCTest is handled)
>
--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
Basic tcp networking can be implemented with with posix api, but the http/https protocol needs an implementation?
Antoine
On 13 janv. 2016, at 7:16 PM, Robert Stephen Thompson <robert.thompson@willowtreeapps.com> wrote:
I might be wrong (I obviously don’t work for Apple or anything), but when I looked at it, the main stumbling block at this point is the lack of libdispatch. I mean, all the basic network stuff could be implemented with posix APIs no problem, but basically all of what makes NSURLSession et al actually cool comes from GCD. Which does make me wonder what the status of libdispatch is now… I did try building it on Linux and it appears to now build “out of the box” (after installing some additional dependencies), but tests hang. How will we go about integrating it with the corelibs-foundation project once it’s ready to try doing that?
Thanks,
RobertOn Jan 13, 2016, at 1:02 PM, Antoine Palazzolo via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
I’ve been thinking about the NSURLSession implementation and maybe we should include CFNetwork in the project ? We can also use it for NSStream implementation (CFStream dynamically link with CFNetwork).
It can save us a lot of time, but it comes with other dependencies (Security, SystemConfiguration …) and I don’t think the sources have been updated recently (http://www.opensource.apple.com/source/CFNetwork/\).Antoine
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
I believe another task for the list of libdispatch integration items is
the implementation of the Dispatch overlay for Linux. I've been taking a
stab at that based on changes made (and then reverted) by Slava in
November - hopefully I'll have something worth looking at for it tomorrow.
<swift-corelibs-dev@swift.org>
From: Pierre Habouzit via swift-corelibs-dev
To: Philippe Hausler <phausler@apple.com>
Cc: Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 13/01/2016 20:04
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration
to the rest of swift-corelibs
Sent by: swift-corelibs-dev-bounces@swift.org
it needs pthread wq changes that are still in review (the pull request is
tagged as such).
-Pierre
On Jan 13, 2016, at 11:11 AM, Philippe Hausler via swift-corelibs-dev < swift-corelibs-dev@swift.org> wrote:
Splitting this off to it’s own thread since this is not per-se networking
related
On Jan 13, 2016, at 11:07 AM, David P Grove <groved@us.ibm.com> wrote:
phausler@apple.com wrote on 01/13/2016 01:43:17 PM:
Getting GCD up and rolling is definitely a bottleneck for a few
things (not just NSURLSession).
Thanks Philippe! I was looking for things to do :)
There are a few things that we need as checkpoints:
A set of build instructions tailored for the swift project
GCD needs to be able to build on Ubuntu 14 as well as Ubuntu 15
I can take a look at Ubuntu 14. I know that one of the issues was the old
version of libpthread_workqueue (under discussion for other reasons; we
need fixes that aren't in the Ubuntu 15 version). I'll see if there is
anything else standing in the way for Ubuntu 14.
I am getting a build failure on 14:
/bin/bash ../libtool --tag=CC --mode=compile clang -DHAVE_CONFIG_H -I.
-I../config -I.. -I.. -I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall
-fvisibility=hidden -momit-leaf-frame-pointer -I/usr/include/kqueue
-isystem /usr/include/bsd -DLIBBSD_OVERLAY -fblocks -g -O2 -c -o
queue.lo queue.c
libtool: compile: clang -DHAVE_CONFIG_H -I. -I../config -I.. -I..
-I../private -I../os -DDISPATCH_USE_DTRACE=0 -Wall -fvisibility=hidden
-momit-leaf-frame-pointer -I/usr/include/kqueue -isystem /usr/include/bsd
-DLIBBSD_OVERLAY -fblocks -g -O2 -c queue.c -fPIC -DPIC -o .libs/queue.o
queue.c:216:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:227:22: error: use of undeclared identifier 'WORKQ_BG_PRIOQUEUE'
.dgq_wq_priority = WORKQ_BG_PRIOQUEUE,
^
queue.c:928:18: error: invalid application of 'sizeof' to an incomplete
type 'struct dispatch_root_queue_context_s '
We need to move Availability.h and friends down underneath both GCD
and Foundation since they both use them
We need to move the blocks runtime out of Foundation and underneath
or inside of GCD.
The master build script in swift/utils/build-script needs to be able
to build GCD
The master build script in swift/utils/build-script to ensure GCD
and all required components are built before Foundation and the
appropriate linker paths are passed to Foundation (see how XCTest is
handled)
--dave
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.
So the change would be effectively something like this:
struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};
and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.
Then we would need to find some way to have the values for:
void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };
these guys emitted as objects as well so that the isa’s would be treated correctly.
I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com> wrote:
phausler@apple.com wrote on 01/13/2016 06:04:23 PM: > > > > Tony brought up an important point about the prep for integration > > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.Hi,
I'm guessing RR expands to Retain Release?
I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.
thanks,
--dave
>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>
incompatible with the layout of blocks referenced from swift since
the object size there is 2 uint32’s bigger to handle the RR (we
don’t have objc). Without modifying clang and the runtime we won’t
have a way to properly handoff blocks back and forth w/o objc.
Hi,
I'm guessing RR expands to Retain Release?
I've started exploring the source in
swift-corelibs-foundation/closure. Any chance there is a design doc laying
out the object model used by the Swift implementation? I'm sure I could
figure it out from the source code, but it would be nice to be able to
cheat and start with a an overview.
thanks,
--dave
phausler@apple.com wrote on 01/13/2016 06:04:23 PM: > > Tony brought up an important point about the prep for integration > this morning: the blocks runtime from libblocksruntime-dev will be
I am currently taking a look at this to see what we can do to add an
option to the clang code-gen to properly emit this structural
difference. This isn’t a big issue for Foundation to CF since we
don’t have many block APIs but dispatch is mostly blocks and that
might pose an issue.
There are a few other things to consider: SSL for streams, certificates, and asynchronous host name resolution.
On Jan 13, 2016, at 10:57 AM, Antoine Palazzolo via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
Basic tcp networking can be implemented with with posix api, but the http/https protocol needs an implementation?
Antoine
On 13 janv. 2016, at 7:16 PM, Robert Stephen Thompson <robert.thompson@willowtreeapps.com> wrote:
I might be wrong (I obviously don’t work for Apple or anything), but when I looked at it, the main stumbling block at this point is the lack of libdispatch. I mean, all the basic network stuff could be implemented with posix APIs no problem, but basically all of what makes NSURLSession et al actually cool comes from GCD. Which does make me wonder what the status of libdispatch is now… I did try building it on Linux and it appears to now build “out of the box” (after installing some additional dependencies), but tests hang. How will we go about integrating it with the corelibs-foundation project once it’s ready to try doing that?
Thanks,
RobertOn Jan 13, 2016, at 1:02 PM, Antoine Palazzolo via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
I’ve been thinking about the NSURLSession implementation and maybe we should include CFNetwork in the project ? We can also use it for NSStream implementation (CFStream dynamically link with CFNetwork).
It can save us a lot of time, but it comes with other dependencies (Security, SystemConfiguration …) and I don’t think the sources have been updated recently (http://www.opensource.apple.com/source/CFNetwork/\).Antoine
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the captured function pointer.
This is used pervasively, through this macro (internal.h):
#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)
And we need to know the size of the Block_layout for a simple block (abused in _dispatch_block_get_data(), see inline_internal.h).
technically speaking dispatch only cares about the offsets/sizes and not about the rest of the layout.
-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.
So the change would be effectively something like this:
struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.
Then we would need to find some way to have the values for:
void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };these guys emitted as objects as well so that the isa’s would be treated correctly.
I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 06:04:23 PM: >> > >> > Tony brought up an important point about the prep for integration >> > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.Hi,
I'm guessing RR expands to Retain Release?
I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.
thanks,
--dave
>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
I'd like to make progress and push this along.
On Philippe's list of work items from last week were:
The master build script in swift/utils/build-script needs to be able to
build GCD
The master build script in swift/utils/build-script to ensure GCD
and all required components are built before Foundation and the
appropriate linker paths are passed to Foundation (see how XCTest is
handled)
Currently libdispatch (and libpthread_workqueue) are outliers in that they
use autotools instead of CMake. Do we need to convert dispatch (and
eventually workqueue) to using CMake to get the build system integration?
Or do you want to leave libdispatch using autotools to drive its build and
get swift-build to invoke a more or less unchanged autotools build for
dispatch?
I have time to work on this, but would appreciate some guidance on what the
desired outcome is.
thanks,
--dave
Hi,
Following up on the blocks discussion. I looked at the various Block header files and it looked there was a good chance things would just work. So, I did a quick & dirty hack of copying the .c and .h from foundation/closure into libdispatch/src/closure, building them, and linking with them instead of with libBlocksRuntime. All of the libdispatch tests that were working before, still work (not surprising), so it seems promising.
I speculate we would prefer to not have multiple copies of the blocks runtime used by an application. Therefore we need to get libdispatch and libFoundation to use the same one. The natural path would be to just build a blocks runtime library from the sources in foundation/closure and have foundation and dispatch link against that library and compile against its headers. I'd further guess that it would be preferred to leave the files in foundation/closure (not create some new project) and tweak the foundation build process so that it is possible to invoke it to build just the blocks runtime as a separate step.
Does this sound plausible? If so, I can start working on it. Or is some other approach better?
yes, I think we should have libdispatch build-system have an option to use the libclosure built by swift, as opposed to the libbllocksruntime.
Hopefully the blocks runtime for swift understands C closures generated by clang too (but I would assume the reverse is not true).
the things that will IMO require adaptaitons is if you call dispatch_async() from swift, that will likely break because dispatch only understands C closures today and my understanding is that the swift generated ones have a different layout.
If the legacy blocks runtime is used, then you don’t get the swift variants working for you and that’s fine.
Foundation/Swift people, is my expectation valid?
On Jan 22, 2016, at 2:22 PM, David P Grove <groved@us.ibm.com> wrote:
thanks,
--dave
<graycol.gif>Pierre Habouzit ---01/14/2016 10:09:28 PM---well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the ca
From: Pierre Habouzit <phabouzit@apple.com>
To: Philippe Hausler <phausler@apple.com>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 01/14/2016 10:09 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: phabouzit@apple.comwell dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the captured function pointer.
This is used pervasively, through this macro (internal.h):
#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)And we need to know the size of the Block_layout for a simple block (abused in _dispatch_block_get_data(), see inline_internal.h).
technically speaking dispatch only cares about the offsets/sizes and not about the rest of the layout.
-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.
So the change would be effectively something like this:
struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.
Then we would need to find some way to have the values for:
void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };these guys emitted as objects as well so that the isa’s would be treated correctly.
I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 06:04:23 PM: > > > > Tony brought up an important point about the prep for integration > > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.Hi,
I'm guessing RR expands to Retain Release?
I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.
thanks,
--dave
>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
Hi,
Following up on the blocks discussion. I looked at the various Block
header files and it looked there was a good chance things would just work.
So, I did a quick & dirty hack of copying the .c and .h from
foundation/closure into libdispatch/src/closure, building them, and linking
with them instead of with libBlocksRuntime. All of the libdispatch tests
that were working before, still work (not surprising), so it seems
promising.
I speculate we would prefer to not have multiple copies of the blocks
runtime used by an application. Therefore we need to get libdispatch and
libFoundation to use the same one. The natural path would be to just
build a blocks runtime library from the sources in foundation/closure and
have foundation and dispatch link against that library and compile against
its headers. I'd further guess that it would be preferred to leave the
files in foundation/closure (not create some new project) and tweak the
foundation build process so that it is possible to invoke it to build just
the blocks runtime as a separate step.
Does this sound plausible? If so, I can start working on it. Or is some
other approach better?
thanks,
--dave
From: Pierre Habouzit <phabouzit@apple.com>
To: Philippe Hausler <phausler@apple.com>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs
<swift-corelibs-dev@swift.org>
Date: 01/14/2016 10:09 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to
the rest of swift-corelibs
Sent by: phabouzit@apple.com
well dispatch also uses a bit of the blocks runtime because it uncorks the
blocks and gets to the captured function pointer.
This is used pervasively, through this macro (internal.h):
#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)
And we need to know the size of the Block_layout for a simple block (abused
in _dispatch_block_get_data(), see inline_internal.h).
technically speaking dispatch only cares about the offsets/sizes and not
about the rest of the layout.
-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
As a note: our closure implementation in Foundation does NOT adhere
to this (since it would mean that we would need to alter the c
compiler to do so and that was not something that I have gotten to
look at beyond a cursory glance.
So the change would be effectively something like this:
struct Block_layout {
void *isa;
#if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};
and making the retain and release methods for blocks just call
swift_retain and swift_release. But since blocks may be emitted by
the compiler we would need to make certain that the compiler had a
flag to emit those with the new layout and emit correct values for
the refCount fields to give it immortality for things like global
static blocks etc.
Then we would need to find some way to have the values for:
void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };
these guys emitted as objects as well so that the isa’s would be
treated correctly.
I am not certain on how much of this lofty goal is actually needed;
perhaps we should loop in one of the compiler team in on this to
query what really needs to be done to properly support the
@convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com> wrote:
phausler@apple.com wrote on 01/13/2016 06:04:23 PM: > > Tony brought up an important point about the prep for integration > this morning: the blocks runtime from libblocksruntime-dev
will be
> incompatible with the layout of blocks referenced from swift
since
> the object size there is 2 uint32’s bigger to handle the RR
(we
> don’t have objc). Without modifying clang and the runtime we
won’t
> have a way to properly handoff blocks back and forth w/o
objc.
Hi,
I'm guessing RR expands to Retain Release?
I've started exploring the source in
swift-corelibs-foundation/closure. Any chance there is a
design doc laying out the object model used by the Swift
implementation? I'm sure I could figure it out from the source
code, but it would be nice to be able to cheat and start with a
an overview.
thanks,
--dave
>
> I am currently taking a look at this to see what we can do to
add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since
we
> don’t have many block APIs but dispatch is mostly blocks and
that
> might pose an issue.
>
_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev