NSURLSession & CFNetwork

Hi Dave,

···

On Jan 19, 2016, at 5:49 AM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

I'd like to make progress and push this along.

On Philippe's list of work items from last week were:

> The master build script in swift/utils/build-script needs to be able to build GCD
> The master build script in swift/utils/build-script to ensure GCD
> and all required components are built before Foundation and the
> appropriate linker paths are passed to Foundation (see how XCTest is handled)

Currently libdispatch (and libpthread_workqueue) are outliers in that they use autotools instead of CMake. Do we need to convert dispatch (and eventually workqueue) to using CMake to get the build system integration? Or do you want to leave libdispatch using autotools to drive its build and get swift-build to invoke a more or less unchanged autotools build for dispatch?

I have time to work on this, but would appreciate some guidance on what the desired outcome is.

thanks,

--dave

I don’t think using autotools is a barrier to integration into the larger build script. swift-corelibs-foundation actually builds with a pile of custom python right now. I think we want to preserve as much of dispatch’s current build system as possible. The swift/utils/build-script process is pretty flexible (since it’s a bunch of scripts itself).

Thanks,
- Tony

Hi Dave,

What kind of layout does the Swift compiler emit for a Swift closure? If I call into a dispatch function that has a block argument, and that function calls Block_copy (from the blocks library), is everything retained correctly? The closure runtime.c calls out into a function emitted by the compiler to do that work. I would expect C code compiled by clang to follow the existing conventions but I’m not sure about Swift code compiled by swiftc.

- Tony

···

On Jan 22, 2016, at 2:22 PM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi,

Following up on the blocks discussion. I looked at the various Block header files and it looked there was a good chance things would just work. So, I did a quick & dirty hack of copying the .c and .h from foundation/closure into libdispatch/src/closure, building them, and linking with them instead of with libBlocksRuntime. All of the libdispatch tests that were working before, still work (not surprising), so it seems promising.

I speculate we would prefer to not have multiple copies of the blocks runtime used by an application. Therefore we need to get libdispatch and libFoundation to use the same one. The natural path would be to just build a blocks runtime library from the sources in foundation/closure and have foundation and dispatch link against that library and compile against its headers. I'd further guess that it would be preferred to leave the files in foundation/closure (not create some new project) and tweak the foundation build process so that it is possible to invoke it to build just the blocks runtime as a separate step.

Does this sound plausible? If so, I can start working on it. Or is some other approach better?

thanks,

--dave

<graycol.gif>Pierre Habouzit ---01/14/2016 10:09:28 PM---well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the ca

From: Pierre Habouzit <phabouzit@apple.com>
To: Philippe Hausler <phausler@apple.com>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 01/14/2016 10:09 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: phabouzit@apple.com

well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the captured function pointer.

This is used pervasively, through this macro (internal.h):

#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)

And we need to know the size of the Block_layout for a simple block (abused in _dispatch_block_get_data(), see inline_internal.h).

technically speaking dispatch only cares about the offsets/sizes and not about the rest of the layout.

-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.

So the change would be effectively something like this:

struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};

and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.

Then we would need to find some way to have the values for:

void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };

these guys emitted as objects as well so that the isa’s would be treated correctly.

I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 06:04:23 PM: > > > > Tony brought up an important point about the prep for integration > > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.

Hi,

I'm guessing RR expands to Retain Release?

I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.

thanks,

--dave

>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

FWIW today dispatch has that code when the objective-C runtime is used os OS X (in src/object.m).

  Block_callbacks_RR callbacks = {
    sizeof(Block_callbacks_RR),
    (void (*)(const void *))&objc_retain,
    (void (*)(const void *))&objc_release,
    (void (*)(const void *))&_os_objc_destructInstance
  };
  _Block_use_RR2(&callbacks);

we may have to set it up with a similar trick for swift and that would solve the retain/release concern IMO.

but that’s not the only concern, I sent a mail the other day, dispatch needs to:
- get to the function pointer inside the block (so know at which offset it is)
- if we want dispatch_block_create()d blocks to work, get some kind of support for it, to implement this function the right way:

DISPATCH_ALWAYS_INLINE
static inline dispatch_block_private_data_t
_dispatch_block_get_data(const dispatch_block_t db)
{
  if (!_dispatch_block_has_private_data(db)) {
    return NULL;
  }
  // Keep in sync with _dispatch_block_create implementation
  uint8_t *x = (uint8_t *)db;
  // x points to base of struct Block_layout
  x += sizeof(struct Block_layout);
  // x points to base of captured dispatch_block_private_data_s object
  dispatch_block_private_data_t dbpd = (dispatch_block_private_data_t)x;
  if (dbpd->dbpd_magic != DISPATCH_BLOCK_PRIVATE_DATA_MAGIC) {
    DISPATCH_CLIENT_CRASH(dbpd->dbpd_magic,
        "Corruption of dispatch block object");
  }
  return dbpd;
}

Though presumably, since dispatch_create_block() are created by the C++ runtime, these are C-like and probably don’t need anything. so the the only two issues should be retain/release of swift issued blocks, and the offset to the invoke function. If that offset could be the same in both worlds then that would be perfect.

-Pierre

···

On Jan 22, 2016, at 2:47 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Hi Dave,

What kind of layout does the Swift compiler emit for a Swift closure? If I call into a dispatch function that has a block argument, and that function calls Block_copy (from the blocks library), is everything retained correctly? The closure runtime.c calls out into a function emitted by the compiler to do that work. I would expect C code compiled by clang to follow the existing conventions but I’m not sure about Swift code compiled by swiftc.

- Tony

On Jan 22, 2016, at 2:22 PM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi,

Following up on the blocks discussion. I looked at the various Block header files and it looked there was a good chance things would just work. So, I did a quick & dirty hack of copying the .c and .h from foundation/closure into libdispatch/src/closure, building them, and linking with them instead of with libBlocksRuntime. All of the libdispatch tests that were working before, still work (not surprising), so it seems promising.

I speculate we would prefer to not have multiple copies of the blocks runtime used by an application. Therefore we need to get libdispatch and libFoundation to use the same one. The natural path would be to just build a blocks runtime library from the sources in foundation/closure and have foundation and dispatch link against that library and compile against its headers. I'd further guess that it would be preferred to leave the files in foundation/closure (not create some new project) and tweak the foundation build process so that it is possible to invoke it to build just the blocks runtime as a separate step.

Does this sound plausible? If so, I can start working on it. Or is some other approach better?

thanks,

--dave

<graycol.gif>Pierre Habouzit ---01/14/2016 10:09:28 PM---well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the ca

From: Pierre Habouzit <phabouzit@apple.com <mailto:phabouzit@apple.com>>
To: Philippe Hausler <phausler@apple.com <mailto:phausler@apple.com>>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>
Date: 01/14/2016 10:09 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: phabouzit@apple.com <mailto:phabouzit@apple.com>

well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the captured function pointer.

This is used pervasively, through this macro (internal.h):

#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)

And we need to know the size of the Block_layout for a simple block (abused in _dispatch_block_get_data(), see inline_internal.h).

technically speaking dispatch only cares about the offsets/sizes and not about the rest of the layout.

-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.

So the change would be effectively something like this:

struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};

and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.

Then we would need to find some way to have the values for:

void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };

these guys emitted as objects as well so that the isa’s would be treated correctly.

I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 06:04:23 PM: >> > >> > Tony brought up an important point about the prep for integration >> > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.

Hi,

I'm guessing RR expands to Retain Release?

I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.

thanks,

--dave

>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

FWIW today dispatch has that code when the objective-C runtime is used os OS X (in src/object.m).

  Block_callbacks_RR callbacks = {
    sizeof(Block_callbacks_RR),
    (void (*)(const void *))&objc_retain,
    (void (*)(const void *))&objc_release,
    (void (*)(const void *))&_os_objc_destructInstance
  };
  _Block_use_RR2(&callbacks);

we may have to set it up with a similar trick for swift and that would solve the retain/release concern IMO.

but that’s not the only concern, I sent a mail the other day, dispatch needs to:
- get to the function pointer inside the block (so know at which offset it is)
- if we want dispatch_block_create()d blocks to work, get some kind of support for it, to implement this function the right way:

DISPATCH_ALWAYS_INLINE
static inline dispatch_block_private_data_t
_dispatch_block_get_data(const dispatch_block_t db)
{
  if (!_dispatch_block_has_private_data(db)) {
    return NULL;
  }
  // Keep in sync with _dispatch_block_create implementation
  uint8_t *x = (uint8_t *)db;
  // x points to base of struct Block_layout
  x += sizeof(struct Block_layout);
  // x points to base of captured dispatch_block_private_data_s object
  dispatch_block_private_data_t dbpd = (dispatch_block_private_data_t)x;
  if (dbpd->dbpd_magic != DISPATCH_BLOCK_PRIVATE_DATA_MAGIC) {
    DISPATCH_CLIENT_CRASH(dbpd->dbpd_magic,
        "Corruption of dispatch block object");
  }
  return dbpd;
}

Though presumably, since dispatch_create_block() are created by the C++ runtime, these are C-like and probably don’t need anything. so the the only two issues should be retain/release of swift issued blocks, and the offset to the invoke function. If that offset could be the same in both worlds then that would be perfect.

-Pierre

If the swift compiler emits closures that look like Swift objects (and are retained/released with Swift ref counting functions), then the layout will probably be different.

In any case, I think the correct location for this stuff is not a library created by Foundation and imported by dispatch but instead exported functions from the Swift runtime (C code) that both libraries use as the truth for “how Swift blocks act when used in C code”.

- Tony

···

On Jan 22, 2016, at 2:59 PM, Pierre Habouzit <phabouzit@apple.com> wrote:

On Jan 22, 2016, at 2:47 PM, Tony Parker via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi Dave,

What kind of layout does the Swift compiler emit for a Swift closure? If I call into a dispatch function that has a block argument, and that function calls Block_copy (from the blocks library), is everything retained correctly? The closure runtime.c calls out into a function emitted by the compiler to do that work. I would expect C code compiled by clang to follow the existing conventions but I’m not sure about Swift code compiled by swiftc.

- Tony

On Jan 22, 2016, at 2:22 PM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

Hi,

Following up on the blocks discussion. I looked at the various Block header files and it looked there was a good chance things would just work. So, I did a quick & dirty hack of copying the .c and .h from foundation/closure into libdispatch/src/closure, building them, and linking with them instead of with libBlocksRuntime. All of the libdispatch tests that were working before, still work (not surprising), so it seems promising.

I speculate we would prefer to not have multiple copies of the blocks runtime used by an application. Therefore we need to get libdispatch and libFoundation to use the same one. The natural path would be to just build a blocks runtime library from the sources in foundation/closure and have foundation and dispatch link against that library and compile against its headers. I'd further guess that it would be preferred to leave the files in foundation/closure (not create some new project) and tweak the foundation build process so that it is possible to invoke it to build just the blocks runtime as a separate step.

Does this sound plausible? If so, I can start working on it. Or is some other approach better?

thanks,

--dave

<graycol.gif>Pierre Habouzit ---01/14/2016 10:09:28 PM---well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the ca

From: Pierre Habouzit <phabouzit@apple.com <mailto:phabouzit@apple.com>>
To: Philippe Hausler <phausler@apple.com <mailto:phausler@apple.com>>
Cc: David P Grove/Watson/IBM@IBMUS, Swift Core Libs <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>
Date: 01/14/2016 10:09 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to the rest of swift-corelibs
Sent by: phabouzit@apple.com <mailto:phabouzit@apple.com>

well dispatch also uses a bit of the blocks runtime because it uncorks the blocks and gets to the captured function pointer.

This is used pervasively, through this macro (internal.h):

#define _dispatch_Block_invoke(bb) \
((dispatch_function_t)((struct Block_layout *)bb)->invoke)

And we need to know the size of the Block_layout for a simple block (abused in _dispatch_block_get_data(), see inline_internal.h).

technically speaking dispatch only cares about the offsets/sizes and not about the rest of the layout.

-Pierre
On Jan 14, 2016, at 3:00 PM, Philippe Hausler via swift-corelibs-dev <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>> wrote:

As a note: our closure implementation in Foundation does NOT adhere to this (since it would mean that we would need to alter the c compiler to do so and that was not something that I have gotten to look at beyond a cursory glance.

So the change would be effectively something like this:

struct Block_layout {
void *isa;
if DEPLOYMENT_RUNTIME_SWIFT
uint32_t refCount;
uint32_t weakRefCount;
#else // not certain this branch can happen or not. we may need both
volatile int32_t flags; // contains ref count
int32_t reserved;
#endif
void (*invoke)(void *, ...);
struct Block_descriptor_1 *descriptor;
// imported variables
};

and making the retain and release methods for blocks just call swift_retain and swift_release. But since blocks may be emitted by the compiler we would need to make certain that the compiler had a flag to emit those with the new layout and emit correct values for the refCount fields to give it immortality for things like global static blocks etc.

Then we would need to find some way to have the values for:

void * _NSConcreteStackBlock[32] = { 0 };
void * _NSConcreteMallocBlock[32] = { 0 };
void * _NSConcreteAutoBlock[32] = { 0 };
void * _NSConcreteFinalizingBlock[32] = { 0 };
void * _NSConcreteGlobalBlock[32] = { 0 };
void * _NSConcreteWeakBlockVariable[32] = { 0 };

these guys emitted as objects as well so that the isa’s would be treated correctly.

I am not certain on how much of this lofty goal is actually needed; perhaps we should loop in one of the compiler team in on this to query what really needs to be done to properly support the @convention(block) syntax on linux. Perhaps I am overthinking this.
On Jan 14, 2016, at 2:49 PM, David P Grove <groved@us.ibm.com <mailto:groved@us.ibm.com>> wrote:
phausler@apple.com <mailto:phausler@apple.com> wrote on 01/13/2016 06:04:23 PM: >>> > >>> > Tony brought up an important point about the prep for integration >>> > this morning: the blocks runtime from libblocksruntime-dev will be
> incompatible with the layout of blocks referenced from swift since
> the object size there is 2 uint32’s bigger to handle the RR (we
> don’t have objc). Without modifying clang and the runtime we won’t
> have a way to properly handoff blocks back and forth w/o objc.

Hi,

I'm guessing RR expands to Retain Release?

I've started exploring the source in swift-corelibs-foundation/closure. Any chance there is a design doc laying out the object model used by the Swift implementation? I'm sure I could figure it out from the source code, but it would be nice to be able to cheat and start with a an overview.

thanks,

--dave

>
> I am currently taking a look at this to see what we can do to add an
> option to the clang code-gen to properly emit this structural
> difference. This isn’t a big issue for Foundation to CF since we
> don’t have many block APIs but dispatch is mostly blocks and that
> might pose an issue.
>

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Hi again. Sorry for my silence on this blocks/libdispatch conversation for
over a week.

I've been experimenting with taking the Swift and Objective C code from the
Dispatch overlay (swift/stdlib/public/SDK/Dispatch/) and getting it built
into libdispatch using libdispatch's existing autotools-based build system.
This is my understanding of the direction Tony suggested we pursue in pull
request #974 [1]. I have a branch of libdispatch [2] which can produce a
Dispatch.swiftmodule and libdispatch.so that can be used to satisfy an
import Dispatch request from a Swift program and allow Swift blocks to be
submitted to dispatch functions like dispatch_async. The resulting
executable runs successfully and uses libdispatch. So perhaps this is
getting close, but I had some questions.

One of the not nice things is that just like the existing overlay, it uses
@convention(block) to cause the compiler to generate code that bridges
between Swift blocks and the Objective C compatible blocks that dispatch is
expecting/using internally. This (a) is less efficient than I would like
and (b) forces the Swift program that imports Dispatch to be compiled with
-Xcc -fblocks -Xlinker -lBlocksRuntime. I don't see a good way of
avoiding this absent some compiler work that allow all of the C code in
dispatch & its test suite to be compiled to use Swift compatible blocks.
I'm assuming this would take a while, so we should try to not gate progress
on integrating dispatch with this compiler work. Is this approach
reasonable for now? Can we do better?

Another not nice thing is that the libdispatch build uses libtool to get
portable compilation of C/C++ code. I tried and failed to get libtool to
integrate reasonable with swiftc. As a result the rules that produce
Dispatch.o from Dispatch.swift so it can be linked into libdispatch.la is
not going to be very portable. If someone has better ideas, that would be
good...

I have a little more work to do so that the install target includes
DIspatch.swiftmodule and module.map in the set of things that are
installed. Is there more that needs to be done before a pull request to
get this into libdisaptch makes sense?

thanks,

--dave

[1] Enable Dispatch overlay on Linux by seabaylea · Pull Request #974 · apple/swift · GitHub
[2]
GitHub - dgrove-oss/swift-corelibs-libdispatch at swift-overlay

Thanks Tony,

I will work on putting something together along those lines (leave dispatch
internals unchanged; optionally invoke its build from build-script)

--dave

···

From: Tony Parker <anthony.parker@apple.com>
To: David P Grove/Watson/IBM@IBMUS
Cc: Swift Core Libs <swift-corelibs-dev@swift.org>
Date: 01/19/2016 01:11 PM
Subject: Re: [swift-corelibs-dev] libdispatch prep for integration to
            the rest of swift-corelibs
Sent by: anthony.parker@apple.com

Hi Dave,

      On Jan 19, 2016, at 5:49 AM, David P Grove via swift-corelibs-dev < swift-corelibs-dev@swift.org> wrote:

      I'd like to make progress and push this along.

      On Philippe's list of work items from last week were:

      > The master build script in swift/utils/build-script needs to be
      able to build GCD
      > The master build script in swift/utils/build-script to ensure GCD
      > and all required components are built before Foundation and the
      > appropriate linker paths are passed to Foundation (see how XCTest
      is handled)

      Currently libdispatch (and libpthread_workqueue) are outliers in that
      they use autotools instead of CMake. Do we need to convert dispatch
      (and eventually workqueue) to using CMake to get the build system
      integration? Or do you want to leave libdispatch using autotools to
      drive its build and get swift-build to invoke a more or less
      unchanged autotools build for dispatch?

      I have time to work on this, but would appreciate some guidance on
      what the desired outcome is.

      thanks,

      --dave

I don’t think using autotools is a barrier to integration into the larger
build script. swift-corelibs-foundation actually builds with a pile of
custom python right now. I think we want to preserve as much of dispatch’s
current build system as possible. The swift/utils/build-script process is
pretty flexible (since it’s a bunch of scripts itself).

Thanks,
- Tony

Hi Dave,

Thanks for the update! It seems like you’ve really made great progress on this.

Hi again. Sorry for my silence on this blocks/libdispatch conversation for over a week.

I've been experimenting with taking the Swift and Objective C code from the Dispatch overlay (swift/stdlib/public/SDK/Dispatch/) and getting it built into libdispatch using libdispatch's existing autotools-based build system. This is my understanding of the direction Tony suggested we pursue in pull request #974 [1]. I have a branch of libdispatch [2] which can produce a Dispatch.swiftmodule and libdispatch.so that can be used to satisfy an import Dispatch request from a Swift program and allow Swift blocks to be submitted to dispatch functions like dispatch_async. The resulting executable runs successfully and uses libdispatch. So perhaps this is getting close, but I had some questions.

One of the not nice things is that just like the existing overlay, it uses @convention(block) to cause the compiler to generate code that bridges between Swift blocks and the Objective C compatible blocks that dispatch is expecting/using internally. This (a) is less efficient than I would like and (b) forces the Swift program that imports Dispatch to be compiled with -Xcc -fblocks -Xlinker -lBlocksRuntime. I don't see a good way of avoiding this absent some compiler work that allow all of the C code in dispatch & its test suite to be compiled to use Swift compatible blocks. I'm assuming this would take a while, so we should try to not gate progress on integrating dispatch with this compiler work. Is this approach reasonable for now? Can we do better?

I think we can absolutely do better — but I’m also a believer in incremental development. If we get this started by requiring Swift apps to put in the special flags and link the runtime, then we can tackle that problem separately.

Another not nice thing is that the libdispatch build uses libtool to get portable compilation of C/C++ code. I tried and failed to get libtool to integrate reasonable with swiftc. As a result the rules that produce Dispatch.o from Dispatch.swift so it can be linked into libdispatch.la is not going to be very portable. If someone has better ideas, that would be good…

Hm, not sure about this one. If we can get support for our two Ubuntu platforms though, then I think others will be happy to help jump in and help solve portability problems.

I have a little more work to do so that the install target includes DIspatch.swiftmodule and module.map in the set of things that are installed. Is there more that needs to be done before a pull request to get this into libdisaptch makes sense?

thanks,

—dave

Let’s get the PR started and then make improvements from there.

Thanks again for your hard work on this! Swift on Linux will be much better with integrated support for dispatch. I can’t wait to remove some #ifdefs in Foundation.

- Tony

···

On Feb 2, 2016, at 3:05 PM, David P Grove via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:
[1] Enable Dispatch overlay on Linux by seabaylea · Pull Request #974 · apple/swift · GitHub
[2] GitHub - dgrove-oss/swift-corelibs-libdispatch at swift-overlay

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev