swift (ABI) and Windows

Development Compiler
1

Hi,

I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics should
get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is dependent
on how the module(s) are being built. For example, something may change
from the exported storage to default if being built into a static library
rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build system
so that we know whether a given SIL module is external or internal.

Given that this would potentially effect ABI stability, it seems like this
is a good time to tackle it so that we can push this into the resilience
work that is being done for swift 3.

I would appreciate any pointers and suggestions as to how to best go about
handling this.

···

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

2

Hey, Saleem. How do you expect this to differ from normal symbol visibility? It seems to me that in a shared library, any public symbol is either exported or imported, depending on whether it has a definition, and any non-public symbol is default. (Unlike C, we expect to have sensible rules here.) I guess there's the difference between "a public symbol from elsewhere in this library" and "a public symbol from some other library". Is that it?

The difference between static and shared libraries seems unfortunate to have to expose to IRGen, but we may end up needing that anyway to handle Mach-O/ELF-style symbol visibility.

Jordan

···

On Apr 6, 2016, at 10:21, Saleem Abdulrasool via swift-dev <swift-dev@swift.org> wrote:

Hi,

I was playing around with the idea of swift and Windows since there are some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules (DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks in Windows parlance). Fortunately, LLVM has a nice way to model this: GlobalValues have an associated "DLLStorageClass" which indicates whether something is "imported" (provided by an external module), "exported" (provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics should get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is dependent on how the module(s) are being built. For example, something may change from the exported storage to default if being built into a static library rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build system so that we know whether a given SIL module is external or internal.

Given that this would potentially effect ABI stability, it seems like this is a good time to tackle it so that we can push this into the resilience work that is being done for swift 3.

I would appreciate any pointers and suggestions as to how to best go about handling this.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

3

As Jordan noted, we probably want to thread this information through for ELF and Mach-O builds too, for a couple of reasons. One, if you're building a static library as opposed to a .so or .dylib, it's not desirable to reexport that static library's API by default through any executables or dynamic libraries that use it. We currently get this wrong; if the compiler knew it was building for a static library, it could give public symbols in the .a 'hidden' visibility so that they do the right thing when linked into dylibs. Second, ELF's default visibility is problematic for Swift, since it allows default-visibility symbols to be interposed at load time by other dynamic libraries. This imposes a performance penalty on shared libraries accessing their own data, interferes with some of the load-time optimizations we do with metadata that assume we can hardcode relative references within a linkage unit, and is flat-out dangerous in the face of interprocedural optimization. For these reasons, we export symbols with "protected" visibility when we're emitting them as part of the current dylib, but import external public symbols with default visibility. Since we already need to track this distinction, it should be possible to approximate the right thing for Windows by using dllexport where we set protected visibility on ELF, or dllimport otherwise.

-Joe

···

On Apr 6, 2016, at 10:21 AM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org> wrote:

Hi,

I was playing around with the idea of swift and Windows since there are some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules (DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks in Windows parlance). Fortunately, LLVM has a nice way to model this: GlobalValues have an associated "DLLStorageClass" which indicates whether something is "imported" (provided by an external module), "exported" (provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics should get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is dependent on how the module(s) are being built. For example, something may change from the exported storage to default if being built into a static library rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build system so that we know whether a given SIL module is external or internal.

Given that this would potentially effect ABI stability, it seems like this is a good time to tackle it so that we can push this into the resilience work that is being done for swift 3.

I would appreciate any pointers and suggestions as to how to best go about handling this.

4

Hi,

I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is dependent
on how the module(s) are being built. For example, something may change
from the exported storage to default if being built into a static library
rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build system
so that we know whether a given SIL module is external or internal.

To the DLL Storage semantics support, Ive taken a quick first stab at it.
Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a Pull
Request at https://github.com/apple/swift/pull/2080 .

However, as I expected, this is going to cause problems for building some
of the core libraries. In particular, there are mismatches between what
gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.

It seems to me, at least initially, that we need a way to treat SwiftModule
as a container (a la llvm::Module) and indicate which of the TopLevelDecls
are meant to be a single "module" (DSO, DLL, whatever you want to call it)
so that we can properly track the DLL storage associated with them. Am I
confusing something there?

Is there a preference on a means to handle this?

···

On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

Given that this would potentially effect ABI stability, it seems like this
is a good time to tackle it so that we can push this into the resilience
work that is being done for swift 3.

I would appreciate any pointers and suggestions as to how to best go about
handling this.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

5

Hey, Saleem. How do you expect this to differ from normal symbol
visibility? It seems to me that in a shared library, any public symbol is
either exported or imported, depending on whether it has a definition, and
any non-public symbol is default. (Unlike C, we expect to have sensible
rules here.) I guess there's the difference between "a public symbol from
elsewhere in this library" and "a public symbol from some other library".
Is that it?

Well, there are four cases to consider:
- externally available: imported
- defined (and available for others): exported
- defined (statically): default -- won't even show up, so this is a no-op
- defined (non-statically defined for internal use): default

The thing is that there is no modeling for internal symbols which other
shared objects can use. The closest thing you can do is anonymize the
symbol (so you don't have a name that you can call it by, but you have an
integral ID).

A public symbol from elsewhere in this library is treated as a local symbol
defined elsewhere in this library. However, a public symbol from another
library is treated differently. These get a local symbol which is
synthesized to perform the indirect addressing. The symbol uses the form
__USER_LABEL_PREFIX__ ## _imp_ ## name. This will contain the real address
or an indirect jump for the public symbol.

The difference between static and shared libraries seems unfortunate to

have to expose to IRGen, but we may end up needing that anyway to handle
Mach-O/ELF-style symbol visibility.

Yes, it is unfortunate, but sounds like it could end up being beneficial.

···

On Wed, Apr 6, 2016 at 11:18 AM, Jordan Rose <jordan_rose@apple.com> wrote:

Jordan

> On Apr 6, 2016, at 10:21, Saleem Abdulrasool via swift-dev < > swift-dev@swift.org> wrote:
>
> Hi,
>
> I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> Given that this would potentially effect ABI stability, it seems like
this is a good time to tackle it so that we can push this into the
resilience work that is being done for swift 3.
>
> I would appreciate any pointers and suggestions as to how to best go
about handling this.
>
> --
> Saleem Abdulrasool
> compnerd (at) compnerd (dot) org
> _______________________________________________
> swift-dev mailing list
> swift-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

6

In concept, it's more-or-less the same as symbol visibility. My understanding is that the PE/COFF model is simply a little less forgiving about getting it wrong.

As you say, this is something we want to be getting right anyway because it allows IRGen to use more efficient code patterns.

John.

···

On Apr 6, 2016, at 11:18 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

Hey, Saleem. How do you expect this to differ from normal symbol visibility? It seems to me that in a shared library, any public symbol is either exported or imported, depending on whether it has a definition, and any non-public symbol is default. (Unlike C, we expect to have sensible rules here.) I guess there's the difference between "a public symbol from elsewhere in this library" and "a public symbol from some other library". Is that it?

7

>
> Hi,
>
> I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> Given that this would potentially effect ABI stability, it seems like
this is a good time to tackle it so that we can push this into the
resilience work that is being done for swift 3.
>
> I would appreciate any pointers and suggestions as to how to best go
about handling this.

As Jordan noted, we probably want to thread this information through for
ELF and Mach-O builds too, for a couple of reasons. One, if you're building
a static library as opposed to a .so or .dylib, it's not desirable to
reexport that static library's API by default through any executables or
dynamic libraries that use it. We currently get this wrong; if the compiler
knew it was building for a static library, it could give public symbols in
the .a 'hidden' visibility so that they do the right thing when linked into
dylibs. Second, ELF's default visibility is problematic for Swift, since it
allows default-visibility symbols to be interposed at load time by other
dynamic libraries. This imposes a performance penalty on shared libraries
accessing their own data, interferes with some of the load-time
optimizations we do with metadata that assume we can hardcode relative
references within a linkage unit, and is flat-out dangerous in the face of
interprocedural optimization. For these reasons, we export symbols with
"protected" visibility when we're emitting them as part of the current
dylib, but import external public symbols with default visibility. Since we
already need to track this distinction, it should be possible to
approximate the right thing for Windows by using dllexport where we set
protected visibility on ELF, or dllimport otherwise.

Yeah, I can absolutely see this information being useful in a non-COFF
environment. I think that approaching it where we can implement it
generically enough to be shared across COFF, ELF, (and possibly MachO? --
sorry, not as familiar with the object format as the other two), would be
the best thing to do. But, I wanted to raise the issue on swift-dev so
that all the concerns can get ironed out before I take a stab at doing
something like this. Not to mention, some pointers would probably be time
saving as well.

···

On Wed, Apr 6, 2016 at 11:39 AM, Joe Groff <jgroff@apple.com> wrote:

> On Apr 6, 2016, at 10:21 AM, Saleem Abdulrasool via swift-dev < > swift-dev@swift.org> wrote:

-Joe

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

8

Hi,

I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.

To the DLL Storage semantics support, Ive taken a quick first stab at it.
Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a
Pull Request at https://github.com/apple/swift/pull/2080 .

However, as I expected, this is going to cause problems for building some
of the core libraries. In particular, there are mismatches between what
gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.

Playing around with this, I was trying to special case the building of the
standard library (as the runtime will be statically linked into it, the
symbols that it is expecting to be externally available are actually
private linkage. Not hacking up the compiler like this causes issues since
there are inverse dependencies (swiftCore gets dllimport interfaces from
swiftRuntime, which has dependencies on swiftCore). The crux of the
problem is that we do not have a way to represent that in swift.

The easiest answer that seems to come to mind is to actually introduce an
attribute to indicate that an interface is part of a specific module and
assume that everything else is locally defined. This would also
potentially allow us to handle things like @inline(always) @transparent
interfaces which get imported to ensure that a static inline function is
given local visibility rather than a DLL Import storage.

Unfortunately, I believe that currently Im stuck as I do not have a good
way to determine what type of dll storage class a symbol should be given
(since currently, theres no way to determine if we will have a symbol
available locally or not when actually linking).

···

On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool <compnerd@compnerd.org > > wrote:

It seems to me, at least initially, that we need a way to treat
SwiftModule as a container (a la llvm::Module) and indicate which of the
TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you
want to call it) so that we can properly track the DLL storage associated
with them. Am I confusing something there?

Is there a preference on a means to handle this?

Given that this would potentially effect ABI stability, it seems like
this is a good time to tackle it so that we can push this into the
resilience work that is being done for swift 3.

I would appreciate any pointers and suggestions as to how to best go
about handling this.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

9

What is an "internal symbol which other shared objects can use"? That sounds like a self-contradiction.

Jordan

···

On Apr 6, 2016, at 11:31, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

On Wed, Apr 6, 2016 at 11:18 AM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:
Hey, Saleem. How do you expect this to differ from normal symbol visibility? It seems to me that in a shared library, any public symbol is either exported or imported, depending on whether it has a definition, and any non-public symbol is default. (Unlike C, we expect to have sensible rules here.) I guess there's the difference between "a public symbol from elsewhere in this library" and "a public symbol from some other library". Is that it?

Well, there are four cases to consider:
- externally available: imported
- defined (and available for others): exported
- defined (statically): default -- won't even show up, so this is a no-op
- defined (non-statically defined for internal use): default

The thing is that there is no modeling for internal symbols which other shared objects can use. The closest thing you can do is anonymize the symbol (so you don't have a name that you can call it by, but you have an integral ID).

10

The runtime is linked as part of the standard library, and its ABI interface should be exported from libswiftCore.dylib/so/dll like the standard library's. We should already mark up the ABI entry points with the SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not sufficient to expand these macros to __dllexport?

-Joe

···

On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org> wrote:

On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
Hi,

I was playing around with the idea of swift and Windows since there are some interesting differences between COFF/PE and (ELF and MachO).

PE/COFF does not directly address symbols in external modules (DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks in Windows parlance). Fortunately, LLVM has a nice way to model this: GlobalValues have an associated "DLLStorageClass" which indicates whether something is "imported" (provided by an external module), "exported" (provided to external modules), or "default" (everything else).

Adjusting the IRGen to correctly annotate this part of the semantics should get us part of the way to supporting swift on PE/COFF.

The thing to consider with this is that the DLL storage class is dependent on how the module(s) are being built. For example, something may change from the exported storage to default if being built into a static library rather than a shared object and is not meant to be re-exported.

Part of this information really needs to be threaded from the build system so that we know whether a given SIL module is external or internal.

To the DLL Storage semantics support, Ive taken a quick first stab at it. Ive pushed the changes to GitHub - compnerd/apple-swift at dllstorage and created a Pull Request at https://github.com/apple/swift/pull/2080 .

However, as I expected, this is going to cause problems for building some of the core libraries. In particular, there are mismatches between what gets compiled and is desired. The swiftStubs and swiftRuntime are statically compiled and then merged into swiftCore. There is also the concern of the the support modules (e.g. Platform). If there are stubs that are being used (e.g. via _silgen_name) then there are issues with calculating the correct DLL storage for the associated global values.

Playing around with this, I was trying to special case the building of the standard library (as the runtime will be statically linked into it, the symbols that it is expecting to be externally available are actually private linkage. Not hacking up the compiler like this causes issues since there are inverse dependencies (swiftCore gets dllimport interfaces from swiftRuntime, which has dependencies on swiftCore). The crux of the problem is that we do not have a way to represent that in swift.

The easiest answer that seems to come to mind is to actually introduce an attribute to indicate that an interface is part of a specific module and assume that everything else is locally defined. This would also potentially allow us to handle things like @inline(always) @transparent interfaces which get imported to ensure that a static inline function is given local visibility rather than a DLL Import storage.

Unfortunately, I believe that currently Im stuck as I do not have a good way to determine what type of dll storage class a symbol should be given (since currently, theres no way to determine if we will have a symbol available locally or not when actually linking).

It seems to me, at least initially, that we need a way to treat SwiftModule as a container (a la llvm::Module) and indicate which of the TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you want to call it) so that we can properly track the DLL storage associated with them. Am I confusing something there?

Is there a preference on a means to handle this?

11

Hey, Saleem. How do you expect this to differ from normal symbol
visibility? It seems to me that in a shared library, any public symbol is
either exported or imported, depending on whether it has a definition, and
any non-public symbol is default. (Unlike C, we expect to have sensible
rules here.) I guess there's the difference between "a public symbol from
elsewhere in this library" and "a public symbol from some other library".
Is that it?

Well, there are four cases to consider:
- externally available: imported
- defined (and available for others): exported
- defined (statically): default -- won't even show up, so this is a no-op
- defined (non-statically defined for internal use): default

The thing is that there is no modeling for internal symbols which other
shared objects can use. The closest thing you can do is anonymize the
symbol (so you don't have a name that you can call it by, but you have an
integral ID).

What is an "internal symbol which other shared objects can use"? That
sounds like a self-contradiction.

This is used for things like providing private hooks across shared
objects. These would usually be anonymized (nameless), but you get an
integral ID that gives you an index into a table of pointers to the
function or variable that you are interested in.

···

On Wed, Apr 6, 2016 at 11:34 AM, Jordan Rose <jordan_rose@apple.com> wrote:

On Apr 6, 2016, at 11:31, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:
On Wed, Apr 6, 2016 at 11:18 AM, Jordan Rose <jordan_rose@apple.com> > wrote:

Jordan

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

12

The definitions can be marked as __declspec(dllexport) but the compiler
generated references need to be dllimport for the wrapped runtime functions
(easy for the most part -- see my changes). There's also the concern of
stubs for the aliases (via silgen_name). Those are defined externally with
no indication that they are locally available and thus should have default
rather than dllimport storage. Similar things for standard library
metadata (type, witness tables, etc).

-Joe

···

On Monday, April 11, 2016, Joe Groff <jgroff@apple.com> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev < > swift-dev@swift.org <javascript:;>> wrote:
>
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool < > compnerd@compnerd.org <javascript:;>> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool < > compnerd@compnerd.org <javascript:;>> wrote:
> Hi,
>
> I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at
it. Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a
Pull Request at https://github.com/apple/swift/pull/2080 .
>
> However, as I expected, this is going to cause problems for building
some of the core libraries. In particular, there are mismatches between
what gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of
the standard library (as the runtime will be statically linked into it, the
symbols that it is expecting to be externally available are actually
private linkage. Not hacking up the compiler like this causes issues since
there are inverse dependencies (swiftCore gets dllimport interfaces from
swiftRuntime, which has dependencies on swiftCore). The crux of the
problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce
an attribute to indicate that an interface is part of a specific module and
assume that everything else is locally defined. This would also
potentially allow us to handle things like @inline(always) @transparent
interfaces which get imported to ensure that a static inline function is
given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a good
way to determine what type of dll storage class a symbol should be given
(since currently, theres no way to determine if we will have a symbol
available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat
SwiftModule as a container (a la llvm::Module) and indicate which of the
TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you
want to call it) so that we can properly track the DLL storage associated
with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI
interface should be exported from libswiftCore.dylib/so/dll like the
standard library's. We should already mark up the ABI entry points with the
SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not
sufficient to expand these macros to __dllexport?

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

13

>
> Hi,
>
> I was playing around with the idea of swift and Windows since there are
some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at
it. Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a
Pull Request at [windows] IRGen: add support for DLL Storage semantics by compnerd · Pull Request #2080 · apple/swift · GitHub .
>
> However, as I expected, this is going to cause problems for building
some of the core libraries. In particular, there are mismatches between
what gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of
the standard library (as the runtime will be statically linked into it, the
symbols that it is expecting to be externally available are actually
private linkage. Not hacking up the compiler like this causes issues since
there are inverse dependencies (swiftCore gets dllimport interfaces from
swiftRuntime, which has dependencies on swiftCore). The crux of the
problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce
an attribute to indicate that an interface is part of a specific module and
assume that everything else is locally defined. This would also
potentially allow us to handle things like @inline(always) @transparent
interfaces which get imported to ensure that a static inline function is
given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a
good way to determine what type of dll storage class a symbol should be
given (since currently, theres no way to determine if we will have a symbol
available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat
SwiftModule as a container (a la llvm::Module) and indicate which of the
TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you
want to call it) so that we can properly track the DLL storage associated
with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI
interface should be exported from libswiftCore.dylib/so/dll like the
standard library's. We should already mark up the ABI entry points with the
SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not
sufficient to expand these macros to __dllexport?

The definitions can be marked as __declspec(dllexport) but the compiler
generated references need to be dllimport for the wrapped runtime functions
(easy for the most part -- see my changes). There's also the concern of
stubs for the aliases (via silgen_name). Those are defined externally with
no indication that they are locally available and thus should have default
rather than dllimport storage. Similar things for standard library
metadata (type, witness tables, etc).

A gentle reminder on this topic. I would like to get something sorted out
so that we can try to get this resolved, preferably before the swift 3
release.

···

On Tue, Apr 12, 2016 at 9:32 AM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

On Monday, April 11, 2016, Joe Groff <jgroff@apple.com> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev < >> swift-dev@swift.org> wrote:
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool < >> compnerd@compnerd.org> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool < >> compnerd@compnerd.org> wrote:

-Joe

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

14

I think it's reasonable to assume that @_silgen_name is not being used for objects that are external to the declaring module. That is, the implementation model is that those declarations really are declaring a Swift function; it just happens that the actual body (if not provided) is provided magically. That should work for all the standard library use cases.

John.

···

On Apr 26, 2016, at 8:43 AM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
On Tue, Apr 12, 2016 at 9:32 AM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:
On Monday, April 11, 2016, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org <>> wrote:
>
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool <compnerd@compnerd.org <>> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool <compnerd@compnerd.org <>> wrote:
> Hi,
>
> I was playing around with the idea of swift and Windows since there are some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules (DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks in Windows parlance). Fortunately, LLVM has a nice way to model this: GlobalValues have an associated "DLLStorageClass" which indicates whether something is "imported" (provided by an external module), "exported" (provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is dependent on how the module(s) are being built. For example, something may change from the exported storage to default if being built into a static library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at it. Ive pushed the changes to GitHub - compnerd/apple-swift at dllstorage and created a Pull Request at [windows] IRGen: add support for DLL Storage semantics by compnerd · Pull Request #2080 · apple/swift · GitHub .
>
> However, as I expected, this is going to cause problems for building some of the core libraries. In particular, there are mismatches between what gets compiled and is desired. The swiftStubs and swiftRuntime are statically compiled and then merged into swiftCore. There is also the concern of the the support modules (e.g. Platform). If there are stubs that are being used (e.g. via _silgen_name) then there are issues with calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of the standard library (as the runtime will be statically linked into it, the symbols that it is expecting to be externally available are actually private linkage. Not hacking up the compiler like this causes issues since there are inverse dependencies (swiftCore gets dllimport interfaces from swiftRuntime, which has dependencies on swiftCore). The crux of the problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce an attribute to indicate that an interface is part of a specific module and assume that everything else is locally defined. This would also potentially allow us to handle things like @inline(always) @transparent interfaces which get imported to ensure that a static inline function is given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a good way to determine what type of dll storage class a symbol should be given (since currently, theres no way to determine if we will have a symbol available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat SwiftModule as a container (a la llvm::Module) and indicate which of the TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you want to call it) so that we can properly track the DLL storage associated with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI interface should be exported from libswiftCore.dylib/so/dll like the standard library's. We should already mark up the ABI entry points with the SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not sufficient to expand these macros to __dllexport?

The definitions can be marked as __declspec(dllexport) but the compiler generated references need to be dllimport for the wrapped runtime functions (easy for the most part -- see my changes). There's also the concern of stubs for the aliases (via silgen_name). Those are defined externally with no indication that they are locally available and thus should have default rather than dllimport storage. Similar things for standard library metadata (type, witness tables, etc).

A gentle reminder on this topic. I would like to get something sorted out so that we can try to get this resolved, preferably before the swift 3 release.

15

Just to chime in here (as asked by Saleem, um, a month ago) I think this is the right way to go. We should handle the DLL-ish case (mark public things ‘dllexport’ and references outside the Swift module ‘dllimport’), and not worry about static linking (unnecessary dllimports).

(It seems __declspec(dllexport) is exactly the same as LLVM ‘public’ vs ‘hidden’—at least for Swift’s uses, even if that isn’t true generally.)

We can then come back later and design / add some kind of “I know this object file is going to be statically linked into the executable” mode, which will drop all the dllexports. We’d also want to encode this flag into the serialized “swiftmodule” files (a library’s public interface), so that we know not to use dllimport for anything we use from there.

Thanks for pushing on this!
Jordan

···

On Apr 26, 2016, at 08:43, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

On Tue, Apr 12, 2016 at 9:32 AM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:
On Monday, April 11, 2016, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev <swift-dev@swift.org <>> wrote:
>
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool <compnerd@compnerd.org <>> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool <compnerd@compnerd.org <>> wrote:
> Hi,
>
> I was playing around with the idea of swift and Windows since there are some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules (DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks in Windows parlance). Fortunately, LLVM has a nice way to model this: GlobalValues have an associated "DLLStorageClass" which indicates whether something is "imported" (provided by an external module), "exported" (provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is dependent on how the module(s) are being built. For example, something may change from the exported storage to default if being built into a static library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at it. Ive pushed the changes to GitHub - compnerd/apple-swift at dllstorage and created a Pull Request at [windows] IRGen: add support for DLL Storage semantics by compnerd · Pull Request #2080 · apple/swift · GitHub .
>
> However, as I expected, this is going to cause problems for building some of the core libraries. In particular, there are mismatches between what gets compiled and is desired. The swiftStubs and swiftRuntime are statically compiled and then merged into swiftCore. There is also the concern of the the support modules (e.g. Platform). If there are stubs that are being used (e.g. via _silgen_name) then there are issues with calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of the standard library (as the runtime will be statically linked into it, the symbols that it is expecting to be externally available are actually private linkage. Not hacking up the compiler like this causes issues since there are inverse dependencies (swiftCore gets dllimport interfaces from swiftRuntime, which has dependencies on swiftCore). The crux of the problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce an attribute to indicate that an interface is part of a specific module and assume that everything else is locally defined. This would also potentially allow us to handle things like @inline(always) @transparent interfaces which get imported to ensure that a static inline function is given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a good way to determine what type of dll storage class a symbol should be given (since currently, theres no way to determine if we will have a symbol available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat SwiftModule as a container (a la llvm::Module) and indicate which of the TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you want to call it) so that we can properly track the DLL storage associated with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI interface should be exported from libswiftCore.dylib/so/dll like the standard library's. We should already mark up the ABI entry points with the SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not sufficient to expand these macros to __dllexport?

The definitions can be marked as __declspec(dllexport) but the compiler generated references need to be dllimport for the wrapped runtime functions (easy for the most part -- see my changes). There's also the concern of stubs for the aliases (via silgen_name). Those are defined externally with no indication that they are locally available and thus should have default rather than dllimport storage. Similar things for standard library metadata (type, witness tables, etc).

A gentle reminder on this topic. I would like to get something sorted out so that we can try to get this resolved, preferably before the swift 3 release.

16

Hi,

I made an experimental MSVC port. Of cause, dllimport/dllexport and the
driver for linking and many other part is not implemented. But dynamic
linking was possible with some trick.

I think it is useful for designing, my observation about the experimental
building of libswiftCore.dll, libswiftSwiftOnoneSupport.dll and linking of
Hello.exe - its source has only 'print("Hello")'.

1) SWIFT_RUNTIME_EXPORT was not enough for dllexport.
  Hello.obj needed defined in libswift*.dll
    _swift_getExistentialTypeMetadata,
    _TFs5printFTGSaP__9separatorSS10terminatorSS_T_,
    _TMSS,
    _TZvOs7Process5_argcVs5Int32,
    swift_bufferAllocate, ....
  Some of above are dllexported by the macro, but _T* are not. Maybe, it
generated by swiftc.exe.
  I used the utility 'dlltool.exe' from Cygwin/MinGW world. It extracts all
symbols and generates 'allsymbol.def'.
  With that .def, I could build the all-symbol-dllexported libswiftCore.dll.
  (I'm hoping we can build it without this trick.)

2) Imediate mode just worked with libswift*.dll.
  Using the libswiftCore.dll, I could just run 'swift -O Hello.swift' and
it worked. I was happy.
  But I don't know how Immediate mode work. How could this work without any
consideration about dllimport or import library?
  Anyway, after building libswiftSwiftOnoneSupport.dll, I could run 'swift
Hello.swift' without -O.

3) Building Hello.exe with some trick.
  I compiled the Hello.swift to Hello.ll, and simply replace the string
"external global/constant" to "external dllimport global/contant" in the
file, and pass it to 'clang' for Hello.obj, and link with previously built
import library of swiftCore.
  It worked well. It will be happy swiftc could generate the dllimport.

4) In building libswiftSwiftOnoneSupport.dll, there is SOME MORE.
  To build libswiftSwiftOnoneSupport.dll, it needed to link with
libswiftCore.dll. I used the same method - 'injecting dllimport into *.ll'.
  When linking swiftOnone, only _TWVBo was not resolved.
  To make it linkable, in the SwiftOnoneSupport.ll,
    @_TWVBo = external global i8*, align 8
  should be replaced to
    @__imp__TWVBo = external global i8*, align 8
  not
    @_TWVBo = external dllimport global i8*, align 8.

  I don't know well what _TWVBo is. I only guess this related to indirect
access the variable _TWVBo.
  But I think we should know what the root cause to this.

- Han Sangjin

···

2016-05-04 10:34 GMT+09:00 Jordan Rose via swift-dev <swift-dev@swift.org>:

On Apr 26, 2016, at 08:43, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:

On Tue, Apr 12, 2016 at 9:32 AM, Saleem Abdulrasool <compnerd@compnerd.org > > wrote:

On Monday, April 11, 2016, Joe Groff <jgroff@apple.com> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev < >>> swift-dev@swift.org> wrote:
>
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool < >>> compnerd@compnerd.org> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool < >>> compnerd@compnerd.org> wrote:
> Hi,
>
> I was playing around with the idea of swift and Windows since there
are some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at
it. Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a
Pull Request at https://github.com/apple/swift/pull/2080 .
>
> However, as I expected, this is going to cause problems for building
some of the core libraries. In particular, there are mismatches between
what gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of
the standard library (as the runtime will be statically linked into it, the
symbols that it is expecting to be externally available are actually
private linkage. Not hacking up the compiler like this causes issues since
there are inverse dependencies (swiftCore gets dllimport interfaces from
swiftRuntime, which has dependencies on swiftCore). The crux of the
problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce
an attribute to indicate that an interface is part of a specific module and
assume that everything else is locally defined. This would also
potentially allow us to handle things like @inline(always) @transparent
interfaces which get imported to ensure that a static inline function is
given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a
good way to determine what type of dll storage class a symbol should be
given (since currently, theres no way to determine if we will have a symbol
available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat
SwiftModule as a container (a la llvm::Module) and indicate which of the
TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you
want to call it) so that we can properly track the DLL storage associated
with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI
interface should be exported from libswiftCore.dylib/so/dll like the
standard library's. We should already mark up the ABI entry points with the
SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not
sufficient to expand these macros to __dllexport?

The definitions can be marked as __declspec(dllexport) but the compiler
generated references need to be dllimport for the wrapped runtime functions
(easy for the most part -- see my changes). There's also the concern of
stubs for the aliases (via silgen_name). Those are defined externally with
no indication that they are locally available and thus should have default
rather than dllimport storage. Similar things for standard library
metadata (type, witness tables, etc).

A gentle reminder on this topic. I would like to get something sorted out
so that we can try to get this resolved, preferably before the swift 3
release.

Just to chime in here (as asked by Saleem, um, a month ago) I think this
is the right way to go. We should handle the DLL-ish case (mark public
things ‘dllexport’ and references outside the Swift module ‘dllimport’),
and not worry about static linking (unnecessary dllimports).

(It seems __declspec(dllexport) is exactly the same as LLVM ‘public’ vs
‘hidden’—at least for Swift’s uses, even if that isn’t true generally.)

We can then come back later and design / add some kind of “I know this
object file is going to be statically linked into the executable” mode,
which will drop all the dllexports. We’d also want to encode this flag into
the serialized “swiftmodule” files (a library’s public interface), so that
we know not to use dllimport for anything we use from there.

Thanks for pushing on this!
Jordan

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

17

>
> Hi,
>
> I was playing around with the idea of swift and Windows since there
are some interesting differences between COFF/PE and (ELF and MachO).
>
> PE/COFF does not directly address symbols in external modules
(DSOs/dylibs/DLLs). Instead, there is an indirect addressing model (thunks
in Windows parlance). Fortunately, LLVM has a nice way to model this:
GlobalValues have an associated "DLLStorageClass" which indicates whether
something is "imported" (provided by an external module), "exported"
(provided to external modules), or "default" (everything else).
>
> Adjusting the IRGen to correctly annotate this part of the semantics
should get us part of the way to supporting swift on PE/COFF.
>
> The thing to consider with this is that the DLL storage class is
dependent on how the module(s) are being built. For example, something may
change from the exported storage to default if being built into a static
library rather than a shared object and is not meant to be re-exported.
>
> Part of this information really needs to be threaded from the build
system so that we know whether a given SIL module is external or internal.
>
> To the DLL Storage semantics support, Ive taken a quick first stab at
it. Ive pushed the changes to
GitHub - compnerd/apple-swift at dllstorage and created a
Pull Request at https://github.com/apple/swift/pull/2080 .
>
> However, as I expected, this is going to cause problems for building
some of the core libraries. In particular, there are mismatches between
what gets compiled and is desired. The swiftStubs and swiftRuntime are
statically compiled and then merged into swiftCore. There is also the
concern of the the support modules (e.g. Platform). If there are stubs
that are being used (e.g. via _silgen_name) then there are issues with
calculating the correct DLL storage for the associated global values.
>
> Playing around with this, I was trying to special case the building of
the standard library (as the runtime will be statically linked into it, the
symbols that it is expecting to be externally available are actually
private linkage. Not hacking up the compiler like this causes issues since
there are inverse dependencies (swiftCore gets dllimport interfaces from
swiftRuntime, which has dependencies on swiftCore). The crux of the
problem is that we do not have a way to represent that in swift.
>
> The easiest answer that seems to come to mind is to actually introduce
an attribute to indicate that an interface is part of a specific module and
assume that everything else is locally defined. This would also
potentially allow us to handle things like @inline(always) @transparent
interfaces which get imported to ensure that a static inline function is
given local visibility rather than a DLL Import storage.
>
> Unfortunately, I believe that currently Im stuck as I do not have a
good way to determine what type of dll storage class a symbol should be
given (since currently, theres no way to determine if we will have a symbol
available locally or not when actually linking).
>
> It seems to me, at least initially, that we need a way to treat
SwiftModule as a container (a la llvm::Module) and indicate which of the
TopLevelDecls are meant to be a single "module" (DSO, DLL, whatever you
want to call it) so that we can properly track the DLL storage associated
with them. Am I confusing something there?
>
> Is there a preference on a means to handle this?

The runtime is linked as part of the standard library, and its ABI
interface should be exported from libswiftCore.dylib/so/dll like the
standard library's. We should already mark up the ABI entry points with the
SWIFT_RUNTIME_EXPORT and SWIFT_RUNTIME_STDLIB_INTERFACE macros. Is it not
sufficient to expand these macros to __dllexport?

The definitions can be marked as __declspec(dllexport) but the compiler
generated references need to be dllimport for the wrapped runtime functions
(easy for the most part -- see my changes). There's also the concern of
stubs for the aliases (via silgen_name). Those are defined externally with
no indication that they are locally available and thus should have default
rather than dllimport storage. Similar things for standard library
metadata (type, witness tables, etc).

A gentle reminder on this topic. I would like to get something sorted out
so that we can try to get this resolved, preferably before the swift 3
release.

Just to chime in here (as asked by Saleem, um, a month ago) I think this
is the right way to go. We should handle the DLL-ish case (mark public
things ‘dllexport’ and references outside the Swift module ‘dllimport’),
and not worry about static linking (unnecessary dllimports).

(It seems __declspec(dllexport) is exactly the same as LLVM ‘public’ vs
‘hidden’—at least for Swift’s uses, even if that isn’t true generally.)

Yeah, for the most part, they serve similar purposes.

We can then come back later and design / add some kind of “I know this
object file is going to be statically linked into the executable” mode,
which will drop all the dllexports. We’d also want to encode this flag into
the serialized “swiftmodule” files (a library’s public interface), so that
we know not to use dllimport for anything we use from there.

Unfortunately, ATM, the runtime is compacted into the stdlib, which has the
small problem of making the stdlib have references to the runtime functions
as if it were being dynamically linked. We *could* just assume that the
any time that the LLVM module name is "Swift.*" we are building the stdlib
and not mark the runtime functions for external (import) dll storage, but
this seems rather fragile. The problem is that at IRGen time, we no longer
have access to the SILModule (AFAIK), so we don't have a way to identify
if we are the stdlib or not. Suggestions on how to handle that would be
appreciated.

···

On Tue, May 3, 2016 at 6:34 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Apr 26, 2016, at 08:43, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:
On Tue, Apr 12, 2016 at 9:32 AM, Saleem Abdulrasool <compnerd@compnerd.org > > wrote:

On Monday, April 11, 2016, Joe Groff <jgroff@apple.com> wrote:

> On Apr 11, 2016, at 3:19 PM, Saleem Abdulrasool via swift-dev < >>> swift-dev@swift.org> wrote:
> On Thu, Apr 7, 2016 at 2:12 PM, Saleem Abdulrasool < >>> compnerd@compnerd.org> wrote:
> On Wed, Apr 6, 2016 at 10:21 AM, Saleem Abdulrasool < >>> compnerd@compnerd.org> wrote:

Thanks for pushing on this!
Jordan

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

18

So what's the cleanest way to make this work in practice? I
experimented with the following hack, which worked for building the
stdlib and the SDK overlay but which is full of layer violations:

- Add a symbol tracking hook in SILGenNameAttr, such that when the
hook is active, whenever we encounter a _silgen_name() declaration we
record it in a set.

- Enable this symbol tracking just for the duration of parsing the
main module. Thus we end up with a set of all _silgen_name attributes
on functions declared in the current module, but not _silgen_name
attributes on functions imported from other modules.

- At various points in lib/IRGen/GenDecl.cc
(updateLinkageForDefinition, LinkInfo::createFunction,
LinkInfo::createVariable) check whether the symbol being used is in
the list of _silgen_name attributes from the current module, and if so
ignore the dll-import tag on the symbol when setting up the
llvm::GlobalValue.

It would be nicer to walk across the main module after parsing it to
find SILGenNameAttr instances, but I don't have a good idea how to
start with that. Then we could perhaps add the list of attributes to
the llvm::Module as some kind of llvm::Metadata object?

Paul

···

On Tue, Apr 26, 2016 at 8:49 AM, John McCall via swift-dev <swift-dev@swift.org> wrote:

I think it's reasonable to assume that @_silgen_name is not being used for
objects that are external to the declaring module. That is, the
implementation model is that those declarations really are declaring a Swift
function; it just happens that the actual body (if not provided) is provided
magically. That should work for all the standard library use cases.

19

The _T symbols are emitted by the Swift compiler. You should modify swiftc's IRGen to generate public symbols with LLVM's "dllexport" storage class when targeting Windows.

-Joe

···

On May 5, 2016, at 4:18 PM, Sangjin Han via swift-dev <swift-dev@swift.org> wrote:

Hi,

I made an experimental MSVC port. Of cause, dllimport/dllexport and the driver for linking and many other part is not implemented. But dynamic linking was possible with some trick.

I think it is useful for designing, my observation about the experimental building of libswiftCore.dll, libswiftSwiftOnoneSupport.dll and linking of Hello.exe - its source has only 'print("Hello")'.

1) SWIFT_RUNTIME_EXPORT was not enough for dllexport.
  Hello.obj needed defined in libswift*.dll
    _swift_getExistentialTypeMetadata,
    _TFs5printFTGSaP__9separatorSS10terminatorSS_T_,
    _TMSS,
    _TZvOs7Process5_argcVs5Int32,
    swift_bufferAllocate, ....
  Some of above are dllexported by the macro, but _T* are not. Maybe, it generated by swiftc.exe.
  I used the utility 'dlltool.exe' from Cygwin/MinGW world. It extracts all symbols and generates 'allsymbol.def'.
  With that .def, I could build the all-symbol-dllexported libswiftCore.dll.
  (I'm hoping we can build it without this trick.)

20

>
> Hi,
>
> I made an experimental MSVC port. Of cause, dllimport/dllexport and the
driver for linking and many other part is not implemented. But dynamic
linking was possible with some trick.
>
> I think it is useful for designing, my observation about the
experimental building of libswiftCore.dll, libswiftSwiftOnoneSupport.dll
and linking of Hello.exe - its source has only 'print("Hello")'.
>
> 1) SWIFT_RUNTIME_EXPORT was not enough for dllexport.
> Hello.obj needed defined in libswift*.dll
> _swift_getExistentialTypeMetadata,
> _TFs5printFTGSaP__9separatorSS10terminatorSS_T_,
> _TMSS,
> _TZvOs7Process5_argcVs5Int32,
> swift_bufferAllocate, ....
> Some of above are dllexported by the macro, but _T* are not. Maybe, it
generated by swiftc.exe.
> I used the utility 'dlltool.exe' from Cygwin/MinGW world. It extracts
all symbols and generates 'allsymbol.def'.
> With that .def, I could build the all-symbol-dllexported
libswiftCore.dll.
> (I'm hoping we can build it without this trick.)

The _T symbols are emitted by the Swift compiler. You should modify
swiftc's IRGen to generate public symbols with LLVM's "dllexport" storage
class when targeting Windows.

https://github.com/apple/swift/pull/2080 is a first cut attempt to do this.

···

On Thu, May 5, 2016 at 5:26 PM, Joe Groff via swift-dev <swift-dev@swift.org > wrote:

> On May 5, 2016, at 4:18 PM, Sangjin Han via swift-dev < > swift-dev@swift.org> wrote:

-Joe
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org