[RFC] Moving to gold linker

Hi All,

Orlando (included in the to line) and I have been looking at using the gold linker in the swift toolchain. Orlando is really the lead on this project, I’m mostly just doing the plumbing.

The open questions I have are the following:

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

2. Should the swift toolchain support the selection of linkers on the command line, and if so, what’s the default?

3. If a command-line switch for swift is not desired, should a switch in the build process select the supported linker?

4. Can it be assumed that gold will exist in user installations?

For some additional context, we’re investigating the use of gold because on armv6/v7 and aarch64 platforms the standard linker has bugs that require the "big hammer,” as Dmitri called it, of -Bsymbolic to work around them. It would be nice to solve these issues a better way.

Thoughts, Ideas, Flames, etc?
- Will

Hi All,

Orlando (included in the to line) and I have been looking at using the gold linker in the swift toolchain. Orlando is really the lead on this project, I’m mostly just doing the plumbing.

The open questions I have are the following:

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

I can't speak for the project, but I can speak for myself.
My naive understanding is that swift uses two linkers because 'INSERT
AFTER' linker script directive isn't supported (in gold). I read a lot about
that -- and I felt like the gold developers didn't care about adding
this command as it's mainly of use for the default linker script
shipped with ld.bfd.
They also provided an alternative way to obtain the same semantics
without providing an additional directive. Other than that, I don't
know any other reasons why gold shouldn't be used.

2. Should the swift toolchain support the selection of linkers on the command line, and if so, what’s the default?

If you mean something like -fuse-ld for clang, I'd like to see that
happen (if not supported already). As lld developer I found that very
useful and I hope to be able to link swift and swiftc-generated
executables using lld, so, yes.

···

On Thu, Jan 14, 2016 at 9:54 AM, William Dillon via swift-dev <swift-dev@swift.org> wrote:

3. If a command-line switch for swift is not desired, should a switch in the build process select the supported linker?

4. Can it be assumed that gold will exist in user installations?

For some additional context, we’re investigating the use of gold because on armv6/v7 and aarch64 platforms the standard linker has bugs that require the "big hammer,” as Dmitri called it, of -Bsymbolic to work around them. It would be nice to solve these issues a better way.

Thoughts, Ideas, Flames, etc?
- Will
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

I can't speak for the project, but I can speak for myself.
My naive understanding is that swift uses two linkers because 'INSERT
AFTER' linker script directive isn't supported (in gold). I read a lot about
that -- and I felt like the gold developers didn't care about adding
this command as it's mainly of use for the default linker script
shipped with ld.bfd.
They also provided an alternative way to obtain the same semantics
without providing an additional directive. Other than that, I don't
know any other reasons why gold shouldn't be used.

That’s right, AFAIK. Orlando developed a system using two assembler files that bookend the objects during link to achieve the same result as the ld script. Additionally, this method works with BFD, so swift.ld can be discarded entirely.

2. Should the swift toolchain support the selection of linkers on the command line, and if so, what’s the default?

If you mean something like -fuse-ld for clang, I'd like to see that
happen (if not supported already). As lld developer I found that very
useful and I hope to be able to link swift and swiftc-generated
executables using lld, so, yes.

That is the expectation, yes.

That sounds like a great improvement. Supporting gold for the entire build process would be great. Any idea if the assembler input would also work on FreeBSD or other platforms?

-Joe

···

On Jan 14, 2016, at 11:07 AM, William Dillon via swift-dev <swift-dev@swift.org> wrote:

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

I can't speak for the project, but I can speak for myself.
My naive understanding is that swift uses two linkers because 'INSERT
AFTER' linker script directive isn't supported (in gold). I read a lot about
that -- and I felt like the gold developers didn't care about adding
this command as it's mainly of use for the default linker script
shipped with ld.bfd.
They also provided an alternative way to obtain the same semantics
without providing an additional directive. Other than that, I don't
know any other reasons why gold shouldn't be used.

That’s right, AFAIK. Orlando developed a system using two assembler files that bookend the objects during link to achieve the same result as the ld script. Additionally, this method works with BFD, so swift.ld can be discarded entirely.

Yes, as long as you’re using clang, it should work in FreeBSD. Darwin doesn’t require this at all AFAIK. So, that would support Linux, FreeBSD, and Darwin, all without swift.ld.

- Will

···

On Jan 14, 2016, at 9:19 PM, Joe Groff <jgroff@apple.com> wrote:

On Jan 14, 2016, at 11:07 AM, William Dillon via swift-dev <swift-dev@swift.org> wrote:

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

I can't speak for the project, but I can speak for myself.
My naive understanding is that swift uses two linkers because 'INSERT
AFTER' linker script directive isn't supported (in gold). I read a lot about
that -- and I felt like the gold developers didn't care about adding
this command as it's mainly of use for the default linker script
shipped with ld.bfd.
They also provided an alternative way to obtain the same semantics
without providing an additional directive. Other than that, I don't
know any other reasons why gold shouldn't be used.

That’s right, AFAIK. Orlando developed a system using two assembler files that bookend the objects during link to achieve the same result as the ld script. Additionally, this method works with BFD, so swift.ld can be discarded entirely.

That sounds like a great improvement. Supporting gold for the entire build process would be great. Any idea if the assembler input would also work on FreeBSD or other platforms?

Thanks for the input about the gold linker so far, everyone.

I have a few follow-up questions.

First up is whether there is any desire to keep swift.ld at all. I believe that I’ll be able to use the same object files using the binutils linker, so the mechanics in place for gold will function in both cases. I don’t know if there is another compelling reason for keeping it. Thoughts?

Second, Orlando believes that it’s possible to use C++ to generate the object files rather that assembler. This would make these source files much more portable and easier to maintain. There is a catch, however. In the assembler version there is no runtime impact at all; the C++ version requires a subtraction at load time. The cost is pretty minimal, but it’s worth considering. Changing from one implementation to another is very simple.

Finally, tying in the discussion of multi-architecture/multi-sdk cross compilation on linux, I noticed that swift.ld is copied to a 2-d array of SDKs and Architectures. Considering that effory this effort (may?) not support a functional use case anyway, is it desirable to keep this behavior? It would certainly simplify the generation of the object files. I’ve already done the work, so it’s not that I’m trying to avoid it. ;)

Thanks again for your comments and thoughts,
- Will

···

On Jan 14, 2016, at 9:26 PM, William Dillon via swift-dev <swift-dev@swift.org> wrote:

On Jan 14, 2016, at 9:19 PM, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

On Jan 14, 2016, at 11:07 AM, William Dillon via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

1. Is moving to or supporting Gold within the swift toolchain (not building swift itself) a goal, non-goal, or not desired?

I can't speak for the project, but I can speak for myself.
My naive understanding is that swift uses two linkers because 'INSERT
AFTER' linker script directive isn't supported (in gold). I read a lot about
that -- and I felt like the gold developers didn't care about adding
this command as it's mainly of use for the default linker script
shipped with ld.bfd.
They also provided an alternative way to obtain the same semantics
without providing an additional directive. Other than that, I don't
know any other reasons why gold shouldn't be used.

That’s right, AFAIK. Orlando developed a system using two assembler files that bookend the objects during link to achieve the same result as the ld script. Additionally, this method works with BFD, so swift.ld can be discarded entirely.

That sounds like a great improvement. Supporting gold for the entire build process would be great. Any idea if the assembler input would also work on FreeBSD or other platforms?

Yes, as long as you’re using clang, it should work in FreeBSD. Darwin doesn’t require this at all AFAIK. So, that would support Linux, FreeBSD, and Darwin, all without swift.ld.

- Will
_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

Thanks for the input about the gold linker so far, everyone.

I have a few follow-up questions.

First up is whether there is any desire to keep swift.ld at all. I believe that I’ll be able to use the same object files using the binutils linker, so the mechanics in place for gold will function in both cases. I don’t know if there is another compelling reason for keeping it. Thoughts?

swift.ld is a means to an end. I don't think we'd keep it if we had a superior alternative, and we wouldn't want the maintenance burden of having to keep multiple techniques working.

Second, Orlando believes that it’s possible to use C++ to generate the object files rather that assembler. This would make these source files much more portable and easier to maintain. There is a catch, however. In the assembler version there is no runtime impact at all; the C++ version requires a subtraction at load time. The cost is pretty minimal, but it’s worth considering. Changing from one implementation to another is very simple.

How different is the .s file across platforms? I'd expect it to just contain data directives.

Finally, tying in the discussion of multi-architecture/multi-sdk cross compilation on linux, I noticed that swift.ld is copied to a 2-d array of SDKs and Architectures. Considering that effory this effort (may?) not support a functional use case anyway, is it desirable to keep this behavior? It would certainly simplify the generation of the object files. I’ve already done the work, so it’s not that I’m trying to avoid it. ;)

Dmitri would have to comment about this part.

-Joe

···

On Jan 18, 2016, at 9:46 PM, William Dillon <wdillon@coas.oregonstate.edu> wrote:

Second, Orlando believes that it’s possible to use C++ to generate the object files rather that assembler. This would make these source files much more portable and easier to maintain. There is a catch, however. In the assembler version there is no runtime impact at all; the C++ version requires a subtraction at load time. The cost is pretty minimal, but it’s worth considering. Changing from one implementation to another is very simple.

How different is the .s file across platforms? I'd expect it to just contain data directives.

There are very minor differences. Here are the current files:

swift_begin.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_start
.swift2_protocol_conformances_start:

if defined(__x86_64__)
.quad (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
#else
.long (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
.long 0
#endif

swift_end.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_end
.swift2_protocol_conformances_end:

Whereas the C++ version would be something like this (for begin):

__attribute__((section(".swift2_protocol_conformances")))
extern char swift2_protocol_conformances_end __asm__(".swift2_protocol_conformances_end");

__attribute__((section(".swift2_protocol_conformances")))
long long swift2_protocol_conformances_start __asm__(".swift2_protocol_conformances_start") =
(long long)((char *)&swift2_protocol_conformances_end - (char *)&swift2_protocol_conformances_start - sizeof(swift2_protocol_conformances_start));

So, it’s not necessary to break-out the different sections, the generated initializer for swift2_protocol_conformances_start would have a subtraction.

- Will

A .ll file containing LLVM IR might be able to adequately abstract away the per-platform syntactical differences in .s files, while allowing enough control to avoid the extra relocation implied by the C++ code.

-Joe

···

On Jan 19, 2016, at 8:31 AM, William Dillon <william@housedillon.com> wrote:

Second, Orlando believes that it’s possible to use C++ to generate the object files rather that assembler. This would make these source files much more portable and easier to maintain. There is a catch, however. In the assembler version there is no runtime impact at all; the C++ version requires a subtraction at load time. The cost is pretty minimal, but it’s worth considering. Changing from one implementation to another is very simple.

How different is the .s file across platforms? I'd expect it to just contain data directives.

There are very minor differences. Here are the current files:

swift_begin.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_start
.swift2_protocol_conformances_start:

if defined(__x86_64__)
.quad (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
#else
.long (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
.long 0
#endif

swift_end.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_end
.swift2_protocol_conformances_end:

Whereas the C++ version would be something like this (for begin):

__attribute__((section(".swift2_protocol_conformances")))
extern char swift2_protocol_conformances_end __asm__(".swift2_protocol_conformances_end");

__attribute__((section(".swift2_protocol_conformances")))
long long swift2_protocol_conformances_start __asm__(".swift2_protocol_conformances_start") =
(long long)((char *)&swift2_protocol_conformances_end - (char *)&swift2_protocol_conformances_start - sizeof(swift2_protocol_conformances_start));

So, it’s not necessary to break-out the different sections, the generated initializer for swift2_protocol_conformances_start would have a subtraction.

Thanks for the idea, Joe!

We’ll look into it. :)

···

On Jan 20, 2016, at 12:53 PM, Joe Groff <jgroff@apple.com> wrote:

On Jan 19, 2016, at 8:31 AM, William Dillon <william@housedillon.com> wrote:

Second, Orlando believes that it’s possible to use C++ to generate the object files rather that assembler. This would make these source files much more portable and easier to maintain. There is a catch, however. In the assembler version there is no runtime impact at all; the C++ version requires a subtraction at load time. The cost is pretty minimal, but it’s worth considering. Changing from one implementation to another is very simple.

How different is the .s file across platforms? I'd expect it to just contain data directives.

There are very minor differences. Here are the current files:

swift_begin.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_start
.swift2_protocol_conformances_start:

if defined(__x86_64__)
.quad (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
#else
.long (.swift2_protocol_conformances_end - .swift2_protocol_conformances_start) - 8
.long 0
#endif

swift_end.S:

if defined(__arm__)
.section .swift2_protocol_conformances, "aw",%progbits
#else
.section .swift2_protocol_conformances, "aw",@progbits
#endif
.global .swift2_protocol_conformances_end
.swift2_protocol_conformances_end:

Whereas the C++ version would be something like this (for begin):

__attribute__((section(".swift2_protocol_conformances")))
extern char swift2_protocol_conformances_end __asm__(".swift2_protocol_conformances_end");

__attribute__((section(".swift2_protocol_conformances")))
long long swift2_protocol_conformances_start __asm__(".swift2_protocol_conformances_start") =
(long long)((char *)&swift2_protocol_conformances_end - (char *)&swift2_protocol_conformances_start - sizeof(swift2_protocol_conformances_start));

So, it’s not necessary to break-out the different sections, the generated initializer for swift2_protocol_conformances_start would have a subtraction.

A .ll file containing LLVM IR might be able to adequately abstract away the per-platform syntactical differences in .s files, while allowing enough control to avoid the extra relocation implied by the C++ code.