SPM and dynamic linking

Hi,

Probably this is a misunderstanding on my part, but I'd like to convince SPM to do more dynamic library generation/linking, and am finding it hard to do so.

I know I can explicit request a dynamic library in my own Package.swift using "type: .dynamic,", and that works fine and produces the .so as I'd expect (its a .so as I'm on linux for this).

However, if I clone an external package that doesn't specify either static or dynamic, only the static .swiftmodule is generated - is there any way to request a dynamic library without changing the Package.swift?

Similarly though libraries can request dynamic linking, an executable pulling in a package (which again doesn't specify dynamic or static, leaving it to its 'client' AIUI), also gets statically linked, and I can't see an option to request dynamic linking.

Any help much appreciated,
Thanks,
Jon

1 Like

I started looking into the SPM source, and it turned out to be not too hard to get at least a minimum of what I was looking for working - I have it both producing a dynlib for a external package, and also dynamic linking a sample executable. Is there likely to be any interest in accepting these changes if I looked to tidy them up further?

Thanks,
Jon

I'd definitely be interested in seeing that merged - static linking is still not properly available on Windows (mostly from the SPM side). So, getting this support would be great.

I've pushed what I have so far to my fork.

It needs more work - at the least, more comments, documentation and testing.

However if anyone has any comments at this stage, or wants to give it a try, that'd be great. In particular, whether the approach of keeping things as 'automatic' libraries, but modifying the build plan to create products for them, is correct?

Thanks,
Jon

I definitely would like to take a closer look and possibly help out if it looks promising, but I do not have time at this very moment. I am posting this to let you know so that if I haven’t remembered to get back to you in a week or so, you can ping me with a reminder.

Okay, so I see you have added --prefer-dynamic, and you have enabled processing of .automatic in several places. (I think it would be better to resolve early whether to divert it into .static or .dynamic mode and then let it simply flow through all the same code branches as the explicit options, but we can get to that later.)

It is not yet clear to me what the precise intended meaning of --prefer-dynamic is. What is supposed to be dynamic?

  • Are the toolchain libraries dynamic?
  • Are dependency products dynamic? All of them, or just conjunction nodes where it would be useful?
  • Should it split .automatic libraries into targets for more fine‐grained heuristics?
  • What about libraries in the same package?

Some other important questions:

  • Should it (or a companion flag) override explicit .static?
  • What happens when an explicit .static ends up in two .dynamic libraries linked into the same executable?
  • How does this interplay with inter‐module optimization?
  • How much of this would be better as heuristics improvements for the default instead of an explicit option?
  • Are we in over our heads and all we really wanted is to produce a .so or .a for a top‐level .automatic library that we then expect to use outside of SwiftPM?

The answers to these questions affect whether it will need to go through the full evolution process and in turn, how much work would be ahead of you.

Thanks for taking the time to look at this, these are all great questions. Let me have a first swing at answering them:

(Apologies if answering inline ends up confusing, but it seemed the easiest way to try and make sure I'd looked at all the questions.)

Yes, that is what I meant. They originate from the Swift toolchain and not from the dependency graph. I ask because there has been recent discussion about changing the defaults for these, which might have some overlap with what we are doing here.

Right now a manifest cannot declare a dependency from a target on a product in the same package. It is already confusing to some that a .dynamic product creates a .so, but a neighbouring executable statically links the targets instead. (It is less confusing when the target and product names don’t match, but in many cases they do.) The flag controls you propose would make this quirk more visible and possibly more painful. It might need to be fixed/rethought first.

It appears to happen far more often than necessary. SwiftSyntax itself is an example. I suspect this is due to autocomplete presenting the field for type, and users reaching for .static without knowing about the .automatic enum case. (They are not even the same; .static creates a .a in the products directory, automatic does not.)

That is only part of what I mean. Already static linking can do some dead stripping, specialization, etc. and it is getting better with time. Theoretically the same optimizations ought to be possible for dynamic‐linked libraries built together in the dependency graph, but I do not think any of this has been implemented yet. So is the worse size and speed acceptable in the meantime, and does it need to be communicated to users somehow? Also, if a package is an executable‐dynamic library pair, where one uses the other, should the dynamic library be stripped down to only what the executable actually uses or not? Do you know of any other ramifications I have not thought of yet?

We use Swift on ARMv7 Linux (Yocto) on a commercial IoT device for a couple years now. For resource constrained devices having the runtime as a dynamic library is optimal, and we even try to build SwiftPM libraries as dynamic when it makes sense. Making the stdlib static by default would increment binary size a lot, and should not be the default given that you do need more resources to run that larger binary. The default setting should always favor minimal resource usage. Also its not only SwiftCore that would be bundled, but Dispatch, Foundation and SwiftConcurrency modules. What's the story for CURL, ICU and LibXML2? I think having every single Swift library statically link all that by default is insane, and just linking the stdlib is inconsistent since a simple Hello world program with DateFormatter requires all of the above.

I have a custom Buildroot fork that runs Swift 5.6 on an Armv5 chip with 64mb of RAM. If this proposal is accepted, I would have to make an extra effort to efficiently run on resource constrained devices that work incredibly well right now.

Furthermore, with the plans for distributing official RPM and DEB packages, this makes absolutely no sense. Not all usage of Swift on Linux is server or Docker containers. Even though the Swift stdlib is not ABI stable on Linux officially, I have been able to run the same binary (dynamically linked) with different compiled std libs (Yocto and Buildroot recipes, Debian as well) without missing symbols, at least for new patch versions (e.g. Swift 5.5 -> 5.5.3, Swift 5.6 -> 5.6.1)