Ah, thank you for the clarification! Approaching it that way would definitely be much nicer IMO.
1 Like
Thanks @John_McCall for chatting with Sharon and me during the dev meeting! Capturing some of the discussions here:
1> The build system can generate information via a build file and pass it over to the compiler to enable further optimizations and improve accuracy of visibility.
2> One potential proposal for visibility control in the build file is a setting for each module:
// - export both public and package symbols
// - export only public symbols
// - export nothing
3> A first step can be implementing a flag to set these levels
4> The build file can guide some of the Swift optimizations, one example is the possibility of using relative references in more places.
I looked a bit at using relative references in Swift metadata, it seems that some types of metadata can use relative references, while others can't:
-
Metadata requiring rebases: witness tables, type metadata, full type metadata
-
Metadata not requiring rebases: protocol conformance descriptor, reflection metadata field, nominal type descriptor
We are interested in understanding relative references to better understand the startup performance due to rebases/fixups etc.
CC @Joe_Groff on relative references and other potential optimizations that can be enabled by the build file.
Thanks!
1 Like
We use relative references for things that both are likely to be compiled into the same final binary product, and which also can be put in constant memory. Giving the compiler accurate information about what packages are destined for the same binary would definitely let it make more aggressive use of direct relative references (as opposed to indirect relative references where the relative pointer really just points at a full absolute pointer somewhere else) everywhere we currently use relative references. However, type metadata and witness tables as currently designed are instantiated on demand for generic types, so they don't meet either criterion; they can be allocated anywhere in the process address space, and often live in writable memory. The compiler also emits a lot of code that projects directly out of type metadata and witness table records, so there would be ABI difficulties in changing their layout.
For witness tables, we could still probably transition to a design based on relative pointers to the method implementations. Witness tables don't have a uniqueness requirement like type metadata does, and if we didn't try to store associated type metadata pointers in them, then there would be no fundamental reason to instantiate them at runtime. We could instead reference a mangled name for associated type witnesses. As far as compatibility with the current ABI is concerned, I can think of a few options. For one, we could use a new relative witness table ABI for protocols that are only defined and used internally or among packages that all agree to use the new ABI. For existing ABI-stable protocols, we could perhaps provide both the existing and new ABIs, allowing existing code to request instantiated absolute witness tables while also providing relative witness tables for new code. @Mike_Ash and @Arnold had done some exploratory implementation work for relative witness tables for embedded platforms that might be usable.
For type metadata, the uniqueness requirement seems to me like it makes runtime instantiation hard to avoid. But maybe we could make it so that type metadata is needed less often. Most of the core value operations on a type only need the value witness table, which @drexin has been working on compressing into a single "value witness bytecode" string. It might be interesting to experiment with a generics ABI where, instead of passing the type metadata, we pass only the value witness bytecode and maybe a mangled name that can be used to lazily instantiate the metadata when it's needed. Code that never does dynamic casting, type equality tests, or other reflection operations might be able to avoid ever touching the type metadata.
3 Likes
IIRC, the relative references we emit from mangled type-ref strings are indirected when they need to cross module boundaries. We use type-ref strings in several places in the ABI, such as associated type entries in protocol witness tables, although to my surprise we don't seem to use them when just emitting type metadata references from code.
I may be overestimating how large of an impact we'd get from knowing that other modules shared the current image (beyond just enabling much smarter builds on Windows, of course). It's probably something we could pretty easily measure with some sort of hard-coded experiment, though. We'd need to find all the places where we currently query "is this in the current module" for this purpose and unify that logic into a single function, and then we could run the experiment by hacking that function to know that module "abc" is in the same image and see how much good it does.
3 Likes
Hi all,
I've done some analysis on the proposed assembly instructions and want to discuss a little bit about how a module granularity symbol visibility solution isn't likely to work for us. We currently use an exported symbols list to control visibility of Swift symbols in our dylibs. For one such dylib, I took the existing exported symbols list and found all Swift modules which defined at least one symbol included in the current exported symbol list. The penalty of exporting all symbols from those modules is significant: we would go from exporting ~400 Swift symbols to exporting ~27,000, and I'm only looking at Swift symbols, not considering related symbols for ObjC interop.
It seems there have been a couple other requests for more fine grain symbol control: https://forums.swift.org/t/convention-thin-function-pointers/65180/8, https://forums.swift.org/t/pitch-low-level-linkage-control-attributes-used-and-section/65877?fbclid=IwAR1SSGYGOkZiD7hDnn9TH9Wh0v4S1NJa8FN5kiv6rtJ3TXP8Pu9Drl0tHi0, including suggesting an annotation that controls attributes including visibility.
@mren already mentioned it, but the concept of LTO visibility is quite important. For example, because Swift currently lacks something like this, Swift's LLVM-VFE implementation currently uses the visibility of otherwise unnecessary dispatch thunks to reason about the existence of virtual calls outside the LTO unit.
I know @mren already suggested a visibility annotation, but if any of this is compelling perhaps we might consider at least an underscored attribute to start. For instance the previous suggestion of @symbolName(swift: "swift_foo", visibility: internal) would seem suitable.
1 Like
Do you happen to know how this export list was derived in the first place? Another possibility to keep things working in terms of modules, once internal import is the default import behavior, might be to allow for selective reexport of symbols, so that the primary interface corresponding to your dylib can reexport the APIs it needs to transitively expose.
1 Like
To start, we essentially run nm on all our images and aggregate any undefined symbols defined in dylib Foo, which then becomes the exported symbols list for that dylib.
once
internal importis the default import behavior, might be to allow for selective reexport of symbols, so that the primary interface corresponding to your dylib can reexport the APIs it needs to transitively expose.
I'm not sure rexporting of symbols is the problem though, when calculating the symbols we'd need to export, I'm running nm -g --defined-only on the image for each Swift module we'd need to export. IIUC that wouldn't include a symbol rexported from another module.
It's just that operating at the module level is too coarse grain. I suppose I don't see why the symbol visibility across a package boundary is done with symbol granularity, but across a dylib boundary, it needs to be at module granularity.
In a perhaps-too-idealized world, packages are generally built from source and statically linked together into an executable, so nothing ultimately needs external visibility, and the binary can be pruned down to include only the parts of the packages that were actually used. By contrast, when a module is built as a dynamic library that gets shipped in an Apple OS, then its whole public API needs to be exported to any possible client that may want to link against it, so all of its public ABI entry points need to be visible. Swift (and ObjC before it, to be fair) don't really want there to be more than one runtime manifestation of any type, protocol conformance, or other runtime entity, so although in the uncontrolled real world there can be a lot of space in between those two points, it nonetheless becomes more problematic the more you get away from either of them. That's why I'd like to understand what's leading to the situation you're describing, where a single dynamic library is exporting only part of the public API of multiple modules. Is the dylib only distributed as part of a closed application distribution, so you're using the export list to prune it down to only the API used by its clients within that distribution?
1 Like
Yes exactly.
Okay, I can see how module-level controls are too coarse. Are you sure that automatically creating an exports list isn't already the right tooling design for what you're trying to do, though? You're doing closed application distribution, but IIRC you have mutiple closed application distributions, and they presumably have slightly different optimal exports lists. Setting the visibility on the declaration seems like it'd lead to sub-optimal behavior because something would have to be exported if it was used in any closed distribution. Basically, I think this information is inherently extrinsic to the source and belongs in a separate input rather than being a new level of access control. In theory, that input could be more semantic, i.e. not using mangled names; but that just seems like unnecessary complexity and a likely source of bugs since the easiest way to produce it is from the imports list of other modules, which is naturally going to be a bunch of mangled names. At that point, we're basically talking about an exports list.
Now, I think we could probably take better advantage of the exports list in the compiler if you can feed that into us. Like, you could change up your build system so that you just build modules and .tbds for dependencies but don't actually generate code, and then you generate code in reverse order using your knowledge of what symbols are actually used outside of the module. There are probably performance optimizations that would take advantage of that, but if nothing else, you'd probably end up with a significant compile-time win from skipping the emission of a bunch of symbols.
2 Likes
Yes, we're aware of this, but we're willing to at least try to accept that in-efficiency (ie. visibility attributes would be conservative enough to accommodate all our applications).
Are you sure that automatically creating an exports list isn't already the right tooling design for what you're trying to do, though?
Maybe to provide some more clarity as to our motivation. We currently only do this automatic export list calculation for release builds because it is too slow for regular iteration (we have to build some of our dylibs twice). This leads to issues creeping into releases, as developers don't test with the release configuration. To mitigate this, and make the whole symbol visibility situation a little more observable to developers, our idea is to have explicit source based visibility annotations that the developer can control and the consequences of which are evident as they develop.
Like, you could change up your build system so that you just build modules and
.tbds for dependencies but don't actually generate code, and then you generate code in reverse order using your knowledge of what symbols are actually used outside of the module.
That's certainly an interesting suggestion, and something we've considered. It would likely help mitigate some of our performance issues, but we'd still have the observability issues. I'll discuss it with my peers.
2 Likes
I understand the idea, but I don't currently see a path for it actually being added to the language, which is why I'd like to focus on alternative approaches.
On macOS, bundles can link against the host programâs symbols. RTLD_MAIN_ONLY also lets you look them up. Bundles have fallen out of favor and dlsym is very hard to use with Swift symbols, though, so that might not matter.
1 Like
Yeah, I think for situations where you're using bundles/reverse-linking .so's to the executable, or using dlsym on your main module, then having explicit symbol visibility control would be reasonable, though it probably also needs to come with symbol name and calling convention control to be fully usable the way it is in C.
1 Like