I think that the concern that I had was for section merging. The constant data is being merged into .rdata which then prevents us from isolating the tables. The segment,section which is common on MachO is not really there on PE/COFF and ELF, which only have the "segment" component. Of course we could use custom "segments" for the metadata to work around the limitation, but that does add more difference between the various formats.
1 Like
@compnerd and I have discussed this a bit. It's not quite sufficient for the runtime's needs to just passively read these sections since, as you've noted, the runtime needs to know when new images are loaded and ready to inspect. But we could minimalize the static constructor to simply "ping" the runtime like how it's done on Darwin.
That sounds familiar. I admit to not having dug into the ELF case much.
I think we already do? Or rather, we already emit data into various custom PE sections anyway, and the use of segments with Mach-O doesn't really make much of a difference here.
I'm playing with fire here @Joe_Groff, but does dl_iterate_phdr() help us? The dlpi_phdr field points to the ELF header for each loaded image.
1 Like
This field points to the the array of program headers (segment headers), not the ELF header. What we need are the section headers (which are retrievable from the ELF header) but that's stripped when it's mapped into memory.
1 Like
Edit: I have a proof of concept that's fully fleshed out for Darwin, Linux (ELF), and Windows (COFF) now.
6 Likes
There's no reason we can't work with glibc / musl / etc. on this to eventually eliminate the need for static constructors. If ELF loaders don't want to map section headers dynamically, that's an understandable choice, but there ought to be some system to find information passively in all loaded shared objects. That could just be symbol lookup as long as there's some way to recursively do the lookup in every shared object — that would make it a static problem to construct a well-formed, self-delimited object for that symbol, but that's a solvable problem.
1 Like
I believe, sadly, that it's the case that the section headers don't even have to be in the ELF image on disk (for a loadable image, that is). So I don't think what you have for ELF will really work — also, it should probably be using the build ID to check that the image is the same as the thing you're looking for sections in.
The things you can look for in ELF are program headers (which broadly correspond to segments in Mach-O) and notes. Both of those get mapped into memory for executables. If you need access to sections, you have to use symbols (either linker-generated ones, or explicit ones).
1 Like
It isn't just that they don't get mapped dynamically; they can be completely omitted from the loadable ELF binary, because it's only the program headers that are relevant at that point.
I am broadly aware of the problem. Doesn't leave a whole lot of solutions available though.
Perhaps we can teach the linker to emit a specialized swift5_sections section containing relative ranges for all other sections, and then that could be the sole section that the static constructor (SwiftRT+ELF) needs to worry about. Basically reconstruct the section headers.
Since we already need a static constructor to alert the runtime to the presence of new images, it isn't clear to me that there's a benefit to putting the metadata section information in a particular place in the binary, when the static constructor itself can be linked to directly reference a table containing the information like we do now.
Well yes, the data in that scenario doesn't necessarily need its own section per se, but it would be difficult to collate information about all sections in the binary in a single constructor function like we have today.
Teaching the linker things on Linux is hard, because there are at least three commonly used linkers (admittedly one of them, the BFD linker, won't work for Swift anyway at the moment), and most of them are non-permissively licensed which puts additional obstacles in the way.
In principle it might be possible to use a custom linker script, in which case we could stuff all of whatever you want into a custom segment with its program header type set to some magic number in the PT_LOOS..PT_HIOS range (maybe we could reserve 0x6531f7xx, for instance, which would give us 256 program header types we could allocate as needed).
1 Like
I suspect Swift does not have the ability to reserve anything on Linux.
Perhaps if we reserve our guaranteed knowledge to "things emitted by Swift", the compiler could be in charge of creating the appropriate table rather than the linker. Then whatever API we expose to pair with @_section doesn't need to concern itself with non-Swift sections or symbols.
It isn't really reserving things "on Linux". The PT_OS constants are an ELF thing. The GNU ones are used everywhere the GNU tools exist, including non-Linux ELF systems; we'd just be doing the same, hence picking 0x6531f7xx (because it seems unlikely anyone else will choose a number that "looks like" Swift).
Anyway, my feeling is that the @linkage specification is useful, but it might be too low-level for some use cases (because of things like this) and I agree that we should probably have some combination of the compiler/runtime/some package build some useful and easy to use abstraction over the top for the kind of thing you're thinking about.
1 Like