Do nominal type descriptors have to be in .rodata to avoid breaking things?

Development Compiler
1

I'm investigating optimising the linking of metadata records on my platform. I use -function-sections and -data-sections when compiling bitcode with llc together with gc-sections in the linker to eliminate code and data I'm not using. However I'm finding a lot of unused stuff is still getting linked sometimes.

My investigation seems to point to the fact that nominal type descriptors (and other types of metadata record) are stored in .rodata in each object file, which I think effectively overrides -data-sections, is that right?

I'm far from an LLVM expert so, not sure.

On a preliminary analysis of GenMeta.cpp (etc), it looks like maybe the cause is the call to setTrueConstGlobal where the nominal type descriptor metadata record is emitted...

    llvm::Constant *emit() {
      asImpl().layout();
      auto addr = IGM.getAddrOfTypeContextDescriptor(Type, HasMetadata,
                                                     B.finishAndCreateFuture());
      auto var = cast<llvm::GlobalVariable>(addr);

      if (IGM.getOptions().VirtualFunctionElimination) {
        asImpl().addVTableTypeMetadata(var);
      }

      var->setConstant(true);
      IGM.setTrueConstGlobal(var);
      return var;
    }

For the sake of argument, would it be harmful to remove this call? Then perhaps llc would be free to store each record in its own .data section and the linker could only link the parts it needed?

Sorry if I'm missing something dumb and obvious!

Any help anyone can give, gratefully received!

Carl

2

I think this is intended to be a pure optimization, and removing it would not be harmful except to performance. I’d also say that in theory being marked as a “true const global” shouldn’t mean “must go in a section called ‘.rodata’”, so maybe this should be treated as an LLVM issue in the implementation of -data-sections.

That said, I don’t think -gc-sections is safe with Swift code because of its relative references, which to a non-LTO linker will not look like symbol references. Swift is definitely very conservative now in what it marks live-for-linking, but there’s still a reason for it.

3

We do consider it import for security that nominal type descriptors end up in a section that will be mapped read-only. Nominal type descriptors contain a significant amount of data that could be used to take over a process if overwritten. We don't care exactly which section that will be, but usually we can't force specific memory permissions without picking a specific section.

5 Likes
4

That makes a lot of sense, thank you.

As an interesting experiment I tried commenting out IGM.setTrueConstGlobal(var); in TypeContextDescriptorBuilderBase.emit(). This "worked" for me. It put the nominal type descriptor for the class type I was investigating into a section .rodata.$ss29_AVRArrayBufferStorageManagerCMn in the object file. This might seem like a trivial change, but it allowed the tools to "untangle" the necessary symbols for metadata records needed in my trivial sample programs and the symbols that aren't needed. What seemed to be happening was when metadata records from my standard library were referenced, the unresolved symbol chain ended up linking nominal type descriptors from .rodata in one of the objects in the static library, that would also have other things that weren't actually needed but were in the same file, all their symbols got resolved, linked out to other object files and so on... before I knew it, 450kb of unused code was linked. Now my simple test program is back down to 3-4k, which is tractable.
(I think @dragomirecky saw some of the same issues in his investigations)

At the moment, the platform is in a slightly strange state because we have just started to introduce a very small amount of runtime, including a trimmed down ARC. We still don't support the vast majority of things you'd expect to see, but it's not an issue in the very simple programs we are creating. And we very much emphasise writing in such a way that things like generic type params are known at compile time and fully LTO optimised away. It's a work in progress of course. But in a lot of ways, most of the metadata records being emitted in our program are linked and never really used by the runtime in any meaningful way I think.

With regard to security, it's a really interesting point. We aren't going to have the same memory protections available as "big" processors do. If and when we start to really meaningfully need to access a lot of these metadata records, we'll probably need to resolve a lot of platform bugs in AVR to do with RAM/flash address spaces. At that point, I think it would be good to move all the metadata records the compiler emits on our platform into flash memory. That should help solve the issue of security for the nominal type descriptor (AVR micros have security protection for flash writes).

One thing I still don't quite understand, from a security point of view, shouldn't the above arguments apply also to the full type metadata?

When I looked at the llvm IR for the full type metadata and for the nominal type descriptor, they looked something like this...

@"$ss29_AVRArrayBufferStorageManagerCMf" = linkonce_odr hidden global <{ void (%Ts29_AVRArrayBufferStorageManagerC*) addrspace(1)*, i8**, i16, %swift.type*, i32, i32, i32, i16, i16, i32, i32, <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>*, i8 addrspace(1)*, i16, i8 addrspace(1)*, i8 addrspace(1)*, i8 addrspace(1)*, %Ts29_AVRArrayBufferStorageManagerC* (i8*, %swift.type*) addrspace(1)* }> 
<{
void (%Ts29_AVRArrayBufferStorageManagerC*) addrspace(1)* @"$ss29_AVRArrayBufferStorageManagerCfD",
i8** @"$sBoWV",
i16 0,
%swift.type* null,
i32 2,
i32 0,
i32 6,
i16 1,
i16 0,
i32 46,
i32 4,
<{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>* @"$ss29_AVRArrayBufferStorageManagerCMn",
i8 addrspace(1)* null,
i16 4, 
i8 addrspace(1)* bitcast (void () addrspace(1)* @swift_deletedMethodError to i8 addrspace(1)*), 
i8 addrspace(1)* bitcast (void () addrspace(1)* @swift_deletedMethodError to i8 addrspace(1)*), 
i8 addrspace(1)* bitcast (void () addrspace(1)* @swift_deletedMethodError to i8 addrspace(1)*), 
%Ts29_AVRArrayBufferStorageManagerC* (i8*, %swift.type*) addrspace(1)* @"$ss29_AVRArrayBufferStorageManagerCyABBpcfC" }>, align 2



@"$ss29_AVRArrayBufferStorageManagerCMn" = protected constant <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }> 
<{
 i32 -2147483568,
 i16 sub (
   i16 ptrtoint (<{ i32, i32, i16 }>* @"$ssMXM" to i16),
   i16 ptrtoint (i16* getelementptr inbounds (
      <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>,
      <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>* @"$ss29_AVRArrayBufferStorageManagerCMn",
      i32 0,
      i32 1) to i16)),
 i16 sub (
    i16 ptrtoint ([30 x i8]* @.str.29._AVRArrayBufferStorageManager to i16),
    i16 ptrtoint (i16* getelementptr inbounds (
       <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>,
       <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>* @"$ss29_AVRArrayBufferStorageManagerCMn",
       i32 0,
       i32 2) to i16)),
 i16 sub (
    i16 ptrtoint (%swift.metadata_response (i16) addrspace(1)* @"$ss29_AVRArrayBufferStorageManagerCMa" to i16),
    i16 ptrtoint (i16* getelementptr inbounds (
       <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>,
       <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>* @"$ss29_AVRArrayBufferStorageManagerCMn",
       i32 0,
       i32 3) to i16)),
 i32 0,
 i32 0,
 i32 2,
 i32 21,
 i32 5,
 i32 1,
 i32 16,
 i32 17,
 i32 4,
 %swift.method_descriptor { i32 18, i16 0 },
 %swift.method_descriptor { i32 19, i16 0 },
 %swift.method_descriptor { i32 20, i16 0 },
 %swift.method_descriptor { i32 1,
    i16 sub (
       i16 ptrtoint (%Ts29_AVRArrayBufferStorageManagerC* (i8*, %swift.type*) addrspace(1)* @"$ss29_AVRArrayBufferStorageManagerCyABBpcfC" to i16),
       i16 ptrtoint (i16* getelementptr inbounds (
          <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>,
          <{ i32, i16, i16, i16, i32, i32, i32, i32, i32, i32, i32, i32, i32, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor, %swift.method_descriptor }>* @"$ss29_AVRArrayBufferStorageManagerCMn",
          i32 0,
          i32 16,
          i32 1) to i16))
 }
}>,
section ".rodata",  <<< IF WE DONT PUT IT IN RODATA, WHERE DOES IT GO?
align 4

I've added some line wrapping/whitespace to try and make it a bit easier to read!

But anyway, the point I'm not quite understanding is the full type metadata is IRGen'ed as @"$ss29_AVRArrayBufferStorageManagerCMf" = linkonce_odr hidden global ... but the nominal type descriptor is IRGen'ed as @"$ss29_AVRArrayBufferStorageManagerCMn" = protected constant ... section ".rodata". So does that mean that the type metadata is "in regular data RAM" but the nominal type descriptor is in "protected read only memory"?

It's probably me not understanding the subleties of LLVM again!

Thanks very much again guys!
Carl

(p.s. @jrose the linking so far hasn't caused me any issues on metadata records but, like I mention above, I barely use them at the moment. When I later implement runtime that needs them I may well run into the issues you mention!)

5

Depending on the OS, metadata objects can’t necessarily be in a read-only section because they contain pointers and thus relocations (assuming PIC). Not all OSes support changing page permissions after applying relocations.

1 Like
6

It took me a while to reply because I had to go and understand it better! It sounds like with nominal type descriptors, all the offsets they point to will be within the same object file (NTDs ignore generic type params) and should all point to the rodata section, so the fixups created during lowering can all be resolved to offsets in the same .rodata section, and thus all resolved before the object file is emitted. None become relocations in the final object file? I might be missing the subtleties!

On our architecture, it's all statically linked during build and not PIC, so the relocations between sections and between object files can all get resolved fully during the link. Plus all code effectively becomes read only as it's a Harvard Architecture!

Long story short, I think I've resolved my linker issue, which is a huge win for us. And I've learned a lot more about metadata on the journey.

Much, much thanks guys. Really, really helpful!

Carl

1 Like
7

That sounds right to me, except that IIRC it isn’t just the same object file; relative references may be used for anything in the same Swift module, which the compiler assumes will always be linked together. (In your case that seems trivially true, since you are statically linking everything.) Those will show up as relocations in the object file, but should go away at link time.

8

Interesting stuff!

I suppose the salient background here is the world you guys mostly inhabit would see all of the object files linked together into a dynamic library (libSwiftCore.dylib or libswiftcore.so), some sections of which will be read-only and some will be data. When the dll is mapped into virtual memory in a process, relocations are applied on the data segment and all the rest of it.The ideal is the .rodata segment loads in without relocations being performed. This means it can be shared between processes and is secure from tampering. (I just had to look this up! I hope it's useful for someone else.)

In my world, the ELF is fully statically linked, so the resulting program has no relocations left to do on load, everything is at a predictable address, and is all programmed into read-only memory (flash memory on the microcontroller) before the device is booted up.

Plus of course the general background is on unix machines a 7MB dynamic library loaded into RAM permanently is a trivial overhead, and is shared between all processes that run programs written in Swift. On our platform static linking and gc-sections is essential to get the program size down to 2-30k programs!

9

If sections are GC’d before relative references are resolved, you’ll be okay, but otherwise you are very likely to see things break:

A +========+
  | B @ +4 |
  +========+
B +===========+
  | some data |
  +===========+
C +============+
  | other data |
  +===========+

silently becomes

A +========+
  | B @ +4 | # oops!
  +========+
C +============+
  | other data |
  +===========+

If you’re relying on things being compiled away so you never get in this scenario, I strongly recommend you rip out whatever bits of the runtime read those fields, so you can at least get a deterministic crash.

10

Yeah. Good point. Roughly What parts of the runtime should I pay attention to?

11

I think my last reply was a bit vague and not so well thought through!... we have already pretty much done what you suggested! Almost our entire runtime is removed so we have either linker errors or defined stub actions (pretty similar to a trap but suitable for our platform) as options for the developer.

I think I've got a bit more handle on a bit more of the problem in our builds today. Most compiled units have "llvm.used" pointing to a set of records, even if there's nothing in our program that uses them. So if I include a symbol that's in CoreOperators.o for example, then the second that object file joins the link, the "llvm.used" list causes a set of records to be included in the ELF, those then link in other files which do the same, etc. and before I know it, I've linked 1/3 of the standard library just to service one small thunk!

Although it's clearly not going to fly in the longer run and we will have to solve this kind of thing "properly", I'm going to attempt to remove the "llvm.used" from the bitcode we use to generate the object files in our standard library archive and see what happens!

p.s. I could not work out how "llvm.used" works! I thought it might be some kind of lld magic but it seems not! In my toolchain I compile the bitcode to AVR object files with llc then link them with avr-ld from avr-binutils and I still see the same behaviour! So llc must somehow mark the symbols or the sections they're in as "immune" to garbage collection! More undocumented mysteries of the world! :laughing: