COFF indirect addressing and protocol conformance tables ABI issues


(Saleem Abdulrasool) #1

Hello again,

I come bearing more problems :-).

I seem to have found another point in the ABI which prevents an easy port
of swift to Windows without an emulation layer underneath. This time it
deals with the protocol conformance table. The table is constructed with a
direct reference to protocol. However, because the protocol lies in an
external module, this ends up being a problem as it must be indirectly
addressed.

There is a workaround in place for generating a GOT equivalent entry,
however, that still doesnt indirect through the pointer, which needs to be
done to address the executable model on Windows.

This along with the DLL storage changes (
https://github.com/apple/swift/pull/2080) are the currently two known ABI
issues preventing the port to windows (with cross-compiling!). Im hopeful
that the former can be merged soon. If we can work out a way to address
this last issue, I think that it should be possible to get a complete
windows port of swift working (modulo API completeness) shortly thereafter
without the need for any emulation layer (i.e. cygwin or MinGW)!

···

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org


(John McCall) #2

Is it not possible to emit the GOT equivalent entry as a reference to the appropriate entry in the import lookup table?

John.

···

On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
Hello again,

I come bearing more problems :-).

I seem to have found another point in the ABI which prevents an easy port of swift to Windows without an emulation layer underneath. This time it deals with the protocol conformance table. The table is constructed with a direct reference to protocol. However, because the protocol lies in an external module, this ends up being a problem as it must be indirectly addressed.

There is a workaround in place for generating a GOT equivalent entry, however, that still doesnt indirect through the pointer, which needs to be done to address the executable model on Windows.


(Saleem Abdulrasool) #3

> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy
port of swift to Windows without an emulation layer underneath. This time
it deals with the protocol conformance table. The table is constructed
with a direct reference to protocol. However, because the protocol lies in
an external module, this ends up being a problem as it must be indirectly
addressed.
>
> There is a workaround in place for generating a GOT equivalent entry,
however, that still doesnt indirect through the pointer, which needs to be
done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to the
appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the
value rather than the synthetic. Perhaps I am misunderstanding something?

···

On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com> wrote:

> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:

John.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org


(John McCall) #4

I don't know what you mean. I assume this is the reference to the protocol structure from the global protocol-conformances list. This reference is a relative offset, either directly to the protocol structure or indirectly to a global containing a pointer to an external protocol structure. PE's import lookup table is an array of pointers to external objects.

I could easily believe that COFF doesn't allow us to express a relative offset to an entry in the import lookup table. However, it sounds like you think this needs to be *doubly* indirected, i.e. the conformance list needs to be a relative offset to a variable containing a pointer to a variable containing a pointer to the import lookup table. I'm not sure why that would be.

John.

···

On Jul 5, 2016, at 5:56 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:
> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:
> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy port of swift to Windows without an emulation layer underneath. This time it deals with the protocol conformance table. The table is constructed with a direct reference to protocol. However, because the protocol lies in an external module, this ends up being a problem as it must be indirectly addressed.
>
> There is a workaround in place for generating a GOT equivalent entry, however, that still doesnt indirect through the pointer, which needs to be done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to the appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the value rather than the synthetic. Perhaps I am misunderstanding something?


(Saleem Abdulrasool) #5

> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy
port of swift to Windows without an emulation layer underneath. This time
it deals with the protocol conformance table. The table is constructed
with a direct reference to protocol. However, because the protocol lies in
an external module, this ends up being a problem as it must be indirectly
addressed.
>
> There is a workaround in place for generating a GOT equivalent entry,
however, that still doesnt indirect through the pointer, which needs to be
done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to the
appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the
value rather than the synthetic. Perhaps I am misunderstanding something?

I don't know what you mean. I assume this is the reference to the
protocol structure from the global protocol-conformances list. This
reference is a relative offset, either directly to the protocol structure
or indirectly to a global containing a pointer to an external protocol
structure. PE's import lookup table is an array of pointers to external
objects.

Exactly, in the case that identified the issue, it is _TMps12OutputStream

I could easily believe that COFF doesn't allow us to express a relative
offset to an entry in the import lookup table.

I don't think that there is any way to represent that in IR, and I don't
know if COFF allows us to encode such a thing. I will see if I can
construct an object file/library and see if we can get away with that.

However, it sounds like you think this needs to be *doubly* indirected,
i.e. the conformance list needs to be a relative offset to a variable
containing a pointer to a variable containing a pointer to the import
lookup table. I'm not sure why that would be.

John.

···

On Tue, Jul 5, 2016 at 6:28 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 5, 2016, at 5:56 PM, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:
On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com> wrote:

> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org> >> wrote:

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org


(John McCall) #6

> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy port of swift to Windows without an emulation layer underneath. This time it deals with the protocol conformance table. The table is constructed with a direct reference to protocol. However, because the protocol lies in an external module, this ends up being a problem as it must be indirectly addressed.
>
> There is a workaround in place for generating a GOT equivalent entry, however, that still doesnt indirect through the pointer, which needs to be done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to the appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the value rather than the synthetic. Perhaps I am misunderstanding something?

I don't know what you mean. I assume this is the reference to the protocol structure from the global protocol-conformances list. This reference is a relative offset, either directly to the protocol structure or indirectly to a global containing a pointer to an external protocol structure. PE's import lookup table is an array of pointers to external objects.

Exactly, in the case that identified the issue, it is _TMps12OutputStream

I could easily believe that COFF doesn't allow us to express a relative offset to an entry in the import lookup table.

I don't think that there is any way to represent that in IR, and I don't know if COFF allows us to encode such a thing. I will see if I can construct an object file/library and see if we can get away with that.

The way to represent it in IR is using the same private unnamed_addr representation that we special-case for ELF/Mach-O. I guess it would be a mandatory transformation for COFF.

But if it's not representable in COFF, that's pretty conclusive.

John.

···

On Jul 6, 2016, at 7:33 AM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
On Tue, Jul 5, 2016 at 6:28 PM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:

On Jul 5, 2016, at 5:56 PM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:
On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com <mailto:rjmccall@apple.com>> wrote:
> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:

However, it sounds like you think this needs to be *doubly* indirected, i.e. the conformance list needs to be a relative offset to a variable containing a pointer to a variable containing a pointer to the import lookup table. I'm not sure why that would be.
John.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org


(Saleem Abdulrasool) #7

> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy
port of swift to Windows without an emulation layer underneath. This time
it deals with the protocol conformance table. The table is constructed
with a direct reference to protocol. However, because the protocol lies in
an external module, this ends up being a problem as it must be indirectly
addressed.
>
> There is a workaround in place for generating a GOT equivalent entry,
however, that still doesnt indirect through the pointer, which needs to be
done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to
the appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the
value rather than the synthetic. Perhaps I am misunderstanding something?

I don't know what you mean. I assume this is the reference to the
protocol structure from the global protocol-conformances list. This
reference is a relative offset, either directly to the protocol structure
or indirectly to a global containing a pointer to an external protocol
structure. PE's import lookup table is an array of pointers to external
objects.

Exactly, in the case that identified the issue, it is _TMps12OutputStream

I could easily believe that COFF doesn't allow us to express a relative
offset to an entry in the import lookup table.

I don't think that there is any way to represent that in IR, and I don't
know if COFF allows us to encode such a thing. I will see if I can
construct an object file/library and see if we can get away with that.

The way to represent it in IR is using the same private unnamed_addr
representation that we special-case for ELF/Mach-O. I guess it would be a
mandatory transformation for COFF.

But if it's not representable in COFF, that's pretty conclusive.

I did look into this. It is, as I suspected, not possible to represent an
offset to the value in the ILT or IAT.

···

On Wed, Jul 6, 2016 at 9:43 AM, John McCall <rjmccall@apple.com> wrote:

On Jul 6, 2016, at 7:33 AM, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:
On Tue, Jul 5, 2016 at 6:28 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 5, 2016, at 5:56 PM, Saleem Abdulrasool <compnerd@compnerd.org> >> wrote:
On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com> wrote:

> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org> >>> wrote:

John.

However, it sounds like you think this needs to be *doubly* indirected,
i.e. the conformance list needs to be a relative offset to a variable
containing a pointer to a variable containing a pointer to the import
lookup table. I'm not sure why that would be.

John.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org


(Saleem Abdulrasool) #8

> Hello again,
>
> I come bearing more problems :-).
>
> I seem to have found another point in the ABI which prevents an easy
port of swift to Windows without an emulation layer underneath. This time
it deals with the protocol conformance table. The table is constructed
with a direct reference to protocol. However, because the protocol lies in
an external module, this ends up being a problem as it must be indirectly
addressed.
>
> There is a workaround in place for generating a GOT equivalent entry,
however, that still doesnt indirect through the pointer, which needs to be
done to address the executable model on Windows.

Is it not possible to emit the GOT equivalent entry as a reference to
the appropriate entry in the import lookup table?

I had tried that, but it would still just be a constant reference to the
value rather than the synthetic. Perhaps I am misunderstanding something?

I don't know what you mean. I assume this is the reference to the
protocol structure from the global protocol-conformances list. This
reference is a relative offset, either directly to the protocol structure
or indirectly to a global containing a pointer to an external protocol
structure. PE's import lookup table is an array of pointers to external
objects.

Exactly, in the case that identified the issue, it is _TMps12OutputStream

I could easily believe that COFF doesn't allow us to express a relative
offset to an entry in the import lookup table.

I don't think that there is any way to represent that in IR, and I don't
know if COFF allows us to encode such a thing. I will see if I can
construct an object file/library and see if we can get away with that.

The way to represent it in IR is using the same private unnamed_addr
representation that we special-case for ELF/Mach-O. I guess it would be a
mandatory transformation for COFF.

But if it's not representable in COFF, that's pretty conclusive.

I did look into this. It is, as I suspected, not possible to represent an
offset to the value in the ILT or IAT.

This seems to be the only blocking issue for the Windows port. There is a
minor build wrinkle that needs to be worked out, but everything else seems
to be more or less ready for enabling a Windows port.

···

On Sat, Jul 9, 2016 at 4:07 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:

On Wed, Jul 6, 2016 at 9:43 AM, John McCall <rjmccall@apple.com> wrote:

On Jul 6, 2016, at 7:33 AM, Saleem Abdulrasool <compnerd@compnerd.org> >> wrote:
On Tue, Jul 5, 2016 at 6:28 PM, John McCall <rjmccall@apple.com> wrote:

On Jul 5, 2016, at 5:56 PM, Saleem Abdulrasool <compnerd@compnerd.org> >>> wrote:
On Tue, Jul 5, 2016 at 10:30 AM, John McCall <rjmccall@apple.com> wrote:

> On Jul 3, 2016, at 2:40 PM, Saleem Abdulrasool <compnerd@compnerd.org> >>>> wrote:

John.

However, it sounds like you think this needs to be *doubly* indirected,
i.e. the conformance list needs to be a relative offset to a variable
containing a pointer to a variable containing a pointer to the import
lookup table. I'm not sure why that would be.

John.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org