SwiftGlibc: Use VFS overlay instead of -fmodule-map-file

Development Compiler
1

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right
thing in the code.

Any thought?

2

Hi, Rintaro. That’s a clever solution; it would mean we wouldn’t be blocked by talking to the Clang folks about making module map search paths SDKROOT-relative. That said, the VFS is a fairly, well, hacky piece of Clang, and I’m not sure we’d want to add a new dependency on it. Ben, Daniel, what do you think?

Jordan

···

On May 21, 2016, at 05:36, rintaro ishizaki via swift-dev <swift-dev@swift.org> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
Generate native glibc.modulemap when cross-compiling by karwa · Pull Request #2486 · apple/swift · GitHub

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

3

I’m not sure if we could dynamically generate the YAML because of multiarch, and because the GCC triple isn’t always the same as the LLVM triple.

For example, you might built ‘armv7-linux-gnueabihf’ for Swift/LLVM, but GCC only has ‘arm-linux-gnueabihf’.

But yeah, thats a really good find, it’s a flexible solution that would allow the same glibc.modulemap to be used by a host compiler when cross-compiling, and natively on the target.

Karl

···

On 21 May 2016, at 14:36, rintaro ishizaki via swift-dev <swift-dev@swift.org> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
Generate native glibc.modulemap when cross-compiling by karwa · Pull Request #2486 · apple/swift · GitHub

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

4

That said, the VFS is a fairly, well, hacky piece of Clang, and I’m not

sure we’d want to add a new dependency on it. Ben, Daniel, what do you
think?

Actually, VFS overlay seems to be relatively new feature and looks unstable.
For instance,
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay#diff-3d555304611b40b626f0d2abd95b8e53R412
Because, with "-sysroot /", Clang tries to find the module map
with path string //usr/include/module.map.
In *real* filesystem, //usr/include/module.map is usually equivalent to
/usr/include/module.map.
But in overlaid VFS, that needs exact string match. i.e. /usr doesn't match
//usr.

Nevertheless, I think, it's worth to implement this solution.
Even if Clang would have been modified to search headers SYSROOT
relative, it would still hard to import Clang builtin headers, I think.
To *properly* import them, as far as I understand, Clang requires bare
filename
in module map, such as header "limits.h".

···

2016-05-24 1:20 GMT+09:00 Jordan Rose <jordan_rose@apple.com>:

Hi, Rintaro. That’s a clever solution; it would mean we wouldn’t be
blocked by talking to the Clang folks about making module map search paths
SDKROOT-relative. That said, the VFS is a fairly, well, hacky piece of
Clang, and I’m not sure we’d want to add a new dependency on it. Ben,
Daniel, what do you think?

Jordan

On May 21, 2016, at 05:36, rintaro ishizaki via swift-dev < > swift-dev@swift.org> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
Generate native glibc.modulemap when cross-compiling by karwa · Pull Request #2486 · apple/swift · GitHub

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right
thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

5

Hi Karl,
(Just re-sending with CC: swift-dev)

I agree. This solution does not solve multiarch problem.
The installed module map still contains hardcoded arch directory name
of building machine.

e.g.
lib/swift/linux/x86_64/glibc.modulemap built on Ubuntu 15.10

    module ioctl {
      header "x86_64-linux-gnu/sys/ioctl.h"
      export *
    }

···

2016-05-28 4:49 GMT+09:00 Karl Wagner <razielim@gmail.com>:

I’m not sure if we could dynamically generate the YAML because of
multiarch, and because the GCC triple isn’t always the same as the LLVM
triple.

For example, you might built ‘armv7-linux-gnueabihf’ for Swift/LLVM, but
GCC only has ‘arm-linux-gnueabihf’.

But yeah, thats a really good find, it’s a flexible solution that would
allow the same glibc.modulemap to be used by a host compiler when
cross-compiling, and natively on the target.

Karl

On 21 May 2016, at 14:36, rintaro ishizaki via swift-dev < > swift-dev@swift.org> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
Generate native glibc.modulemap when cross-compiling by karwa · Pull Request #2486 · apple/swift · GitHub

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right
thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

6

> That said, the VFS is a fairly, well, hacky piece of Clang, and I’m not sure we’d want to add a new dependency on it. Ben, Daniel, what do you think?

Sorry for not replying earlier. Daniel, I believe you considered building on top of the VFS for module maps like this before, and decided against it. Can you share your reasoning?

Actually, VFS overlay seems to be relatively new feature and looks unstable.
For instance,
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay#diff-3d555304611b40b626f0d2abd95b8e53R412
Because, with "-sysroot /", Clang tries to find the module map
with path string //usr/include/module.map.
In *real* filesystem, //usr/include/module.map is usually equivalent to /usr/include/module.map.
But in overlaid VFS, that needs exact string match. i.e. /usr doesn't match //usr.

Neat! The problem is almost certainly that “//usr” is also a net name, so it would be treated as a single path component by llvm’s path handling code. We would need to explicitly add some fallback case to handle this. Worth a bug report!

We certainly are not just doing exact string matches against paths :-)

Nevertheless, I think, it's worth to implement this solution.
Even if Clang would have been modified to search headers SYSROOT
relative, it would still hard to import Clang builtin headers, I think.
To properly import them, as far as I understand, Clang requires bare filename
in module map, such as header "limits.h".
https://github.com/apple/swift-clang/blob/b9c42fe/lib/Lex/ModuleMap.cpp#L1860

Correct.

···

On May 25, 2016, at 8:04 PM, rintaro ishizaki <fs.output@gmail.com> wrote:

2016-05-24 1:20 GMT+09:00 Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>>:
Hi, Rintaro. That’s a clever solution; it would mean we wouldn’t be blocked by talking to the Clang folks about making module map search paths SDKROOT-relative. That said, the VFS is a fairly, well, hacky piece of Clang, and I’m not sure we’d want to add a new dependency on it. Ben, Daniel, what do you think?

Jordan

On May 21, 2016, at 05:36, rintaro ishizaki via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
https://github.com/apple/swift/pull/2486

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing right thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

7

Neat! The problem is almost certainly that “//usr” is also a net name,

so it would be treated as a single path component by llvm’s path handling
code. We would need to explicitly add some fallback case to handle this.
Worth a bug report!

We certainly are not just doing exact string matches against paths :-)

Thank you for clarifying!
Filed a bug report:
https://llvm.org/bugs/show_bug.cgi?id=27909

···

2016-05-27 1:41 GMT+09:00 Ben Langmuir <blangmuir@apple.com>:

On May 25, 2016, at 8:04 PM, rintaro ishizaki <fs.output@gmail.com> wrote:

> That said, the VFS is a fairly, well, hacky piece of Clang, and I’m not
sure we’d want to add a new dependency on it. Ben, Daniel, what do you
think?

Sorry for not replying earlier. Daniel, I believe you considered building
on top of the VFS for module maps like this before, and decided against
it. Can you share your reasoning?

Actually, VFS overlay seems to be relatively new feature and looks
unstable.
For instance,

https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay#diff-3d555304611b40b626f0d2abd95b8e53R412
Because, with "-sysroot /", Clang tries to find the module map
with path string //usr/include/module.map.
In *real* filesystem, //usr/include/module.map is usually equivalent to
/usr/include/module.map.
But in overlaid VFS, that needs exact string match. i.e. /usr doesn't
match //usr.

Neat! The problem is almost certainly that “//usr” is also a net name, so
it would be treated as a single path component by llvm’s path handling
code. We would need to explicitly add some fallback case to handle this.
Worth a bug report!

We certainly are not just doing exact string matches against paths :-)

Nevertheless, I think, it's worth to implement this solution.
Even if Clang would have been modified to search headers SYSROOT
relative, it would still hard to import Clang builtin headers, I think.
To *properly* import them, as far as I understand, Clang requires bare
filename
in module map, such as header "limits.h".

https://github.com/apple/swift-clang/blob/b9c42fe/lib/Lex/ModuleMap.cpp#L1860

Correct.

2016-05-24 1:20 GMT+09:00 Jordan Rose <jordan_rose@apple.com>:

Hi, Rintaro. That’s a clever solution; it would mean we wouldn’t be
blocked by talking to the Clang folks about making module map search paths
SDKROOT-relative. That said, the VFS is a fairly, well, hacky piece of
Clang, and I’m not sure we’d want to add a new dependency on it. Ben,
Daniel, what do you think?

Jordan

On May 21, 2016, at 05:36, rintaro ishizaki via swift-dev < >> swift-dev@swift.org> wrote:

Hi all,

Recently, a couple of PR are posted regarding
glibc.modulemap in cross-compiling environment.

[Platform] Simplify GLIBC_INCLUDE_PATH detection by rintaro · Pull Request #2473 · apple/swift · GitHub
https://github.com/apple/swift/pull/2486

The problem is that glibc.modulemap contains hardcoded SDKROOT in it.
To resolve that, how about using virtual file system feature in Clang?

I mean, prepare YAML like this:

{
  "use-external-names": false,
  "roots": [
    {
      "type": "file",
      "name": "${SYSROOT}/usr/include/module.map",
      "external-contents": "${RSRC}/${platform}/${arch}/glibc.modulemap"
    }
  ]
}

Then, invoke Clang with -ivfsoverlay argument.

Of course, we have to dynamically create YAML based on -sdk and -target
argument of the Swift compiler.
Luckily, Clang provides convenient YAML builder for this:
http://clang.llvm.org/doxygen/classclang_1_1vfs_1_1YAMLVFSWriter.html
It's easy and trivial work to build that dynamically.

Using this feature, glibc.modulemap can be rather simple.
No need to specify absolute path.
It can be simple as /usr/include/module.map in Darwin platforms:

    module ctype {
      header "ctype.h"
      export *
    }

And, it makes easy to import Clang builtin headers like "limits.h".

Here is the PoC code:
https://github.com/apple/swift/compare/master...rintaro:clang-vfsoverlay
It works, and passes all Swift test suite.

Current my concerns are:
* The VFS overlay is the right way in the first place?
* Since I'm a very newbie in C++ programming, I'm not sure I'm doing
right thing in the code.

Any thought?

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev