Creating C Language Targets with complicated include layouts or headers

jeudesprits · November 29, 2023, 5:32am

I'm trying to write a wrapper over lexbor C library. But there are some difficulties with it, it has no include folder, and all header files are scattered around the project as follows:

Simply creating soft links to header files inside the include folder does not produce results:

ln -s ../lexbor/source/lexbor/utils/utils.h
ln -s ../lexbor/source/lexbor/html/html.h
...

What should be done in this situation?

At the moment the Package.swift file looks like this:

let lexborExclude: [String] = [
    "./lexbor/examples",
    "./lexbor/packaging",
    "./lexbor/test",
    "./lexbor/utils",
    "./lexbor/CHANGELOG.md",
    "./lexbor/CMakeLists.txt",
    "./lexbor/config.cmake",
    "./lexbor/feature.cmake",
    "./lexbor/INSTALL.md",
    "./lexbor/LICENSE",
    "./lexbor/NOTICE",
    "./lexbor/pvs_studio.sh",
    "./lexbor/README.md",
    "./lexbor/version",
    
    "./lexbor/source/lexbor/core/config.cmake",
    "./lexbor/source/lexbor/css/config.cmake",
    "./lexbor/source/lexbor/dom/config.cmake",
    "./lexbor/source/lexbor/encoding/config.cmake",
    "./lexbor/source/lexbor/html/config.cmake",
    "./lexbor/source/lexbor/ns/config.cmake",
    "./lexbor/source/lexbor/ports/posix/config.cmake",
    "./lexbor/source/lexbor/ports/windows_nt",
    "./lexbor/source/lexbor/punycode/config.cmake",
    "./lexbor/source/lexbor/selectors/config.cmake",
    "./lexbor/source/lexbor/tag/config.cmake",
    "./lexbor/source/lexbor/unicode/config.cmake",
    "./lexbor/source/lexbor/url/config.cmake",
    "./lexbor/source/lexbor/utils/config.cmake",
]

let package = Package(
    name: "Lexbor",
    platforms: [
        .iOS(.v17),
    ],
    products: [
        .library(name: "Lexbor", targets: ["Lexbor"]),
    ],
    targets: [
        .target(name: "Lexbor", dependencies: ["CLexbor"]),
        .target(
            name: "CLexbor",
            exclude: lexborExclude,
            sources: ["./lexbor/source"],
            cSettings: [
                .headerSearchPath("./lexbor/source/**"),
//                .headerSearchPath("./lexbor/source/lexbor/core"),
//                .headerSearchPath("./lexbor/source/lexbor/css"),
//                .headerSearchPath("./lexbor/source/lexbor/dom"),
//                .headerSearchPath("./lexbor/source/lexbor/encoding"),
//                .headerSearchPath("./lexbor/source/lexbor/html"),
//                .headerSearchPath("./lexbor/source/lexbor/ports/posix"),
//                .headerSearchPath("./lexbor/source/lexbor/punycode"),
//                .headerSearchPath("./lexbor/source/lexbor/selectors"),
//                .headerSearchPath("./lexbor/source/lexbor/tag"),
//                .headerSearchPath("./lexbor/source/lexbor/unicode"),
//                .headerSearchPath("./lexbor/source/lexbor/url"),
//                .headerSearchPath("./lexbor/source/lexbor/utils"),
            ]
        ),
        .testTarget(name: "LexborTests", dependencies: ["Lexbor"]),
    ],
    cLanguageStandard: .gnu17
)

code-per-day · November 29, 2023, 2:10pm

Taking a quick look at lexbor's github repo, you could use publicHeadersPath pointed at CLexbor/lexbor/source. Then you at least wouldn't need to go through symlinks or headerSearchPath.

You can also omit the ./ in all your paths, no leading slashes means relative path.

I've had some issues when using #include "..." rather than #include <...> when the package was used as a dependency.

jeudesprits · November 29, 2023, 2:45pm

Just tried it, but it also doesn't work. The only workable way is to copy all header files to include folder (which I really don't want to do) and add shim.h file:

code-per-day · November 29, 2023, 3:51pm

publicHeadersPath essentially renames the include folder, you want to point that to the source folder since all their includes start from there, what's your new Package.swift?

jeudesprits · November 29, 2023, 6:15pm

When I specify "lexbor/sources" as publicHeadersPath it stops seeing the headers.

code-per-day · November 29, 2023, 6:22pm

I've had some issues when using #include "..." rather than #include <...> when the package was used as a dependency.

Maybe it would work with angled includes? I eventually gave up in my specific case and just symlinked all headers, in your case you'd have to place the symlinks in the right directory structure as well.

I might give this a try at the end of my workday, I'm curious why this isn't working!

jeudesprits · November 29, 2023, 6:41pm

I'd try to do that, but there are such a huge number of headers that it seems like too long an undertaking.

Just copied all the headers in include folder and merged(only public ) them into one shim.h. Everything worked, but for some reason I can't even pass a simple test, application crashes.

@_implementationOnly
import CLexbor
import Darwin

public func test() {
    let html = "<div>Works fine!</div>"
    guard let document = lxb_html_document_create() else { preconditionFailure() }
    assert(html.withCString({ lxb_html_document_parse(document, $0, strlen($0)) }) == LXB_STATUS_OK.rawValue)
    guard let tagName = document
        .pointer(to: \.body)!
        .withMemoryRebound(to: lxb_dom_element_t.self, capacity: 1, {
            lxb_dom_element_qualified_name($0, nil)
        })
    else { preconditionFailure() }
    print(String(cString: tagName))
    lxb_html_document_destroy(document)
}

Lancelotbronner · November 29, 2023, 10:35pm

The following screenshot shows a package that builds using no symlinks, header copies, shims or module map.

It seems to import windows.h so it stops compiling at that point. I'll need another few hours to spin up another Windows VM. The previous one.. huh.. fell down the stairs...

Lancelotbronner · November 30, 2023, 3:59am

It crashes because you weren't dereferencing the right thing. I've published a Github repo which builds the target and successfully executes that test you sent (with modifications to avoid the crash).

I also had to modify the res.h files generated by the python scripts, they're relying on textual inclusion and Swift builds C targets as modules. You'll probably want to modify the python scripts and upstream the changes.

The package currently ignores the Windows port, you'd probably want to figure out a way to provide a target which builds with it.

 // lexbor/css/selectors/pseudo_ref.h
 #include "lexbor/core/shs.h"
 #include "lexbor/css/selectors/pseudo_const.h"
+#include "lexbor/css/selectors/pseudo_state.h"

 // lexbor/css/at_rule/ref.h
 #include "lexbor/core/shs.h"
 #include "lexbor/css/at_rule/const.h"
+#include "lexbor/css/at_rule/state.h"

 // lexbor/css/property/ref.h
 #include "lexbor/core/shs.h"
 #include "lexbor/css/property/const.h"
+#include "lexbor/css/property/state.h"

If you want I can transfer the Github repo's ownership to you. You can just message me your Github username.

jeudesprits · November 30, 2023, 4:44am

Thank you so much for the work you've done!

jeudesprits · November 30, 2023, 6:45am

There was a silly mistake with the pointer dereferencing... Everything worked! The only thing I didn't quite understand is the difference between publicHeadersPath and cSettings: [ .headerSearchPath("...") ]?

And whether it is necessary to specify in exclude everything that lies next to source, other folders, scripts and so on. The documentation says that this path is relative to target, not sources

Lancelotbronner · November 30, 2023, 1:17pm

The difference between public headers path and header search path is subtle in name but makes a big difference.

Public headers path is where the API headers go, they'll be the declarations available in Swift. If imported into another C/C++ target it will be the structure to use in includes. They're also compiled as individual modules rather than textually, which is the reason behind my patch.

The headers I patched were always included after the header containing their declarations so they always had them. With modules enabled, the headers themselves are compilation units and needed to include the missing declarations.

Header search paths is, well, the paths used to resolve includes, the public header path implicitly adds itself as a header search path. They only apply to their own target so packages depending on this one won't have access to them.

Lancelotbronner · November 30, 2023, 1:31pm

As for sources and excludes, I haven't specified a path for the target so it is implicitly Sources/lexbor (based on its name). Any path within the target will be relative to that. You can change the target's path using the path parameter, be aware that targets cannot overlap.

The sources and exclude parameters are filters and are applied in that order. I'm specifying lexbor's source directory in sources (resolves to Sources/lexbor/source) which will automatically exclude everything else, that's why I don't have to explicitly exclude everything else.

jeudesprits · November 30, 2023, 7:33pm

Everything is crystal clear now. Thank you so much!

I was thinking, in general, those patched headers are not public. If you don't add them to
publicHeadersPath, they won't be a compilation unit and everything will work without a patch.