Jextract-swift high-level design discussion

Development Java interoperability
5

Hi everyone,
this is a small update on the design of the tooling around exposing Swift libraries to java.

We previously relied on mangled names that we experimentally were emitting in swift interface files. This has hit a number of limitations (primarily because we cannot express Swift's calling convention in Java's new Foreign Function APIs), and instead we pursued a direction where we don't need these.

The new design: Gradle and SwiftPM collaboration

We've added a new example here: swift-java/Samples/SwiftKitSampleApp at main · swiftlang/swift-java · GitHub which showcases how this will work eventually, although there is still too much boilerplate which we'll hide in plugins eventually.

The new design approach from an user's perspective changes to rely more on a closer gradle (we can support other build tools eventually, for now we picked gradle because it seems most folks are using that), and swiftpm collaboration.

In short, a mixed language project that wants to expose Swift code to Java easily, would do the following:

  • have a Package.swift for the Swift project

And ensure the products we want to use from java are dynamic libraries:

  products: [
    .library(
      name: "MySwiftLibrary",
      type: .dynamic,
      targets: ["MySwiftLibrary"]
    ),

Targets that we want to expose to Java, would depend on the SwiftJava SwiftPM plugin. I believe we'll offer both a command and pre-build plugin here. (The example in the repository currently drives the sample using a command plugin.):


    .target(
      name: "MySwiftLibrary",
      dependencies: [
        .product(name: "SwiftKitSwift", package: "swift-java"),
      ],
      exclude: [
        "swift-java.config",
      ],
      swiftSettings: ...
      plugins: [
        .plugin(name: "JExtractSwiftPlugin", package: "swift-java"),
      ]

TODO: We're going to converge plugins and call them SwiftJavaPlugin most likely, but that's still work in progress.

This plugin is responsible for generating Swift and Java sources necessary for the bridging.

On the Java project side, the Gradle build is basically informed how to generate sources, and its compileJava depends on the new compileSwift task.

In essence, this means for multi language projects using Java, the way to develop would be "call the gradle build" as it would also invoke the Swift build when necessary.

Then, we just ./gradlew run and all those steps are triggered in order:

  • we build the swift plugins
  • we build the targets which have swift-java.config files
  • we invoke the swift package jextract <targets> --options with a number of options that the gradle build determined
  • this generates both swift thunks with "well known names" that we'll invoke from Java using FFM, as well as Java "wrappers" that target those thunks.
  • optionally: we rebuild the generated swift sources to make sure things are correct and build fine
  • now all dependency tasks of the compileJava task have completed and the normal Gradle "java" build builds the java sources
  • the run task also has automatically configured java.library.path dependent on the targets that we depend on.
  • the "java app" with swift dynamic library dependencies just works™

Such app's main looks like this:

package com.example.swift;

// Import swift-extract generated sources
import com.example.swift.MySwiftLibrary;
import com.example.swift.MySwiftClass;

// Import support libraries
import org.swift.swiftkit.*;

import java.util.Arrays;

public class HelloJava2Swift {

    public static void main(String[] args) {
        boolean traceDowncalls = Boolean.getBoolean("jextract.trace.downcalls");
        System.out.println("Property: jextract.trace.downcalls = " + traceDowncalls);

        System.out.print("Property: java.library.path = " +SwiftKit.getJavaLibraryPath());

        examples();
    }

    static void examples() {
        MySwiftLibrary.helloWorld();

        MySwiftLibrary.globalTakeInt(1337);

        // Example of using an arena; MyClass.deinit is run at end of scope
        try (var arena = SwiftArena.ofConfined()) {
            MySwiftClass obj = new MySwiftClass(arena, 2222, 7777);

            // just checking retains/releases work
            SwiftKit.retain(obj.$memorySegment());
            SwiftKit.release(obj.$memorySegment());

            obj.voidMethod();
            obj.takeIntMethod(42);
        }

        System.out.println("DONE.");
    }
}

You'll notice that the generated sources ended up in com.example.swift:

  • there is one type for the static and global Swift values and functions, with the same module as the target that we built
  • and wrapper types for every public type

Those wrapper types then expose "looks like normal methods" methods on those wrappers, and perform native calls.

Solution to Object Lifetimes: Swift Arenas

You'll also notice the previous snippet contained a SwiftArena type.

This actually is not implementing the JDK's Arena type, but it serves a very similar purpose: management of natively allocated memory / objects.

We leave the allocation of objects up to Swift so far, so for a class the allocating initializer is invoked.

The arena is a concept that manages the lifetime of the object, and this example has a Confined arena:

try (var arena = SwiftArena.ofConfined()) {
    MySwiftClass obj = new MySwiftClass(arena, 2222, 7777);
}

what this does, is that at the end of this try-with-resources scope the Java side asserts that "i should be able to destroy this object now", checks the reference count is just 1 and invokes the right destroy through the object's witness table. This is native integration into Swift by modeling and calling through Swift's witness tables (!).

This is a very good arena type for short lived objects when we know we will be destroying them at the end of a scope. It has predictable performance and it avoids leaking objects by accident.

We are also offering an AutoArena type (names here follow established names from the JDK), which relies on the GC to track these objects and can be used like this:

// java
final var arena = SwiftArena.ofConfined();
MySwiftClass obj = new MySwiftClass(arena, 2222, 7777);
obj = null; // obj no longer referenced!
// when it is GC-ed, we'll swift_release the underlying object

Because the wrapper MySwiftClass around the native instance of this class is no longer referenced in the Java runtime, as soon as the GC gets to checking it, we will use phantom references and queues to decrement the reference count of this object.

Note: This technique has performance implications on the JDK, as phantom references are costly to maintain for the GC, but it does offer a simple way of using Swift objects, especially long lived ones.

It also is not predictable anymore when we'll release the resource. So in general confined arenas are more predictable and should be preferred when possible.

Next steps for jextract

We'd like to converge the various plugins swift-java currently has accumulated. We currently have Java2Swift and jextract-swift which work in opposite "directions". Instead, we should offer a swift-java tool and plugins which just do the right thing depending on what sources and configuration they are used on.

We need to further develop the jextract mechanism to handle: enums, properties, throwing functions, and more.

We will provide a way to make a jar that includes the .dylib/.so file as well as the generated java sources. This way you will be able to publish your my-swift-library.jar and make it very easy to consume for java only developers which don't even have to know you wrote some of the library in swift.

If any of this sounds exciting help is most welcome. There's also much work on the JNI (JavaKit) side of the library, so don't be shy to join in. :slight_smile:

Future Gradle plugin work

We will eventually need to write a gradle plugin that encapsulates a lot of the logic that today is just done in the one example file: swift-java/Samples/SwiftKitSampleApp/build.gradle at main · swiftlang/swift-java · GitHub

The good news is that this is very possible, and eventually we'll just be able to add a plugin and get this "just works" workflow going.

If someone is more familiar with Gradle, help on this front would be more than welcome.

Hope you've enjoyed this update and we'll keep in touch about ongoing developments :slight_smile:

7 Likes