PythonLambda โ€“ adding Python Lambdas to Swift / PythonKit

Hi all, I am interested in using Swift for data science, and have of course started using PythonKit to interface with numpy, pandas etc.

A challenge that one quickly faces when using many Python APIs is their reliance on Python lambdas, and this can quickly become a showstopper.

To fix this, I've built a small library which works together with PythonKit to provide Python lambda functionality to Swift. Here's an example:

Python code: map(lambda(x:x*2), [10,12,14] )

Swift code: Python.map( PythonLambda {x in x*2} , [10,12,14] )

As a convenience, and to improve the look of the code, the special character ๐บ is provided as a typealias for PythonLambda. So the above can be written as:

Swift code: Python.map( ๐บ{x in x*2} , [10,12,14] )

A second example: sum all the rows in a Pandas dataframe:

let summer = ๐บ{(row:PythonObject) in Double(np.sum(row)) ?? 0}

let result = df.apply( summer, axis: 0 )

There are a few limitations to the implementation, in that only certain function shapes are supported (though generally sufficient to cover normal usage of lambdas in Python), see the documentation for details. However, if more complex lambdas are needed, it's possible to create a lambda using Python code directly in Swift:

    let doubler = PythonStringLambda(lambda: "x:x*2")

    // x*2 is interpreted as Python code. Obviously, care is needed!

    df.apply( doubler.pythonObject )

If you're interested, please see the github repo here:

https://github.com/jrsonline/PythonLambda

This relies on a forked version of PythonKit to add various features that PythonLambda uses. If people find PythonLambda useful, I will look to get those added to PythonKit.

https://github.com/jrsonline/PythonKit

To use in your own code, add dependencies to the below Swift packages:

GitHub - jrsonline/PythonLambda , branch master

GitHub - jrsonline/PythonKit: Swift framework to interact with Python. , branch py_c_tools

and add at the top of your code

import PythonKit

import PythonLambda

The library has only been tested on macOS (but doesn't rely on system-specific features, so should "in theory" work anywhere PythonKit is supported); and it only works on Python3 installations. It's "beta" at the moment, awaiting feedback from a larger audience!

I'd be very interested in thoughts and feedback.

2 Likes

Hi @strictlyswift

I am checking out your library for use with optuna + PythonKit. I'll let you know if we run into any issues. The initial comment I have is that it looks really nice! And it certainly works with a very simple example I threw at it.

After a brief look at its implementation I did have the impression it may be worth trying to reduce the API surface area to reduce complexity in the codebase overall, also to get the necessary bits merged into PythonKit. For example, it may be worth only allowing lambdas with PythonConvertible arguments rather than a host of (Int) -> Int etc. combinations.

edit: I do have one question off the bat: how do we raise a python exception from within the Swift lambda? I'm imagining something like try raisePythonException(optuna.TrialPruned()). If this functionality does not yet exist I might be able to put together a PR for it

Thanks! Let me know.

Many thanks
Jonathan

Not sure if you saw my question about exceptions (it was in an edit)- do you have an idea how to raise a Python exception from a swift PythonLambda?

Hi - no I missed the question. I'm afraid I haven't tried raising Python exceptions from Swift at all (curious as to what the use case is, actually)

Re: the API surface - I'll experiment with using PythonConvertible ; the reason I went for the strongly-typed initializers is actually down to the Python-C bridge requiring strong types in order to be able to "interpret" the lambda it gets passed. It may be possible to recreate this at runtime vs relying on the type checker to do this. Strong types also mean you can pass lambdas and get results back without having to cast back to specific types, which feels a bit more "swifty".

Lastly, I fixed a somewhat esoteric bug (__name__ not working on lambdas) and pushed back to github so if you get chance you might want to update the package version.

Re: the use case. We are optimising a machine learning classification task. For that we want to use the optuna Python package to find optimal input parameters for some of our decision logic.

optuna allows you to "prune" and bail out early (saving processing time) when testing parameter sets that promise to perform worse than other parameter sets it's already tested. We need to raise a specific python exception from our lambda in order to signal to optuna that it should abandon the run. (See https://optuna.readthedocs.io/en/stable/tutorial/pruning.html - our PythonLambda is the objective function in that example)

Fair enough about the typed lambdas, I'm not very deep into the topic of PythonKit yet in general. I had the impression that PythonKit keeps everything as either a PythonObject or a PythonConvertible to keep their APIs simple, but it's entirely possible that lambdas have different requirements.

Thanks for pushing the update! I will let you know if we get any further this week


edit: To avoid double-posting I'm writing here. It turns out the pruning functionality (and therefore the ability to throw Python exceptions from the Swift function) is quite essential for us.

I found it difficult to add this functionality to PythonLambda because of the amount of code repetition and because it uses various targets (including a C target), which I found quite difficult to get my head around and set up a development environment for. Instead, I opted to add the functionality directly to PythonKit, here: Allow defining Swift functions to be called from Python by ephemer ยท Pull Request #25 ยท pvieito/PythonKit ยท GitHub.

It doesn't do exactly what you set out to do with PythonLambda (in particular, it doesn't allow devs to define various Swift function signatures to be called from Python, e.g. (Int) -> String, etc.). Looking at it a bit closer though, that didn't seem like a good tradeoff to me because it seemed to be largely implementing what PythonKit already does (converting PythonObject to Float, String, Int, etc.).

Since the PythonFunction will always be called from Python, what you "actually" get will always be a PythonObject, so I don't think there's a performance benefit in allow varying function signatures. I do see the benefit in making the API look more Swifty, but I think the tradeoff of having much less (10x less) code is worth it to devs using PythonKit, who have to convert between Python and Swift types all the time anyway.

I haven't tried it myself, but creating the functions should also be thread-safe in my implementation. Just in case anyone needs this (is Python even thread-safe itself?).

Anyway, I hope we can get a good solution to calling Swift functions from Python soon. Maybe a third person will join this thread and come up with something even simpler :slight_smile: I would be happy to hear any feedback people have about the approach I took.

Hi, yes I am looking into doing the same. I also realized that having the return type of the lambda as the correct type doesnt give much benefit!

Will have an update shortly.

Again, not sure if you saw my edit here. Have created a PR on PythonKit directly with this functionality.