Alamofire on Linux: Possible but not release ready!

millenomi · 2020-03-15T18:21:06.268Z

We are testing FoundationNetworking with XCTest right now without specific workarounds, so it may need more investigation on your part for your specific threading setup. Certainly we want to fix this if it causes a behavior difference on Linux vs Darwin.

Jon_Shier · 2020-03-15T18:21:59.866Z

millenomi:

Note that you’re doing a disservice to command-line (unbundled) Darwin processes as well. What’s the behavior there? Do you require an __infoplist… segment for them?

Until recently it wasn't easy to use Alamofire with command line projects anyway, so it's not something we've ever tested (not being able to easily run tests against them is another issue). What would you suggest instead?

Jon_Shier · 2020-03-15T18:25:40.670Z

SlaunchaMan:

A more serious issue for this port is that when run in XCTest, we never see the callbacks for requests. I can compile a simple project using Alamofire that runs fine on macOS and Linux and fetches a simple JSON payload from the network, but the tests never seem to get any data. I don’t know if this is XCTest or something else about the Linux implementation. Maybe a runloop or threading issue? A lot of the default callbacks use the main queue, so it could be something to do with how XCTest waits for expectations, too.

It looks like the request tests are running and many passing, there are just failures where tests expect certain events. Some behavioral and equality differences in certain tests too.

millenomi · 2020-03-15T18:26:05.941Z

You can get the process name from ProcessInfo, and have an API (which you probably have somewhere) for setting version information explicitly. Both Darwin and Linux command-line processes can have an Info.plist added to them if needed (the former via linker settings to embed it into a section, the latter via freestanding bundles), but it’s not the default.

Jon_Shier · 2020-03-15T18:28:28.468Z

millenomi:

You can get the process name from ProcessInfo, and have an API (which you probably have somewhere) for setting version information explicitly. Both Darwin and Linux command-line processes can have an Info.plist added to them if needed (the former via linker settings to embed it into a section, the latter via freestanding bundles), but it’s not the default.

Sounds like an additional level of fallback would work well there.

In general, there are changes in this PR that can be played separately, like adding a mutex lock, rearranging some methods to be easier to work around, and things like the IANA parsing and this bundle workaround.

millenomi · 2020-03-15T18:28:52.747Z

Jon_Shier:

It looks like the request tests are running and many passing, there are just failures where tests expect certain events. Some behavioral and equality differences in certain tests too.

Writing down the differences you’re seeing as issues would help tremendously in figuring out how to approach this!

Jon_Shier · 2020-03-15T18:37:13.715Z

You can see the results of the latest test run here. From the top it looks like:

Tests using URLCredential fail.
Some tests for URLCache behavior fail due to timestamp and other issues.
Tests around download resumeData fail.
Tests around file operations fail.
Tests of number handling in our URLEncodedFormEncoder fail, likely due to the NSNumber issues pointed out earlier.
Various lifetime event failures due to missing metrics callbacks which haven't be avoided, possibly other changes to behavior.
Tests of URLSession's behavior when using invalidateAndCancel fail.
Somehow there are failures on the number of times adapters and retriers are called, which is weird.
Eventually the tests crash.

Investigating those failures will need a full Linux instance, so it may be a while.

millenomi · 2020-03-15T18:38:20.412Z

The link and the list already help a lot, thank you!

Jon_Shier · 2020-03-15T23:05:56.180Z

@millenomi If I could get a little more advice around the User-Agent I'd appreciate it. Here's the current code:

/// Example: `iOS Example/1.0 (org.alamofire.iOS-Example; build:1; iOS 13.0.0) Alamofire/5.0.0`
public static let defaultUserAgent: HTTPHeader = {
    let userAgent: String = {
        if let info = Bundle.main.infoDictionary {
            let executable = info[kCFBundleExecutableKey as String] as? String ?? "Unknown"
            let bundle = info[kCFBundleIdentifierKey as String] as? String ?? "Unknown"
            let appVersion = info["CFBundleShortVersionString"] as? String ?? "Unknown"
            let appBuild = info[kCFBundleVersionKey as String] as? String ?? "Unknown"

            let osNameVersion: String = {
                let version = ProcessInfo.processInfo.operatingSystemVersion
                let versionString = "\(version.majorVersion).\(version.minorVersion).\(version.patchVersion)"
                // swiftformat:disable indent
                let osName: String = {
                #if os(iOS)
                    return "iOS"
                #elseif os(watchOS)
                    return "watchOS"
                #elseif os(tvOS)
                    return "tvOS"
                #elseif os(macOS)
                    return "macOS"
                #elseif os(Linux)
                    return "Linux"
                #elseif os(Windows)
                    return "Windows"
                #else
                    return "Unknown"
                #endif
                }()
                // swiftformat:enable indent

                return "\(osName) \(versionString)"
            }()

            let alamofireVersion = "Alamofire/\(version)"

            return "\(executable)/\(appVersion) (\(bundle); build:\(appBuild); \(osNameVersion)) \(alamofireVersion)"
        }

        return "Alamofire"
    }()

    return .userAgent(userAgent)
}()

I'm assuming that if there's no Info.plist at all that Bundle.main.infoDictionary will be nil. I also know that I can use ProcessInfo to get the executable name through arguments.first, but there doesn't seem to be any additional information available, like integrating app version (Alamofire has its own version info) or build numbers.

Jon_Shier · 2020-03-15T23:28:25.307Z

One other bit of missing functionality: MIME type parsing. Alamofire tries to infer the type of a file used in multipart form data uploads to properly represent it in the form boundaries using API from CoreServices. Obviously that won't be available but is there any sort of equivalent API for:

if
    let id = UTTypeCreatePreferredIdentifierForTag(kUTTagClassFilenameExtension, pathExtension as CFString, nil)?.takeRetainedValue(),
    let contentType = UTTypeCopyPreferredTagWithClass(id, kUTTagClassMIMEType)?.takeRetainedValue() {
    return contentType as String
}

Jon_Shier · 2020-03-16T00:00:40.866Z

I've replaced the implementation of our isBool check with String(cString: objCType) == "c" which seems to work. That's present on Linux too, despite no Obj-C runtime?

I think it's okay if we go through NSNumber for everything, though there seems to be encoding differences in the Linux tests, so we may need to look at this again later.

Jon_Shier · 2020-03-16T02:14:59.153Z

I've put up a PR which boils down many of the foundational changes into a single set of changes. It should make it easier to ultimately support Linux while not affecting Alamofire now.

millenomi · 2020-03-16T04:52:56.266Z

Jon_Shier:

I've replaced the implementation of our isBool check with String(cString: objCType) == "c" which seems to work. That's present on Linux too, despite no Obj-C runtime?

Indeed. Linux NSValue simulates encodings for known types, though it cannot handle encoding C structs the way the Darwin version can. It works for numbers the way you’d expect, though!

I will take a look at the PR at some point this week. Thank you!

Jon_Shier · 2020-03-21T04:10:07.171Z

The test suite has completed for the first time! With the probable Progress bug out of the way just 57 failures to investigate!

SlaunchaMan · 2020-03-21T04:15:41.928Z

It looks like the current issue is some inconsistent behavior of the Progress type between macOS and Linux. I’ve filed this as SR-12391 so we can track it separately.

Jon_Shier · 2020-03-31T21:29:17.277Z

Another additional wrinkle: with the upcoming DataStreamRequest we now use Stream.getBoundStreams, which is unimplemented in swift-corelibs-foundation. Luckily it should be easy to workaround.

SlaunchaMan · 2020-04-01T21:03:51.122Z

I've conditionally compiled out the call to NSStream.getBoundStreams(withBufferSize:inputStream:outputStream:) for now, with an additional todo marked in the PR to see if that API can be made to work on Linux.

SlaunchaMan · 2020-04-27T01:14:13.303Z

Another issue as I go through the test failures: it looks like URLSession on Linux isn’t calling methods for URLAuthenticationChallenge. I’ve created SR-12676 to track it.

dnadoba · 2020-04-28T08:03:59.857Z

Apple has released a new security framework, called SwiftCrypto, with an open source implementation for Linux a while ago. It is also available on all Apple platforms as system framework but only with the newest OS version (iOS 13, macOS 10.15, etc.).
This could be a better alternative than OpenSSL on Linux.

Jon_Shier · 2020-04-28T18:20:46.776Z

Unfortunately SwiftCrypto isn't really suitable for the type of work Alamofire needs to do, like certificate pinning. Additionally, there isn't an alternate path for security challenges in URLSession, so there's not really a way to expose them on Linux unless Apple's willing to create Linux-only APIs, or a new API that can be used on all platforms. I'd prefer the second if it means being able to stop using the Security framework.