URLSession download of a large file took ages

Using Swift
,
1

Hi,

can someone tell why downloading the same file in Safari is fast (8sec), but from within my script it takes ages (40min)?

    private func download() async -> Data? {
        do {
            let configuration = URLSessionConfiguration.default
            configuration.timeoutIntervalForRequest = .infinity
            configuration.requestCachePolicy = .returnCacheDataElseLoad
            
            let session = URLSession(configuration: configuration)
            
             let (asyncBytes, urlResponse) = try await session.bytes(from: resourcesURL)
             let length = (urlResponse.expectedContentLength)
             let gb_total = Double(length) / 1024 / 1024 / 1024
            
             var data = Data()
             data.reserveCapacity(Int(length))
            
            for try await byte in asyncBytes {
                 data.append(byte)
                 
                 let progress = Double(data.count) / Double(length)
                 
                 let gb = Double(data.count) / 1024 / 1024 / 1024
                 let formatted_gb_total = String(format: "%.3f", gb_total)
                 let formatted_gg_loaded = String(format: "%.3f", gb)
                 
                 
                 taskLabel.stringValue = "Downloading (\(formatted_gg_loaded) of \(formatted_gb_total) GB)\(getDot(for: data.count))"
                 progressBar.doubleValue = progress
             }
             return data
         } catch {
             return nil
         }
    }
2
  • How big is the file in question?
  • I'd first check that expectedContentLength returns a sensible result.
  • Then I'd try to comment out label/progress bar updating to see how that affects the timing.
  • Then check the timing with formatting removed as well.
  • Then with data.append commented out.
  • Then checking the timing if you switch from "bytes" to "data" (to grab the whole data at once).
  • Then switching from "async" versions of those to a callback.

I guess one of the steps above will reveal the culprit and you'd know where to go from there.

1 Like
3

You’re streaming byte by byte. Unless the optimizer likes you this can very slow. Better to use URLSession’s actual download method.

2 Likes
4

yap, i undeerstand and see it. Thanks a lot. Download-task and delegates are even more the code that i like. ;)

5

Yes... but 300x slowdown?! If I read a file byte by byte with fread it could be, say, 3x time slower overall than reading reading it with fread by 100K blocks, 300x would be quite unexpected; I expect the same with URLSession.

That's a very good advice. Note that your app doesn't even have to be running for the download to happen – once the download is finished your app will be relaunched if needed.

6

Ok, thanks for your help.
I have to admit, that I cant get URLSession working with delegates correctly. The Task did not start. So I switch over to what I know: AsyncHTTPClient works as it should.

Hm, so URLSession did not work correctly on Linux, and on MacOS it's hard to get it working correctly. :thinking: (at least for me). So I think I stick to AsyncHTTPClient, is there is any downside?

7

You don't need the "delegate" version specifically... I suggested to try with a callback version to see if there's a significant speed difference.

The typical gotcha is forgetting calling resume():

URLSession.shared.dataTask(with: request) { data, response, error in
    // data or error here
}.resume() // !

similar for downloadTask, you are just getting the temporary file URL in the callback instead of data.

The thing I was talking above was a combination of using downloadTask and "background" session:

URLSession(configuration: .background(withIdentifier: "myBackgroundSession"))

I can't comment on AsyncHTTPClient or using URLSession on linux as I never tried those.

1 Like
8

I assume you need the file at the end of the process, not necessarily Data in memory? If so use downloadTask, this will save you of extra work of writing the accumulated data to a file. Here's downloadTask example with progress tracking:

class Test: NSObject, URLSessionDownloadDelegate {
    func download(_ url: URL) {
        let session = URLSession(configuration: .default, delegate: self, delegateQueue: .main)
        session.downloadTask(with: URLRequest(url: url)).resume()
    }
    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
        print("finihsed \(location)")
    }
    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) {
        print("\(totalBytesWritten/1024/1024)MB of \(totalBytesExpectedToWrite/1024/1024)MB")
    }
}

Note that for some URL resources totalBytesExpectedToWrite (taken from response's expectedContentLength would be -1, be ready to handle that case correctly (e.g. with an indefinite progress).

9

Never assume! This test shows 7600x slowdown in one of the cases!

10

It's allocating three Strings for every byte in the download, I'm surprised it's just 300x slower

4 Likes