[Demo Code of Vapor v4+] Image processing: upload, store download

Related Projects Vapor
1

I'm a professional Swift coder, Vapor Neophyte.
I'm trying to learn how to upload & download an arbitrary image.
I'm using Vapor v4.77 (Concurrency).
I need complete demo code to piece the parts into a cohesive whole to understand.
Reference, info is greatly appreciated.
Thanks!

2

For downloading an image you can use Vapor's stream API

You have a number of options for uploading, but the efficient way is to stream it to a file as that doesn't load the whole thing into memory. There's an example in Vapor:

1 Like
3

I have piecemeal code snippets, most are dated (pre-Vapor v4.7).
Most of the tutorials merely cover introductory text like ‘Hello World.’

I want to get an in-depth grasp of working with binaries (images) and forms between Vapor and the UI.

Thanks for the lead.
I’ll study it.

~ Ric.

1 Like
4

Hi @0xTim,

Thanks for the code pointer.

Quick question about that code vs a comment a bit above:
in lines 226 and 234 it's returning req.eventLoop.makeSucceededFuture(()) whereas lines 39-41 say that "returning pre-completed future would balloon in memory."

Could you please help me understand if I should just return a "futureResult" or "success"?

Thanks

5

That's possible related to the artificial slowing of the stream to demonstrate back pressure.

Those routes need updating as I highly recommend you migrate to async/await and NIOFileSystem - you don't have to deal with futures at all and the code is much cleaner

6

Yes, and I LOVE how the code is now much much simpler with async/await.

However, I couldn't find anywhere an example of how to use the req.body.drain with async/await. Basically, I want to have an upload endpoint using streams and if you have an example that you could share - it would be much appreciated :smile:

7

The request body now conforms to AsyncSequence so you should be able to do

for try away part in req.body {
  // ...
}

See vapor/Tests/VaporTests/AsyncRequestTests.swift at d9fa0d3f7831ae055077fbcc166eb8baa94a19d0 · vapor/vapor · GitHub as an example

1 Like
8

Thank you for this!

Do you happen to have a snippet where it shows how to write the buffer to the file as a stream?

I tried writing to file like this, but it was collecting the entire buffer into the memory and wasn't cleaning up after the write was completed:
FileManager.default.createFile(atPath: path, contents: Data(buffer: receivedBuffer))

I also tried using a handle which was slightly (~0.5MB) increasing in memory, but sending the second request before the first one completes is crashing the app:

let fileHandle = FileHandle(forWritingAtPath: path)
for try await part in req.body {
    fileHandle.seekToEndOfFile()
    fileHandle.write(Data(buffer: part))
}
fileHandle.closeFile()

The [file I'm trying](https://commons.wikimedia.org/ https://upload.wikimedia.org/wikipedia/commons/2/2a/"A_present_for_the_mayor."_Washington%2C_D.C.%2C_Nov._17._Mayor_Joseph_K._Carson_of_Portland%2C_Ore._today_presented_a_chair_to_Mayor_Fiorello_La_Guardia%2C_of_New_York_City._Mayor_Carson_presented_LCCN2016872616.tif) ~160MB

I don't really need this large, but it's easier to see the spikes with large files

9

What's the crash you're seeing? Because it definitely shouldn't be crashing

10

Sorry for the late reply @0xTim - I tried to create a sample project to simulate the crash, but for some reason the sample app works fine with almost identical code:

Sample app that doesn't crash:

func routes(_ app: Application) throws {
    app.on(.POST, "stream", body: .stream, use: stream)

    @Sendable
    func stream(req: Request) async throws -> HTTPStatus {
        let path = app.directory.publicDirectory + UUID().uuidString
        FileManager.default.createFile(atPath: path, contents: nil)
        let fileHandle = FileHandle(forWritingAtPath: path)!
        for try await part in req.body {
            fileHandle.seekToEndOfFile()
            fileHandle.write(Data(buffer: part))
        }
        fileHandle.closeFile()
        return .ok
    }
}

Reproducible crash:

public func configure(_ app: Application) async throws {
    // ...

    app.on(.POST, "stream", body: .stream, use: stream)

    @Sendable
    func stream(req: Request) async throws -> HTTPStatus {
        let path = req.application.directory.publicDirectory + UUID().uuidString
        FileManager.default.createFile(atPath: path, contents: nil)
        let fileHandle = FileHandle(forWritingAtPath: path)!
        for try await part in req.body {
            fileHandle.seekToEndOfFile()
            fileHandle.write(Data(buffer: part))
        }
        fileHandle.closeFile()
        return .ok
    }
}

The app consistently crashes here:

crash
crash1619×1277 435 KB

11

Are you sending the same request to both servers? And do you have any other middleware running?

The main difference I can see is that the one that crashes does so when the route is defined in configure(_:) - is that correct?

12

Was there ever a follow-up on this ?

I believe I'm experiencing the same issue.

I did create the smallest possible test project, only consuming a streamed body, see GitHub - nelcea/Test_FileUpload: Example showing a crash using Vapor for file upload over HTTP
I can reproduce the issue mostly consistently (sometimes need to perform the query multiple times but it eventually crashes).

And the route is defined in a controller, so opposite from what I can see above.

Thanks

13

Just to chime back in on this - we have a fix for the crash here Fix `.noSignalReceived` body streaming crash by ptoffy · Pull Request #3232 · vapor/vapor · GitHub

We just need to get some extra eyes from the NIO team on it to make sure we're not masking a more serious problem, but feel free to try it out and let us know if it does or doesn't fix the issue!