Read text file line by line

Funnily enough I needed to do this last week. I first looked at NSFileHandle but had issues with it, I think there is a memory leak somewhere but I've not had time to investigate too deeply. But using it caused any data read to never be released from memory which isn't ideal when reading in a 3.3GB file.

I ended up using the excellent work by the SwiftNIO developers and their version of FileHandle. It's not reading a file line by line but chunk by chunk which you can then subsequently split each chunk down by line. Try looking at the NonBlockingFileIO object, you can then read in the chunked data and get given a ByteBuffer object which you can then in turn use to parse out primitive types or raw bytes. It was probably never intended to read in multiple GB size files (I have a 70GB file I need to test it on soon), but it works great for what I'm doing. A basic macOS app can read through the entire file and never go above 30mb of memory usage if you are just processing the data, but obviously if you start caching then your memory usage will increase. One thing to note is that you will find a HUGE speed difference between running in debug mode compared to releases mode.

@lukasa I was going to message you separately about this, but since this thread came up I don't suppose there is any desire to split out the file IO work from SwiftNIO so it can be used independently of SwiftNIO? It's really nice to use for low level file reading work, but as you can see in my use case I don't really need all of the rest of SwiftNIO.

5 Likes