Data.withUnsafeBytes and type annotations

Using Swift
,
1

Hi everyone,

I'm searching through multi-gigabyte files for newlines, starting with

let raw = try Data(contentsOf: url, options: .alwaysMapped)

The following implementation compiles and works, but seems slow.

raw.withUnsafeBytes { (buf: UnsafeRawBufferPointer) in
  if let ba: UnsafeRawPointer = buf.baseAddress {
    let ptr = ba.bindMemory(to: UInt8.self, capacity: raw.count)
    for i in 0..<raw.count {
      if ptr[i] == 0x0a {newlines.append(i)}
    }
  }
}

I want to try memchr but the following doesn't compile, with Type of expression is ambiguous without a type annotation on the .withUnsafeBytes:

raw.withUnsafeBytes { (buf: UnsafeRawBufferPointer) in
  if let ba: UnsafeRawPointer = buf.baseAddress {
    let ptr = ba.bindMemory(to: UInt8.self, capacity: raw.count)
    var offset: Int = 0
    while true {
      guard offset < raw.count else { return }
      let start = ptr + offset
      let found = memchr(start, 0x0a, raw.count - offset)
      guard let found = found else { return }
      guard Int(bitPattern: found) > 0 else { return }
      offset += found - start
      newlines.append(offset)
      offset += 1
    }
  }
}

I tried several variations on a theme of type annotation but couldn't satisfy the compiler.

What am I doing wrong?

2

This is a frequent issue with type checking errors in long closures, and the solution is usually to break them up.

In this case, the problem seems to be that memchr returns a mutable pointer. In the following I moved the distance calculation to a helper function that takes care of type-matching between found and start.

import Glibc

let raw: [UInt8] = []

let locations: [Int] = raw.withUnsafeBytes { b -> [Int] in
  b.withMemoryRebound(to: UInt8.self) { bytes -> [Int] in
    var newlines = [Int]()
    var offset = 0
    while offset < bytes.count {
      let start = bytes.baseAddress! + offset
      let found = memchr(start, 0xa, bytes.count - offset)
      guard let found, Int(bitPattern: found) > 0 else { break }
      offset += distance(from: start, to: found)
      newlines.append(offset)
      offset += 1
    }
    return newlines
  }
}

func distance(
    from start: UnsafeRawPointer, to end: UnsafeRawPointer
) -> Int {
    start.distance(to: end)
}
1 Like
3

@glessard covered handling memchr directly but an alternative: since Data is already a Collection of UInt8, you should be able to use Collection.indices(of:) to get effectively this exact result:

let raw = try Data(contentsOf: url, options: .alwaysMapped)
let newlines = raw.indices(of: 0x0A)

May be worth comparing the performance.

2 Likes
4

I reduced the problem and filed an issue: Bad error message after type-checking error · Issue #92306 · swiftlang/swift · GitHub

5

You folks are amazing thank you. (I was about to throw a tantrum and just do it in C, but this kept me here!)