# Shared Substrings

**URL:** <https://forums.swift.org/t/shared-substrings/38547>\
**Category:** Pitches\
**Tags:** string\
**Created:** [July 18, 2020, 1:49pm UTC](https://forums.swift.org/t/shared-substrings/38547 "2020-07-18T13:49:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 18, 2020, 1:49pm UTC](https://forums.swift.org/t/shared-substrings/38547/1 "2020-07-18T13:49:26Z")

</div>

Hi! I'd like to get some feedback on a new feature for the standard library: shared Substrings.

It's based on @allevato's [prototype implementation](https://github.com/apple/swift/pull/23050), but with some significant changes.

## Introduction

Shared Substrings give us a way to interpret buffers of bytes as unicode text, without allocating new storage exclusively owned by a `String` object. It would, for example, allow developers to receive data from a file or network connection as an `Array<UInt8>` or Foundation `Data` object, and parse that data as text without copying it. Additionally, it gives developers of structured text objects (like URLs) greater control over how they organise their storage.

## Motivation

It is common for applications to receive and manipulate text as blocks of bytes. For example, when reading data from a file or over a network connection, applications will typically traffic in types such as `Data` or `ByteBuffer`, some or all of which may be interpretable as text. The Swift standard library includes several utilities and algorithms for interpreting unicode text - however, making use of these algorithms currently requires that the data is stored in a buffer created and managed by the `String` type.

Let's imagine a function which receives some payload data from a network connection as an Array and counts the number of characters it contains:

```swift
func countCharacters(in utf8Bytes: [UInt8]) -> Int { 
  let text = String(decoding: utf8Bytes, as: UTF8.self)
  return text.count
}

```

Unfortunately, this copies the entire payload in to new, owned storage - even though all we _really_ want is to apply String's grapheme-traversal algorithms to the bytes in the existing buffer.

Furthermore, many libraries which make heavy use of Strings may want finer control over how they organise their storage. Take URLs for instance: a type representing a parsed URL might look something like this:

```swift
struct MyURL {
  var urlString: String
  var schemeEndIndex: String.Index
  var usernameEndIndex: String.Index?
  var passwordEndIndex: String.Index?
  var hostnameEndIndex: String.Index?
  var portEndIndex: String.Index?
  var pathEndIndex: String.Index?
  var queryStringEndIndex: String.Index?
  var fragmentEndIndex: String.Index?
}

```

This is roughly what [Rust's URL type](https://github.com/servo/rust-url/blob/master/src/lib.rs#L154) looks like. But just looking at it: every `String.Index` is 8 bytes on x86\_64 (9 bytes for an optional), and `String` is 16 bytes - resulting in a minimum 87 bytes _plus_ whatever dynamic storage the `String` might own. That's a very heavy object.

Even if we made this object a class, we end up with 2 heap allocations: one for `MyURL` itself, containing the indexes and the `String`, and again for `String`'s storage. Maybe we would prefer to group these in to a single allocation, like `ManagedBuffer` does, and maybe we'd like more control over the size of the String, so that we can use a smaller index type. Shared Substrings allow developers to explore these kinds of designs.

## Proposed solution

This proposal would add the following API to the standard library, allowing developers to create `Substring`s from either: a pointer-owner pair, a `ManagedBuffer` instance, or an `ArraySlice`.

```swift
extension Substring {

  /// Creates an immutable `Substring` whose backing store is shared with the UTF-8 data
  /// referenced by the given buffer pointer.
  ///
  /// The `owner` argument should manage the lifetime of the shared buffer.
  /// The `Substring` instance created by this initializer retains `owner` so that deallocation
  /// may occur after the substring is no longer in use. The buffer _must not_ be
  /// mutated while there are any strings sharing it.
  ///
  /// This initializer does not try to repair ill-formed UTF-8 code unit
  /// sequences. If any are found, the result of the initializer is `nil`.
  ///
  /// - Parameters:
  /// - buffer: An `UnsafeBufferPointer` containing the UTF-8 bytes that
  /// should be shared with the created `Substring`.
  /// - owner: An optional object that owns the memory referenced by `buffer`.
  ///
  public init?(sharingStorage buffer: UnsafeBufferPointer<UInt8>, owner: AnyObject)

  /// Creates an immutable `Substring` whose UTF-8 backing store is shared with
  /// the given `ManagedBuffer` instance's elements.
  ///
  /// The `Substring` instance created by this initializer retains `owner` so that deallocation
  /// may occur after the substring is no longer in use. The buffer _must not_ be
  /// mutated while there are any strings sharing it.
  ///
  /// This initializer does not try to repair ill-formed UTF-8 code unit
  /// sequences. If any are found, the result of the initializer is `nil`.
  ///
  /// - Parameters:
  /// - buffer: A `ManagedBuffer` whose elements are UTF-8 bytes that
  /// should be shared with the created `Substring`.
  /// - range: The range of elements which should be included in the `Substring`.
  ///
  public init?<Header>(sharingElements buffer: ManagedBuffer<Header, UInt8>, range: Range<UnsafeBufferPointer<UInt8>>)

  /// Creates an immutable `Substring` whose backing store is shared with the UTF-8 data
  /// in the given region of an array.
  ///
  /// This initializer does not try to repair ill-formed UTF-8 code unit
  /// sequences. If any are found, the result of the initializer is `nil`.
  ///
  /// - Parameters:
  /// - array: An `ArraySlice` containing the UTF-8 bytes that
  /// should be shared with the created `Substring`.
  ///
  public init?(sharingStorage array: ArraySlice<UInt8>)
}

```

There are several interesting things about this API:

1. It allows creating a `Substring`, not a `String`. This aligns with [SE-0163](https://github.com/apple/swift-evolution/blob/master/proposals/0163-string-revision-1.md), which introduced `Substring` specifically to differentiate strings which own their backing storage from those which share a buffer and should not be stored long-term. A similar principle applies to shared strings. From SE-0163:

2. Sharing storage requires value semantics/copy-on-write. Even though we ask people to not store `Substrings` long-term, that doesn't mean they won't, and an object which exposes `Substring` views of its storage must be mindful that those references may indeed escape. The `owner` object lets other references determine if any such references exist, and whether or not it is safe to mutate or deallocate the buffer.

3. The returned `Substring`s are immutable. This means that standard library will not mutate the bytes in the referenced buffer, and any attempted in-place modifications (like calling `append`) will copy to String-owned storage.

4. `ManagedBuffer` and `Array` are special-cased. Even though we typically don't add convenience functions for low-level APIs, implementing these requires access to standard library internals. The special-casing is in terms of `ArraySlice` in order to reduce the number of entrypoints.

## Detailed design

With the above API, the character-counting example could be written as follows:

```swift
func countCharacters(in utf8Bytes: [UInt8]) -> Int { 
  let text = Substring(sharingStorage: utf8Bytes)
  return text?.count ?? 0
}

```

Here, the creation of the shared `Substring` allows us to avoid having to allocate new storage.

The URL example consisting of a String and lots of indexes could look something like:

```swift
struct MyURL {
  struct Header {
    var schemeEndOffset: UInt16
    // ...
  }
  var storage: ManagedBuffer<Header, UInt8> = ...
  
  var scheme: Substring {
    let schemeEndIndex = Int(storage.header.schemeEndOffset)
    return Substring(sharingStorage: storage, range: 0..<schemeEndIndex)!
  }
  // etc.
}

```

Here, a single allocation contains both the indexes, and the storage itself. Additionally, we have gained control over the index storage, by representing them as offsets in to the UTF8 bytes and limiting them to 2 bytes each. Using 0 to represent `nil`, this representation amounts to a 16 byte in-line prefix to the UTF8 data.

## Source compatibility

These APIs are additive.

## Effect on ABI stability

The functionality is already part of String's ABI since Swift 5.0.

## Effect on API resilience

This proposal does not include any language features which would affect API resilience.

## Alternatives considered

### Create `String`s rather than `Substring`s

Many languages distinguish between owned and borrowed strings: for example, Rust includes both `String` (owned) and `str` (borrowed) types, C++ includes `std::string` (owned) and `std::string_view` (borrowed). Indeed, distinguishing between owned and borrowed storage is precisely why Swift defines both `String` (owned) and `Substring` (borrowed) types in the first place. Therefore, this proposal considers that `Substring` is the "correct" place for this API to live.

### A closure-based API

The 2 most important things for users of this API to get right are the `owner` object's management of the buffer lifetime, and that all mutations first ensure that the buffer is not shared. Typically, Swift does this with closure-based APIs:

```swift
extension String {

  static func withStringView<Source, Result>(
    of: Source, perform: (String?) throws -> Result
  ) rethrows -> Result where Source: Sequence, Source.Element == UInt8 {
    // use withContiguousStorageIfAvailable, otherwise copy or return nil.
    // No need for 'owner' because the String should not escape.
  }
}

// Example:
func countCharacters(in utf8Bytes: [UInt8]) -> Int { 
  return String.withStringView(of: utf8Bytes) { string in
    return string?.count ?? 0
  }
}

```

Unfortunately, this would severely hurt the usability of some of the motivating use-cases. For example, it would be rather unfortunate if a URL type could only expose its `scheme` or `path` string inside a closure scope.

### Promote `Substring` to `String`

@allevato's [prototype](https://github.com/apple/swift/pull/23050) shows another API we could implement using shared storage: the ability to "promote" a `Substring` to a `String` for a limited scope without copying:

```swift
extension Substring {

  /// Calls the given closure, passing it a `String` whose contents are equal to
  /// the substring but which shares the substring's storage instead of copying
  /// it.
  ///
  /// The `String` value passed into the closure is only valid durings its
  /// execution.
  /// ...
  public func withSharedString<Result>(
       _ body: (String) throws -> Result
  ) rethrows -> Result
}

```

This proposal considers such an API to be a separate issue. It might be nice to have a way to break the split owned/borrowing type model in limited situations, but isn't necessary for the motivating use-cases in this proposal.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 18, 2020, 1:55pm UTC](https://forums.swift.org/t/shared-substrings/38547/2 "2020-07-18T13:55:19Z")

</div>

CC @Michael_Ilseman

One thing that might be nice to add is some way to bypass UTF8 verification. String would already be relying on me not to mutate the buffer under its feet: it's perfectly reasonable to ask that it also just trust me that my buffer contains only ASCII text.

Another nice thing would be if `String.Index` had methods which exposed UTF8 offsets, but that might better be left to a separate proposal.

For example, right now, String.Index has the following methods to convert to/from integer UTF16 offsets:

```swift
extension String.Index {
  // https://developer.apple.com/documentation/swift/string/index/3151848-init
  init<S>(utf16Offset offset: Int, in s: S) where S : StringProtocol

  // https://developer.apple.com/documentation/swift/string/index/3151849-utf16offset
  func utf16Offset<S>(in s: S) -> Int where S : StringProtocol
}

```

But there is strangely no equivalent for UTF8.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [July 18, 2020, 5:15pm UTC](https://forums.swift.org/t/shared-substrings/38547/3 "2020-07-18T17:15:09Z")

</div>

Thanks for driving this forward!

Unfortunately, I think using `Substring` instead of `String` for this API is the wrong choice for many situations—it's limiting to such a degree that the API would not be useful for many of its intended use cases.

The original use case that drove my implementation was writing bridging code for C/C++ APIs that returned `char` buffers or `std::string` objects (for which ownership transferred to the caller) that I wanted to get into a Swift `String` without a copy. The problem with using `Substring` as the currency type for this operation is simply that most Swift APIs that consume text operate on `String` values, store `String` values, etc. Therefore, the moment you want to do anything truly useful with the string, you'd have to convert it using `String(substring)`, and you've lost every performance benefit that the shared storage API was supposed to provide.

The issue is with the following statement:

> [@](#):
>
> It allows creating a `Substring` , not a `String` . This aligns with [SE-0163](https://github.com/apple/swift-evolution/blob/master/proposals/0163-string-revision-1.md), which introduced `Substring` specifically to differentiate strings which own their backing storage from those which share a buffer and should not be stored long-term. **A similar principle applies to shared strings.**

The bolded sentence is not always true. Consider the following function from Cmark (the C Markdown library):

```cpp
/** Render a 'node' tree as an HTML fragment. It is up to the user
 * to add an appropriate header and footer. It is the caller's
 * responsibility to free the returned buffer.
 */
CMARK_EXPORT
char *cmark_render_html(cmark_node *root, int options);

```

The C API has explicitly transferred ownership of the returned `char` buffer to the caller. "Strings which own their backing storage" doesn't have to mean that they initially **created** that backing storage, just that they manage it once they're given ownership of it. In this case, what is desired is an ability to create a `String` that takes ownership of an existing buffer to efficiently create one without a copy:

```swift
class FreeingOwner {
  private let buffer: UnsafeBufferPointer<UInt8>
  init(_ buffer: UnsafeBufferPointer<UInt8>) { self.buffer = buffer }
  deinit { free(buffer.baseAddress) }
}

let pointer = cmark_render_html(treeRoot, 0)
let buffer = UnsafeBufferPointer(base: pointer, count: strlen(pointer))
let htmlString = String(sharingStorage: buffer, owner: FreeingOwner(buffer))

```

If I'm forced to create a `Substring` here, then I'm much more limited in what I can do with that value unless I create a copy or unless all the APIs I want to use support `Substring` or are generic over `StringProtocol`, which is extremely unlikely.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 18, 2020, 5:21pm UTC](https://forums.swift.org/t/shared-substrings/38547/4 "2020-07-18T17:21:07Z")

</div>

[SE-0248](https://github.com/apple/swift-evolution/blob/master/proposals/0248-string-gaps-missing-apis.md) introduced `Substring.base`, so you could still get a `String` type if you’re sure there’s no cost to storing the result long-term (like if it represents or takes ownership of the entire buffer).

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [July 18, 2020, 5:35pm UTC](https://forums.swift.org/t/shared-substrings/38547/5 "2020-07-18T17:35:00Z")

</div>

That would be a misuse of that API, though, and going through such hoops would unnecessarily obfuscate the code at the call site. The problem being solved is "I have an existing buffer _that I own_ and I want to create a `String` and transfer ownership to it". I shouldn't have to go through `Substring` for that. Instead of giving people an escape hatch to get a `String`, let's just give them a clear API.

> [@Karl](#):
>
> Many languages distinguish between owned and borrowed strings: for example, Rust includes both `String` (owned) and `str` (borrowed) types, C++ includes `std::string` (owned) and `std::string_view` (borrowed). Indeed, distinguishing between owned and borrowed storage is precisely why Swift defines both `String` (owned) and `Substring` (borrowed) types in the first place. Therefore, this proposal considers that `Substring` is the "correct" place for this API to live.

I can't speak for Rust's types, but the analogy between the proposed shared strings and C++'s `std::string_view` isn't correct. A `std::string_view` is nothing more than a pointer into an existing buffer and a length; there is no concept of ownership involved with `string_view` (so it's also unsafe). The equivalent in Swift would be a `Substring` initializer that takes a buffer pointer but no owner. But since the proposed initializers **do** take an owner object, we have the ability to transfer ownership completely to the object, which is what makes `String` a viable target type.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 18, 2020, 5:42pm UTC](https://forums.swift.org/t/shared-substrings/38547/6 "2020-07-18T17:42:14Z")

</div>

> [@allevato](#):
>
> That would be a misuse of that API, though, and going through such hoops would unnecessarily obfuscate the code at the call site. The problem being solved is "I have an existing buffer _that I own_ and I want to create a `String` and transfer ownership to it". I shouldn't have to go through `Substring` for that. Instead of giving people an escape hatch to get a `String` , let's just give them a clear API.

Perhaps - that is something I’d like to get feedback on. `String` is certainly convenient, but `Substring` specifically exists to represent text from a shared buffer. It’s up to the community if they care about strictly enforcing that model or just letting everything be `String`s because convenience is more important. They’re both viable options.

> [@allevato](#):
>
> the analogy between the proposed shared strings and C++'s `std::string_view` isn't correct. A `std::string_view` is nothing more than a pointer into an existing buffer and a length; there is no concept of ownership involved with `string_view` (so it's also unsafe). The equivalent in Swift would be a `Substring` initializer that takes a buffer pointer but no owner.

Perhaps it would be more correct to say that `string_view` is “unowned” rather than “borrowed“. It’s a similar concept, but of course it isn’t a direct equivalent.

---

<div class="post-metadata">

**Author:** ![woolsweater](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/woolsweater/32/2402_2.png) [@woolsweater](https://forums.swift.org/u/woolsweater)\
**Post date:** [July 18, 2020, 5:57pm UTC](https://forums.swift.org/t/shared-substrings/38547/7 "2020-07-18T17:57:18Z")

</div>

I like the direction here very much!

One point, perhaps minor, that springs to mind is the occasional mismatch in `String`'s API between signed/unsigned bytes. I think this interface may want to include entry points that use `CChar`/`Int8` to avoid friction talking to C APIs.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [July 18, 2020, 6:01pm UTC](https://forums.swift.org/t/shared-substrings/38547/8 "2020-07-18T18:01:57Z")

</div>

> [@Karl](#):
>
> `String` is certainly convenient, but `Substring` specifically exists to represent text from a shared buffer.

`Substring` exists to represent a slice of a `String` where the memory is owned by the base string—I don't think one should try to extrapolate semantics about other external memory buffers from that. The [documentation for `Substring`](https://developer.apple.com/documentation/swift/substring) says simply:

> # Substring
> 
> A slice of a string.

If you wished to implement the proposed APIs on `Substring`, then you would still need to create a `String` under the hood to act as the `base`, that `String` would be the owner of the buffer, and then your `Substring` is a "view" over the entire string. IMO, such an API would be a lie—it's returning something other than what it's actually creating, and it's returning something that is not what documentation of `Substring` says it is.

* * *

In my opinion, the crux of the issue is that Swift makes it extremely easy _at the compiler level_ to interop with C APIs; there is no other language I can think of that has such a clean approach (thanks to embedding Clang directly), because all you have to do is import the module.

When it comes to working with those APIs at the _runtime level_, however, string-based APIs are where C interop suffers a great deal more, and it's one of the biggest obstacles today to Swift being used as a replacement for C and C++ in situations where performance is critical. In order to eliminate those obstacles, not only should we remove the performance penalties, but the APIs we create to do so _must be as ergonomic as possible._

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 18, 2020, 6:55pm UTC](https://forums.swift.org/t/shared-substrings/38547/9 "2020-07-18T18:55:12Z")

</div>

The ergonomic difficulties that come from using Substring are not new. The responses in this thread summarise the situation well, and why it is the way it is: [Swift is still sometimes so cumbersome - #4 by Ben\_Cohen](https://forums.swift.org/t/swift-is-still-sometimes-so-cumbersome/24772/4)

But those responses do convince me that sharing should be expressed as a `Substring` by default. Maybe we could add a separate API on `String` for taking ownership of (not sharing) an external buffer?

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [July 18, 2020, 9:55pm UTC](https://forums.swift.org/t/shared-substrings/38547/10 "2020-07-18T21:55:43Z")

</div>

Ok, I think the link above provided some additional context that wasn't as clear before. Is your concern that the following would happen?

1. You have a `MyURL` type like the one above (slices into a managed buffer), but if the shared buffer initializers are on `String`, then the person implementing the type would also make the properties (like `scheme`) type `String`.
2. Someone accesses `myURL.scheme` and gets a `String` with the value `"http"`.
3. They no longer need the rest of the URL, so they let it go out of scope.
4. They think they're hanging on to this small 4-byte string, but what they're really retaining is the original buffer which 1MB worth of query parameters.
5. They would have realized this possible memory inefficiency if the type of the property had been `Substring`.

That's a reasonable concern, but it's a problem in this specific case because the initializer that takes a `ManagedBuffer` also takes a slice of the buffer. I think there's a different way to resolve this:

- An initializer on `String` takes only a `ManagedBuffer` and no range argument, and assumes the entire buffer is the string contents.
- An initializer on `String` takes an `Array` instead of an `ArraySlice` and uses the entire array.

Is that what you mean by "a separate API on `String` for taking ownership of (not sharing) an external buffer"? (I can't think of another way to distinguish owning vs. sharing in these situations, since both must retain the underlying buffer via an owner for the string or substring to be safe.)

Then, properties like those in `MyURL` would create a string that wraps the entire buffer and the get the relevant substring of that. This would require `String.Index` initializers with UTF-8 offsets, as you already pointed out, and I don't _think_ it should be less efficient.

I personally would prefer this because it maintains the property that a `Substring` is a slice of a `String`, as opposed to something that is created from "thin air".

Now, in total fairness, the above problem can't be avoided for the `UnsafeBufferPointer` initializer, because once you have one of those, you can move it wherever you want, and there's no way to guarantee that the buffer passed in is exactly the same as what is owned by `owner`. But such is the nature of unsafe APIs.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 19, 2020, 5:00pm UTC](https://forums.swift.org/t/shared-substrings/38547/12 "2020-07-19T17:00:30Z")

</div>

> [@allevato](#):
>
> - An initializer on `String` takes only a `ManagedBuffer` and no range argument, and assumes the entire buffer is the string contents.
> - An initializer on `String` takes an `Array` instead of an `ArraySlice` and uses the entire array.

Unfortunately, slicing shared Strings of the entire buffer has a couple of drawbacks:

- It means the entire buffer contents are checked as being valid UTF8, every time we get any component. We'd need an option to bypass that (e.g. URLs do percent-escaping and IDN transformations, so we can guarantee the contents are ASCII).
- There are still plenty of use-cases (e.g. buffers in text editors) when we can't give any meaningful "base" for the string to be sliced from, other than the entire string itself. So "base" doesn't always have much meaning - library authors need to document what (if anything) users can assume the slice is part of.

> [@allevato](#):
>
> Is that what you mean by "a separate API on `String` for taking ownership of (not sharing) an external buffer"? (I can't think of another way to distinguish owning vs. sharing in these situations, since both must retain the underlying buffer via an owner for the string or substring to be safe.)

Actually I kind of misspoke there, that's not what I mean. Here's hopefully a clearer explanation:

I think Substring is the "natural place" for sharing to live, as it already represents text data from a possibly larger allocation that shouldn't be stored long-term. Currently, that buffer only comes from a String, but this proposal would expand that. This is reinforced as I read posts like the one I linked to above, and by research I've done about other String models and how similar functionality is exposed there.

Take Rust, where `&str` represents a string slice and is actually a language-primitive type. It includes methods from `from_utf8` and `from_utf8_unchecked`, and from what I can tell, it is by far the most common string representation you will see in libraries. The owned string type, `String`, is a library type which basically consists of a vector that can be implicitly converted to a string slice (`&str`). Rust developers seem to [really like](https://news.ycombinator.com/item?id=16546494) their model, and besides syntax, there are direct parallels to Swift's `String` vs `Substring` model. It’s not exactly the same (Rust’s string slice doesn’t take an owner, but lifetime is enforced through borrow scopes). If we were to extrapolate from Rust, I think these methods would be on `Substring`.

So when you say:

> [@allevato](#):
>
> The problem with using `Substring` as the currency type for this operation is simply that most Swift APIs that consume text operate on `String` values, store `String` values, etc.

I wonder if that's a problem with this feature, or the ecosystem/language? Again, we have an almost-identical model to Rust, but their developers love this split and use their version of `Substring` for everything, while Swift developers (as evidenced in that post, and many like it) find it awkward. Maybe this calls for some broader thinking beyond shared Strings.

> [@allevato](#):
>
> ```swift
> class FreeingOwner {
> private let buffer: UnsafeBufferPointer<UInt8>
> init(_ buffer: UnsafeBufferPointer<UInt8>) { self.buffer = buffer }
> deinit { free(buffer.baseAddress) }
> }
> 
> let pointer = cmark_render_html(treeRoot, 0)
> let buffer = UnsafeBufferPointer(base: pointer, count: strlen(pointer))
> let htmlString = String(sharingStorage: buffer, owner: FreeingOwner(buffer))
> 
> ```

You're right that it makes sense to create a `String` here - that despite the very good reasons we have to distinguish `String` and `Substring` in the type system, there are cases when the sharing doesn't really add up to much/any additional memory use in practice, and we want to explicitly allow long-term storage.

Besides your example: for a URL, the entire-URL string consisting of the buffer's entire contents can safely be exposed as a `String` and stored long-term without worrying about sharing causing memory leaks. There's a small cost for the header, but the library author can decide if it's worth copying to remove that.

So maybe what we need is something like I mentioned in the last part of "alternatives considered": an API to promote `Substring`s in to `String`. Would something like this work?

```swift
extension String {

  /// Creates a String referencing the same storage as the given Substring.
  ///
  /// - Warning: Using this initializer erases the signal which the type `Substring`
  /// gives to developers, informing them that a piece of text is part of a shared
  /// and potentially large allocation. Users are advised to only use this
  /// when long-term storage of the result would not keep much more memory alive
  /// than is necessary.
  /// 
  public init(allowingLongTermStorageOf substring: Substring)
}

```

Allowing you to write:

```swift
let pointer = cmark_render_html(treeRoot, 0)
let buffer = UnsafeBufferPointer(base: pointer, count: strlen(pointer))
let htmlString = String(allowingLongTermStorageOf: 
  Substring(sharingStorage: buffer, owner: FreeingOwner(buffer))
)

```

It's a little bit involved, but I think it gives sharing the best API (using `Substring`), and is nice and explicit about what the lifting to `String` entails.

* * *

One more thing I've been thinking about is adding an API to get the owner _back out_ of the `String`. This could enable some really funky stuff, for example:

```swift
extension MyURL {
  var storage: ManagedBuffer<Header, UInt8>

  var urlString: String {
    let stringView = Substring(sharingStorage: storage, range: 0..<header.count)
    return String(allowingLongTermStorageOf: stringView)
  }
  
  init?(_ string: String) {
    // If this is a MyURL.urlString, reference the existing storage.
    if let existingStorage = string.owner as? ManagedBuffer<MyURL.Header, UInt8>,
      string.withUTF8({ $0.count == existingStorage.header.count }) 
    {
      self.storage = existingStorage
      return
    }
    // Ordinary String. Create new storage and parse.
  }
}

```

So basically we can cache arbitrary associated data in `String`s. There are lots of very cool things you can do with that, like caching Regex match information to speed up subsequent searches.

---

<div class="post-metadata">

**Author:** ![Jon\_Shier](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jon_shier/32/921_2.png) [@Jon\_Shier](https://forums.swift.org/u/Jon_Shier)\
**Post date:** [July 19, 2020, 5:22pm UTC](https://forums.swift.org/t/shared-substrings/38547/13 "2020-07-19T17:22:02Z")

</div>

> [@Karl](#):
>
> Again, we have an almost-identical model to Rust, but their developers love this split and use their version of `Substring` for everything, while Swift developers (as evidenced in that post, and many like it) find it awkward. Maybe this calls for some broader thinking beyond shared Strings.

I don't have the expertise for the larger conversation, but I do run into `String` / `Substring` issues quite a bit, so I wanted to speak briefly to this point. Perhaps the biggest issue with `Substring` is the lack of support within Swift and Apple's SDKs, as almost nothing uses `StringProtocol`. There isn't much Swift-native `String` functionality, and none of the imported Foundation APIs work with it. None of Apple's other APIs work with it either, so every time we have one it must be converted into a `String`.

I think if Swift were to make it easier to use `Substring`s where we currently use `String`s, were to move much of Foundation's `String` API onto `StringProtocol`, and were to provide some way to easily use API's unaware of `Substring`, we would see much higher use of the type.

---

<div class="post-metadata">

**Author:** ![allevato](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/allevato/32/385_2.png) [@allevato](https://forums.swift.org/u/allevato)\
**Post date:** [July 19, 2020, 9:25pm UTC](https://forums.swift.org/t/shared-substrings/38547/14 "2020-07-19T21:25:59Z")

</div>

> [@Karl](#):
>
> - It means the entire buffer contents are checked as being valid UTF8, every time we get any component. We'd need an option to bypass that (e.g. URLs do percent-escaping and IDN transformations, so we can guarantee the contents are ASCII).

Then we should add that option, which I think is a requirement for this API to be minimally viable anyway. At least in the case of the `UnsafeBufferPointer` initializer, we're **already** in unsafe land (i.e., `String` has to trust that the caller won't modify the memory underneath it because there's no CoW like the `ManagedBuffer` or `Array` cases), so having an option that says "trust me, the API that gave me this buffer already guaranteed it's valid UTF-8" doesn't make things any _less_ safe and is necessary for the performance that users need. Adding it to the `ManagedBuffer` and `Array` cases seems fine too.

> [@](#):
>
> - There are still plenty of use-cases (e.g. buffers in text editors) when we can't give any meaningful "base" for the string to be sliced from, other than the entire string itself. So "base" doesn't always have much meaning - library authors need to document what (if anything) users can assume the slice is part of.

I don't follow this argument; if the only meaningful base is the entire string itself, then just let that be the base. I'm not sure how to interpret "what (if anything) [...] the slice is part of" because _by definition_ a slice _has to be_ sliced from something. This goes back to my point providing an API that creates `Substring`s from "thin air"; if the base isn't obvious, we shouldn't provide APIs that gloss over that. Make the caller create the whole `String` and slice it themselves so it's crystal clear.

But these two bullet points make me think we should drill deeper into the `MyURL` example because they leave me unclear about what some of the invariants of these shared strings would be under your proposed implementation.

Above, `MyURL` wraps a buffer and provides efficient non-copying access to non-overlapping slices of that buffer. It's not mentioned in the snippet, but presumably there's also a way to access the entire URL as a string, and based on the representation, I imagine that would also be a non-copying shared string covering the whole buffer.

You mentioned concern about the "create a `String` and then slice it" approach incurring UTF-8 validation of the entire string, absent an option to skip it. But consider the following properties:

1. All strings in Swift are valid UTF-8. Therefore, all substrings are also valid UTF-8.
2. For any substring `s`, there is a base string `S` accessible using the `Substring.base` property.

Let's imagine the user accesses `MyURL.scheme` for a URL that is 1MB of data. What are you proposing the behavior to be here?

1. _The substring is the scheme and the base is the entire buffer, but only the scheme is UTF-8-validated or known to be valid UTF-8; validation is deferred for the entire buffer until the base is accessed (if at all)._

2. _The substring is the scheme and the base is also just that slice of the buffer, so only the requested portion needs to be UTF-8-validated or known to be valid UTF-8._

I claim that the **only** answer to this that can satisfy both the required relationships between `String` and `Substring` and provide a `Substring` that is semantically meaningful/correct to the caller is:

1. _The substring is the scheme and the base is the entire buffer, and the entire buffer has been UTF-8-validated or is known to be valid UTF-8._

Therefore, there is no advantage to providing a `Substring` initializer. The relationship and validation behavior is far clearer if the callee creates a `String` first and then slices it to get the `Substring`. All that's needed are the initializers to get indices from UTF-8 offsets.

So, to this point:

> [@Karl](#):
>
> I think Substring is the "natural place" for sharing to live, as it already represents text data from a possibly larger allocation that shouldn't be stored long-term. Currently, that buffer only comes from a String, but this proposal would expand that.

Since Swift `Substring` has the user-accessible `base` property that is a non-optional `String`, how would you expand that without creating a `String` anyway at some point (either internally, or as a consequence of the user accessing that property) or without making the API more complicated to support different kinds of bases?

I'm not sure whether the Rust analogies hold because I can't find a way to take a `&str` and get the base `String` from it, because one may not exist (e.g., if the `&str` is a literal). I'm by _no means_ a Rust expert so it's entirely possible I've missed something, but since we have the `base` property, we need to work within that constraint. If the claim is that Swift's string model is awkward, then trying to make `Substring` a currency type for slices of non-`String` buffers is only going to make it **more** complex, not less, without a much bigger overhaul of the API than you're proposing here.

---

<div class="post-metadata">

**Author:** ![saeta](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/saeta/32/6264_2.png) [@saeta](https://forums.swift.org/u/saeta)\
**Post date:** [July 20, 2020, 3:55pm UTC](https://forums.swift.org/t/shared-substrings/38547/15 "2020-07-20T15:55:58Z")

</div>

> [@Karl](#):
>
> One thing that might be nice to add is some way to bypass UTF8 verification.

+1 to this. I have found UTF-8 verification to be a significant enough performance challenge that I had to abandon using Swift `String`s in [a library I wrote](https://github.com/saeta/penguin/commit/feca5379413207a35bf5e30ce1a1fc7b4917259d).

---

<div class="post-metadata">

**Author:** ![dabrahams](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/dabrahams/32/9815_2.png) [@dabrahams](https://forums.swift.org/u/dabrahams)\
**Post date:** [July 20, 2020, 4:08pm UTC](https://forums.swift.org/t/shared-substrings/38547/16 "2020-07-20T16:08:00Z")

</div>

Please no. `String` and `SubString` are safe types: when you write an API receiving them, you know you can store and manipulate them in perpetuity without violating memory safety. The whole point of `StringProtocol` is that you can build unsafe strings without violating the safety of these core types. It would be as wrong to create arrays that point to unmanaged memory.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 20, 2020, 5:14pm UTC](https://forums.swift.org/t/shared-substrings/38547/17 "2020-07-20T17:14:41Z")

</div>

> [@dabrahams](#):
>
> `String` and `SubString` are safe types: when you write an API receiving them, you know you can store and manipulate them in perpetuity without violating memory safety.

Could you explain how this makes either of those types unsafe? The concerns about storage lifetimes and which type to use are about leakage, not safety.

You can totally store a shared Substring for as long as like: it’s just discouraged, is all.

---

<div class="post-metadata">

**Author:** ![MrMage](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/mrmage/32/3829_2.png) [@MrMage](https://forums.swift.org/u/MrMage)\
**Post date:** [July 20, 2020, 5:46pm UTC](https://forums.swift.org/t/shared-substrings/38547/18 "2020-07-20T17:46:47Z")

</div>

Another +1. My app spends a considerable amount of time validating UTF-8 strings that are expected to be valid in the first place.

I am a bit concerned about the usefulness of `Substring` though. Having a similar option to obtain a `String` would be great. My use case would mostly involve using entire strings, so wasting memory by retaining a long string while only using a small slice of it would not be a concern.

---

<div class="post-metadata">

**Author:** ![dabrahams](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/dabrahams/32/9815_2.png) [@dabrahams](https://forums.swift.org/u/dabrahams)\
**Post date:** [July 20, 2020, 6:03pm UTC](https://forums.swift.org/t/shared-substrings/38547/19 "2020-07-20T18:03:42Z")

</div>

> [@Karl](#):
>
> > [@dabrahams](#):
> >
> > `String` and `SubString` are safe types: when you write an API receiving them, you know you can store and manipulate them in perpetuity without violating memory safety.
> 
> Could you explain how this makes either of those types unsafe? The concerns about storage lifetimes and which type to use are about leakage, not safety.

This API:

```swift
extension SubString {
  public init?(sharingStorage buffer: UnsafeBufferPointer<UInt8>, owner: AnyObject)
}

```

Allows me to create a new `SubString` that can outlive the buffer it references. If the buffer is deallocated or overwritten while (a copy of) the `SubString` is alive, you now have a “safe” type all of whose interesting operations violate memory safety.

---

<div class="post-metadata">

**Author:** ![MrMage](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/mrmage/32/3829_2.png) [@MrMage](https://forums.swift.org/u/MrMage)\
**Post date:** [July 20, 2020, 6:07pm UTC](https://forums.swift.org/t/shared-substrings/38547/20 "2020-07-20T18:07:48Z")

</div>

> [@dabrahams](#):
>
> This API:
> 
> ```swift
> extension SubString {
> public init?(sharingStorage buffer: UnsafeBufferPointer<UInt8>, owner: AnyObject)
> }
> 
> ```
> 
> Allows me to create a new `SubString` that can outlive the buffer it references. If the buffer is deallocated or overwritten while (a copy of) the `SubString` is alive, you now have a “safe” type all of whose interesting operations violate memory safety.

My understanding is that the `SubString` would retain `owner`, and the user is expected to guarantee that the buffer does not get deallocated before `owner` is deallocated. In that case, `SubString` should automatically extend the buffer's lifetime as needed.

---

<div class="post-metadata">

**Author:** ![Karl](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@Karl](https://forums.swift.org/u/Karl)\
**Post date:** [July 20, 2020, 6:17pm UTC](https://forums.swift.org/t/shared-substrings/38547/21 "2020-07-20T18:17:18Z")

</div>

> [@MrMage](#):
>
> My understanding is that the `SubString` would retain `owner`

Correct. The only way you would violate memory safety is if there was a bug in your code.

We already have plenty of unsafe APIs which can cause undefined behaviour if not used correctly, and UB anywhere is UB everywhere: an `UnsafeMutableBufferPointer` overflow somewhere could corrupt a neighbouring Array's memory, and even though Array is "safe", this object's behaviour is unknown. No code can be resilient even to bugs.

An example with Array:

```swift
class SomeClass {}
let boom = Array<SomeClass>(unsafeUninitializedCapacity: 10) { ptr, count in
  count = 5 
}

```

Maybe it's worth adding the word `unsafe` to the name for visibility? `init(sharingUnsafeStorage:)`?

[Next page](https://forums.swift.org/t/shared-substrings/38547.md?page=2)