Unexpected error: "cannot use metatype of type 'UInt8' in embedded Swift" when initialising a simple array from an array literal

[carlos42421 (Carl Peto)]

Hiya, after a hiatus (great word), I finally got the Swift for Arduino IDE up to date with Swift 6 (from trunk a few days ago) and got embedded mode with embedded runtime ARC working. Pleasing.

However, I'm having a lot of trouble with arrays.

Even this simple, one-line program won't compile now...

let dummyArray: [UInt8] = [1,5,9,112,0,1,4]

I get the surprising (to me) error:

main.swift:1:27: error: cannot use metatype of type 'UInt8' in embedded Swift
let dummyArray: [UInt8] = [1,5,9,112,0,1,4]
                          ^
in function specialized static Array._allocateUninitialized(_:)

I suspect this might be partly related to us having a different version of Array in our standard library, but it's not all that different. I've changed the internal storage type that implements _ArrayBufferProtocol but it looks like the code is breaking way before that? Are there new parts of the standard library that I should be making sure we have brought over? Some extra #if !$Embedded that I need to add around some parts? Sorry it's a bit of a general question...

As an alternative, I saw in the PRs merged to trunk recently a lot of work on things called "fixed sized arrays", seemingly for global top level things. They sound amazingly useful for us! Should I be using them instead? If so, how do I "activate" them?

Thanks for any advice and help anyone can give!

Regards,
Carl

[kubamracek]

a different version of Array in our standard library

Right, this is likely the reason. I’m mostly curious now if we could find a way to not need a non-standard Array but also to not need a non-standard standard library as well. What kind of modifications do you actually need? And if you really need a different array type, is it concievable to perhaps be a non-stdlib addition (with a different name than Swift.Array)?

[carlos42421 (Carl Peto)]

Yeah, I see where you're coming from, it's a completely fair question.

I've sort of dodged the question a few times before now, for a few reasons. Firstly, I went to the effort to try to make a really minimal micro standard library for us, because I wanted to strip out lots of parts to avoid programmers on our platform "accidentally" including in their program very expensive and bulky parts that we really don't need for super simple (1-20kb) microcontroller programs. So I worked on the concept of starting with the absolute bare minimum and adding things as needed, rather than taking the normal library and taking bits away if that makes sense?

I think it's somewhat less important as a design approach now that embedded mode and all of the embedded variants of things exist, which I hope achieves some of the same goals in a more rigorous way.

But, Secondly, I'm a little reluctant to jettison our work and go back to the drawing board, just using the shipped standard library, just because it might expand to be a lot of work and might break a lot of things our side. I've fairly carefully crafted the existing "micro" standard library variant and it's a bit daunting throwing it away. Until we turned on embedded mode it worked and was stable. A known quantity. Tricky...

Also, we do also have behaviourally different things, such as a reference type, fixed size Array that's safer to use (it won't suddenly allocate more memory when you .append a new element or cause a COW fault). And our arithmetic is wrapping by default, to avoid trapping on overflow all over the place... we work with UInt8/Int8 far more than people usually do!

For the sake of argument. If we went the whole hog, abandoned our own standard library, changed our "safe for very small embedded devices Array" to be called something like RestrictedArray, added it in a third party library then told everyone "don't use Array unless you know exactly what you're doing, use RestrictedArray instead"... I fear that people would forget and we could end up with a lot of unintended consequences and programs constantly blowing past their flash memory footprints or (far worse) running rampant through the extremely limited RAM on devices and causing all sorts of stack overflows etc. It feels like we would be removing crucial guard rails that have served us well over the years if you see what I mean?

Finally, just in practical terms, swift doesn't support AVR. We have a compiler fork (GitHub - carlos4242/swift: The Swift Programming Language - avr-mods-4 branch) with all the things needed (including a lot of pointer tricks to support address space 1 for function pointers) that are unlikely to get upstreamed any time soon. So we have this multi stage process. We build the compiler, then we compile our own standard library afterwards, with the AVR compatible swiftc. So we can't hook into the normal CMake build that builds the standard library directly after the compiler. We sort of have to do our standard library as a standalone compile (currently with hand crafted Makefiles). A lot of it could just be my inexperience, but given all this, it's difficult to see how we'd make this work with the normal standard library and again I fear far too many late nights for not much benefit?

Sorry for the long-winded reply, I feel like I haven't answered this properly in the past.

I'm absolutely not trying to be awkward and 100% if there's no way forward without abandoning our fork then we will have to do that... but the path of least resistance is we try to work out a few simple fixes to get it working again! We only have very limited resources as a team, sadly.

However, all the above said... just switching Array back to the standard variant (if it's the best approach) feels like a manageable change and worth a try, if you think it will solve all our problems?

I hope that all makes sense?

[kubamracek]

@carlos42421

However, all the above said... just switching Array back to the standard variant (if it's the best approach) feels like a manageable change and worth a try, if you think it will solve all our problems?

I'd say it's definitely worth the try.

IIUC, your custom array has reference semantics -- if that's true, then I think it's even more important to have Swift.Array be the standard array (COW) and have a custom one with a different name. Otherwise correct Swift code will behave incorrectly with your AVR toolchain.

[carlos42421 (Carl Peto)]

Yes, that's right. Reference semantics. Our arrays are really more like C arrays... by design. When you create them, they're fixed size, you can change the contents of elements but not append or delete elements. So it allocates a buffer at creation then deallocates at release.

To implement this, I created my own array buffer _AVRArrayBuffer which I used instead of _ArrayBuffer or _ContiguousArrayBuffer.

The first iteration of this was heap stored but unmanaged and the user was responsible for deallocating arrays by hand...

@usableFromInline
@frozen
internal struct _AVRArrayBuffer<Element> : _ArrayBufferProtocol {
  public var _countAndCapacity: _ArrayBody

  @usableFromInline
  internal var _storage: UnsafeMutableBufferPointer<Element>

  // this is a temporary hack, we will get rid of this when we
  // have either reference counted heap memory or stack memory
  // the latter probably via a heap-to-stack promotion optimisation
  @inlinable
  public mutating func deallocate() {
    _storage.deallocate()
  }

  // AVR/uSwift note: if there is no memory available, this initialiser will 'fail'
  // and return nil (because the underlying UnsafeMutableBufferPointer.allocate will return nil)

  // we use a fixed buffer malloced on the heap
  // in later implementation, promote to alloca
  @inlinable
  @inline(__always)
  internal init?(
    _uninitializedCount uninitializedCount: Int,
    minimumCapacity: Int
  ) {
    let realMinimumCapacity = Swift.max(uninitializedCount, minimumCapacity)
    // malloc_size doesn't exist on avr gnu libc and allocations are
    // assumed byte aligned, so assume the allocation matches the request
    let realCapacity = realMinimumCapacity

    guard let buffer = UnsafeMutableBufferPointer<Element>.allocate(capacity: realCapacity) else {
      return nil
    }

    _storage = buffer
    _countAndCapacity = _ArrayBody(
      count: uninitializedCount,
      capacity: realCapacity,
      elementTypeIsBridgedVerbatim: false)
  }

  @inlinable
  internal init(count: Int, storage: UnsafeMutableBufferPointer<Element>) {
    _storage = storage
    _countAndCapacity = _ArrayBody(
      count: count,
      capacity: count,
      elementTypeIsBridgedVerbatim: false)
  }

  @inlinable
  internal init(_ storage: UnsafeMutableBufferPointer<Element>) {
    _storage = storage
    _countAndCapacity = _ArrayBody(
      count: storage.count,
      capacity: storage.count,
      elementTypeIsBridgedVerbatim: false)
  }

Later on, for the version 5.0 IDE, I created a basic, stripped down ARC runtime, copied from the full ARC C++ runtime and stripped back to basics. At this point it would probably have made more sense in an ideal world to go back to something closer to the normal Swift _ArrayBuffer using allocWithTailElems_1 to create reference counted storage, etc. But I had fairly limited time, so I added a simple "sentinel" class that deallocated the internal storage when the instance of _AVRArrayBuffer was deinited...

import uSwiftShims

#if !UNMANAGED_ARRAYS
@usableFromInline
internal class _AVRArrayBufferStorageManager {
  // this is the raw pointer for the internal buffer of the AVR Array Buffer storage
  // we use it here to avoid the complexities/overhead of generic type parameters on this class
  // the deinit will internally destroy this buffer as deallocate() would
  // use of the buffer or array after a manual deallocate will create havoc
  // the microcontroller will probably let you write into random memory... don't do it
  internal var _rawValue: Builtin.RawPointer

  @usableFromInline
  init(_ _rawValue: Builtin.RawPointer) {
    self._rawValue = _rawValue
  }

  deinit {
    // check not already deallocated
    guard Int(Builtin.ptrtoint_Word(_rawValue)) != 0 else {
      return
    }

    Builtin.deallocRaw(_rawValue, (-1 as Int)._builtinWordValue, (0 as Int)._builtinWordValue)
  }
}
#endif

@usableFromInline
@frozen
internal struct _AVRArrayBuffer<Element> : _ArrayBufferProtocol {
  public var _countAndCapacity: _ArrayBody

  @usableFromInline
  internal var _storage: UnsafeMutableBufferPointer<Element>

#if !UNMANAGED_ARRAYS
  @usableFromInline
  internal var _storageSentinel: _AVRArrayBufferStorageManager?
#endif

  @inlinable
  public mutating func deallocate() {
#if UNMANAGED_ARRAYS
    _storage.deallocate()
#else
    _storageSentinel = nil
#endif
  }

  // AVR/uSwift note: if there is no memory available, this initialiser will 'fail'
  // and return nil (because the underlying UnsafeMutableBufferPointer.allocate will return nil)

  // we use a fixed buffer malloced on the heap
  // in later implementation, promote to alloca
  @inlinable
  @inline(__always)
  internal init?(
    _uninitializedCount uninitializedCount: Int,
    minimumCapacity: Int
  ) {
    let realMinimumCapacity = Swift.max(uninitializedCount, minimumCapacity)
    // malloc_size doesn't exist on avr gnu libc and allocations are
    // assumed byte aligned, so assume the allocation matches the request
    let realCapacity = realMinimumCapacity

    guard let buffer = UnsafeMutableBufferPointer<Element>.allocate(capacity: realCapacity) else {
      return nil
    }

    _storage = buffer
#if !UNMANAGED_ARRAYS
    _storageSentinel = _AVRArrayBufferStorageManager(buffer._position!._rawValue)
#endif
    _countAndCapacity = _ArrayBody(
      count: uninitializedCount,
      capacity: realCapacity,
      elementTypeIsBridgedVerbatim: false)
  }
...

This is the version we have today.

I know it's hard to understand this approach if you're not used to our platform. These two versions were definitely good enough for our developers and got people moving. The most common use case for arrays was effectively a static, read-only collection of constants... things like fixed timings for a set of physical components, or initial values to set voltages for DAC lines on system reboot of a developer's program on an atmega328p. There were probably other approaches we could have taken but for expediency the first approach got the version 4.1 IDE out of the door a few years ago when I did it and it gave people familiar semantics. And the second approach in 5.0 to version 5.3.5 was a welcome improvement. Although it might surprise you to hear it, no one has ever got tripped up with the absence of Value type / COW semantics because our programs are so small and self contained.

I don't in any way describe our approach because I think "it's better" or the Apple approach was "wrong"... our approach is right for us and has worked to build a platform where people can make commercial grade products with it. I completely sympathise that it's probably the wrong approach for most platforms.

Also, it has been really, really interesting and fun playing with the internals of things like this, seeing how the mechanisms of the internals of Array work in Swift and creating new alternatives. I hope it is inspiring and interesting for people to see what we did... even if it ends up getting consigned to history!

Carl

[carlos42421 (Carl Peto)]

Hi @kubamracek ... another little update. I reworked Array to use _ContiguousArrayBuffer as normal but I'm still getting the same compiler error. I used swift-ide-test to decompile my Swift module, just to make sure, and the old AVRArrayBuffer has gone, replaced by the normal array buffer.

So this is the definition of Array in the most recent clone of swift I have...

@frozen
@_eagerMove
public struct Array<Element>: _DestructorSafeContainer {
  #if _runtime(_ObjC)
  @usableFromInline
  internal typealias _Buffer = _ArrayBuffer<Element>
  #else
  @usableFromInline
  internal typealias _Buffer = _ContiguousArrayBuffer<Element>
  #endif

  @usableFromInline
  internal var _buffer: _Buffer

  /// Initialization from an existing buffer does not have "array.init"
  /// semantics because the caller may retain an alias to buffer.
  @inlinable
  internal init(_buffer: _Buffer) {
    self._buffer = _buffer
  }
}

...and the definition I'm working with is...

@frozen
public struct Array<Element>: _DestructorSafeContainer {

#if FORCE_MAIN_SWIFT_ARRAYS
  @usableFromInline
  internal typealias _Buffer = _ContiguousArrayBuffer<Element>
#else
  @usableFromInline
  internal typealias _Buffer = _AVRArrayBuffer<Element>
#endif

  @usableFromInline
  internal var _buffer: _Buffer

  @inlinable
  internal init(_buffer: _Buffer) {
    self._buffer = _buffer
  }
}

(With FORCE_MAIN_SWIFT_ARRAYS defined in the builds I'm using.)

This simple program still fails to compile:
let arr = [3]

---

Given this is leading away from my standard library experiments a bit, I thought it was time to dive into the SIL and debugging the compiler.

The raw SIL...

sil_stage raw

import Builtin
import Swift
import SwiftShims

@_hasStorage @_hasInitialValue let arr: [UInt8] { get }

// arr
sil_global hidden [let] @$s4main3arrSays5UInt8VGvp : $Array<UInt8>

// main
sil [ossa] @main : $@convention(c) (Int32, UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>) -> Int32 {
bb0(%0 : $Int32, %1 : $UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>):
  alloc_global @$s4main3arrSays5UInt8VGvp         // id: %2
  %3 = global_addr @$s4main3arrSays5UInt8VGvp : $*Array<UInt8> // user: %16
  %4 = integer_literal $Builtin.Word, 1           // user: %6
  // function_ref _allocateUninitializedArray<A>(_:)
  %5 = function_ref @$ss27_allocateUninitializedArrayySayxG_BptBwlF : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer) // user: %6
  %6 = apply %5<UInt8>(%4) : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer) // user: %7
  (%7, %8) = destructure_tuple %6 : $(Array<UInt8>, Builtin.RawPointer) // users: %16, %9, %9
  %9 = mark_dependence %8 : $Builtin.RawPointer on %7 : $Array<UInt8> // user: %10
  %10 = pointer_to_address %9 : $Builtin.RawPointer to [strict] $*UInt8 // user: %15
  %11 = integer_literal $Builtin.IntLiteral, 3    // user: %14
  %12 = metatype $@thin UInt8.Type                // user: %14
  // function_ref UInt8.init(_builtinIntegerLiteral:)
  %13 = function_ref @$ss5UInt8V22_builtinIntegerLiteralABBI_tcfC : $@convention(method) (Builtin.IntLiteral, @thin UInt8.Type) -> UInt8 // user: %14
  %14 = apply %13(%11, %12) : $@convention(method) (Builtin.IntLiteral, @thin UInt8.Type) -> UInt8 // user: %15
  store %14 to [trivial] %10 : $*UInt8            // id: %15
  store %7 to [init] %3 : $*Array<UInt8>          // id: %16
  %17 = integer_literal $Builtin.Int32, 0         // user: %18
  %18 = struct $Int32 (%17 : $Builtin.Int32)      // user: %19
  return %18 : $Int32                             // id: %19
} // end sil function 'main'

// _allocateUninitializedArray<A>(_:)
sil [serialized] [always_inline] [_semantics "array.uninitialized_intrinsic"] @$ss27_allocateUninitializedArrayySayxG_BptBwlF : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer)

// UInt8.init(_builtinIntegerLiteral:)
sil [transparent] [serialized] @$ss5UInt8V22_builtinIntegerLiteralABBI_tcfC : $@convention(method) (Builtin.IntLiteral, @thin UInt8.Type) -> UInt8



// Mappings from '#fileID' to '#filePath':
//   'main/main.swift' => 'main.swift'


Emitted build/main.sil

Canonical SIL cannot be made, the error occurs before then...

main.swift:1:11: error: cannot use metatype of type 'UInt8' in embedded Swift
let arr = [3]
          ^
in function specialized static Array._allocateUninitialized(_:)
make: *** [build/main.csil] Error 1

Breaking on the diagnostic being emitted in the compiler and looking at the SIL function in question...

 e function->dump()
// specialized static Array._allocateUninitialized(_:)
sil shared [always_inline] [_semantics "array.uninitialized"] [ossa] @$sSa22_allocateUninitializedySayxG_SpyxGtSizFZs5UInt8V_Tgm5 : $@convention(thin) (@inout Int) -> (@owned Array<UInt8>, UnsafeMutablePointer<UInt8>) {
// %0                                             // users: %32, %13, %10
bb0(%0 : $*Int):
  %1 = metatype $@thin Array<UInt8>.Type          // user: %8
  %2 = metatype $@thick UInt8.Type                // user: %4
  %3 = metatype $@thick UInt8.Type                // user: %5
  %4 = unchecked_trivial_bit_cast %2 : $@thick UInt8.Type to $Builtin.Word // user: %6
  %5 = unchecked_trivial_bit_cast %3 : $@thick UInt8.Type to $Builtin.Word // user: %6
  %6 = builtin "cmp_eq_Word"(%4 : $Builtin.Word, %5 : $Builtin.Word) : $Builtin.Int1 // user: %7
  cond_br %6, bb1, bb2                            // id: %7

bb1:                                              // Preds: bb0
  %8 = unchecked_trivial_bit_cast %1 : $@thin Array<UInt8>.Type to $@thin Array<UInt8>.Type // user: %10
  // function_ref specialized static Array._allocateUninitialized(_:)
  %9 = function_ref @$sSa22_allocateUninitializedySayxG_SpyxGtSizFZs5UInt8V_Tgq5 : $@convention(method) (@inout Int, @thin Array<UInt8>.Type) -> (@owned Array<UInt8>, UnsafeMutablePointer<UInt8>) // user: %10
  %10 = apply %9(%0, %8) : $@convention(method) (@inout Int, @thin Array<UInt8>.Type) -> (@owned Array<UInt8>, UnsafeMutablePointer<UInt8>) // user: %11
  %11 = unchecked_value_cast %10 : $(Array<UInt8>, UnsafeMutablePointer<UInt8>) to $(Array<UInt8>, UnsafeMutablePointer<UInt8>) // user: %12
  br bb3(%11 : $(Array<UInt8>, UnsafeMutablePointer<UInt8>)) // id: %12

bb2:                                              // Preds: bb0
  %13 = load [trivial] %0 : $*Int                 // users: %63, %58, %45, %16
  %14 = integer_literal $Builtin.Int8, 0          // user: %50
  %15 = integer_literal $Builtin.Int16, 0         // users: %52, %56, %22, %28, %17
  %16 = struct_extract %13 : $Int, #Int._value    // users: %52, %48, %22, %17
  %17 = builtin "cmp_slt_Int16"(%16 : $Builtin.Int16, %15 : $Builtin.Int16) : $Builtin.Int1 // user: %18
  cond_br %17, bb4, bb5                           // id: %18

// %19                                            // user: %20
bb3(%19 : @owned $(Array<UInt8>, UnsafeMutablePointer<UInt8>)): // Preds: bb7 bb1
  return %19 : $(Array<UInt8>, UnsafeMutablePointer<UInt8>) // id: %20

bb4:                                              // Preds: bb2
  br bb6                                          // id: %21

bb5:                                              // Preds: bb2
  %22 = builtin "cmp_eq_Int16"(%16 : $Builtin.Int16, %15 : $Builtin.Int16) : $Builtin.Int1 // user: %23
  cond_br %22, bb8, bb9                           // id: %23

bb6:                                              // Preds: bb14 bb8 bb4
  // function_ref specialized _ContiguousArrayBuffer.init()
  %24 = function_ref @$ss22_ContiguousArrayBufferVAByxGycfCs5UInt8V_Tgm5 : $@convention(thin) () -> @owned _ContiguousArrayBuffer<UInt8> // user: %25
  %25 = apply %24() : $@convention(thin) () -> @owned _ContiguousArrayBuffer<UInt8> // users: %30, %26, %31
  %26 = copy_value %25 : $_ContiguousArrayBuffer<UInt8> // user: %27
  %27 = move_value [lexical] %26 : $_ContiguousArrayBuffer<UInt8> // user: %33
  %28 = struct $Int (%15 : $Builtin.Int16)        // users: %30, %32
  // function_ref specialized _ContiguousArrayBuffer.count.setter
  %29 = function_ref @$ss22_ContiguousArrayBufferV5countSivss5UInt8V_Tg5 : $@convention(method) (Int, @guaranteed _ContiguousArrayBuffer<UInt8>) -> () // user: %30
  %30 = apply %29(%28, %25) : $@convention(method) (Int, @guaranteed _ContiguousArrayBuffer<UInt8>) -> ()
  destroy_value %25 : $_ContiguousArrayBuffer<UInt8> // id: %31
  store %28 to [trivial] %0 : $*Int               // id: %32
  br bb7(%27 : $_ContiguousArrayBuffer<UInt8>)    // id: %33

// %34                                            // user: %35
bb7(%34 : @owned $_ContiguousArrayBuffer<UInt8>): // Preds: bb13 bb6
  %35 = struct $Array<UInt8> (%34 : $_ContiguousArrayBuffer<UInt8>) // user: %36
  %36 = move_value [lexical] [var_decl] %35 : $Array<UInt8> // users: %42, %37
  %37 = copy_value %36 : $Array<UInt8>            // user: %38
  %38 = destructure_struct %37 : $Array<UInt8>    // users: %40, %41
  // function_ref specialized _ContiguousArrayBuffer.firstElementAddress.getter
  %39 = function_ref @$ss22_ContiguousArrayBufferV19firstElementAddressSpyxGvgs5UInt8V_Tg5 : $@convention(method) (@guaranteed _ContiguousArrayBuffer<UInt8>) -> UnsafeMutablePointer<UInt8> // user: %40
  %40 = apply %39(%38) : $@convention(method) (@guaranteed _ContiguousArrayBuffer<UInt8>) -> UnsafeMutablePointer<UInt8> // user: %42
  destroy_value %38 : $_ContiguousArrayBuffer<UInt8> // id: %41
  %42 = tuple (%36 : $Array<UInt8>, %40 : $UnsafeMutablePointer<UInt8>) // user: %43
  br bb3(%42 : $(Array<UInt8>, UnsafeMutablePointer<UInt8>)) // id: %43

bb8:                                              // Preds: bb5
  br bb6                                          // id: %44

bb9:                                              // Preds: bb5
  %45 = destructure_struct %13 : $Int             // user: %46
  %46 = builtin "truncOrBitCast_Int16_Int8"(%45 : $Builtin.Int16) : $Builtin.Int8 // users: %50, %47
  %47 = builtin "zextOrBitCast_Int8_Int16"(%46 : $Builtin.Int8) : $Builtin.Int16 // user: %48
  %48 = builtin "cmp_eq_Int16"(%47 : $Builtin.Int16, %16 : $Builtin.Int16) : $Builtin.Int1 // user: %49
  cond_br %48, bb10, bb11                         // id: %49

bb10:                                             // Preds: bb9
  %50 = builtin "cmp_ult_Int8"(%14 : $Builtin.Int8, %46 : $Builtin.Int8) : $Builtin.Int1 // user: %51
  br bb12(%50 : $Builtin.Int1)                    // id: %51

bb11:                                             // Preds: bb9
  %52 = builtin "cmp_slt_Int16"(%15 : $Builtin.Int16, %16 : $Builtin.Int16) : $Builtin.Int1 // user: %53
  br bb12(%52 : $Builtin.Int1)                    // id: %53

// %54                                            // user: %55
bb12(%54 : $Builtin.Int1):                        // Preds: bb11 bb10
  cond_br %54, bb13, bb14                         // id: %55

bb13:                                             // Preds: bb12
  %56 = struct $Int (%15 : $Builtin.Int16)        // user: %58
  // function_ref specialized _ContiguousArrayBuffer.init(_uninitializedCount:minimumCapacity:)
  %57 = function_ref @$ss22_ContiguousArrayBufferV19_uninitializedCount15minimumCapacityAByxGSi_SitcfCs5UInt8V_Tgm5 : $@convention(thin) (Int, Int) -> @owned _ContiguousArrayBuffer<UInt8> // user: %58
  %58 = apply %57(%56, %13) : $@convention(thin) (Int, Int) -> @owned _ContiguousArrayBuffer<UInt8> // user: %59
  %59 = move_value [lexical] [var_decl] %58 : $_ContiguousArrayBuffer<UInt8> // users: %63, %64, %60
  %60 = copy_value %59 : $_ContiguousArrayBuffer<UInt8> // user: %61
  %61 = move_value [lexical] %60 : $_ContiguousArrayBuffer<UInt8> // user: %65
  // function_ref specialized _ContiguousArrayBuffer.count.setter
  %62 = function_ref @$ss22_ContiguousArrayBufferV5countSivss5UInt8V_Tg5 : $@convention(method) (Int, @guaranteed _ContiguousArrayBuffer<UInt8>) -> () // user: %63
  %63 = apply %62(%13, %59) : $@convention(method) (Int, @guaranteed _ContiguousArrayBuffer<UInt8>) -> ()
  destroy_value %59 : $_ContiguousArrayBuffer<UInt8> // id: %64
  br bb7(%61 : $_ContiguousArrayBuffer<UInt8>)    // id: %65

bb14:                                             // Preds: bb12
  br bb6                                          // id: %66
} // end sil function '$sSa22_allocateUninitializedySayxG_SpyxGtSizFZs5UInt8V_Tgm5'

...the SIL instruction that is causing the diagnostic to emit is...

%2 = metatype $@thick UInt8.Type

...is this making any sense to you why it's emitting a diagnostic for me, but I guess works in normal embedded swift?

Any help greatly appreciated.
Carl

[kubamracek]

  %1 = metatype $@thin Array<UInt8>.Type          // user: %8
  %2 = metatype $@thick UInt8.Type                // user: %4
  %3 = metatype $@thick UInt8.Type                // user: %5
  %4 = unchecked_trivial_bit_cast %2 : $@thick UInt8.Type to $Builtin.Word // user: %6
  %5 = unchecked_trivial_bit_cast %3 : $@thick UInt8.Type to $Builtin.Word // user: %6
  %6 = builtin "cmp_eq_Word"(%4 : $Builtin.Word, %5 : $Builtin.Word) : $Builtin.Int1 // user: %7

This looks like the result of EagerSpecializer SIL pass creating a type check for a @_specialize-annotated function. Do you have such an attribute in your stdlib? CC @Erik_Eckstein

[carlos42421 (Carl Peto)]

Interesting. The answer is sort of yes/no!

Firstly, I do have @_specialize in my stdlib, so for a minute I thought this might be the answer...

...

  @_specialize(kind: full, where Element == UInt8)
  @inlinable
  @inline(__always)
  @_semantics("array.uninitialized")
  internal static func _allocateUninitialized(
    _ count: inout Int
  ) -> (Array, UnsafeMutablePointer<Element>) {
    let result = Array(_uninitializedCount: &count)
    return (result, result._buffer.firstElementAddress)
  }

...

  @_specialize(kind: full, where Element == UInt8)
  // @inline(never)
  @inlinable
  @inline(__always)
  @usableFromInline
  internal static func _allocateBufferUninitialized(
    minimumCapacity: Int
  ) -> _Buffer? {
#if FORCE_MAIN_SWIFT_ARRAYS
    return _ContiguousArrayBuffer<Element>(_uninitializedCount: 0, minimumCapacity: minimumCapacity)
#else    
    return _AVRArrayBuffer<Element>(_uninitializedCount: 0, minimumCapacity: minimumCapacity)
#endif
  }

...

  @_specialize(kind: full, where Element == UInt8)
  @inlinable
  @inline(__always)
  internal init(_uninitializedCount count: inout Int) {
    _precondition(count >= 0)
    // Note: Sinking this constructor into an else branch below causes an extra
    // Retain/Release.
    // _buffer = _Buffer()
    if count > 0, let buffer = Array._allocateBufferUninitialized(minimumCapacity: count) {
      // Creating a buffer instead of calling reserveCapacity saves doing an
      // unnecessary uniqueness check. We disable inlining here to curb code
      // growth.
      _buffer = buffer
      _buffer.count = count
    } else {
      // this case is when either count is zero, or we were unable to allocate a buffer
      // we need to zero the count in either case
      _buffer = _Buffer() // we are using a different memory model/no ref count
      _buffer.count = 0
      count = 0
    }
    // Can't store count here because the buffer might be pointing to the
    // shared empty array.
  }

...

  @_specialize(kind: full, where Element == UInt8)
  @inlinable
  @inline(__always)
  @_semantics("array.uninitialized")
  internal static func _allocateUninitialized(
    _ count: inout Int
  ) -> (Array, UnsafeMutablePointer<Element>) {
    let result = Array(_uninitializedCount: &count)
    return (result, result._buffer.firstElementAddress)
  }

...

...so I changed the test program to create an array of UInt16 instead to see if this resolves the issue... and I still get the same class of diagnostic:

public typealias IntegerLiteralType = UInt16

let arr: [UInt16] = [3]

...produces...

main.swift:3:21: 
error: cannot use metatype of type 'UInt16' in embedded Swift

let arr: [UInt16] = [3]
                    ^

in function specialized static Array._allocateUninitialized(_:)

And it produces basically the same raw SIL...

carl@LT1152 why does embedded mode break array % make build/main.sil
set -o pipefail && "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/swift/swiftc" -target avr-atmel-linux-gnueabihf -emit-silgen  -O -enforce-exclusivity=unchecked   -O -no-link-objc-runtime -Xfrontend -disable-reflection-metadata -Xfrontend -disable-stack-protector -enable-experimental-feature Embedded -nostdimport -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwift-AVR/Embedded" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwiftShims/Embedded"  -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libgcc/include" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libc/include" -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -I"/Users/carl/Library/Application Support/SwiftForArduino/Extensions/Modules"/Embedded -I"/Users/carl/Library/Application Support/SwiftForArduino/S4A/206/Modules"/Embedded -Xcc -DAVR_LIBC_DEFINED -Xcc -DLIBC_DEFINED -Xcc -D__AVR_ATmega328P__ -Xcc -DF_CPU=16000000 -DAVR_LIBC_DEFINED_SWIFT  -module-name main -whole-module-optimization \
	-num-threads 4 -output-file-map build/outputMap.json main.swift  2>&1|tee -a build/build_log.txt
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: '/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/swift/swift-driver-new'
<unknown>:0: warning: libc not found for 'avr-atmel-linux-gnueabihf'; C stdlib may be unavailable
sil_stage raw

import Builtin
import Swift
import SwiftShims

public typealias IntegerLiteralType = UInt16

@_hasStorage @_hasInitialValue let arr: [UInt16] { get }

// arr
sil_global hidden [let] @$s4main3arrSays6UInt16VGvp : $Array<UInt16>

// main
sil [ossa] @main : $@convention(c) (Int32, UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>) -> Int32 {
bb0(%0 : $Int32, %1 : $UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>):
  alloc_global @$s4main3arrSays6UInt16VGvp        // id: %2
  %3 = global_addr @$s4main3arrSays6UInt16VGvp : $*Array<UInt16> // user: %16
  %4 = integer_literal $Builtin.Word, 1           // user: %6
  // function_ref _allocateUninitializedArray<A>(_:)
  %5 = function_ref @$ss27_allocateUninitializedArrayySayxG_BptBwlF : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer) // user: %6
  %6 = apply %5<UInt16>(%4) : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer) // user: %7
  (%7, %8) = destructure_tuple %6 : $(Array<UInt16>, Builtin.RawPointer) // users: %16, %9, %9
  %9 = mark_dependence %8 : $Builtin.RawPointer on %7 : $Array<UInt16> // user: %10
  %10 = pointer_to_address %9 : $Builtin.RawPointer to [strict] $*UInt16 // user: %15
  %11 = integer_literal $Builtin.IntLiteral, 3    // user: %14
  %12 = metatype $@thin UInt16.Type               // user: %14
  // function_ref UInt16.init(_builtinIntegerLiteral:)
  %13 = function_ref @$ss6UInt16V22_builtinIntegerLiteralABBI_tcfC : $@convention(method) (Builtin.IntLiteral, @thin UInt16.Type) -> UInt16 // user: %14
  %14 = apply %13(%11, %12) : $@convention(method) (Builtin.IntLiteral, @thin UInt16.Type) -> UInt16 // user: %15
  store %14 to [trivial] %10 : $*UInt16           // id: %15
  store %7 to [init] %3 : $*Array<UInt16>         // id: %16
  %17 = integer_literal $Builtin.Int32, 0         // user: %18
  %18 = struct $Int32 (%17 : $Builtin.Int32)      // user: %19
  return %18 : $Int32                             // id: %19
} // end sil function 'main'

// _allocateUninitializedArray<A>(_:)
sil [serialized] [always_inline] [_semantics "array.uninitialized_intrinsic"] @$ss27_allocateUninitializedArrayySayxG_BptBwlF : $@convention(thin) <τ_0_0> (Builtin.Word) -> (@owned Array<τ_0_0>, Builtin.RawPointer)

// UInt16.init(_builtinIntegerLiteral:)
sil [transparent] [serialized] @$ss6UInt16V22_builtinIntegerLiteralABBI_tcfC : $@convention(method) (Builtin.IntLiteral, @thin UInt16.Type) -> UInt16



// Mappings from '#fileID' to '#filePath':
//   'main/main.swift' => 'main.swift'


Emitted build/main.sil

And again fails when I try to produce canonical SIL...

carl@LT1152 why does embedded mode break array % make build/main.csil
set -o pipefail && "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/swift/swiftc" -target avr-atmel-linux-gnueabihf -emit-sil  -O -enforce-exclusivity=unchecked   -O -no-link-objc-runtime -Xfrontend -disable-reflection-metadata -Xfrontend -disable-stack-protector -enable-experimental-feature Embedded -nostdimport -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwift-AVR/Embedded" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwiftShims/Embedded"  -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libgcc/include" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libc/include" -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -I"/Users/carl/Library/Application Support/SwiftForArduino/Extensions/Modules"/Embedded -I"/Users/carl/Library/Application Support/SwiftForArduino/S4A/206/Modules"/Embedded -Xcc -DAVR_LIBC_DEFINED -Xcc -DLIBC_DEFINED -Xcc -D__AVR_ATmega328P__ -Xcc -DF_CPU=16000000 -DAVR_LIBC_DEFINED_SWIFT  -module-name main -whole-module-optimization \
	-num-threads 4 -output-file-map build/outputMap.json main.swift  2>&1|tee -a build/build_log.txt
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: '/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/swift/swift-driver-new'
<unknown>:0: warning: libc not found for 'avr-atmel-linux-gnueabihf'; C stdlib may be unavailable
main.swift:3:21: error: cannot use metatype of type 'UInt16' in embedded Swift
let arr: [UInt16] = [3]
                    ^
in function specialized static Array._allocateUninitialized(_:)
make: *** [build/main.csil] Error 1

And again, breaking in the debugger for this updated test code gives the same call stack as before and roughly the same SIL in the function being optimised...

(lldb) up
frame #1: 0x0000000101a3d343 swift-frontend`(anonymous namespace)::PerformanceDiagnostics::visitFunctionEmbeddedSwift(this=0x000000031bb58ee8, function=0x00007fc03a8753d0, parentLoc=0x000000031bb58b20) at PerformanceDiagnostics.cpp:180:11
   177 	
   178 	  for (SILBasicBlock &block : *function) {
   179 	    for (SILInstruction &inst : block) {
-> 180 	      if (visitInst(&inst, PerformanceConstraints::None, parentLoc)) {
   181 	        if (inst.getLoc().getSourceLoc().isInvalid()) {
   182 	          auto demangledName = Demangle::demangleSymbolAsString(
   183 	              inst.getFunction()->getName(),
(lldb) e function->dump()
// specialized static Array._allocateUninitialized(_:)
sil shared [always_inline] [_semantics "array.uninitialized"] [ossa] @$sSa22_allocateUninitializedySayxG_SpyxGtSizFZs6UInt16V_Tgm5 : $@convention(thin) (@inout Int) -> (@owned Array<UInt16>, UnsafeMutablePointer<UInt16>) {
// %0                                             // users: %32, %13, %10
bb0(%0 : $*Int):
  %1 = metatype $@thin Array<UInt16>.Type         // user: %8
  %2 = metatype $@thick UInt16.Type               // user: %4
  %3 = metatype $@thick UInt8.Type                // user: %5
  %4 = unchecked_trivial_bit_cast %2 : $@thick UInt16.Type to $Builtin.Word // user: %6
  %5 = unchecked_trivial_bit_cast %3 : $@thick UInt8.Type to $Builtin.Word // user: %6
  %6 = builtin "cmp_eq_Word"(%4 : $Builtin.Word, %5 : $Builtin.Word) : $Builtin.Int1 // user: %7
  cond_br %6, bb1, bb2                            // id: %7

...

(lldb) e inst.dump()
  %2 = metatype $@thick UInt16.Type               // user: %4

---

Would the @_specialize attribute still be causing this error even though it (seems) like it only applies when Element == UInt8?

Carl

[carlos42421 (Carl Peto)]

EDIT: it turns out the answer to my final question is "yes"! Somehow the above SIL is emitted comparing metatype $@thick UInt16.Type with metatype $@thick UInt8.Type. I commented out all the @_specialize attributes on the Array constructors and it all works now! That was the issue all along.

It's progress to get back to using the same Array guts as "main" swift with our MicroSwiftStdlib.

All good. Thanks so much guys!

[kubamracek]

Nice! Glad to see progress :)

[carlos42421 (Carl Peto)]

Really, really cool! Thanks so much!

p.s. I'm curious about these static read only arrays and if we can use them, but I'll ask in a different post to avoid mixing the threads.