How to use Static, Read-only arrays

carlos42421 · 2024-06-17T16:07:53.392Z

I think this isn't strictly an embedded question.

I'm curious about the new static, read-only arrays that I've seen mentioned in the compiler and standard library code base, introduced over the last few months.

It looks like this is something like an optimisation, perhaps allowing read-only (let) top-level globally defined array constants to be defined more optimally, and the data stored in global memory rather than on the heap, possibly (given that they're immortal for the life of the program).

Is that something we would be able to use? It would be very handy not having malloc and free in our code because "tables" of top-level global constant values, are a common idiom in our production micro controller code. If we could make them more efficient (and also more visible because we can report the size of global memory precisely to the developer at build time).

A quick glance/grep in the code makes me think they're not yet available outside of Darwin?

bool IRGenModule::canMakeStaticObjectReadOnly(SILType objectType) {
  if (getOptions().DisableReadonlyStaticObjects)
    return false;

  // TODO: Support constant static arrays on other platforms, too.
  // See also the comment in GlobalObjects.cpp.
  if (!Triple.isOSDarwin())
    return false;

When I create top level array constants on our platform they use _ContiguousArrayStorage as normal at the moment.

Any information hungrily and excitedly received.

Carl

Erik_Eckstein · 2024-06-17T17:14:16.682Z

Do you build with -parse-as-library? (It only works in this mode)

kubamracek · 2024-06-17T17:47:19.275Z

If you still observe this in -parse-as-library mode, would you mind filing a Github issue for this?

carlos42421 · 2024-06-17T21:44:36.724Z

Ah, no. This is our embedded programs. So the array in question is a top level let variable at the top of main.swift in something like a command line executable. (We actually link it into an on the metal binary, that runs at microcontroller boot up and always executes an infinite loop... almost certainly the same thing Kuba is doing with embedded swift programs.)

I'll try to create a standalone module linked with parse-as-library to link into this executable and see if it works there.

carlos42421 · 2024-06-18T16:41:50.665Z

Confirmed it's still an issue with parse-as-library when targeting AVR. I'll try to raise an issue tomorrow.

carlos42421 · 2024-06-19T13:51:52.261Z

github.com/apple/swift

Fixed-size, constant, global array of Ints not promoted to StaticArray storage on some targets

opened 01:51PM - 19 Jun 24 UTC

carlos4242

bug triage needed

### Description According to the developers on the forum, in the above case, wh…en built as a library, an array constant like this should automatically be promoted from heap allocated to global storage. This is not happening when I build for AVR. Given that our code is not very different from trunk (it's a couple of weeks old, from just before WWDC), I suspect it may not work on other platforms (e.g. Linux). ### Reproduction ```swift let pots = [41,52,99,1,4,4] public enum ValidState { case yes case no case undefined } public func checkPot(i: UInt8, testValue: UInt8) -> ValidState { guard i < pots.count else { return .undefined } return pots[Int(i)] < testValue ? .yes : .no } ``` When I compile the above code using -parse-as-library, I should expect the memory to be promoted to global memory but I am still seeing calls to swift_allocObject and swift_slowAlloc in decompiled code. It still seems to be using _ContiguousArrayStorage internally and therefore using a heap allocated class internally. ### Expected behavior storage promoted to global memory, no heap usage in the program ### Environment Swift version 6.0-dev (LLVM c7c87ee42989d4b, Swift ebfddb42549f691) ### Additional information _No response_

HTH

kubamracek · 2024-06-19T15:53:00.512Z

Thanks, I'll take a look!

carlos42421 · 2024-06-20T12:42:59.956Z

I saw the bug is fixed. I'll rebase and rebuild a compiler over the weekend to check it works. Thanks!

carlos42421 · 2024-06-23T20:22:51.248Z

I'm not 100% sure if it's working or not as I'm not sure of the expected behaviour?

Rebased on cdfeefc68b689b90653b20136a53c0858adc154c the compiler still seems to be creating a heap based object behind the scenes I think?

EDIT: I tested carefully to make sure it was parsed as library and I still think I''m seeing it.

I built tests as described in the GitHub issue: Fixed-size, constant, global array of Ints not promoted to StaticArray storage on some targets · Issue #74549 · swiftlang/swift · GitHub

This library code...

let pots = [41,52,99,1,4,4]

public enum ValidState {
    case yes
    case no
    case undefined
}

public func checkPot(i: UInt8, testValue: UInt8) -> ValidState {
    guard i < pots.count else { return .undefined }
    return pots[Int(i)] < testValue ? .yes : .no
}

...is compiled with the new compiler, including PR [embedded] Enable read-only static array promotion in embedded mode by kubamracek · Pull Request #74551 · swiftlang/swift · GitHub and I see this function for the array initialiser:

; Function Attrs: nounwind
define protected void @"$s5sats24pots_WZ"(ptr nocapture readnone %0) addrspace(1) #0 {
  %2 = tail call noalias addrspace(1) ptr @swift_allocObject(ptr nonnull @"$ss23_ContiguousArrayStorageCys5UInt8VGN", i16 14, i16 poison) #1
  %3 = getelementptr inbounds %Ts28__ContiguousArrayStorageBaseC, ptr %2, i16 0, i32 1
  store i16 6, ptr %3, align 2
  %4 = getelementptr inbounds %Ts28__ContiguousArrayStorageBaseC, ptr %2, i16 0, i32 1, i32 0, i32 1
  store i16 12, ptr %4, align 2
  %5 = getelementptr inbounds i8, ptr %2, i16 8
  store i8 41, ptr %5, align 1
  %6 = getelementptr inbounds i8, ptr %2, i16 9
  store i8 52, ptr %6, align 1
  %7 = getelementptr inbounds i8, ptr %2, i16 10
  store i8 99, ptr %7, align 1
  %8 = getelementptr inbounds i8, ptr %2, i16 11
  store i8 1, ptr %8, align 1
  %9 = getelementptr inbounds i8, ptr %2, i16 12
  store i8 4, ptr %9, align 1
  %10 = getelementptr inbounds i8, ptr %2, i16 13
  store i8 4, ptr %10, align 1
  store ptr %2, ptr @"$s5sats24potsSays5UInt8VGvp", align 2
  ret void
}

...which looks like it's still storing the array guts on the heap I think?

I'm happy to build a debug compiler and step through the optimisation to try and work out where it's not going to plan if that's useful? I might just need a few pointers how best to do that?

p.s. The compile command is...
"/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/swift/swiftc" -target avr-atmel-linux-gnueabihf -emit-bc -O -enforce-exclusivity=unchecked -parse-as-library -import-underlying-module -I . -O -no-link-objc-runtime -Xfrontend -disable-reflection-metadata -Xfrontend -disable-stack-protector -enable-experimental-feature Embedded -nostdimport -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwift-AVR/Embedded" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/uSwiftShims/Embedded" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libgcc/include" -I "/Users/carl/Library/Developer/Xcode/DerivedData/Swift_For_Arduino-ekjmfkrcyudnlpemyalpewpvtamj/Build/Products/Debug/Swift For Arduino.app/Contents/XPCServices/BuildEngine.xpc/Contents/Resources/gpl-tools-avr/lib/avr-libc/include" -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -I"/Users/carl/Documents/SwiftForArduino/Exports"/Embedded -I"/Users/carl/Library/Application Support/SwiftForArduino/Extensions/Modules"/Embedded -I"/Users/carl/Library/Application Support/SwiftForArduino/S4A/206/Modules"/Embedded -Xcc -DAVR_LIBC_DEFINED -Xcc -DLIBC_DEFINED -Xcc -D__AVR_ATmega328P__ -Xcc -DF_CPU=16000000 -DAVR_LIBC_DEFINED_SWIFT $(cat build/packages-swift-includes-list.txt) -module-name sats2 -whole-module-optimization -num-threads 4 -output-file-map build/outputMap.json constants.swift

carlos42421 · 2024-06-24T10:28:05.907Z

Footnote: I'm puzzling over this a little bit because you even have a unit test for a non standard platform (armv7em-none-none-eabi) and it works there, with my code. I was thinking, maybe the best way forwards is for me to focus on submitting PRs to get AVR as a supported target for Swift on the mainline compiler? Then it's going to be a lot easier to find and fix issues like this, because we'll have a working test suite, it'll be easy to build a testbed, etc.? Do you think the team and maintainers involved will be supportive of me adding AVR as a platform? I'm going to submit a fairly small bug fix PR today for 16-bit pointer support, which is independent of this. But I can also create a separate PR afterwards for adding AVR. I'll keep things like fixes for Harvard architecture etc out of the PR, to keep it simple.

Philippe_Hausler · 2024-06-24T14:53:39.174Z

I would welcome 16 bit support in the compiler, from a maintenance standpoint it seems like a better proposition for you and the language in the long term. That is a great first step to getting AVR at the compiler level supported.

Erik_Eckstein · 2024-06-25T14:10:05.844Z

carlos42421:

I'm happy to build a debug compiler and step through the optimisation to try and work out where it's not going to plan if that's useful? I might just need a few pointers how best to do that?

Cool! The ObjectOutliner pass should create the "outlined" array buffer global

// outlined variable #0 of one-time initialization function for pots
sil_global private @$s5sats24pots_WZTv_ : $_ContiguousArrayStorage<Int> = {
...

Looks like something going wrong there

carlos42421 · 2024-06-25T14:17:24.030Z

Yeah, I thought so too. I was expecting to see __StaticArrayStorage for the array storage.

(The unit test from [embedded] Enable read-only static array promotion in embedded mode by kubamracek · Pull Request #74551 · swiftlang/swift · GitHub confirms it too.)

I'm currently trying to build and test 16 bit support (GitHub - carlos4242/swift at 16bp) once I've done that and created a PR, I'll try to go back and build a debug swift supporting AVR targets and see if I can debug what's happening.

Thank you!

carlos42421 · 2024-06-25T17:43:24.954Z

OK @Philippe_Hausler here's a first stab at a PR. I think these might be the only 16 bit pointer fixes left. I upstreamed some work with Joe Groff and others a few years back.

github.com/swiftlang/swift

16 bit pointer fixes

swiftlang:main ← carlos4242:16bp

opened 05:40PM - 25 Jun 24 UTC

carlos4242

+11 -7

These changes fix some crashes when trying to build programs for 16-bit targets.…  Resolves #NNNNN, fix apple/llvm-project#MMMMM.

It says waiting for smoke tests to run but I think a contributor has to authorise that to make them run.

Locally, the tests seem to be mostly passing. I got a fail in /Users/carl/Documents/Code/swift/swift/test/Backtracing/CrashAsync.swift:41:11: error: CHECK: expected string not found in input but I'm guessing that's not caused by this PR. Also my tests seem to have hung at 99% so I'm not 100% sure. Hopefully when someone kicks off the smoke test we'll get passes.

Carl

carlos42421 · 2024-06-26T15:01:55.782Z

I think I'm going to try and focus on getting AVR as a proper build target for Swift if possible first, then see if this issue still occurs after we've done that. I won't debug it until we've got to that point and found that AVR still doesn't produce static arrays.