Hi all,
I'd like to pitch a more scalable representation for resources embedded in code by SwiftPM. I have a prototype implemented across SwiftPM and Swift Build, and I'd appreciate feedback on the API and the build-system approach.
Edit, September 6: I've updated the pitch and prototypes after incorporating feedback from this thread. Thanks to @OneSadCookie for suggesting C #embed, which replaces the format-specific objcopy and assembly machinery, and @benlings for suggesting RawSpan as the interface for the embedded bytes. I've also updated the alternatives section to discuss preserving the existing array interface and implementing this through a plugin. The benchmark numbers below are from the original prototype and are labeled accordingly.
Scalable embedInCode resources using object files
- Proposal: SE-NNNN
- Author: Gonzalo Larralde
- Review Manager: TBD
- Status: Pitch
- Implementation:
Introduction
This proposal adds an opt-in object-file representation for SwiftPM resources embedded in code. Instead of generating and compiling a Swift array literal containing every byte, SwiftPM generates a small C source file using #embed, compiles it into a target-native object file, and exposes its contents as an immutable RawSpan.
Motivation
SwiftPM currently supports embedding a resource directly into executable code:
.target(
name: "Assets",
resources: [
.embedInCode("payload.bin"),
]
)
For this resource, SwiftPM generates code equivalent to:
struct PackageResources {
static let payload_bin: [UInt8] = [
123, 0, 33, 100, 99, // ...
]
}
This is straightforward and works well for small resources. However, it scales poorly because every resource byte becomes a decimal literal in generated Swift source. SwiftPM must generate a source file several times larger than the original resource, and the compiler must then parse, type-check, and compile the resulting array expression.
This is particularly relevant to Embedded Swift and other environments without a filesystem, where embedding assets directly into the executable is a normal part of working with the platform.
In a local benchmark of the original prototype using a 1 MiB resource:
| Representation | Generated Swift | Build time | Peak build memory |
|---|---|---|---|
| Byte array | 3.7 MiB | ~70 seconds | ~7.2 GiB |
| Object file | 476 bytes | ~1.6 seconds | ~105 MiB |
With an 8 MiB resource, the byte-array representation generated approximately 30 MiB of Swift source and failed because the compiler could not type-check the expression in reasonable time. The object-file representation completed in approximately 1.3 seconds with similar memory usage to the 1 MiB case.
The proposal is primarily about build scalability. In the optimized cases measured, the compiler already lowered the existing byte-array literal to static storage, so there was no meaningful runtime memory difference between the two representations. This does not imply that every configuration avoids materializing an array in writable memory. The object-file representation provides a non-owning view of linked storage rather than constructing a Swift Array value.
Proposed solution
This proposal adds an explicit representation parameter to Resource.embedInCode:
.target(
name: "Assets",
resources: [
.embedInCode(
"payload.bin",
representation: .objectFile
),
],
swiftSettings: [
.enableExperimentalFeature("Lifetimes"),
]
)
When .objectFile is selected, SwiftPM generates an interface equivalent to:
struct PackageResources {
static var payload_bin: RawSpan {
// Immutable view into the linked image.
}
}
The generated C source defines a constant byte array initialized with #embed. The target C compiler determines its object-file representation, and the generated Swift accessor creates a RawSpan over the linked bytes. Accessing the resource does not allocate or copy its contents.
RawSpan represents the resource as untyped bytes without imposing an element type. Consumers can create a typed view, such as Span<UInt8>, when that is appropriate for their use case.
The existing spelling remains unchanged:
.embedInCode("payload.bin")
It continues to generate an [UInt8]. The same behavior can be requested explicitly:
.embedInCode(
"payload.bin",
representation: .byteArray
)
Detailed design
Package manifest API
The following API is added to PackageDescription:
extension Resource {
/// The representation used for a resource embedded in executable code.
public enum EmbeddedInCodeRepresentation: Sendable {
/// Generate a Swift source file containing the resource as a
/// byte-array literal.
case byteArray
/// Embed the resource in a native object file and expose it as an
/// immutable `RawSpan` backed directly by the linked image.
case objectFile
}
/// Applies the embed rule to a resource using the requested representation.
///
/// The `objectFile` representation avoids generating and compiling a Swift
/// byte-array literal. Package Manager instead compiles a C `#embed`
/// declaration with the target C compiler and generates an immutable `RawSpan`
/// accessor into the linked image. Targets using this representation must
/// explicitly enable Swift's experimental `Lifetimes` feature.
///
/// - Parameters:
/// - path: The path for a resource.
/// - representation: The representation generated for the resource.
/// - Returns: A `Resource` instance.
public static func embedInCode(
_ path: String,
representation: EmbeddedInCodeRepresentation
) -> Resource
}
The API is gated by the Swift tools version in which it is introduced.
Generated interface
For .byteArray, SwiftPM preserves the existing generated interface:
static let payload_bin: [UInt8]
For .objectFile, it generates:
static var payload_bin: RawSpan
The C source defines an immutable pointer to the embedded byte array. The Swift accessor loads that pointer through a private binding to an implementation-defined symbol and creates an immortal span over the pointed-to storage. For a 1,024-byte resource, the generated Swift is equivalent to:
// Both the pointer and its statically embedded bytes are immutable.
@_silgen_name("swiftpm_resource_Assets_payload_bin_data")
nonisolated(unsafe) private let _payload: UnsafeRawPointer
struct PackageResources {
static var payload_bin: RawSpan {
@_lifetime(immortal)
get {
let span = unsafe RawSpan(
_unsafeStart: _payload,
byteCount: 1024
)
return unsafe _overrideLifetime(span, copying: ())
}
}
}
The resource size is substituted as a literal when the accessor is generated. The pointer and byte storage have static lifetime. The private binding and lifetime annotations are implementation details, not additional APIs that package authors need to use.
An empty resource receives addressable placeholder storage in the object file, but its generated RawSpan has a byte count of zero.
Lifetimes requirement
The prototype uses experimental lifetime annotations to express that the returned RawSpan has an immortal lifetime. This requirement is about the generated accessor, not a claim that every use of RawSpan requires experimental features.
A target using .objectFile must explicitly enable Lifetimes:
swiftSettings: [
.enableExperimentalFeature("Lifetimes"),
]
SwiftPM does not enable the feature implicitly. Attempting to use object-file embedding without the opt-in produces a diagnostic explaining the required setting.
This requirement can be removed once the lifetime functionality used by the accessor is available without experimental opt-in.
Build behavior
The prototype implements object-file embedding through Swift Build.
The accessor-generation task produces the shared Swift accessor file and a small C source file for each object-file resource. The C source is equivalent to:
static const unsigned char resource_bytes[] = {
#embed "embedded_resource.bin" if_empty(0)
};
__attribute__((visibility("hidden")))
const unsigned char *const swiftpm_resource_Assets_payload_bin_data = resource_bytes;
The actual generated source also checks for #embed support and suppresses Clang's C23 extension warning locally, allowing a supporting compiler to use the directive without changing the language mode of the target's other C sources.
Swift Build stages each resource next to its generated C file under a safe, generated basename. This handles resource paths containing characters that cannot be represented directly in a #embed header name. Staging is a build-time file copy; it does not introduce a runtime copy.
The generated C files enter Swift Build's ordinary C compilation pipeline. The target C compiler selects the architecture and object format, and the resulting objects are linked alongside the compiled Swift objects.
The resource symbols are namespaced using the effective module name and resource name. The backing byte array is local to its C translation unit; the pointer symbol provides the link between the separately compiled C and Swift sources. These symbols are implementation details rather than part of the package's public interface.
The generated accessor contains an undefined reference to the resource pointer symbol. When the target is packaged as a static library, this reference causes the linker to extract the corresponding resource object from the archive.
Resource changes participate in normal dependency tracking. Changing a resource, including changing its size, regenerates the accessor when necessary and rebuilds the affected resource object.
Tool selection and cross-compilation
The implementation uses the target C compiler selected by the existing build configuration. There is no objcopy lookup, assembly seed, or separate mapping from Swift target triples to object-file formats.
The compiler must support #embed. The generated source diagnoses a compiler that does not support the directive rather than silently reverting to a Swift byte-array literal.
Cross-compilation follows the normal C compilation configuration for the target, including its architecture and SDK settings.
Supported object formats and validation
The same generated C source is intended to support:
- Mach-O
- ELF
- COFF/Windows
- WebAssembly/WASI
The local proof of concept compiled empty and 1 MiB resources for each of these formats, including ARM ELF and multiple COFF architectures. Additional checks linked ELF and COFF outputs and linked and instantiated a Wasm module, verifying the resource bytes in linear memory.
Native macOS checks verified Swift RawSpan access through a static archive with dead stripping at both -Onone and -O. Package-level tests cover resource access, empty resources, static-library linking, same-size and different-size resource updates, the missing-Lifetimes diagnostic, and preservation of the existing byte-array behavior.
These cross-format checks validate the embedding mechanism, not complete SwiftPM builds and Swift execution on every target. End-to-end package validation on Linux, Windows, and WASI remains necessary before claiming full platform support.
The interface is immutable, but physical memory protection and image loading follow the target's normal behavior. In particular, Wasm resources occupy linear memory and participate in normal data-segment initialization; they are not read-only custom sections. The no-copy guarantee applies to accessing the resource through the generated accessor, not to platform startup or loading the executable.
The prototype currently requires SwiftPM's Swift Build backend. When the llbuild-based backend is selected, SwiftPM diagnoses the unsupported representation rather than silently changing its semantics.
Security
This proposal does not execute resource contents or package-provided build tools. Resource objects are generated using the target C compiler selected by the build configuration.
The generated interface exposes immutable storage. The implementation-defined resource symbol is not part of the package's public Swift interface.
As with existing embedded resources, placing a resource in the executable does not provide confidentiality. Its contents remain inspectable in the resulting binary.
Impact on existing packages
This is an additive API gated by a new Swift tools version.
Existing uses of:
.embedInCode("payload.bin")
retain their current behavior and continue to produce an [UInt8].
Packages only receive the new RawSpan interface when they explicitly request .objectFile. Those targets must also opt in to the experimental Lifetimes feature while that requirement remains necessary.
Packages selecting .objectFile will initially require the Swift Build backend and a target C compiler with #embed support.
Alternatives considered
Continue generating byte-array literals
The current representation is simple and portable, but its build time and memory consumption grow poorly with resource size. It becomes impractical for resources that are otherwise reasonable to include in an executable.
Preserve the array interface with different backing storage
Another direction would be to preserve the existing [UInt8] interface while generating the equivalent static array storage without a large Swift source literal. This could make the improvement an implementation change rather than requiring a new representation option.
Constructing an array through the ordinary public buffer initializer copies the bytes. Constructing native array storage directly instead requires the array/object header to precede the elements and involves private layout and runtime details.
A local experiment initialized static array storage from Swift and verified shared payload addresses and copy-on-write behavior on the tested macOS runtime. However, it depended on private runtime APIs and required writable storage for header initialization. That is not yet a portable replacement for the current implementation. Compiler or standard-library support could make preserving the array interface a viable alternative without encoding those private details in the build system.
Changing the existing generated property to RawSpan would be source-incompatible. The explicit representation option avoids changing existing packages while exposing a non-owning view of the embedded bytes.
Expose Span<UInt8>
The initial version proposed Span<UInt8>, but the revised interface uses RawSpan, which represents a binary resource without imposing an element type. Consumers can form a typed view when needed.
Expose an unsafe pointer or buffer pointer
The generated property could return an UnsafeBufferPointer<UInt8> or a pointer-and-length pair. This avoids the lifetime annotations used by the prototype, but exposes an unsafe interface for storage whose lifetime and size are already known to the generator.
RawSpan provides a lifetime-aware, bounds-checked interface over the same bytes.
Generate objects with objcopy or assembly
The initial prototype used target-specific object-file construction through objcopy and assembly. That required handling format-specific section names and other details separately for Mach-O, ELF, COFF, and Wasm.
Generating C with #embed delegates those choices to the target compiler and allows the build system to use its existing C compilation pipeline. This is the approach used by the revised prototype.
Implement this using a build-tool plugin
A plugin-based implementation is worth exploring, but it would require extending how SwiftPM integrates plugin outputs into the build.
Avoiding the large Swift array literal means generating something else. A C file using #embed is now a viable direction with mixed-language targets, but accepting plugin-generated C into a Swift target still requires additional work. Alternatively, a plugin could generate an object file directly, but SwiftPM would need to accept that output and arrange for it to be linked, including handling the appropriate target architectures.
This proposal aims to make an existing SwiftPM feature scale more reliably. Ideally, that could be an internal implementation change with no new public API. However, preserving [UInt8] without copying the payload involves the array-storage considerations described above, while changing the existing accessor to RawSpan would be source-breaking. The explicit representation option allows the new implementation without changing existing packages.
Extending the plugin system could provide a more general solution and allow iteration outside the toolchain, but it is not currently a straightforward replacement for this change.