Policy on hiding implementation details from Swift module interface

Hi all.

I continue to work on bringing about this vision for a stable @_implementationOnly via abstract type layouts, but I've reached a point where I'd like to discuss the direction.

Specifically, I want to discuss how much information we are willing to hide, potentially at the cost of diagnostics quality. Consider the following:

Library Module:

public class PublicClass {
  public init() {}

  internal func internalMethod() {}
}

Client Module:

import Library

func test(_ value: PublicClass) {
  value.internalMethod()
}

Today, if you build Client against Library as normal, you'll get the following error:

/Users/nuriamari/local/git/swift-project/access-control-diagnostic-example/Client.swift:4:9: error: 'internalMethod' is inaccessible due to 'internal' protection level
2 |
3 | func test(_ value: PublicClass) {
4 |   value.internalMethod()
  |         `- error: 'internalMethod' is inaccessible due to 'internal' protection level
5 | }
6 |

However, if we first produce a .swiftinterface file from Library, we get the following: (private API is dropped):

// swift-interface-format-version: 1.0
// swift-compiler-version: Swift version 6.5-dev (LLVM 270e638cf7ca80b, Swift 8b4449a3749e274)
// swift-module-flags: -module-name Library -language-mode 6 -enable-library-evolution
// swift-module-flags-ignorable:  -formal-cxx-interoperability-mode=off -interface-compiler-version 6.5
import Swift
import _Concurrency
import _StringProcessing
import _SwiftConcurrencyShims
public class PublicClass {
  public init()
  @objc deinit
}

If we then convert this back into a .swiftmodule file and build Client again, we get the following less helpful error:

/Users/nuriamari/local/git/swift-project/access-control-diagnostic-example/Client.swift:4:9: error: value of type 'PublicClass' has no member 'internalMethod'
2 |
3 | func test(_ value: PublicClass) {
4 |   value.internalMethod()
  |         `- error: value of type 'PublicClass' has no member 'internalMethod'
5 | }
6 |

Are we happy with the second degraded diagnostic? Personally I am, but I'd like to hear other's thoughts.

I would like to bring .swiftmodule files closer to .swiftinterface files via a change similar to @tshortli's experimental change, to exclude module internal declarations from .swiftmodule files: Frontend: Introduce `-experimental-serialize-external-decls-only` option by tshortli · Pull Request #68166 · swiftlang/swift · GitHub

This comes into play with @_implementationOnly types. Consider I have:

@_implementationOnly import HiddenModule


public struct PublicStruct {
    internal func internalMethod(_ foo: HiddenModule.HiddenType) {}
}

Today clients of this module that don't have access to HiddenModule will drop internalMethod during de-serialization, as they can't resolve its entire signature, and attempts to call it will result in the degraded .swiftinterface style diagnostics above.

Are we ok with this? I suggest we should be, based on the precedent set by .swiftinterface files.

If we wish to preserve the same diagnostics, we could instead serialize unavailable stub declarations that carry only a name, and exist solely to differentiate between an API which does not exist, and one which is not accessible outside the module.

cc @tshortli @Slava_Pestov @Xi_Ge

3 Likes

IMO, as far as Client is concerned, the diagnostic "value of type 'PublicClass' has no member 'internalMethod'" is true. The fact that it can be diagnosed better is a sign that the compiled .swiftmodule contains more information than it should.

This still wouldn't be ideal because it would cause caching build systems to rebuild more than necessary (as they do today). If module A depends on module B and module B makes a change to the name of an internal method, that's going to change the content of B.swiftmodule and tell the build system that A needs to be rebuilt, even though it doesn't need to.

This sounds similar to what .swiftsourceinfo files were created for; to provide file/line number information in diagnostics coming from dependencies. Could that information be added to those files instead? Then build systems can choose: if they want the good diagnostics, they can make those .swiftsourceinfo files visible to downstream compilations at the cost of potentially more rebuilds, and build systems that prioritize minimal rebuilds can omit those files.

1 Like

Hm, from the perspective of developer experience when building libraries and clients that belong to the same package/project, I think the diagnostic degradation is unfortunate. It's nice for the compiler to recognize your mistake more directly if what happened is that you forgot that the declaration wasn't made public yet. The thing that justifies the behavior of -experimental-serialize-external-delcs-only is that it is designed to be used when producing a binary module for a library evolution enabled module and that binary module is going to be distributed in an SDK. In that scenario, the clients that might load the emitted module are typically never from the same project. That said, if the tradeoff for this diagnostic degradation is that we get true dependency hiding for non-resilient builds it's probably worth it.

As an aside, this is all going to be sensitive to testability, right? When a binary module might be used by testable clients, we can't actually hide its internal declarations or dependencies, otherwise it would be source breaking for clients with @testable imports.

2 Likes

Yes, just chiming in that I've benefitted from this diagnostic specificity.

2 Likes

If leveraging .swiftsourceinfo isn't feasible, I wonder if having an .internal.swiftmodule sidecar would be a viable option. Kind of like how .private.swiftinterface and .package.swiftinterface files exist for clients who need SPIs/package decls that you don't want to distribute as part of the public API.

I'm sympathetic to users who have benefited from this diagnostic and think it would be fine to give users the ability to get that if they want it, but I also want to make sure we don't over-index on it and that we give folks who care more about build performance a way to achieve those goals too. For our builds that involve hundreds of modules in deep dependency graphs, the time that could be saved by minimizing unnecessary rebuilds far exceeds the amount of time lost by trying to hunt down a mistaken API usage.

6 Likes

I hadn't thought much about that, but yes. In that case though, I think all information hiding goes out the window. For example, I think that a module compiled with -enable-testing should declare non optional dependencies on all imported modules, even @_implementationOnly imported modules, since those modules may be required for the client to interact with private APIs.

I wonder if having an .internal.swiftmodule sidecar would be a viable option. Kind of like how .private.swiftinterface and .package.swiftinterface files exist for clients who need SPIs/package decls that you don't want to distribute as part of the public API.

I think this idea seems more straightforward, the format is already designed to encode the information. I would say we might even go one step further, and just produce the same one existing .swiftmodule file that controlled by flag either includes private APIs or not. Let's say the flag is called -include-private-decls. This flag would be implied by -enable-testing, force inclusion of private declarations, and also make private module dependencies non optional. That way users that want the more descriptive diagnostics still have that option. I'm also happy for the default to be more descriptive diagnostics, and hiding private decls to be an opt in option.

Part of my motivation for this is that current the module serializer is not consistent about which declarations it serializes and which module dependencies it makes optional. We currently serialize private declarations that can refer to @_implementationOnly imported types, which the client may not be able to resolve. To make this work, the de-serializer has recovery mechanisms where instead of emitting a fatal error when it can't resolve @_implementationOnly types, we drop parts of the AST, often in problematic ways (leads to miscompilation).

I think with this design we can adopt a more hard line stance: if we serialize a decl into a .swiftmodule we expect the client to be able to load it entirely, given the depended upon modules we have marked non-optional. This will simplify the design and make it less error prone, as we can drop some of the recovery mechanisms.

Remember that the primary motivation for swiftinterface files is to ship closed-source libraries. For that use case, revealing internal method names could be a secrecy leak for the publisher. (For Objective-C this wasn't as much of a thing because selectors were always present in the binary regardless, but Swift methods are often strippable.)

3 Likes

I'd like to split this into two separate questions, because I think the answers are different.

What belongs in the binary .swiftmodule?

I think the binary module should carry only what clients need to compile against it, and keeping it lean is worth it. Where it's safe, I'm fine with dropping declarations that can't be reached from what the module exports.

One consumer to watch is LLDB. It uses the .swiftmodule to rebuild types, including non-public ones. If the module is lean, the debugger needs another way to get that information, or debug builds need different behavior.

How do we keep the diagnostics?

I agree with @tshortli and @xwu that we shouldn't regress "inaccessible due to 'internal' protection level" into "has no member." In the same-project or same-package case, that precision helps people. I think .swiftsourceinfo is the right place to recover it, I'm not a big fan of adding another .internal.swiftmodule module since the module loading story for Swift is already too complicated. .swiftsourceinfo is the right tool because:

  • It already exists to support the developer experience of a module's clients without affecting compilation.
  • It's a local, non-distributed file: it goes in the Project/ directory and doesn't ship with a framework.

This needs some work, though. Today, swiftsourceinfo file contains only mapping from USR to source locations. We need to empower this file, so it could help compiler diagnostics in a more integrated and deeper way. For instance, we could potentially teach the .swiftsourceinfo file to:

  1. Emit entries for internal declarations (name, kind, access level, location), keyed by the parent type plus the member name, so the file can answer the questions a failed lookup asks.
  2. When member lookup fails against a module that has sourceinfo, check it before giving up. If there's a match, emit the inaccessibility diagnostic and point to the declaration's source location.
2 Likes

I didn't comment on the swiftsourceinfo approach earlier, but I do want to share dissenting opinion about it being the right way to avoid regressing diagnostics. To me, trying to work backward to diagnose that some internal declaration might have matched had it been visible sounds both complex and seems impossible to do in a generally correct way. I think the only way we can be sure the compiler would have picked that declaration is if it is part of the AST when the compiler does name lookup and overload resolution. Without that, it's unclear to me how to avoid both under-diagnosing and over-diagnosing potential matches from swiftsourceinfo, where all we have is the declaration's name and a bit of other metadata perhaps. Both name lookup and overload resolution have all kinds of rules that are just not going to be re-implementable in this scheme and I think an attempt to replicate prior diagnostics is unlikely to be able to recover the experience accurately in many cases. It seems likely to me that it would result in suggesting internal overloads that wouldn't match at all based on types, had the compiler had the chance to see them in the AST.

I think if we're going down the path of removing these decls from serialized modules then I'd rather we just accept the regression and don't attempt to solve it by adding a potentially poor approximation.

3 Likes

Ultimately I will attempt to implement whatever the community builds consensus on, but I want to echo some of @tshortli's sentiment.

I think ensuring that these two parallel lookup paths through either the .swiftmodule representation or the .sourceinfo representation are equivalent will be difficult to test and maintain.

That is why I suggest to have one .swiftmodule format and one file (no .internal.swiftmodule). If we wish to support two configurations, one where all private decls are hidden, and another where they are not, I think we should do our best to keep divergences between these two "codepaths" limited.

I don't really see what the alternatives buy us. We will still have two codepaths to test and maintain, but now they also differ in format and or loading behavior.

We also already need a .swiftmodule format that preserves private decls to support -enable-testing, so I would suggest that using .swiftsourceinfo files, would still require two versions of .swiftmodule anyways.

You might be saying this already, but I also want to clarify that I don't see the testability requirement as implying that there are multiple swiftmodule formats or multiple swiftmodules produced by the build system for the same module. If the build system determines that a module must support testability, then I think the swiftmodule that is produced during the build should look largely like it does today; full type and dependency information should be present in the main swiftmodule for non-public declarations. When there is no testability requirement, though, then the compiler is able to start emitting the new abstract representations of declarations that are hidden. Either way, there's just one "format" but in the testability case there just won't be any abstract representations encoded in the swiftmodule.

2 Likes

Right I agree.

We already have to support emitting .swiftmodule files in two configurations. One for normal module emission, and one for -enable-testing. I am suggesting that normal emission mode should now exclude private decls, and that if we want to preserve diagnostic quality, we introduce a third "fidelity level" that preserves private decls, but doesn't have the unwanted effects of enable-testing. It can also be that normal emission stays as it is today, and the new fidelity mode be the one that excludes private decls.

My point is that I think we should stick with the one existing .swiftmodule file in the current format, emitted with various degrees of detail, rather than a sidecar. I don't see the benefit to the sidecar, especially if it is in a different format. If we are going to introduce a side car and expand the .sourceinfo format, I think we need a compelling reason to do it.

3 Likes