No, it should work fine. (I haven't tested this, but I'm pretty sure what I'm writing is correct. Did you test it for visionOS and got an error?)
The platforms: parameter is confusingly named. It’s not an allowlist of supported platforms, but rather a list of the respective minimum deployment target for each listed platform. If a platform is not mentioned in the list, it means that the package does not constrain the minimum deployment target beyond SwiftPM's per-platform default. In other words, adding .visionOS(.v1) to the list is the same as omitting it outright because visionOS 1.0 is the default min deployment target anyway.
As I understand it, the feature was designed in this way for exactly this purpose: when a new platform is added, all existing packages should support the new platform by default without having to make changes to their packages. Swift's availability annotations work the same way. You always have to include the asterisk, e.g. @available(iOS 17.0, *), meaning "this should be available on iOS 17.0+, and on all versions of all other platforms".
SwiftPM provides no API that would allow a package to exclude a certain platform completely (be it an Apple platform like visionOS, or Linux, or Windows, or …).
Ahh… interesting! That makes sense. I did find the test that seems to confirm that.[1]:
// default platforms will be auto-added during package build
let expectedDerivedPlatforms = defaultDerivedPlatforms.merging(expectedDeclaredPlatforms, uniquingKeysWith: { lhs, rhs in rhs })
I think maybe the documentation is a little ambiguous on this specific question[2]:
/// By default, Swift Package Manager assigns a predefined minimum deployment version for each
/// supported platforms unless you configure supported platforms using the
/// `platforms` API.
It's a little ambiguous whether defining any platform opts you out of the default behavior for other platforms… but it looks like the test confirms that platforms are not dropped.