If I explicitly set the Swift language mode to v6 (in my Package.swift), I have to remove all upcoming feature flag settings, and then it doesn't work in version 5 because those features are necessary.
Build settings can be conditionalised but as far as I can tell that's only on build mode (debug vs release) or platform.
Is there no way to support both Swift 5 & 6 in one package?
Put all your Swift 6 stuff in the Package with the version 6 manifest, move everything else to the Swift versioned packages in Package@swift-5.<X>.swift files. Unless you really need to build using Swift 6 in 5 mode, which should never actually be necessary (modules built by the Swift 6 compiler are compatible regardless of mode).
My concern is more about building "Swift 6 mode" with Swift 5. There's a lot of feature/experimental flags and beyond the logistical nightmare of enumerating and enabling those, I'm not sure that actually equates to the same thing as actually building with Swift 6. I want to ensure someone using Swift 6 has all the features and correctness they'd expect from my package, without breaking compatibility with folks still using Swift 5.
If I do have to basically duplicate my Package.swift, that would be a bit disappointing - that's a bit of a maintenance annoyance and likely source of mistakes in future.
Using the v6 constant already means your package is only usable via 6.0 tools, doesn't it?
If the goal is one package that builds in 6 language mode if the client uses Swift 6 tools and 5 if they use Swift 5 tools, this should be achievable by staying on the 5 tools-version and utilizing the stringly typed initializer for the language mode and #if swift to conditionalize the manifest.
The one caveat here is that this won't work correctly if the consumer of your package uses different compiler versions for compiling the manifest vs. the package's code itself.
Yes, but one can use .version("6") instead to work around that.
In any case, am I approaching this wrong? It seems to me like I should want my package to build in full Swift 6 mode wherever possible (based on the user's toolchain version etc), and only use Swift 5 mode as a fallback. Is that not a good goal?
it might be considered a hot take, but I think you could
It seems like it's less "nice" for users, but I think if most packages did this, the entire ecosystem will upgrade to v6 faster.
(unrelated story of a similar case in JS: I originally come from JS land and Sindresorhus is a prominent open-sourcer in our community who started doing this for his 1000+ packages when nodeJS supported ESM. If he wouldn't have done this the entire community might still be on CJS. 
but I digress)