Change Xcode SwiftPM checkout path

I have a SPM dependency that relies on a local binary, that local binary is just referenced in the manifest, but not published on git (for plenty of reason).
To download this binary I use a swift command plugin (command type, because is the only one that can write in the package and access to the network).
After dowloaded the binary the spm project builds fine.
Unfortunately I can't do the same when the package is added as a dependency in an Xcode project.
Once Xcode starts resolving the dependencies fails, due to the missing local package, but it can be resolved manually by going into:
DerivedData/App-random_string/SourcePackages/chekouts/mylibrary/
and launch the plugin from here.
This can't be done on CI, because I don't know in advance which will be the path of the spm checkouts, the random_string changes.
I've tried by launching before xcodebuild -showBuildSetting | grep BUILD_DIR, but still Xcode tries to resolve the packages and fails without showing any output.
Is there a build settings in Xcode to make the checkout of packages in a specific directory?
Or, doesn't exist a way in the Package manifest using ProcessInfo to understand if the package is resolved by Xcode or from swift package?

.target(
        name: "LocalBinaryWrapper",
        dependencies: [
            "LocalBinary"
        ]
    ),
    .binaryTarget(
        name: "LocalBinary",
        path: "./LocalBinary/LocalBinary.xcframework"
    ),