Change Xcode SwiftPM checkout path

I have a SPM dependency that relies on a local binary, that local binary is just referenced in the manifest, but not published on git (for plenty of reason).
To download this binary I use a swift command plugin (command type, because is the only one that can write in the package and access to the network).
After dowloaded the binary the spm project builds fine.
Unfortunately I can't do the same when the package is added as a dependency in an Xcode project.
Once Xcode starts resolving the dependencies fails, due to the missing local package, but it can be resolved manually by going into:
DerivedData/App-random_string/SourcePackages/chekouts/mylibrary/
and launch the plugin from here.
This can't be done on CI, because I don't know in advance which will be the path of the spm checkouts, the random_string changes.
I've tried by launching before xcodebuild -showBuildSetting | grep BUILD_DIR, but still Xcode tries to resolve the packages and fails without showing any output.
Is there a build settings in Xcode to make the checkout of packages in a specific directory?
Or, doesn't exist a way in the Package manifest using ProcessInfo to understand if the package is resolved by Xcode or from swift package?

.target(
        name: "LocalBinaryWrapper",
        dependencies: [
            "LocalBinary"
        ]
    ),
    .binaryTarget(
        name: "LocalBinary",
        path: "./LocalBinary/LocalBinary.xcframework"
    ),

In our CI using jenkins we set derived data as subfolder of the cloned project, that way you know always where derived data is.
https://docs.fastlane.tools/actions/setup_jenkins/
This is the action we use with Fastlane to set it. You can check how fastlane does it if you want to replicate it yourself if you do not have that setup.

I would recommend using binaryTarget(name:url:checksum:) | Apple Developer Documentation instead and let SwiftPM handle the download if at all possible.