Unable to compile universal binary for command line swift package

I have been using swift build -c release --arch arm64 --arch x86_64 to create a universal binary for a command line swift package to run the same binary on both apple silicon and intel based Macs. However, when I run the same command on M1 Max machine I get the following error and the build fails.

error: remark: Incremental compilation has been disabled: it is not compatible with whole module optimization

Not sure what changed.

I also maintain a universal binary CLI tool using swift package on my M1 machine. And it works fine for me.

I encounter this hint too, but just a warning not an error

remark: Incremental compilation has been disabled: it is not compatible with whole module optimization

Maybe you could try delete .build folder first and put the full log here to see if anyone can help.