Thank you for the response, its really encouraging.
Here is my test setup - I have 2 separate clones of the project, one I use for Xcode 15, another for 14, each time before building I'm changing command line tools to the respective one ( when testing with 15 - command line tools are set to 15 ) just in case. I do incremental build and adding/removing print("hello") in the same file in the same place, I do multiple builds with both Xcode to make sure its not accidental result, trying to wait a little before building so Xcode will complete the indexing (although I could not understand if its finished or not )) ) and I'm not doing anything with the laptop during test run.
Attaching results with screenshots, tldl I got 38 sec(Xcode 14) vs 45 sec (Xcode 15)
Next I will try to use Xcode's diagnostic options and will update here the result
