tvOS xibs do not support target device type "iphone"

I have a swift package which has tvOS and iOS xibs. For some reason i cant use separate targets. Whenever i try to build SPM this error comes.

tvOS xibs do not support target device type "iphone".

PS: Dont want to move xibs to a separate package as well.

Then you can't use xibs. Either create UIKit views programmatically or use SwiftUI (I'd do the latter).