MKMapView/UIViewRepresentable Calls UIViewUpdate Unnecessarily

Does anyone know how to prevent UIViewRepresentable from calling updateUIView?

I have a bunch of annotations in a MKMapView and updateUIView is where I remove/add the annotations (there doesn't seem to be another place to remove/add them). Every time any @State values in the MapView (UIViewRepresentation), parent struct and/or sibling struct changes, the updateUIView gets called even tho nothing changed for the UIViewRepresentable.

This causes the all the annotations to be refreshed.