Swift is open-source but is it patent-free as well?

In what context are you talking about? Are you thinking about lifting pieces of code of the Swift compiler into your own work? Because with that, I think you might run into issues with the license and any patents used in the software.

Or are you talking about using Swift on your server/app/IoT device? Because if they could do this, and did, I could almost guarantee a tremendous backlash that would almost certainly cause Swift adoption to drop to next to nothing. So I wouldn't worry too much about it.