Well, I've solved it with help of your hint, but it was not that straightforward ...
I looked into the path to Xcode but I could find any other version available. I uninstalled Xcode and re-installed, same issue.
I then renamed the /usr/bin/swift exec to see if that helped, but strangely no help here as well.
I deleted in Xcode the Swift 3.1.1 entries
After entering swift --version the reply was then:
swiftenv: version `3.1.1' (set by /Users/thewall/.swiftenv/version) is not installed
That hinted me to that path and there was a file named version (with only 3.1.1 as text in it). I renamed that file and issued swift --version again:
Apple Swift version 4.1.2 (swiftlang-902.0.54 clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
EUREKA!
So, it boiled down to the fact that there is a version file in /Users//.swiftenv/ called version and that holds the info on what version it has.