This is messier than you expect. The right answer depends a bit on exactly what you're trying to do.
Remember that you can only use a Swift framework if that framework was built with the exact same version of Swift as the one compiling your app. That means, in practice (for now), the same version of Xcode, and that in turn means you're better off (for now) building the frameworks as sub-projects of your main project (which is basically the solution that @brokaw suggested).
If, however, you really want to use pre-built frameworks (assuming you are ensuring Swift version compatibility), then you likely want to link them as *embedded frameworks. That is, you want to include those frameworks inside your built app. (Otherwise, you can link against them, but you would have to make sure they were already installed on the user's system in a standard location. This is not a direction you really want to go right now.)
To embed your frameworks, you shouldn't use the build settings directly, but go to the General tab of the project editor for your app target (select the project item in the navigator, then select the target in the editor pane, then select the General tab). Add your framework under "Embedded Binaries". It should then show up under "Linked Frameworks & Binaries" as well.
That much takes care of what you need to link your app, but it isn't enough to get your "import" statements to work. For that, you need to go to the build settings for your app target, and the "Frameworks Search Paths" in the Search Paths section. Add a project-relative path to your frameworks, if you don't see that Xcode has automatically added the correct path(s) already.
I believe that'll be enough to allow you to build your app.
Yes, that's a limitation of using pre-built frameworks. You don't get to specify which particular configuration of the framework build to use. If you need this, the only supported way is to add the framework projects as subprojects of your app project, which means that Xcode will re-build the frameworks whenever you change platforms (device or simulator) or configurations (debug or release) for your app project.