I've made it my goal to be able to contribute to the Swift compiler by the end of 2019.
It's an admirable goal too. I'd like to help you reach it however I can.
Is there a particular part of C++ that would be handy to know? Are there things about compiler development in particular I ought to study?
C++ is a notoriously dense language that can be a real pain to get a hold on. That said, LLVM-style projects have carved out their own subset of the language, and are very consistent about sticking to that subset. If you want to get a feel for the minutiae, the LLVM Coding Standards document is a great way to get a feel for the aesthetics. If you're looking for a technical start, including programming patterns, data structures, algorithms, and more, look to the LLVM Programmer's Manual.
I learn best by doing, so I'm sparse on book recommendations and such, but I think if you spend enough time with it that you can get up and working with LLVM-style structures pretty quickly.
I will say from personal experience that a formal compilers course or something won't be much help to you here. Resources like The Dragon Book, The Tiger Book, PFPL, etc. that form the foundation of these courses are good for grounding in theory, but they will not help you in practice. These books do not accurately capture the state of real-world compiler development. But if you're just looking for "what is a compiler and how do I put one together", they're each a pretty good answer.
Basically, don't worry about it. Compilers have this reputation for being dense black boxes that they earned over decades of arcana and implementation details leaking out into the broader programming world, or from overly-rigorous college courses leaving a bad taste in people's mouths. It's a terrible misconception that you have to be some kind of wizard to even approach one, and I hope projects like Swift, Rust, and Go can all help to actively challenge it.
Also I don't know how feasible is it to contribute code to Swift as a hobby? Does it take a huge amount of time to understand enough to make small contributions if you never worked on compilers before and aren't a C++ expert?
Just pick a problem and get going. Please feel free to reach out to me personally if you have any questions or need guidance or just wanna talk really! You can get ahold of me through private forum messages or on Twitter usually. My DMs are always open for Swift questions, no matter what.
Finally, a little shameless self-promotion. My interactions with LLVM these days are mostly improving its C API to support Swift bindings. @harlanhaskins has ported parts of LLVM's famous Kaleidoscope tutorial to Swift and has a series of blog posts about it. I'm hoping to update those blog posts and expand them to support LLVM 7.0 and the new JIT in the coming days/weeks/months.