That might be the right way to think about it, but I don't think most Swift programmers will think about it that way. This has led to confusion before. And not only does the grammar get this subtlety wrong — the syntax highlighters for the Swift Playgrounds app and Visual Studio Code also get it wrong, highlighting this expression incorrectly.
Personally, I'd be in favor of adding a warning to expressions like -13.incremented() that can be suppressed by adding parentheses around a subexpression e.g. -(13.incremented()) or (-13).incremented().