Surely you don't mean that we should create a whole new literal syntax entirely as a shorthand for UInt(ascii:)?
The use of a character literal, as I see it, is in being able to represent to the reader the assertion that I expect a certain Unicode sequence to be a single extended grapheme cluster. This is not merely a model of text being promoted by Swift (although that is salient here because we're talking about proposals for Swift evolution, which should be compatible with the direction of Swift) but also one promoted by Unicode, and therefore intended for much wider adoption. The reason it would be such an important feature is precisely that what's a single Unicode extended grapheme cluster can be non-obvious yet of huge importance when working with Unicode (for example, in text editing, the caret should advance one extended grapheme cluster at a time); having a syntax to assert that something is one such cluster and not a string of them is highly relevant for a Unicode-correct language.
That Swift allows essentially arbitrarily many types to be expressible by the same kind of literal makes possible the use case of restricting the content of a character literal to those that can be represented by a single code point--which is certainly a nice benefit.
However, I certainly don't find the use case of shorthands for UInt(ascii:) compelling. And, yes, I have actually written code that requires such APIs. (In fact, I've thought long enough about this in the past that I renamed a C string API in SE-0134.)
Moreover, some of the examples above show likelihood of active harm in the proposed design. Specifically I would argue that it would be a nonstarter for the proposal if the following behavior didn't hold:
'1' + '1' // "11"
'1' - '1' // Error: binary operator '-' cannot be applied to two 'Character' operands