One thing that worries me is that async let and TaskGroup has different behaviours if the program exits the scope with unused task(s).
TaskGroup cancels its (unused) child tasks if its body exits by throwing, while async let cancels the (unused) child task regardless.
I really think that such a difference will lead to inconsistent feelings when using API as a whole. Further, it turns into something you need to remember on an API-by-API basis, which isn't great.
I think it'd be better if async let behaviour is more similar to task group in this direction as well. After all, it is advertised as a "sugar, but also more optimized" version of the task group.