Yeah, this is expected behavior. The type inferred for an expression in a function builder does not affect the types inferred for the other expressions.
Unlike most expressions in Swift, function builders use something called one-way constraints for type inference (this is why a regular function call to buildBlock behaves differently wrt type inference). A one-way constraint means that type information can flow only in one direction, instead of the usual bi-directional type inference. So, in this case, the type information Conv2D<Float> will affect the overall type inferred for model, but not for the other expressions inside Sequential { ....
@Douglas_Gregor wrote up an explanation of one-way constraints and why they're used in his PR that added them: [Constraint solver] Introduce one-way binding constraints. by DougGregor · Pull Request #25983 · apple/swift · GitHub.