So the unfortunate truth of the matter is that Swift Crypto does not give you the tools you need to properly implement chacha20-poly1305@openssh.com.
The spec for that cipher suite is stored in the OpenSSH repository. The reason that Swift Crypto's implementation is not suitable is not immediately obvious, but this sentence provides a clue:
The construction used is based on that proposed for TLS by Adam Langley in [3], but differs in the layout of data passed to the MAC and in the addition of encryption of the packet lengths.
You also have a comment here that points to the core issue:
// For ChaCha20-Poly1305, the length is encrypted along with the data, so there is no need to decrypt the first block separately.
So let's try to nail down the problem.
Swift Crypto provides ChaCha20-Poly1305 as an AEAD construction. An AEAD has two operations: seal and open. Seal takes a key, a nonce, a plaintext, and any additional authenticated data (AAD), and computes as its output a ciphertext and a tag. Open reverses the process: it takes a key, a nonce, a ciphertext, a tag, and the AAD, and will validate the tag and, assuming validation passes, return the plaintext.
Importantly, the whole point of an AEAD is that it is not possible to encrypt without integrity checking. In general, encryption without authentication is very bad and opens you up to a wide range of attacks, so AEAD constructions are designed to make it impossible to forget that authentication step. Putting this more simply: the ChaCha20-Poly1305 AEAD requires that you perform an integrity check over the ciphertext.
With that, let's look at the protocol. The first two paragraphs of the "Detailed Construction" section look like this:
The chacha20-poly1305@openssh.com cipher requires 512 bits of key material as output from the SSH key exchange. This forms two 256 bit keys (K_1 and K_2), used by two separate instances of chacha20. The first 256 bits constitute K_2 and the second 256 bits become K_1.
The instance keyed by K_1 is a stream cipher that is used only to encrypt the 4 byte packet length field. The second instance, keyed by K_2, is used in conjunction with poly1305 to build an AEAD (Authenticated Encryption with Associated Data) that is used to encrypt and authenticate the entire packet.
This reveals the crux of our issue. chacha20-poly1305@openssh.com performs ChaCha20 encryption/decryption twice, with two different keys. One instance is used without Poly1305 to encrypt the packet length field. The other is used to encrypt the payload, and also in conjunction with Poly1305 to authenticate the entire payload.
But Swift Crypto does not provide ChaCha20 as a standalone stream cipher, only as an AEAD mode. There is no API for doing encryption/decryption without integrity checking. However, the OpenSSH API requires this for the length: the integrity check of the packet length is done later.
Fundamentally, implementing this requires access to ChaCha20 and Poly1305 as standalone implementations, rather than as a combined AEAD, so that they can be combined in this unusual way. This is not a use-case that Swift Crypto is likely to support. We could consider adding it to Crypto Extras, but even that isn't a slam dunk.
In the short term, you could look for alternative implementations, or file a ticket against Swift Crypto. CryptoExtras also offers an AES block function construction, which you could use to implement AES counter mode.