# Option to make Connect Proxy work with HTTP traffic?

**URL:** <https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737>\
**Category:** SwiftNIO\
**Created:** [June 18, 2021, 1:56pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737 "2021-06-18T13:56:51Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 22, 2021, 1:10pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/23 "2021-06-22T13:10:58Z")

</div>

You mean the `GlueHandler` from CONNECT proxy example? No, I thought this is needed for the HTTPS variant only.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 22, 2021, 1:11pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/24 "2021-06-22T13:11:45Z")

</div>

Nope, the `GlueHandler` is perhaps the most important part of the example: it's how the data gets from the inbound connection (where your proxy handler lives) to the outbound connection you just created. Without them, no data can flow.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 22, 2021, 1:13pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/25 "2021-06-22T13:13:12Z")

</div>

Big oops on my part then. I thought even since you mentioned previously that the HTTP should be much simpler, that lot of the things from the connect proxy aren't needed.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 22, 2021, 4:26pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/26 "2021-06-22T16:26:07Z")

</div>

I think I am still confused as to what the intended sequence is. I have tried having another detailed look at the Connext Proxy example but I am not sure why the `glue` method is called in two places?

Also the `handleInitialMessage` method which parses the `uri` from the head is called once, just for the connection to be made in the `connectTo` method but above in the steps, you said that I should repeat the steps 1-5 for each request. But it appears this only happens once.

**Conceptually what I need to do is this** (?):

_Initial request comes to proxy_ -\> Parse the head from the request -\> Save the head (with modified uri) & `connectTo` to the host on port 80 -\> _connection succeeded_ -\> Add the `GlueHandler` -\> write the cached/buffered head with the `context.fireChannelRead(self.wrapInboundOut(.head(head)))` –\> Done?

From the Connect proxy it appears that after it is "glued", there is nothing else to do?

I have tried creating another handler based on the `ConnectHandler` from Connect proxy but a lot of stuff there is quite different with the `awaitingEnd` and `awaitingConnection` + manipulating `pendingBytes` so I got lost.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 22, 2021, 5:36pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/27 "2021-06-22T17:36:05Z")

</div>

> [@nemecek\_f](#):
>
> I think I am still confused as to what the intended sequence is. I have tried having another detailed look at the Connext Proxy example but I am not sure why the `glue` method is called in two places?

`glue` is called in two places because the state machine needs to wait for two things to happen, and they could happen in either order:

1. Receive the `.end` of the `CONNECT` request
2. Receive a completed connection to the remote server

In your case you do not need to wait for (1) (as you are passing the request you received on), so you only need to call `glue` in the (2) case.

> [@nemecek\_f](#):
>
> Also the `handleInitialMessage` method which parses the `uri` from the head is called once, just for the connection to be made in the `connectTo` method but above in the steps, you said that I should repeat the steps 1-5 for each request. But it appears this only happens once.

It only happens once _per request_. HTTP/1.1 uses connection keep-alive, where a connection will be re-used for subsequent requests to the same host. Your code should tolerate that possibility. Thus:

> [@nemecek\_f](#):
>
> From the Connect proxy it appears that after it is "glued", there is nothing else to do?

this is incorrect, as you are responsible to continually rewrite the HTTP requests as they come in.

> [@nemecek\_f](#):
>
> **Conceptually what I need to do is this** (?):
> 
> _Initial request comes to proxy_ -\> Parse the head from the request -\> Save the head (with modified uri) & `connectTo` to the host on port 80 -\> _connection succeeded_ -\> Add the `GlueHandler` -\> write the cached/buffered head with the `context.fireChannelRead(self.wrapInboundOut(.head(head)))` –\> Done?

If you want to tear the connection down each time, this is absolutely fine. You do have some extension points:

1. You could to pass on the body, if you have one.
2. You could handle keep-alive.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 23, 2021, 6:11am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/28 "2021-06-23T06:11:18Z")

</div>

That clarifies some things.. So after the `connectTo` succeeded, I need to `glue` the channels which makes sense. But at what point am I supposed to send along the buffered head from the initial request?

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 23, 2021, 10:50am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/29 "2021-06-23T10:50:58Z")

</div>

After the glue handlers have been added.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 23, 2021, 11:34am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/30 "2021-06-23T11:34:01Z")

</div>

Right, I meant more like what is the correct sequence.

I have tried calling it manually with completion handler which gets called after the `glue` method is done but I am getting a fatal error:

> Fatal error: tried to decode as type IOData but found HTTPPart\<HTTPRequestHead, ByteBuffer\> with contents other(NIOHTTP1.HTTPPart\<NIOHTTP1.HTTPRequestHead, NIO.ByteBuffer\>.head(HTTPRequestHead

The `glue` method from the original Connect handler starts with these lines, which I commented out, because my handler is a bit different and I don't have access to these.

```swift
        // Ok, upgrade has completed! We now need to begin the upgrade process.
        // First, send the 200 message.
        // This content-length header is MUST NOT, but we need to workaround NIO's insistence that we set one.
        let headers = HTTPHeaders([("Content-Length", "0")])
        let head = HTTPResponseHead(version: .init(major: 1, minor: 1), status: .ok, headers: headers)
        context.write(self.wrapOutboundOut(.head(head)), promise: nil)
        context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)

```

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 23, 2021, 4:01pm UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/31 "2021-06-23T16:01:43Z")

</div>

Ah, yes, you need HTTP encoders and decoders in the other pipeline.

Concretely, the problem you have is that you're sending HTTP structured data into the new channel you just created. That's fine, there's nothing wrong with that, but we need to re-encode them to binary data before we send them on.

In the `channelInitializer` for the `ClientBootstrap` you create, add a `HTTPRequestEncoder` and a `HTTPResponseDecoder`.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 24, 2021, 8:39am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/32 "2021-06-24T08:39:13Z")

</div>

That makes sense, are you sure I need to add the `HTTPResponseDecoder` because this does not work, since this does not conform to the ChannelHandler protocol.

When I added just the `HTTPRequestEncoder`, I am getting slightly different crash:

> NIO/NIOAny.swift:200: Fatal error: tried to decode as type HTTPPart\<HTTPRequestHead, IOData\> but found HTTPPart\<HTTPRequestHead, ByteBuffer\> with contents other(NIOHTTP1.HTTPPart\<NIOHTTP1.HTTPRequestHead, NIO.ByteBuffer\>.head(HTTPRequestHead

This looks like I am closer, since the only difference is now `IOData` vs `ByteBuffer` in the generic parameters list.

I have added the encoder like this:

```swift
let channelFuture = ClientBootstrap(group: context.eventLoop)
            .channelInitializer { channel in
                channel.pipeline.addHandler(HTTPRequestEncoder())
            }
            .connect(host: host, port: port)

```

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 24, 2021, 11:01am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/33 "2021-06-24T11:01:18Z")

</div>

Just tried another approach:

```swift
let channelFuture = ClientBootstrap(group: context.eventLoop)
            .channelInitializer { channel in
                channel.pipeline.addHandler(ByteToMessageHandler(HTTPResponseDecoder(leftOverBytesStrategy: .forwardBytes))).flatMap {
                    channel.pipeline.addHandler(HTTPRequestEncoder())
                }
            }
            .connect(host: host, port: port)

```

But still the same `fatalError` about `IOData` x `ByteBuffer` mismatch

---

<div class="post-metadata">

**Author:** ![wonderfulsuccess](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/wonderfulsuccess/32/18397_2.png) [@wonderfulsuccess](https://forums.swift.org/u/wonderfulsuccess)\
**Post date:** [June 25, 2021, 2:50am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/34 "2021-06-25T02:50:44Z")

</div>

I am trying to build a http/https proxy with SwiftNIO , stucked a few days. The most amazing tutorial I have found is [https://www.alwaysrightinstitute.com/microexpress-nio2](https://www.alwaysrightinstitute.com/microexpress-nio2/)

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 28, 2021, 9:39am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/35 "2021-06-28T09:39:05Z")

</div>

I have tried all the possible combinations that I could but I still get the above mentioned fatal error.

How can I turn the `ByteBuffer` into `IOData`? This seems weird, since `IOData` is abstraction over `ByteBuffer` or `FileRegion` according to the docs.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 28, 2021, 10:06am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/36 "2021-06-28T10:06:48Z")

</div>

> [@nemecek\_f](#):
>
> That makes sense, are you sure I need to add the `HTTPResponseDecoder` because this does not work, since this does not conform to the ChannelHandler protocol.

`HTTPResponseDecoder` is wrapped in `ByteToMessageHandler`.

Concretely, we need to discuss what is happening here.

Your server channel is reading data from the network and decoding the data into `HTTPServerRequestPart`. The problem you have is that you need to transform these into `HTTPClientRequestPart` for the `HTTPRequestEncoder` to tolerate it (that's what the type mismatch you saw above was). You can do this my modifying your proxy handler to change its `InboundOut` type to `HTTPClientRequestPart`. In general this won't be a problem, but you will need to wrap each of your `.body` payloads in `.body(.byteBuffer())`.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 28, 2021, 10:26am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/37 "2021-06-28T10:26:48Z")

</div>

Regarding the `ByteToMessageHandler` I figured that out later and tried this:

```swift
let channelFuture = ClientBootstrap(group: context.eventLoop)
            .channelInitializer { channel in

                channel.pipeline.addHandler(ByteToMessageHandler(HTTPResponseDecoder(leftOverBytesStrategy: .forwardBytes))).flatMap {
                    channel.pipeline.addHandler(HTTPResponseEncoder())
                }
            }
            .connect(host: host, port: port)

```

So far I haven't implemented anything related to `.body`. This should happen once the state is `connected` right?

Or would it be enough to modify this guard:

```swift
guard case .head(var head) = self.unwrapInboundIn(data) else {
            context.fireChannelRead(data)
            return
        }

```

?

Since I don't think I need to touch the `body`, as I am only interested in the `head` for proxy connection.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 28, 2021, 10:48am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/38 "2021-06-28T10:48:12Z")

</div>

Yeah, you only need to touch the body to get the types to line up. Essentially your `guard` probably wants to be a `switch` to make sure you make the appropriate transformations.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 28, 2021, 11:14am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/39 "2021-06-28T11:14:57Z")

</div>

So after modifying the `InboundOut` to be `HTTPClientRequestPart`, I am no longer getting fatal errors. I see connect succeeded and then I am sending the buffered head, however nothing else happens after this point.

curl gets stuck on this output:

```swift
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET http://apple.com/ HTTP/1.1
> Host: apple.com
> User-Agent: curl/7.64.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 

```

I have implemented the wrapping of `.body` stuff, like this for now to quickly try it out:

```swift
guard case .head(var head) = self.unwrapInboundIn(data) else {
            
            let unwrapped = self.unwrapInboundIn(data)
            
            switch unwrapped {
            case .body(let buffer):
                context.fireChannelRead(self.wrapInboundOut(.body(.byteBuffer(buffer))))
            default:
                context.fireChannelRead(data)
                break
            }
            return
        }

```

It is not super pretty code, but I think the basic logic should work. If I don't get `head`, I try to get `.body` and wrap it if it succeeds. 🤔

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 28, 2021, 11:23am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/40 "2021-06-28T11:23:47Z")

</div>

Are you buffering the other data objects?

While we're waiting for the connection to succeed you may receive `.body` and `.end`. They all have to be buffered, and can be unbuffered together once you unbuffer the `.head`.

---

<div class="post-metadata">

**Author:** ![nemecek\_f](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/nemecek_f/32/18316_2.png) [@nemecek\_f](https://forums.swift.org/u/nemecek_f)\
**Post date:** [June 28, 2021, 11:32am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/41 "2021-06-28T11:32:29Z")

</div>

Oops. Can I just store them and once connection is finished send them out with the `fireChannelRead` or is more needed?

Do I need to wrap the `.end` in similar fashion like with `.body`?

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [June 28, 2021, 11:34am UTC](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737/42 "2021-06-28T11:34:44Z")

</div>

You can just store them. You do need to wrap the `.end` as well: it likely seems fairly foolish, but I `HTTPServerRequestPart.end` and `HTTPClientRequestPart.end` are technically different types.

[Previous page](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737.md?page=1)

[Next page](https://forums.swift.org/t/option-to-make-connect-proxy-work-with-http-traffic/49737.md?page=3)
