Example of SNIHandler and Connect-Proxy

NghiaTranUIT · August 21, 2023, 9:44am

Hello everyone,

I'm trying to use SNIHandler in Connect-Proxy.

My goal:

Able to proxy HTTPS traffic as connect-proxy example has already been done
Able to get SNI from TLS Data (By using SNIHandler).

Here is my code:

    private func glue(_ peerChannel: Channel, context: ChannelHandlerContext) {
        self.logger.debug("Gluing together \(ObjectIdentifier(context.channel)) and \(ObjectIdentifier(peerChannel))")

        // Ok, upgrade has completed! We now need to begin the upgrade process.
        // First, send the 200 message.
        // This content-length header is MUST NOT, but we need to workaround NIO's insistence that we set one.
        let headers = HTTPHeaders([("Content-Length", "0")])
        let head = HTTPResponseHead(version: .init(major: 1, minor: 1), status: .ok, headers: headers)
        context.write(self.wrapOutboundOut(.head(head)), promise: nil)
        context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)

        // Now remove the HTTP encoder.
        self.removeEncoder(context: context)
        context.pipeline.removeHandler(self, promise: nil)

        // ✅ NEW LOGIC Here
        // try to add SNIHandler
        context.pipeline.addHandler(ByteToMessageHandler(SNIHandler(sniCompleteHandler: { result in

            print("SNI = \(result)")

            // try to glue again
            let (localGlue, peerGlue) = GlueHandler.matchedPair()
            return context.channel.pipeline.addHandler(localGlue).and(peerChannel.pipeline.addHandler(peerGlue))
                .map { (_, _) -> Void in
                    return ()
                }
        })))
        .whenComplete { _ in
            context.fireChannelReadComplete()
        }
    }

Result:

I can retrieve the SNI correctly, but the HTTPS connection just hangs.
I guess that SNIHandler consumes all data and doesn't pass through the GlueHandlers, so the socket just hangs.
Or maybe I'm doing wrong here

Hope anyone here points me in to correct direction. Thanks in advance.

lukasa · August 21, 2023, 11:15am

Hmm, the SNI handler should forward on the bytes when it is removed from the pipeline. Can you add a logging handler that you insert into the pipeline just before localGlue and print what you see?

NghiaTranUIT · August 21, 2023, 2:24pm

If you means print(context.pipeline) before the localGlue

2023-08-21T21:22:59+0700 info com.apple.nio-connect-proxy.main : [ConnectProxy] Listening on Optional([IPv4]127.0.0.1/127.0.0.1:9090)
2023-08-21T21:23:02+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102804080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:52619) [ConnectProxy] CONNECT httpbin.proxyman.app:443 HTTP/1.1
2023-08-21T21:23:02+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102804080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:52619) [ConnectProxy] Connecting to httpbin.proxyman.app:443
2023-08-21T21:23:02+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102804080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:52619) [ConnectProxy] Connected to Optional([IPv6]2606:4700:3032::ac43:b89b/2606:4700:3032::ac43:b89b:443)
SNI = hostname("httpbin.proxyman.app")
ChannelPipeline[ObjectIdentifier(0x000060000210c050)]:
                              [I] ↓↑ [O]
 ByteToMessageHandler<SNIHandler> ↓↑  [handler3]

NghiaTranUIT · August 21, 2023, 2:25pm

I tried some variations, but it doesn't work too.

My attempt: After the SNIHandler is removed, I add the GlueHandlers.

    private func glue(_ peerChannel: Channel, context: ChannelHandlerContext) {
        self.logger.debug("Gluing together \(ObjectIdentifier(context.channel)) and \(ObjectIdentifier(peerChannel))")

        // Ok, upgrade has completed! We now need to begin the upgrade process.
        // First, send the 200 message.
        // This content-length header is MUST NOT, but we need to workaround NIO's insistence that we set one.
        let headers = HTTPHeaders([("Content-Length", "0")])
        let head = HTTPResponseHead(version: .init(major: 1, minor: 1), status: .ok, headers: headers)
        context.write(self.wrapOutboundOut(.head(head)), promise: nil)
        context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)

        // Now remove the HTTP encoder.
        self.removeEncoder(context: context)
        context.pipeline.removeHandler(self, promise: nil)

        // ✅ NEW LOGIC Here
        // try to add SNIHandler
        context.pipeline.addHandler(ByteToMessageHandler(SNIHandler(sniCompleteHandler: { result in

            print("SNI = \(result)")
            return context.eventLoop.next().makeSucceededVoidFuture()
        })))
        .flatMap { _ -> EventLoopFuture<Void> in
            // try to glue again
            let (localGlue, peerGlue) = GlueHandler.matchedPair()
            return context.channel.pipeline.addHandler(localGlue).and(peerChannel.pipeline.addHandler(peerGlue)).map { (_, _) -> Void in
                return ()
            }
        }
        .whenComplete { result in
            switch result {
            case .success(_):
                context.pipeline.removeHandler(self, promise: nil)
            case .failure(_):
                // Close connected peer channel before closing our channel.
                peerChannel.close(mode: .all, promise: nil)
                context.close(promise: nil)
            }
        }
    }

Not sure what I'm missing here.

lukasa · August 21, 2023, 2:52pm

No, sorry, I mean that I'd like you to change context.pipeline.addHandler(ByteToMessageHandler(SNIHandler)) to:

let sniHandler = ByteToMessageHandler(SNIHandler(sniCompleteHandler: { result in
    print("SNI = \(result)")
    return context.eventLoop.next().makeSucceededVoidFuture()
}))
let debugInboundHandler = DebugInboundEventsHandler()
let debugOutboundHandler = DebugOutboundEventsHandler()
context.pipeline.addHandlers([sniHandler, debugInboundHandler, debugOutboundHandler])        
    .whenComplete { _ in
        context.fireChannelReadComplete()
    }

You can get the debug events handlers from NIOExtras.

NghiaTranUIT · August 22, 2023, 1:47am

Here is my full code:

        let sniHandler = ByteToMessageHandler(SNIHandler(sniCompleteHandler: { result in
            print("✅ SNI = \(result)")
            return context.eventLoop.next().makeSucceededVoidFuture()
        }))

        let debugInboundHandler = DebugInboundEventsHandler()
        let debugOutboundHandler = DebugOutboundEventsHandler()
        context.pipeline.addHandlers([sniHandler, debugInboundHandler, debugOutboundHandler])
            .flatMap { _ -> EventLoopFuture<Void> in
                // try to glue again
                let (localGlue, peerGlue) = GlueHandler.matchedPair()
                return context.channel.pipeline.addHandler(localGlue).and(peerChannel.pipeline.addHandler(peerGlue)).map { (_, _) -> Void in
                    return ()
                }
            }
            .whenComplete { _ in
                print(context.pipeline)
                context.fireChannelReadComplete()
            }

Output:

2023-08-22T08:45:57+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102204080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:50286) [ConnectProxy] CONNECT httpbin.proxyman.app:443 HTTP/1.1
2023-08-22T08:45:57+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102204080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:50286) [ConnectProxy] Connecting to httpbin.proxyman.app:443
2023-08-22T08:45:58+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102204080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:50286) [ConnectProxy] Connected to Optional([IPv4]104.21.68.11/104.21.68.11:443)
ChannelPipeline[ObjectIdentifier(0x00006000021100a0)]:
                              [I] ↓↑ [O]
 ByteToMessageHandler<SNIHandler> ↓↑                            [handler3]
        DebugInboundEventsHandler ↓↑                            [handler4]
                                  ↓↑ DebugOutboundEventsHandler [handler5]
                      GlueHandler ↓↑ GlueHandler                [handler6]
✅ SNI = hostname("httpbin.proxyman.app")
Channel completed reading in handler4
Reading in handler5
Channel read NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 330, readableBytes: 330, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000103009200 (2048 bytes) } } in handler4

// 👉 Hang 10 seconds
// Then, ...
Flushing in handler5

Closing with mode all in handler5
Channel became inactive in handler4
Channel unregistered in handler4

lukasa · August 22, 2023, 9:31am

Can you try applying a quick patch to the SNIHandler? I have a theory.

Can you change:

    public func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
        context.fireChannelRead(NIOAny(buffer))
        return .needMoreData
    }

to:

    public func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
        context.fireChannelRead(NIOAny(buffer))
        context.fireChannelReadComplete()
        return .needMoreData
    }

NghiaTranUIT · August 23, 2023, 1:36am

Thanks. I've confirmed that it's fixed. I'm able to see the SNI and the HTTPS Connection works fine

My patch: Fire read complete · apple/swift-nio@6d04a4a · GitHub
Log:

2023-08-23T08:34:05+0700 info com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000102a04080) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:50014) [ConnectProxy] Connected to Optional([IPv4]104.21.68.11/104.21.68.11:443)
ChannelPipeline[ObjectIdentifier(0x000060000210c0a0)]:
                              [I] ↓↑ [O]
 ByteToMessageHandler<SNIHandler> ↓↑                            [handler3]
        DebugInboundEventsHandler ↓↑                            [handler4]
                                  ↓↑ DebugOutboundEventsHandler [handler5]
✅ SNI = hostname("httpbin.proxyman.app")
Channel completed reading in handler4
First: channelReadComplete
Reading in handler5
Channel read NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 330, readableBytes: 330, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } } in handler4
First: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 330, readableBytes: 330, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } }
Second: partnerWrite
Channel completed reading in handler4
First: channelReadComplete
Second: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 2048, readableBytes: 2048, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000103809e00 (2048 bytes) } }
First: partnerWrite
Writing NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 2048, readableBytes: 2048, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000103809e00 (2048 bytes) } } in handler5
Second: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 2648, readableBytes: 2648, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } }
First: partnerWrite
Writing NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 2648, readableBytes: 2648, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } } in handler5
Second: channelReadComplete
Flushing in handler5
Channel read NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 226, readableBytes: 226, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } } in handler4
First: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 226, readableBytes: 226, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } }
Second: partnerWrite
Channel completed reading in handler4
First: channelReadComplete
Reading in handler5
Second: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 71, readableBytes: 71, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } }
First: partnerWrite
Writing NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 71, readableBytes: 71, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } } in handler5
Second: channelReadComplete
Flushing in handler5
Channel read NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 31, readableBytes: 31, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } } in handler4
First: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 31, readableBytes: 31, capacity: 2048, storageCapacity: 2048, slice: _ByteBufferSlice { 0..<2048 }, storage: 0x0000000102009800 (2048 bytes) } }
Second: partnerWrite
Channel completed reading in handler4
First: channelReadComplete
Reading in handler5
Second: channelRead, data = NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 981, readableBytes: 981, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } }
First: partnerWrite
Writing NIOAny { ByteBuffer { readerIndex: 0, writerIndex: 981, readableBytes: 981, capacity: 4096, storageCapacity: 4096, slice: _ByteBufferSlice { 0..<4096 }, storage: 0x000000010380c400 (4096 bytes) } } in handler5
Second: channelReadComplete
Flushing in handler5
Channel caught error: read(descriptor:pointer:size:): Connection reset by peer (errno: 54) in handler4

Closing with mode all in handler5
Channel became inactive in handler4
Channel unregistered in handler4

Is it a bug from SNIHandler? If yes, can you release a fix soon.

lukasa · August 24, 2023, 9:10am

Yes, it is: would you be interested in contributing this fix to the upstream repo? It would also need a unit test.

NghiaTranUIT · August 27, 2023, 7:30am

Thanks. I'm unsure if I can write Unit Tests under Apple standards, but I will try it.