Issue with Connect Proxy - some connections don't work (probably sockets)?

Hello,

before I start with the topic, I just want to mention that I don't have a lot of experience with networking stuff. So apologies for any mistakes on my part while explaing the problem :slight_smile:

I am creating a NetworkExtension for iOS app which is using connect proxy implemented as shown in this official example - swift-nio-examples/connect-proxy at main · apple/swift-nio-examples · GitHub

The network extension works pretty well but some apps don't, namely Signal and WhatsApp. Via Charles Proxy I found that Signal seems to use websockets for the actual mesages and I think here lies the issue with my app.

I am not exactly sure how sockets fit in with NetworkExtension and this connect proxy.

The NetworkExtension uses the NEPacketTunnelProvider (Apple Developer Documentation) where you read incoming packets, and write them to the TUN interface.

Here is my server bootstrap setup:

bootstrap = ServerBootstrap(group: group)
        .serverChannelOption(ChannelOptions.socket(SOL_SOCKET, SO_REUSEADDR), value: 1)
        .childChannelOption(ChannelOptions.socket(SOL_SOCKET, SO_REUSEADDR), value: 1)
        .childChannelInitializer { channel in
            channel.pipeline.addHandler(ByteToMessageHandler(HTTPRequestDecoder(leftOverBytesStrategy: .forwardBytes))).flatMap {
                channel.pipeline.addHandler(HTTPResponseEncoder()).flatMap {
                    channel.pipeline.addHandler(ConnectHandler())
                }
            }
        }

So far my hunch is that sockets aren't kept alive, but my naive usage of SO_KEEPALIVE in the ChannelOptions.socket did not help.

Thanks to any pointers! I am more than happy to clarify things.

Thanks for reaching out! This is a tricky area of code because two quite different places are interacting. Before we can dive in we need to know a bit more about what you're actually trying to achieve. Does your server bootstrap run in your app? If so, have you changed the ConnectHandler in any way compared to upstream? Presumably you must have, otherwise your NEPacketTunnelProvider would have no relationship to the server bootstrap you're running here.

Yes, apologies! Forgot about this detail.

I have the server running inside the NetworkExtension. When I initially got this working, I was creating a TCP connection via createTCPConnection available in the NEPacketTunnelProvider, I then also tried switching this to the UDP session. Both worked for the "standard" stuff, but none for Signal.

Currently I am using the method packetFlow.readPacketObjects and here I am just calling packetFlow.writePacketObjects(packets) and then again. This also works. I am honestly quite surprised that I could try three methods without much experience and the result seems to be the same, that most of the traffic works correctly.

ConnectHandler is the same, I modified just the logging part to work with iOS 13.

I have added additional logging and there seems to be no issues logged by the ConnectHandler when trying to send messages via Signal.

Is there a way that would help me verify that socket connections are actually what causes an issue?

Signal has some troubleshooting info here: https://support.signal.org/hc/en-us/articles/360007320291-Firewall-and-Internet-settings They are specifically mentioning WebSockets, but this test seems to work fine when running my network extension: https://www.websocket.org/echo.html

It is highly unlikely that websockets are the specific problem you're having here. The CONNECT proxy is entirely agnostic to websockets (they run entirely within the opaque HTTPS tunnel).

Note, however, that the CONNECT proxy will only work for HTTPS traffic. That's why I'm having trouble grasping exactly what the intended data flow is here. Who is making HTTP connections to you? NEPacketTunnelProvider implies that it should be able to tunnel arbitrary IP packets, not purely HTTP packets. Who is making HTTP connections to your server? What should your extension do for packets that are not HTTP?

This network extension works like a kind of VPN configuration. So basically all the iOS traffic goes through my NEPacketTunnelProvider. It is basically similar to apps like Charles Proxy or Proxyman which help with debugging network traffic.

So I think the flow is something like this:

iOS app -> NetworkExtension -> SwiftNIO server (with the connect proxy) -> internet

Again, apologies for the confusion. I wanted to keep the first posts clear and did you accurately choose what is important and what isn't.

OK, let me start by making sure I understand you correctly:

  • You have an iOS packet tunnel provider.

  • Your provider claims the default route.

  • Your concern is the traffic from some apps is not ‘seen’ by your provider.

Is that right?

If so, this is an issue that lots of folks have bumped in to. I’ll start by explaining why it happens and then go on to discuss the big picture.

The reason why it happens is that packet tunnel providers work at the IP level and thus traffic is routed to them based on the destination IP address (hence their routingMethod property being set to .destinationIP). When a provider claims the default route is only sees traffic routed via the default route. If an app goes out of its way to route traffic explicitly, the packet tunnel provider won’t see that traffic.

My experience is that a lot of these ‘chat’ apps explicitly bind their connections to the WWAN interface, which means they don’t use the default route and thus aren’t seen by the packet tunnel provider.

Which brings us to the big picture. You wrote:

This network extension [is to] help with debugging network traffic.

Packet tunnel providers were designed to support VPN. They were not designed for creating network debugging tools and when you use them for that you will encounter a wide variety of problems, of which this is just the first.

The best way to see all traffic on an iOS device is an RVI packet trace. The main drawback to this is that you need to tether the device to your Mac. If you want to build an iOS-based product to do the equivalent, a packet tunnel provider is your only choice and it’s not a good one. If you’d like to see this improve in the future, I encourage you to file an enhancement request describing your requirements.

ps SwiftNIO is an innocent bystander in this. If you have further questions about Network Extension providers, it’d be best to ask them over on Apple Developer Forums. Use NetworkExtension so that the right folks see your question.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

2 Likes

Thanks, this helps a lot.

Yes, the traffic is apparently not seen (going by no error logs) and therefore can't get through and Signal messages do not work.

The use-case I am working on is more a malicious sites and adblocker kind of an app. I referenced the network debugging tools because I tried to see what Signal is doing and find the issue this way.

I will move the discussion to the Apple Developer Forum and try to phrase it a bit better :+1:

This by definition is a Content Filter action[1], and not a case to be handled by NEPacketTunnelProvider. If you traffic is claimed by the tunnel it is expected to be routed, and not blocked or denied. I didn't see your post on the Developer Forums so I thought I would chime in here.

[1] Content filters on iOS only work with supervised devices.

The content filter was what I tried at first but it has the limitation regarding supervised devices.

Yes, I know this usecase is not ideal, but I am afraid I don't have a choice.

Btw here is the new post at the Apple Developer Forum - NEPacketTunnelProvider does not se… | Apple Developer Forums

I followed up on the Developer Forums. Thank you!