UDP proxy relay issue

I want to implement a UDP proxy (like socks5 UDP associate part).
The pseudo code is as follows:

routeMap = [String : String]

public func channelRead(context: ChannelHandlerContext, data: NIOAny) {
	if fromLocal {
		// send request
		let info = self.unwrapInboundIn(data)
		let server = read_address()
		let data = read_data()
		routeMap[server] = info.remoteAddress
		let envelope = AddressedEnvelope(remoteAddress: server, data: data, metadata: nil)
		context.writeAndFlush(self.wrapOutboundOut(envelope), promise: nil)
	} else {
		// send back response
		let info = self.unwrapInboundIn(data)
		let data = read_data()
		let address = routeMap[info.remoteAddress] // ? conflict
		let envelope = AddressedEnvelope(remoteAddress: address, data: data, metadata: nil)
		context.writeAndFlush(self.wrapOutboundOut(envelope), promise: nil)
	}
}

When two clients request same host and a response come , should send to whom?

For UDP, if you don't remap the port this question is unsolvable. This is why NAT boxes also do port rewriting.