Ahaaaa, makes sense. This is how IRC worked.
To have a "private" chat, you have 2 options:
1, Look in the source of the OnReceive event of the server. Instead of sending the received message to everyone, if the message contains a special flag, send it to one recipient only. For example, if you start your message with "@IP" then it will look for all connected clients, and where the IP address matches - send the message only there
2, A more secure way is if you initiate a DCC-like (again, IRC did it this way) channel. If the message starts with @IP DCC, the server will send your IP address to @IP. Then, on your client, you open up a server socket and wait for a connection from @IP.
Long story short, wither your message will travel around the network twice (thus, more vulnerable) or you have to establish a client-server connection directly between the TWO CLIENTS: