26th
9/90: All communications should be two-way
It is not uncommon for streaming media (one-way streaming or conferencing) systems to be built as a set of sender-receiver pairs. Sometimes, these systems use protocols like RTP that lend themselves to two-way communication with the sender blasting packets and the receiver sending some sort of report or acknowledgement back. This is the right way to do things. Many of these systems are also built with the senders spewing UDP packets at the receivers and hoping they take it, with no messaging from receiver back to the sender. This is the wrong way.
I’m now quite adamant about this because I’ve just spent a lot of time and grief debugging a very difficult problem that resulted from a one-way UDP application. I’ve worked with this application for years and it has generally worked flawlessly in a number of very challenging demonstration scenarios. More specifically, it has worked great over routed networks. Over switched networks, where all communication is at Layer 2, it turns out that switches will prune the receiver addresses from their MAC address forwarding tables, causing the packets from a given sender to flood all the ports on its local switch until the receiver is re-added to the forwarding table. In cases where the media streams are over half of the link capacity, this means that the receivers will experience seemingly random and catastrophic packet loss when the forwarding tables are incorrect. To make it more difficult, the receiving computers will show reduced data rates, even though the problem is actually excessive traffic.
This was a very difficult problem to solve. Hopefully someone with this problem will fortuitously stumble across this post, just as I was fortunate enough to remember hearing Cees de Laat describe this switch flooding problem several months ago.
