Facebook friend Donn Lee presented on the need for 100Gbps and 1Tbps Ethernet at a recent Ethernet Alliance meeting. (found via Data Center Knowledge) You can watch the video here:
http://link.brightcove.com/services/player/bcpid1701276884?bclid=1622640422&bctid=40363249001
In addition to Donn’s presentation there are very good presentation videos from Google and Brocade that you can view at the above link. Bikash Koley from Google gave a great talk, and hints at some of the deep architectural problems posed by cloud services during his post-talk Q&A, so be sure to watch until the end. While watching Donn’s talk I was pleased to see that he can present real details as a Facebook employee, as opposed to censored presentations I’ve seen him give when he worked for Google. In particular it’s nice to see numbers on each axis of his graphs. 
In any case, his discussion centers around the port bandwidth needed for non-blocking Ethernet fabrics at high bisection bandwidths. His argument is that the bandwidth needs of Internet-scale properties like Facebook are outpacing the progress of the vendor and standards community. I especially liked one of his example network topologies, which illustrates how crazy things can get when you need lots of bandwidth throughout a datacenter:
This is a topic that I’ve been struggling with over the past couple years, as I’ve worried about building a non-blocking datacenter fabric that can support arbitrary workload distribution. And after all this time worrying, I’m not convinced this is the right strategy. Please don’t misunderstand; I’m certainly not convinced that it’s a wrong strategy. But, considering the technical and economic challenges, this approach seems pretty weak on its own. My opinion (until I’m convinced otherwise) is that more bandwidth and clever fabric topologies must be supplemented with some form of workload distribution management.
In enterprise or single-tenant environments this should be much easier than in a service provider cloud, because the relationship between each node can be determined and/or planned. But when anybody can throw any sort of workload at the infrastructure, creating opaque relationship topologies, what’s a network architect to do? That’s a real question–please send me pointers if you’re aware of existing work. Otherwise, I guess I should start wishing for the appearance of 1Tbps Ethernet in my future.
