An interview with the 10-gig E team leaders

When the Institute of Electrical and Electronics Engineers (IEEE) recently adopted a new 10-gigabit-per-second standard for the Ethernet, the most widely installed local area network technology, the speed of Ethernet operations increased by an order of magnitude -- at least on paper. A gigabit is a billion bits, and actually achieving that ten-fold increase in Ethernet performance remains a challenge that can only be met with leading-edge equipment and expertise.

Lawrence Berkeley National Laboratory operates some of the world's most powerful computing, data storage, and networking resources for the U.S. Department of Energy. Recently the Lab teamed with a number of commercial vendors, including Force10 Networks for switches, SysKonnect for network interfaces, FineTec Computers for clusters, Quartet Network Storage for online storage, and Ixia for line rate monitors, to assemble a "10-gig E" demonstration system. The system runs a true scientific application on one 11-processor cluster, then sends the resulting data across a 10-gigabit Ethernet connection to another cluster, where it is rendered for visualization.

	In Berkeley Lab's Access Grid Node, a Cactus code simulation of in-spiraling black holes is rendered by Visapult, demonstrating data transfer at ten billion bits per second.

Ten-gigabit Ethernet capability was showcased in a demonstration held at Berkeley Lab on Tuesday, July 2, 2002. Ixia's line-monitoring equipment showed that performance peaked at an actual line rate of 10.6 gigabits per second, with the total amount of real data transferred during the demonstration and preceding 12 hours of trial runs reaching nearly 60 terabytes (60 trillion bytes).

When it comes to moving huge amounts of scientific data quickly across networks, the team from Berkeley Lab has been the undisputed champion of the high-performance computing and networking world for two years running. Last November, at the SC2001 conference in Denver, the Berkeley Lab team took top honors in the High-Performance Bandwidth Challenge, moving data across the network at a sustained rate of 3.3 Gigabits in a live demonstration of computational steering and visualization. The team made use of the Albert Einstein Institute's "Cactus" simulation code and Berkeley Lab's Visapult parallel visualization system, running on hardware provided by Force10, SysKonnect, and FineTec.

The July 2 10-gigabit Ethernet demonstration at Berkeley Lab was assembled with help from the same vendors and served as a test run for this year's High-Performance Bandwidth Challenge at SC2002 in Baltimore. The Berkeley Lab team and its partners will be seeking their third straight win.

The team primarily responsible for assembling the 10-gigabit demonstration system consists of four Berkeley Lab staffers, network engineers Mike Bennett and John Christman, and computer systems engineers John Shalf and George "Chip" Smith. Science Beat had a chance to talk with Bennett, Shalf, and Smith about the effort as the demonstration was being prepared.

From left, Mike Bennett, George "Chip" Smith, Raju Shah of Force10 Networks, John Shalf, and John Christman. The screens indicate a total data transfer rate of 10.6 gigabits per second.

Science Beat: "First of all, why is Lawrence Berkeley National Laboratory leading a demonstration project like this?"

Bennett: "We had been asked in January to serve as a technical advisor to a conference planned for March. The goal of the conference was to highlight the new IEEE standard for 10-gig E. For various reasons, the conference was pushed back until June to coincide with adoption of the standard. We were then asked what kind of demo we could put together that would show the difference that having 10-gig capability would make. I immediately thought of the Lab group that won the Bandwidth Challenge at SC2001 -- they had a real scientific application that was bandwidth intensive.

"We put the demo system together for the conference, which was again delayed. Since we had a room full of equipment, we decided to salvage our effort and do a demo run here. It turned out to really successful. Force 10 loaned us the switches, FineTec donated enough computers to make it interesting, and Chip Smith worked with SysKonnect to get very high performance from their network interfaces. Quartet provided the network storage for the data to be visualized.

"The result is we proved that 10-gig E is a reality, not just a bunch of back-of-the-envelope calculations."

Smith: "Also, Berkeley Lab has a long history of being on the forefront of networking, from putting the first supercomputer on ARPANET, to helping develop TCP and IP protocols, to posting one of earliest sites at the dawn of the World Wide Web. We're carrying on that work by extension, to keep the Lab at the forefront of technology and to continue to push the capabilities of that technology."

Science Beat: "In lay terms, what does 10-gigabit Ethernet represent?"

Bennett: "In order to put 10-gigabit Ethernet in perspective, consider that the average desktop machine connects at 100 megabits per second [100 million bits per second]. In essence, the higher speed technology is 100 times faster.

"Here's an example of the advantage of faster data transfer: the file size of a raw digital version of The Matrix in AVI format is approximately 236 gigabits. With 10-gigabit Ethernet, transferring the entire movie file takes 23.6 seconds. In contrast, the average desktop machine transfer using Fast Ethernet takes 2,360 seconds, or roughly 39 minutes. The same transfer over a DSL line takes 66 hours. Still, the full benefit of 10-gigabit Ethernet has yet to be fully appreciated."

Science Beat: "Is this the first real-world demonstration of 10-gigabit Ethernet capability?"

Bennett: "As far as I know. A lot of the tests that have been publicized have been interoperability-based, to show that a product from Vendor A can interoperate with equipment from Vendor B, which is the aim of the IEEE standard. What the interoperability standard doesn't address is whether you can take one vendor's equipment and plug it into a cluster connected to a network and get that 10-gig level of performance.

"What we are demonstrating is that it does work in the real world. And it has real-world benefits. From a network engineering perspective, 10-gig E makes building a network is much easier. You have one point-to-point connection, rather than ten 1-gig E connections to install and maintain."

Ten billion bits a second, Part 2