Next Generation 100G Ethernet (with corrections)

Ever since the 40/100G Ethernet standard was completed in 2010, the IEEE standards group has been working on ways to improve it. In my opinion, there were two very serious holes in the original standard. The 40G long-reach variant did not match the existing telecom standard for 40G so some type of conversion equipment would be needed. This was fixed when the IEEE 802.3bg 40GBASE-FR single mode fiber standard was released in 2010. The second concern, which still exists, is the 100GBASE variants. Four standardized and one MSA currently exist and are shown in the following table.

Ethernet Variant	Data rate (Gbps)	Min. Reach (meters)	Form Factors	Media Wavelength	Standard IEEE 802.3
100GBASE-
CR10	10x10	7	CXP Direct Attach Copper	Twinax Copper	ba
SR10		100/150	CXP, CFP	LOMF 850nm
LR4	4x25	10,000	QSFP+, CFP	SMF 1310nm
ER4		40,000	CFP	SMF 1310nm
LR10	10x10	2,000	CFP	SMF 1550nm	Not supported

40/100G, the IEEE did not want to make the mistake of too many variants and form factors again (like they did for 10G) so consciously limited them. But, in our opinion, may have restricted them too much. By reducing the laser-optimized multi-mode fiber (LOMF) optical reach to 100m over OM3 and 150m for OM4, the IEEE left a huge gap in distance covered for data center applications – in fact, a two orders of magnitude gap – from 100m to 10km. This results in an enormous difference in cost as well. For example, a 100GBASE-SR10 CXP module average selling price is about $200, while the 100GBASE-LR4 average price is more than $20,000. So it is currently cost-prohibitive to design a data center with connections longer than 100m. This is not realistic. In order to address this shortcoming, the top transceiver manufacturers are working on SR4 products that have the potential to reach to 300m. Recently, the IEEE has recognized this issue and is looking to address it in its next generation study group. It is called the Next Generation 100Gb/s Optical Ethernet Study Group and its charter is to investigate 25G-per-lane standards and to explore lower-cost solutions to cover reaches perhaps up to a kilometer.

Any 100G variant using 25G signaling is still under development. While the optical devices are almost ready to go, there are long-term projects to ascertain how 25G is going to run on a printed-circuit board (PCBs) or on twinax cable. The group that was studying this has just officially been named a task force in the IEEE – the P802.3bj 100 Gb/s Backplane and Copper Cable Task Force. There are chip sets available to run 25G signals over PCBs that will be available in the coming months. Texas Instruments was demonstrating this at SC11 and Altera, Amphenol, Semtech/Gennum, IBM, Inphi, TE Connectivity and Xilinx showed 25G products in the OIF booth at OFC/NFOEC 2012.

Notice in the table above that there are different signaling schemes and form factors between 100GBASE-CR10, SR10 and 100GBASE-LR4. The CXP that was chosen for short-reach copper and LOMF is not suitable for longer-reach SMF operation. Even though most of them were involved in the IEEE process, equipment manufacturers are not happy about this because that means their products must support two different form factors at the same time. It may also doom CXP to only the initial products until another, better form factor can be developed that will cover both cost effectively – maybe a CFP2 or CFP4? Or the 25G signaling matures and the SR4 and CR4 variants are created in the QSFP28 (now being worked on in the SFF committee) is used.

The LR10 variant is not standardized, but is backed by a consortium of vendors and end users – including Google and Facebook. Whether this will take hold in the industry at large remains to be seen, but some of the industry leaders are boasting that it is actually selling very well currently at more than 2,500 units already.

So, while we talk about Terabit Ethernet being on the horizon and there have been multi-vendor demonstrations of 25G signaling for 100G operation, plenty of work remains to bring 100G to fruition prior to the next speed bump.

Interop, New York - Software Show Now

I hadn’t been to Interop since 2003 and I’d never been to the New York show so I decided this year I would give it a shot. I was woefully disappointed. I was expecting to see great product demonstrations from all the top equipment manufacturers, but instead received inquiries for meetings from software vendors who didn’t even bother to see that I cover data centers, optical components, structured cabling and interconnects.

The exhibit hall only had eight rows. Cisco was there, but half of its booth was taken up by its channel partners and the other half had virtual demos, not actual equipment running. Brocade was there, but had a much smaller booth and pretty much legacy equipment on a tabletop display. Most telling of course were the companies that didn’t participate in the exhibition – Extreme Networks and IBM to name just two.

Some of the programming was interesting, though, and maybe made it worth the travel costs. I sat in on the second day of the Enterprise Cloud Summit so actually got to meet some of the gurus in the industry of cloud computing. I also sat in on the “Evaluating New Data Center LAN Architectures” technical session which was a panel of equipment manufacturers that responded to an RFI from Boston Scientific for a data center expansion project. Interesting to note is that while Cisco was asked to respond, it did not. The panel consisted of Alcatel-Lucent, Extreme Networks, Force10 and Hewlett Packard. It is also interesting to note that the vendors responded with different architectures – some including top-of-rack solutions and others with end-of-row.

All-in-all, I think my time would have been better spent staying home and working on my optical components research…

DARPAs Chip-to-Chip Optical Interconnects (C2OI) Program

The C2OI DARPA program is funding on-going optical components projects. Its end goal is to “demonstrate optical interconnections between multiple silicon chips that will enable data communications between chips to be as seamless as data communication within a chip.”

This program grew out of work initially done by Agilent (now Avago Technologies) under the DARPA Parallel Optical Network Interconnect (PONI) project. Agilent/Avago developed a 30 Gbps transmitter (2.5 Gbps/lane) that was eventually standardized as the SNAP-12.

IBM (with help from Avago) extended the work originally done by Agilent/Avago into inter-chip connections and in 2009 achieved optical interconnection with 16 parallel lanes of 10G. By early 2010, IBM was extending this work into board-to-board applications which resulted in the new Avago MicroPOD™ product that was specifically designed for IBM’s POWER7™ supercomputer.

While it was designed for HPC server interconnects, the MicroPOD could be used for on-board or chip-to-chip interconnects as well. As mentioned in previous posts, the devices use a newly designed miniature detachable connector from US CONEC called PRIZM™ LightTurn™. The system has separate transmitter and receiver modules that are connected through a 12-fiber ribbon. Each lane supports up to 12.5 Gbps. It uses 850nm VCSEL and PIN diode arrays. The embedded modules can be used for any board-level or I/O-level application by either using two PRIZM LightTurn connectors or one PRIZM LightTurn and one MPO.

While MicroPOD is targeted at high-density HPC environments, a natural expansion of its market reach would be into switches and routers in high-density Ethernet data center environments. While this may not happen in the next few years, for me it looks like it could be a more cost-effective solution than say a 40G serial one.

Gigabit Transcievers

In our rush to want to discuss all the new technologies, it seems to me that analysts have forgotten that part of our job is to also point out ongoing trends in existing products. So while talking about Gigabit transceivers might not be as appealing as talking about Terabit Ethernet, it’s also a necessity – especially since, without these devices and the continuing revenue they produce, we wouldn’t have 40/100G or even 10G Ethernet. So what are the important points to make about Gigabit transceivers?

• The market for Gigabit Ethernet transceivers (copper and optical) is expected to be about $2.5-billion in 2010 according to CIR, but it is also supposed to start declining in 2011 when more 10GigE will take its place.
• Pricing for a 1000BASE-SX SFP module is now at about $20 for OEMs. End users still pay Cisco or Brocade or their agents about 8x that much (more about this later).
• Low pricing makes it difficult on profit margins so transceiver vendors hope to make it up in volume.
• While SFP is certainly the preferred form factor, there is still a decent amount of GBIC modules being sold.
• SFP direct-attach copper cable assemblies have become an option for top-of-rack switches to servers instead of using UTP Category patch or fiber cabling, although the majority of implementations today are still UTP patch cords, mainly because the connections within the rack are still 100M with the uplink being Gigabit Ethernet of the 1000BASE-SX variety.
• While 10/100/1000 ports are the norm for desktop and laptop computers, most of these devices are still connected back through standard Category 5e or 6 cabling to 100M switch ports in the telecom room.
• Gigabit Fibre Channel business is pretty much non-existent now. It was quickly replaced by 2G and has progressed through 4G and 8G is expected to become the volume application this year. Look for more on Fibre Channel in future posts.
• Avago Technologies and Finisar top the list of vendors for 1000BASE-SR devices. JDSU has all but disappeared from the scene, mainly because they have de-emphasized this business in favor of their telecom products. In fact, rumor has it that JDSU is shopping its datacom transceiver business and has been for some time.

A note on JDSU: It appears that the optical components giant has taken the technology that was developed at IBM, E2O and Picolight and thrown it away. Picolight was once a leader in parallel optics and, along with E2O, long-wavelength VCSELs. IBM pioneered v-groove technology and the oxide layer that enabled the next leap in speed and improved reliability for 850nm VCSELs. All of these technologies look like they are destined to die a slow, painful death after being acquired by JDSU. The company’s attention is clearly focused on its tunable technology and telecom applications, which is where, of course, it started. JDSU has never had a good reputation for assimilating acquisitions, so none of this should be a surprise. I was optimistic when JDSU bought these companies thinking that now these emerging technologies would be supported by a larger pocketbook. What is the reasoning for JDSU deemphasizing the technologies it acquired? Is it trying to get rid of short-reach competition in hopes that all optical networking would move towards long-wavelength devices? This would have been naïve; the likes of Finisar, Avago, MergeOptics and others would still be supporting 850nm optics and there remains a healthy market for them in enterprise networks and data centers—albeit a very competitive one as stated above.