PCIe – An I/O Optical Interconnect Soon?

The Peripheral Component Interconnect (PCI) is that bus in computers that connects everything back to the processor. It has been around for as long as I can remember having workstations. But in recent years, it has been morphing in response to the need to connect to the processor at higher data rates.

PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.

PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.

CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.

Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.

Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.

Opportunities for CXP AOCs and Transceivers

For those of you that believe that 100-Gigabit Ethernet is just around the corner, I have a bridge I want to sell you. But seriously, we haven’t even seen the height of 10-Gigabit Ethernet adoption yet, and there are some equipment companies saying they will sell 100’s-of-thousands of CXP 100GBASE-SR ports in 2011. Are you kidding? What is the application and where is the need?

First, 10GE has taken more than eight years to get to a million ports – we believe it will take 40G and 100G even longer. Second, even for clustering applications, which could potentially drive demand faster, 100GE port-adoption won’t be that quick. Ethernet architecture is different than the InfiniBand (IB) one – the density of an IB director-type switch provides over a Terabit per second, whereas the newly released 40GE ones are around 250G (due to both slower data rate and fewer ports). IB is also based on a CLOS architecture where you have equal bandwidth everywhere, while Ethernet is more often used in an aggregated network so ends up having a lot less higher-speed ports than lower speed ones. This is further supported by clustering applications that use ToR switches that are currently Gigabit connections to the servers with 10G uplinks to the network core. These will be upgraded to 10G downlinks and 40G uplinks first and this won’t happen quickly.

While several IP router manufacturers claim to have the need for 100’s of thousands of 100GBASE-SR CXP ports in 2011, I have found no evidence of this. Who are their customers? In fact, even those companies that could use 100G ports today, i.e. Google, Facebook , IXCs, etc., would need six months to a year to evaluate IP router products before they would deploy them. Since these devices do not yet exist, the reality is that the market will really not begin to materialize until at least 2012. Right now, the majority of router connections are still Gigabit Ethernet or OC-48 (2.5G) or below with OC-192 (10G) or 10GE being implemented on an as-needed basis. Until routers transition through 10G, then probably 40G, 100G installations will be few and far between.

But, there is a market for CXP AOCs today – InfiniBand. This is becoming a volume market now and will continue to be the best opportunity for CXP AOCs for at least the next few years and probably over the lifetime of the CXP products. In fact, we expect the volume of InfiniBand CXP AOCs to be at about six million by 2015. By comparison, the total volume of Ethernet CXP AOCs is expected to be less than 100-thousand. While 100G Ethernet clustering applications will initially use CXP AOCs, customers in these markets prefer to use pluggable modules mainly because they are used to structured cabling solutions and like their flexibility and ease of use, so AOCs will quickly give way to pluggable modules as they are developed. 100GE CXP ports may eventually eclipse InfiniBand once it permeates most data center distribution and core networks, but this will take longer than any of these equipment vendors anticipate I think.

Datacom Transceiver Vendors Transitioning into New Businesses

Have you noticed that it seems like all of the top datacom transceiver suppliers are transitioning their businesses? I’ve already talked about Avago’s new venture with its MicroPOD technology. They seemed to have supplanted Finisar as the technology leader in the space. Finisar has expanded its offerings into more telecom markets and JDSU is all but gone from the scene and focusing more on telecom again.

Finisar seems to be enjoying what may well be short-term success with its Laserwire offering. Since it is a non-standards based solution, it is difficult to believe it will become a mainstream one. While Finisar is offering other AOCs—C.wire (CXP-based) and Quadwire (QSFP-based), it does not seem to be participating in what seems to be a chip-to-chip optical interconnection revolution like Avago and Luxtera are (see previous posts for details) . Finisar used to be the technology leader in the optical transceiver space, it has veered off-course from that strength in preference for market diversification instead—now covering telecom and HPC standards-based solutions as an alternative. But perhaps this is the right move for Finisar, since it has not seemed to hurt its revenue position at all.

JDSU seems to be absent from the short-reach module market. It appears that the optical components giant has taken the technology that was developed at IBM, E2O and Picolight and thrown it away. Picolight was once a leader in parallel optics and, along with E2O, long-wavelength VCSELs. IBM pioneered v-groove technology and the oxide layer that enabled the next leap in speed and improved reliability for 850nm VCSELs. All of these technologies look like they are destined to die a slow, painful death after being acquired by JDSU. The company’s attention is clearly focused on its tunable technology and telecom applications, which is where, of course, it started. JDSU has never had a good reputation for assimilating acquisitions, so none of this should be a surprise. I was optimistic when JDSU bought these companies thinking that now these emerging technologies would be supported by a larger pocketbook. What was the reasoning for JDSU deemphasizing the technologies it acquired? Was it trying to get rid of short-reach competition in hopes that all optical networking would move towards long-wavelength devices? This would have been naïve; the likes of Finisar, Avago, MergeOptics and others would still be supporting 850nm optics and there remains a healthy market for them in enterprise networks and data centers—albeit a very competitive one.

According to JDSU, it is focusing on the LH and ULH versions of 40G and 100G first because it does not see the value in the CXP module for short-reach applications. For short-reach, it is focusing on QSFP+ modules, but development of these will take longer. The company claims it is not de-emphasizing its 850nm technology, but just focusing elsewhere first. I’m not so sure. Rumor has it, and I tend to believe that JDSU is looking for buyers for its short-wavelength business.