The 10X10 MSA: Niche, Distraction or the Right Answer? (Continued)

While Vipul has a point that this new MSA is probably a distraction, it is difficult to deny that there is a market for cost-effective devices with optical reaches between 100m and 10km. In fact, 100m to 300m is the market that multi-mode fiber has served so well for the last 20 years. And, 300m to 2km has been a niche for lower-cost 1310nm single mode products like 1000BASE-LX. So I have a slightly different opinion about this 10x10 MSA and whether it’s a niche, distraction or the right answer.

In a recent article written on Optical Reflection, Pauline Rigby quotes Google’s senior network architect, Bikash Koley. About 100GBASE-SR10, he says 100m isn’t long enough for Google – that it won’t even cover room-to-room connections and that “ribbon fibres are hard to deploy, hard to manage, hard to terminate and hard to connect. We don’t like them.” There is an answer for this ribbon-fiber problem – don’t use it. There are many optical fiber manufacturers that now provide round multi-fiber cables that are only “ribbonized” at the ends for use with the 12-position MPO connector and are much easier to install – Berk-Tek, A Nexans Company, AFL and even Corning have released products that address this concern. But, the 100m optical reach is another matter.

I have to agree with Google about one other thing – 4x25G QSFP+ solutions are at least four years away from reality (and I would say probably even longer). This solution will eventually have the low-cost, low-power and high-density Google requires, but not quick enough. I think something needs to be done to address Google’s and others requirements between 300m and 2km in the short term, but I also believe that it needs to be standardized. There is no IEEE variant that would currently cover a 10x10G single mode device. However, there is an effort currently going on in the IEEE for 40G over SMF up to 2km. Perhaps the members of the MSA should look to work with this group to expand its work or start a new related project to cover 100G for 2km as well? I know this was thrown out of the IEEE before, but so were 1000BASE-T and 10GBASE-T initially.

So what I'm saying is that the market is more than a niche - hundreds of millions of dollars of LOMF sales at 1G and 10G would attest to that. And it's more than a distraction because there is a need. But I don't think it's entirely the right answer without an IEEE variant to back it up.

Let us know what you think.

The 10X10 MSA: Niche, Distraction or the Right Answer?

{For today’s blog, our guest author is Vipul Bhatt. I have known Vipul for several years, since when he was the Director of High Speed Optical Subsystems at Finisar. He has served as the Chair of Optical PMD Subgroup of IEEE 802.3ah Ethernet in the First Mile (EFM), and the Chair of Equalization Ad Hoc of IEEE 802.3ae 10G Ethernet. He can be reached at vjb@SignalOptics.com.}

Last week, Google, JDSU, Brocade and Santur Corp announced the 10X10 Multi-Source Agreement (MSA) to establish sources of 100G transceivers. It will have 10 optical lanes of 10G each. Their focus is on using single mode fiber to achieve a link length of up to 2 km. The key idea is that a transceiver based on 10 lanes of 10G will have lower power consumption and cost because it doesn’t need the 10:4 gearbox and 25G components. But is this a good idea? What is the tradeoff? Based on my conversations with colleagues in the industry, it seems there are three different opinions emerging about how this will play out. I will label them as niche, distraction, or the right answer. Here is a paraphrasing of those three opinions.

It’s a niche: It’s a solution optimized for giant data centers – we’re talking about a minority of data centers (a) that are [already] rich in single mode fiber, (b) where the 100-meter reach of multi-mode 100GBASE-SR10 is inadequate, and (c) where the need for enormous bandwidth is so urgent that the density of 10G ports is not enough, and 100G ports can be consumed in respectable quantities in 2011.

It’s a distraction: Why create another MSA that is less comprehensive in scope than CFP, when the CFP has sufficient support and momentum already? Ethernet addresses various needs – large campuses, metro links, etc. – with specifications like the LR4 that need to support link lengths of well beyond 2 km over one pair of fiber. We [do] need an MSA that implements LR4, and the SR10 meets the needs of a vast majority of data centers, so why not go with CFP that can implement both LR4 and SR10? As for reducing power consumption and cost, the CFP folks are already working on it. And it’s not like we don’t have time – the 10G volume curve hasn’t peaked yet, and may not even peak in 2011. Question: What is the surest way to slow down the decisions of Ethernet switch vendors? Answer: Have one MSA too many.

It’s the right answer: What is the point of having a standard if we can’t implement it for two years? The CFP just isn’t at the right price-performance point today. The 10X10 MSA can be the “here and now” solution because it will be built with 10G components that have already traversed the experience curve. It can be built with power, density and cost figures that will excite the switch vendors, which may accelerate the adoption of 100G Ethernet, not distract it. As for 1-pair vs. 10-pairs of fiber, the first swelling of 100G demand will be in data centers where it’s easier to lay more fiber, if there isn’t plenty installed already. The 2-km length is sufficient to serve small campuses and large urban buildings as well.

Okay, so what do I think? I think the distraction argument is the most persuasive. An implementation that is neither SR10-compliant nor LR4-compliant is going to have a tough time winning the commitment of Ethernet switch vendors, even if it’s cheaper and cooler than the CFP in the short term.

SFP+ - The New Optical RJ45?

For those of you that have been in the industry for what seems to be 100 years, but is really about 25 years, you know that the one “connector” that hasn’t changed much is the RJ45. While there have been improvements by adding compensation for the error that was made way back when AT&T developed the wiring pattern (splitting the pair causing major crosstalk issues), the connector itself has remained intact. Contrastingly, optical connectors for datacom applications have changed several times – ST to SC to MT-RJ to LC. They have finally seemed to settle on the LC and perhaps on a transceiver form factor – the SFP+. The SFP was originally introduced at 1G, was used for 2G and 4G and with slight improvements has become the SFP+ and the dominant form factor now used for 10G. Well, it is in the process of getting some slight improvements again and promises to make it all the way to 32G. That’s six generations of data rates – pretty impressive. But how?

The INCITS T11.2 Committee's Fibre Channel Physical Layer – 5 (FC-PI-5) standard was ratified in September. It specifies 16G Fibre Channel. Meanwhile, the top transceiver manufacturers have been demonstrating pre-standard 16G SFP+ SW devices. But, wait a minute – short-wavelength VCSELs were supposed to be very unstable when trying to modulate them at data rates above 10G right? Well, it seems that at least Avago and Finisar have figured this out. New microcontrollers and adding at least one clock and data recovery (CDR) device in the module to help clean up the signals have proven to be keys. Both vendors believe it is possible to do this and not add too much cost to the modules. In fact, both also think that possibly by adding electronic dispersion compensation (EDC) they can push the SFP+ to 32G as well - which is the next step for Fibre Channel - hoping to stop at 20G and 25G to cover developments in Ethernet and InfiniBand.

And what about long wavelength devices? It has always been a challenge fitting the components needed to drive long distances into such a small package mainly because the lasers need to be cooled. But not anymore – Opnext has figured it out. In fact, it was showing its 10km 16G FC SFP+ devices long before any of the SW ones were out (March 2010). Of course, this isn't surprising considering Opnext has already figured out 100G long haul as well.

These developments are important to datacom optical networking for a few of reasons:  

1. They show that Fibre Channel is not dead.
2. The optical connector and form factor "wars" have seemed to subsided so transceiver manufacturers and optical components vendors can focus on cooperation instead of positioning.
3. They will impact the path other networking technologies are taking – Ethernet and InfiniBand are using parallel optics for speeds above 10G – will they switch back to serial?

Stay tuned for more on these points later.

 

SC10 and Optical Transcievers

It’s a beautiful time of year in New Orleans for the top supercomputing companies to show their wares. While I wouldn’t consider SC10 exactly the place to sell optical components, there were a few new developments there. SCinet – the network that is always built at the top HPC conference – boasted 100G Ethernet as well as OTu4. Alcatel-Lucent, Ciena, Cisco, Force10 and Juniper, among others donated equipment to build this network. Module vendors Avago Technologies, Finisar and Reflex Photonics contributed QSFP and CFP 40G and 100G devices to the cause.

Meanwhile, the Ethernet alliance was showing two demonstrations in its booth – a converged network running FCoE and RoCE over 40GigE and 100GigE. Nineteen different vendors participated in this demo that was run by the University of New Hampshire Interoperability Lab. Both CFPs and QSFPs were used in this demo.

Some of you may wonder why I would attend SC10. I keep my eye on the HPC market because it usually indicates where the broader data center market will be in a few years. And, in fact, even medium-sized businesses’ data centers with higher computational needs are starting to resemble small HPC centers with their server clusters using top-of-rack switching.

Most of the top optical transceiver vendors and even some of the smaller ones see this market as an opportunity as well. While InfiniBand still uses a lot of copper interconnects, for 40G and 120G, this is changing. QSFP was the standout for 40G IB displays and CXP AOCs were shown for 120G as well. Avago Technologies was the first to announce a CXP module at the show.

Some believe that the CXP will be short-lived because there is progress being made on 4x25 technologies – Luxtera announced its 25G receivers to go with its 25G transmitter that it announced earlier this year. But it will still be a few years before all of the components for 25G will be ready for system’s developers to spec in. Tyco Electronics had a demonstration at their booth showing it is possible to run 28G over eight inches of a PCB, but this was still a prototype. And Xilinx has announced a chip for 28G electrical transceivers that can be used with this board design. But, none of these devices are even being tested by equipment manufacturers yet and the CXP has already been adopted by a few. So I think the CXP may have more life in it than some people may think.

PCIe – An I/O Optical Interconnect Soon?

The Peripheral Component Interconnect (PCI) is that bus in computers that connects everything back to the processor. It has been around for as long as I can remember having workstations. But in recent years, it has been morphing in response to the need to connect to the processor at higher data rates.

PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.

PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.

CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.

Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.

Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.

Opportunities for CXP AOCs and Transceivers

For those of you that believe that 100-Gigabit Ethernet is just around the corner, I have a bridge I want to sell you. But seriously, we haven’t even seen the height of 10-Gigabit Ethernet adoption yet, and there are some equipment companies saying they will sell 100’s-of-thousands of CXP 100GBASE-SR ports in 2011. Are you kidding? What is the application and where is the need?

First, 10GE has taken more than eight years to get to a million ports – we believe it will take 40G and 100G even longer. Second, even for clustering applications, which could potentially drive demand faster, 100GE port-adoption won’t be that quick. Ethernet architecture is different than the InfiniBand (IB) one – the density of an IB director-type switch provides over a Terabit per second, whereas the newly released 40GE ones are around 250G (due to both slower data rate and fewer ports). IB is also based on a CLOS architecture where you have equal bandwidth everywhere, while Ethernet is more often used in an aggregated network so ends up having a lot less higher-speed ports than lower speed ones. This is further supported by clustering applications that use ToR switches that are currently Gigabit connections to the servers with 10G uplinks to the network core. These will be upgraded to 10G downlinks and 40G uplinks first and this won’t happen quickly.

While several IP router manufacturers claim to have the need for 100’s of thousands of 100GBASE-SR CXP ports in 2011, I have found no evidence of this. Who are their customers? In fact, even those companies that could use 100G ports today, i.e. Google, Facebook , IXCs, etc., would need six months to a year to evaluate IP router products before they would deploy them. Since these devices do not yet exist, the reality is that the market will really not begin to materialize until at least 2012. Right now, the majority of router connections are still Gigabit Ethernet or OC-48 (2.5G) or below with OC-192 (10G) or 10GE being implemented on an as-needed basis. Until routers transition through 10G, then probably 40G, 100G installations will be few and far between.

But, there is a market for CXP AOCs today – InfiniBand. This is becoming a volume market now and will continue to be the best opportunity for CXP AOCs for at least the next few years and probably over the lifetime of the CXP products. In fact, we expect the volume of InfiniBand CXP AOCs to be at about six million by 2015. By comparison, the total volume of Ethernet CXP AOCs is expected to be less than 100-thousand. While 100G Ethernet clustering applications will initially use CXP AOCs, customers in these markets prefer to use pluggable modules mainly because they are used to structured cabling solutions and like their flexibility and ease of use, so AOCs will quickly give way to pluggable modules as they are developed. 100GE CXP ports may eventually eclipse InfiniBand once it permeates most data center distribution and core networks, but this will take longer than any of these equipment vendors anticipate I think.

Interop, New York - Software Show Now

I hadn’t been to Interop since 2003 and I’d never been to the New York show so I decided this year I would give it a shot. I was woefully disappointed. I was expecting to see great product demonstrations from all the top equipment manufacturers, but instead received inquiries for meetings from software vendors who didn’t even bother to see that I cover data centers, optical components, structured cabling and interconnects.

The exhibit hall only had eight rows. Cisco was there, but half of its booth was taken up by its channel partners and the other half had virtual demos, not actual equipment running. Brocade was there, but had a much smaller booth and pretty much legacy equipment on a tabletop display. Most telling of course were the companies that didn’t participate in the exhibition – Extreme Networks and IBM to name just two.

Some of the programming was interesting, though, and maybe made it worth the travel costs. I sat in on the second day of the Enterprise Cloud Summit so actually got to meet some of the gurus in the industry of cloud computing. I also sat in on the “Evaluating New Data Center LAN Architectures” technical session which was a panel of equipment manufacturers that responded to an RFI from Boston Scientific for a data center expansion project. Interesting to note is that while Cisco was asked to respond, it did not. The panel consisted of Alcatel-Lucent, Extreme Networks, Force10 and Hewlett Packard. It is also interesting to note that the vendors responded with different architectures – some including top-of-rack solutions and others with end-of-row.

All-in-all, I think my time would have been better spent staying home and working on my optical components research…