Data Center Optics - TIA Data Center Workshop and 7x24 Exchange Delaware Valley Chapter Meeting

I just finished a quick trip to Arlington, VA and Philadelphia, PA to participate in two events:  TIA Data Center Workshop and 7x24 Exchange Delaware Valley Chapter Meeting. Both had excellent discussions about the role of optics in the data center.

The panel I moderated at the TIA Workshop had representatives from Microsoft, Google, Dell and Sumitomo. The discussion brought home to me the fact that there is a large gap between the needs of the small, medium and even large enterprise data center and the needs of Internet Data Centers. The chart below, that was provided by Dell, tries to summarize the point.

Here in 2014, we still have many enterprise data centers that run 1G servers with uplinks of 10G. At the same time, Microsoft is moving from a combination of 10G and 40G servers with uplinks of 40G and 100G, to pushing for 25G servers to connect to one 100G top-of-rack (ToR) switch-port (a breakout cable). And, Microsoft is also currently working on 400G and above for the rest of its data center network. Clearly a vast difference in Ethernet application.

The diversity of data center needs will continue to broaden over the next five to ten years so the IEEE is trying to address this in its Ethernet standards development. Data rates like 2.5G, 5G, 25G and 400G are now being considered in order to address both ends of the market "spectrum." Pun intended.

At the 7x24 Exchange Delaware Valley meeting, I discussed this trend and more from our recent study on data center optics. Here is a link to a copy of the presentation.

More to come later on trends in data center optics.

Next Generation 100G Ethernet (with corrections)

Ever since the 40/100G Ethernet standard was completed in 2010, the IEEE standards group has been working on ways to improve it. In my opinion, there were two very serious holes in the original standard. The 40G long-reach variant did not match the existing telecom standard for 40G so some type of conversion equipment would be needed. This was fixed when the IEEE 802.3bg 40GBASE-FR single mode fiber standard was released in 2010. The second concern, which still exists, is the 100GBASE variants. Four standardized and one MSA currently exist and are shown in the following table.

Ethernet Variant	Data rate (Gbps)	Min. Reach (meters)	Form Factors	Media Wavelength	Standard IEEE 802.3
100GBASE-
CR10	10x10	7	CXP Direct Attach Copper	Twinax Copper	ba
SR10		100/150	CXP, CFP	LOMF 850nm
LR4	4x25	10,000	QSFP+, CFP	SMF 1310nm
ER4		40,000	CFP	SMF 1310nm
LR10	10x10	2,000	CFP	SMF 1550nm	Not supported

40/100G, the IEEE did not want to make the mistake of too many variants and form factors again (like they did for 10G) so consciously limited them. But, in our opinion, may have restricted them too much. By reducing the laser-optimized multi-mode fiber (LOMF) optical reach to 100m over OM3 and 150m for OM4, the IEEE left a huge gap in distance covered for data center applications – in fact, a two orders of magnitude gap – from 100m to 10km. This results in an enormous difference in cost as well. For example, a 100GBASE-SR10 CXP module average selling price is about $200, while the 100GBASE-LR4 average price is more than $20,000. So it is currently cost-prohibitive to design a data center with connections longer than 100m. This is not realistic. In order to address this shortcoming, the top transceiver manufacturers are working on SR4 products that have the potential to reach to 300m. Recently, the IEEE has recognized this issue and is looking to address it in its next generation study group. It is called the Next Generation 100Gb/s Optical Ethernet Study Group and its charter is to investigate 25G-per-lane standards and to explore lower-cost solutions to cover reaches perhaps up to a kilometer.

Any 100G variant using 25G signaling is still under development. While the optical devices are almost ready to go, there are long-term projects to ascertain how 25G is going to run on a printed-circuit board (PCBs) or on twinax cable. The group that was studying this has just officially been named a task force in the IEEE – the P802.3bj 100 Gb/s Backplane and Copper Cable Task Force. There are chip sets available to run 25G signals over PCBs that will be available in the coming months. Texas Instruments was demonstrating this at SC11 and Altera, Amphenol, Semtech/Gennum, IBM, Inphi, TE Connectivity and Xilinx showed 25G products in the OIF booth at OFC/NFOEC 2012.

Notice in the table above that there are different signaling schemes and form factors between 100GBASE-CR10, SR10 and 100GBASE-LR4. The CXP that was chosen for short-reach copper and LOMF is not suitable for longer-reach SMF operation. Even though most of them were involved in the IEEE process, equipment manufacturers are not happy about this because that means their products must support two different form factors at the same time. It may also doom CXP to only the initial products until another, better form factor can be developed that will cover both cost effectively – maybe a CFP2 or CFP4? Or the 25G signaling matures and the SR4 and CR4 variants are created in the QSFP28 (now being worked on in the SFF committee) is used.

The LR10 variant is not standardized, but is backed by a consortium of vendors and end users – including Google and Facebook. Whether this will take hold in the industry at large remains to be seen, but some of the industry leaders are boasting that it is actually selling very well currently at more than 2,500 units already.

So, while we talk about Terabit Ethernet being on the horizon and there have been multi-vendor demonstrations of 25G signaling for 100G operation, plenty of work remains to bring 100G to fruition prior to the next speed bump.

Avago’s Interesting Demos at OFC/NFOEC

As always, the top transceiver manufacturers were represented at OFC, but of the top three datacom transceiver providers, Avago Technologies stood out to me. They had two significant demonstrations:

1. Connecting a 40GBASE-SR4 Ethernet port using its QSFP+ to four standard 10GBASE-SR Ethernet ports with its SFP+ modules. On the surface, this seems pretty easy to do, until you realize that the specifications for the transmitters and receivers in these devices have are very different. The 10G devices could easily overpower the 40G receiver if it’s not designed to handle the higher power. Avago has solved this issue with its parts and hopes to be able to be interoperable with anyone’s transceivers in the near future.
2. A VCSEL-based 25G short-wavelength SFP+ working prototype. At first I was puzzled about this because I couldn’t figure out the application. Well, it turns out there really is none for the 25G part, yet, but showing that it could be done makes you realize that 32G Fibre Channel applications using the SFP+ may not be as far-fetched as we think. And, perhaps we can get a 100GBASE-SR4 (4x25G that isn't in the IEEE standard yet) solution soon.

Stay tuned for more on other developments announced or demonstrated at OFC/NFOEC.

Optical Interconnection Players Strengthening Their Businesses

Molex just purchased Luxtera’s AOC business completing the circle that all the other optical interconnect players started. During the telecom bust in the early 2000’s, Amphenol, FCI, Molex and Tyco Electronics all either de-emphasized their optical interconnect businesses or exited them all together. Now, they have all re-entered. Why?

While they are all working on more high-speed copper solutions like the one Tyco showed for 25G and beyond at SC10, I beleive they also see the writing on the wall. While they won’t admit it, I think they know that beyond 100G copper cable interconnects may have FINALLY reached the end of their useful life. At 40G and 100G, for example, there is still no twisted-pair solution and the direct-attach copper can only reach about 7m reliably.

It has been interesting watching the choices these traditional connector companies have made:

• Amphenol: It never exited the optical interconnect business, but left the transceiver products to Avago, Finisar, JDSU and others until recently. It has a stronghold on the short-reach copper direct-attach market so has inroads at customers for its AOCs and modules.
• FCI: Exited the optics business entirely for a few years but then started again from scratch and subsequently purchased MergeOptics in February 2010. MergeOptics is what was left of Infineon Technologies and still has strong technical abilities in short-reach products. It also has the building blocks to provide all-optical interconnects all the way from the chip (see my previous posts on MergeOptics). They can provide both AOCs and transceiver modules so have the ability to cover all high-speed markets in InfiniBand, Ethernet and Fibre Channel.
• Molex: Purchased Luxtera’s AOC business recently. So while FCI and Tyco are stressing short-wavelength technologies, Molex has turned to custom long-wavelength ones. Luxtera’s technology is based on 1490nm devices, which really doesn’t matter if you’re purchasing an AOC, but will matter if you want transceiver modules. According to company representatives, they will eventually get back into supplying transceiver modules, but there has been no evidence of this as of yet. Perhaps the possession of Luxtera AOCs will prompt this.
• Tyco Electronics: Tyco exited the transceiver business in the early 2000’s, but still had a very active fiber-optic interconnect business – especially for premise wiring (AMP NETCONNECT). It acquired Zarlink Semiconductor’s optical products group in May 2010. Zarlink is on the forefront of parallel-optics technology and was one of the first to introduce AOCs. It does not appear that Tyco intends to supply optical transceiver modules again.

I would never bet against copper re-inventing itself in order to meet the demands of future high-speed networks, but with optical 10G dominating the market currently and 40/100G optical products starting to emerge, it will be an uphill battle for copper solutions to gain traction. And beyond 100G, all bets are off. I’m thinking that these companies are reaching the same conclusions and that if they don’t add optical capabilities soon, they may render themselves obsolete within the next ten years or so. That's not to say that there won't be a vibrant businesses in both copper structured cabling and interconnects over the next ten years - there will be. But I think that R&D dollars will be better spent on optical interconnect technologies rather than trying to figure out how to run 25G signals using copper interconnects (including backplanes.) Or how to convince end-user customers in the US that a shielded structured cabling solution for 40G is better than a short-reach optical one because it will be cheaper - but at what cost to power, cooling and space?

What do you think? I'd love to hear your thoughts.

The 10X10 MSA: Niche, Distraction or the Right Answer? (Continued)

While Vipul has a point that this new MSA is probably a distraction, it is difficult to deny that there is a market for cost-effective devices with optical reaches between 100m and 10km. In fact, 100m to 300m is the market that multi-mode fiber has served so well for the last 20 years. And, 300m to 2km has been a niche for lower-cost 1310nm single mode products like 1000BASE-LX. So I have a slightly different opinion about this 10x10 MSA and whether it’s a niche, distraction or the right answer.

In a recent article written on Optical Reflection, Pauline Rigby quotes Google’s senior network architect, Bikash Koley. About 100GBASE-SR10, he says 100m isn’t long enough for Google – that it won’t even cover room-to-room connections and that “ribbon fibres are hard to deploy, hard to manage, hard to terminate and hard to connect. We don’t like them.” There is an answer for this ribbon-fiber problem – don’t use it. There are many optical fiber manufacturers that now provide round multi-fiber cables that are only “ribbonized” at the ends for use with the 12-position MPO connector and are much easier to install – Berk-Tek, A Nexans Company, AFL and even Corning have released products that address this concern. But, the 100m optical reach is another matter.

I have to agree with Google about one other thing – 4x25G QSFP+ solutions are at least four years away from reality (and I would say probably even longer). This solution will eventually have the low-cost, low-power and high-density Google requires, but not quick enough. I think something needs to be done to address Google’s and others requirements between 300m and 2km in the short term, but I also believe that it needs to be standardized. There is no IEEE variant that would currently cover a 10x10G single mode device. However, there is an effort currently going on in the IEEE for 40G over SMF up to 2km. Perhaps the members of the MSA should look to work with this group to expand its work or start a new related project to cover 100G for 2km as well? I know this was thrown out of the IEEE before, but so were 1000BASE-T and 10GBASE-T initially.

So what I'm saying is that the market is more than a niche - hundreds of millions of dollars of LOMF sales at 1G and 10G would attest to that. And it's more than a distraction because there is a need. But I don't think it's entirely the right answer without an IEEE variant to back it up.

Let us know what you think.

Opportunities for CXP AOCs and Transceivers

For those of you that believe that 100-Gigabit Ethernet is just around the corner, I have a bridge I want to sell you. But seriously, we haven’t even seen the height of 10-Gigabit Ethernet adoption yet, and there are some equipment companies saying they will sell 100’s-of-thousands of CXP 100GBASE-SR ports in 2011. Are you kidding? What is the application and where is the need?

First, 10GE has taken more than eight years to get to a million ports – we believe it will take 40G and 100G even longer. Second, even for clustering applications, which could potentially drive demand faster, 100GE port-adoption won’t be that quick. Ethernet architecture is different than the InfiniBand (IB) one – the density of an IB director-type switch provides over a Terabit per second, whereas the newly released 40GE ones are around 250G (due to both slower data rate and fewer ports). IB is also based on a CLOS architecture where you have equal bandwidth everywhere, while Ethernet is more often used in an aggregated network so ends up having a lot less higher-speed ports than lower speed ones. This is further supported by clustering applications that use ToR switches that are currently Gigabit connections to the servers with 10G uplinks to the network core. These will be upgraded to 10G downlinks and 40G uplinks first and this won’t happen quickly.

While several IP router manufacturers claim to have the need for 100’s of thousands of 100GBASE-SR CXP ports in 2011, I have found no evidence of this. Who are their customers? In fact, even those companies that could use 100G ports today, i.e. Google, Facebook , IXCs, etc., would need six months to a year to evaluate IP router products before they would deploy them. Since these devices do not yet exist, the reality is that the market will really not begin to materialize until at least 2012. Right now, the majority of router connections are still Gigabit Ethernet or OC-48 (2.5G) or below with OC-192 (10G) or 10GE being implemented on an as-needed basis. Until routers transition through 10G, then probably 40G, 100G installations will be few and far between.

But, there is a market for CXP AOCs today – InfiniBand. This is becoming a volume market now and will continue to be the best opportunity for CXP AOCs for at least the next few years and probably over the lifetime of the CXP products. In fact, we expect the volume of InfiniBand CXP AOCs to be at about six million by 2015. By comparison, the total volume of Ethernet CXP AOCs is expected to be less than 100-thousand. While 100G Ethernet clustering applications will initially use CXP AOCs, customers in these markets prefer to use pluggable modules mainly because they are used to structured cabling solutions and like their flexibility and ease of use, so AOCs will quickly give way to pluggable modules as they are developed. 100GE CXP ports may eventually eclipse InfiniBand once it permeates most data center distribution and core networks, but this will take longer than any of these equipment vendors anticipate I think.

25G/40G VCSELs Driving Short-reach Optical Interconnects

Just a few years ago, laser designers were struggling with stability of their 10G VCSELs. But now, at least one, VI Systems GmbH, claims it will have production-ready 40G VCSELs within the next few years. The German start-up has developed two products it believes will take VCSELs beyond 10G applications - a directly-modulated (DM) device and an electro-optic modulated (EOM) DBR VCSEL. Both are short-wavelength (850nm) lasers.

In a recent press release, VI Systems explains that it “developed the VCSEL products at a wavelength of 850 nm along with a range of extremely fast integrated circuits based on the SiGe BiCMOS (silicon-germanium bipolar junction transistors in complementary metal-oxide-semiconductor) technology. The company uses a patent pending micro-assembly platform for the integration of the opto-electrical components and for alignment to a standard high performance multi-mode glass-based fiber.” The start-up has been presenting data supporting its claims of highly stable devices for more than a year now. It gets there by changing the laser active region material and structure to InAs quantum dot (QD).

Not only is VI Systems working on innovative laser structures, it has also developed new electro-optic integration methods to further reduce the cost of these devices.

I’ve noted in previous posts how VCSELs are the key to low-cost optical networks in the data center. These new VCSELs and packaging methods would bring an even more cost-effective “serial” solution for 40/100G. They could also be used for very short-reach optical connections like for chip-to-chip, on-board or board-to-board. Perhaps these inventive products will rival Avago’s MicroPOD and Luxtera’s OptoPHY (also in previous posts). Based on the presentations that VI Systems has released, it sure appears that its management completely understand the needs of both the data center and optical interconnect markets so could very well give incumbents in the industry some competition.

How the 40/100G Ethernet Shift to Parallel Optics Affects Data Center Cabling

Most data centers are cabled with at least Category 5e and some MMF. To upgrade to 10G, data center managers need to either test their entire installed base of Category 5e to make sure it is 10G-worthy or replace it with Category 6A or 7. And their MMF should be of at least the OM3 (2000 MHz.km) variety or the 300m optical-reach is in question. Unless, they want to use 10GBASE-LX4 or LRM modules that are about 10x the price of 10GBASE-SR devices. But what happens when you want to look beyond 10G to the next upgrade?

Last month I talked about how at 40/100G there is a shift to parallel-optics. Unlike today’s two-fiber configurations, with one send and one receive, the standards for 40G and 100G Ethernet specify multiple parallel 10G connections that are aggregated. 40GBASE-SR4 will use four 10G fibers to send and four 10G fibers to receive, while 100GBASE-SR10 will use ten 10G fibers in each direction.

What this means to the data center operator is that they may need to install new cable. Unless they’ve started to install pre-terminated fiber assemblies using the 12-position MPO connectors – these can be re-used if polarity is chosen carefully. Polarity is the term used in the TIA-568 standard to explain how to fiber (wire) to make sure each transmitter is connected to a receiver on the other end of a multi-fiber cable.

There are three polarity methods defined in the TIA standard and each has its advantages and disadvantages, but only two of the three will allow you to easily reuse your installed pre-term assemblies for 40/100G – methods A or B. I’ll explain why in my subsequent posts.