Colocation Data Centers Structured Cabling Trends

By Tom Debiec and Lisa Huff

As companies look to embrace cloud computing or backup their existing data centers, many are evaluating colocation as an option.  With this increased demand, colocation data centers are popping up all over the world and becoming a larger part of the overall data center market.  In 2013, the colocation sector is expected to account for about 25-percent of the structured-cabling data center market. During the work to develop the structured cabling forecast for the soon to be released Bishop & Associates report, "Structured Cabling Technology and Market Assessment," we had the chance to talk to project managers that are responsible for implementing 10,000 - 20,000 sqft build-outs in colocation facilities. It was clear to us that a few key trends have emerged:

• Whenever possible contractors recommend the use of pre-terminated copper and fiber cabling.  The benefits of utilizing these components include cost reduction, on-time delivery and the project is easier to manage.
• Although they are installing MPO cassettes on some jobs the cost premium often scares customers away.
• More OM3 fiber is being installed than OM4.  The up-sell to OM4 is difficult since OM3 covers the distances that are typically seen in these facilities (300m at 10G).
• A majority of the copper cabling is being installed is Category 6. 

A typical colocation lease averages about eight years.  Since the clients don't know what their requirements will be in this timeframe, they are less likely to make decisions that "future-proof" the installation for reuse with upgraded active equipment.  Tight budgets further preclude the addition of higher performing cabling products.  They would prefer to re-cable in the future than to pay for it now.  Since many of these installations are based on Top-of-Rack (ToR) architecture, re-cabling is viewed as a much simpler thing to do than to install new equipment when the cabinets are stuffed full of cabling. In view of this, we project that Category 6A and Category 7 cabling will only see very slow growth and that OM3 will be the mainstay over the next few years.

Next Generation 100G Ethernet (with corrections)

Ever since the 40/100G Ethernet standard was completed in 2010, the IEEE standards group has been working on ways to improve it. In my opinion, there were two very serious holes in the original standard. The 40G long-reach variant did not match the existing telecom standard for 40G so some type of conversion equipment would be needed. This was fixed when the IEEE 802.3bg 40GBASE-FR single mode fiber standard was released in 2010. The second concern, which still exists, is the 100GBASE variants. Four standardized and one MSA currently exist and are shown in the following table.

Ethernet Variant	Data rate (Gbps)	Min. Reach (meters)	Form Factors	Media Wavelength	Standard IEEE 802.3
100GBASE-
CR10	10x10	7	CXP Direct Attach Copper	Twinax Copper	ba
SR10		100/150	CXP, CFP	LOMF 850nm
LR4	4x25	10,000	QSFP+, CFP	SMF 1310nm
ER4		40,000	CFP	SMF 1310nm
LR10	10x10	2,000	CFP	SMF 1550nm	Not supported

40/100G, the IEEE did not want to make the mistake of too many variants and form factors again (like they did for 10G) so consciously limited them. But, in our opinion, may have restricted them too much. By reducing the laser-optimized multi-mode fiber (LOMF) optical reach to 100m over OM3 and 150m for OM4, the IEEE left a huge gap in distance covered for data center applications – in fact, a two orders of magnitude gap – from 100m to 10km. This results in an enormous difference in cost as well. For example, a 100GBASE-SR10 CXP module average selling price is about $200, while the 100GBASE-LR4 average price is more than $20,000. So it is currently cost-prohibitive to design a data center with connections longer than 100m. This is not realistic. In order to address this shortcoming, the top transceiver manufacturers are working on SR4 products that have the potential to reach to 300m. Recently, the IEEE has recognized this issue and is looking to address it in its next generation study group. It is called the Next Generation 100Gb/s Optical Ethernet Study Group and its charter is to investigate 25G-per-lane standards and to explore lower-cost solutions to cover reaches perhaps up to a kilometer.

Any 100G variant using 25G signaling is still under development. While the optical devices are almost ready to go, there are long-term projects to ascertain how 25G is going to run on a printed-circuit board (PCBs) or on twinax cable. The group that was studying this has just officially been named a task force in the IEEE – the P802.3bj 100 Gb/s Backplane and Copper Cable Task Force. There are chip sets available to run 25G signals over PCBs that will be available in the coming months. Texas Instruments was demonstrating this at SC11 and Altera, Amphenol, Semtech/Gennum, IBM, Inphi, TE Connectivity and Xilinx showed 25G products in the OIF booth at OFC/NFOEC 2012.

Notice in the table above that there are different signaling schemes and form factors between 100GBASE-CR10, SR10 and 100GBASE-LR4. The CXP that was chosen for short-reach copper and LOMF is not suitable for longer-reach SMF operation. Even though most of them were involved in the IEEE process, equipment manufacturers are not happy about this because that means their products must support two different form factors at the same time. It may also doom CXP to only the initial products until another, better form factor can be developed that will cover both cost effectively – maybe a CFP2 or CFP4? Or the 25G signaling matures and the SR4 and CR4 variants are created in the QSFP28 (now being worked on in the SFF committee) is used.

The LR10 variant is not standardized, but is backed by a consortium of vendors and end users – including Google and Facebook. Whether this will take hold in the industry at large remains to be seen, but some of the industry leaders are boasting that it is actually selling very well currently at more than 2,500 units already.

So, while we talk about Terabit Ethernet being on the horizon and there have been multi-vendor demonstrations of 25G signaling for 100G operation, plenty of work remains to bring 100G to fruition prior to the next speed bump.

Telecom Exchange

I recently attended an event in New York City – Telecom Exchange. The format was originally developed by Hunter Newby and Rory Cutaia when they were at Telx. This year it was hosted by Jamie Scotto & Associates (JSA). Unlike most trade shows, this affair puts large and small companies on equal footing. In order to provide a “network-neutral” environment, JSA arranged the exhibit tables in alphabetical order and they were the same size with the same-sized branding. No giveaways were allowed at the tables. To be frank, to me it was a refreshing change. Instead of spotlighting the next new thing, the event forced you to focus on networking with industry players and real business opportunities.

Some of my thoughts from my meetings:

• Containerized/modularized data centers:  One prominent executive from a data center connectivity supplier said to me:  “Brick and mortar data centers are dead.” We only had a short time to expand on this comment, but what I think he meant was that data center operators will need to move to more modular solutions in order to lower their PUE. According to him, if you move all your high-density applications to a containerized solution, your PUE can be as low as 1.1, whereas, any traditional building would be hard to get below a PUE of 1.5. His premise is that companies will need to lower their total cost of ownership of their data center and therefore will move to these solutions or be out of business. He hasn’t convinced me yet, but I intend to do some more research on the subject.
• AlliedFiber (AF) and Dupont Fabros Technology (DFT):  Allied Fiber is known for connecting data centers nationwide, but has never connected the “last mile” into the facility. That has now changed. AF and Dupont Fabros have struck a deal for AF to connect into DFT’s Piscataway, New Jersey facility with a straight path to Chicago, bypassing Manhattan. The agreement gives AF access to DFTs underground fiber ducting and DFT access to AFs direct fiber link to Chicago, lowering latency for both providers.
• EtherCloud: Tinet, A Neutral Tandem Company, has now taken its Ethernet Exchange one step further. With its EtherCloud offering, it can provide end-to-end international connectivity to any company. It allows global coverage using VPLS through Juniper equipment in the core and Cisco in the access. Tinet is one of less than a handful of companies that can now provide direct Ethernet services on three continents.
• Global reach:  Telehouse America is known for its data center and managed services business in the US, but is quickly growing its reach internationally. It now has facilities on four continents – Asia, Europe, North America and Africa. Similar to Tinet, Telehouse is building out its Ethernet networks globally.

So what does all of this have to do with optical components? It shows that not only are there opportunities within the data center, but also new ones in the telecom/long haul market.

The VCSEL Advantage

There are basically two types of lasers used in fiber optic transmission systems today:  edge-emitters and surface emitters.  The most prevalent high-speed edge-emitters are FP and DFB.  The beam emission for these devices is parallel to the substrate.  In the case of VCSELs, the light is emitted vertical to the substrate.

Short-wavelength VCSELs have been a part of the optical networking world since Honeywell introduced them in the early 1990s.  The devices were adopted quickly to replace unreliable and costly CD lasers in the datacom market.  Due to their inherent low cost, low power and small size, VCSELs became the light source of choice in enterprise networks.  They are also credited with enabling Gigabit transmission in that space and are currently being used in over 90 percent of Fibre Channel and Ethernet transceivers.   By taking what was learned at 850 nm and extending it to 1310 and to 1550-nm wavelengths, companies are now starting to show that they can drastically reduce the cost and size of transmitters.

Historically, longer wavelengths of 1300 to 1700 nm have been more difficult to produce in the VCSEL construction, because of the refractive index of InGaAsP.  This material is generally used for edge-emitters in these wavelengths and does not change very much with composition, which makes it difficult to produce components.  But in the early 2000s, development of different combinations of III-V elements led to long-wavelength VCSELs.  Several manufacturers such as Bandwidth9 with its tunable, 1550-nm VCSEL;  Cielo Communications, E2O and Picolight with their  1310-nm VCSELs had proven that lasers supporting wavelengths higher than 980 nm were possible to produce in volume. In fact, before JDSU acquired Picolight, it had several multi-Gigabit transceivers it produced with 1310-nm VCSELs. JDSU has since discontinued that line and, it seems, its support of 1310-nm VCSELs.

Key advantages of the VCSEL at production-level include the following:

• High Yields. VCSELs can be processed with as many as 20,000 individual lasers on a three-inch wafer.  Even if 20 percent of these are lost due to processing yields (a high number by VCSEL manufacturers standards), this is still a far higher yield than their edge-emitting cousins.
• Testing at the Wafer Level.  VCSELs can be tested before the wafer is diced.  Most edge-emitting lasers (FPs and DFBs) must be cleaved from the wafer and packaged before they can be tested, and are therefore tested individually, which increases processing costs and decreases yields significantly
• Easier Coupling and Packaging.  Another important advantage of the VCSEL structure is that its circular cross-section gives better control over beam size and divergence than for edge-emitters, allowing for much easier coupling of the fiber to the VCSEL output and easier alignment during packaging.

Table I shows a comparison of FP, DFB and VCSEL solutions, an availability status, and a list of some of the component manufacturers.

 

Table I:  Comparison of  VCSEL, FP and DFB Technologies
Attribute	VCSEL	FP	DFB
Cost	Low	High
Optical Output Power
Power Consumption
Size	Small (vertical construction)	Large (planar construction)
Mode Stability	Good	Fair
Testing	On chip	Packaged assembly
Manufacturing	Easy (20, 000 devices on 3 inch wafer)	Difficult
Packaging	Easy
Coupling to Fiber	Efficient	Inefficient
Modulation	Direct up to 12 Gbps	Direct only up to 2.5 Gbps then must be external
Drive circuitry	Simple	Complicated
Monolithic Integration	With receiver and electronic driver components	With other optical components
Suppliers	850 nm:  Agilent, Aerius Photonics, Applied Optoelectronics, Inc., Oclaro, Optowell, Emcore, EpiWorks, FCI/MergeOptics, Finisar, JDSU, Raycan, TE Connectivity, VI Systems 1310 nm:  Alight Technologies, Beam Express, JDSU, Raycan, Vertilas, VI Systems 1550 nm:  Raycan, Princeton Optronics, Vertilas	Agilent, Excelight, Finisar, JDSU, Modulight, Oclaro, OpNext	Agilent, Bookham,  JDSU, Excelight, Finisar, Fujitsu, Oclaro, OpNext

Long-wavelength VCSELs have started to emerge again mainly due to some new process technologies now being leveraged. Two companies stand out to me with their technology developments of long-wavelength VCSELs – Vertilas and VI Systems.