Data Center Optics - TIA Data Center Workshop and 7x24 Exchange Delaware Valley Chapter Meeting

I just finished a quick trip to Arlington, VA and Philadelphia, PA to participate in two events:  TIA Data Center Workshop and 7x24 Exchange Delaware Valley Chapter Meeting. Both had excellent discussions about the role of optics in the data center.

The panel I moderated at the TIA Workshop had representatives from Microsoft, Google, Dell and Sumitomo. The discussion brought home to me the fact that there is a large gap between the needs of the small, medium and even large enterprise data center and the needs of Internet Data Centers. The chart below, that was provided by Dell, tries to summarize the point.

Here in 2014, we still have many enterprise data centers that run 1G servers with uplinks of 10G. At the same time, Microsoft is moving from a combination of 10G and 40G servers with uplinks of 40G and 100G, to pushing for 25G servers to connect to one 100G top-of-rack (ToR) switch-port (a breakout cable). And, Microsoft is also currently working on 400G and above for the rest of its data center network. Clearly a vast difference in Ethernet application.

The diversity of data center needs will continue to broaden over the next five to ten years so the IEEE is trying to address this in its Ethernet standards development. Data rates like 2.5G, 5G, 25G and 400G are now being considered in order to address both ends of the market "spectrum." Pun intended.

At the 7x24 Exchange Delaware Valley meeting, I discussed this trend and more from our recent study on data center optics. Here is a link to a copy of the presentation.

More to come later on trends in data center optics.

Review of Data Centre World London 2013

I haven’t attended an AFCOM Data Center World Conference in almost five years because when I did, my impression was that the exhibits were not well attended by would-be clients of the company I was working for at the time. In fact, that year, we saw only two potential customers in our booth. People that have attended it recently have told me much of the same. They received less than a handful of good prospects to target.

Fast forward to 2013 when I’m temporarily living in the UK and have an opportunity to attend Data Centre World in London. I was pleasantly surprised to find, in my opinion, a much better event – and it was free for qualified attendees! Perhaps that was the difference. I found the conference program to be comparable to AFCOM’s and the exhibit floor to  be much more crowded. The other refreshing thing was that while software defined networks (SDN) were mentioned, they were not the overwhelming theme. The program stuck to its intended subject matter, the data center market, its networks, facilities, clouds and hardware.

Some of my observations from attended presentations and visits to some of the booths:

• Data center space demand throughout the world is expected to level off within the next five years due to server virtualization. SSE Telecoms showed this chart to illustrate this:

           It was reiterated by CBRE in its presentation – a snapshot is below.

 

• More than 75% of network traffic stays within the data center. Thus the trend to higher data rates and flattening of the data center network.
• The vertical market customer drives data center needs. In other words, if you’re a healthcare organization, your data center may look much different from a state/local government data center.
•  “Shadow IT” is driving change in local area networks (LAN) and data centers. Bring your own device (BYOD) forces company networks to be more open to the employees using them, but poses possible security issues for the IT department.
• Co-location is slowly becoming the norm for even smaller businesses due to the expenses associated with maintaining ownership of the data center.
• Many connectivity vendors were there and they seemed to be getting healthy traffic in their booths. Some that I visited were 3M, Brand-Rex, CommScope, Corning, Draka/Prysmium Group, Fujikura, HellermannTyton, Lynx Data Cabling, Methode, Ortronics and Telegärtner. All were showing either copper and fiber solutions (or both) for the data center.

Companies Seriously Considering Colocation Data Centers

By Thomas Debiec

Colocation data centers were the topic at the February Philadelphia meeting of 7x24 Exchange Delaware Valley Chapter.  Representatives from the University of Pennsylvania, Wilmington University, Comcast and Vanguard shared their opinions on this topic.   Here are a few takeaways from this meeting:

• Considering colocation to support new and to improve existing services.
• Comcast is going to regional data centers to support new and existing services in various regions of the country enabling them to increase performance and roll out new products quickly. Services are moving closer to the edge.
• University libraries are moving away from hard books.
• Universities are moving to cloud-based grading services such as Blackboard.
• Mission critical recovery – a separate location is needed for disaster recovery and backup
• To support cloud services
• To take advantage of new technologies that might be very costly in-house
• Companies no longer want to be in the DC business
• Data centers are a huge expense
• If you host it you own it
• Growing comfort levels in not owning and operating
• Security is not as much of a barrier due to proven case studies
• Companies are still feeling their way around
• Vanguard representative feels that larger banks are moving to colocation perhaps more quickly than other vertical market sectors.

Colocation Data Centers Structured Cabling Trends

By Tom Debiec and Lisa Huff

As companies look to embrace cloud computing or backup their existing data centers, many are evaluating colocation as an option.  With this increased demand, colocation data centers are popping up all over the world and becoming a larger part of the overall data center market.  In 2013, the colocation sector is expected to account for about 25-percent of the structured-cabling data center market. During the work to develop the structured cabling forecast for the soon to be released Bishop & Associates report, "Structured Cabling Technology and Market Assessment," we had the chance to talk to project managers that are responsible for implementing 10,000 - 20,000 sqft build-outs in colocation facilities. It was clear to us that a few key trends have emerged:

• Whenever possible contractors recommend the use of pre-terminated copper and fiber cabling.  The benefits of utilizing these components include cost reduction, on-time delivery and the project is easier to manage.
• Although they are installing MPO cassettes on some jobs the cost premium often scares customers away.
• More OM3 fiber is being installed than OM4.  The up-sell to OM4 is difficult since OM3 covers the distances that are typically seen in these facilities (300m at 10G).
• A majority of the copper cabling is being installed is Category 6. 

A typical colocation lease averages about eight years.  Since the clients don't know what their requirements will be in this timeframe, they are less likely to make decisions that "future-proof" the installation for reuse with upgraded active equipment.  Tight budgets further preclude the addition of higher performing cabling products.  They would prefer to re-cable in the future than to pay for it now.  Since many of these installations are based on Top-of-Rack (ToR) architecture, re-cabling is viewed as a much simpler thing to do than to install new equipment when the cabinets are stuffed full of cabling. In view of this, we project that Category 6A and Category 7 cabling will only see very slow growth and that OM3 will be the mainstay over the next few years.

Telecom Exchange

I recently attended an event in New York City – Telecom Exchange. The format was originally developed by Hunter Newby and Rory Cutaia when they were at Telx. This year it was hosted by Jamie Scotto & Associates (JSA). Unlike most trade shows, this affair puts large and small companies on equal footing. In order to provide a “network-neutral” environment, JSA arranged the exhibit tables in alphabetical order and they were the same size with the same-sized branding. No giveaways were allowed at the tables. To be frank, to me it was a refreshing change. Instead of spotlighting the next new thing, the event forced you to focus on networking with industry players and real business opportunities.

Some of my thoughts from my meetings:

• Containerized/modularized data centers:  One prominent executive from a data center connectivity supplier said to me:  “Brick and mortar data centers are dead.” We only had a short time to expand on this comment, but what I think he meant was that data center operators will need to move to more modular solutions in order to lower their PUE. According to him, if you move all your high-density applications to a containerized solution, your PUE can be as low as 1.1, whereas, any traditional building would be hard to get below a PUE of 1.5. His premise is that companies will need to lower their total cost of ownership of their data center and therefore will move to these solutions or be out of business. He hasn’t convinced me yet, but I intend to do some more research on the subject.
• AlliedFiber (AF) and Dupont Fabros Technology (DFT):  Allied Fiber is known for connecting data centers nationwide, but has never connected the “last mile” into the facility. That has now changed. AF and Dupont Fabros have struck a deal for AF to connect into DFT’s Piscataway, New Jersey facility with a straight path to Chicago, bypassing Manhattan. The agreement gives AF access to DFTs underground fiber ducting and DFT access to AFs direct fiber link to Chicago, lowering latency for both providers.
• EtherCloud: Tinet, A Neutral Tandem Company, has now taken its Ethernet Exchange one step further. With its EtherCloud offering, it can provide end-to-end international connectivity to any company. It allows global coverage using VPLS through Juniper equipment in the core and Cisco in the access. Tinet is one of less than a handful of companies that can now provide direct Ethernet services on three continents.
• Global reach:  Telehouse America is known for its data center and managed services business in the US, but is quickly growing its reach internationally. It now has facilities on four continents – Asia, Europe, North America and Africa. Similar to Tinet, Telehouse is building out its Ethernet networks globally.

So what does all of this have to do with optical components? It shows that not only are there opportunities within the data center, but also new ones in the telecom/long haul market.

Thoughts on OFC/NFOEC

I’ve never been enamored with Los Angeles, but when OFC/NFOEC decides to go there, I really have no choice but to follow them. OFC/NFOEC is the premier optical components conference and is starting to move its way up the food chain again. Those of us who have been in the industry since before 2000 know that equipment manufacturers and service providers were regularly a part of OFC. But after 2000, this changed and the optical value chain was split – components relegated to OFC and/or NFOEC (until they combined into one), equipment manufacturers concentrating on Interop and Supercom and service providers opting for Supercom. Now, Supercom has closed its doors and Interop has become more and more software-centric. So both OEMs and service providers are looking for a trade show of value and they may have found it in OFC/NFOEC.

Notable communications equipment OFC/NFOEC exhibitors this year were ADVA, Ciena, Cisco, Fujitsu, Hitachi, Huawei, Infinera, Juniper, Mellanox Technolgies, Nokia Siemens Networks and Optelian. Many others gave speeches including Alcatel-Lucent, ADVA, Brocade, Ciena, Cisco, Cray, Force10 Networks, Fujitsu, Hewlett Packard, IBM, Infinera, Juniper, Nokia Siemens Networks and SunLabs/Oracle. Service providers participated by way of technical and business presentations as well – among them were AT&T, Deutsche Telecom, NTT and Verizon.

End users of networking equipment even showed up – NYSE Euronext, USA provided the plenary speaker for the Service Provider Summit and Facebook and Google again told us how much more bandwidth they need – Terabit Ethernet. There was an entire afternoon dedicated to large data center business issues at The Optical Business Forum, which included speakers from Abovenet, Allied Fiber, CENX, Equinix, Juniper, PacketExchange, Verizon, XO Communications Zayo Bandwidth and Zayo Networks.

So this seems to be a transition year for OFC/NFOEC where it has started to include more practical programming to expand its audience into data communications, data centers and up the value chain with equipment manufacturers and service providers. It appears to be working too, because the attendance was up by all accounts.

I’ll review some exciting new developments by systems and components suppliers in future posts.

Bringing the Optical Networking Supply Chain Back Together: The Optical Business Forum at OFC/NFOEC

As the networking equipment supply chain has broken up into intellectual property suppliers, semiconductor suppliers, optical component manufacturers, and full system manufacturers, telecom carriers are often 2-4 steps away from the technologies that can alter the economics of running their networks. This was not the case ten years ago during OFC's peak attendance years when many equipment vendors still owned components divisions. Yet as the large carriers turn to business accounts to cover the costs of expanding their optical networks and increasingly depend on technologies from their suppliers' supplier's suppliers, there has been no central trade event where they can gather to meet with this expanding set of companies entering their networks, or to examine the latest advances in optical technologies. OFC/NFOEC has seized the opportunity to bring the telecom supply chain back together to serve the community with much greater depth than any other show. It now has The Optical Business Forum.

Major telecom providers are in significant competition with each other for large business accounts. With wireless taking up a greater share of telephony revenue, and households having limited budgets to spend on service regardless of bandwidth offered, service providers see their future lying with the business customer. While they do not publicize much of their activity with commercial accounts due to regulatory pressure, businesses are a major factor behind carriers expanding the capacity of their optical networks.

Verizon's sale of its northern New England and West Virginia lines was carried out because of the limited base of commercial accounts in those regions. AT&T built nine 40-Gigabit MPLS nodes outside of its incumbent territory, where it does not offer U-Verse or any video service, in order to increase capacity for its corporate data and IP/VPN customers. It has kept quiet in the press regarding this upgrade, in spite of tens of millions invested. In an addition to corporations continuing to make heavy investments in bandwidth, carrier IP cores are carrying more traffic, with Petabytes transferred still growing 30 to 40% a year. And in order to maintain their dividends, as well as their returns on capital investments, carriers need revenue from the high margin, high budget business customer.

The Optical Business Forum was developed by analysts associated with DataCenterStocks.com. It is the first forum in what we’re hoping to be a yearly affair covering the most important topics in the area of optical transport for businesses data communications. This year’s summit is on the exhibit floor and consists of a key note address on High-Bandwidth Ethernet Services given by Rajiv Datta, Senior Vice President and Chief Technology Officer of AboveNet Inc. Following the keynote are three focused sessions:

1. Who is Buying Optical Bandwidth Services?
2. The Economics & Business Case for Connecting Data Centers
3. Carrier Ethernet Exchanges

The program includes speakers from AboveNet, Allied Fiber, CENX, Equinix, Juniper Networks, Packet Exchange, Telx Ethernet Exchange, Verizon Digital Media Services, XO Communications, Zayo Fiber Solutions and zColo. Moderators include Craig Clausen from NPRG, Michael Howard from Infonetics and myself.

Join us at OFC/NFOEC and hear what you’re customers’ customers are saying.

10GBASE-T versus 10GBASE-SR – Tradeoffs in Costs Make the Difference

The 10GBASE-T standard was finalized some four years ago now, but, as I’ve mentioned before, equipment using these ports is really just starting to be deployed in actual networks. The main reason being that you couldn’t get a switch with these ports on it. So early implementations of 10G are 10GBASE-SR, 10GBASE-LR or 10GBASE-LRM, with the vast majority now being the SR. But now that switch manufacturers the likes of Blade Networks, Cisco and Extreme Networks have products offering up to 24 ports of 10GBASE-T, the market dynamics may change.

With Ethernet, history tends to repeat itself so let’s take a minute to review what happened at Gigabit. Early products were 1000BASE-CX, SX and LX because the 100m 1000BASE-T had not yet been standardized. But, as soon as it was and the switch manufacturers started adopting it, it took over the shorter-reach Gigabit Ethernet market. In fact, it still dominates today.

So, why would 10GBASE-T be any different? Well, my belief is that eventually, it won’t. Even though data center managers concerns have shifted from space, to power availability per rack and cooling hot spots, when they see the price tag difference between SR and T (still about 2:1 per installed port), it causes them to pause and rethink the T scenario. So although data center managers want to reduce their headaches with fat CAT6A cables, most are still not willing to pay that much more for the optical solution until they have to because of distances. So even though the T ports may push electricity bills up, for most, the increase isn’t significant enough to justify the up-front cost of SR.

10G Copper versus Fiber – Is Power Consumption Really the Issue?

For decades now, fiber has been slated to take over the data networking world, but somehow, some way, copper keeps reinventing itself. But are the ways in which copper can compensate for its lower bandwidth capacity coming to an end at 10G due to what seem to be astronomical power consumption issues? Probably not. I have listened to the rhetoric from the fiber-optic companies for more than five years now, and have conducted my own research to see if what they say is true. Their argument was that at 8 to 14W per port, copper is just too costly. But, now that the chips have reduced power consumption to less than 3W per port, 10GBASE-T is a viable data center networking solution. Actually, even at 14W per port, it was viable, just not practical for switch manufacturers to incorporate in their designs because they couldn’t get the port density they needed and actually have room to cool the devices. Now that doesn’t seem to be an issue, evidenced by 24-port 10GBASE-T configurations that have been released by all the major players.

I believe decisions on copper versus fiber will be made around other parameters as well, such as latency. In a recent study, Data Center Cabling Cost Analysis - Copper Still Has Its Place, that we released at DataCenterStocks.com, we looked at the cost of 10G copper versus fiber and added in the higher power consumption. Using the specific example focused on a rack of EoR Cisco switches, copper was still more cost-effective even when considering higher electricity costs.

But our next area to study will be a rack of servers with a ToR switch. In this scenario, the power consumption difference may be enough to justify the cost of installing fiber over copper. The above referenced report and this next research are part of a series of research reports for our Data Center Network Operator Service.

Why Polarity Matters (Part 2)

If you’ve read my previous posts on the subject, you know that polarity can be a tricky matter and it’s even more complicated when you try to choose it for your data center cabling. You really have to choose based on several factors:

1. Patch cords – method A has two different patch cords that you have to stock, but the upside is that it’s pretty simple to follow where the signal is going and if you happen to be out of one type of patch cord, you can really take the one you have and just flip the fibers as a temporary fix until you can get the other patch cords. Of course this isn’t recommended, but if you’re in a bind and need to get up-and-running right away, it will work. With methods B and C you have the same patch cord on each end so no need to worry about this, but if you happen to have the wrong cassettes or backbone, nothing will work and you'll have to wait to get the correct ones.
2. Cassettes and backbone cables – you need to make sure you buy all of one method of polarity or your system won’t work. If you’re concerned about supply, all three polarity methods are available from multiple vendors, but Method A is “preferred” by most.
3. Upgradability – this is where it can get dicey. Typically your pre-terminated assemblies are running Gigabit applications today and a few may be running 10G. Any of the polarities will work at these data rates. But when you move to 40/100G, methods A and B have straight forward paths, while C does not. Also, you’ll want to make sure you use the highest grade of LOMF available, which is OM4 – this will give you the best chance of being able to reuse your backbones up to 125m. If you need something longer, you’ll need to go to SMF.

If you are thinking about installing pre-terminated cassette-based assemblies now for 10G with an upgrade path to 40 and 100G, you need to consider the polarity method you use. Unlike today's 2-fiber configurations, with one send and one receive, the standards for 40G and 100G Ethernet implementations use multiple parallel 10G connections that are multiplexed. While 40/100G equipment vendors will tell you that polarity is not an issue, care must be taken if you want to reuse this installed base.

40G will use four 10G fibers to send and four 10G fibers to receive, while 100G uses either four 25G fibers or ten 10G fibers in each direction. Because 40 and 100G will be using the MPO connector, if the polarity method is carefully chosen, you will be able to reuse your backbone cables. This is enabled by the fact that the IEEE took much care in specifying the system so that you can connect any transmit within a connection on one end of the channel to any receive on the other end.

Those selecting fiber to support 10G now and 40G in the near future need to understand what will be involved in transitioning and repurposing their cable plant. In order to upgrade using method A, you can replace the cassettes with MPO-to-MPO patch panels and MPO-to-MPO patch cords and it will enable flexibility to address moves, adds and changes as well as promoting proper installation best practices. The polarity flip will need to be accomplished in either an A-to-A patch cord or possibly with a key up/key down patch panel.

Method B multimode backbone cables can also readily support 40G applications. For a structured cabling approach, method B will still use a patch panel and patch cords, though as with current method B, both patch cords could be A-to-B configuration. While Method C backbones could be used, they are not recommended for 40G as completing the channel involves complex patch cord configurations.

It appears that 100G will use either the 12-fiber (4x25G) or the 24-fiber (10x10G) MPO connector. With transmits in the top row and receives in the bottom row, the connection will still be best made using a standardized structured cabling approach as described above.

There are many suppliers of pre-terminated optical assemblies including Belden, Berk-Tek, a Nexans Company, CommScope, Corning, Panduit, Siemon, Tyco Electronics NetConnect as well as many smaller shops that give quick-turn assemblies like Cxtec CablExpress and Compulink.