It’s a beautiful time of year in New Orleans for the top supercomputing companies to show their wares. While I wouldn’t consider SC10 exactly the place to sell optical components, there were a few new developments there. SCinet – the network that is always built at the top HPC conference – boasted 100G Ethernet as well as OTu4. Alcatel-Lucent, Ciena, Cisco, Force10 and Juniper, among others donated equipment to build this network. Module vendors Avago Technologies, Finisar and Reflex Photonics contributed QSFP and CFP 40G and 100G devices to the cause.
Meanwhile, the Ethernet alliance was showing two demonstrations in its booth – a converged network running FCoE and RoCE over 40GigE and 100GigE. Nineteen different vendors participated in this demo that was run by the University of New Hampshire Interoperability Lab. Both CFPs and QSFPs were used in this demo.
Some of you may wonder why I would attend SC10. I keep my eye on the HPC market because it usually indicates where the broader data center market will be in a few years. And, in fact, even medium-sized businesses’ data centers with higher computational needs are starting to resemble small HPC centers with their server clusters using top-of-rack switching.
Most of the top optical transceiver vendors and even some of the smaller ones see this market as an opportunity as well. While InfiniBand still uses a lot of copper interconnects, for 40G and 120G, this is changing. QSFP was the standout for 40G IB displays and CXP AOCs were shown for 120G as well. Avago Technologies was the first to announce a CXP module at the show.
Some believe that the CXP will be short-lived because there is progress being made on 4x25 technologies – Luxtera announced its 25G receivers to go with its 25G transmitter that it announced earlier this year. But it will still be a few years before all of the components for 25G will be ready for system’s developers to spec in. Tyco Electronics had a demonstration at their booth showing it is possible to run 28G over eight inches of a PCB, but this was still a prototype. And Xilinx has announced a chip for 28G electrical transceivers that can be used with this board design. But, none of these devices are even being tested by equipment manufacturers yet and the CXP has already been adopted by a few. So I think the CXP may have more life in it than some people may think.
Thursday, November 18, 2010
Tuesday, November 9, 2010
PCIe – An I/O Optical Interconnect Soon?
The Peripheral Component Interconnect (PCI) is that bus in computers that connects everything back to the processor. It has been around for as long as I can remember having workstations. But in recent years, it has been morphing in response to the need to connect to the processor at higher data rates.
PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.
PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.
CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.
Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.
Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.
PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.
PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.
CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.
Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.
Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.
Thursday, November 4, 2010
Opportunities for CXP AOCs and Transceivers
For those of you that believe that 100-Gigabit Ethernet is just around the corner, I have a bridge I want to sell you. But seriously, we haven’t even seen the height of 10-Gigabit Ethernet adoption yet, and there are some equipment companies saying they will sell 100’s-of-thousands of CXP 100GBASE-SR ports in 2011. Are you kidding? What is the application and where is the need?
First, 10GE has taken more than eight years to get to a million ports – we believe it will take 40G and 100G even longer. Second, even for clustering applications, which could potentially drive demand faster, 100GE port-adoption won’t be that quick. Ethernet architecture is different than the InfiniBand (IB) one – the density of an IB director-type switch provides over a Terabit per second, whereas the newly released 40GE ones are around 250G (due to both slower data rate and fewer ports). IB is also based on a CLOS architecture where you have equal bandwidth everywhere, while Ethernet is more often used in an aggregated network so ends up having a lot less higher-speed ports than lower speed ones. This is further supported by clustering applications that use ToR switches that are currently Gigabit connections to the servers with 10G uplinks to the network core. These will be upgraded to 10G downlinks and 40G uplinks first and this won’t happen quickly.
While several IP router manufacturers claim to have the need for 100’s of thousands of 100GBASE-SR CXP ports in 2011, I have found no evidence of this. Who are their customers? In fact, even those companies that could use 100G ports today, i.e. Google, Facebook , IXCs, etc., would need six months to a year to evaluate IP router products before they would deploy them. Since these devices do not yet exist, the reality is that the market will really not begin to materialize until at least 2012. Right now, the majority of router connections are still Gigabit Ethernet or OC-48 (2.5G) or below with OC-192 (10G) or 10GE being implemented on an as-needed basis. Until routers transition through 10G, then probably 40G, 100G installations will be few and far between.
But, there is a market for CXP AOCs today – InfiniBand. This is becoming a volume market now and will continue to be the best opportunity for CXP AOCs for at least the next few years and probably over the lifetime of the CXP products. In fact, we expect the volume of InfiniBand CXP AOCs to be at about six million by 2015. By comparison, the total volume of Ethernet CXP AOCs is expected to be less than 100-thousand. While 100G Ethernet clustering applications will initially use CXP AOCs, customers in these markets prefer to use pluggable modules mainly because they are used to structured cabling solutions and like their flexibility and ease of use, so AOCs will quickly give way to pluggable modules as they are developed. 100GE CXP ports may eventually eclipse InfiniBand once it permeates most data center distribution and core networks, but this will take longer than any of these equipment vendors anticipate I think.
First, 10GE has taken more than eight years to get to a million ports – we believe it will take 40G and 100G even longer. Second, even for clustering applications, which could potentially drive demand faster, 100GE port-adoption won’t be that quick. Ethernet architecture is different than the InfiniBand (IB) one – the density of an IB director-type switch provides over a Terabit per second, whereas the newly released 40GE ones are around 250G (due to both slower data rate and fewer ports). IB is also based on a CLOS architecture where you have equal bandwidth everywhere, while Ethernet is more often used in an aggregated network so ends up having a lot less higher-speed ports than lower speed ones. This is further supported by clustering applications that use ToR switches that are currently Gigabit connections to the servers with 10G uplinks to the network core. These will be upgraded to 10G downlinks and 40G uplinks first and this won’t happen quickly.
While several IP router manufacturers claim to have the need for 100’s of thousands of 100GBASE-SR CXP ports in 2011, I have found no evidence of this. Who are their customers? In fact, even those companies that could use 100G ports today, i.e. Google, Facebook , IXCs, etc., would need six months to a year to evaluate IP router products before they would deploy them. Since these devices do not yet exist, the reality is that the market will really not begin to materialize until at least 2012. Right now, the majority of router connections are still Gigabit Ethernet or OC-48 (2.5G) or below with OC-192 (10G) or 10GE being implemented on an as-needed basis. Until routers transition through 10G, then probably 40G, 100G installations will be few and far between.
But, there is a market for CXP AOCs today – InfiniBand. This is becoming a volume market now and will continue to be the best opportunity for CXP AOCs for at least the next few years and probably over the lifetime of the CXP products. In fact, we expect the volume of InfiniBand CXP AOCs to be at about six million by 2015. By comparison, the total volume of Ethernet CXP AOCs is expected to be less than 100-thousand. While 100G Ethernet clustering applications will initially use CXP AOCs, customers in these markets prefer to use pluggable modules mainly because they are used to structured cabling solutions and like their flexibility and ease of use, so AOCs will quickly give way to pluggable modules as they are developed. 100GE CXP ports may eventually eclipse InfiniBand once it permeates most data center distribution and core networks, but this will take longer than any of these equipment vendors anticipate I think.
Monday, October 25, 2010
Interop, New York - Software Show Now
I hadn’t been to Interop since 2003 and I’d never been to the New York show so I decided this year I would give it a shot. I was woefully disappointed. I was expecting to see great product demonstrations from all the top equipment manufacturers, but instead received inquiries for meetings from software vendors who didn’t even bother to see that I cover data centers, optical components, structured cabling and interconnects.
The exhibit hall only had eight rows. Cisco was there, but half of its booth was taken up by its channel partners and the other half had virtual demos, not actual equipment running. Brocade was there, but had a much smaller booth and pretty much legacy equipment on a tabletop display. Most telling of course were the companies that didn’t participate in the exhibition – Extreme Networks and IBM to name just two.
Some of the programming was interesting, though, and maybe made it worth the travel costs. I sat in on the second day of the Enterprise Cloud Summit so actually got to meet some of the gurus in the industry of cloud computing. I also sat in on the “Evaluating New Data Center LAN Architectures” technical session which was a panel of equipment manufacturers that responded to an RFI from Boston Scientific for a data center expansion project. Interesting to note is that while Cisco was asked to respond, it did not. The panel consisted of Alcatel-Lucent, Extreme Networks, Force10 and Hewlett Packard. It is also interesting to note that the vendors responded with different architectures – some including top-of-rack solutions and others with end-of-row.
All-in-all, I think my time would have been better spent staying home and working on my optical components research…
The exhibit hall only had eight rows. Cisco was there, but half of its booth was taken up by its channel partners and the other half had virtual demos, not actual equipment running. Brocade was there, but had a much smaller booth and pretty much legacy equipment on a tabletop display. Most telling of course were the companies that didn’t participate in the exhibition – Extreme Networks and IBM to name just two.
Some of the programming was interesting, though, and maybe made it worth the travel costs. I sat in on the second day of the Enterprise Cloud Summit so actually got to meet some of the gurus in the industry of cloud computing. I also sat in on the “Evaluating New Data Center LAN Architectures” technical session which was a panel of equipment manufacturers that responded to an RFI from Boston Scientific for a data center expansion project. Interesting to note is that while Cisco was asked to respond, it did not. The panel consisted of Alcatel-Lucent, Extreme Networks, Force10 and Hewlett Packard. It is also interesting to note that the vendors responded with different architectures – some including top-of-rack solutions and others with end-of-row.
All-in-all, I think my time would have been better spent staying home and working on my optical components research…
Thursday, October 14, 2010
SFP+ Markets
SFP+ is primarily being used in two markets currently – 10G Ethernet and 8G Fibre Channel. The Fibre Channel market is almost exclusively optical SFP+ modules while, the 10G Ethernet market is expected to see its largest growth in SFP+ direct-attach copper (DAC) products. SFP+ DAC will mainly be used in top-of-rack (ToR) switches that connect to servers within a rack. Multi-million port forecasts that are predicted over the next few years are predicated on an anticipated high-rate of adoption of this architecture in the data center. Even if this topology is used, once 10GBASE-T becomes more readily available, SFP+ DAC will be sharing its market with this less costly variant. Some OEMs believe quantities of each will be about the same, but I have my doubts. If history tells us anything, which it usually does, once the less expensive BASE-T variant takes hold, it usually pushes the more costly alternatives to much smaller volume.
But, right now it is a matter of timing. I am skeptical that the 10GBASE-T adoption in switches will be able to keep pace with the port-density need in the data center in the short-term. Both products will be needed in the long term – 10GBASE-T for inexpensive connections and 10GBASE-CR (SFP+ DAC) for lower latency, lower power consumption and more flexibility. Currently, if you want both copper and fiber ports on your 10G switch, you need to use SFP+ because there are no switches that contain both 10GBASE-T and optical.
Friday, October 8, 2010
DARPAs Chip-to-Chip Optical Interconnects (C2OI) Program
The C2OI DARPA program is funding on-going optical components projects. Its end goal is to “demonstrate optical interconnections between multiple silicon chips that will enable data communications between chips to be as seamless as data communication within a chip.”
This program grew out of work initially done by Agilent (now Avago Technologies) under the DARPA Parallel Optical Network Interconnect (PONI) project. Agilent/Avago developed a 30 Gbps transmitter (2.5 Gbps/lane) that was eventually standardized as the SNAP-12.
IBM (with help from Avago) extended the work originally done by Agilent/Avago into inter-chip connections and in 2009 achieved optical interconnection with 16 parallel lanes of 10G. By early 2010, IBM was extending this work into board-to-board applications which resulted in the new Avago MicroPOD™ product that was specifically designed for IBM’s POWER7™ supercomputer.
While it was designed for HPC server interconnects, the MicroPOD could be used for on-board or chip-to-chip interconnects as well. As mentioned in previous posts, the devices use a newly designed miniature detachable connector from US CONEC called PRIZM™ LightTurn™. The system has separate transmitter and receiver modules that are connected through a 12-fiber ribbon. Each lane supports up to 12.5 Gbps. It uses 850nm VCSEL and PIN diode arrays. The embedded modules can be used for any board-level or I/O-level application by either using two PRIZM LightTurn connectors or one PRIZM LightTurn and one MPO.
While MicroPOD is targeted at high-density HPC environments, a natural expansion of its market reach would be into switches and routers in high-density Ethernet data center environments. While this may not happen in the next few years, for me it looks like it could be a more cost-effective solution than say a 40G serial one.
This program grew out of work initially done by Agilent (now Avago Technologies) under the DARPA Parallel Optical Network Interconnect (PONI) project. Agilent/Avago developed a 30 Gbps transmitter (2.5 Gbps/lane) that was eventually standardized as the SNAP-12.
IBM (with help from Avago) extended the work originally done by Agilent/Avago into inter-chip connections and in 2009 achieved optical interconnection with 16 parallel lanes of 10G. By early 2010, IBM was extending this work into board-to-board applications which resulted in the new Avago MicroPOD™ product that was specifically designed for IBM’s POWER7™ supercomputer.
While it was designed for HPC server interconnects, the MicroPOD could be used for on-board or chip-to-chip interconnects as well. As mentioned in previous posts, the devices use a newly designed miniature detachable connector from US CONEC called PRIZM™ LightTurn™. The system has separate transmitter and receiver modules that are connected through a 12-fiber ribbon. Each lane supports up to 12.5 Gbps. It uses 850nm VCSEL and PIN diode arrays. The embedded modules can be used for any board-level or I/O-level application by either using two PRIZM LightTurn connectors or one PRIZM LightTurn and one MPO.
While MicroPOD is targeted at high-density HPC environments, a natural expansion of its market reach would be into switches and routers in high-density Ethernet data center environments. While this may not happen in the next few years, for me it looks like it could be a more cost-effective solution than say a 40G serial one.
Tuesday, October 5, 2010
DARPAs Ultraperformance Nanophotonic Intrachip Communitcations (UNIC)
UNIC is a DARPA-funded project that started in 2008 and is slated to run for about five years. Sun/Oracle, Kotura and Luxtera are working to develop this chip-to-chip "high-performance, CMOS-compatible photonic technology for high-throughput, non-blocking and power-efficient intrachip photonic communications networks."
The first application of such technology is targeted for optical interconnects for microprocessors. The goal is to replace high-performance computing clusters with computers that consist of these arrays of microprocessors interconnected by optics. Another goal of the project is to make sure the new devices are "compatible" with CMOS processes in order to also integrate the associated electronic devices. Using its now proven Silicon CMOS Photonics technology, Luxtera has developed transmitters and receivers for the project. Kotura supported the project with new low-power, high-speed modulators made of silicon photonics.
Potential new products are expected by the end of 2012. While these will be initial products, commercialization is not expected until quite some time later, perhaps not until 2016 or so. Meanwhile, Luxtera will continue to use its technology to sell 10, 40 and 100G transceivers and AOCs.
With data rates increasing beyond 10G, chip-to-chip, on-board and board-to-board optical interconnects will become progressively more significant. Even at 10G, traditional printed-circuit boards cannot support transmission beyond about 12 inches without needing re-timers. As I’ve mentioned in previous posts, instead of spending money to develop more exotic PCBs using complicated digital signal processing (DSP), it may be time to embrace optical interconnects for both board-level and chip-level.
The first application of such technology is targeted for optical interconnects for microprocessors. The goal is to replace high-performance computing clusters with computers that consist of these arrays of microprocessors interconnected by optics. Another goal of the project is to make sure the new devices are "compatible" with CMOS processes in order to also integrate the associated electronic devices. Using its now proven Silicon CMOS Photonics technology, Luxtera has developed transmitters and receivers for the project. Kotura supported the project with new low-power, high-speed modulators made of silicon photonics.
Potential new products are expected by the end of 2012. While these will be initial products, commercialization is not expected until quite some time later, perhaps not until 2016 or so. Meanwhile, Luxtera will continue to use its technology to sell 10, 40 and 100G transceivers and AOCs.
With data rates increasing beyond 10G, chip-to-chip, on-board and board-to-board optical interconnects will become progressively more significant. Even at 10G, traditional printed-circuit boards cannot support transmission beyond about 12 inches without needing re-timers. As I’ve mentioned in previous posts, instead of spending money to develop more exotic PCBs using complicated digital signal processing (DSP), it may be time to embrace optical interconnects for both board-level and chip-level.
Subscribe to:
Posts (Atom)