- Connecting a 40GBASE-SR4 Ethernet port using its QSFP+ to four standard 10GBASE-SR Ethernet ports with its SFP+ modules. On the surface, this seems pretty easy to do, until you realize that the specifications for the transmitters and receivers in these devices have are very different. The 10G devices could easily overpower the 40G receiver if it’s not designed to handle the higher power. Avago has solved this issue with its parts and hopes to be able to be interoperable with anyone’s transceivers in the near future.
- A VCSEL-based 25G short-wavelength SFP+ working prototype. At first I was puzzled about this because I couldn’t figure out the application. Well, it turns out there really is none for the 25G part, yet, but showing that it could be done makes you realize that 32G Fibre Channel applications using the SFP+ may not be as far-fetched as we think. And, perhaps we can get a 100GBASE-SR4 (4x25G that isn't in the IEEE standard yet) solution soon.
Showing posts with label 10G. Show all posts
Showing posts with label 10G. Show all posts
Monday, March 21, 2011
Avago’s Interesting Demos at OFC/NFOEC
As always, the top transceiver manufacturers were represented at OFC, but of the top three datacom transceiver providers, Avago Technologies stood out to me. They had two significant demonstrations:
Tuesday, November 9, 2010
PCIe – An I/O Optical Interconnect Soon?
The Peripheral Component Interconnect (PCI) is that bus in computers that connects everything back to the processor. It has been around for as long as I can remember having workstations. But in recent years, it has been morphing in response to the need to connect to the processor at higher data rates.
PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.
PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.
CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.
Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.
Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.
PCI Express (PCIe) GEN1 defined PCIe over cable implementations in 2007. Molex was instrumental in helping to define this and up until now, it has been a purely copper solution using its iPass™ connection system. This system has been used mainly for “inside-the-box” applications first at 2.5G (GEN1) and then at 5G (GEN2). The adoption rate for PCIe over cable has been slow. It is mainly used for high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment and medical imaging systems.
PCIe GEN3 is running at 8G and some physical layer component vendors are looking to use an optical solution instead of the current copper cable along with trying to move it into a true I/O technology for the data center connections – servers to switches and eventually storage. While component vendors are excited about these applications, mainstream OEMs do not seem interested in supporting it. I believe it is because they see it as a threat to their Ethernet equipment revenue.
CXP AOCs seem to be a perfect fit for this GEN3 version of PCIe, but neither equipment manufacturers nor component suppliers believe it will reach the price level needed for this low-cost system. It is expected that the optical interconnect should cost 10’s of dollars, not 100’s. However, CXP AOCs may be used for PCIe GEN3 prototype testing for proof of concept. But if the first demonstrations are any indication, this will not be the case. PCIe GEN3 over optical cable was recently shown by PLX Technology using just a one channel optical engine next to its GEN3 chip with standard LC jumpers. PLX and other vendors are looking towards using optical engines with standard MPO patch cords to extend this to 4x and 8x implementations.
Columbia University and McGill University also demonstrated PCIe GEN3, but with eight lanes over a WDM optical interconnect. This is obviously much more expensive than even the CXP AOCs and is not expected to get any traction in real networks.
Another factor against PCIe as an I/O is the end user. In a data center, there are typically three types of support personnel – networking (switches), storage/server managers and data center managers. While the server managers are familiar with PCIe from an “inside-the-box” perspective, I’m not sure they are ready to replace their Ethernet connections outside the box. And, the others may have heard of PCIe, but probably aren’t open to changing their Ethernet connections either. They can run 10-Gigabit on their Ethernet connections today so really don’t see a need to learn an entirely new type of interconnect in their data center. In fact, they are all leaning towards consolidation instead – getting to one network throughout their data center like FCoE. But, as I've stated in previous posts, this won’t happen until it is shown to be more cost effective than just using 10GigE to connect their LANs and SANs. The fact that PCIe I/O could be cheaper than Ethernet may not be enough because Ethernet is pretty cost-effective itself and has the luxury of being the installed base.
Thursday, October 14, 2010
SFP+ Markets
SFP+ is primarily being used in two markets currently – 10G Ethernet and 8G Fibre Channel. The Fibre Channel market is almost exclusively optical SFP+ modules while, the 10G Ethernet market is expected to see its largest growth in SFP+ direct-attach copper (DAC) products. SFP+ DAC will mainly be used in top-of-rack (ToR) switches that connect to servers within a rack. Multi-million port forecasts that are predicted over the next few years are predicated on an anticipated high-rate of adoption of this architecture in the data center. Even if this topology is used, once 10GBASE-T becomes more readily available, SFP+ DAC will be sharing its market with this less costly variant. Some OEMs believe quantities of each will be about the same, but I have my doubts. If history tells us anything, which it usually does, once the less expensive BASE-T variant takes hold, it usually pushes the more costly alternatives to much smaller volume.
But, right now it is a matter of timing. I am skeptical that the 10GBASE-T adoption in switches will be able to keep pace with the port-density need in the data center in the short-term. Both products will be needed in the long term – 10GBASE-T for inexpensive connections and 10GBASE-CR (SFP+ DAC) for lower latency, lower power consumption and more flexibility. Currently, if you want both copper and fiber ports on your 10G switch, you need to use SFP+ because there are no switches that contain both 10GBASE-T and optical.
Thursday, September 16, 2010
10GBASE-T versus 10GBASE-SR – Tradeoffs in Costs Make the Difference
The 10GBASE-T standard was finalized some four years ago now, but, as I’ve mentioned before, equipment using these ports is really just starting to be deployed in actual networks. The main reason being that you couldn’t get a switch with these ports on it. So early implementations of 10G are 10GBASE-SR, 10GBASE-LR or 10GBASE-LRM, with the vast majority now being the SR. But now that switch manufacturers the likes of Blade Networks, Cisco and Extreme Networks have products offering up to 24 ports of 10GBASE-T, the market dynamics may change.
With Ethernet, history tends to repeat itself so let’s take a minute to review what happened at Gigabit. Early products were 1000BASE-CX, SX and LX because the 100m 1000BASE-T had not yet been standardized. But, as soon as it was and the switch manufacturers started adopting it, it took over the shorter-reach Gigabit Ethernet market. In fact, it still dominates today.
So, why would 10GBASE-T be any different? Well, my belief is that eventually, it won’t. Even though data center managers concerns have shifted from space, to power availability per rack and cooling hot spots, when they see the price tag difference between SR and T (still about 2:1 per installed port), it causes them to pause and rethink the T scenario. So although data center managers want to reduce their headaches with fat CAT6A cables, most are still not willing to pay that much more for the optical solution until they have to because of distances. So even though the T ports may push electricity bills up, for most, the increase isn’t significant enough to justify the up-front cost of SR.
With Ethernet, history tends to repeat itself so let’s take a minute to review what happened at Gigabit. Early products were 1000BASE-CX, SX and LX because the 100m 1000BASE-T had not yet been standardized. But, as soon as it was and the switch manufacturers started adopting it, it took over the shorter-reach Gigabit Ethernet market. In fact, it still dominates today.
So, why would 10GBASE-T be any different? Well, my belief is that eventually, it won’t. Even though data center managers concerns have shifted from space, to power availability per rack and cooling hot spots, when they see the price tag difference between SR and T (still about 2:1 per installed port), it causes them to pause and rethink the T scenario. So although data center managers want to reduce their headaches with fat CAT6A cables, most are still not willing to pay that much more for the optical solution until they have to because of distances. So even though the T ports may push electricity bills up, for most, the increase isn’t significant enough to justify the up-front cost of SR.
Friday, September 10, 2010
10G Copper versus Fiber – Is Power Consumption Really the Issue?
For decades now, fiber has been slated to take over the data networking world, but somehow, some way, copper keeps reinventing itself. But are the ways in which copper can compensate for its lower bandwidth capacity coming to an end at 10G due to what seem to be astronomical power consumption issues? Probably not. I have listened to the rhetoric from the fiber-optic companies for more than five years now, and have conducted my own research to see if what they say is true. Their argument was that at 8 to 14W per port, copper is just too costly. But, now that the chips have reduced power consumption to less than 3W per port, 10GBASE-T is a viable data center networking solution. Actually, even at 14W per port, it was viable, just not practical for switch manufacturers to incorporate in their designs because they couldn’t get the port density they needed and actually have room to cool the devices. Now that doesn’t seem to be an issue, evidenced by 24-port 10GBASE-T configurations that have been released by all the major players.
I believe decisions on copper versus fiber will be made around other parameters as well, such as latency. In a recent study, Data Center Cabling Cost Analysis - Copper Still Has Its Place, that we released at DataCenterStocks.com, we looked at the cost of 10G copper versus fiber and added in the higher power consumption. Using the specific example focused on a rack of EoR Cisco switches, copper was still more cost-effective even when considering higher electricity costs.
But our next area to study will be a rack of servers with a ToR switch. In this scenario, the power consumption difference may be enough to justify the cost of installing fiber over copper. The above referenced report and this next research are part of a series of research reports for our Data Center Network Operator Service.
I believe decisions on copper versus fiber will be made around other parameters as well, such as latency. In a recent study, Data Center Cabling Cost Analysis - Copper Still Has Its Place, that we released at DataCenterStocks.com, we looked at the cost of 10G copper versus fiber and added in the higher power consumption. Using the specific example focused on a rack of EoR Cisco switches, copper was still more cost-effective even when considering higher electricity costs.
But our next area to study will be a rack of servers with a ToR switch. In this scenario, the power consumption difference may be enough to justify the cost of installing fiber over copper. The above referenced report and this next research are part of a series of research reports for our Data Center Network Operator Service.
Friday, July 30, 2010
Laser-optimized Multi-mode Fiber (LOMF)
It occurred to me as I was writing the last post that many of you may not be aware of the different grades of multi-mode fiber and that for the purposes of this blog, it would be good to present the differences so many of my points can be thoroughly understood.
Right now, there are three standardized types of LOMF as well as what most of us in the industry call FDDI-grade fiber, which is not laser optimized. So first, what does laser-optimized actually mean? In basic terms, it just means that the fiber was designed to be used with lasers, and in the case of MMF, typically VCSELs. FDDI-grade fiber pre-dated the use of VCSELs so is not laser-optimized - it was intended for utilization with LEDs. Lasers were adopted as the light source of choice when scientists and engineers realized that LEDs became very unstable when trying to modulate them at data rates beyond 100 Mbps. They originally tried to use the same lasers that were being used in CD players, but these turned out to be unstable at Gigabit data rates as well. In the early 1990s, the development of the VCSEL enabled these higher data rates.
As the light sources evolved, the fiber progressed with them. So, for 850nm operation today we have four choices:
All this leads to a very important point – when specifying a cabling system for your networks and data centers, it is important to understand not only the fiber you’re going to install, but also the equipment you’re trying to connect. Just because you're "only" installing Gigabit systems and you've used OM1 fiber for years, doesn't mean it's the best solution (or even the most economical) for today and tomorrow.
Right now, there are three standardized types of LOMF as well as what most of us in the industry call FDDI-grade fiber, which is not laser optimized. So first, what does laser-optimized actually mean? In basic terms, it just means that the fiber was designed to be used with lasers, and in the case of MMF, typically VCSELs. FDDI-grade fiber pre-dated the use of VCSELs so is not laser-optimized - it was intended for utilization with LEDs. Lasers were adopted as the light source of choice when scientists and engineers realized that LEDs became very unstable when trying to modulate them at data rates beyond 100 Mbps. They originally tried to use the same lasers that were being used in CD players, but these turned out to be unstable at Gigabit data rates as well. In the early 1990s, the development of the VCSEL enabled these higher data rates.
As the light sources evolved, the fiber progressed with them. So, for 850nm operation today we have four choices:
- OM1 (FDDI): Minimum OFL Bandwidth of 200 MHz•km; 10G Minimum Optical Reach of 33m
- OM2: Minimum OFL BW of 500; 10G Minimum Optical Reach of 82m
- OM3: Minimum OFL BW of 1500; 10G Minimum Optical Reach of 300m
- OM4: Minimum OFL BW of 3500; 10G Minimum Optical Reach of 550m
All this leads to a very important point – when specifying a cabling system for your networks and data centers, it is important to understand not only the fiber you’re going to install, but also the equipment you’re trying to connect. Just because you're "only" installing Gigabit systems and you've used OM1 fiber for years, doesn't mean it's the best solution (or even the most economical) for today and tomorrow.
Wednesday, July 28, 2010
Other 10GigE Transceiver Markets – ER, LX4 and LRM
Some of you may have noticed that in my last post I neglected to talk about three other 10-Gigabit Ethernet variants – LX4, LRM and ER. That’s because the content was already long enough and I wanted to focus on the volume data center applications. Now, I’ll discuss the others.
10GBASE-ER runs over SMF at 1550nm for up to 40km. While there are a few service providers that might choose to do this, the vast majority of them choose WDM through the OTN instead. There may be some private networks that have need for this variant as well which is what prompted the IEEE include it in the standard. These transceivers are priced out of typical budgets for the average enterprise at over $4000.
10GBASE-LX4 was originally introduced to address installed base of FDDI-grade (not laser-optimized) MMF, but can also be used with higher-grades of LOMF as well as SMF. It uses CWDM to send four wavelengths (thus the X4) between 1269.0 to 1355.9nm running at data rates of 2.5 Gbps each. LX4 modules are available in both X2 and XENPAK form factors and from at least a couple of sources such as Emcore and Hitachi Cable. As you can imagine, because these devices have four lasers and four detectors with their associated electronics, they cost appreciably more than either the SR or LR transceivers. The module alone retails for about $2000, which means that the per port cost would probably around $2500. But, you most likely wouldn’t fill your switch with these modules of course; you would only use them as needed where you wanted to re-use existing installed fiber.
In response to the high priced LX4, 10GBASE-LRM variant was developed. It was enabled by chip companies such as Clariphy and Scintera with some new technology, electronic dispersion compensation (EDC), that could push 10G serial to longer distances on MMF. It took a while to develop the standard and the consequence was that LX4 really took much of the market it was intended for. However, once products were released and supported by the top-tier transceiver manufacturers (Avago, Finisar, Opnext), it has really taken much of the business away from LX4. LRM modules are now available in SFP+ packages as well, which clearly indicates the vendors think there will be an ongoing market for them.
One note of caution – if you intend to use either the LX4 or the LRM modules, you need to make sure that both ends of your channel have them, otherwise it won’t work.
10GBASE-ER runs over SMF at 1550nm for up to 40km. While there are a few service providers that might choose to do this, the vast majority of them choose WDM through the OTN instead. There may be some private networks that have need for this variant as well which is what prompted the IEEE include it in the standard. These transceivers are priced out of typical budgets for the average enterprise at over $4000.
10GBASE-LX4 was originally introduced to address installed base of FDDI-grade (not laser-optimized) MMF, but can also be used with higher-grades of LOMF as well as SMF. It uses CWDM to send four wavelengths (thus the X4) between 1269.0 to 1355.9nm running at data rates of 2.5 Gbps each. LX4 modules are available in both X2 and XENPAK form factors and from at least a couple of sources such as Emcore and Hitachi Cable. As you can imagine, because these devices have four lasers and four detectors with their associated electronics, they cost appreciably more than either the SR or LR transceivers. The module alone retails for about $2000, which means that the per port cost would probably around $2500. But, you most likely wouldn’t fill your switch with these modules of course; you would only use them as needed where you wanted to re-use existing installed fiber.
In response to the high priced LX4, 10GBASE-LRM variant was developed. It was enabled by chip companies such as Clariphy and Scintera with some new technology, electronic dispersion compensation (EDC), that could push 10G serial to longer distances on MMF. It took a while to develop the standard and the consequence was that LX4 really took much of the market it was intended for. However, once products were released and supported by the top-tier transceiver manufacturers (Avago, Finisar, Opnext), it has really taken much of the business away from LX4. LRM modules are now available in SFP+ packages as well, which clearly indicates the vendors think there will be an ongoing market for them.
One note of caution – if you intend to use either the LX4 or the LRM modules, you need to make sure that both ends of your channel have them, otherwise it won’t work.
Monday, July 26, 2010
Cost Effective 10G Data Center Networks
For networks running Gigabit Ethernet, it’s a no-brainer to use Category 5e or 6 cabling with low-cost copper switches for less than 100m connections because they are very reliable and cost about 40-percent less per port than short-wavelength optical ones. But for 10G, there are other factors to consider.
While 10GBASE-T ports are now supposedly available on switches (at least the top Ethernet switch vendors, Brocade, Cisco and Extreme Networks say they have them), is it really more cost effective to use these instead of 10GBASE-SR, 10GBASE-CX4 or 10GBASE-CR (SFP+ direct attach copper)? Well, again, it depends on what your network looks like and how well your data center electrical power and cooling structures are designed.
First, 10GBASE-CX4 is really a legacy product and is only available on a limited number of switches so you may want to rule this out right away. If you’re looking for higher density, but you can’t support too many higher power devices, I would opt for 10GBASE-SR because it has the lowest power consumption – usually less than 1W/port. And also, the useful life of LOMF is longer; it’s smaller so won’t take up as much space or block cooling airflow if installed under a raised floor.
If you don’t have a power or density problem and can make do with just a few 10G ports for a short distance, you may choose to use 10GBASE-CR (about $615/port). But, if you don’t have a power or density issue and you need to go longer than about 10m, you’ll still need to use 10GBASE-SR and if you need a reach of more than 300m you’ll need to either install OM4 cable (which will get you up to 600m in some cases) to use with your SR devices; or look at 10GBASE-LR modules ($2500/port) that will cost you about twice as much as the SR transceivers (about $1300/port). If your reach needs to be less than 100m and you can afford higher power, but you need the density, 10GBASE-T (<$500/port) may be your solution. If you have a mix of these requirements, you may want to make sure you opt for an SFP-based switch so you can mix long and short reaches and copper and fiber modules/cables for maximum flexibility.
So what’s the bottom line? Do an assessment of your needs in your data center (and the rest of your network for that matter) and plan them out in order to maximize the cost effectiveness of your 10G networks. One more thing – if you can wait a few months, you may want to consider delaying implementation of 10G until later this year when most of the 10GBASE-T chip manufacturers promise to have less than 2.5W devices commercially available, which will drastically reduce (about half) its per port power consumption.
While 10GBASE-T ports are now supposedly available on switches (at least the top Ethernet switch vendors, Brocade, Cisco and Extreme Networks say they have them), is it really more cost effective to use these instead of 10GBASE-SR, 10GBASE-CX4 or 10GBASE-CR (SFP+ direct attach copper)? Well, again, it depends on what your network looks like and how well your data center electrical power and cooling structures are designed.
First, 10GBASE-CX4 is really a legacy product and is only available on a limited number of switches so you may want to rule this out right away. If you’re looking for higher density, but you can’t support too many higher power devices, I would opt for 10GBASE-SR because it has the lowest power consumption – usually less than 1W/port. And also, the useful life of LOMF is longer; it’s smaller so won’t take up as much space or block cooling airflow if installed under a raised floor.
If you don’t have a power or density problem and can make do with just a few 10G ports for a short distance, you may choose to use 10GBASE-CR (about $615/port). But, if you don’t have a power or density issue and you need to go longer than about 10m, you’ll still need to use 10GBASE-SR and if you need a reach of more than 300m you’ll need to either install OM4 cable (which will get you up to 600m in some cases) to use with your SR devices; or look at 10GBASE-LR modules ($2500/port) that will cost you about twice as much as the SR transceivers (about $1300/port). If your reach needs to be less than 100m and you can afford higher power, but you need the density, 10GBASE-T (<$500/port) may be your solution. If you have a mix of these requirements, you may want to make sure you opt for an SFP-based switch so you can mix long and short reaches and copper and fiber modules/cables for maximum flexibility.
So what’s the bottom line? Do an assessment of your needs in your data center (and the rest of your network for that matter) and plan them out in order to maximize the cost effectiveness of your 10G networks. One more thing – if you can wait a few months, you may want to consider delaying implementation of 10G until later this year when most of the 10GBASE-T chip manufacturers promise to have less than 2.5W devices commercially available, which will drastically reduce (about half) its per port power consumption.
Thursday, July 22, 2010
SFP+ Marks a Shift in Data Center Cabling
With the advent of top-or-rack (ToR) switching and SFP+ direct attach copper cables, more data centers are able to quickly implement cost-effective 10G and beyond connections. ToR designs are currently one of two configurations:
So what makes these companies believe they can compete with the likes of Amphenol Interconnect, Molex and Tyco Electronics? Well, it might not be the fact that they think they can compete, but that they see some erosion of their patch cord businesses and view this as the only way to make sure the “interconnect” companies don’t get into certain customers. So, protecting their customer base by offering products they won’t necessarily make any money on – because, after all, many of them are actually private-labeled from the very companies they are trying to oust. Smart or risky? Smart, I think, because it seems to me that the future of the data center will be in short-reach copper and mid-range fiber in the form of laser-optimized multi-mode fiber (LOMF).
- GigE Category cabling (CAT5e, 6, or 6A) connection to each server with a 10G SFP+ or XFP uplink to either an EoR switch or back to a switch in the main distribution area (MDA)
- SFP direct attach cabling connection to each server with a 10G SFP+ or XFP uplink to either an EoR switch or back to a switch in the MDA
So what makes these companies believe they can compete with the likes of Amphenol Interconnect, Molex and Tyco Electronics? Well, it might not be the fact that they think they can compete, but that they see some erosion of their patch cord businesses and view this as the only way to make sure the “interconnect” companies don’t get into certain customers. So, protecting their customer base by offering products they won’t necessarily make any money on – because, after all, many of them are actually private-labeled from the very companies they are trying to oust. Smart or risky? Smart, I think, because it seems to me that the future of the data center will be in short-reach copper and mid-range fiber in the form of laser-optimized multi-mode fiber (LOMF).
Subscribe to:
Posts (Atom)