Sandia’s new fiber optic network is world’s largest

Sandia Labs News Releases

February 28, 2013

Sandia’s new fiber optic network is world’s largest

Fiber optic network saves energy, money

 

ALBUQUERQUE, N.M. — Sandia National Laboratories has become a pioneer in large-scale passive optical networks, building the largest fiber optical local area network in the world. 

Steve Gossage, a senior engineer at Sandia National Laboratories, looks at fiber optics in a cable box that replaced heavier and bulkier copper cable for high-speed communications throughout much of the labs. Fiber offers more capacity and is more reliable than copper. (Photo by Randy Montoya) Click on the thumbnail for a high-resolution image.

The network pulls together 265 buildings and 13,000 computer network ports and brings high-speed communication to some of the labs’ most remote technical areas for the first time. And it will save an estimated $20 million over five years through energy and other savings and not having to buy replacement equipment. Sandia expects to reduce energy costs by 65 percent once the network is fully operational.

Fiber offers far more capacity, is more secure and reliable and is less expensive to maintain and operate than the traditional network using copper cables.

An optical local area network (LAN) gives people phone, data and video services using half-inch fiber optic cables made of 288 individual fibers, instead of the conventional 4-inch copper cables. Copper cables used to fill up underground conduits and required steel overhead racks of connecting cable, along with distribution rooms filled with separate frames for copper voice and data cables. The fiber distribution system uses only part of the conduit and needs only a 2- by 3-foot cable box.

“The frames go away, and the walls are bare and the tray empties,” said senior engineer Steve Gossage, who has spent his 36-year career at Sandia in advanced information and network systems engineering.

The national laboratory has always pushed for speed beyond the fastest transmission rate available, Gossage said. “When people were working in much slower data rates, kilobit-type rates at short distances, we were trying to get 10 times the distance and 10 times the speed,” he said.

Adopting fiber optics

Sandia began looking at fiber optics early in the technology’s development because of its promise of higher bandwidth — greater communication speed — at longer distances. The labs started converting from copper in the 1980s, first installing then-emerging fiber optics in a single building and bumping that facility to megabit speeds. “Today we’re way past that. We’re at 10 gigabit-type rates and looking hard at 100,” Gossage said.

After years of planning, Sandia completed a formal network plan in late 2008 and sought competitive bids the following year. Sandia selected Tellabs of Naperville, Ill., as the equipment vendor for the network, and Gossage and his colleagues simultaneously began to jumpstart the deployment of the fiber infrastructure and set up a test lab to validate the performance of configurations for the equipment and various network functions. The technology began moving to desktops in 2011, and by the end of 2012, Sandia had converted more than 90 percent of bulky copper cable to a fiber optics LAN.

Sandia, which will spend about $15 million on the project, needs superb computing capability for the problems it tackles as part of its support for the mission for the National Nuclear Security Administration.

“Whether it’s a materials science problem or modeling an event, we need a lot of data and a lot of processing capability,” Gossage said. “We need to be able to see it, we need to be able to view it, we need to be able to put teams together. This is a large laboratory, deeply stocked with scientists and engineers and test labs. For the analyses we get, the problems are not small and they’re not easy.”

Since its first experience with fiber optics, Sandia envisioned being able to use multiple wavelengths in a very high bandwidth single strand reaching the farthest tech areas. But decades ago, when Sandia began putting in single-mode fiber to desks and adding underground fiber capabilities, the technology wasn’t quite mature enough to take advantage of fiber optics’ inherent multiple wavelengths and speeds.

So Sandia continued to install the fiber optics cable foundation and waited as the technology developed, and moved quickly when commercial optical networks began deploying voice, data and video to large collections of homes and offices.

“There weren’t that many unknowns for us because we had been thinking about ways to do this on a large scale for quite a while,” Gossage said. “We had already thought through what this might mean to us, what it might mean to our lifecycle costs and where the investments would be, and we were already pretty comfortable with fiber and the technologies that go with it.”

Copper versus fiber optics

Buildings with conventional copper LANs have separate networks for phones, computers, wireless, security and so on. Fiber optics puts everything in a single network cable. That eliminates a large number of power-consuming switches and routers and makes the network simpler to operate and cheaper to install. Since it requires less space, energy and maintenance costs go down.

“As we research and deploy new technologies, our main objectives are to enable the labs’ mission, decrease life-cycle costs and if possible reduce our footprint on the environment. With the deployment of passive optical networks we have been able to meet and exceed all of these objectives,” said Sandia manager Jeremy Banks.

Where a conventional LAN serving 900 customers requires a space the size of three double ovens, an optical network serving 8,000 requires a microwave oven-sized space. Where copper cable required Sandia to maintain and manage 600 separate switches in the field, optical LAN allows it to operate a data center in one building and simple, standard ports to offices. Because fiber optics reaches beyond the 100-meter radius that once was the standard from a wiring closet to a desktop, remote areas such as the National Solar Thermal Test Facility have high-speed communications for the first time.

The only copper wire for most of Sandia today is a short connection from the wall to the desktop. Everything behind the wall is fiber.

Moving away from copper wasn’t easy. It required new technology for the core communication system and made Sandia its own network provider, Gossage said. He credited a central team of about 10 people across Sandia who worked together every day throughout 2011, plus sub-teams totaling about 40 people. The effort included engineering design, information technology, network systems, computing, facilities, security and people in the field pulling cable and connecting ports.

Still to come

Sandia is recycling copper as it’s replaced, which keeps tons of valuable material out of a landfill. The estimated $80,000 for the copper will offset some of the fiber optics cost.

The labs also must turn off hundreds of switches before it can fully realize the energy savings. That will take longer because it depends on such things as staffing, Gossage said.

More change could be coming. A small trial is under way for voice-over-fiber — putting data and voice in one system rather than the two Sandia uses today. Testing shows Sandia can protect voice running through a congested circuit — what Gossage calls “a Mother’s Day test,” when everyone calls at the same time. The Gigabit Passive Optical Network standard Sandia works with can dedicate part of the bandwidth and give priority to selected traffic such as voice. So calls would go through even with heavy competition from data.

Sandia also is working with a small number of researchers who need more bandwidth than they’re getting. The labs’ needs are ahead of the market but it’s pushing for next-generation increases in speed, Gossage said.

Communication speed improves every five to eight years. With copper, each improvement required replacing large, heavy bundles of jacketed cable to re-engineer them to perform at the new speed, he said. Fiber optical cable offers a bandwidth good for 25 years or more. 

“We change the wavelength, we change the modulation rate, we don’t get back in the ceiling, we don’t get back in the customer’s office,” Gossage said. “So our return on investment, our capital investment, our operational investment, the impact on our customers — everything gets better.” 

---

Sandia National Laboratories is a multiprogram laboratory operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration. With main facilities in Albuquerque, N.M., and Livermore, Calif., Sandia has major R&D responsibilities in national security, energy and environmental technologies, and economic competitiveness.