Lighting Up the Network

Nationally and internationally, there has been significant activity in building large, cross-continent and worldwide high-speed telecommunications backbones to act as the main arteries that flow information throughout the higher education community. But large backbones are not the whole the equation; distribution of that connectivity to individual campuses or consortia is just as critical for meeting the near-insatiable demand for more bandwidth in the 21st century. A key to developing that distribution network is dark fiber.

Two terms making their way around higher education information technology discussions these days are bandwidth and dark fiber. Bandwidth is a term that meant very little to anyone outside of a telecommunications function in colleges and universities a decade ago. Today, bandwidth is the key ingredient to the advancement of nearly every initiative and service within higher education institutions. Bandwidth is the coin of the realm for research, teaching and learning, distance education, and service to campus residents. Bandwidth is the gas that makes information technology (and the information economy) go. Bandwidth is everything.

Dark fiber is how your institution can get bandwidth—in quantities needed to serve higher education with high-speed telecommunications at costs that you can manage in these tough economic times.

To understand dark fiber, it is helpful to learn another bit of telecom terminology: lit service. This is, in the parlance, the opposite of dark fiber. Lit service is a connection that you purchase from a telecommunications provider (e.g., SBC, AT&T, Qwest, or Level3 Communications). With a lit service, you are buying a specified amount of bandwidth, usually in terms of speed/capacity—such as OC-3, which represents how much data can be sent across the connection (in the case of an OC-3, ~155 million bits per second). You usually sign contracts for this service—anywhere from one-year to three years in length—where the provider guarantees to give you the service for a specified monthly cost. With a lit service, the provider handles all the elements between your campus core router and the core router at the destination (e.g., the Internet, Internet2, or another institution). You simply hook-up and use it, and what’s between you and where you are connecting is not really your concern. But that is a canard, because it is your concern…you are paying for it. You are paying the provider (and likely paying more than you would if you controlled the "what’s between" yourself).

Dark fiber is a fiber optic connection path that has not been lit. In telecom terms, dark fiber is something you acquire, either through buying unused fiber that is already in place, underground or aerial (which means attached to telephone poles) or that you contract to have installed, underground or aerial. You own the path (or route), and you are responsible for the integrity of that path, and for acquiring equipment to light the fiber and make it actually carry your information. You own it either outright, or via another term: Indefeasible Right to Use (or IRU). An IRU is simply a lease of the fiber, meaning it is yours to use (and to be responsible for) for a set period of time, usually 20 years, though shorter IRUs down to five-year contracts can be found.

Need a non-technical analogy? Consider water. Your home has a connection to a nearby water facility. The water company paid for the pipes to get water to your neighborhood and then to your house (though you likely paid a one-time connection fee); and they own those pipes (infrastructure). You pay the water company to provide water to your house, for which they charge you by how much you use (by the gallon). That is the ‘lit service’ model. But if you dig a well, and tap into an aquifer and then pump your own water to your house, you no longer pay by the gallon—but you do pay for the well, the pump, the purifier, and so on. And of course, in the 'well' model, if something g'es wrong you can’t just call one number and get it handled. You are responsible for figuring out what’s wrong and getting it fixed (though there are companies in both this analogy and in the telecom reality that will do this for you too). The water analogy is a little over-simplified, because while you can draw aquifer water for free, you can’t draw Internet drainage (your connection to the Internet) for free. But it’s close enough to get you on the track to understanding what’s going on.

I-Light: Expanding Access in Indiana

To meet the steadily increasing demand for network capacity supporting teaching and learning, research, and residents of their campuses, Indiana University and Purdue University collaborated to create I-Light, a university-owned fiber optic network linking the State’s three core campuses in Bloomington, Indianapolis, and West Lafayette to each other, and to the Internet, Internet2, and advanced research networks beyond.

Funded in 1999 by a state appropriation of $5 million and completed in 2001, I-Light has yielded a 20-fold increase in inter-campus connectivity and a five-fold increase in the campuses’ Internet bandwidth without increasing operating costs. It supports significant high-end collaborative applications and faculty research programs in such areas as grid computing, human genomics, remote instrumentation, and visualization.

I-Light has enabled the development of a distributed three campus supercomputing "grid" with broad computation, storage, and data resources called IP-grid, which was selected in September 2003 by the National Science Foundation as one of three new sites added to the national TeraGrid Facility. In addition to research applications and Internet connectivity, I-Light also supports inter-campus voice communications, e-mail, and video conferencing.

I-WIRE: Connecting Researchers in Illinois
I-WIRE represents a new type of emerging advanced infrastructure. It is an innovative distributed facility, much more powerful than those found on traditional national research networks (NRNs). Unlike NRNs, I-WIRE is designed specifically for, and dedicated to, a community of researchers, for investigation and experimentation not possible on traditional routed networks.

I-WIRE is a dark fiber-based communications infrastructure in Illinois, which interconnects research facilities at multiple sites: Argonne National Laboratory, the University of Illinois (Chicago and Urbana campuses, including the National Center for Supercomputing Applications and the Electronic Visualization Laboratory), the University of Chicago, Illinois Institute of Technology, Northwestern University, the Illinois Century Network, and collocation facilities. I-WIRE provides point-to-point Layer 1 data transport services based on Dense Wave Division Multiplexing, enabling each organization to have at least one 2.5Gbps optical channel. Projects using I-WIRE include the TeraGrid project, OptIPuter, and the Distributed Optical Testbed.

CENIC’s CalREN: A Statewide Network Infrastructure for California
January 2003 saw the deployment of the nation’s first multi-tiered, statewide optical network, comprised of 1,100 fiber miles and long-haul equipment supporting multiple 10 Gigabit per second (Gbps) Ethernet connections as well as many separate links at 1Gbps or greater. The new optical infrastructure became fully operational with the California State University system; the community colleges and the K-12 county nodes expect to transition to the new backbone by the end of the first quarter of 2004.

Educational networks in California, and nationwide, have typically depended on leasing circuits from telephone or cable companies. Utilizing leased circuits requires long-term planning to allow for the development of detailed specifications, bidding, and then building of the network, making what should be a simple upgrade time consuming. By making the transition to an "owned" fiber optic infrastructure with scalable network components, additional bandwidth may be added incrementally without the need to redesign the underlying technology.

The Demands for Bandwidth
Colleges and universities have seen an explosion in their need for bandwidth in the past five years. Residential institutions have been wrestling with the need to increase their bandwidth to the Internet to deal with the Peer-to-Peer issue (i.e., Napster, KazAa, and others). While control of these applications has been one approach, in reality while you can cage the beast, it remains a beast that must be fed. Most institutions have combined controls with expansion in their connectivity. But while these recreational uses grab the headlines, there has been a steady increase in other, more mission-centric needs for bandwidth.

Research in particular is driving the need for additional bandwidth. Research has always been a driver of institutional function—but it is perhaps more so today, as dwindling budgets have focused institutional direction on bringing in more research funding to drive overall institutional activities. As well, advances in the tools used for teaching and learning have also driven an increase in demand beyond research (and for institutions that don’t do research). More courses are online today, whether they are distance education focused or a part of on-campus course management systems. And the tools used by these systems in support of the vital teaching and learning function have advanced—more graphical data, more multimedia (audio/video), more collaboration (video conferencing), and greater access to digitized library data. Campuses today are no longer isolated islands of information; they are plugged into a growing grid of information and resources beyond the boundaries of the campus, on the Internet. And many institutions are becoming net exporters of information, feeding more information out to the scholarly world than they draw in.

With these increases in demand for bandwidth comes the need to get control over this new piece of the infrastructure: the connections off our campuses. In the lit service model, institutions pay a provider by the bit (in simplest terms). The more you need, the more you buy, and the more you need to spend. While there are quantity discounts to be sure, the explosion of need has rendered the attempts to stay at a constant cost-state futile. Expansion far outpaces the volume discounts of buying bigger lit service ‘pipes’.

A good analogy is to look at how your institution deals with other forms of infrastructure. D'es your institution have its own telephone system? If so, it is likely that it was more cost effective to have control over the infrastructure rather than paying (per phone) to a provider. D'es your institution have its own power plant? If so, it is likely that it was more cost effective to generate and distribute all (or a portion) of the institution’s need for electricity than to pay by the kilowatt to the local power company. D'es your institution own its own land and buildings? If so, it is likely that it was more cost effective to do so, rather than lease space from a provider. Telecommunications infrastructure—dark fiber—is a similar infrastructure resource, to be considered as a ‘buy’ or ‘build’ decision point.

Dark fiber has become a way for institutions to get control of a new, vital resource—bandwidth. Bandwidth to feed its needs for teaching and learning, bandwidth to feed its needs for advancing research, and bandwidth to feed its need to be a provider for its campus community of scholars and residents.

Connecting the Campus
Colleges and universities long ago made the decision to pay close attention to their on-campus networks. This included installing fiber optic backbones, advanced communications wiring to each desk and dorm room, and acquiring the equipment to establish a campus network. However, now the equation has shifted, because the balance of flow has shifted off campus (i.e., not just between users and buildings on campus) to connections to the Internet world beyond. Dark fiber becomes a way of expanding the campus network to the point where it connects to the national and global Internet(s) and to other collaborating, regional institutions. In the latter case, this could be connected within a multi-campus institution. And it could also be to connect to neighboring campuses within a given state or region of the country.

Dark fiber removes your institution’s dependence upon the telecommunications providers, who bring a vast array of complicating issues from tariff to competition (or lack thereof) to the institution. There is a national information superhighway in place: this is the Internet (commodity Internet for the faithful, the place where all those dotcoms are), and the advanced higher education research Internet community (Internet2, National LambdaRail, global advanced network peering points, and others).

Institutions or consortiums of institutions have benefited from dark fiber—to build their own on-ramps to the information superhighway. Dark fiber is a way for institutions to expand their domain of control, to the point where the campus network more easily connects to the rest of the world. There will always be charges associated with accessing the rest of the world—because they have an infrastructure that needs to be supported. But one more element is put under the control of the institutions: the path from their campus to the points of connection to the rest of the world—GigaPoPs.

Dark Fiber and You
You may be asking yourself, “What should my institution’s dark fiber strategy be?” This varies by institution. There is no one-size-fits-all dark fiber strategy. Variables include the size of the institution, its focus (research, teaching, service, etc.), its relation to other institutions in its state or region, and most importantly its location.

These are varying facets of building a custom strategy for your institution. The overriding conceptual strategy, however, is consistent—to get control of the infrastructure resources that best position your institution to succeed in the 21st century.

Large, research institutions should look for the nearest significant point of access to the Internet(s). They should then look to buy or build dark fiber infrastructure from their campuses to these access points. From these major GigaPoPs, they can then work strategies to connect themselves into the commodity Internet and advanced Internet. With regard to the former, the larger and more diverse GigaPoP, and the greater the competition for cost-intensive services (like commodity Internet connectivity). Competition alone means you can bid out your Internet drainage demands and look to acquire Internet drainage at substantially reduced rates.

Smaller institutions should look for partners in their geographic areas to build a Web of dark fiber, connecting several institutions to each other and then to these significant points of access to the Internet(s). Combinations include finding a large partner to lead the effort. The advantage here is in pooling needs and budgets, to get more volume (which leads to better pricing on top of the competition aspect mentioned above). Building a consortium also helps spread the costs of these solutions across more entities.

The State of Indiana is one example of how these strategies were deployed. The I-Light Network allowed two major institutions, constituted from three major core campuses, to build infrastructure that essentially combined their campuses onto one large campus network, and then through connections to the state’s existing higher education network, bring in its demand to pool for bidding for service. The results have been astounding. The I-Light consortium now gets commodity Internet service for about one-fifth of what it used to pay (per million bits per second capacity), due to the competition between providers facilitated by being in their main points of presence, and through the pooling of need which established a very large (more than 1.5 billion bits per second) jewel for these providers to bid upon.

National LambdaRail: Lighting the Future

In May 2003, a consortium of U.S. research universities and private sector technology companies formed a non-profit corporation, National LambdaRail, Inc. (NLR). NLR is dedicated to building a national scale infrastructure for research and experimentation in networking technologies and applications. NLR is the largest higher education-owned and managed optical networking and research facility in the world, with approximately 10,000 route miles of dark fiber and four 10 Gbps light waves or lambdas provisioned at the outset.

NLR is not a single network; rather it is a unique and rich set of facilities, capabilities, and services being built to support a set of multiple, distinct, experimental, and production networks for the U.S. research community. On NLR, these various networks will exist side-by-side in the same fiber optic cable pair but will be operationally independent of each other.

NLR is the first national infrastructure that will allow researchers to dedicate wavelengths to a set of different problems, giving the community the freedom and flexibility to stress the network in order to identify solutions and applications.

On November 18, 2003, National LambdaRail announced that it successfully lit the first path on the national footprint between Chicago’s StarLight facility and the Pittsburgh Supercomputing Center (from the University of Pittsburgh and Carnegie Mellon University). NLR is currently working on the Seattle to Portland, Oregon path as well as a path between Portland and Sunnyvale, Calif.

For more information, visit www.nationallambdarail.org.

When Should We Act?
Institutions should act now to develop a communications infrastructure expansion strategy that includes dark fiber. In the 90s, telecommunications companies installed a large amount of dark fiber, which then went unused once the dotcom/telecom bubble burst. There has been a glut of existing dark fiber in place, and if your institution is located where unused dark fiber exists, you should seek to lay your hands on it now. Why? Because others are looking to buy low, right now! In fact, there are dark fiber initiatives at varying points of development or deployment in nearly 20 states today. And as the economy has started to re-ignite, this unused commodity will disappear soon enough.

As well, national initiatives such as National LambdaRail are driving the broader research component. Advances at the national level are linked closely to the development of regional optical networks (RONs); in fact, they are co-motivators. Dark fiber is an enabler of the national and regional initiatives. In this regard, Internet2 has established the National Research and Education Fiber Company (FiberCo) to support regional fiber optical networking initiatives dedicated to research and higher education.

Funding a Dark Fiber Strategy
Funding strategies will vary, depending upon the dark fiber strategy selected to meet the needs of the institution or consortium, the local market for dark fiber, and the timing when such a strategy is implemented. What those who have successfully implemented such networks have found, is that existing budgets that previously provided for lit services can usually accommodate the ongoing costs of lighting and maintaining the dark fiber plant. These costs include co-location (costs to rent space in the Internet access locations), route monitoring and maintenance (to help ensure the fiber is not cut during excavation), and life cycling of optical hardware.

What is often needed, though, is a source of one-time funding to provide for the initial outlay of capital needed to acquire (or install) the dark fiber, whether this be from special state appropriations (as was the case in Indiana), from grant funding, or from other sources of one-time-only capital. Of course, depending upon the current budget costs for lit services, bonding (payback of a loan for construction) is possible. All of this will vary by the specific situation.

But in the end, experience of the successful has proven that the investment, regardless the source, has been a wise one. Getting control of the communications infrastructure means campuses can ride the hardware curve and deploy more advanced hardware capable of driving higher communication speeds (i.e., more bandwidth) as they become affordable. This means that institutions can grow their bandwidth, and do so in a fairly cost-neutral manner. And in so doing, keep pace with this important facet of the continuing evolution of information technology in higher education.

Making the decision to develop a dark fiber strategy is the important first step. Building that strategy will be complex, and building it well will be critical to its ultimate success. There are many questions to be answered—build versus buy, length of IRU, where to get connectivity to the Internet(s), and how or if to form consortiums. But if you make the decision now to start such a process, you will be able to stand on the shoulders of those who have gone before you, and learn from what they did right, and what they want to do differently. There are many places to learn, and many places to get help. The Internet2 community via its many initiatives (including FiberCo) as well as National LambdaRail can provide advice and guidance on the entire scope of the issue, and help you make contact with your colleagues around the country already engaged in dark fiber networking initiatives. But the first step is to make the decision to incorporate a dark fiber strategy into your overall campus strategic plan.

Resources

CENIC (Corporation for Education Network Initiatives in California)
www.cenic.org

Internet2
www.internet2.edu

I-WIRE (Illinois Wired/Wireless Infrastructure for Research and Education)
www.i-wire.org

National LambdaRail
www.nationallambdarail.org

The National Research and Education Fiber Company (FiberCo)
www.fiberco.org

National Science Foundation (NSF)
www.nsf.gov

comments powered by Disqus