Lighting Up the Network
Nationally and internationally,
there has been significant activity in building large, cross-continent and worldwide
high-speed telecommunications backbones to act as the main arteries that flow
information throughout the higher education community. But large backbones are
not the whole the equation; distribution of that connectivity to individual
campuses or consortia is just as critical for meeting the near-insatiable demand
for more bandwidth in the 21st century. A key to developing that distribution
network is dark fiber.
Two terms making their way around higher education information technology discussions
these days are bandwidth and dark fiber. Bandwidth is a term that meant very
little to anyone outside of a telecommunications function in colleges and universities
a decade ago. Today, bandwidth is the key ingredient to the advancement of nearly
every initiative and service within higher education institutions. Bandwidth
is the coin of the realm for research, teaching and learning, distance education,
and service to campus residents. Bandwidth is the gas that makes information
technology (and the information economy) go. Bandwidth is everything.
Dark fiber is how your institution can get bandwidth—in quantities needed
to serve higher education with high-speed telecommunications at costs that you
can manage in these tough economic times.
To understand dark fiber, it is helpful to learn another bit of telecom terminology:
lit service. This is, in the parlance, the opposite of dark fiber. Lit service
is a connection that you purchase from a telecommunications provider (e.g.,
SBC, AT&T, Qwest, or Level3 Communications). With a lit service, you are
buying a specified amount of bandwidth, usually in terms of speed/capacity—such
as OC-3, which represents how much data can be sent across the connection (in
the case of an OC-3, ~155 million bits per second). You usually sign contracts
for this service—anywhere from one-year to three years in length—where
the provider guarantees to give you the service for a specified monthly cost.
With a lit service, the provider handles all the elements between your campus
core router and the core router at the destination (e.g., the Internet, Internet2,
or another institution). You simply hook-up and use it, and what’s between
you and where you are connecting is not really your concern. But that is a canard,
because it is your concern…you are paying for it. You are paying the provider
(and likely paying more than you would if you controlled the "what’s
between" yourself).
Dark fiber is a fiber optic connection path that has not been lit. In telecom
terms, dark fiber is something you acquire, either through buying unused fiber
that is already in place, underground or aerial (which means attached to telephone
poles) or that you contract to have installed, underground or aerial. You own
the path (or route), and you are responsible for the integrity of that path,
and for acquiring equipment to light the fiber and make it actually carry your
information. You own it either outright, or via another term: Indefeasible Right
to Use (or IRU). An IRU is simply a lease of the fiber, meaning it is yours
to use (and to be responsible for) for a set period of time, usually 20 years,
though shorter IRUs down to five-year contracts can be found.
Need a non-technical analogy? Consider water. Your home has a connection to
a nearby water facility. The water company paid for the pipes to get water to
your neighborhood and then to your house (though you likely paid a one-time
connection fee); and they own those pipes (infrastructure). You pay the water
company to provide water to your house, for which they charge you by how much
you use (by the gallon). That is the ‘lit service’ model. But if
you dig a well, and tap into an aquifer and then pump your own water to your
house, you no longer pay by the gallon—but you do pay for the well, the
pump, the purifier, and so on. And of course, in the 'well' model,
if something g'es wrong you can’t just call one number and get it handled.
You are responsible for figuring out what’s wrong and getting it fixed
(though there are companies in both this analogy and in the telecom reality
that will do this for you too). The water analogy is a little over-simplified,
because while you can draw aquifer water for free, you can’t draw Internet
drainage (your connection to the Internet) for free. But it’s close enough
to get you on the track to understanding what’s going on.
I-Light: Expanding Access in Indiana
To meet the steadily increasing demand for network capacity supporting
teaching and learning, research, and residents of their campuses, Indiana
University and Purdue University collaborated to create I-Light, a university-owned
fiber optic network linking the State’s three core campuses in Bloomington,
Indianapolis, and West Lafayette to each other, and to the Internet, Internet2,
and advanced research networks beyond.
Funded in 1999 by a state appropriation of $5 million and completed
in 2001, I-Light has yielded a 20-fold increase in inter-campus connectivity
and a five-fold increase in the campuses’ Internet bandwidth without
increasing operating costs. It supports significant high-end collaborative
applications and faculty research programs in such areas as grid computing,
human genomics, remote instrumentation, and visualization.
I-Light has enabled the development of a distributed three campus supercomputing
"grid" with broad computation, storage, and data resources
called IP-grid, which was selected in September 2003 by the National Science
Foundation as one of three new sites added to the national TeraGrid Facility.
In addition to research applications and Internet connectivity, I-Light
also supports inter-campus voice communications, e-mail, and video conferencing.
I-WIRE: Connecting Researchers in Illinois
I-WIRE represents a new type of emerging advanced infrastructure. It is
an innovative distributed facility, much more powerful than those found
on traditional national research networks (NRNs). Unlike NRNs, I-WIRE
is designed specifically for, and dedicated to, a community of researchers,
for investigation and experimentation not possible on traditional routed
networks.
I-WIRE is a dark fiber-based communications infrastructure in Illinois,
which interconnects research facilities at multiple sites: Argonne National
Laboratory, the University of Illinois (Chicago and Urbana campuses, including
the National Center for Supercomputing Applications and the Electronic
Visualization Laboratory), the University of Chicago, Illinois Institute
of Technology, Northwestern University, the Illinois Century Network,
and collocation facilities. I-WIRE provides point-to-point Layer 1 data
transport services based on Dense Wave Division Multiplexing, enabling
each organization to have at least one 2.5Gbps optical channel. Projects
using I-WIRE include the TeraGrid project, OptIPuter, and the Distributed
Optical Testbed.
CENIC’s CalREN: A Statewide Network Infrastructure for
California
January 2003 saw the deployment of the nation’s first multi-tiered,
statewide optical network, comprised of 1,100 fiber miles and long-haul
equipment supporting multiple 10 Gigabit per second (Gbps) Ethernet connections
as well as many separate links at 1Gbps or greater. The new optical infrastructure
became fully operational with the California State University system;
the community colleges and the K-12 county nodes expect to transition
to the new backbone by the end of the first quarter of 2004.
Educational networks in California, and nationwide, have typically depended
on leasing circuits from telephone or cable companies. Utilizing leased
circuits requires long-term planning to allow for the development of detailed
specifications, bidding, and then building of the network, making what
should be a simple upgrade time consuming. By making the transition to
an "owned" fiber optic infrastructure with scalable network
components, additional bandwidth may be added incrementally without the
need to redesign the underlying technology.
The Demands for Bandwidth
Colleges and universities have seen an explosion in their need for bandwidth
in the past five years. Residential institutions have been wrestling with the
need to increase their bandwidth to the Internet to deal with the Peer-to-Peer
issue (i.e., Napster, KazAa, and others). While control of these applications
has been one approach, in reality while you can cage the beast, it remains a
beast that must be fed. Most institutions have combined controls with expansion
in their connectivity. But while these recreational uses grab the headlines,
there has been a steady increase in other, more mission-centric needs for bandwidth.
Research in particular is driving the need for additional bandwidth. Research
has always been a driver of institutional function—but it is perhaps more
so today, as dwindling budgets have focused institutional direction on bringing
in more research funding to drive overall institutional activities. As well,
advances in the tools used for teaching and learning have also driven an increase
in demand beyond research (and for institutions that don’t do research).
More courses are online today, whether they are distance education focused or
a part of on-campus course management systems. And the tools used by these systems
in support of the vital teaching and learning function have advanced—more
graphical data, more multimedia (audio/video), more collaboration (video conferencing),
and greater access to digitized library data. Campuses today are no longer isolated
islands of information; they are plugged into a growing grid of information
and resources beyond the boundaries of the campus, on the Internet. And many
institutions are becoming net exporters of information, feeding more information
out to the scholarly world than they draw in.
With these increases in demand for bandwidth comes the need to get control
over this new piece of the infrastructure: the connections off our campuses.
In the lit service model, institutions pay a provider by the bit (in simplest
terms). The more you need, the more you buy, and the more you need to spend.
While there are quantity discounts to be sure, the explosion of need has rendered
the attempts to stay at a constant cost-state futile. Expansion far outpaces
the volume discounts of buying bigger lit service ‘pipes’.
A good analogy is to look at how your institution deals with other forms of
infrastructure. D'es your institution have its own telephone system? If so,
it is likely that it was more cost effective to have control over the infrastructure
rather than paying (per phone) to a provider. D'es your institution have its
own power plant? If so, it is likely that it was more cost effective to generate
and distribute all (or a portion) of the institution’s need for electricity
than to pay by the kilowatt to the local power company. D'es your institution
own its own land and buildings? If so, it is likely that it was more cost effective
to do so, rather than lease space from a provider. Telecommunications infrastructure—dark
fiber—is a similar infrastructure resource, to be considered as a ‘buy’
or ‘build’ decision point.
Dark fiber has become a way for institutions to get control of a new, vital
resource—bandwidth. Bandwidth to feed its needs for teaching and learning,
bandwidth to feed its needs for advancing research, and bandwidth to feed its
need to be a provider for its campus community of scholars and residents.
Connecting the Campus
Colleges and universities long ago made the decision to pay close attention
to their on-campus networks. This included installing fiber optic backbones,
advanced communications wiring to each desk and dorm room, and acquiring the
equipment to establish a campus network. However, now the equation has shifted,
because the balance of flow has shifted off campus (i.e., not just between users
and buildings on campus) to connections to the Internet world beyond. Dark fiber
becomes a way of expanding the campus network to the point where it connects
to the national and global Internet(s) and to other collaborating, regional
institutions. In the latter case, this could be connected within a multi-campus
institution. And it could also be to connect to neighboring campuses within
a given state or region of the country.
Dark fiber removes your institution’s dependence upon the telecommunications
providers, who bring a vast array of complicating issues from tariff to competition
(or lack thereof) to the institution. There is a national information superhighway
in place: this is the Internet (commodity Internet for the faithful, the place
where all those dotcoms are), and the advanced higher education research Internet
community (Internet2, National LambdaRail, global advanced network peering points,
and others).
Institutions or consortiums of institutions have benefited from dark fiber—to
build their own on-ramps to the information superhighway. Dark fiber is a way
for institutions to expand their domain of control, to the point where the campus
network more easily connects to the rest of the world. There will always be
charges associated with accessing the rest of the world—because they have
an infrastructure that needs to be supported. But one more element is put under
the control of the institutions: the path from their campus to the points of
connection to the rest of the world—GigaPoPs.
Dark Fiber and You
You may be asking yourself, “What should my institution’s dark fiber
strategy be?” This varies by institution. There is no one-size-fits-all
dark fiber strategy. Variables include the size of the institution, its focus
(research, teaching, service, etc.), its relation to other institutions in its
state or region, and most importantly its location.
These are varying facets of building a custom strategy for your institution.
The overriding conceptual strategy, however, is consistent—to get control
of the infrastructure resources that best position your institution to succeed
in the 21st century.
Large, research institutions should look for the nearest significant point
of access to the Internet(s). They should then look to buy or build dark fiber
infrastructure from their campuses to these access points. From these major
GigaPoPs, they can then work strategies to connect themselves into the commodity
Internet and advanced Internet. With regard to the former, the larger and more
diverse GigaPoP, and the greater the competition for cost-intensive services
(like commodity Internet connectivity). Competition alone means you can bid
out your Internet drainage demands and look to acquire Internet drainage at
substantially reduced rates.
Smaller institutions should look for partners in their geographic areas to
build a Web of dark fiber, connecting several institutions to each other and
then to these significant points of access to the Internet(s). Combinations
include finding a large partner to lead the effort. The advantage here is in
pooling needs and budgets, to get more volume (which leads to better pricing
on top of the competition aspect mentioned above). Building a consortium also
helps spread the costs of these solutions across more entities.
The State of Indiana is one example of how these strategies were deployed.
The I-Light Network allowed two major institutions, constituted from three major
core campuses, to build infrastructure that essentially combined their campuses
onto one large campus network, and then through connections to the state’s
existing higher education network, bring in its demand to pool for bidding for
service.
The results have been astounding. The I-Light consortium now gets commodity
Internet service for about one-fifth of what it used to pay (per million bits
per second capacity), due to the competition between providers facilitated by
being in their main points of presence, and through the pooling of need which
established a very large (more than 1.5 billion bits per second) jewel for these
providers to bid upon.
National LambdaRail: Lighting the Future
In May 2003, a consortium of U.S. research universities and private sector
technology companies formed a non-profit corporation, National LambdaRail,
Inc. (NLR). NLR is dedicated to building a national scale infrastructure
for research and experimentation in networking technologies and applications.
NLR is the largest higher education-owned and managed optical networking
and research facility in the world, with approximately 10,000 route miles
of dark fiber and four 10 Gbps light waves or lambdas provisioned at the
outset.
NLR is not a single network; rather it is a unique and rich set of facilities,
capabilities, and services being built to support a set of multiple, distinct,
experimental, and production networks for the U.S. research community.
On NLR, these various networks will exist side-by-side in the same fiber
optic cable pair but will be operationally independent of each other.
NLR is the first national infrastructure that will allow researchers
to dedicate wavelengths to a set of different problems, giving the community
the freedom and flexibility to stress the network in order to identify
solutions and applications.
On November 18, 2003, National LambdaRail announced that it successfully
lit the first path on the national footprint between Chicago’s StarLight
facility and the Pittsburgh Supercomputing Center (from the University
of Pittsburgh and Carnegie Mellon University). NLR is currently working
on the Seattle to Portland, Oregon path as well as a path between Portland
and Sunnyvale, Calif.
For more information, visit www.nationallambdarail.org.
When Should We Act?
Institutions should act now to develop a communications infrastructure expansion
strategy that includes dark fiber. In the 90s, telecommunications companies
installed a large amount of dark fiber, which then went unused once the dotcom/telecom
bubble burst. There has been a glut of existing dark fiber in place, and if
your institution is located where unused dark fiber exists, you should seek
to lay your hands on it now. Why? Because others are looking to buy low, right
now! In fact, there are dark fiber initiatives at varying points of development
or deployment in nearly 20 states today. And as the economy has started to re-ignite,
this unused commodity will disappear soon enough.
As well, national initiatives such as National LambdaRail are driving the
broader research component. Advances at the national level are linked closely
to the development of regional optical networks (RONs); in fact, they are co-motivators.
Dark fiber is an enabler of the national and regional initiatives. In this regard,
Internet2 has established the National Research and Education Fiber Company
(FiberCo) to support regional fiber optical networking initiatives dedicated
to research and higher education.
Funding a Dark Fiber Strategy
Funding strategies will vary, depending upon the dark fiber strategy selected
to meet the needs of the institution or consortium, the local market for dark
fiber, and the timing when such a strategy is implemented. What those who have
successfully implemented such networks have found, is that existing budgets
that previously provided for lit services can usually accommodate the ongoing
costs of lighting and maintaining the dark fiber plant. These costs include
co-location (costs to rent space in the Internet access locations), route monitoring
and maintenance (to help ensure the fiber is not cut during excavation), and
life cycling of optical hardware.
What is often needed, though, is a source of one-time funding to provide for
the initial outlay of capital needed to acquire (or install) the dark fiber,
whether this be from special state appropriations (as was the case in Indiana),
from grant funding, or from other sources of one-time-only capital. Of course,
depending upon the current budget costs for lit services, bonding (payback of
a loan for construction) is possible. All of this will vary by the specific
situation.
But in the end, experience of the successful has proven that the investment,
regardless the source, has been a wise one. Getting control of the communications
infrastructure means campuses can ride the hardware curve and deploy more advanced
hardware capable of driving higher communication speeds (i.e., more bandwidth)
as they become affordable. This means that institutions can grow their bandwidth,
and do so in a fairly cost-neutral manner. And in so doing, keep pace with this
important facet of the continuing evolution of information technology in higher
education.
Making the decision to develop a dark fiber strategy is the important first
step. Building that strategy will be complex, and building it well will be critical
to its ultimate success. There are many questions to be answered—build
versus buy, length of IRU, where to get connectivity to the Internet(s), and
how or if to form consortiums. But if you make the decision now to start such
a process, you will be able to stand on the shoulders of those who have gone
before you, and learn from what they did right, and what they want to do differently.
There are many places to learn, and many places to get help. The Internet2 community
via its many initiatives (including FiberCo) as well as National LambdaRail
can provide advice and guidance on the entire scope of the issue, and help you
make contact with your colleagues around the country already engaged in dark
fiber networking initiatives. But the first step is to make the decision to
incorporate a dark fiber strategy into your overall campus strategic plan.