Click here to receive your FREE subscription to Campus Technology

Home > Sharing Large Data Files

Current News

Sharing Large Data Files

10/29/2003

By Terry CalhounBy , Joe St Sauver

To move large files efficiently, you really want to get the data distributed around the network, with redundant copies topologically close to those who need it. Staging the data close to those who need it means that observed throughput *will* typically go up and network hot spots *can* usually be avoided. A free side benefit of distributing large files this way is that distribution of files via a series of geographically dispersed nodes grants the content of those files a degree of resistance to network denial of service attacks, or to data loss due to simple hardware failure.

All this is wonderful, so far as it g'es, but if you are like me, you probably have a well-developed skeptical side:

  • "What d'es it cost to use LoRS?";
  • "Are there running production-quality applications, on the platforms I need?"; and
  • "Can I trust my data to some bunch of random LoRS nodes?"

    For once, the answers are all pretty good:

  • You can use LoRS for free, in part because the LoCI project has received federal funding as well as support from many volunteers who host storage nodes;
  • LoRS is available in graphical and command line form for Windows PCs, Mac OS X boxes, including source to build on Solaris, Linux and other Unix systems; and
  • LoRS uses encryption to protect your data from storage depot operators (and storage depot operators from your data); protection against arbitrary loss of a node, or data on a node comes from the fact that you will typically make redundant copies of the same data (stored in chunks) on a variety of different storage nodes, so that if one node g'es down you can get a copy of the missing chunks from a different node.

    All this sounds pretty cool, d'esn't it?

    Having said that, I won't kid you: installing LoRS may take a little noodling around depending on what you've already got installed (you'll need to download and install Tcl/Tk and Perl, for example, if you don't already have them installed on your system); it is not yet a download-click-and-go operation (but neither d'es it typically require expert knowledge to do an install). I'd put it at the level where it helps to be technically minded, but you don't need to be a hard core geek (or even a minor league geek) to make it work.

    As you begin working with LoRS, you'll need to absorb some new concepts, such as:

  • The idea of an exNode (an XML-formated file with pointers to the chunks of your dataset that have been written to storage depots by LoRS); or the fact that . . .
  • Soft (gratis, as-available) storage allocations which are made by LoRS are of limited duration (a day by default), sort of like a giant temp or scratch disk on some Unix system. If you need to, you can request that your storage allocation be refreshed for additional time, subject to space availability and provided your storage allocations haven't already expired.

    Getting Started with LoRS

    If you'd like to try LoRS, and I'd encourage you to do so, you should begin by checking out the LoRS section of the LoCI Web site. Download the documentation (yes, do RTFM, because in this case the manual is quite good), then download and install the LoRS software. Try uploading and downloading a sample file or two. See what you think - I suspect you'll be as impressed as I was.



    Recommended Reading
    • IT Trends :: Thursday, August 28, 2008

      :::::: CAMPUS WIFI

      :: Saint Joseph Builds Out Wireless Network in Multi-year Upgrade

      :::::: IT NEWS

      :: California Community Colleges Partner with Waterfall Mobile on Statewide Emergency Notification Coverage
      :: King's College and ASU Add e2Campus for Improved Emergency Notifications
      :: Vista Ramp Up Is Happening Now, Study Says
      :: Talisma Launches New Version of CRM with Built-in Application Management
      :: Ferrum College Enrolls Juniper Networks To Extend 10 Gigabit Ethernet
      :: California Community Colleges Adopt SunGard Banner Software
      :: Ball State U Web Sites Now Managed with Sitecore

    • C-Level View :: August 27, 2008

      :::::: EXECUTIVE VIEW

      : Let the Games Begin! Google vs. Microsoft

      :::::: WORTH NOTING

      : California Community Colleges Adopt SunGard Banner Software
      : McGill U Library Scanning Rare Books with Kirtas
      : Ball State U Web Sites Now Managed with Sitecore
      : Report: Green Efforts Improving on Campuses
      : Oracle Releases Student Administration Integration Pack

    • SmartClassroom :: Wednesday, August 27, 2008

      :::::: COLLABORATION

      : Bringing Composers into Classrooms Through Skype

      :::::: NEWS and PRODUCT UPDATES

      : Columbia U Going Live on iTunes U
      : Tiffin U's New Online College to Use Pearson's eCollege for Course Management
      : Luidia Releases eBeam Interact 2.1 for Interactive Whiteboards
      : McGill U Library Scanning Rare Books with Kirtas
      : Ball State U Web Sites Now Managed with Sitecore

    • SmartClassroom :: Wednesday, August 27, 2008

      :::::: COLLABORATION

      : Bringing Composers into Classrooms Through Skype

      :::::: NEWS and PRODUCT UPDATES

      : Columbia U Going Live on iTunes U
      : Tiffin U's New Online College to Use Pearson's eCollege for Course Management
      : Luidia Releases eBeam Interact 2.1 for Interactive Whiteboards
      : McGill U Library Scanning Rare Books with Kirtas
      : Ball State U Web Sites Now Managed with Sitecore

    • News Update :: Tuesday, August 26, 2008

      :::::: NEWS

      : Report: Green Efforts Improving on Campuses
      : Polytechnic Institute of NYU Deploys Array Networks Equipment for Access Control
      : Oracle Releases Student Administration Integration Pack
      : Red Hat Hacked, Company Issues Security Advisory
      : Sun Open Sources Mobile Toolkit LWUIT
      : Vulnerability Management Needed for Security, Study Says
      : Microsoft Details SharePoint-SQL 2008 Integration
      : Higher Ed Growing into BI, Data Warehousing
      : LectureShare Updates Free Course Management System

    • Campus Security :: August 22, 2008

      :::::: CASE STUDY

      : Corralling Identity Management

      :::::: CAMPUS SECURITY NEWS

      : Vulnerability Management Needed for Security, Study Says
      : Wayne State Deploys Q1 Labs QRadar to Manage and Secure Network
      : KU Medical Center Installs Real-time Beacon System
      : Virginia Tech Tries 'Compliance Sheriff' To Improve Web Site Accessibility
      : Microsoft, BearingPoint Team Up To Provide Risk-Based Compliance Solution
      : Collaboration Key to Security, Microsoft Says
      : IBM Unveils New Software Designed To Streamline eDiscovery
      : Security Woes Up, as PHP and OSS Make the List

    • Related Articles

    send e-mail link to articleEmail this article

    print this articlePrintable Format