Big Data

In the first installment of a two-part series, CT explains what Big Data is and its potential for improving student learning and success.

• By John K. Waters
• 11/01/12

Colleges and universities are swimming in an ever-widening sea of data. We all are. Human beings and machines together generate about 2.5 quintillion (1018) bytes every day, according to IBM's latest estimate. The sources of all that data are dizzyingly diverse: e-mail, blogs, click streams, security cameras, weather sensors, social networks, academic research, and student portfolios, to name just a few. And it's all coming at us at warp speed: Google alone reportedly processes 24 petabytes (that's a quadrillion--1015--bytes) every day.

The industry buzz phrase for this phenomenon is "Big Data," which loosely refers to data sets too large and/or diverse for conventional tools to manage and mine efficiently. For colleges and universities, Big Data presents a challenge that will only get...well...bigger. But approached with the right tools and strategies, Big Data also offers an incredibly rich resource for improving retention rates, fine-tuning curricula, and supporting students, faculty, and administration in myriad ways.

In higher education, Big Data may be seen in two distinct contexts: 1) as a product of research institutions that are charged with gathering, managing, and curating a wide range of structured and unstructured data; and 2) as a resource for predictive analytics.

The former is not exactly a new phenomenon, although the sources and velocities of the data streams are expanding and accelerating. But the latter has emerged as a way to leverage a variety of data sources--some new, some not--to help guide students along course and degree paths that will lead to higher graduation rates.

Predictive analytics, which applies statistical techniques to data to forecast likely outcomes, is not a new process either. The difference now is scale: Applied to vast amounts of data from a huge variety of sources, predictive analytics now seems capable of achieving its crystal ball promise.

New Possibilities
"The thing you have to understand about Big Data is that it gives us something we haven't had before," says Gerry McCartney, CIO at Purdue University (IN). "Analyzing these massive data sets allows us to extract patterns, which you just can't obtain from smaller data sets, that allow us to predict, say, how a student will do in class. That predictive capability is a direct result of the volumes of data being analyzed."

In other words, you can see more useful things in the data when more of it's available. Of course, "big" is a relative adjective. One organization's terabyte is another's gigabyte. Traditionally, in most organizations, predictive analytics has involved data volumes in the high gigabytes and low terabytes. In small and mid-sized organizations today, it probably remains in that realm. But don't expect it to stay that way.

"In the last few years, we've crossed a threshold," says James Kobielus, a former Forrester analyst who now serves as Big Data evangelist at IBM. "Suddenly, we're seeing multi-terabytes--tens, hundreds, even thousands of terabytes, aka petabytes--coming into analytic application environments in many industries. In many ways it's still a specialized environment; it's not as if every business intelligence application needs that much data under the covers just yet. But it's moving toward that volume over the next several years over a wide range of use cases."

But it would be a mistake to think that the promise--and challenge--of Big Data is simply tied up in the size of the data sets available for analysis. "In my opinion, 'Big Data' is kind of a misnomer," says Darren Catalano, associate vice president of business intelligence for the University of Maryland University College (UMUC). "We've always had systems that have generated a whole lot of data. It's just that now we're paying attention to that data. We used to look at it as something that facilitates business processes--something very operational. But now we're applying advanced analytic techniques to these large data sets."

And it is these analytic techniques that represent the golden key to unlocking the secrets of giant data sets. As analysts peer into these very large data sets, patterns emerge, nuances appear, and trends reveal themselves. Big Data provides more granularity than smaller data sets, and--if you've got enough of the right type of data--you get what Kobielus calls a 360-degree portrait of a student, his world, what's going on in his mind, and what he's likely to do.

"You can get a deeper and more nuanced portrait of what your customers--the students--like and don't like, or the kinds of courses they would like to sign up for, or the kinds of majors they want to pursue," explains Kobielus. "This isn't just a way of selling a customer more stuff or signing up students for more expensive courses; it's about making those students happier and more fulfilled, and maybe leading them on their way to faster completion of a course of study, based on giving them fine-tuned guidance throughout their time at the university."

The Three V's
The analytic techniques and algorithms needed to identify actionable trends are far more complex than similar efforts of previous decades, for one simple reason: The data sets encompass data points that go way beyond easily quantified measurements.

"If you go by the numbers, most of the data being generated now is unstructured and semi-structured data," says Anjul Bhambhri, IBM's vice president of Big Data. "It's not just social data, although you hear a lot of talk about social data in Big Data circles. But enterprises are collecting a lot of log data, and there's a structured and unstructured component to this type of data. For example, enterprises are doing a lot of work around network-performance management, predicting maintenance-schedule requirements, and analyzing logs. I was at a conference recently where a company with a solution in the log analytics space talked about 2.5 exabytes [2.5 quintillion bytes] of log data being getting generated every two days."

The sheer variety of data is one of three defining characteristics of Big Data, commonly referred to as the Three V's--variety, volume, and velocity. For many higher education institutions, velocity and variety--not volume--are the trickier aspects of Big Data to manage, says Charles Thornburgh, CEO and founder of Civitas Learning, an Austin, TX-based predictive analytics company.

"The variety of the data causes the first hurdle," notes Thornburgh. "Many different autonomous systems are collecting very different types of data. Even schools that have deployed a data warehouse are unlikely to be centralizing all of the data in a manner that supports empirical mining."

Today, school decisions about what data to collect are often based on compliance needs, rather than a desire to be able to use the data predictively. "Additionally," adds Thornburgh, "the ability to scale the processing of this data in a near-real-time or even a nightly job is something that most schools could find challenging."

Indeed, the velocity at which this data is collected is only going to get faster. "We're now moving more toward real-time, continuous-streaming acquisition of data from various sources for a lot of different applications," explains Kobielus.

To a certain degree, schools are beneficiaries of the fact that so much of their student information is stored in relational databases, making it easy to slice and dice the data. "Most of the data in higher education is dramatically more structured," Thornburgh notes, citing student information systems (SIS) in particular. "This is precise information about all these students: exactly who they are, which classes they took, what grades they received in those classes. Each of those events is not just the click of a keystroke; it's the manifestation of months of work on both the faculty member's and the student's behalf."

Unfortunately, the very structure that makes it easy to analyze aspects of student data stands at odds with another underlying concept behind Big Data: flexibility. While relational databases are great at serving up data for preconfigured purposes, it can be a bear to set them up to generate different results, even when the amount of data involved is of moderate size.

"That's one of the key limitations of traditional designs today," insists Charles Zedlewski, vice president of the products group at Cloudera, one of the leading commercial supporters of Hadoop and a range of Big Data solutions and services. "Typically, once you set up a database, it's difficult and expensive to change later. In Big Data, the whole point is that you're acquiring so much data that it's not realistic to assume up front all the different ways that you're going to use it. You can't possibly predict that. So how do you make it possible to experiment and change a lot at very little cost?"

Pressure to Change
Until recently, the relational database has worked fine, because the level of analysis was fairly superficial. Traditionally, higher education has tended to think of analytics as a matter of slicing and dicing basic outcomes information by criteria such as ethnicity, geography, or Pell status. Such an approach can identify big-picture gaps in equity and effectiveness, but it's not really predictive.

Looking ahead, such a broad-brush approach is unlikely to be viable. "Higher education is facing a lot of pressure these days from an accountability standpoint," notes Cole Clark, global vice president in database giant Oracle's education and research industries group. "It's really being pushed hard to improve student outcomes and demonstrate that the money spent on higher education is producing the kinds of outcomes we all want to see. That pressure has pushed schools to look at ways to pull meaningful data out of their SISs, LMSs, and, increasingly, social media."

Purdue is one school that has invested heavily in the development of homegrown systems that leverage these kinds of resources, but its CIO believes that the majority of colleges and universities have just scratched the surface of what might be possible with Big Data analytics.

"Every higher education institution has this data, but it just sits there like gold in the ground," explains McCartney. "They have to do more than just manage it. Big Data and the new tools we're seeing now are about mining that gold. It's about extracting real value from the data that we're now accumulating at a ferocious rate."

And, in McCartney's eyes, the payoff promised by Big Data exceeds even the initial hype. "I think Big Data is going to have a bigger impact on regular people than the internet," he declares. "The internet is basically just a delivery mechanism. Now I can read my newspaper online, I've got on-demand movies, I've got VoIP, I can query Google. There's not a lot in there I couldn't do before--the internet just gave me more convenient and universal access to it. But Big Data gives me something I've never had before."