Disaster Recovery
After Hurricane Maria, Puerto Rico University Turns to the Cloud to Restore Student Services
When September's massive storm knocked out access to electricity, clean water and communications for the entire island, Universidad del Sagrado Corazón needed to get up and running fast. Thanks to an extraordinary IT team and the resources of the cloud, the school was back in action within a few weeks.
- By Dian Schaffhauser
- 11/16/17
Universidad del Sagrado Corazón was able to resume classes both on campus and online within weeks of the disaster.
On Sept. 20, when Hurricane Maria slammed into Puerto Rico with 155 mile-per-hour winds, the storm knocked out the island's energy grid, eliminating access to electricity and clean water for almost everybody. It also tore down 1,360 of the 1,600 cellphone towers on the island, leaving many people in communication isolation. As the territory's governor, Ricardo Rosselló, declared in a prepared statement, "What Puerto Rico is experiencing after Hurricane Maria is an unprecedented disaster." The "devastation," he added, "is vast. Our infrastructure and energy distribution systems suffered great damages."
Amid the wide-scale destruction of homes and businesses, colleges and universities on the island also closed down. But the head of one school, Universidad del Sagrado Corazón, wasn't going to wait for federal action to get up and running again. President Gilberto Marxuach turned to Dynamic Campus, a service provider on the mainland the institution had been in recent negotiations with for an ERP integration project.
Richard Middaugh, senior VP and founding member of Dynamic Campus, remembered the call from Marxuach. It came on Sept. 29, nine days after Maria. His main question: "Could you guys get me to the cloud? I'm not going to have electricity here for months or maybe a year, and I want to continue to deliver coursework and services to our students."
A miraculous two and a half weeks after that call, Sagrado was back in action, registering students, issuing financial aid and delivering courses on campus and online. In fact, enrollment data shows an increase from the previous semester of more than a hundred undergraduates.
Getting Back to Business
Sagrado is a four-year private not-for-profit in Santurce, an 11-minute drive from Old San Juan. The institution, one of the oldest and largest in the territory, didn't lose buildings, but it did lose trees — about a hundred of them, many of which had toppled two weeks earlier, victims of Hurricane Irma.
When the call from Marxuach came in, Middaugh and Mike Glubke, Dynamic Campus president and CEO, sprang into action to perform a quick assessment.
There were four key applications the university needed in place to be able to operate:
- The enterprise resource planning system, Jenzabar EX, to handle admissions, registrations and transcripts;
- The learning management system, Moodle;
- The financial aid management system, the College Board's PowerFAIDS; and
- The institutional finance system, Kuali.
This software had been hosted in a data center on 20 servers. In normal circumstances, Dynamic Campus would have figured out what was on each server, gained access to those systems and planned the migration to Amazon Web Services. But nothing was normal for Sagrado at this time. For example, Middaugh's team had to work around generator runtimes. The generator in place at the data center was intended to provide power during intermittent outages, not continuously. As a consequence, he said, "every two days it has to have about eight hours of maintenance done on it." That included changing the oil and filters, which school personnel had to wait to have shipped to them since there were no filters left on an island where everybody was relying on generators for their power.
On top of that Middaugh and his staff couldn't simply fly to Sagrado to do the work. For a long time, all commercial flights had been suspended and the airport closed. But even if they'd been able to get to the island, all the hotels were either damaged or occupied by FEMA workers. "We couldn't get a hotel room until the first of November, which is when we went down," he said.
So, in the early days of the project, they relied mostly on text messages, "which for some reason worked better than phone calls. Phone calls were choppy and hard to understand," Middaugh recalled. Eventually, the two teams — Middaugh's on the mainland and Sagrado's in the data center — would hold daily scheduled phone calls. When that wasn't enough, they continued texting and using e-mail. "It took a lot of logistical coordination," he noted.
But no matter what form the communications took, there was an extra layer of challenge to the project. While the senior people at the university spoke both Spanish and English, "junior" people were primarily Spanish language speakers. While Dynamic Campus has bilingual speakers, there weren't as many as the company needed for this initiative, Middaugh acknowledged. Google's translation features "helped some," he said.
A Three-Week Migration Timeline
On the same day that the call from the university president came in, Dynamic Campus kicked off its planning, which was firmed up by Oct. 1.
Between Oct. 1 and 3, the company set up the structure on AWS that would host Sagrado's applications and data, including servers, storage, backups and internet links.
On Oct.3 and 4, the service provider worked with a San Juan computer security firm to set up a virtual private networking tunnel through which all data could be transferred in encrypted form.
Then the generator ran out of fuel. Since road closures prevented resupply trucks from arriving, the project was put on hold until Oct. 7, when the fuel could finally be delivered.
Between Oct. 7 and 10, Dynamic Campus copied everything from Sagrado to AWS. That was tackled in two "major bursts," said Middaugh. First, the backup data was copied in order to get an estimate how long the transfer of live data would take. Then Sagrado staffers were scheduled to stay out of the system for that same period to ensure the completeness and integrity of the transfer of about 8 terabytes of that data.
Starting on Oct. 10 and ending on Oct. 14, the company undertook database configuration and installation of the software.
During the course of the migration, the university was moved to the newest versions of its software. That's where minor problems surfaced, said Middaugh. PCs weren't set up to work with the newer applications, so those had to be tweaked. In addition, the three systems undertaken first — Jenzabar, PowerFAIDS and Moodle — were tied to Microsoft Active Directory for user account management, and that needed to be set up.
Starting on Oct. 14 Sagrado began performing user testing to make sure transactions could be completed and that users could do what they expected to be able to do.
On Oct. 16, classes resumed.
Students attending classes on campus mostly sit outdoors under tents, in gazebos and in breezeways.
Let the Classes Begin
Because there's no air conditioning and few fans for the classrooms, students attending classes on campus mostly sit outdoors under tents and in breezeways. The school now has running water for the restrooms — but it's not potable, so the university makes bottled water available.
Those attending courses online are doing so through Moodle Mobile on their phones and tablets. Anybody who has access to generated power can register and take classes. Last year, the institution offered 175 Moodle courses with 778 sections; currently, the school has 67 courses with 117 sections. In spite of the gap between what was and what is, said Middaugh, campus leaders are "amazed" by the attendance numbers.
While many of the previous online offerings were courses for dedicated online programs, now the school is creating core courses as quickly as it can. In fact, added Middaugh, Sagrado has put the call out for foundational bilingual courseware already designed for Moodle or that can be ported from another LMS. The campus is working with a learning consultant from the City University of New York on this effort, Middaugh noted, but access to additional offerings would enable it to get more courses up more quickly.
According to the institution, undergraduate enrollment has grown semester-over-semester. The 2016 enrollment for undergrads was 4,302; in 2017, that number increased to 4,411 students. Graduate enrollment, however, a took a dive from 519 to 413 over the same period. One reason for the overall increase: Sagrado is operational even as other institutions on the island are still struggling to open, so it's seeing transfer students come in to continue their studies.
Peering into the Future
In the intervening weeks, Dynamic Campus has undertaken the move of the university's financial system, Kuali, to AWS as well. Once that's done, Kuali will be connected to the campus's ADP service, which handles payroll.
But the work doesn't end there, said Middaugh. Marxuach has expressed concern regarding the relative vulnerability of the fiber optic submarine data communications cables that link Puerto Rico to the rest of the region. That cabling runs right over the Puerto Rico Trench, an oceanic trench that some geophysicists consider overdue for a major earthquake. Should that kind of disruption occur, the university president wants to be ready with alternatives for keeping internet services running.
"We don't know what we're going to do right now," Middaugh admitted. "But our initial thoughts are to test satellite communications with AWS from the island." On an upcoming trip, his crew will travel to San Juan armed with satellite modems, to see whether transactions back to Amazon will work via satellite.
A team of 10 at Dynamic Campus was put on the project, working 24/7 during the weeks it took to get the university's systems onto the cloud. But even then, Middaugh emphasized, that still doesn't come close to what Sagrado accomplished for itself. "I have to give it to those people," he said. "They did [what we did] with no running water, no way to cook, no air conditioning and in 90-degree heat in closed, confined buildings. They had it a lot worse than we did. They have done a remarkable job."
