Learning Management Systems
Community Source Evaluation Strategies: Is Sakai Right for Your Institution?
The Sakai Project represents an emerging “community source” model for the design, development, and maintenance of a new breed of innovative learning environment. Marist College, a mid-size comprehensive liberal arts institution located in Poughkeepsie, New York, has been involved with the Sakai community since 2005 through the Sakai Education Partners program. Beginning in fall 2005, Marist engaged in a strategic planning and assessment process to evaluate Sakai as a possible long-term Course Management System (CMS) solution. After completing a yearlong pilot and assessment effort, Marist decided to adopt Sakai, branded locally as iLearn or Innovative Learning Environment and Research Network, to replace its existing CMS. The College-wide transition began in Fall 2008 and has just completed, after one academic year, with the start of the Fall 2009 semester.
Given that Sakai is considered an enterprise-level solution developed primarily by large research institutions, there is often a myth that it does not match the needs and resources of smaller non-research focused colleges and universities. My purpose here is to outline the rigorous evaluation process implemented by Marist College to determine whether Sakai was in fact the right solution for an institution its size and type. In addition, I will explain why Marist has concluded that Sakai will be a vital resource in its ongoing strategic focus on technology innovation.
As Marist began to look at Sakai it established five core “evaluation criteria” against which to assess and benchmark it with regards to two important questions: (1) Was Sakai the right strategic solution for the College?; and (2) If so, what was the appropriate timeline and transition strategy to move the institution to Sakai? These evaluation criteria were broken down into five primary categories: Functionality Requirements, Support Requirements, Sakai Community Health, Reliability/Scalability, and Innovation Drivers.
One of the most critical of the five primary criteria categories, particularly for the Marist faculty, was the area of Functionality Requirements. This category focused on the functionality or capabilities of Sakai and benchmarked them against both the capabilities of the College’s existing CMS solution and the new capabilities needed to meet objectives established in Marist’s Strategic Plan (e.g. ability to provide students with electronic portfolios). An extensive Functionality Gap Analysis was developed as part of this benchmarking process. The analysis was used to identify and track critical gaps in capabilities as Sakai evolved and new versions were released. Closing these gaps, which happened more rapidly than initially expected, was a vital part of the decision to adopt Sakai.
The most common concern regarding the adoption of open source solutions, and in particular Sakai, is the perception that additional development resources (i.e. programmers) will be required. This was the impetus for establishing “support requirements” as one of the major evaluation criteria. This criterion benchmarked what, if any, additional support resources were needed over and beyond the ones already in place to support the existing CMS solution. This included a staffing analysis involving interviews with IT offices at other institutions and discussions with commercial support providers (e.g. rSmart, Unicon, etc.). As Marist concluded research into this area it became clear that running Sakai was like “consuming” any other enterprise level system and that the myth that new development resources would be needed was just that, a myth.
When considering purchasing an enterprise-level system from a commercial provider there is the need to investigate the financial standing of that company given the significant investment of the purchase. The equivalent assessment in an open source or community source project involves looking at the “health” of the community. With this in mind the “Sakai Community Health” evaluation criteria were developed to research trends over time in such areas as the number of developers with commit access, the number of conference participants, and the number and type of institutional members. It also examined less quantifiable issues such as the Sakai Foundation governance structure and risk factors such as the Blackboard patient lawsuit. As we looked at these issues closely it was clear that the Sakai was a vibrant and growing community that would be stable for the long term.
With any mission critical system, reliability and scalability are important criteria against which to benchmark any strategic long-term decisions. Establishing the major evaluation criteria was necessary when considering a transition to Sakai. Reliability benchmarks were fairly standard and they included outcomes from performance, load, and quality assurance testing. Scalability benchmarks were initially established based on historical growth rates in use in the college’s course management system technology. When beginning to pilot Sakai, these benchmarks were significantly overhauled as the strategic value of Sakai to the college became apparent. For example, a “portfolio-for-life” initiative was developed, which allowed alumni to keep their electronic portfolios after graduating and enabled them to continue to add to the portfolios as they began their first job. This one initiative radically increased the potential scale the system would need to be able to support over time.
The most unique aspect of the evaluation criteria developed was the “Innovation Drivers” category. These innovation factors included: code-service decoupling, the concept of “built by educators, for educators,” “crowd sourcing,” and Service Orientated Architecture (SOA) that Sakai has followed. These factors were identified from early discussions with Sakai community members as potential innovations that would flow from adopting a community source solution. Understanding these factors and assessing the real potential of benefiting from them became the basis for establishing the Innovation Driver benchmark. The following paragraphs examine these innovation factors in detail and why they were critical benchmarks for Marist.
“Code-services decoupling” refers to the separation of system code from the support services that are commonly bundled together by commercial vendors in today’s market place. This bundling often leads to the phenomena of “vendor lock-in” sometimes resulting in lower quality support services, a frequent complaint among institutions using the dominant CMS commercial solutions. In community source models, the community provides the system code and the support services can be provided in-house or purchased from a commercial provider. This “decoupling” had the potential to drive significant increases in the quality of support services.
The fact that Sakai is not driven by a profit-margin and is instead “built by educators, for educators” is another important factor that had the potential to drive innovation. Since adoption rates for new and innovative instructional technologies tend to be low (due to human resistance to change) they often do not result in quick profits. In a community source model, decisions on building innovative teaching and learning tools are not made based solely on the financial analysis of their profitability. Instead, educators have direct input into the process and if there is enough interest in a new capability it will generally be built. Thus, once the profit-margin is removed from the “development equation” there is potential for significant innovation to result in the design and development of new tools and capabilities.
Crowd sourcing is the concept of distributing work and innovation across large populations. As more and more individuals and groups begin to contribute to community source projects a massive collective of brain power begins to assemble. As numbers grow from 100s to 1,000s to 10,000s of contributors, the chance that a few radically innovative ideas will develop increases significantly. By becoming part of a community source effort, institutions are able to tap this massively large collective knowledgebase and benefit from the innovation that emerges.
Finally, Marist saw service orientated architecture or SOA as another major potential innovation factor because it allows for highly customized systems without the high price tag that comes with such customization today. The ability to deploy something as complex as a course management system in ways that are customized to meet the unique needs of different learner populations has huge a potential to improve the teaching and learning process. Today, such customization is cost prohibitive and thus the innovation potential is not realized at most institutions. SOA could eliminate or reduce the cost barrier allowing institutions of all sizes to customize their deployments of complex systems.
In Fall 2008, when at we Marist began our transition to Sakai, we initially set the goal of having 20 percent of our faculty “opt in” to using it during that first semester. In the end, we had 65 percent opt to convert over to Sakai, which we feel is great testimony from our faculty and students that the rigorous evaluation process we deployed was worthwhile and successful.
