Qualified Free Access to Advanced Compute Resources with NSF's Jetstream2 and ACCESS
A Q&A with David Hancock and Jennifer Taylor
NSF-funded Jetstream and now Jetstream2 resource providers have, for about a decade, served up advanced and high performance compute resources to researchers and education programs approved for free allocations through NSF's ACCESS program.
This coordination of Jetstream2 and ACCESS has helped many otherwise under-resourced research and education projects succeed. And now, with the growing emphasis on data- and compute-intensive programs on our campuses, and a need to find an "on-ramp" to the nation's advanced cyberinfrastructure, many more institutions will obtain ACCESS allocations and look to the cloud-based Jetstream2 program for a wide variety of in-demand high performance compute resources.
For some background and a look at the process and where to start, CT visited with Jetstream2 PI David Hancock and Jetstream2 Education, Outreach, and Training Leader Jennifer Taylor, speaking to us from their Jetstream2 home base at Indiana University-Bloomington.
Mary Grush: David, as the Jetstream/Jetstream2 PI, could you tell us a little about how Jetstream and Jetstream2 began and the history of establishing them? How are these programs funded?
David Hancock: The initial conception of Jetstream occurred about 10 years ago with a National Science Foundation award in 2014, starting as a pilot and production system for deploying a new high performance infrastructure that would broaden access for individual researchers and education programs that need advanced or HPC resources.
Building on that, the project that we currently reference most today, is Jetstream2. It began in 2020 and is now in its third year of operation. Jetstream and Jetstream2 are separate awards from different NSF programs, but they have operated concurrently, each contributing to the momentum.
These NSF awards are intended for five operational years. So we currently have two more years of operations — actually about two and a half years at this point. After its third year, Jetstream2 will be up for potential renewal of the resource.
Grush: Jenn, given your extensive interactions with users, could you speak about how Jetstream and now especially Jetstream2 work, and tell us about obtaining allocations?
Jennifer Taylor: Sure. The original Jetstream was part of an NSF program called XSEDE, which later became, and is still called ACCESS. The process of getting access to the resources provided by these programs starts with creating your ACCESS ID and applying for and receiving an ACCESS allocation. So the model of access is for potential users to apply for allocations, which are kind of like small grants. The allocations are typically awarded for one year.
Jetstream and now Jetstream2 are referred to as resource providers for the ACCESS program, so ACCESS and Jetstream/Jetstream2 are very closely enmeshed.
Grush: These days, are the resources in the cloud?
Taylor: Jetstream2 is a cloud computing environment. So all of our resources exist in the cloud, available on demand once you've applied for and received your allocation.
Jetstream2 is a cloud computing environment. So all of our resources exist in the cloud, available on demand once you've applied for and received your allocation.
At that point, you have the ability to launch virtual machines, also called instances, on Jetstream2. And you have full admin access to them. You can install whatever software that you might have the license for, as well as applications from our software collection. You have a lot of ownership of these resources.
Grush: Can users define the kind of resources they'll need? How can they qualify for the level of resources they decide they'll need?
Taylor: With ACCESS, there are different levels, based on the kind of computing the user might need to be doing. Basically there are four tiers. The higher the tier you'll have, the more credits and more compute time and power you can get.
One of the biggest draws for our users is that there's no cost to users. You don't have to pay anything. But at that point you've earned it: You really do have to spend time and effort talking about your project and applying for what you plan to do.
Once you have applied for an allocation, assuming that you are approved, you are then awarded credits; virtual credits, which you exchange for your use of resources. Again, setting up your allocation is all under the ACCESS program.
Assuming that you are approved, you are then awarded credits; virtual credits, which you exchange for your use of resources.
I also want to mention that Jetstream2 is actually four separate kinds of resources: standard compute, GPUs, large memory, and storage. Users can allocate credits toward some, or all, of these different options, depending on what their work calls for.
Grush: So the credits available to you from your ACCESS award you apply as you go, so to speak, for use of your desired Jetstream2 resources?
Taylor: Yes. You exchange your credits toward your use of resources. Once you have an active allocation, and you've exchanged credits toward Jetstream2 resources, that's when you can get started actually using Jetstream2 resources.
It's also notable that there's a variety of resources available from other resource providers as well, that your credits can be used for, as the full resource catalog shows.
For those who are looking into Jetstream2, there's ample information on how to get started on the "Get Started" page of our website.
For those who are looking into Jetstream2, there's ample information on how to get started on the "Get Started" page of our website.
Grush: David, today, who is Jetstream2 positioned to serve?
Hancock: The National Science Foundation has a broad mandate for science and is serving many agencies and institutions — not just NSF-funded research. Jetstream and Jetstream2 have been targeted at researchers,
educators, and engineers from the hard sciences;
those in fields like physics or chemistry; and some representative of the other disciplines, like biology or sociology.
With the rise of high-end interactive tools and innovative field work in research and education programs, as well as experimental labs with workstations that connect to a central infrastructure or science gateway, we may be serving individuals who are using advanced computing resources through a web portal and browser that's catered specifically to their domain or educational need.
Still, as inviting as that sounds, I don't want say we are offering "user-friendly" interfaces because an interface cannot be friendly… but through Jetstream2 we are providing an on-ramp to the national infrastructure that's a lot easier to use and trying to meet people where they are, technically. So we have, I think, a real innovation — while also bringing cloud computing paradigms and cloud-native technologies into the research and education ecosystem.
Through Jetstream2 we are providing an on-ramp to the national infrastructure… while also bringing cloud computing paradigms and cloud-native technologies into the research and education ecosystem.
Grush: As HPC and research technologies become more pervasive in higher education — in undergraduate as well as graduate programs — will research and education programs have access to the advanced compute resources they'll need? Will Jetstream2 be able to help?
Hancock: Absolutely. I think that is one of our key missions, and our calling to help with — including the workforce development aspects of that whole picture.
This is already a very active area beyond our efforts. There are specific calls and other resources, such as the National AI Research Resource call. More specifically, NSF has put out a "dear colleague" letter that invites people to apply for modest funding to support some of this education and workforce development.
And we are here not only to support individual researchers and educators, but also to help the operators that may want to provide similar infrastructure at their institutions.
Grush: How could advances in cyberinfrastructure affect the future of how Jetstream2 provides access to resources for those who need them?
Hancock: Great question. This is something quite unique about the Jetstream2 project compared with other supercomputing or cluster deployments: Often times you'll see a project deployed well at the outset, but that's the way it is, ongoing, with only some minor revisions, for many years.
Whereas with Jetstream and Jetstream2, we specifically and purposefully embrace the changeable nature of piloting new services. All the software environments and interfaces that are on top of a system and updated as the community needs have been adapted since we first started Jetstream.
With Jetstream and Jetstream2, we specifically and purposefully embrace the changeable nature of piloting new services.
For example, we didn't think there was going to be a large demand for API interfaces or for jumping into the back end of cloud computing. But we found that about half of the people coming to the system were interested in that sort of access, so we enabled that very quickly.
We've also partnered with other projects to provide new interfaces. For example, we were early in deploying the ability for people to use Jupyter notebooks — and now JupyterHubs — with orchestrated environments.
And in gesturing today through interfaces like Exosphere as well as CACAO, these are tools that can be orchestrated with larger deployments of resources, whether that's dozens of virtual machines or deploying through Kubernetes or other specific technologies.
It wasn't clear when we started the Jetstream project what technologies would take off. Docker Swarm was probably the most popular at the time in addition to Apache Mesos. And now, Kubernetes and containers are everywhere. So, we've been able to meet all of that in terms of thinking ahead to what's next.
Of course, we don't very often really know what's next. This is where the folks in the classroom or lab may come to us with ideas or challenges that they have. And that's why we have advisory boards with our partners, and forms and surveys to watch for what the next pilot needs may be. And we watch for what we can do to continue to develop and make that process easier. We'll continue to push to evolve these types of things, because the demand to evolve is our hope.
We have advisory boards with our partners, and forms and surveys to watch for what the next pilot needs may be… The demand to evolve is our hope.
Grush: What makes that kind of evolution possible for a project like this? What's the "secret sauce" for Jetstream2's success in this evolution, and its success in general?
Hancock: It's because of a great team and project partners. We have partners on this project from the University of Hawaii, University of Arizona, Arizona State University, University of Texas/Texas Advanced Computing Center, Cornell, the San Diego Supercomputer Center, and of course, Indiana University. Such partners and advisors really make these large-scale projects work, and I think that together we can all truly magnify the capabilities of any one institution.
[Editor's note: Jetstream2 logo and image courtesy of Jetstream2 and Indiana University. With permission.]