Intel Intros HPC Distribution for Apache Hadoop for Storing and Processing Big Data

Intel has rolled out its new Intel HPC Distribution for Apache Hadoop software, an enterprise-grade solution designed for storing and processing large data sets.

The HPC (high-performance computing) distribution combines the Intel Distribution for Apache Hadoop software with the Intel Enterprise Edition of Lustre software. According to the company, the Intel Distribution for Apache Hadoop "provides distributed processing and data management for enterprise applications that analyze massive amounts of diverse data," while the Enterprise Edition of Lustre is designed "to make performance-based storage solutions easier to deploy and manage."

According to the company, Lustre is the most widely deployed file system for HPC, and Apache Hadoop can transform big data into manageable, distributed datasets for easier analysis. By combining these two tools, the Intel HPC Distribution for Apache Hadoop software can help organizations analyze and manage their big data by enabling them to run their MapReduce applications directly on Lustre-powered storage for improved task performance, storage scalability, and storage management. They can access their Lustre files directly from Hadoop, without first copying them to the Hadoop environment.

Intel is also collaborating with its partners in the HPC community to deliver customized HPC products "to meet the diverse needs of customers," according to the company. Intel hopes these customized HPC products will help organizations achieve scientific, industrial, and academic breakthroughs.

Further information about the Intel HPC Distribution for Apache Hadoop software can be found at intel.com.

About the Author

Leila Meyer is a technology writer based in British Columbia. She can be reached at [email protected].

Featured

  • blue and green lines intersecting and merging in an abstract pattern against a light gray background with a subtle grid design

    Data Integration Market: Cloud Giants Down, AI Up

    "By 2027, AI assistants and AI-enhanced workflows incorporated into data integration tools will reduce manual intervention by 60 percent and enable self-service data management," according to research firm Gartner.

  • abstract geometric pattern of glowing interconnected triangles, hexagons, and circles in blue, gold, and white, spread across a dark navy-to-black gradient background

    OpenAI Unveils 'Operator' AI for Performing Web Tasks

    OpenAI has launched "Operator," an AI agent designed to perform web-based tasks autonomously using its own browser. Currently available as a research preview for Pro users in the United States, the tool aims to automate everyday activities such as filling out forms, ordering groceries, and even creating memes.

  • silhouetted human figures stand opposite a glowing digital brain, surrounded by abstract circuits and shadowy shapes

    Global Tech Execs Expect Advancements in AI to Increase Security Threats

    Forty-one percent of global tech executives in a recent NetApp survey said they believe advancements in AI will significantly increase security threats. The firm's second annual Data Complexity Report points to 2025 as "AI's make or break year."

  • glowing hexagons and circles connected by illuminated lines

    Microsoft Intros Open Source Multi-Agent AI System

    Microsoft researchers have unveiled a new open source multi-agent AI system, Magnetic-One, aimed to help enterprises automate complex tasks typically requiring human intervention.