Big data science accessing high-end HPC

Overview:

Big data science now demands processing capacity at the scale of the largest HPC systems in the world. A traditional highly distributed model is likely to be cost expensive. Research must therefore consider the very high-end HPC option to explore the full scientific potential of the collected data. Switzerland currently has the highest ranked Top 500 HPC machine in Europe (and the 8th worldwide), operated by the Swiss National Supercomputing Center (CSCS). We report on a project investigating the possibility of using Cray machines for the processing of data from the Large Hadron Collider at CERN in Geneva, a typical big science and big data case. The long existing Advanced Resource Connector (ARC) from the Nordugrid collaboration has a non-intrusive design and data management capabilities that enable the fulfilment of restrictive policies and I/O challenges related to high-end HPC systems. The report considers the setup solutions, the observed performance and is thus an example on how to use world leading HPC systems for data intensive science.

Target Audience:

Large data scientists and projects, high-end HPC providers

Benefits for Audience:

How to use the Advanced Resource Connector (ARC) to access high-end HPC systems

Topic 1: Challenges facing users and service providers