Search results

(1 - 2 of 2)
ACCELERATING I/O USING DATA LABELS: A CONTENTION-AWARE, MULTI-TIERED, SCALABLE, AND DISTRIBUTED I/O PLATFORM
Title
ACCELERATING I/O USING DATA LABELS: A CONTENTION-AWARE, MULTI-TIERED, SCALABLE, AND DISTRIBUTED I/O PLATFORM
Creator
Kougkas, Antonios
Date
2019
Description
Parallel file systems (PFS) have been the dominant storage solution in High-Performance Computing (HPC) for several years. However, as we move...
Show moreParallel file systems (PFS) have been the dominant storage solution in High-Performance Computing (HPC) for several years. However, as we move towards the exascale era, PFS have several limitations, such as scalability, complexity, metadata, data synchronization, and access latency, which can seriously affect storage's performance. These challenges along with the unprecedented data explosion accentuated the research conundrum known as I/O bottleneck. Moreover, the extreme computing scale, that exascale machines promise, brings forward another important limitation of the existing I/O path. Multiple large scientific applications will be accessing shared storage resources at the same time, and thus, will be competing. This phenomenon is known as cross-application I/O interference and is one of the most challenging performance degradation factors, even in today's petascale. To address some of the above issues, modern system designs have introduced a new memory and storage hierarchy, filled with novel special hardware technologies, that aims to ease, in a sense, the I/O bottleneck. However, software for management, I/O scheduling, and efficient data movement in this new complicated landscape of multi-tiered I/O infrastructure is limited at best. The added complexity of data access using buffering resources needs to be addressed and is of the utmost priority of several scientific sites and communities. This study makes steps towards I/O acceleration in HPC by proposing: a) a new subsystem for the I/O convergence between HPC and BigData storage ecosystems, b) a new subsystem equipped with several advanced I/O buffering techniques for the deep memory and storage hierarchy, and c) a new subsystem that implements several I/O scheduling algorithms to prevent the negative effects of I/O contention, and d) a new storage system that relies on a novel abstract notion of a data label that allows the I/O system to provide storage flexibility, versatility, agility, and malleability. The proposed work has been evaluated and results suggest that substantial improvements in I/O performance have been achieved.
Show less
A Multi-level Data Integration Approach for the Convergence of HPC and Big Data Systems
Title
A Multi-level Data Integration Approach for the Convergence of HPC and Big Data Systems
Creator
Feng, Kun
Date
2020
Description
HPC is moving towards exascale (10^18 operations per second) following the trend that has continued for over half a century. Such an extremely...
Show moreHPC is moving towards exascale (10^18 operations per second) following the trend that has continued for over half a century. Such an extremely compelling computing power brings huge opportunities for scientists to explore their problems with larger sizes and finer granularity. As a result, the data volume produced and consumed by extreme-scale computing has increased dramatically. To gain useful scientific insights, scientists analyze tremendous amounts of data, which stresses the storage systems and requires efficient data access. Besides the data volume increase, the variety of I/O subsystems grows as well to meet the drastically different, often conflicting I/O requirements of numerous applications. HPC and BD, as two major camps of extreme-scale computing, have been developed separately for a long time and diverged from computing and storage paradigms. However, recent developments have proven the convergence of them leads to more efficient scientific output. Hence, unification between these ecosystems is necessary to accelerate extreme-scale computing with the collaboration of applications from both camps. Therefore, integrated I/O has become a major issue that needs to be addressed as the extreme computing community moves forward.This study explores improvement by proposing a new integrated data access system for extreme-scale computing. We enhance the BD framework to adapt to the change of integrated data access requirement by enabling direct processing of scientific data from PFS at the HPC site. Our framework can perform up to 8x faster than the state-of-the-art solutions in representative workloads. We design a new advanced I/O middleware service to utilize data aggregation resources to facilitate integrated data access in scientific workflows with both HPC and BD applications. Our middleware service can reach up to 10x speedup against the default solution and 133% better performance than existing solutions. We propose a novel storage integration solution on the storage side to unite all the storage resources, to unify the namespace across all the storage systems, and provide an ultimate integrated data access service. The integrated solution can speed up a real workflow with integrated data access requirements by up to 6.86x over existing solutions. The three-level integration at the application level, middleware level, and storage level provide us a systematic hierarchical I/O integration. Our implementation results show that the three-level optimized design and implementation is feasible and effective. It improves the state-of-the-art solutions and helps us to achieve an enhanced I/O system towards extreme-scale computing to support both HPC and BD applications.
Show less