Parallel file systems (PFS) have been the dominant storage solution in High-Performance Computing (HPC) for several years. However, as we move... Show moreParallel file systems (PFS) have been the dominant storage solution in High-Performance Computing (HPC) for several years. However, as we move towards the exascale era, PFS have several limitations, such as scalability, complexity, metadata, data synchronization, and access latency, which can seriously affect storage's performance. These challenges along with the unprecedented data explosion accentuated the research conundrum known as I/O bottleneck. Moreover, the extreme computing scale, that exascale machines promise, brings forward another important limitation of the existing I/O path. Multiple large scientific applications will be accessing shared storage resources at the same time, and thus, will be competing. This phenomenon is known as cross-application I/O interference and is one of the most challenging performance degradation factors, even in today's petascale. To address some of the above issues, modern system designs have introduced a new memory and storage hierarchy, filled with novel special hardware technologies, that aims to ease, in a sense, the I/O bottleneck. However, software for management, I/O scheduling, and efficient data movement in this new complicated landscape of multi-tiered I/O infrastructure is limited at best. The added complexity of data access using buffering resources needs to be addressed and is of the utmost priority of several scientific sites and communities. This study makes steps towards I/O acceleration in HPC by proposing: a) a new subsystem for the I/O convergence between HPC and BigData storage ecosystems, b) a new subsystem equipped with several advanced I/O buffering techniques for the deep memory and storage hierarchy, and c) a new subsystem that implements several I/O scheduling algorithms to prevent the negative effects of I/O contention, and d) a new storage system that relies on a novel abstract notion of a data label that allows the I/O system to provide storage flexibility, versatility, agility, and malleability. The proposed work has been evaluated and results suggest that substantial improvements in I/O performance have been achieved. Show less
