Test Whats new message.
Scheduled Maintenance on Monday 1/13/2025
Single article sales and account management will be unavailable from 5:00 AM - 7:00 PM ET (09:00 - 23:00 UTC). We apologize for the inconvenience.
Showing 1-25 of 399 results
- Sort
Filter Results
Show
Sort By
Results
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
Lang Xu;Quentin Anthony;Jacob Hatef;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
Scaling up Large Language Model(LLM) training involves fitting a tremendous amount of training parameters across a limited number of workers. However, methods like ZeRO-3 that drastically reduce GPU memory pressure often incur heavy communication to ensure global synchronization and consistency. Established efforts such as ZeRO++ use secondary partitions to avoid inter-node communications, given t...Show More
HyperSack: Distributed Hyperparameter Optimization for Deep Learning using Resource-Aware Scheduling on Heterogeneous GPU Systems
Nawras Alnaasan;Bharath Ramesh;Jinghan Yao;Aamir Shafi;Hari Subramoni;Dhabaleswar K Panda
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
Hyperparameter Optimization (HPO) can unlock the full potential of Deep Learning (DL) models; however, it is considered one of the most compute-intensive tasks in the DL do-main due to multi-dimensional search spaces and complex neural network architectures. A common method for accelerating HPO workloads is parallelizing training jobs on multiple computing devices, such as modern GPUs in High-Perf...Show More
Using BlueField-3 SmartNICs to Offload Vector Operations in Krylov Subspace Methods
Kaushik Kandadi Suresh;Benjamin Michalowicz;Nick Contini;Bharath Ramesh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar Panda
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
Modern SmartNICs are capable of performing both computation and communication operations. In this context, past works on accelerating HPC/DL applications have manually selected some computational phases for offloading them to the SmartNICs. In this work, we identify Vector Multiply-Adds (VMA), Distributed Dot Products (DDOT), and Sparse Matrix-Vector Multiplication (Matvec) as three fundamental op...Show More
Design and Implementation of Kernel-based MPI Reduction Operations for Intel GPU s
Chen-Chun Chen;Goutham Kalikrishna Reddy Kuncham;Hari Subramoni;Dhabaleswar K. Panda
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
The demand for computing power in high- performance computing and deep learning applications is steadily increasing, leading to a noticeable inclination toward equipping modern exascale clusters with accelerators. In particular, dis-tributed Deep Learning training necessitates high-performance G PU -aware MPI operations, with reduction operations being widely employed. Unlike data movement-based M...Show More
Effective and Efficient Offloading Designs for One-Sided Communication to SmartNICs
Ben Michalowicz;Kaushik Kandadi Suresh;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar K. Panda;Steve Poole
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
One-sided communication is one of many approaches to use for data transfer in High-Performance Computing (HPC) applications. One-sided operations require less demand on parallel programming libraries and do not require HPC hardware to issue acknowledgments of successful data transfer. Thanks to its inherently non-blocking nature, one-sided communication is also useful for improving overlap between...Show More
MPI Allgather Utilizing CXL Shared Memory Pool in Multi-Node Computing Systems
Hooyoung Ahn;Seonyoung Kim;Yoomi Park;Woojong Han;Shinyoung Ahn;Tu Tran;Bharath Ramesh;Hari Subramoni;Dhabaleswar K. Panda
2024 IEEE International Conference on Big Data (BigData)
Year: 2024 | Conference Paper |
In Artificial Intelligence (AI) and high-performance computing (HPC), growing data and model sizes require distributed processing across multiple nodes due to single-node limitations, increasing inter-node communication. To address these challenges, we propose a novel MPI allgather method leveraging CXL technology, which supports composable architectures and dynamic resource allocation in data cen...Show More
Accelerating Large Language Model Training with Hybrid GPU-based Compression
Lang Xu;Quentin Anthony;Qinghua Zhou;Nawras Alnaasan;Radha Gulhane;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Year: 2024 | Conference Paper |
Cited by: Papers (1)
Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) are the three strategies widely adopted to enable fast and efficient Large Language Model (LLM) training. However, these approaches rely on data-intensive communication routines to collect, aggregate, and re-distribute gradients, activations, and other important model information, which pose significant overhead. Co-desi...Show More
Characterizing Communication in Distributed Parameter-Efficient Fine-Tuning for Large Language Models
Nawras Alnaasan;Horng-Ruey Huang;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
2024 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2024 | Conference Paper |
Parameter-efficient Fine-tuning (PEFT) methods have emerged as powerful techniques for adapting pre-trained Large Language Models (LLMs) to specific tasks with reduced computational and memory overhead. However, despite their promising potential, there remains a gap in understanding how these methods perform in distributed computing settings. In this paper, we present a comprehensive characterizat...Show More
Demystifying the Communication Characteristics for Distributed Transformer Models
Quentin Anthony;Benjamin Michalowicz;Jacob Hatef;Lang Xu;Mustafa Abduljabbai;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
2024 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2024 | Conference Paper |
Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication beha...Show More
OHIO: Improving RDMA Network Scalability in MPI_Alltoall Through Optimized Hierarchical and Intra/Inter-Node Communication Overlap Design
Tu Tran;Goutham Kalikrishna Reddy Kuncham;Bharath Ramesh;Shulei Xu;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar K. DK Panda
2024 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2024 | Conference Paper |
The presence of exascale computers has pushed a new boundary in computing capability, which poses performance challenges in parallel programming models on how to exploit such systems efficiently. A dominant programming model for running parallel programs is the Message Passing Interface. Among primitives provided by MPI, Alltoall is a communication-intensive operation, which is utilized by many ap...Show More
Towards Accelerating k-NN with MPI and Near-Memory Processing
Hooyoung Ahn;Seonyoung Kim;Yoomi Park;Woojong Han;Nick Contini;Bharath Ramesh;Mustafa Abduljabbar;Dhabaleswar K. Panda
2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2024 | Conference Paper |
Message Passing Interface (MPI) is a common parallel programming model in High-Performance Computing fields. Recently, it has been widely used in Artificial Intelligence (AI) applications. However, the performance of those applications is limited by the memory wall problem, which is the performance gap between the processor and memory. To address these problems, we propose a novel computing archit...Show More
PML-MPI: A Pre-Trained ML Framework for Efficient Collective Algorithm Selection in MPI
Mingzhe Han;Goutham Kalikrishna Reddy Kuncham;Ben Michalowicz;Rahul Vaidya;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2024 | Conference Paper |
The Message Passing Interface is the de facto standard in high-performance computing (HPC) for inter-process communication. MPI libraries employ numerous algorithms for each collective communication pattern whose behavior is largely affected by the underlying hardware, communication pattern, message size, and number of processes involved. Choosing the “best” algorithm for every possible scenario i...Show More
Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
Jinghan Yao;Quentin Anthony;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2024 | Conference Paper |
Cited by: Papers (2)
In the realm of large language models (LLMs) like the Generative Pre-trained Transformer (GPT), the Mixture of Experts (MoE) paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, the deployment of GPT MoE models for parallel inference on distributed systems presents significant challenges, primarily due to the extensive Alltoall communication requir...Show More
HINT: Designing Cache-Efficient MPI_Alltoall using Hybrid Memory Copy Ordering and Non-Temporal Instructions
Bharath Ramesh;Nick Contini;Nawras Alnaasan;Kaushik Kandadi Suresh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. D K Panda
2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2024 | Conference Paper |
Modern multi/many-core processors in HPC systems have hundreds of cores with deep memory hierarchies. HPC applications run at high core counts often experience contention between processes/threads on shared resources such as caches, leading to degraded performance. This is especially true for dense collective patterns, such as MPI_Alltoall, that have many concurrent memory transactions. The orderi...Show More
Accelerating MPI AllReduce Communication with Efficient GPU-Based Compression Schemes on Modern GPU Clusters
Qinghua Zhou;Bharath Ramesh;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
ISC High Performance 2024 Research Paper Proceedings (39th International Conference)
Year: 2024 | Conference Paper |
With the increasing scale of High-Performance Computing (HPC) and Deep Learning (DL) applications through GPU adaptation, the seamless communication of data stored on GPUs has become a critical factor in enhancing overall application performance. AllReduce is a communication collective operation that is commonly used in HPC applications and distributed DL training, especially Data Parallelism. Dat...Show More
Flover: A Temporal Fusion Framework for Efficient Autoregressive Model Parallel Inference
Jinghan Yao;Nawras Alnaasan;Tian Chen;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
Autoregressive models, despite their commendable performance in a myriad of generative tasks, face challenges stemming from their inherently sequential structure. Inference on these models, by design, harnesses a temporal dependency, where the current token's probability distribution is conditioned on preceding tokens. This inherent characteristic severely impedes computational efficiency during i...Show More
Optimized All-to-All Connection Establishment for High-Performance MPI Libraries Over InfiniBand
Shulei Xu;Goutham Kalikrishna Reddy Kuncham;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2023 | Conference Paper |
In modern multi-/many-core HPC systems, the increasing number of processor cores presents new challenges in managing parallel compute workloads across multiple nodes. One crucial aspect that significantly impacts the startup phase of parallel MPI jobs is the methodology used for connection establishment. In this paper, we investigate the limitations of existing all-to-all connection establishment ...Show More
Performance Characterization of Using Quantization for DNN Inference on Edge Devices
Hyunho Ahn;Tian Chen;Nawras Alnaasan;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
2023 IEEE 7th International Conference on Fog and Edge Computing (ICFEC)
Year: 2023 | Conference Paper |
Cited by: Papers (4)
Quantization is a popular technique used in Deep Neural Networks (DNN) inference to reduce the size of models and improve the overall numerical performance by exploiting native hardware. This paper attempts to conduct an elaborate performance characterization of the benefits of using quantization techniques-mainly FP16/INT8 variants with static and dynamic schemes-using the MLPerf Edge Inference b...Show More
MPI4Spark Meets YARN: Enhancing MPI4Spark through YARN support for HPC
Kinan Al-Attar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
2023 IEEE International Conference on Big Data (BigData)
Year: 2023 | Conference Paper |
The MPI4Spark effort was able to reconcile disparities that existed between High-Performance Computing (HPC) environments and Big Data stacks, by adopting an MPI-based solution inside of A pache Spark’s Netty communication layer that was capable of better utilizing high-speed interconnects — such as InfiniBand (IB), Intel Omni-Path (OPA), and HPE Slingshot — across a variety of HPC systems. Apache...Show More
HARVEST: High-Performance Artificial Vision Framework for Expert Labeling using Semi-Supervised Training
Nawras Alnaasan;Matthew Lieber;Aamir Shafi;Hari Subramoni;Scott Shearer;Dhabaleswar K Panda
2023 IEEE International Conference on Big Data (BigData)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
Supervised Deep Learning (DL) thrives on Big Data; however, it inherits a major limitation—training and testing datasets must be fully annotated to train Deep Neural Networks (DNNs). To mitigate this bottleneck, we propose HARVEST—a distributed computer-vision framework that employs state-of-the-art semi-supervised learning (SSL) algorithms to train accurate DNNs using Distributed Data Parallelism...Show More
Battle of the BlueFields: An In-Depth Comparison of the BlueField-2 and BlueField-3 SmartNICs
Benjamin Michalowicz;Kaushik Kandadi Suresh;Hari Subramoni;Dhabaleswar K. DK Panda;Steve Poole
2023 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2023 | Conference Paper |
Cited by: Papers (4)
Over the past several years, Smart Network Interface Cards (NIC/SmartNICs) have rapidly evolved in popularity. In particular, NVIDIA’s BlueField line of SmartNICs has been effective in a wide variety of uses: Offloading communication in High-Performance Computing applications (HPC), various stages of the Deep Learning (DL) pipeline, and is designed especially for Datacenter/virtualization uses. Th...Show More
Designing In-network Computing Aware Reduction Collectives in MPI
Bharath Ramesh;Goutham Kalikrishna Reddy Kuncham;Kaushik Kandadi Suresh;Rahul Vaidya;Nawras Alnaasan;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
The Message-Passing Interface (MPI) provides convenient abstractions such as MPI_Allreduce for inter-process collective reduction operations. With the advent of deep learning and large-scale HPC systems, it is ever so important to optimize the latency of the MPI_Allreduce operation for large messages. Due to the amount of compute and communication involved in MPI_Allreduce, it is beneficial to off...Show More
In-Depth Evaluation of a Lower-Level Direct-Verbs API on InfiniBand-based Clusters: Early Experiences
Benjamin Michalowicz;Kaushik Kandadi Suresh;Bharath Ramesh;Aamir Shafi;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar Panda
2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
Many High-Performance Computing (HPC) clusters around the world use some variation of InfiniBand interconnects, all of which are powered by the ‘‘Verbs’’ API. Verbs supply a quick, efficient, and developer-friendly method of passing data buffers between nodes through their interconnect(s). In more recent years, the MLX5-DV (Direct Verbs) API has made itself known as a method of providing mechanism...Show More
ScaMP: Scalable Meta-Parallelism for Deep Learning Search
Quentin Anthony;Lang Xu;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Dk Panda
2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)
Year: 2023 | Conference Paper |
In this paper, we propose Scalable Meta-Parallelism for Deep Learning Search (ScaMP): a distributed Hyperparameter Optimization (HPO) and Neural Architecture Search (NAS) framework that supports out-of-core models with flexible parallelism schemes. SCaMP is integrated into the modern DL ecosystem, and enables both efficient parallel training of concurrent candidate architectures and aggregate devi...Show More
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Quentin Anthony;Ammar Ahmad Awan;Jeff Rasley;Yuxiong He;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar Panda
2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2023 | Conference Paper |
Cited by: Papers (5)
In recent years, the training requirements of many state-of-the-art Deep Learning (DL) models have scaled beyond the compute and memory capabilities of a single processor, and necessitated distribution among processors. Training such massive models necessitates advanced parallelism strategies [1], [2] to maintain efficiency. However, such distributed DL parallelism strategies require a varied mixt...Show More
IEEE Account
Purchase Details
Profile Information
Need Help?
- US & Canada: +1 800 678 4333
- Worldwide: +1 732 981 0060
- Contact & Support
- About IEEE Xplore
- Contact Us
- Help
- Accessibility
- Terms of Use
- Nondiscrimination Policy
- Sitemap
- Privacy & Opting Out of Cookies
A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.