Author details
Co-Authors:
Author's Published Works
Scheduled Maintenance on Monday 1/13/2025
Single article sales and account management will be unavailable from 5:00 AM - 7:00 PM ET (09:00 - 23:00 UTC). We apologize for the inconvenience.
Author details
Showing 1-23 of 23 results
- Sort
Sort By
Results
Understanding and Characterizing Communication Characteristics for Distributed Transformer Models
Quentin Anthony;Benjamin Michalowicz;Jacob Hatef;Lang Xu;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
IEEE Micro
Year: 2025 | Early Access Article |
The transformer architecture has revolutionized many applications such as large language models. This progress has been largely enabled by distributed training, yet communication remains a significant bottleneck. This paper examines the communication behavior of transformer models, focusing on how different parallelism schemes in multi-node/multi-GPU training communicate data. We use GPT-based lan...Show More
Using BlueField-3 SmartNICs to Offload Vector Operations in Krylov Subspace Methods
Kaushik Kandadi Suresh;Benjamin Michalowicz;Nick Contini;Bharath Ramesh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar Panda
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
Modern SmartNICs are capable of performing both computation and communication operations. In this context, past works on accelerating HPC/DL applications have manually selected some computational phases for offloading them to the SmartNICs. In this work, we identify Vector Multiply-Adds (VMA), Distributed Dot Products (DDOT), and Sparse Matrix-Vector Multiplication (Matvec) as three fundamental op...Show More
Effective and Efficient Offloading Designs for One-Sided Communication to SmartNICs
Ben Michalowicz;Kaushik Kandadi Suresh;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar K. Panda;Steve Poole
2024 IEEE 31st International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2024 | Conference Paper |
One-sided communication is one of many approaches to use for data transfer in High-Performance Computing (HPC) applications. One-sided operations require less demand on parallel programming libraries and do not require HPC hardware to issue acknowledgments of successful data transfer. Thanks to its inherently non-blocking nature, one-sided communication is also useful for improving overlap between...Show More
OHIO: Improving RDMA Network Scalability in MPI_Alltoall Through Optimized Hierarchical and Intra/Inter-Node Communication Overlap Design
Tu Tran;Goutham Kalikrishna Reddy Kuncham;Bharath Ramesh;Shulei Xu;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar K. DK Panda
2024 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2024 | Conference Paper |
The presence of exascale computers has pushed a new boundary in computing capability, which poses performance challenges in parallel programming models on how to exploit such systems efficiently. A dominant programming model for running parallel programs is the Message Passing Interface. Among primitives provided by MPI, Alltoall is a communication-intensive operation, which is utilized by many ap...Show More
Towards Accelerating k-NN with MPI and Near-Memory Processing
Hooyoung Ahn;Seonyoung Kim;Yoomi Park;Woojong Han;Nick Contini;Bharath Ramesh;Mustafa Abduljabbar;Dhabaleswar K. Panda
2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2024 | Conference Paper |
Message Passing Interface (MPI) is a common parallel programming model in High-Performance Computing fields. Recently, it has been widely used in Artificial Intelligence (AI) applications. However, the performance of those applications is limited by the memory wall problem, which is the performance gap between the processor and memory. To address these problems, we propose a novel computing archit...Show More
PML-MPI: A Pre-Trained ML Framework for Efficient Collective Algorithm Selection in MPI
Mingzhe Han;Goutham Kalikrishna Reddy Kuncham;Ben Michalowicz;Rahul Vaidya;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2024 | Conference Paper |
The Message Passing Interface is the de facto standard in high-performance computing (HPC) for inter-process communication. MPI libraries employ numerous algorithms for each collective communication pattern whose behavior is largely affected by the underlying hardware, communication pattern, message size, and number of processes involved. Choosing the “best” algorithm for every possible scenario i...Show More
HINT: Designing Cache-Efficient MPI_Alltoall using Hybrid Memory Copy Ordering and Non-Temporal Instructions
Bharath Ramesh;Nick Contini;Nawras Alnaasan;Kaushik Kandadi Suresh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. D K Panda
2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2024 | Conference Paper |
Modern multi/many-core processors in HPC systems have hundreds of cores with deep memory hierarchies. HPC applications run at high core counts often experience contention between processes/threads on shared resources such as caches, leading to degraded performance. This is especially true for dense collective patterns, such as MPI_Alltoall, that have many concurrent memory transactions. The orderi...Show More
Accelerating MPI AllReduce Communication with Efficient GPU-Based Compression Schemes on Modern GPU Clusters
Qinghua Zhou;Bharath Ramesh;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
ISC High Performance 2024 Research Paper Proceedings (39th International Conference)
Year: 2024 | Conference Paper |
With the increasing scale of High-Performance Computing (HPC) and Deep Learning (DL) applications through GPU adaptation, the seamless communication of data stored on GPUs has become a critical factor in enhancing overall application performance. AllReduce is a communication collective operation that is commonly used in HPC applications and distributed DL training, especially Data Parallelism. Dat...Show More
Optimized All-to-All Connection Establishment for High-Performance MPI Libraries Over InfiniBand
Shulei Xu;Goutham Kalikrishna Reddy Kuncham;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2023 | Conference Paper |
In modern multi-/many-core HPC systems, the increasing number of processor cores presents new challenges in managing parallel compute workloads across multiple nodes. One crucial aspect that significantly impacts the startup phase of parallel MPI jobs is the methodology used for connection establishment. In this paper, we investigate the limitations of existing all-to-all connection establishment ...Show More
Performance Characterization of Using Quantization for DNN Inference on Edge Devices
Hyunho Ahn;Tian Chen;Nawras Alnaasan;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
2023 IEEE 7th International Conference on Fog and Edge Computing (ICFEC)
Year: 2023 | Conference Paper |
Cited by: Papers (3)
Quantization is a popular technique used in Deep Neural Networks (DNN) inference to reduce the size of models and improve the overall numerical performance by exploiting native hardware. This paper attempts to conduct an elaborate performance characterization of the benefits of using quantization techniques-mainly FP16/INT8 variants with static and dynamic schemes-using the MLPerf Edge Inference b...Show More
Designing In-network Computing Aware Reduction Collectives in MPI
Bharath Ramesh;Goutham Kalikrishna Reddy Kuncham;Kaushik Kandadi Suresh;Rahul Vaidya;Nawras Alnaasan;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
The Message-Passing Interface (MPI) provides convenient abstractions such as MPI_Allreduce for inter-process collective reduction operations. With the advent of deep learning and large-scale HPC systems, it is ever so important to optimize the latency of the MPI_Allreduce operation for large messages. Due to the amount of compute and communication involved in MPI_Allreduce, it is beneficial to off...Show More
In-Depth Evaluation of a Lower-Level Direct-Verbs API on InfiniBand-based Clusters: Early Experiences
Benjamin Michalowicz;Kaushik Kandadi Suresh;Bharath Ramesh;Aamir Shafi;Hari Subramoni;Mustafa Abduljabbar;Dhabaleswar Panda
2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
Many High-Performance Computing (HPC) clusters around the world use some variation of InfiniBand interconnects, all of which are powered by the ‘‘Verbs’’ API. Verbs supply a quick, efficient, and developer-friendly method of passing data buffers between nodes through their interconnect(s). In more recent years, the MLX5-DV (Direct Verbs) API has made itself known as a method of providing mechanism...Show More
MCR-DL: Mix-and-Match Communication Runtime for Deep Learning
Quentin Anthony;Ammar Ahmad Awan;Jeff Rasley;Yuxiong He;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar Panda
2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2023 | Conference Paper |
Cited by: Papers (5)
In recent years, the training requirements of many state-of-the-art Deep Learning (DL) models have scaled beyond the compute and memory capabilities of a single processor, and necessitated distribution among processors. Training such massive models necessitates advanced parallelism strategies [1], [2] to maintain efficiency. However, such distributed DL parallelism strategies require a varied mixt...Show More
Accelerating Distributed Deep Learning Training with Compression Assisted Allgather and Reduce-Scatter Communication
Qinghua Zhou;Quentin Anthony;Lang Xu;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. DK Panda
2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Year: 2023 | Conference Paper |
Cited by: Papers (4)
Fully Sharded Data Parallel (FSDP) technology achieves higher performance by scaling out data-parallel training of Deep Learning (DL) models. It shards the model parameters, gradients, and optimizer states of the model among multiple GPUs. Consequently, this requires data-intensive Allgather and Reduce-Scatter communication to share the model parameters, which becomes a bottleneck. Existing scheme...Show More
Implementing and Optimizing a GPU-aware MPI Library for Intel GPUs: Early Experiences
Chen-Chun Chen;Kawthar Shafie Khorassani;Goutham Kalikrishna Reddy Kuncham;Rahul Vaidya;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
Year: 2023 | Conference Paper |
Cited by: Papers (1)
As the demand for computing power from High-Performance Computing (HPC) and Deep Learning (DL) applications increase, there is a growing trend of equipping modern exascale clusters with accelerators, such as NVIDIA and AMD GPUs. GPU-aware MPI libraries allow the applications to communicate between GPUs in a parallel environment with high productivity and performance. Although NVIDIA and AMD GPUs h...Show More
Network-Assisted Noncontiguous Transfers for GPU-Aware MPI Libraries
Kaushik Kandadi Suresh;Kawthar Shafie Khorassani;Chen Chun Chen;Bharath Ramesh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
The importance of graphics processing units (GPUs) in accelerating HPC applications is evident by the fact that a large number of supercomputing clusters are GPU enabled. Many of these HPC applications use message passing interface (MPI) as their programming model. These MPI applications frequently exchange data that is noncontiguous in GPU memory. MPI provides derived datatypes (DDTs) to represen...Show More
Designing Efficient Pipelined Communication Schemes using Compression in MPI Libraries
Bharath Ramesh;Qinghua Zhou;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
2022 IEEE 29th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2022 | Conference Paper |
Cited by: Papers (1)
The emergence of trillion-parameter models in AI, and the deployment of dense Graphics Processing Unit (GPU) systems with high-bandwidth inter-GPU and network interconnects underscores the need to design efficient architecture-aware large message communication operations. GPU-based on-the-fly compression communication designs help reduce the amount of data transferred across processes, thereby imp...Show More
Efficient Personalized and Non-Personalized Alltoall Communication for Modern Multi-HCA GPU-Based Clusters
Kaushik Kandadi Suresh;Akshay Paniraja Guptha;Benjamin Michalowicz;Bharath Ramesh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar Panda
2022 IEEE 29th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Year: 2022 | Conference Paper |
Cited by: Papers (1)
Graphics Processing Units (GPUs) have become ubiquitous in today’s supercomputing clusters primarily because of their high compute capability and power efficiency. Message Passing Interface (MPI) is a widely adopted programming model for large-scale GPU-based applications used in such clusters. Modern GPU-based systems have multiple HCAs. Previously, scientists have leveraged multi-HCA systems to ...Show More
STEER: Asymmetry-aware Energy Efficient Task Scheduler for Cluster-based Multicore Architectures
Jing Chen;Madhavan Manivannan;Bhavishya Goel;Mustafa Abduljabbar;Miquel Pericàs
2022 IEEE 34th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)
Year: 2022 | Conference Paper |
Cited by: Papers (1)
Reducing the energy consumption of parallel applications is becoming increasingly important. Current chip multiprocessors (CMPs) incorporate asymmetric cores (i.e. static asymmetry) and DVFS (i.e. dynamic asymmetry) to enable energy efficient execution. To reduce cost and complexity, designs typically organize asymmetric cores into core-clusters supporting the same DVFS setting across cores in a c...Show More
Network Assisted Non-Contiguous Transfers for GPU-Aware MPI Libraries
Kaushik Kandadi Suresh;Kawthar Shafie Khorassani;Chen Chun Chen;Bharath Ramesh;Mustafa Abduljabbar;Aamir Shafi;Hari Subramoni;Dhabaleswar K. Panda
2022 IEEE Symposium on High-Performance Interconnects (HOTI)
Year: 2022 | Conference Paper |
Cited by: Papers (4)
The importance of GPUs in accelerating HPC applications is evident by the fact that a large number of super-computing clusters are GPU-enabled. Many of these HPC applications use MPI as their programming model. These MPI applications oftentimes exchange data that is non-contiguous in GPU memory. MPI provides Derived Datatypes(DDTs) to represent such data. In the past, researchers have proposed sol...Show More
Spark Meets MPI: Towards High-Performance Communication Framework for Spark using MPI
Kinan Al-Attar;Aamir Shafi;Mustafa Abduljabbar;Hari Subramoni;Dhabaleswar K. Panda
2022 IEEE International Conference on Cluster Computing (CLUSTER)
Year: 2022 | Conference Paper |
Cited by: Papers (5)
There are several popular Big Data processing frameworks including Apache Spark, Dask, and Ray. The Apache Spark software provides an easy-to-use high-level API in different languages including Scala, Java, and Python. Spark supports parallel and distributed execution of user workloads by supporting communication using an event-driven framework called Netty. Some efforts - including RDMA-Spark and...Show More
LEGaTO: Low-Energy, Secure, and Resilient Toolset for Heterogeneous Computing
B. Salami;K. Parasyris;A. Cristal;O. Unsal;X. Martorell;P. Carpenter;R. De La Cruz;L. Bautista;D. Jimenez;C. Alvarez;S. Nabavi;S. Madonar;M. Pericàs;P. Trancoso;M. Abduljabbar;J. Chen;P. N. Soomro;M Manivannan;M. Berge;S. Krupop;F. Klawonn;Al Mekhlafi;S. May;T. Becker;G. Gaydadjiev;H. Salomonsson;D. Dubhashi;O. Port;Y. Etsion;Le Quoc Do;Christof Fetzer;M. Kaiser;N. Kucza;J. Hagemeyer;R. Griessl;L. Tigges;K. Mika;A. Hüffmeier;M. Pasin;V. Schiavoni;I. Rocha;C. Göttel;P. Felber
2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)
Year: 2020 | Conference Paper |
Cited by: Papers (3)
The LEGaTO project leverages task-based programming models to provide a software ecosystem for Made in-Europe heterogeneous hardware composed of CPUs, GPUs, FPGAs and dataflow engines. The aim is to attain one order of magnitude energy savings from the edge to the converged cloud/HPC, balanced with the security and resilience challenges. LEGaTO is an ongoing three-year EU H2020 project started in ...Show More
LEGaTO: Low-Energy, Secure, and Resilient Toolset for Heterogeneous Computing
B. Salami;K. Parasyris;A. Cristal;O. Unsal;X. Martorell;P. Carpenter;R. De La Cruz;L. Bautista;D. Jimenez;C. Alvarez;S. Nabavi;S. Madonar;M. Pericàs;P. Trancoso;M. Abduljabbar;J. Chen;P. N. Soomro;M Manivannan;M. Berge;S. Krupop;F. Klawonn;Al Mekhlafi;S. May;T. Becker;G. Gaydadjiev;H. Salomonsson;D. Dubhashi;O. Port;Y. Etsion;Le Quoc Do;Christof Fetzer;M. Kaiser;N. Kucza;J. Hagemeyer;R. Griessl;L. Tigges;K. Mika;A. Hüffmeier;M. Pasin;V. Schiavoni;I. Rocha;C. Göttel;P. Felber
Year: 2020 | Conference Paper |
Abstraction Layer For Standardizing APIs of Task-Based Engines
We introduce AL4SAN, a lightweight library for abstracting the APIs of task-based runtime engines. AL4SAN unifies the expression of tasks and their data dependencies. It supports various dynamic runtime systems relying on compiler technology and user-defined APIs. It enables a single application to employ different runtimes and their respective scheduling components, while providing user-oblivious...Show More
IEEE Account
Purchase Details
Profile Information
Need Help?
- US & Canada: +1 800 678 4333
- Worldwide: +1 732 981 0060
- Contact & Support
- About IEEE Xplore
- Contact Us
- Help
- Accessibility
- Terms of Use
- Nondiscrimination Policy
- Sitemap
- Privacy & Opting Out of Cookies
A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.