Distributed training (multi-node) of a Transformer model

Messaging and state layer for distributed serverless applications

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)

Blink+: Increase GPU group bandwidth by utilizing across tenant NVLink.

collectives library for upc++

Audit GPU cluster communication schedules from NCCL logs. Zero dependencies. CI-ready.

TileXR (eXtreme Rendezvous for Asynchronous Tile Communication) is a data-centric asynchronous communication runtime for Huawei Ascend NPUs.

Interactive web visualization for understanding collective communication algorithms (as used in NCCL, RCCL, MPI). Learn how AllReduce, Broadcast, Reduce, AllGather and more work step by step.

Research prototype investigating adaptive collective communication optimization for MPI workloads using runtime performance feedback.

From-scratch Ring-AllReduce on CPU (C++20) and GPU (CUDA, multi-device): real multi-GPU benchmarks, Nsight profiling, and a bandwidth-optimal ring implementation built without NCCL.

Simple quick test to benchmark your pytorch + nccl/ncclx setup

AllReduce/AllGather scaling in ASTRA-sim across torus vs switch topologies on the analytical + ns-3 backends — latency-bound vs bandwidth-bound over message size and node count. Reproducible Docker/Chakra harness + write-up.

This repository contains simple programs of MPI_Bcast, MPI_Reduce, MPI_Scatter and MPI_Gather. Download the repository and test your self.

Modelling of MPI collective operations latencies: Broadcast and Reduce operations. UniTS, SDIC, 2023-2024

A reduction algorithm for MPI using only peer to peer communication

Ring-allreduce collective built from scratch in C++20 over raw MPI point-to-point primitives, benchmarked against Open MPI 5.0.10's MPI_Allreduce across 8B-128MiB and N=2..16. An alpha-beta cost model shows the ring's small-message loss is the algorithm's 2(N-1) step count, not implementation quality: Open MPI's own forced ring is just as slow.

MPI laboratory project demonstrating collective communication primitives to perform distributed numerical computations on a vector. Implements broadcast, scatter, gather, reduce, and scan operations while managing vector segments across multiple processes (Introduction to Parallel Computing, UNIWA).

HPC course practice assignments for parallel-programming

Summary of call graphs and data structures of collective communication plugin in NVIDIA TensorRT-LLM

Develop high-performance parallel applications in C++ using the Partitioned Global Address Space model and asynchronous communication primitives.