Root Mean Square Layer Normalization

249M-param MoE transformer built from scratch in PyTorch. GQA, RoPE, SwiGLU, sparse MoE with 3 aux losses, AMP training loop no Trainer abstractions. Architecture mirrors LLaMA/Mistral/Mixtral decisions, fully inspectable.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

Efficient kernel for RMS normalization with fused operations, includes both forward and backward passes, compatibility with PyTorch.

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Optimized CUDA kernels for NVIDIA GB10 Blackwell (sm_121, DGX Spark). RMSNorm + GELU. First sm_121 kernel on HuggingFace Kernel Hub.

Fused Triton kernels for Transformer inference: RMSNorm+RoPE, Gated MLP, FP8 GEMM — CPU-testable references, autotuning, and benchmarking

Optimized Fused RMSNorm implementation with CUDA. Features vectorized memory access (float4), warp-level reductions, and efficient backward pass for LLM training

A non-official implementation of Qwen 3.5, as there doesn’t seem to be a paper or any code available that I can find, so I decided to implement it just for fun.

LLM pretraining from scratch on FineWeb dataset (architecture and all components explained), plus optimal use of GPU on SLURM cluster

A 36M-parameter goldfish language model with a 10-second memory + pixel-art PWA desk pet. Runs in your browser, fully offline. Adopt it at den-sec.github.io/glublm/desk-pet/

Simple character level Transformer

A CPU-friendly character-level GPT from scratch in PyTorch featuring modern architecture (RoPE, RMSNorm, GQA) and a modular RAG pipeline with Chroma DB.

A transformer language model built from scratch, from byte-level BPE tokenization through pretraining and QA fine-tuning.

Generative models nano version for fun. No STOA here, nano first.

From-scratch PyTorch GPT: the GPT-2 skeleton plus the components of current open LLMs — RMSNorm, RoPE, SwiGLU, GQA, no-bias + tied embeddings. Each isolated and self-checking.

A from-scratch PyTorch LLM implementing Sparse Mixture-of-Experts (MoE) with Top-2 gating. Integrates modern Llama-3 components (RMSNorm, SwiGLU, RoPE, GQA) and a custom-coded Byte-Level BPE tokenizer. Pre-trained on a curated corpus of existential & dark philosophical literature.

Hands-on CUDA kernel engineering on LLM decode bottlenecks (RMSNorm, GEMV): PyTorch/Triton/CUDA C++ kernels profiled with Nsight and evaluated against a real vLLM backend.

Correctness-first Triton RMSNorm and SwiGLU kernels with cache-controlled GPU benchmarks and regression gates

Build an LLM in PyTorch: BPE tokenizer, GPT-1/2 + LLaMA, end-to-end train/infer