Skip to content

README.md

Latest commit

聽

History

History
75 lines (61 loc) 路 4.1 KB

README.md

File metadata and controls

75 lines (61 loc) 路 4.1 KB
Raw

Coding Quantization Methods

if any one want to contribute to this repo,

This section provides an overview of various quantization methods used in LLMs, along with their implementation progress.

Mapping:

Method Release Date Link Progress
LLM.int8() Aug 2022 馃敆 Link 馃毀 Under Progress
GPTQ Oct 2022 馃敆 Link 馃毀 Under Progress
QLoRA May 2023 馃敆 Link 馃毀 Under Progress
AWQ Jun 2023 馃敆 Link 馃毀 Under Progress
Quip# Jul 2023 馃敆 Link 馃毀 Under Progress
GGUF Aug 2023 馃敆 Link 馃毀 Under Progress
HQQ Nov 2023 馃敆 Link 馃毀 Under Progress
AQLM Feb 2024 馃敆 Link 馃毀 Under Progress

Installation Steps

  1. Clone the repo:

    https://github.com/Ahmed-El-Zainy/coding_quantization_methods.git
cd coding_quantization_methods

  2. Install the dependencies: for starting direct to the running:

    . ./setup.sh --new-env

    Before running the following command there are somethings to note:

    • By adding --new-env, a new conda environment named trellis will be created. If you want to use an existing conda environment, please remove this flag.
    • By default the trellis environment will use pytorch 2.4.0 with CUDA 11.8. If you want to use a different version of CUDA (e.g., if you have CUDA Toolkit 12.2 installed and do not want to install another 11.8 version for submodule compilation), you can remove the --new-env flag and manually install the required dependencies. Refer to PyTorch for the installation command.
    • If you have multiple CUDA Toolkit versions installed, PATH should be set to the correct version before running the command. For example, if you have CUDA Toolkit 11.8 and 12.2 installed, you should run export PATH=/usr/local/cuda-11.8/bin:$PATH before running the command.
    • By default, the code uses the flash-attn backend for attention. For GPUs do not support flash-attn (e.g., NVIDIA V100), you can remove the --flash-attn flag to install xformers only and set the ATTN_BACKEND environment variable to xformers before running the code. See the Minimal Example for more details.
    • The installation may take a while due to the large number of dependencies. Please be patient. If you encounter any issues, you can try to install the dependencies one by one, specifying one flag at a time.
    • If you encounter any issues during the installation, feel free to open an issue or contact us.

    Create a new conda environment named trellis and install the dependencies:

    . ./setup.sh --new-env --basic --xformers --flash-attn --diffoctreerast --spconv --mipgaussian --kaolin --nvdiffrast

    The detailed usage of setup.sh can be found by running . ./setup.sh --help.

    Usage: setup.sh [OPTIONS]
Options:
    -h, --help              Display this help message
    --new-env               Create a new conda environment
    --basic                 Install basic dependencies
    --xformers              Install xformers
    --flash-attn            Install flash-attn
    --diffoctreerast        Install diffoctreerast
    --vox2seq               Install vox2seq
    --spconv                Install spconv
    --mipgaussian           Install mip-splatting
    --kaolin                Install kaolin
    --nvdiffrast            Install nvdiffrast
    --demo                  Install all dependencies for demo

References: