SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

Reimplement RetinaFace use C++ and TensorRT

tensorrt int8 量化yolov5 onnx模型

An innovative library for efficient LLM inference via low-bit quantization

TensorRT int8 量化部署 yolov5s 模型，实测3.3ms一帧！

RepVGG TensorRT int8 量化，实测推理不到1ms一帧！

nanodet int8 量化，实测推理2ms一帧！

NCNN+Int8+YOLOv4 quantitative modeling and real-time inference

👀 Apply YOLOv8 exported with ONNX or TensorRT(FP16, INT8) to the Real-time camera

a simple pipline of int8 quantization based on tensorrt.

quantization example for pqt & qat

LLM-Lora-PEFT_accumulate explores optimizations for Large Language Models (LLMs) using PEFT, LORA, and QLORA. Contribute experiments and implementations to enhance LLM efficiency. Join discussions and push the boundaries of LLM optimization. Let's make LLMs more efficient together!

TensorRT Int8 Python version sample. TensorRT Int8 Python 实现例子。TensorRT Int8 Pythonの例です

RISCV Vector Kernel C/LLVM-IR generator

A LLaMA2-7b chatbot with memory running on CPU, and optimized using smooth quantization, 4-bit quantization or Intel® Extension For PyTorch with bfloat16.

mxnet GluonCV quantization binary ternary models

Size (in bytes) of an 8-bit signed integer.

Convert a Node-API value representing a two-dimensional strided array to a signed 8-bit integer array.