.. Copyright (C) 2025, Advanced Micro Devices, Inc. All rights reserved.

Welcome to AMD Quark Documentation!
===================================

**AMD Quark** is a comprehensive cross-platform deep learning toolkit designed to simplify and enhance the quantization
of deep learning models. Supporting both PyTorch and ONNX models, AMD Quark empowers developers to optimize their models for
deployment on a wide range of hardware backends, achieving significant performance gains without compromising accuracy.

.. figure:: _static/quark_stack.png
   :align: center
   :alt: Quark Stack

AMD Quark for PyTorch: Flexible and Efficient Quantization for PyTorch Models
-----------------------------------------------------------------------------

AMD Quark for PyTorch provides developers with a flexible, efficient, and easy-to-use toolkit for quantizing deep learning
models from PyTorch. The current quantization method is based on PyTorch in-place operator replacement.
In particular, the tool provides the key features and verified models as below:

Key Features
^^^^^^^^^^^^

* **Comprehensive Quantization Support**:
   - **Eager Mode Post-Training Quantization (PTQ):** Quantize pre-trained models without the need for retraining data.
   - **FX Graph Mode PTQ and Quantization-Aware Training (QAT):** Optimize models during training for superior accuracy on quantized hardware.
   - **Optimized QAT Methods:** Support Trained Quantization Thresholds For Accurate And Efficient Fixed-Point Inference Of Deep Neural Networks (TQT), Learned Step Size Quantization (LSQ) for better QAT result.
   - **Flexible Quantization Strategies:** Choose from symmetric/asymmetric, weight-only/static/dynamic quantization, and various quantization levels (per tensor/channel) to fine-tune performance and accuracy trade-offs.
   - **Extensive Data Type Support:** Quantize models using a wide range of data types, including `float16`, `bfloat16`, `int4`, `uint4`, `int8`, `fp8 (e4m3fn and e5m2)`, Shared Micro exponents with Multi-Level Scaling (`MX6`, `MX9`), and `Microscaling (MX)` data types with `int8`, `fp8_e4m3fn`, `fp8_e5m2`, `fp4`, `fp6_e3m2`, and `fp6_e2m3` elements.
   - **Configurable Calibration Methods:** Optimize quantization accuracy with `MinMax`, `Percentile`, and `MSE` calibration methods.
* **Advanced Capabilities:**
   - **Large Language Model Optimization:** Specialized support for quantizing large language models with `kv-cache` quantization.
   - **Cutting-Edge Algorithms:** Leverage state-of-the-art algorithms like `SmoothQuant`, `AWQ`, `GPTQ`, `Qronos` for `uint4` quantization on GPUs, achieving optimal performance for demanding tasks.
* **Seamless Integration and Deployment:**
   - **Export to multiple formats:** Export quantized models to `ONNX`, `JSON-safetensors`, and `GGUF` formats for deployment on a wide range of platforms.
   - **APL Integration:** Seamlessly integrate with AMD Pytorch-light (APL) for optimized performance on AMD hardware, to provide `INT-K`, `BFP16`, and `BRECQ` support.
   - **Experimental Brevitas Integration:** Explore seamless integration with Brevitas for quantizing Stable Diffusion and ImageNet classification models.
* **Examples included:** Benefit from practical examples for LLM models, SDXL models (Eager Mode), and CNN models (FX Graph Mode), accelerating your quantization journey.
* **Cross-Platform Support:** Develop and deploy on both Linux (CPU and GPU) and Windows (CPU mode) operating systems.

AMD Quark for ONNX: Streamlined Quantization for ONNX models
------------------------------------------------------------

AMD Quark for ONNX leverages the power of the ONNX Runtime Quantization tool,
providing a robust and flexible solution for quantizing ONNX models.

Key Features
^^^^^^^^^^^^

* **Comprehensive Quantization Support**:
   - **Post-Training Quantization (PTQ):** Quantize pre-trained models without the need for retraining data.
   - **Flexible Quantization Strategies:** Choose from symmetric/asymmetric, weight-only/static/dynamic quantization, and various quantization levels (per tensor/channel) to fine-tune performance and accuracy trade-offs.
   - **Extensive Data Type Support:** Quantize models using a wide range of data types, including `uint32`, `int32`, `float16`, `bfloat16`, `int16`, `uint16`, `int8`, `uint8`, `int4` `uint4`, `int3`, Block Floating Point (typical `BFP16`), and `Microscaling (MX)` data types with `int8`, `fp8_e4m3fn`, `fp8_e5m2`, `fp4`, `fp6_e3m2`, and `fp6_e2m3` elements.
   - **Configurable Calibration Methods:** Optimize quantization accuracy with `MinMax`, `Entropy`, `Distribution`, `Percentile`, `MinMSE` and `LayerwisePercentile` calibration methods.
* **Advanced Capabilities:**
   - **Multiple Deployment Targets:** Target a variety of hardware platforms, including `NPU_CNN`, `NPU_Transformer`, and `CPU`.
   - **Cutting-Edge Algorithms:** Leverage state-of-the-art algorithms like `QuaRot`, `SmoothQuant`, `CLE`, `BiasCorrection`, `AdaQuant`, `AdaRound`, and `GPTQ` achieving optimal performance for demanding tasks.
   - **Flexible Scale Types:** Support quantization with `float scale`, `int16 scale`, and `power-of-two scale` options.
   - **Automatic Mixed Precision:**  Achieve an optimal balance between accuracy and performance through automatic mixed precision.

.. toctree::
   :hidden:
   :caption: Release Notes
   :maxdepth: 1

   Release Information <release_note>

.. toctree::
   :hidden:
   :caption: Getting Started with AMD Quark
   :maxdepth: 1

   Introduction to Quantization <intro>
   Installation <install>
   Getting started: Introduction <basic_usage>
   Getting started: Quark for ONNX <onnx/basic_usage_onnx>
   Getting started: Quark for PyTorch <pytorch/basic_usage_pytorch>
   PyTorch Examples <pytorch/pytorch_examples>
   ONNX Examples <onnx/onnx_examples>

.. supported-accelerators:
.. toctree::
   :hidden:
   :caption: Supported accelerators
   :maxdepth: 1

   AMD Ryzen AI <supported_accelerators/ryzenai/index>
   AMD Instinct <supported_accelerators/mi_gpus/index>

.. _advanced-quark-features-pytorch:
.. toctree::
   :hidden:
   :caption: Advanced AMD Quark Features for PyTorch
   :maxdepth: 1

   Configuring PyTorch Quantization for Large Language Models <pytorch/user_guide_config_for_llm>
   Configuring PyTorch Quantization from Scratch <pytorch/user_guide_config_description>
   Save and Load Quantized Models <pytorch/quark_save_load>
   Exporting Quantized Models <pytorch/export/quark_export>
   Best Practices for Post-Training Quantization (PTQ) <pytorch/quark_torch_best_practices>
   Debugging quantization Degradation <pytorch/debug>
   File-to-File LLM Quantization <pytorch/file2file_quantization>
   Language Model Optimization <pytorch/llm_quark>
   Activation/Weight Smoothing (SmoothQuant) <pytorch/smoothquant>
   Auto SmoothQuant <tutorials/torch/auto_smoothquant_document_and_example>
   Activation-aware Weight Quantization (AWQ) <pytorch/awq_document>
   Block Floating Point 16 <pytorch/tutorial_bfp16>
   Extensions <pytorch/extensions>
   Using MX (Microscaling) <pytorch/adv_mx>
   Two Level Quantization Formats <pytorch/adv_two_level>

.. _advanced-quark-features-onnx:
.. toctree::
   :hidden:
   :caption: Advanced Quark Features for ONNX
   :maxdepth: 1


   Configuring ONNX Quantization <onnx/user_guide_config_description>
   Data and OP Types <onnx/user_guide_supported_optype_datatype>
   Accelerate with GPUs <onnx/gpu_usage_guide>
   Mixed Precision <onnx/tutorial_mix_precision>
   Block Floating Point 16 (BFP16) <onnx/tutorial_bfp16_quantization>
   BF16 Quantization <onnx/tutorial_bf16_quantization>
   Microscaling (MX) <onnx/tutorial_microscaling_quantization>
   Microexponents (MX) <onnx/tutorial_microexponents_quantization>
   Accuracy Improvement Algorithms <onnx/accuracy_improvement_algorithms>
   Automatic Search for Model Quantization <onnx/user_guide_auto_search>
   Automatic Search Pro for Model Quantization <onnx/user_guide_auto_search_pro>
   Latency and Memory profiling <onnx/tutorial_profiling>
   Using ONNX Model Inference and Saving Input Data in NPY Format <onnx/config/user_guide_onnx_model_inference_save_input_npy>
   Optional Utilities <onnx/optional_utilities>
   Tools <onnx/tools>

.. toctree::
   :hidden:
   :caption: Tutorials
   :maxdepth: 1

   Quark for PyTorch <tutorials_pytorch>
   Quark for ONNX <tutorials_onnx>

.. toctree::
   :hidden:
   :caption: Third-party contributions
   :maxdepth: 1

   Introduction and guidelines <intro_contrib>

.. toctree::
   :hidden:
   :caption: Experimental Features
   :maxdepth: 1

   Quark CLI <quark_cli>

.. toctree::
   :hidden:
   :caption: APIs
   :maxdepth: 1

   PyTorch APIs <autoapi/pytorch_apis>
   ONNX APIs <autoapi/onnx_apis>

.. toctree::
   :hidden:
   :caption: Troubleshooting and Support
   :maxdepth: 1

   PyTorch Troubleshooting <pytorch/pytorch_troubleshooting>
   ONNX Troubleshooting <onnx/onnx_troubleshooting>

.. toctree::
   :hidden:
   :maxdepth: 1

   versions
   license
