https://jjl357.github.io/blog/tags/svd/ Recent content in SVD on JJ's Blog Hugo -- 0.152.2 zh-cn Thu, 30 Oct 2025 00:00:00 +0000 https://jjl357.github.io/blog/posts/qsvd----efficient-low-rank-approximation-for-unified-query-key-value-weight-compression-in-low-precision-vision-language-models---neurips25-spotlight/ Thu, 30 Oct 2025 00:00:00 +0000 https://jjl357.github.io/blog/posts/qsvd----efficient-low-rank-approximation-for-unified-query-key-value-weight-compression-in-low-precision-vision-language-models---neurips25-spotlight/ <p><img loading="lazy" src="https://jjl357.github.io/blog/image/QSVD/title.png"></p> <p><strong>Conference:</strong> <strong>NeurIPS'25 Spotlight</strong></p> <p><strong>Github:</strong> <a href="https://github.com/SAI-Lab-NYU/QSVD">https://github.com/SAI-Lab-NYU/QSVD</a></p> <hr> <h2 id="1-motivation">1. Motivation</h2> <p>Vision–Language Models (VLMs) 如 LLaVA、BLIP2 等在图像描述、视觉问答 (VQA) 等任务中表现卓越，但这些模型需要极大的计算与存储开销，尤其在推理时：</p> <ul> <li><strong>KV Cache 占用高</strong>：注意力机制中需存储 Key、Value，每层缓存大小随序列长度线性增长。</li> <li><strong>Q/K/V 投影重复计算</strong>：三组权重矩阵独立计算，造成算力浪费。</li> <li><strong>模型量化困难</strong>：激活分布存在极端 outliers，难以稳定进行低比特量化。</li> </ul> <p>QSVD 的目标是<strong>统一地对 Q/K/V 权重矩阵进行低秩近似</strong>并结合<strong>后训练量化 (PTQ)</strong>，实现以下三点：</p> <ol> <li>减少参数量、计算量、缓存占用；</li> <li>保持模型性能；</li> <li>支持低精度硬件部署。</li> </ol> <p><img loading="lazy" src="https://jjl357.github.io/blog/image/QSVD/figure1.png"></p> <hr> <h2 id="2-related-work">2. Related Work</h2> <h3 id="21-svd-in-large-models">2.1 SVD in Large Models</h3> <p><strong>Singular Value Decomposition (SVD)</strong> 是经典的矩阵分解方法。 对于矩阵 ( W \in \mathbb{R}^{m \times n} )，可分解为：</p> <p>$$ W = U \Sigma V^T $$</p> <p>其中：</p> <ul> <li>(U, V) 为正交矩阵；</li> <li>(\Sigma) 为奇异值对角矩阵；</li> <li>保留前 (r) 个奇异值可得到 rank-(r) 近似：</li> </ul> <p>$$ W \approx U_r \Sigma_r V_r^T $$</p>