Decoding LLM Fine-Tuning Math via Architecture Flaws

This review of a Shenshen RAG system reveals "data discontinuities" in full-link architecture. Using dual-track retrieval, RRF, CoT distillation, and QLoRA, it exposes the mathematical logic and engineering challenges of fine-tuning, offering insights for high-precision vertical AI systems. LLM fine

Back to news

Tech Insights AI 编辑Published Nov 17, 2026 87 views

Overview

In a recent review of building an "Intelligent RAG System for Shenshen Summarization Questions," the technical team conducted a deep examination from macro-architecture to micro-mathematical essence. This process began not with coding, but with a meticulous review of the full-link data flow architecture. While mapping the system's data flow, designers identified two seemingly minor yet critical "data discontinuities." These were not simple technical failures but revealed structural contradictions in current large language models (LLMs) when migrating from general capabilities to specific domain knowledge for complex logical tasks in vertical fields.

配图

The core subject was an intelligent system designed to solve Shenshen summarization questions. This was not a simple chatbot but a complex engineering system integrating Retrieval-Augmented Generation (RAG), hybrid re-ranking, knowledge distillation, and Parameter-Efficient Fine-Tuning (PEFT). During the dual-track retrieval phase, the system employed parallel BM25 keyword matching and Dense vector semantic retrieval. In the re-ranking phase, the Reciprocal Rank Fusion (RRF) algorithm was introduced to merge results from different retrievers. In the training phase, DeepSeek Chain of Thought (CoT) data was used for knowledge distillation, followed by QLoRA fine-tuning via the Unsloth framework. However, stitching these mature modules exposed logical breaks in data flow, forcing the team to reconsider the mathematical essence of LLM fine-tuning: how to precisely map general language understanding to specific Shenshen answering logic within limited parameter space through mathematical optimization.

插图1

插图2

Background

To understand the depth of this review, one must trace the common dilemmas of LLMs in vertical applications. Historically, the industry relied on massive general pre-training data, hoping models would "emerge" with specific task capabilities. However, in fields like Shenshen, which demand high logical rigor, formatting standardization, and political literacy, general models' "emergence" often manifests as frequent hallucinations or logical leaps. This explains why simple Prompt Engineering or Few-shot Learning fails to meet production-grade standards.

In this context, RAG technology became key to addressing knowledge timeliness and factual accuracy. However, building a Shenshen RAG system revealed the shortcomings of single retrieval methods. While BM25 precisely matches keywords, it struggles with semantic associations; Dense retrieval captures semantics but is prone to noise. Thus, dual-track retrieval became inevitable. Yet, retrieval is only the first step; filtering content that best fits Shenshen answering logic from massive fragments became a new pain point. The introduction of the RRF algorithm addressed this sorting problem by mathematically fusing ranking results from different retrievers to achieve more robust sorting.

A deeper challenge lay in transforming the model's "thinking mode." Shenshen answers require not just correctness but reasoning that aligns with specific administrative logic. Traditional fine-tuning often trains directly on input-output pairs, ignoring intermediate reasoning processes. Therefore, DeepSeek CoT data was introduced for knowledge distillation to teach the model "how to think," not just "what to output." Finally, Unsloth QLoRA was adopted for efficient training on consumer or low-cost hardware. QLoRA reduces VRAM usage and computational costs through 4-bit quantization and low-rank adaptation matrices. However, during this integration, "data discontinuities" emerged in the architecture: the lack of unified mathematical constraints between frontend retrieval quality, mid-stage sorting logic, and backend fine-tuning data distribution caused capability decay during end-to-end inference. This trajectory clearly demonstrates the cognitive upgrade from "patching technologies" to "understanding mathematical essence."

插图3

Impact

This deep review of the Shenshen intelligent RAG system extends beyond a specific product's development, offering broad industry implications for LLM application development. First, it reveals that "data discontinuities" are a core bottleneck restricting vertical LLM deployment. Many projects fail not due to inferior technology but because semantic alignment and mathematical consistency across data flow stages are ignored. This shifts developer focus from "module functionality" to "data flow integrity," emphasizing the need for unified data standards and evaluation metrics during early architectural design.

Second, regarding technology selection, the combination of dual-track retrieval and RRF re-ranking proves the necessity of hybrid retrieval in complex tasks. As LLM parameter scales grow, relying solely on internal memory is unrealistic;协同 optimization of external retrieval and internal reasoning will become the mainstream paradigm. Simultaneously, the application of knowledge distillation on CoT data highlights the importance of "process supervision." Future model fine-tuning will focus less on final answer accuracy and more on the rationality and interpretability of reasoning paths, driving evaluation systems from "result-oriented" to "process-oriented."

Finally, technologies like Unsloth QLoRA indicate that efficient fine-tuning is lowering the barrier to LLM customization. However, low cost does not mean low complexity. Conversely, it demands developers possess deeper mathematical foundations and architectural design skills to compensate for model limitations through precise data engineering and algorithm optimization under resource constraints. For players and users, this means future vertical AI assistants will be more logically rigorous, responsive, and cost-controlled. They will evolve from simple information retrieval tools into intelligent partners with deep reasoning capabilities. This review case provides valuable experience on building high-reliability RAG systems from a mathematical perspective, signaling that LLM application development is entering a new stage emphasizing underlying logic and system integration.