zh_CN/Step-1-PDF预处理.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f67a6a6",
   "metadata": {},
   "source": "## Notebook 1: PDF 预处理"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "在这一系列中，我们将使用开源模型将PDF转换为播客。\n",
    "\n",
    "获取播客的第一步是找到一个脚本，目前我们的步骤如下：\n",
    "- 使用任意主题的PDF\n",
    "- 使用 [mlx-community/Qwen2.5-3B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-3B-Instruct-4bit) 模型，将其处理成文本文件。英文用的是1.5B，中文使用1.5B效果不太好。\n",
    "- 在下一个 Notebook 中将其改写为播客稿件。\n",
    "\n",
    "在这个 Notebook 中，我们将上传一个PDF，使用 `PyPDF2` 将其保存为 `.txt` 文件，后续我们会用轻量模型处理文本。"
   ],
   "id": "30b6d43441645f5b"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "务必记得安装此项目依赖，不然跑不起来～🙂‍↔️",
   "id": "582680da8eac6c30"
  },
  {
   "cell_type": "code",
   "id": "f4fc7aef-3505-482e-a998-790b8b9d48e4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.318414Z",
     "start_time": "2024-10-31T16:04:58.316740Z"
    }
   },
   "source": "#!pip install -r requirements.txt",
   "outputs": [],
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "id": "7b23d509",
   "metadata": {},
   "source": [
    "这里设置需要处理的PDF文件路径。\n",
    "\n",
    "另外，如果你想要发挥GPU极致性能，求你更换大参数的模型，虽然轻量级模型对于此任务也能胜任。"
   ]
  },
  {
   "cell_type": "code",
   "id": "60d0061b-8b8c-4353-850f-f19466a0ae2d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.339613Z",
     "start_time": "2024-10-31T16:04:58.337923Z"
    }
   },
   "source": [
    "pdf_path = '../resources/2402.13116v4.pdf'\n",
    "DEFAULT_MODEL = \"mlx-community/Qwen2.5-3B-Instruct-4bit\""
   ],
   "outputs": [],
   "execution_count": 17
  },
  {
   "cell_type": "code",
   "id": "21029232-ac5f-42ca-b26b-baad5b2f49b7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.351074Z",
     "start_time": "2024-10-31T16:04:58.349366Z"
    }
   },
   "source": [
    "import os\n",
    "import warnings\n",
    "from typing import Optional\n",
    "\n",
    "import PyPDF2\n",
    "from mlx_lm import load, generate\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ],
   "outputs": [],
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "id": "203c22eb",
   "metadata": {},
   "source": "这里会检查文件有没有什么毛病～"
  },
  {
   "cell_type": "code",
   "id": "153d9ece-37a4-4fff-a8e8-53f923a2b0a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.366385Z",
     "start_time": "2024-10-31T16:04:58.364431Z"
    }
   },
   "source": [
    "def validate_pdf(file_path: str) -> bool:\n",
    "    if not os.path.exists(file_path):\n",
    "        print(f\"Error: 此文件找不到啊🤔: {file_path}\")\n",
    "        return False\n",
    "    if not file_path.lower().endswith('.pdf'):\n",
    "        print(\"Error: 此文件不是 PDF 😵‍💫\")\n",
    "        return False\n",
    "    return True"
   ],
   "outputs": [],
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "id": "5a362ac3",
   "metadata": {},
   "source": "这里就简单的读取 PDF 保存为`.txt`文件。默认的最大字符数是10万，如果你有长篇大论要处理，随时可以调整哦！不过，记得要考虑模型的长度限制～"
  },
  {
   "cell_type": "code",
   "id": "b57c2d64-3d75-4aeb-b4ee-bd1661286b66",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.376960Z",
     "start_time": "2024-10-31T16:04:58.373410Z"
    }
   },
   "source": [
    "def extract_text_from_pdf(file_path: str, max_chars: int = 100000) -> Optional[str]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "\n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            # 创建 PDF reader 对象\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "\n",
    "            # 获取总页数\n",
    "            num_pages = len(pdf_reader.pages)\n",
    "            print(f\"处理 {num_pages} 页的 PDF...\")\n",
    "\n",
    "            extracted_text = []\n",
    "            total_chars = 0\n",
    "\n",
    "            # 遍历所有页面\n",
    "            for page_num in range(num_pages):\n",
    "                # 提取当前页面的文本\n",
    "                page = pdf_reader.pages[page_num]\n",
    "                text = page.extract_text()\n",
    "\n",
    "                # 检查文本是否超过最大字符数\n",
    "                if total_chars + len(text) > max_chars:\n",
    "                    # 仅截取到最大字符数\n",
    "                    remaining_chars = max_chars - total_chars\n",
    "                    extracted_text.append(text[:remaining_chars])\n",
    "                    print(f\"在第 {page_num + 1} 页超过 {max_chars} 字符限制啦！！\")\n",
    "                    break\n",
    "\n",
    "                extracted_text.append(text)\n",
    "                total_chars += len(text)\n",
    "                print(f\"已处理 {page_num + 1}/{num_pages} 页\")\n",
    "\n",
    "            final_text = '\\n'.join(extracted_text)\n",
    "            print(f\"\\n提取完成！总字数：{len(final_text)}\")\n",
    "            return final_text\n",
    "\n",
    "    except PyPDF2.PdfReadError:\n",
    "        print(\"Error: 这 PDF 有毛病啊，请仔细看看～\")\n",
    "        return None\n",
    "    except Exception as e:\n",
    "        print(f\"完了，挂了: {str(e)}\")\n",
    "        return None\n"
   ],
   "outputs": [],
   "execution_count": 20
  },
  {
   "cell_type": "markdown",
   "id": "e023397b",
   "metadata": {},
   "source": "获取 PDF 元信息"
  },
  {
   "cell_type": "code",
   "id": "0984bb1e-d52c-4cec-a131-67a48061fabc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.386444Z",
     "start_time": "2024-10-31T16:04:58.384045Z"
    }
   },
   "source": [
    "# 获取 PDF 元信息\n",
    "def get_pdf_metadata(file_path: str) -> Optional[dict]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "\n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "            metadata = {\n",
    "                'num_pages': len(pdf_reader.pages),\n",
    "                'metadata': pdf_reader.metadata\n",
    "            }\n",
    "            return metadata\n",
    "    except Exception as e:\n",
    "        print(f\"提取元信息失败: {str(e)}\")\n",
    "        return None"
   ],
   "outputs": [],
   "execution_count": 21
  },
  {
   "cell_type": "markdown",
   "id": "6019affc",
   "metadata": {},
   "source": [
    "Finally, we can run our logic to extract the details from the file"
   ]
  },
  {
   "cell_type": "code",
   "id": "63848943-79cc-4e21-8396-6eab5df493e0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.801603Z",
     "start_time": "2024-10-31T16:04:58.392756Z"
    }
   },
   "source": [
    "# 提取元信息\n",
    "print(\"正在提取元信息...\")\n",
    "metadata = get_pdf_metadata(pdf_path)\n",
    "if metadata:\n",
    "    print(\"\\nPDF 元信息:\")\n",
    "    print(f\"页数: {metadata['num_pages']}\")\n",
    "    print(\"文档信息:\")\n",
    "    for key, value in metadata['metadata'].items():\n",
    "        print(f\"{key}: {value}\")\n",
    "\n",
    "# 提取文本\n",
    "print(\"\\n提取文本...\")\n",
    "extracted_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# 预览前500字符文本\n",
    "if extracted_text:\n",
    "    print(\"\\n预览提取后的文本 (前500字符):\")\n",
    "    print(\"-\" * 50)\n",
    "    print(extracted_text[:500])\n",
    "    print(\"-\" * 50)\n",
    "    print(f\"\\n提取的总字符数: {len(extracted_text)}\")\n",
    "\n",
    "# 将提取的文本保存到 txt 文件中\n",
    "if extracted_text:\n",
    "    output_file = './resources/extracted_text.txt'\n",
    "    with open(output_file, 'w', encoding='utf-8') as f:\n",
    "        f.write(extracted_text)\n",
    "    print(f\"\\n提取的文本已保存到 {output_file}\")"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在提取元信息...\n",
      "\n",
      "PDF 元信息:\n",
      "页数: 43\n",
      "文档信息:\n",
      "/Author: \n",
      "/CreationDate: D:20241022021202Z\n",
      "/Creator: LaTeX with hyperref\n",
      "/Keywords: \n",
      "/ModDate: D:20241022021202Z\n",
      "/PTEX.Fullbanner: This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5\n",
      "/Producer: pdfTeX-1.40.25\n",
      "/Subject: \n",
      "/Title: \n",
      "/Trapped: /False\n",
      "\n",
      "提取文本...\n",
      "处理 43 页的 PDF...\n",
      "已处理 1/43 页\n",
      "已处理 2/43 页\n",
      "已处理 3/43 页\n",
      "已处理 4/43 页\n",
      "已处理 5/43 页\n",
      "已处理 6/43 页\n",
      "已处理 7/43 页\n",
      "已处理 8/43 页\n",
      "已处理 9/43 页\n",
      "已处理 10/43 页\n",
      "已处理 11/43 页\n",
      "已处理 12/43 页\n",
      "已处理 13/43 页\n",
      "已处理 14/43 页\n",
      "已处理 15/43 页\n",
      "已处理 16/43 页\n",
      "在第 17 页超过 100000 字符限制啦！！\n",
      "\n",
      "提取完成！总字数：100016\n",
      "\n",
      "预览提取后的文本 (前500字符):\n",
      "--------------------------------------------------\n",
      "1\n",
      "A Survey on Knowledge Distillation of Large\n",
      "Language Models\n",
      "Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1,\n",
      "Can Xu5, Dacheng Tao6, Tianyi Zhou2\n",
      "1The University of Hong Kong2University of Maryland3Microsoft\n",
      "4University of Technology Sydney5Peking University6The University of Sydney\n",
      "{shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu\n",
      "ckcheng@cs.hku.hk jl0725@connect.hku.hk\n",
      "Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati\n",
      "--------------------------------------------------\n",
      "\n",
      "提取的总字符数: 100016\n",
      "\n",
      "提取的文本已保存到 ./resources/extracted_text.txt\n"
     ]
    }
   ],
   "execution_count": 22
  },
  {
   "cell_type": "markdown",
   "id": "946d1f59",
   "metadata": {},
   "source": [
    "### 预处理\n",
    "现在我们来说说为啥不用正则表达式的提取，而使用大型语言模型的：\n",
    "目前，我们已经从 PDF 中提取出一个文本文件。一般来说，由于字符、格式、Latex、表格等原因，PDF 提取出的内容可能会很凌乱。\n",
    "解决这个问题的一种方法是使用正则表达式，但我们也可以使用轻量模型帮助清洗文本。\n",
    "你可以试试修改 `SYS_PROMPT`"
   ]
  },
  {
   "cell_type": "code",
   "id": "7c0828a5-964d-475e-b5f5-40a04e287725",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.809790Z",
     "start_time": "2024-10-31T16:04:58.807967Z"
    }
   },
   "source": [
    "SYS_PROMPT = \"\"\"您是一流的文本预处理器。以下是来自 PDF 的原始数据，请解析并以清晰可用的方式返回给播客作者。\n",
    "原始数据中包含换行符、Latex 数学公式以及一些可以完全删除的冗余内容。请去掉对播客作者记录无用的信息。\n",
    "不论主题如何，以上问题并不详尽，因此请聪明选择要删除的信息，发挥创造力。\n",
    "请注意，您的任务仅限于清理文本和必要时重写，在删除细节时要果断。当您进行文本处理时，请确保输出中文。\n",
    "如果遇到英文或其他语言，请翻译为中文后再输出。\n",
    "请勿添加 Markdown 格式化或特殊字符。\n",
    "一开始便直接开始响应处理后的文本，不做任何确认，谢谢！\n",
    "这里是文本：\n",
    "\"\"\""
   ],
   "outputs": [],
   "execution_count": 23
  },
  {
   "cell_type": "markdown",
   "id": "fd393fae",
   "metadata": {},
   "source": "为了避免模型一次处理整个文件，我们将分块处理文件。"
  },
  {
   "cell_type": "code",
   "id": "24e8a547-9d7c-4e2f-be9e-a3aea09cce76",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:04:58.818065Z",
     "start_time": "2024-10-31T16:04:58.815695Z"
    }
   },
   "source": [
    "def create_word_bounded_chunks(text, target_chunk_size):\n",
    "    \"\"\"\n",
    "    这里会根据 target_chunk_size 分块\n",
    "    \"\"\"\n",
    "    words = text.split()\n",
    "    chunks = []\n",
    "    current_chunk = []\n",
    "    current_length = 0\n",
    "\n",
    "    for word in words:\n",
    "        word_length = len(word) + 1  # +1 for the space\n",
    "        if current_length + word_length > target_chunk_size and current_chunk:\n",
    "            # Join the current chunk and add it to chunks\n",
    "            chunks.append(' '.join(current_chunk))\n",
    "            current_chunk = [word]\n",
    "            current_length = word_length\n",
    "        else:\n",
    "            current_chunk.append(word)\n",
    "            current_length += word_length\n",
    "\n",
    "    # Add the last chunk if it exists\n",
    "    if current_chunk:\n",
    "        chunks.append(' '.join(current_chunk))\n",
    "\n",
    "    return chunks"
   ],
   "outputs": [],
   "execution_count": 24
  },
  {
   "cell_type": "markdown",
   "id": "5d74223f",
   "metadata": {},
   "source": "让我们加载模型并开始处理文本块吧！✨"
  },
  {
   "cell_type": "code",
   "id": "d04a4f07-b0b3-45ca-8f41-a433e1abe050",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:05:01.171211Z",
     "start_time": "2024-10-31T16:04:58.824378Z"
    }
   },
   "source": "model, tokenizer = load(DEFAULT_MODEL)",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "21c87d0ec45847b192e1bab42da9d65d"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "execution_count": 25
  },
  {
   "cell_type": "code",
   "id": "bbda5241-e890-4402-87dd-514d6761bb9c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:05:01.180497Z",
     "start_time": "2024-10-31T16:05:01.177917Z"
    }
   },
   "source": [
    "def process_chunk(text_chunk, chunk_num):\n",
    "    \"\"\"处理文本块，并返回模型处理好的文本\"\"\"\n",
    "    conversation = [\n",
    "        {\"role\": \"system\", \"content\": SYS_PROMPT},\n",
    "        {\"role\": \"user\", \"content\": text_chunk},\n",
    "    ]\n",
    "\n",
    "    prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)\n",
    "    processed_text = generate(\n",
    "        model=model,\n",
    "        tokenizer=tokenizer,\n",
    "        prompt=prompt,\n",
    "        max_tokens=512,\n",
    "        temp=0.7\n",
    "    )\n",
    "\n",
    "    # Print chunk information for monitoring\n",
    "    #print(f\"\\n{'='*40} Chunk {chunk_num} {'='*40}\")\n",
    "    print(f\"INPUT TEXT:\\n{text_chunk[:500]}...\")  # Show first 500 chars of input\n",
    "    print(f\"\\nPROCESSED TEXT:\\n{processed_text[:500]}...\")  # Show first 500 chars of output\n",
    "    print(f\"{'=' * 90}\\n\")\n",
    "\n",
    "    return processed_text"
   ],
   "outputs": [],
   "execution_count": 26
  },
  {
   "cell_type": "code",
   "id": "a0183c47-339d-4041-ae83-77fc34931075",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:05:01.192345Z",
     "start_time": "2024-10-31T16:05:01.187401Z"
    }
   },
   "source": [
    "INPUT_FILE = \"./resources/extracted_text.txt\"  \n",
    "CHUNK_SIZE = 1000\n",
    "\n",
    "# 读取文件\n",
    "with open(INPUT_FILE, 'r', encoding='utf-8') as file:\n",
    "    text = file.read()\n",
    "\n",
    "chunks = create_word_bounded_chunks(extracted_text, CHUNK_SIZE)\n",
    "num_chunks = len(chunks)"
   ],
   "outputs": [],
   "execution_count": 27
  },
  {
   "cell_type": "code",
   "id": "bb36814f-9310-4734-bf54-e16a5032339e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:05:01.200630Z",
     "start_time": "2024-10-31T16:05:01.198749Z"
    }
   },
   "source": [
    "num_chunks"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 28
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:05:01.311376Z",
     "start_time": "2024-10-31T16:05:01.309706Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 创建输出文件路径\n",
    "output_file = f\"./resources/clean_{os.path.basename(INPUT_FILE)}\"\n",
    "\n",
    "processed_text = \"\""
   ],
   "id": "59d994601d77baa8",
   "outputs": [],
   "execution_count": 29
  },
  {
   "cell_type": "code",
   "id": "7917dfdd-b3af-44fc-a8c0-2760ace9363e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:12:11.049886Z",
     "start_time": "2024-10-31T16:05:01.390828Z"
    }
   },
   "source": [
    "with open(output_file, 'w', encoding='utf-8') as out_file:\n",
    "    for chunk_num, chunk in enumerate(tqdm(chunks, desc=\"Processing chunks\")):\n",
    "        # 处理文本块\n",
    "        processed_chunk = process_chunk(chunk, chunk_num)\n",
    "        processed_text += processed_chunk + \"\\n\"\n",
    "\n",
    "        # 将处理后的文本写入输出文件\n",
    "        out_file.write(processed_chunk + \"\\n\")\n",
    "        out_file.flush()"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Processing chunks:   0%|          | 0/101 [00:00<?, ?it/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "d97ce7b87fa64629804aefc590a71a41"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "1 A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1, Can Xu5, Dacheng Tao6, Tianyi Zhou2 1The University of Hong Kong2University of Maryland3Microsoft 4University of Technology Sydney5Peking University6The University of Sydney {shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu ckcheng@cs.hku.hk jl0725@connect.hku.hk Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在大规模语言模型（LLMs）的时代，知识蒸馏（KD）作为将顶级专有LLM如GPT-4等先进能力转移给开源模型如LLaMa和Mistral的关键方法出现了。随着开源模型的蓬勃发展，KD在压缩这些模型和通过自身进行自我提升方面扮演了重要角色。本文全面回顾了KD在LLM领域中的作用，强调了其在赋予模型知识方面的重要功能。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "advanced knowledge to smaller models and its utility in model compression and self- improvement. Our survey is meticulously structured around three foundational pillars: algorithm ,skill, and verticalization – providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the interaction between data augmentation (DA) and KD, illustrating how DA emerges as a powerful ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "通过将高级知识传授给较小的模型，并将其应用于模型压缩和自我提升方面具有实用价值。我们的调查围绕三个基础支柱展开：算法、技能和垂直化，提供了一个全面的评估知识传授机制、特定认知能力的提升及其在多个领域中的实际影响。至关重要的是，调查探讨了数据增强（DA）与知识传授（KD）之间的相互作用，展示了如何将DA作为KD框架中的强大范式来增强模型性能。通过利用DA生成富含上下文且特定技能的训练数据，知识传授超越了传统界限，使得开源模型能够模仿其专用版本的上下文敏感性、伦理一致性以及深层语义洞察。本文旨在为研究人员和实践者提供有价值的指南，详细介绍当前在知识传授方面的研究方法。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distillation and proposing future research directions. By bridging the gap between proprietary and open-source LLMs, this survey underscores the potential for more accessible, efficient, and powerful AI solutions. Most importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs. Index Terms —...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "本文通过将专有和开源大型语言模型（LLMs）之间的差距进行桥梁连接，强调了更具访问性、效率和强大性的AI解决方案的潜在价值。我们强烈倡导遵守用于LLMs的法律条款，确保知识蒸馏（KD）的伦理和合法性应用。有关知识蒸馏的GitHub存储库可在https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs处获取。关键术语——大型语言模型，知识蒸馏，数据增强，技能蒸馏，监督微调。1 引言 在人工智能（AI）不断发展的背景下，GPT-3.5（Ouyang et al., 2022）、GPT-4（OpenAI et al., 2023）、Gemini（Team et al., 2023）和Claude2等专有大型语言模型（LLMs）已经成为颠覆性的技术，极大地改变了我们对自然语言处理（NLP）的理解。这些模型以其庞大的规模而著称。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "complexity, have unlocked new realms of possibility, from generating human-like text to offering sophisticated problem-solving capabilities. The core significance of these LLMs lies in their emergent abil- ities (Wei et al., 2022a,b; Xu et al., 2024a), a phenomenon where the models display capabilities beyond their explicit training objectives, enabling them to tackle a diverse array of tasks with remarkable proficiency. These models excel in understanding and generation, driving applications fr...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这些大型语言模型（LLMs）在解锁新的可能性方面具有核心意义。它们不仅能够生成类似人类的文本，还能够提供高级问题解决能力。这些模型的核心能力在于其涌现能力（Wei et al., 2022a,b; Xu et al., 2024a），即模型在超出其明确训练目标的情况下展现出超出预期的能力，从而能够高效地解决多样化的任务。这些模型在理解和生成方面表现出色，从创意生成到复杂问题解决的应用皆可（OpenAI et al., 2023; Liang et al., 2022）。这些模型的潜力远远超出了当前的应用，有望重塑行业，增强人类的创造力。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "redefine our interaction with technology. Despite the remarkable capabilities of proprietary LLMs like GPT-4 and Gemini, they are not without their shortcom- ings, particularly when viewed in light of the advantages offered by open-source models. A significant drawback is their limited accessibility and higher cost (OpenAI et al., 2023). These proprietary models often come with substantial usage fees and restricted access, making them less attain- able for individuals and smaller organizations. ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "重新定义我们与技术的互动。尽管专有模型如GPT-4和Gemini具有出色的性能，但它们仍然存在一些不足之处，尤其是在与开源模型相比时。一个显著的缺点是访问受限且成本较高（OpenAI等，2023年）。这些专有模型通常包含较高的使用费用且访问受限，使得个人和小型组织难以获得。在数据隐私和安全方面（吴等，2023a），频繁使用这些专有模型往往需要将敏感数据发送至外部服务器，这引发了对数据隐私和安全的担忧。特别是对于处理机密信息的用户来说，这一点尤为重要。此外，专有模型的一般用途设计，虽然强大，但可能并不总是符合特定应用的需求。因此，访问性、成本和适应性等方面的限制至关重要。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "present significant challenges in leveraging the full potential of proprietary LLMs. In contrast to proprietary LLMs, open-source models like LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023a) bring several notable advantages. One of the primaryarXiv:2402.13116v4 [cs.CL] 21 Oct 2024 2 benefits of open-source models is their accessibility and adaptability. Without the constraints of licensing fees or restrictive usage policies, these models are more readily available to a broader rang...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "开放源代码模型如 LLaMA (Touvron et al., 2023) 和 Mistral (Jiang et al., 2023a) 拥有显著优势。这些模型的可访问性和可定制性使其更具吸引力。没有许可费用或限制性使用政策的约束，这些模型更容易被更广泛的研究人员和小型组织使用。这种开放性促进了更加合作和包容的 AI 研究环境，鼓励创新和多样化应用。此外，开放源代码 LLM 的可定制性使其能够提供更加贴合特定需求的解决方案，而这些需求可能是大规模通用模型无法满足的。然而，开放源代码 LLM 也存在一些局限性，主要是由于它们的规模相对较小。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "resources compared to their proprietary counterparts. One of the most significant limitations is the smaller model scale, which often results in lower per- formance on real-world tasks with a bunch of instruc- tions (Zheng et al., 2023a). These models, with fewer pa- rameters, may struggle to capture the depth and breadth of knowledge embodied in larger models like GPT-4. Ad- ditionally, the pre-training investment in these open-source models is typically less substantial. This reduced investmen...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "资源与自有版本相比，最大的限制是模型规模较小，这通常会导致在真实世界任务上的表现不佳，且需要较多的指令（Zheng et al., 2023a）。这些模型由于参数较少，可能难以捕捉像GPT-4这样的大型模型所包含的深度和广度的知识。此外，这些开源模型的预训练投资通常较少。这种投资不足可能导致预训练数据的范围较窄，从而限制模型理解和处理多样化或专门主题的能力（Liang et al., 2022; Sun et al., 2024a）。此外，由于资源限制，开源模型通常接受较少的微调步骤。微调对于优化模型针对特定任务或行业的性能至关重要，缺少微调步骤可能会影响模型在特定应用中的效果。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "particularly evident when these models are compared to the highly fine-tuned proprietary LLMs, which are often tailored to excel in a wide array of complex scenarios (OpenAI et al., 2023). Primarily, recognizing the disparities between propri- etary and open-source LLMs, KD techniques have surged as a means to bridge the performance gap between these models (Gou et al., 2021; Gupta and Agrawal, 2022). Knowl- edge distillation, in this context, involves leveraging the more advanced capabilities o...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "特别是当这些模型与高度定制的专用LLM进行比较时，这些LLM通常被优化以在多种复杂情境中表现出色（OpenAI等，2023年）。主要的是，识别专用和开源LLM之间的差异，KD技术作为缩小这些模型性能差距的一种手段已经兴起（Gou等，2021；Gupta和Agrawal，2022）。在此背景下，知识蒸馏涉及利用领先专用模型如GPT-4或Gemini等先进的能力作为指导框架来提升开源LLM的性能。这一过程类似于将高技能教师的知识传授给学生（例如开源LLM），即学生（例如开源LLM）学习模仿教师（例如专用LLM）的表现特征。与传统的知识蒸馏算法（Gou等，2021年）相比，数据增强（DA）（Feng等，2021）已成为一种普遍的技术。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "paradigm to achieve knowledge distillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a specific skill or domain (Taori et al., 2023). Secondly, KD still retains its fundamental role in compressing LLMs, making them more efficient without significant loss in performance. (Gu et al., 2024; Agarwal et al., 2024). More recently, the strategy of employing open-source LLMs as teachers for their own self-improvement has emerged as a promisi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "知识蒸馏在LLM（大型语言模型）中的应用主要涉及两个方面。首先，通过使用少量的关键知识来引导LLM生成更多与特定技能或领域相关的数据（Taori等人，2023年）。其次，知识蒸馏仍保有其核心作用，即压缩LLM，使其更加高效，同时性能损失不大（Gu等人，2024年；Agarwal等人，2024年）。最近，使用开源LLM作为自身提升的教师，这一策略已经崭露头角，显著提升了其能力（Yuan等人，2024a年；Chen等人，2024a年）。图1展示了知识蒸馏在LLM中的三种关键角色。知识蒸馏的一个关键方面是提升技能，如高级上下文理解（此处似乎有误，应为“高级上下文理解”）。具体来说，图1显示了知识蒸馏在LLM中的三种关键角色：1）主要提升能力，2）提供传统的。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "compression for efficiency, and 3) an emerging trend of self-improvement via self-generated knowledge. (e.g., in-context learning (Huang et al., 2022a) and in- struction following (Taori et al., 2023)), improved align- ment with user intents (e.g., human values/principles (Cui et al., 2023a), and thinking patterns like chain-of-thought (CoT) (Mukherjee et al., 2023)), and NLP task specialization (e.g., semantic understanding (Ding et al., 2023a), and code generation (Chaudhary, 2023)). These ski...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "压缩以提高效率，以及 3） 通过生成知识进行自我提升的趋势（例如，上下文学习（Huang 等人，2022a）和指令跟随（Taori 等人，2023）），提高与用户意图的适配性（例如，人类价值观/原则（Cui 等人，2023a），以及像链式思考（CoT）（Mukherjee 等人，2023）那样的思维方式），以及自然语言处理任务的专门化（例如，语义理解（Ding 等人，2023a），以及代码生成（Chaudhary，2023））。这些技能对于 LLM（Large Language Model）预期要执行的各种应用至关重要，从普通的对话到在特定领域中复杂的问题解决。例如，在医疗（Wang 等人，2023a）、法律（LAW，2023）或科学（Zhang 等人，2024）等垂直领域中，准确性及特定情境的知识是至关重要的，知识蒸馏使得开源模型能够通过学习专有模型的知识，显著提升其性能。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "been extensively trained and fine-tuned in these areas. The benefits of knowledge distillation in the era of LLMs are multifaceted and transformative (Gu et al., 2024). Through a suite of distillation techniques, the gap between proprietary and open-source models is significantly nar- rowed (Chiang et al., 2023; Xu et al., 2023a) and even filled (Zhao et al., 2023a). This process not only streamlines computational requirements but also enhances the environ- mental sustainability of AI operations...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "知识蒸馏在大模型时代带来了多方面的益处。通过一系列蒸馏技术，专有模型和开源模型之间的差距显著缩小甚至被消除（Gu等人，2024年）。这一过程不仅简化了计算需求，还提高了AI操作的环境可持续性，因为开源模型在计算负荷降低的情况下变得更为高效。此外，知识蒸馏促进了更包容和公平的AI环境，让较小的实体和个人研究者也能接触到最先进的技术，从而鼓励更多行业参与和多样性。这一技术的普及推动了各个行业的创新和增长。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and research domains. The escalating need for a comprehensive survey on the knowledge distillation of LLMs stems from the rapidly evolving landscape of AI (OpenAI et al., 2023; Team et al., 2023) and the increasing complexity of these models. As AI continues to penetrate various sectors, the ability to effi- ciently and effectively distill knowledge from proprietary LLMs to open-source ones becomes not just a technical aspiration but a practical necessity. This need is driven by the growing dema...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "研究领域正面临综合调查的挑战，以知识提炼的方式提升LLMs的效能。随着AI（OpenAI等，2023年；Team等，2023年）的发展，以及模型日益复杂，AI的渗透正扩展到各个领域。为了更高效、更有效地从专有LLMs提炼知识到开源模型中，这种需求已不再仅仅是一项技术上的追求，而是成为了一种现实的必要性。这种需求源于对更多易于访问、成本更低、更加灵活的AI解决方案的迫切需求，这些解决方案能够适应多样的场景。带有种子知识的生成知识库示例、数据集、示范任务、上下文关联、对齐代理、NLP任务特化、多模态技能、法律、金融、科学、其他领域的知识提取与提炼算法、训练差异与相似性指导。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "ReinforcementLearningoutputsreward RM!(·)distill SupervisedFine-tuningX,Y preferenceRankOptimizationy,1y,2y3y1y2y3≻≻rank…… DataCuration X,YrawdatasynthesizefeedbackFeedback input outputSelf-Knowledge outputinputinput YlabelLabelingExpansion X,YdemonstrationsexpandFeature featureinput,outputextractSec.4Sec.5 Sec.3.1Sec.3.2①②③④ Fig. 2: An overview of this survey on knowledge distillation of large language models. Note that ‘Section’ is abbreviated as ‘Sec.’ in this figure. RM S(·)denotes the stude...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这是一个关于大规模语言模型知识蒸馏的调查报告概述。知识蒸馏的过程分为若干步骤，每个步骤都有对应的步骤编号。报告还提到了一个名为RM S(·)的模型，它代表学生奖励模型。报告强调了对知识蒸馏领域当前方法、挑战和突破进行综合调查的重要性，这有助于为研究人员和实践者提供方向。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "future research. Survey Organization. The remainder of this survey is orga- nized into several comprehensive sections, each designed to offer a deep dive into the multifaceted aspects of knowledge distillation within the realm ofLLMs. Following this intro- duction, §2 provides a foundational overview of knowledge distillation, comparing traditional techniques with those emerging in the era of LLMs and highlighting the role of data augmentation (DA) in this context. §3 delves into the approaches ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "接下来的调查研究组织。本文的调查内容分为几个综合部分，旨在深入探讨在LLMs领域的知识蒸馏的多方面内容。在介绍之后，§2部分提供了一种基础知识概述，比较了传统方法与LLMs时代新兴方法，并强调了数据增强（DA）在此领域的角色。§3部分深入探讨了从教师LLMs提取知识的方法和核心蒸馏算法，包括从监督微调到涉及差异和相似性、强化学习和优化排序的复杂策略。§4部分则专注于技能蒸馏，探讨如何增强学生模型以提高上下文理解、与用户意图的一致性和在各种NLP任务上的表现。其中包括讨论自然语言的理解。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(NLU), genera- tion (NLG), information retrieval, recommendation systems, and the evaluation of text generation. In §5, we venture into domain-specific vertical distillation, showcasing how knowledge distillation techniques are applied within spe- cialized fields such as law, healthcare, finance, and science,illustrating the practical implications and transformative impact of these approaches. The survey suggests open problems in §6, identifying current challenges and gaps in knowledge distillat...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里的概述介绍了知识蒸馏的概念。知识蒸馏是在人工智能和深度学习领域的一种方法，它涉及将知识从一个大型且复杂的模型（通常称为教师模型）转移到一个较小且更具可解释性的模型（称为学生模型）的过程。通过这种方法，学生模型可以捕捉教师模型的重要特征和知识，从而在保持性能的同时减少模型的复杂度和参数量。\n",
      "\n",
      "在第5节中，我们探讨了特定领域的垂直知识蒸馏，展示了知识蒸馏技术如何应用于法律、医疗、金融和科学等特定领域。通过这种方式，我们可以看到知识蒸馏在这些专业领域的实际应用及其带来的变革影响。\n",
      "\n",
      "最后，在第7节中，我们总结了这些发现，反思了知识蒸馏对更广泛的人工智能和自然语言处理研究社区的影响，并提出了未来研究的方向。图2提供了该调研的概览。\n",
      "\n",
      "图2展示了这个调研的概览。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "model (teacher) to a smaller, more efficient model (student) (Gou et al., 2021). This technique is pivotal in mitigating the challenges posed by the computational demands and resource constraints of deploying large-scale models in practical applications. Historically, knowledge distillation techniques, prior to the era of LLMs, primarily concentrated on transferring knowledge from complex, often cumbersome neural net- works to more compact and efficient architectures (Sanh et al., 2019; Kim and ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "模型 (teacher) 被缩减为一个更小、更高效的模型 (student) (Gou 等人, 2021)。这种方法在缓解大规模模型在实际应用中由于计算需求和资源限制带来的挑战方面至关重要。历史上，知识蒸馏技术，在大规模语言模型 (LLM) 时代之前，主要集中在将复杂、经常复杂的神经网络中的知识转移到更紧凑和效率更高的架构中 (Sanh 等人, 2019; Kir 和 Rush, 2016)。这一过程主要由在资源有限的环境中部署机器学习模型的需要驱动，如移动设备或边缘计算平台，这里的计算能力和内存有限。早期方法主要关注资源受限环境下的单任务神经架构选择和训练目标。这些早期方法 4. 知识蒸馏 (He 等人, 2023a) 和 PandaLM (Wang 等人, 2023b)。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "CoT-Distill (Hsieh et al., 2023) Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023), Baize (Xu et al., 2023b), Mammoth (Yue et al., 2023a), Mixed Distill (Chenglin et al., 2023) ExpansionSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Code Alpaca (Chaudhary, 2023) Self-Align (Sun et al., 2024b), WizardLM (Xu et al., 2023a), WizardCoder (Luo et al., 2023a), WizardMath (Luo et al., 2023b), AugGPT (Dai et al., 2023a), TDG (He et al., 2023b) CurationUltraChat (Ding et al., 2...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这些内容是关于预训练语言模型和生成模型的汇总，主要涉及论文作者及其贡献。根据内容，似乎在列举一些预训练模型和生成模型的名称，但并没有具体解释或详细说明这些模型的功能和性能。以下是整理后的信息：\n",
      "\n",
      "- CoT-Distill (Hsieh et al., 2023)\n",
      "- Orca (Mukherjee et al., 2023)\n",
      "- Orca 2 (Mitra et al., 2023)\n",
      "- Baize (Xu et al., 2023b)\n",
      "- Mammoth (Yue et al., 2023a)\n",
      "- Mixed Distill (Chenglin et al., 2023)\n",
      "- ExpansionSelf-Instruct (Wang et al., 2022a)\n",
      "- Alpaca (Taori et al., 2023)\n",
      "- Code Alpaca (Chaudhary, 2023)\n",
      "- Self-Align (Sun et al., 2024b)\n",
      "- WizardLM (Xu et al., 2023a)\n",
      "- WizardCoder (Luo et al., 2023a)\n",
      "- W...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(Tunstall et al., 2023), CycleAlign (Hong et al., 2023), RLAIF (Lee et al., 2023a), Lion (Jiang et al., 2023b), PERsD (Chen et al., 2023a), GKD (Agarwal et al., 2024) Self-KnowledgeSelf-Instruct (Wang et al., 2022a), Self-Align (Sun et al., 2024b), RLCD (Yang et al., 2024), ImpDistill (Jung et al., 2023), LMSI (Huang et al., 2023a), ReST (Gulcehre et al., 2023), Self-Rewarding (Yuan et al., 2024a), Baize (Xu et al., 2023b), STaR (Zelikman et al., 2022) DistillationSupervised Fine-TuningAlpaca (T...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里是文本：\n",
      "- STaR (Zelikman et al., 2022)\n",
      "- UltraFeedback (Cui et al., 2023a)\n",
      "- MiniLLM (Gu et al., 2024)\n",
      "- MiniLLM (Xu et al., 2023b)\n",
      "- BabyLlama(Timiryasov and Tastet, 2023)...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(Gu et al., 2024), GKD (Agarwal et al., 2024), GPT3 Reward (Kwon et al., 2023) Rank Optimization Zephyr (Tunstall et al., 2023), CycleAlign (Hong et al., 2023), Skill DistillationContext FollowingInstruction FollowingSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023a), Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023), WizardMath (Luo et al., 2023b), Llama-GPT4 (Peng et al., 2023a), Multi-turn DialogueVicuna (Chiang ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "以下是近年来一些重要的预训练模型和方法：\n",
      "\n",
      "- GKD (Agarwal et al., 2024)\n",
      "- GPT3 Reward (Kwon et al., 2023)\n",
      "- Rank Optimization Zephyr (Tunstall et al., 2023)\n",
      "- CycleAlign (Hong et al., 2023)\n",
      "- Skill Distillation Context Following\n",
      "- Self-Instruct (Wang et al., 2022a)\n",
      "- Alpaca (Taori et al., 2023)\n",
      "- Vicuna (Chiang et al., 2023)\n",
      "- WizardLM (Xu et al., 2023a)\n",
      "- Orca (Mukherjee et al., 2023)\n",
      "- Orca 2 (Mitra et al., 2023)\n",
      "- WizardMath (Luo et al., 2023b)\n",
      "- Llama-GPT4 (Peng et al., 2023a)\n",
      "- Multi-turn DialogueVicuna (Chiang...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Reward (Kwon et al., 2023), ILF (Scheurer et al., 2023), ALMoST (Kim et al., 2023a), RLEF (Roit et al., 2023), RLAIF (Lee et al., 2023a), Zephy (Tunstall et al., 2023), UltraFeedback (Cui et al., 2023a), ValueCAI (Bai et al., 2022a), Align Honesty (Yang et al., 2023a), SANDBOX (Liu et al., 2023b), Self-Align (Sun et al., 2024b), UltraFeedback (Cui et al., 2023a), RLCD (Yang et al., 2024) AgentTool UsingToolformer (Schick et al., 2023), Graph-ToolFormer (Zhang, 2023), Gorilla (Patil et al., 2023)...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这些方法分别来自:\n",
      "- Kwon et al. (2023)\n",
      "- Scheurer et al. (2023)\n",
      "- Kim et al. (2023a)\n",
      "- Roit et al. (2023)\n",
      "- Lee et al. (2023a)\n",
      "- Tunstall et al. (2023)\n",
      "- Cui et al. (2023a)\n",
      "- Bai et al. (2022a)\n",
      "- Yang et al. (2023a)\n",
      "- Liu et al. (2023b)\n",
      "- Sun et al. (2024b)\n",
      "- Cui et al. (2023a)\n",
      "- Yang et al. (2024)\n",
      "- Schick et al. (2023)\n",
      "- Zhang (2023)\n",
      "- Patil et al. (2023)\n",
      "- Tang et al. (2023a)\n",
      "- Qin et al. (2023a)\n",
      "- Yuan et al. (2023a)\n",
      "- Gao et al. (2023b)\n",
      "- Wang et al. (2024)\n",
      "- Shen et al. (2024)\n",
      "- Chen et al. (2023b...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023a), Mix Distill (Chenglin et al., 2023), Annollm (He et al., 2023a), UDG (Wang et al., 2021a), ZeroGen (Ye et al., 2022), NLGInheritSumm (Xu et al., 2023c), RECOMP (Xu et al., 2024b), MaRio (Ramnath et al., 2023), ID (Jung et al., 2023), GPT-3 Labeling (Wang et al., 2021b), BioGPT (Guo et al., 2023a), ChatGPT NMT (Yang and Nicolai, 2023), Information RetrievalQUILL (Srinivasan et al., 2022), Promptgator (Dai et al., 2023b), InPars (Bonifacio et al., 2022), AugTriever (Meng et al., 2023), (Su...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2023年，Mix Distill (Chenglin等人, 2023), Annollm (He等人, 2023a), UDG (Wang等人, 2021a), ZeroGen (Ye等人, 2022), NLGInheritSumm (Xu等人, 2023c), RECOMP (Xu等人, 2024b), MaRio (Ramnath等人, 2023), ID (Jung等人, 2023), GPT-3 Labeling (Wang等人, 2021b), BioGPT (Guo等人, 2023a), ChatGPT NMT (Yang和Nicolai, 2023), Information RetrievalQUILL (Srinivasan等人, 2022), Promptgator (Dai等人, 2023b), InPars (Bonifacio等人, 2022), AugTriever (Meng等人, 2023), RankVicuna (Pradeep等人, 2023a), RankZephyr (Pradeep等人, 2023b), ExaRanker (Ferrar...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023) Phi-1 (Gunasekar et al., 2023), PERsD (Chen et al., 2023a), MFTCoder (Liu et al., 2023d), WaveCoder (Yu et al., 2024), Code Clean (Jain et al., 2023), Multi-ModalityLLaVA (Liu et al., 2023e), SVIT (Zhao et al., 2023b), LVIS-Instruct4V (Wang et al., 2023e), Shikra (Chen et al., 2023c), LSKD (Park et al., 2023), DetGPT (Pi et al., 2023; Zhao et al., 2023c), LRV (Liu et al., 2023f), NExT-GPT (Wu et al., 2023b), Valley (Luo et al., 2023d), ILuvUI (Jiang et al., 2023d), StableLLaVA (Li et ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "图3展示了大规模语言模型知识蒸馏的分类。图7详细展示了垂直化蒸馏的分类结构。涉及训练了一个较小的学生网络。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "to mimic the output of a larger teacher network, often through techniques like soft target training, where the student learns from the softened softmax output of the teacher. Please refer to the survey (Gou et al., 2021) for more details on general knowledge distillation techniques in AI and DL. In contrast, the advent of LLMs has revolutionized the knowledge distillation landscape. The current era of knowledge distillation in LLMs shifts the focus from mere architecture compression to knowledge...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "为了模仿大型教师网络的输出，通常通过软目标训练等技术实现。可以参考Gou等人（2021）的调查了解知识蒸馏在AI和DL中的通用知识蒸馏技术。相比之下，大规模语言模型（LLMs）的出现彻底改变了知识蒸馏的领域。当前的LLMs知识蒸馏时代，重点从单纯的架构压缩转向知识提取与转移（Taori等人，2023；Chaudhary，2023；Tunstall等人，2023）。这种范式转变很大程度上归因于大型语言模型如GPT-4和Gemini所拥有的广泛且深刻的知识。而LLMs中不可访问的参数使得通过剪枝（Han等人，2016）或量化（Liu等人，2023a）技术压缩它们变得困难。与早期时代不同，那时的目标是复制教师模型的输出行为或减小模型规模。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "focus in LLM-based knowledge distillation is to elicit the specific knowledge these models have. The key to this modern approach lies in heuristic and carefully designed prompts, which are used to elicit specific knowledge (Ding et al., 2023b) or capabilities (Chaudhary, 2023) from the LLMs. These prompts are crafted to tap into the LLM’s understanding and capabilities in various domains, ranging from natural language understanding (He et al., 2023a) to more complex cognitive tasks like reason- ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "LLM基知識提取的關鍵在於精心設計的提示，用於提取模型具備的特定知識（Ding等，2023b）或能力（Chaudhary，2023）。這些提示會觸發模型在各領域的理解和能力，範圍從自然語言理解（He等，2023a）到更複雜的任務，如推理（Hsieh等，2023）和解決問題（Qiao等，2024）。使用提示進行知識提取提供了一種更加靈活且動態的提取方法。它允許更精目標地提取知識，聚焦於特定技能或興趣領域。這種方法在發揮模型潛在能力方面尤其有效，模型會展現超出其明確訓練目標的能力。此外，這一世節知識提取的時代。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distillation also em- phasizes the transfer of more abstract qualities such as reasoning patterns (Mitra et al., 2023), preference align- ment (Cui et al., 2023a), and value alignment (Sun et al., 2024b). This is in stark contrast to the earlier focus on output replication (Taori et al., 2023), indicating a shift towards a more holistic and comprehensive transfer of cognitive capabilities. The current techniques involve not just the replication of outputs, but also the emulation of the thought p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "深度学习通过强调更多抽象的质量转移，如推理模式（Mitra et al., 2023）、偏好一致（Cui et al., 2023a）以及价值一致（Sun et al., 2024b），来转移更高级的技能。这与早期关注输出复制（Taori et al., 2023）的关注点形成了鲜明对比，表明正在朝向更加全面和综合的技能转移转变。当前的技术不仅涉及输出的复制，还模拟了教师模型的思维过程（Mitra et al., 2023）和决策模式（Asai et al., 2023）。这包括复杂的策略，如链式思考提示，使得学生模型能够学习教师模型的推理过程，从而提升解决问题和决策的能力。在大语言模型（LLMs）的时代，数据增强（DA）（Wang et al., 2022a; Ye et al., 2022）已经成为一个关键的范式，对整个过程至关重要。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "knowledge distillation. Unlike traditional DA techniques such as paraphrasing (Gangal et al., 2022) or back-translation (Longpre et al., 2019), which primarily aim at expanding the training dataset in a somewhat mechanical manner, DA within the context of LLMs focuses on the generation of novel, context-rich training data tailored to specific domains and skills.The relationship between DA and KD in LLMs is both symbiotic and foundational. By leveraging a set of seed knowledge, KD employs DA to p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "知识蒸馏与传统的数据增强（如 paraphrasing 或 back-translation）不同，这些方法主要是通过机械方式扩展训练数据集。在大模型（LLMs）的背景下，知识蒸馏通过数据增强促使大模型生成针对特定领域和技能的详细数据。这种做法既具有共生性又具有基础性。通过利用一组种子知识，知识蒸馏利用数据增强促使大模型生成能够体现特定技能或领域专长的明确数据（Chaudhary, 2023; West et al., 2022）。这种方法在闭源模型和开源模型之间填补知识和能力之间的差距。通过数据增强，大模型被引导生成目标明确、高质量的数据集，这些数据集不仅在规模上更大，而且在多样性和具体性上也更为丰富。这种方法使得知识蒸馏过程更为有效。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "ensuring that the distilled models not only replicate the teacher model’s output behavior but also embody its deep-seated understanding and cognitive strategies. DA acts as a force multiplier, enabling the distilled mod- els to acquire and refine capabilities that would otherwise require exponentially larger datasets and computational re- sources. It facilitates a more effective transfer of knowledge, focusing on the qualitative aspects of learning rather than quantitative expansion. This strate...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "确保通过蒸馏模型不仅复制教师模型的输出行为，还内化其深层次的理解和认知策略。DA作为倍增器，使蒸馏模型能够获取并精进原本需要更大数据集和计算资源才能实现的能力。它有助于更有效的知识转移，专注于学习的质的方面而非量的扩展。在KD流程中战略性使用DA强调了一种更加高效、可持续和易于访问的方法，以充分发挥LLMs的潜力。它使开源模型能够模拟其专业版本的上下文敏感性、伦理一致性以及深层语义洞察，从而扩大先进AI能力的可访问性，并在更广泛的领域和用户中促进创新。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Building on the discussions introduced earlier, this survey aims to comprehensively explore the landscape of knowl- edge distillation within the context of LLMs, following a meticulously structured taxonomy as in Figure 3. The survey’s scope is delineated through three primary facets: KD Algorithms, Skill Distillation, and Verticalization Dis- tillation. Each facet encapsulates a range of subtopics and methodologies. It’s important to note that KD algorithms provide the technical foundations for...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Building on earlier discussions, this survey aims to comprehensively explore knowledge distillation within the context of LLMs, following a meticulously structured taxonomy as in Figure 3. The survey's scope is delineated through three primary facets: KD Algorithms, Skill Distillation, and Verticalization Distillation. Each facet encapsulates a range of subtopics and methodologies. It's important to note that KD algorithms provide the technical foundations for skill distillation and verticalizat...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023), curation (Gu- nasekar et al., 2023), feature understanding (Agarwal et al., 2024), feedback mechanisms (Tunstall et al., 2023), and self- knowledge generation (Wang et al., 2022a). This exploration seeks to uncover the various ways in which knowledge can be identified, expanded, and curated for effective dis- tillation. The ‘ distillation ’ subsection examines learning ap- proaches like supervised fine-tuning (SFT) (Wang et al., 2022a), divergence minimization (Agarwal et al., 2024),...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "al., 2023), feature understanding (Agarwal et al., 2024), feedback mechanisms (Tunstall et al., 2023), and self-knowledge generation (Wang et al., 2022a). 这一探索旨在揭示知识如何被识别、扩展和整理以实现有效提炼的方式。\"提炼\"部分探讨了如监督微调（SFT）（Wang et al., 2022a）、分歧最小化（Agarwal et al., 2024）、强化学习技术（Cui et al., 2023a）和排序优化策略（Tunstall et al., 2023）等学习方法。这些技术展示了KD如何使开源模型能够从专有模型中获取知识。6 技能提炼。这一部分探讨了通过KD增强的具体技能和能力。它涵盖了关于上下文跟随（Taori et al., 2023；Luo et al., 2023c）的详细讨论，包括指令跟随等子话题。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "retrieval-augmented generation (RAG) Capa- bility. In the realm of alignment (Mitra et al., 2023; Tun- stall et al., 2023), the survey investigates thinking patterns, persona/preference modeling, and value alignment. The ‘agent’ category delves into skills such as Tool Using and Planning. NLP task specialization (Dai et al., 2023a; Jung et al., 2023; Chaudhary, 2023) is scrutinized through lenses like natural language understanding (NLU), natural lan- guage generation (NLG), information retrieva...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "retrieval-augmented generation (RAG) 能力。在对齐（Mitra et al., 2023; Tun- stall et al., 2023）的领域中，该调查研究思考模式、个性/偏好建模及价值对齐。‘代理’类别探讨了如工具使用和计划等技能。自然语言任务专业化（Dai et al., 2023a; Jung et al., 2023; Chaudhary, 2023）通过自然语言理解（NLU）、自然语言生成（NLG）、信息检索、推荐系统、文本生成评估及代码生成等视角进行审视。最后，该调查研究多模态（Liu et al., 2023e; Zhao et al., 2023b），探讨如何增强模型的多模态输入整合能力。垂直化蒸馏。该部分评估了在不同垂直领域应用蒸馏的方式，提供了如何将蒸馏模型专门化应用于如法律（LAW, 2023）及医疗与护理（Wang et al., 2023）等领域的见解。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023a), Finance (Zhang and Yang, 2023), Science (Zhang et al., 2024), among others. This exploration not only showcases the practical implications of KD tech- niques but also highlights their transformative impact on domain-specific AI solutions. Through these facets, this survey provides a compre- hensive analysis of KD in LLMs, guiding researchers and practitioners through methodologies, challenges, and op- portunities in this rapidly evolving domain. Declaration. This survey represents our ea...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2023年)，金融(张和杨，2023年)，科学(张等人，2024年)，等等。本文不仅展示了知识蒸馏技术的实用意义，还强调了其对特定领域AI解决方案的变革影响。通过这些方面，本文为知识蒸馏技术在预训练语言模型(LLMs)中的应用提供了全面分析，为研究人员和实践者提供了方法、挑战和机会的指导。声明。本文代表了我们提供全面且具有洞察力地概述将知识蒸馏技术应用于预训练语言模型的领域的努力。尽管该领域的研究非常广泛且迅速发展，特别是在学术界广泛从训练数据中提取知识的情况下，我们承认本文可能不包括所有相关的研究或进展。然而，本文致力于介绍这一快速发展领域的知识。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "foundational paradigms of knowledge dis- tillation, highlighting key methodologies and their impacts across a range of applications. 2.4 Distillation Pipeline in LLM Era SeedKnowledgeSkill/Domain TeacherLLMKnowledgeElicitationStudentModelDistillationAlgorithmsteer driveGeneratedKnowledgeLearningObjectivetrain Fig. 4: An illustration of a general pipeline to distill knowl- edge from a large language model to a student model. The general distillation pipeline of LLMs is a structured and methodical...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "知识提炼的基础范式，重点介绍不同方法及其在各种应用中的影响。LLM时代知识提炼管道示意图。知识提炼管道是一个结构化且方法论化的过程，旨在从高级教师模型（如GPT-4或Gemini）中提炼知识，转移到一个功能更为简化的学生模型。这个管道对于充分利用GPT-4或Gemini等模型的高级能力，使其应用于更加便捷和高效的技术开源模型中至关重要。知识提炼管道大致可分为四个关键阶段，每个阶段都对知识提炼的成功起到关键作用。图4展示了这一管道的概览。详细的管道内容如下所示。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "seen in Figure 2. I. Target Skill or Domain Steering Teacher LLM. The first stage involves directing the teacher LLM towards a specific target skill or domain. This is achieved through care- fully crafted instructions or templates that guide the LLM’s focus. These instructions are designed to elicit responses that demonstrate the LLM’s proficiency in a particular area, be it a specialized domain like healthcare or law, or a skill such as reasoning or language understanding. II. Seed Knowledge as...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "图2中展示的是目标技能或领域引导教师LLM。第一阶段是让教师LLM聚焦于特定的目标技能或领域。这通过精心设计的指令或模板来实现，这些指令或模板引导LLM的注意力。这些指令旨在激发LLM展示其在特定领域的专业能力，无论是医疗、法律这样的专业领域，还是推理、语言理解这样的技能。第二步是提供种子知识给教师LLM。种子知识通常包括一小部分数据集或与目标技能或领域相关的具体数据线索。它作为催化剂，促使教师LLM根据这些初始信息生成更加详尽和详细的输出。种子知识至关重要，因为它为教师模型提供了构建和扩展的基础。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "thereby creating more comprehensive and in-depth knowledge examples. III. Generation of Distillation Knowledge. In response to the seed knowledge and steering instructions, the teacher LLM generates knowledge examples. These examples are predominantly in the form of question-and-answer (QA) dialogues or narrative explanations, aligning with the nat- ural language processing/understanding capabilities of the LLM. In certain specialized cases, the outputs may also in- clude logits or hidden featur...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "生成更全面深入的知识示例。III. 生成蒸馏知识。针对种子知识和引导指令，教师LLM生成知识示例。这些示例主要以问答（QA）对话或叙述解释的形式出现，与LLM的自然语言处理/理解能力相匹配。在某些特定情况下，输出可能会包括 logits 或隐藏特征，尽管这种情况较少见，因为这种数据形式的复杂性和特殊要求所致。生成的知识示例构成了蒸馏知识的核心，包含了教师LLM的高级理解和技能。IV. 根据特定学习目标训练学生模型。最终阶段涉及使用生成的知识示例训练学生模型。训练由与损失函数相匹配的目标引导。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "learning objectives. The loss function quantifies the student model’s performance in replicating or adapting the knowledge from the teacher model. By minimizing this loss, the student model learns to emulate the target skills or domain knowledge of the teacher, thereby acquiring similar capabilities. The process involves iteratively adjusting the student model’s parameters to reduce the discrepancy be- tween its outputs and those of the teacher model, ensuring the effective transfer of knowledge...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "学习目标。损失函数衡量学生模型在再现或适应教师模型中的知识表现。通过最小化这个损失，学生模型学习模拟教师的目标技能或领域知识，从而获得类似的能力。该过程涉及迭代调整学生模型的参数，以减少其输出与教师模型输出之间的差异，确保有效知识转移。本质上，上述四个阶段可以抽象为两个公式。第一个公式表示知识提取过程：D(kd) I={Parse( o, s)|o∼pT(o|I⊕s), ∀s∼ S} (1)，其中⊕表示合并两段文本，I表示任务、技能或领域的指令或模板以引导LLM并提取知识，s∼ S表示种子知识的一个示例，LLM可以基于此探索生成新知识，Parse( o, s) 表示解析浓缩。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "example ( e.g., (x, y)) from the teacher LLM’s output o(plus the input sin some cases), andpTrepresents the teacher LLM with parameters θT. Given the datasets D(kd) Ibuilt for distillation, we then define a learning objective as L=X ILI(D(kd) I;θS), (2) whereP Idenotes there could be multiple tasks or skills being distilled into one student model, LI(·;·)stands for a specific learning objective, and θSparameterizes the student model. Following our exploration of the distillation pipeline and the...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "根据第2.4节的内容，知识蒸馏过程被分为两个主要步骤。“知识”部分专注于从教师LLM中提取知识（公式1），而“蒸馏”部分则专注于将这些知识注入学生模型中（公式2）。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "We will elaborate on these two processes in the subsequent sections. 3.1 Knowledge This section focuses on the approaches to elicit knowledge from teacher LLMs. According to the manners to acquire knowledge, we divided them into Labeling ,Expansion ,Data Curation ,Feature ,Feedback , and Self-Knowledge . Figure 5 shows an illustration of these knowledge elicitation meth- ods. 3.1.1 Labeling Labeling knowledge refers to using a teacher LLM to label the output yfor a given input xas the seed knowl...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在后续部分我们将详细阐述这些过程。3.1 知识\n",
      "本节主要介绍从教师LLM中提取知识的方法。根据获取知识的方式，我们将它们分为标注（Labeling）、扩展（Expansion）、数据整理（Data Curation）、特征提取（Feature）、反馈（Feedback）和自我知识（Self-Knowledge）几类。图5展示了这些知识提取方法的示意图。\n",
      "\n",
      "3.1.1 标注\n",
      "标注知识指的是利用教师LLM对给定输入x的输出y进行标记，作为种子知识，根据指令I或示范c（c=（x1, y1），...，（xn, yn））。从教师LLM中提取知识的方法简单有效，已在多种任务和应用中广泛使用。它仅需要收集输入数据集并将其喂入LLM以获得所需生成内容。此外，通过预先定义的I和c，可以控制y的生成。该过程可以这样形式化表示：\n",
      "D(lab)={x, y|x∼X, y∼Y}...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "y∼pT(y|I⊕c⊕x)}. (3) Input xcould be sourced from existing NLP task datasets, which serve as typical reservoirs for distillation efforts. Numerous works have sought to harness the capa- bilities of powerful LLMs as teachers for annotating dataset samples across a range of tasks. For instance, efforts in natural language understanding involve using LLMs to cat- egorize text (Gilardi et al., 2023; Ding et al., 2023a; He et al., 2023a), while in natural language generation, LLMs assist in generating...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "y∼pT(y|I⊕c⊕x)}. (3) 输入x可以来自现有的NLP任务数据集，这些数据集为知识蒸馏提供了典型的储备。许多工作旨在利用强大的LLM作为标注器，为各种任务标注数据集样本。例如，在自然语言理解任务中，使用LLM对文本进行分类（Gilardi等，2023年；Ding等，2023a年；He等，2023a年），而在自然语言生成任务中，LLM协助生成输出序列（Hsieh等，2023年；Jung等，2023年；Wang等，2021b年）。文本生成评估任务利用LLM对评估结果进行标注（Li等，2024b年；Wang等，2023b年），推理任务则利用LLM标注Chain of Thought（CoT）解释（Hsieh等，2023年；Li等，2022年；Ho等，2023年；Magister等，2023年；Fu等，2023年；Ramnath等，2023年；Li等，2023d年；Liu等，2023g年），等等。而不是集中在特定任务上，许多工作并未如此。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "current works focus on labeling outputs based on instructions, thereby teaching student models to solve tasks in a more flexible way by following in- structions. Collections of various NLP tasks, complemented by instructional templates, serve as valuable input sources forx. For instance, FLAN-v2 collections (Longpre et al., 2023) offers extensive publicly available sets of tasks with instructions, which are labeled with responses generated by teacher LLMs in Orca (Mukherjee et al., 2023; Mitra e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "目前的工作重点是根据指令对输出进行标记，从而教会学生模型通过遵循指令来更灵活地完成任务。各种NLP任务的集合，结合指令模板，为x提供了宝贵输入。例如，FLAN-v2系列（Longpre等人，2023年）提供了大量的任务集合及其指令，这些任务的标签是由强大的LLMs生成的，如Orca（Mukherjee等人，2023年；Mitra等人，2023年）生成的。这些NLP任务的指令来自预定义的模板，缺乏多样性，并且可能与人类自然提问存在差距。人类与聊天模型之间的实际对话提供了大规模数据，其中包含由强大的LLMs标注的真实查询和生成内容，如ShareGPT。此外，Xu等人（2023b）和Anand等人（2023）从论坛如Quora和Stack Overflow中采样的实际问题进行标记。此外，标记过程还可以通过引导进行。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "instructions Ior demonstrations c. A commonly used in- struction type for guiding labeling is chain-of-thought (CoT) prompt (Hsieh et al., 2023; Fu et al., 2023; Magister et al., 2023). Mukherjee et al. (2023) add multiple system messages (e.g. “You must generate a detailed and long answer.” or “explain like I’m five, think step-by-step”) to elicit rich signals. Yue et al. (2023a) and Chenglin et al. (2023) la- bel a hybrid of knowledge of chain-of-thought (CoT) and program-of-thought (PoT) rati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "起始文本中并未包含需要处理或删除的信息。直接提供处理后的文本即可。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "involved. To address these limitations, various expansion methods have been proposed (Wang et al., 2022a; Taori et al., 2023; Chaud- hary, 2023; Si et al., 2023; Ji et al., 2023a; Luo et al., 2023b,a; Wu et al., 2023c; Sun et al., 2024b; Xu et al., 2023a; Guo et al., 2023c; Rozi `ere et al., 2023; West et al., 2022). These methods take the demonstrations as seed knowledge and aim to expand a large scale and various data by in-context learning. A key characteristic of these expansion methods is t...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这些方法利用生成模型的上下文学习能力生成类似提供的示例数据。与标注方法不同，上下文方法中输入 \\(x\\) 和输出 \\(y\\) 都由教师生成模型生成。这个过程可以表示为：\\[D(exp) ={(x, y)|x∼p_T(x|I⊕c), y∼p_T(y|I⊕x)}\\] 其中，上下文 \\(c\\)、输入 \\(I\\) 和上下文 \\(I⊕c\\) 确保生成的数据符合上下文。上下文方法通过上下文学习能力生成新的数据样本，而不仅仅是进行数据标注。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "𝑚Meta-Information𝑐Demonstrations𝑥𝐼 𝑦 FilterFeedback ExtractFeature𝑥𝑦 DistributionIntermediateFeature 𝑥Input𝑦Output𝐼Instruction𝑦! 𝑦\" 𝑦# 𝑥GuideFeedback𝑦#∗ 𝑦# Feedback Self-Knowledge StudentTeacher Generate≻≻𝑦\" 𝑦! 𝑦# 𝑥 𝑥& CorrectExpand𝑐 Fig. 5: An illustration of different knowledge elicitation methods from teacher LLMs. Labeling : The teacher generates the output from the input; Expansion : The teacher generates samples similar to the given demonstrations through in- context learning; Data Curatio...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在不同教师语言模型（LLMs）的方法中，知识提取主要有以下几种方式：\n",
      "\n",
      "1. **Labeling**：教师根据输入生成输出。\n",
      "2. **Expansion**：通过上下文学习，教师生成与给定示范相似的样本。\n",
      "3. **Data Curation**：根据元信息（如主题或实体），教师合成数据。\n",
      "4. **Feedback**：教师提供对学生生成内容的反馈，包括偏好、修正和挑战样本的扩展等。\n",
      "5. **Self-Knowledge**：学生先生成输出，再通过学生自身过滤或评估以获取高质量内容。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "xand yrepresent the new input- output pairs generated by the teacher LLM. The input x is generated based on a set of input-output demonstrations c. The output yis then generated in response to the new input xunder the guidance of an instruction I. Note that the demonstrations could be predefined or dynamically updated by adding the newly generated samples. Expansion techniques have been widely utilized to extract extensive instruction-following knowledge from teacher LLMs. Wang et al. (2022a) fi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "x和y代表根据教师LLM生成的新输入-输出对。输入x基于一组输入-输出演示c生成。输出y在指令I的指导下，根据新的输入x生成。注意到演示可以预先定义或通过添加新生成的样本动态更新。扩展技术已被广泛应用于从教师LLM中提取丰富的指令遵循知识。王等人（2022a）首先引入了迭代式抽样方法，自入教，利用LLM基于几组从175个手动编写指令中抽取的演示来生成多种指令。然后将新生成的指令添加回初始池，以受益于后续扩展迭代。随后，塔里奥等人（2023）将这种方法应用于更强大的教师LLM text-davinci-003，提取了52K高质量数据。为了提升...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "diversity and coverage during expansion, Wu et al. (2023c) and (Sun et al., 2024b) prompt the teacher LLM to generate instructions corresponding to some specific topics. Xu et al. (2023a) propose an Evol-Instruct method to ex- pand the instructions from two dimensions: difficulty (e.g. rewriting the question to be more complex) and diversity (e.g. generating more long-tailed instructions). This Evol- Instruct method is domain-agnostic and has been used to expand the distillation of coding (Luo e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在此扩展现有的任务数据集方面，Wu等人（2023c）和Sun等人（2024b）促使教师LLM生成对应特定主题的指令。Xu等人（2023a）提出了一种Evol-Instruct方法，通过两个维度扩展指令：难度（例如将问题重写得更复杂）和多样性（例如生成更多的末梢指令）。此Evol-Instruct方法是跨领域的，并且已被用于扩展编码（Luo等人，2023a）和数学（Luo等人，2023b）的知识蒸馏。此外，扩展现有的数据集方法可以显著增强类似样本的NLP任务数据集，从而提升任务性能。例如，AugGPT（Dai等人，2023a）利用教师LLM将训练样本中的每个句子重新表述为多个在概念上相似但语义不同的样本，以改善分类性能。类似地，TDG（He等人，2023b）提出了一种目标数据生成（TDG）框架。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "automatically identifies challenging sub- groups within data and generates new samples for these subgroups using LLMs through in-context learning. In summary, the expansion method leverages the in- context learning strengths of LLMs to produce more var- ied and extensive datasets with both inputs and outputs. However, the quality and diversity of the generated data are heavily reliant on the teacher LLMs and the initial seed demonstrations. This dependence can lead to a dataset with inherent bia...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "自动识别数据中的挑战性子群体，并通过上下文学习使用LLMs生成新的样本。总结来说，扩展方法利用LLMs的上下文学习优势，产生更加多样化和扩展性的数据集，既包括输入也包括输出。然而，生成数据的质量和多样性很大程度上依赖于教师的LLMs以及初始种子演示。这种依赖性可能导致数据中固有的偏差（Yu等，2023a；Wei等，2023）以及最终生成内容可能趋于相似的问题，从而限制该方法旨在实现的多样性（Ding等，2023b）。此外，扩展过程可能会无意中放大种子数据中存在的偏差。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "sponse to the limitations observed in both the Labeling and Expansion approaches. These methods often yield data of variable quality and face constraints in quantity. In Labeling, the seed knowledge is sourced from task datasets, leading to potential noise and dirty data. Meanwhile, in Expansion, the input xis derived from seed demonstrations, which can result in homogeneous data when generated in large quantities. To overcome these challenges, the Data Curation method curates high-quality or la...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "针对 Labeling 和 Expansion 方法存在的局限性，这些方法往往产生的数据质量不一且数量有限。在 Labeling 中，种子知识来自任务数据集，可能会引入噪声和脏数据。而在 Expansion 中，由种子演示生成的数据在大量生成时可能变得同质。为克服这些挑战，Data Curation 方法通过深入的元信息来确保高质或大规模数据的质量（Ding 等，2023b；Gunasekar 等，2023；Li 等，2023a；Mar，2023；Liu 等，2023d；Wei 等，2023；Yu 等，2024；Ye 等，2022；Gao 等，2023a；Yang 和 Nicolai，2023）。 Data Curation 的独特之处在于它从零开始合成数据。通过将诸如主题或知识点等多样化的元信息纳入这一过程，可以生成可控的 x 和 y。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "process can be meticulously controlled to yield datasets that are not only large in scale but also of high quality. The formulation for Data Curation can be represented as: D(cur)={(x, y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}.(5) In this formulation, mrepresents the diverse meta- information used to guide the synthesis of x, and Iis the instruction guiding teacher LLMs to generate xory. Different studies primarily vary in their source and method of leveraging meta-information. UltraChat (Ding et al., 2023b)...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "处理可以 meticulously 控制，以产生大规模且高质量的 数据集。数据整理的公式可以表示为：D(cur)={(x, y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}。(5) 在这个公式中，m代表用于指导 x、y 合成的多样化元信息，I是指导教师 LLMs 生成 x 或 y 的指令。不同研究在元信息的来源和利用方法上有所不同。UltraChat（Ding等人，2023b）有效展示了通过知识蒸馏过程，同时整理出高质量且多样化的数据。他们从三个领域收集了广泛的元信息：世界问题、创作与生成以及现有材料的辅助。例如，在世界问题领域，他们探索了诸如“Technology”和“Food and Drink”等 30 个元主题。随后，教师 LLMs 会利用这些元信息提炼出多样化的指令和对话，实现了高质量且多样化的数据整理。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "substantial scale of 1.5 million instances. UltraChat stands out with its lexical and topical diversity. The UltraLLaMA model, fine- tuned on this data, consistently surpasses other open-source models. Another notable series, phi(Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023), focuses on distilling smaller, high-quality datasets akin to ”textbooks.” Phi-1 (Gunasekar et al., 2023) experiments with synthesizing ”textbook qual- ity” data in the coding domain. Their approach involves distillin...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "UltraChat 在实例数量上超过 150 万个。UltraLLaMA 模型通过这个数据集进行微调后，表现始终超越其他开源模型。另一个值得注意的系列，即 phi(Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023)，专注于从较小的质量较高的数据集中提取数据，类似于“教科书”内容。phi-1 (Gunasekar et al., 2023) 系统尝试在编码领域从语言模型中提取具有“教科书质量”的内容。他们的方法通过引导随机主题或函数名称来增强多样性，从 LLM 中提取清晰、完整的、具有指导性的内容。提取的数据包括含有自然语言解释和代码片段的 1 亿个 Python 教科书令牌，以及 1.8 亿个带解决方案的 Python 练习题令牌。令人惊讶的是，尽管模型较小，但在 HumanEval 和 MBPP 等编码基准上 phi-1 模型的表现甚至超过了几乎所有开源模型。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "times smaller in model size and 100 times smaller in dataset size. MFTCoder (Liu et al., 2023d) utilizes hundreds of Python knowledge points as meta-information to create a CodeExercise Dataset. In contrast, Magicoder (Wei et al., 2023) and WaveCoder (Yu et al., 2024) get raw code collections from open-source code datasets, using this as meta-information for generating instructional data. In the context of NLU tasks, certain studies (Ye et al., 2022; Gao et al., 2023a; Wang et al., 2021a) explor...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "模型规模变小了100倍，而数据集规模则缩小了10000倍。MFTCoder（刘等人，2023d）通过数百个Python知识点作为元信息创建了一个CodeExercise数据集。相比之下，Magicoder（韦等人，2023）和WaveCoder（余等人，2024）从开源代码数据集中获取原始代码集合，将其用作元信息来生成教学数据。在自然语言处理（NLU）任务中，某些研究（叶等人，2022；高等人，2023a；王等人，2021a）探索使用标签作为元信息来合成样本，从而实现数据增强。类似地，在信息检索任务中，也有利用文档作为元信息来生成潜在查询，从而构建大规模检索对（邦弗拉西奥等人，2022；孟等人，2023）。总之，通过教师语言模型（LLM）进行数据治理已经成为一种有前景的技术，能够合成不仅高质量多样性的数据集。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "large in scale. The success of models like phi-1 in specialized domains underscores the efficacy of this method. The abilityto create synthetic datasets will become a crucial technical skill and a key area of focus in AI (Li et al., 2023a). 3.1.4 Feature The previously discussed knowledge elicitation methods are typically applied to powerful black-box models, which are expensive and somewhat unreproducible due to calling API. In contrast, white-box distillation offers a more trans- parent and ac...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "大规模的成功表明这种方法的有效性。在特定领域取得成功的模型如phi-1证明了这一点。创建合成数据集将成为AI领域的一项重要技术技能和重点。白-box知识蒸馏提供了一种更透明且易于访问的研究方法。它利用教师大模型的输出分布、中间特征或激活来进行蒸馏，我们将这些知识统称为‘Feature知识’。此前研究主要关注较小的基于编码器的大模型，这些模型通常参数少于1亿（详情见Gou等人（2021））。然而，近期的研究已经开始探索基于生成模型的大模型中的白-box蒸馏方法。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023; Liang et al., 2023a; Gu et al., 2024; Agarwal et al., 2024; Liu et al., 2023a; Wen et al., 2023; Wan et al., 2024a; Zhao and Zhu, 2023; Qin et al., 2023b; Boizard et al., 2024; Zhong et al., 2024). The typical method for acquiring this feature knowledge involves teacher LLMs annotating the output sequence y with its internal representations. These annotations are then distilled into the student model using methods such as Kullback-Leibler Divergence (KLD). The process of eliciting feature ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "文本处理完成如下：\n",
      "\n",
      "2023年，Liang等；2023年，Gu等；2024年，Agarwal等；2023年，Liu等；2023年，Wen等；2024年，Wan等；2023年，Zhao和 Zhu；2023年，Qin等；2024年，Boizard等；2024年，Zhong等）。获取此类特征知识的典型方法是让教师的预训练语言模型（teacher LLMs）在输出序列y的基础上，标注出其内部表示。然后，这些标注通过诸如Kullback-Leibler散度（KLD）等方法进行提炼。特征知识的提取过程可以表示为：D(feat)={(x, y, ϕ feat(x, y;θT))|x∼ X, y∼ Y} 。在这个表达式中，Y是输出集合，可以由教师的预训练语言模型生成，也可以由学生模型生成，或者是直接从数据集中获取的。ϕfeat(·;θT)表示从教师的预训练语言模型中提取特征知识（如输出分布）的操作。最为直接的方法是从固定序列数据集中标记一个带有令牌级概率的固定数据集。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distributions (Sanh et al., 2019; Wen et al., 2023). To leverage the rich semantic and syntactic knowledge in intermediate layers of the teacher model, TED (Liang et al., 2023a) designs task-aware layer-wise distillation. They align the student’s hidden representations with those of the teacher at each layer, selectively extracting knowledge pertinent to the target task. Gu et al. (2024) and Agarwal et al. (2024) introduce a novel approach where the student model first generates sequences, terme...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "多个研究探讨了如何利用教师模型的中间层来提取丰富的语义和语法知识。Sanh等人（2019）和Wen等人（2023）的研究表明了这一点。Liang等人（2023a）提出的TED方法设计了一种任务相关的逐层蒸馏方式。他们令学生模型在每一层的隐藏表示与教师模型对齐，从而有选择地提取与目标任务相关的知识。Gu等人（2024）和Agarwal等人（2024）引入了一种新颖的方法，即让学生模型首先生成序列，称为“自动生成序列”。学生模型通过使用教师模型输出分布的反馈来学习。这种方法尤其适用于学生模型缺乏模仿教师模型分布能力的情况。此外，Tao等人（2022a）、Liu等人（2023a）和Kim等人（2023b）提出了多种基于蒸馏教师模型特征知识的LLM量化方法。这些方法旨在在量化过程中保留原始输出分布。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "quantizing the LLMs, ensuring minimal loss of performance. Additionally, feature knowledge could serve as a potent source for multi-teacher knowledge distil- lation. Timiryasov and Tastet (2023) leverages an ensemble of GPT-2 and LLaMA as teacher models to extract output distributions. Similarly, FuseLLM (Wan et al., 2024a) inno- vatively combines the capabilities of various LLMs through a weighted fusion of their output distributions, integrating them into a singular LLM. This approach has the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "量化语言模型（LLMs）时，确保性能损失最小化。此外，特征知识可以作为多教师知识蒸馏的强大来源。Timiryasov和Tastet（2023）通过将GPT-2和LLaMA的多个模型作为教师模型来提取输出分布。类似地，Wan等人（2024a）通过将各种LLMs的能力通过它们输出分布的加权融合整合到一个单一的LLM中，实现了创新。这种方法有可能显著提升学生模型的能力，超越任何单一教师LLM。总之，使用特征知识提供了一种更透明的替代方法，允许对蒸馏过程有更深的洞察和控制。通过利用教师LLM的特征，如输出分布和中间层特征，白盒方法实现了更丰富的知识转移。尽管展示了潜力，尤其是在...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "smaller models, its application is not suitable for black-box LLMs where internal parame- ters are inaccessible. Furthermore, student models distilled from white-box LLMs may underperform compared to their black-box counterparts, as the black-box teacher LLMs (e.g. GPT-4) tend to be more powerful. 3.1.5 Feedback Most previous works predominantly focus on one-way knowledge transfer from the teacher to the student for imitation, without considering feedback from the teacher on the student’s genera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "对于较小的模型，其应用不适合黑箱语言模型（LLM），在这种模型中，内部参数是不可访问的。此外，来自白箱语言模型的模型从黑箱教师模型（例如GPT-4）中提取时，可能会表现不佳，因为黑箱教师模型往往更强大。3.1.5 反馈 最早的工作大多仅强调从教师到学生的单向知识转移，不考虑从教师对学生的生成进行反馈。教师对学生的生成通常通过提供偏好、评估或纠正信息来提供指导。例如，一种常见的反馈形式是教师对学生的生成进行排名，并通过强化学习自适应反馈（RLAIF）将其偏好整合到学生模型中（Bai等人，2022a）。这是一个用于提取反馈知识的一般化公式： D(fb)={(x, y, ϕ fb(x, y;θT))|x∼ X, y∼pS(y|x)} (7)...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "where ydenotes the output generated by the student model in response to x, and ϕfb(·;θT))represents providing feedback from teacher LLMs. This operation evaluates the student’s output ygiven the input x, by offering assess- ment, corrective information, or other forms of guidance. This feedback knowledge can not only be distilled into the student to also generate feedback (such as creating a student preference model) but, more importantly, enable the student to refine its responses based on the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里的 \\( y \\) 表示对学生模型对 \\( x \\) 的响应生成的输出。\\( \\phi_{fb}(\\cdot;\\theta_{T}) \\) 表示提供来自教师语言模型的反馈。这一操作通过提供评估、纠正信息或其他形式的指导来评估学生的输出 \\( y \\)，给定输入 \\( x \\)。这种反馈知识不仅可以提炼为学生，还可以生成反馈（例如创建学生偏好模型），更重要的是，它能够帮助学生根据反馈来改进他们的回应。已经探索了多种方法来提取这种高级知识（Bai等人，2022a; 洛等人，2023b; 崔等人，2023a; 元等人，2023; 江等人，2023b; 陈等人，2023a; 顾等人，2024; 阿戈瓦尔等人，2024; 陈等人，2024b; 郭等人，2024; 盛等人，2023; 李等人，2023a）。偏好，如之前讨论的，是从教师模型中获得的一种显著的反馈知识形式。可以从各种偏好知识中提炼出不同的知识。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "teachers by prompting it with specific criteria. Bai et al. (2022a) in- troduce RLAIF for distilling harmlessness preferences from LLMs. This involves using an SFT-trained LLM to generate response pairs for each prompt, then ranking them for harmlessness to create a preference dataset. This dataset is distilled into a Preference Model (PM), which then guides the RL training of a more harmless LLM policy. Wizard- Math (Luo et al., 2023b) places emphasis on mathematical reasoning. They employ Chat...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "文本处理完成如下：\n",
      "\n",
      "- Bai et al. (2022a) 提出了一种从 LLM 中提炼危害性偏好（harmlessness preferences）的方法，称为 RLAIF。他们使用经过 SFT（Self-Supervised Fine-Tuning）训练的 LLM 来生成每个提示的响应对，然后根据危害性对这些响应对进行排序，创建一个偏好数据集。这个数据集通过提炼（distillation）转换成一个偏好模型（Preference Model, PM），然后引导更无害的 LLM 政策的 RL（Reinforcement Learning）训练。\n",
      "- 在数学推理方面，Wizard-Math（Luo et al., 2023b）做了强调。他们使用 ChatGPT 作为老师，直接提供过程监督，并评估生成解决方案中的每一步正确性。\n",
      "- 为了扩大高质量提炼的偏好数据集，Cui 等（2023a）开发了一个大规模的偏好数据集，称为 UltraFeedback。他们收集了各种指令和模型，以产生比较数据。之后，GPT-4 被用来从多个方面评估候选者，包括指令遵循、真实性、诚实性。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and helpfulness. Beyond merely assessing student generations, teachers can also furnish extensive feedback on instances where students underperform. In Lion (Jiang et al., 2023b), teacher model pinpoints instructions that pose challenges to the student model, generating new, more difficult instructions aimed at bolstering the student’s abilities. PERsD (Chen et al., 2023a) showcases a method where teacher offers tailored refinement feedback on incorrect code snippets gen- erated by students, gui...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "教师不仅可以评估学生的生成，还可以提供详细的反馈，特别是在学生表现不佳的情况下。例如，Lion (Jiang et al., 2023b) 中，教师模型指出学生模型遇到的挑战性指令，并生成更具挑战性的新指令以提高学生的技能。PERsD (Chen et al., 2023a) 展示了一种方法，即教师根据学生生成的错误代码片段中遇到的具体执行错误提供定制化改进反馈。类似地，SelFee (Ye et al., 2023) 利用ChatGPT生成反馈并根据反馈修订学生的答案。相反地，FIGA (Guo et al., 2024) 是通过将学生的响应与真实响应进行比较来修订学生的响应。此外，教师模型对学生生成的分布本身也可以作为一种反馈形式。MiniLLM (Gu et al., 2024) 和GKD (Agarwal et al., 2024) 提出了一种创新策略。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "wherein the student model initially generates sequences, followed by teacher model producing an output distribution as feedback. This method leverages the teacher’s insight to directly inform and refine the student model’s learning process. 3.1.6 Self-Knowledge The knowledge could also be elicited from the student itself, which we refer to as Self-Knowledge . In this setting, the same model acts both as the teacher and the student, iteratively improving itself by distilling and refining its own ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在该方法中，学生模型最初生成序列，随后教师模型生成一个输出分布作为反馈。这种方法利用教师的洞察直接指导和优化学生模型的学习过程。3.1.6 自我知识 该知识也可以从学生自身提取，我们将其称为自我知识。在这种设置下，同一个模型既是教师也是学生，通过提炼和优化自己之前生成的输出，迭代地自我改进。这种知识可以绕过需要外部的、可能具有专有性质的强大教师模型，如GPT系列的大型语言模型（LLMs）。此外，它允许模型超越传统教师-学生的局限性或“天花板”。自我知识可以表示为： \\(D(sk) = \\{(x, y, ϕ_{sk}(x, y))|x \\sim S, y \\sim p_S(y|I \\oplus x)}\\), 式中，\\(ϕ_{sk}(\\cdot)\\) 是一个表示额外过程的通用函数，用于自动生成输出 \\(y\\)。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "which could include but is not limited to filtering, rewarding, or any other mechanisms for enhancing or evaluating y. It could be governed by external tools or the student itself θS. Recent research in this area has proposed various innovative methodologies to elicit self-knowledge, demonstrating its potential for creating more efficient and autonomous learn- ing systems. (Allen-Zhu and Li, 2020; Wang et al., 2022a; Sun et al., 2024b; Yang et al., 2024; Jung et al., 2023; Huang et al., 2023a; G...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "该机制可能包括但不限于过滤、奖励或其他方法来提升或评估 y。它可能由外部工具或学生自身 θS 管控。近年来，该领域的研究提出了多种创新方法来激发自我认知，展示了其创建更高效、自主学习系统的潜力。（Allen-Zhu 和 Li, 2020; Wang 等人, 2022a; Sun 等人, 2024b; Yang 等人, 2024; Jung 等人, 2023; Huang 等人, 2023a; Gulcehre 等人, 2023; Yuan 等人, 2024a; Xu 等人, 2023b; Zelikman 等人, 2022; Chen 等人, 2024a; Zheng 等人, 2024; Li 等人, 2024c; Zhao 等人, 2024; Singh 等人, 2023; Chen 等人, 2024c; Hosseini 等人, 2024）一个例子就是 Self-Instruct（Wang 等人, 2022a），它利用 GPT-3 通过扩展方法进行数据增强，生成额外的数据样本以增强数据集。随后，该数据集会微调原始模型。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "model. Other methods aim to elicit targeted knowledge 11 from student models by modifying prompts, and leveraging these data for further refinement. In Self-Align (Sun et al., 2024b), they find that models fine-tuned by Self-Instruct data tend to generate short or indirect responses. They prompt this model with verbose instruction to produce in- depth and detailed responses. Then, they employ context- distillation (Askell et al., 2021) to distill these responses paired with non-verbose instructi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "一些方法通过修改提示来诱使其生成目标知识。例如，在 Self-Align (Sun et al., 2024b) 中，他们发现使用 Self-Instruct 数据微调的模型倾向于生成简短或间接的回复。他们给这个模型以详尽的指令来生成深入且详细的回复。然后，他们使用上下文蒸馏（Askell et al., 2021）将这些回复与非详尽的指令配对，重新蒸馏回模型。类似地，RLCD (Yang et al., 2024) 引入对比提示来从不一致的 LLM 中生成偏好对，涵盖优劣示例。一个基于这些对训练的偏好模型则通过强化学习引导不一致模型的提升。其他一些方法通过过滤方法来精制自动生成的数据。例如，Impossible Distillation (Jung et al., 2023) 目标是句子。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "summarization tasks, implementing filters based on entailment, length, and diversity to screen self-generated summaries. LMSI (Huang et al., 2023a) generates multiple CoT reasoning paths and answers for each question, and then retains only those paths that lead to the most consistent answer. Note that refined self-knowledge can be iteratively ac- quired as the student model continuously improves, further enhancing the student’s capabilities. This is Gulcehre et al. (2023) introduces a Reinforced...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "处理后的文本：\n",
      "处理后的文本：总结任务主要通过基于 entailment、长度和多样性的过滤器来筛选自动生成的总结。LMSI（黄等人，2023a）为每个问题生成多条 CoT（Continue to Think）推理路径和答案，然后仅保留那些路径中能够得出最一致答案的路径。需要注意的是，随着学生模型的持续改进，可以迭代地获得精进的知识。这类似于 Gulcehre等人（2023）介绍的强化自训练（ReST）框架，它通过交替进行Grow和Improve阶段来逐步获得更好的自知识，并进一步提升学生的技能。在Grow阶段，学生模型生成多个输出预测；随后在Improve阶段，通过使用评分函数对这些自生成输出进行排序和过滤。最后，语言模型会在经过精心筛选的这个数据集上进行微调，采用离线强化学习目标。自玩游戏（陈等人，2023）则展示了这一过程。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2024a) introduces a framework resembling iterative DPO, where the language model is fine-tuned to differentiate the self-generated responses from the human-annotated data. These self-generated responses could be seen as “negative knowledge” to promote the student to better align with the target distribution. Self-Rewarding (Yuan et al., 2024a) explores a novel and promising approach by utilizing the language model itself as a reward model. It employs LLM- as-a-Judge prompting to autonomously ass...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2024年的一项框架，类似于迭代DPO，其中语言模型被微调以区分自动生成的响应和人工标注数据。这些自动生成的响应可被视为“负面知识”，以帮助学生更好地与目标分布对齐。自奖励（Yuan等人，2024年）通过利用语言模型本身作为奖励模型，探索了一种新颖且有前途的方法。它使用LLM作为裁判提示，自主为自动生成的响应分配奖励。整个过程可以迭代进行，从而提升指令遵循和奖励建模能力。3.2 学徒化 本节探讨了有效从教师LLM中转移提取知识的方法，将其转移到学生模型中。我们探索了一系列的学徒化技术，包括受监督微调策略、差异与相似性策略，以及先进的方法，如强化学习。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and Rank Optimization , as shown in Figure 3. 3.2.1 Supervised Fine-Tuning Supervised Fine-Tuning (SFT), or called Sequence-Level KD (SeqKD) (Kim and Rush, 2016), is the simplest and one of the most effective methods for distilling powerful black-boxDivergence Type D(p, q)Function Forward KLDPp(t) logp(t) q(t) Reverse KLDPq(t) logq(t) p(t) JS Divergence1 2\u0010Pp(t) log2p(t) p(t)+q(t)+Pq(t) log2q(t) p(t)+q(t)\u0011 TABLE 1: Functional forms of Dfor various divergence types. p: reference Similarity Functi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在监督微调（SFT）或称为序列级别知识蒸馏（SeqKD）（Kim和Rush，2016）方法中，学生模型通过最大化生成的序列似然性来与教师模型的预测对齐。这种过程在数学上可以表示为最大化学生模型生成序列的概率。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "formulated as minimizing the objective function: LSFT=Ex∼X,y∼pT(y|x)[−logpS(y|x)], (9) where yis the output sequence produced by the teacher model. This simple yet highly effective technique forms the basis of numerous studies in the field. Numerous re- searchers have successfully employed SFT to train student models using sequences generated by teacher LLMs (Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Luo et al., 2023b). Additionally, SFT has been ex- plored in ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "形式化为最小化目标函数: LSFT=Ex∼X,y∼pT(y|x)[−logpS(y|x)]，(9) 其中y是教师模型生成的输出序列产生的序列。这种简单而高效的技巧构成了领域内众多研究的基础。许多研究人员成功地使用SFT训练学生模型，利用由教师LLM生成的序列（Taori等人，2023；Chiang等人，2023；Wu等人，2023c；Xu等人，2023a；Luo等人，2023b）。此外，SFT已在许多自编码工作中被探索（Wang等人，2022a；Huang等人，2023c；Xu等人，2023b；Zelikman等人，2022）。由于大量使用SFT的迁移学习工作，这里仅列出代表性的例子。更多详细工作可以在第4.3.2.2节中找到。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "groups: those minimizing divergence in probability distributions and those aimed at enhancing the similarity of hidden states. Divergence. Divergence-based methods minimize diver- gence between the probability distributions of the teacher and student models, represented by a general divergence function D: LDiv= E x∼X,y∼Y[D(pT(y|x), pS(y|x))], (10) The specific form of Dvaries depending on the type of divergence employed. Table 1 outlines the functional forms ofDfor different divergence measures....\n",
      "\n",
      "PROCESSED TEXT:\n",
      "groups 包括那些致力于减少概率分布差异的群体和那些旨在增强隐藏状态相似性的群体。减少差异。基于差异的方法通过最小化教师模型和学生模型概率分布之间的差异来减少差异，差异函数为 D: LDiv= E x∼X,y∼Y[D(pT(y|x), pS(y|x))], (10) 差异函数 D 的具体形式取决于所使用的差异类型。表 1 列出了不同差异测量方法下 D 的功能形式。常用的 KL 效果本质上是在近似目标分布时最小化前向 KL 散度 (Sanh 等人，2019; 12 篇论文)。KL 散度最小化 p||q 的 argminqKL(argsminqKL(q||p)) Fig. 6: 目标分布逼近中前向和反向 KL 散度的比较。前向 KL 散度方法倾向于覆盖目标分布的所有模式，但精度较低，即“模式覆盖”行为。反向 KL 散度方法侧重于专注于。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "predom- inantly on the most prominent mode, thereby exhibiting a “mode-seeking” behavior. Wen et al., 2023; Timiryasov and Tastet, 2023; Liang et al., 2023a; Chen et al., 2024d) , which forces pSto cover all the modes of pT. However, when a student model is unable to learn all modes of a highly complex teacher, the re- sultant “mode-covering” behavior might cause the student to assign probability mass to tokens with low probability under the teacher’s distribution (cf. Figure 6 blue curve). This...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "论文中提到，通过迫使学生模型覆盖所有教师模型的模式（predominantly on the most prominent mode），学生模型表现出“模式寻求”行为（Wen et al., 2023; Timiryasov and Tastet, 2023; Liang et al., 2023a; Chen et al., 2024d）。这种行为迫使学生模型覆盖所有教师模型的模式，但当学生模型无法学习高复杂度教师模型的所有模式时，可能会导致学生模型在教师的分布下给概率较低的令牌分配较大的概率质量。这可能导致潜在的幻觉和低质量生成。或者，通过使用反KL散度（reverse KL divergence），可以优先考虑教师模型中分配高概率的令牌，从而减轻低质量输出的风险，但也可能牺牲多样性。Gu等（2024）采用反KL散度来防止学生模型过度估计。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "low-probability regions of the teacher’s distribution, employing Policy Gradient methods for optimization. Both Agarwal et al. (2024) and Sason and Verd ´u (2016) assess the effect of different divergence func- tions in LLM distillation, finding the optimal divergence to be task-dependent. For instance, forward KL divergence is more suitable for tasks like Machine Translation, where the output has fewer modes or variations, while reverse KL divergence is preferable for tasks like dialogue genera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Policy Gradient方法被用于优化低概率区域教师分布。Agarwal等人（2024）和Sason及Verdú（2016）评估了不同差异函数在LLM迁移学习中的效果，发现最优差异函数取决于任务性质。例如，在机器翻译任务中，前向KL差异更适合，因为输出模式较少。而在对话生成和指令调优任务中，后向KL差异效果更好，因为这些任务涉及多个模式和更广泛的可能性响应。因此，任务性质显著影响了差异函数的选择，以实现最佳性能。类似地，知识迁移学习中的相似性方法旨在使学生模型的隐藏状态或特征与教师模型的隐藏状态或特征对齐。这些方法使用各种相似性指标来测量并优化内部表示之间的相关性。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "two models. The objective is to ensure that the student model not only produces similar outputs to the teacher but also processes information in a comparable manner. The formulation for a similarity-based objective might look like this: LSim= E x∼X,y∼Y[LF(ΦT(fT(x, y)),ΦS(fS(x, y)))],(11) where fT(x, y)andfS(x, y)are the feature maps of the teacher and student models, respectively. The transforma-tion functions ΦTandΦSare applied to these feature maps to ensure they are in the same shape, facilit...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "为了确保学生模型不仅在输出上与教师模型相似，还在处理信息的方式上也相似，我们使用如下公式来定义相似性目标：\\[LSim=E_{x∼X,y∼Y}[LF(\\Phi_T(f_T(x, y)),\\Phi_S(f_S(x, y)))],\\]其中，\\(f_T(x, y)\\)和\\(f_S(x, y)\\)分别是教师模型和学生模型的特征图。通过应用变换函数\\(\\Phi_T\\)和\\(\\Phi_S\\)使得这些特征图具有相同的形状，便于直接比较。相似函数\\(LF\\)用于匹配这些变换后的特征图。表2展示了\\(LF\\)的常见选择。目前，较少的工作在大型语言模型（LLMs）的知识蒸馏（KD）中使用基于相似性的方法。其中，Liang等人（2023a）提出了一种任务意识的层次式蒸馏（TED），该方法利用任务意识的滤波器。这些滤波器被设计为从教师模型中选择性地捕捉特定任务最相关的信息。核心目标是最小化这些滤波器的差异。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "discrepancy between the filtered representations in both teacher and student models. While similarity-based approaches are common in encoder-based LMs (Sun et al., 2019, 2020; Jiao et al., 2020; Hou et al., 2020; Zuo et al., 2022; Liang et al., 2021), their application in LLM knowledge distillation is not as widespread. However, considering their effectiveness, we anticipate an increase in research exploring these methods for LLM distillation in the near future. 3.2.3 Reinforcement Learning This...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "关于知识蒸馏，基于增强学习的方法在编码器型语言模型（Sun et al., 2019, 2020; Jiao et al., 2020; Hou et al., 2020; Zuo et al., 2022; Liang et al., 2021）中已有广泛应用，但在语言模型（LLM）的知识蒸馏中则较少见。然而，考虑到其有效性，我们预计未来的研究将探索这些方法在知识蒸馏中的应用。3.2.3 增强学习 本节探讨了利用增强学习（RL）方法将知识注入学生模型的高级方法。这种方法尤其适用于利用教师模型的反馈来训练学生模型（Bai et al., 2022a; Cui et al., 2023a; Luo et al., 2023b; Agarwal et al., 2024; Chen et al., 2024b; Ma et al., 2023a; Pang et al., 2023; Du et al., 2023a）。基于增强学习的知识蒸馏过程通常包括两个主要阶段：蒸馏奖励模型训练。第一阶段...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "involves training a reward model rϕusing the feedback data D(fd) generated by teacher LLMs. Preference data, as one of the typical feedback, is employed to train the student reward model (Bai et al., 2022a; Cui et al., 2023a; Lee et al., 2023a; Kim et al., 2023a). They usually consist of input-output pairs (x, yw, yl). Here, ywandylrepresent “winning” and “losing” outputs relative to the teacher’s preferences. The loss function for the reward model is defined as: LRM(rϕ,D(fd)) =− E (x,yw,yl)∼D(f...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "本文涉及训练奖励模型 \\( \\varphi \\)，使用教师语言模型生成的反馈数据 \\( D(f_d) \\) 来进行训练。偏好数据作为典型的反馈方式被用于训练学生奖励模型（Bai等人，2022a；Cui等人，2023a；Lee等人，2023a；Kim等人，2023a）。它们通常由输入输出对（\\( x, y_w, y_l \\)）组成，其中 \\( y_w \\) 和 \\( y_l \\) 分别表示相对于教师偏好而言的“赢得”和“输掉”输出。奖励模型的损失函数被定义为： \\[\n",
      "LRM(\\varphi, D(f_d)) = - \\mathbb{E}_{(x, y_w, y_l) \\sim D(f_d)}[\\log \\sigma(\\varphi(x, y_w) - \\varphi(x, y_l))] \\quad (12)\n",
      "\\] 这种形式指导奖励模型根据教师的标准正确区分更优和劣质的输出。RLMEC（Chen等人，2024b）则采用不同的方法，通过训练一个生成奖励模型来代替学习实例级别的奖励。它从教师语言模型中提取错误解决方案重写数据，并进行训练以产生词级奖励，用于强化学习。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Learning Optimization. In the second stage, the student model, represented by a policy πθ, is optimized to maximize the expected reward as per the trained reward model. Simultaneously, it minimizes the divergence from a reference policy πref, typically the initial policy of the student model trained by SFT, controlled by a factor β. The RL objective is given by: 13 max πθE x∼X,y∼πθ(y|x)[rϕ(x, y)]−βDKL[πθ(y|x)∥πref(y|x)] (13) This RL framework not only ensures that the student model learns the ex...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在第二阶段，学生模型通过策略πθ进行优化，以最大化期望奖励，并同时最小化与参考策略πref的差异，通常是指学生模型的初始策略通过SFT训练得到的策略。这个差异由因子β控制。RL目标表达为：最大化下式中的期望值：\\[ \\max_{\\pi_{\\theta}} E_{x \\sim X, y \\sim \\pi_{\\theta}(y|x)}[r_{\\phi}(x, y)] - \\beta \\text{DKL}[\\pi_{\\theta}(y|x) \\| \\pi_{ref}(y|x)] \\] 式中，\\(r_{\\phi}(x, y)\\)是奖励模型得到的奖励。当使用PPO（Schulman et al., 2017）算法时，RL提供了一个有效机制，使得学生模型的输出能够与教师保持一致。此外，教师的LLM也可以作为奖励模型直接在RL过程中分配奖励，从而避免训练奖励模型的需要（Lee et al., 2023a; Kwon et al., 2023）。这种做法可以显著提升效果。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "performance, it comes at a higher computational cost compared to employing a smaller distilled reward model. 3.2.4 Ranking Optimization Ranking optimization presents a stable and computationally efficient alternative to RL for injecting preference feedback into language models (Rafailov et al., 2023; Song et al., 2023a; Yuan et al., 2023b). This method, diverging from traditional RL approaches, directly incorporates ranking information into language models from a fixed preference dataset during ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在进行排名优化时，相比于使用较小的精馏奖励模型，这会带来更高的计算成本。3.2.4 排序优化 排序优化为在语言模型中注入偏好反馈提供了一个稳定且计算效率高的替代方案（Rafailov 等人，2023；Song 等人，2023a；Yuan 等人，2023b）。这种方法与传统的 RL 方法不同，它在微调期间直接将排名信息纳入语言模型中。直观上，它直接更新策略以增加偏好响应的相对可能性。无需进行输出采样，这种直接优化偏好使过程更加稳定且高效。最近，有一些工作探索使用排序优化将教师的偏好转移到学生模型中（Tunstall 等人，2023；Hong 等人，2023；Yuan 等人，2024a）。Zephyr（Tunstall 等人，2023）使用直接排序优化。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Preference Optimization (DPO) (Rafailov et al., 2023) to distill the preference alignment in teacher LLMs. DPO streamlines the objective of reinforcement learning (as in Eq. 13), which involves reward maximization with a KL-divergence constraint, into a single-stage policy training. Specifically, DPO’s training goal is to maximize the following expecta- tion: E (x,yw,yl)∼D(fd)\u0014 logσ\u0012 βlogπθ(yw|x) πref(yw|x)−βlogπθ(yl|x) πref(yl|x)\u0013\u0015 , (14) where ywis preferred over ylaccording to the teacher LLM...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Preference Optimization (DPO) (Rafailov et al., 2023)用于提取教师语言模型的偏好一致性。DPO将强化学习（如公式13所示）中基于奖励最大化的KL散度约束的目标简化为单阶段策略训练。具体来说，DPO的目标是最大化以下期望值： $$E_{x,y,w,l}\\sim D(f_d) \\log \\sigma^{\\beta \\log \\pi^{\\theta}(y_w|x) \\pi_{ref}(y_w|x) - \\beta \\log \\pi^{\\theta}(y_l|x) \\pi_{ref}(y_l|x)}$$ 其中，\\(yw\\) 按照教师语言模型的偏好优于 \\(yl\\)。Hong等（2023）（Hong et al., 2023）采用两种基于排名的优化目标，即Align Human Feedback通过响应排名优化（RRHF）（Yuan et al., 2023b）和偏好排名优化（PRO）（Song et al., 2023a），用于偏好提取。RRHF（Yuan et al., 2023b）定义了一个排名损失，公式为： $$L_{RRHF} = \\...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "probabilities under the policy πθ. This approach emphasizes direct comparison and ranking of responses based on the teacher’s preferences. PRO (Song et al., 2023a) expands the concept of pairwisecomparison to handle preference rankings of any length. For a given instruction xand a sequence of responses ordered by teacher preference as y1≻y2≻...≻yn, the RPO training objective is: LPRO=−n−1X k=1logexp (pk)Pn i=kexp (pi), (16) where pkrepresents the conditional log probabilities for ykunder the stu...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "在基于策略πθ的情况下，此方法强调直接基于教师偏好比较和排序响应。PRO（Song等人，2023a）扩展了成对比较的概念，以处理任意长度的偏好排序。给定指令x和按照教师偏好顺序排列的响应序列y1≻y2≻...≻yn，RPO训练目标为： \\( L_{PRO} = -\\sum_{k=1}^{n-1} \\log \\left( \\frac{\\exp(p_k)}{\\sum_{i=k}^{n} \\exp(p_i)} \\right) \\) （16） 式中，\\(p_k\\) 表示在学生策略πθ下，y_k的条件Log概率。通过迭代对比生成响应的似然性，PRO优化学生LM，使其优先考虑最被青睐的响应，并逐步按偏好递减顺序排名其余响应。4 S KILL DISTILLATION 基于第3节关于获取知识和蒸馏算法的概述，我们将注意力转移到如何利用这些技术在大型语言模型（LLMs）中提炼特定技能。我们的探索将涵盖多种方法。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "range of skills exhibited by LLMs, including Context Following ,Alignment ,Agent ,NLP Task Specializa- tion and Multi-Modality .Context Following focuses on the student’s ability to comprehend and respond effectively to input information. Alignment delves into the student’s capability to align its output with the teacher’s responses. Moving forward, Agent underscores the autonomous nature of language models. NLP Task Specialization highlights the LLM’s versatility in specializing across various ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "LMs表现出的技能范围包括：Context Following（理解并有效回应输入信息的能力）、Alignment（确保输出与教师响应一致的能力）、Agent（自主性质）、NLP Task Specialization（在自然语言处理任务上的专精能力）以及Multi-Modality（从教师LMs中转移知识到多模态模型）。Context Following着重于学生的理解和回应输入信息的能力。Alignment则关注学生的输出能够与教师的响应保持一致。接下来，Agent强调了LMs的自主性质。NLP Task Specialization展示了LMs在多种自然语言处理任务上的适应性。最后，Multi-Modality涵盖了从教师LMs中转移知识到多模态模型的知识。表3总结了代表性的作品，包含了涉及的技能、种子知识、教师LMs、学生模型、知识提取方法和训练目标。4.1部分专注于从LMs中提取Context Following的能力。这一过程包括。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "transferring the ability of LLMs to handle a variety of complex contexts — such as few-shot demonstrations, intricate instructions, dia- logue history, and retrieval-augmented information — into smaller models. Many research efforts in this domain aim to imbue smaller models with these sophisticated, context- following capabilities. Our discussion here will dissect this facet of skill distillation, categorizing it based on different types of context and elaborating on how each is distilled and i...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "通过将LLMs的处理复杂情境的能力——例如少量示范、复杂的指示、对话历史和检索增强信息——转移到较小的模型中，从而将处理复杂情境的能力转移到较小的模型中。该领域中的许多研究努力旨在赋予较小模型这些复杂的、基于情境的能力。我们在此讨论将这些能力转移到较小模型中的方面，根据不同的情境类型进行分类，并详细说明如何对其进行提炼和融入。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "can be manually curated by human experts or transformed from existing NLP tasks into instructional 14 Methods Skill Seed Knowledge Teacher LLM Student Model Knowledge Elicitation Objective Context Following Self-Instruct (Wang et al., 2022a) IF 175 human-curated tasks GPT3 GPT3 Expansion + Self-Knowledge SFT Alpaca (Taori et al., 2023) IF 175 human-curated tasks GPT3 LLaMA Expansion + Self-Knowledge SFT LaMini-LM (Wu et al., 2023c) IF3.5K Wikipedia Categories + Mixed DatasetChatGPT Various Model...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "以下是处理后的文本内容：\n",
      "\n",
      "- 这里罗列了多种方法来实现知识提取，包括自监督学习（SFT）和自知学习（Self-Knowledge）等方法。\n",
      "- 主要包括了几项研究和项目，如：\n",
      "  - Wang等人（2022a）的14种方法。\n",
      "  - Taori等人（2023）的175种自定义任务。\n",
      "  - Wu等人（2023c）的3.5K类别和混合数据集。\n",
      "  - Jiang等人（2023b）的10M单词和小型LLaMA。\n",
      "  - Sun等人（2024b）的人工编写原则和小型LLaMA。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Self-Rewarding (Yuan et al., 2024a) IF Human-written Samples LLaMA LLaMA Self-Knowledge SFT + RL STaR (Zelikman et al., 2022) IF Arithmetic + CommonsenseQA + GSM8K GPT-J GPT-J Self-Knowledge SFT Llama-GPT4 (Peng et al., 2023a) IF Alpaca Dataset GPT4 LLaMA Labeling SFT Reflection-Tuning (Li et al., 2023e) IF Alpaca/WizardLM Dataset ChatGPT LLaMA Labeling SFT Selective Reflection-Tuning (Li et al., 2024d) IF Alpaca/WizardLM Dataset ChatGPT LLaMA Labeling SFT Vicuna (Chiang et al., 2023) IF/MD Huma...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这些研究的列表涵盖了多种方法和技术，用于改进语言模型的自我奖励学习。以下是整理后的关键信息：\n",
      "\n",
      "1. **Self-Rewarding (Yuan et al., 2024a)**\n",
      "   - 使用人类编写样本进行训练。\n",
      "\n",
      "2. **Self-Knowledge SFT + RL (Zelikman et al., 2022)**\n",
      "   - 结合算术、常识问答和GSM8K数据集进行自我增强学习。\n",
      "\n",
      "3. **GPT-J GPT-J Self-Knowledge SFT**\n",
      "   - 使用GPT-J进行自我知识增强学习。\n",
      "\n",
      "4. **Alpaca Dataset GPT4 LLaMA Labeling SFT**\n",
      "   - 使用Alpaca数据集和GPT4进行标记学习。\n",
      "\n",
      "5. **Selective Reflection-Tuning (Li et al., 2024d)**\n",
      "   - 使用Alpaca/WizardLM数据集进行选择性回射调优。\n",
      "\n",
      "6. **Koala (Geng et al., 2023)**\n",
      "   - 使用人类对话数据进行标记学习。\n",
      "\n",
      "7. **Baize (Xu ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "et al., 2023) IF/TP Human Conv, Flan/Code/Math Collection ChatGPT LLaMA Labeling SFT CoT-Distill (Hsieh et al., 2023) IF/TP e-SNLI + ANLI + CQA + SVAMP PaLM T5 Labeling SFT KnowPAT (Zhang et al., 2023a) IF/TP CPKG + QA Data ChatGPT + ChatGLM + Vicuna-7B LLaMA Labeling SFT DEBATunE (Li et al., 2024e) IF/TP Controversial Topics ChatGPT LLaMA Labeling SFT Phi-1 (Gunasekar et al., 2023) IF/Code - GPT3.5 phi-1 Curation SFT Phi-1.5 (Li et al., 2023a) IF/Code 20k Topics from Web GPT3.5 phi-1 Curation +...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "处理后的文本：\n",
      "Hsieh et al., 2023 人类对话模型 ChatGPT LLaMA 标注 SFT\n",
      "Zhang et al., 2023a QA 数据 ChatGPT 聊天GLM 聊天模型 Vicuna-7B LLaMA 标注 SFT\n",
      "Li et al., 2024e 争议性话题聊天GPT LLaMA 标注 SFT\n",
      "Gunasekar et al., 2023 GPT3.5 phi-1 编集 SFT phi-1\n",
      "Li et al., 2023a GPT3.5 phi-1 编集+标注 SFT\n",
      "Luo et al., 2023c 阿尔帕卡数据GPT4 LLaMA 标注 SFT\n",
      "Kang et al., 2023b 开放指令GPT4 LLaMA 标注 SFT\n",
      "Wang et al., 2023c 偏好对话模型GPT4 LLaMA 标注 SFT\n",
      "Tunstall et al., 2023 多重数据集GPT4 Mistral 标注 SFT...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Human-written Prompts LLaMA LLaMA Expansion + Labeling SFT + RL RLCD (Yang et al., 2024) IF/Preference Human-written Prompts LLaMA LLaMA Labeling SFT + RL RLAIF (Lee et al., 2023a) IF/Preference Human-written Prompts PaLM 2 PaLM 2 Labeling + Feedback RL GPT3 Reward (Kwon et al., 2023) Preference Human-written Prompts GPT3 GPT3 Labeling RL ILF (Scheurer et al., 2023) Preference Task-specific Datasets GPT3 + FeedME GPT3 Labeling RL ULTRAFEEDBACK (Cui et al., 2023a) Preference Mixed Datasets GPT4 L...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "处理后的文本如下：\n",
      "\n",
      "- LLaMA Expansion + Labeling SFT + RL RLCD (Yang et al., 2024)\n",
      "- PaLM 2 Labeling + Feedback RL GPT3 Reward (Kwon et al., 2023)\n",
      "- GPT3 Labeling RL ILF (Scheurer et al., 2023)\n",
      "- LLaMA Labeling RL Constitutional AI (Bai et al., 2022a)\n",
      "- GPT4 LLaMA Labeling RL SANDBOX (Liu et al., 2023b)\n",
      "- ChatGPTLLaMA Data Curation SFT + RL Agent Toolformer (Schick et al., 2023)\n",
      "- ChatGPT GPT-J + LLaMA Labeling SFT Gorilla (Patil et al., 2023)...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "API Documentation GPT4 LLaMA Expansion SFT GPT4Tools (Yang et al., 2023b) Tool Image Content ChatGPT LLaMA Curation + Expansion SFT ToolAlpaca (Tang et al., 2023a) Tool Public-apis Repository ChatGPT LLaMA Curation SFT ToolLLM (Qin et al., 2023a) Tool Real-world APIs ChatGPT LLaMA Curation SFT MLLM-Tool (Wang et al., 2024) Tool HuggingFace Model Cards GPT4 LLaMA Curation SFT FireAct (Chen et al., 2023b) Planning Mixed QA Dataset GPT4 LLaMA Labeling SFT AgentTuning (Zeng et al., 2023a) Planning 6...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里的文本主要是关于不同工具和方法的介绍，看起来是一份API文档或工具介绍的汇总。以下是处理后的文本内容：\n",
      "\n",
      "- API Documentation GPT4 LLaMA Expansion SFT\n",
      "- GPT4Tools (Yang et al., 2023b) Tool\n",
      "- Image Content ChatGPT LLaMA Curation + Expansion SFT Tool\n",
      "- Alpaca (Tang et al., 2023a) Tool\n",
      "- Public-apis Repository\n",
      "- ChatGPT LLaMA Curation SFT Tool\n",
      "- LLM (Qin et al., 2023a) Tool\n",
      "- Real-world APIs\n",
      "- ChatGPT LLaMA Labeling SFT MLLM-Tool (Wang et al., 2024) Tool\n",
      "- HuggingFace Model Cards\n",
      "- GPT4 LLaMA Curation SFT\n",
      "- FireAct (Chen et al., 2023b) Planning\n",
      "- M...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2021a) NLU NLU Tasks GPT3 BERT Expansion SFT InheritSumm (Xu et al., 2023c) NLG Pile + ArXiv + CNN/DM + WikiHow GPT3.5 ZCode++ Label SFT DIMSUM+ (Jung et al., 2023) NLG None GPT2 + CTRL + BioGPT T5 Curation + Self-Knowledge SFT Genie (Yehudai et al., 2024) NLG ELI5 + ASQA + NQ + CNN/DM Falcon + LLaMA FLAN + LLaMA Label SFT GKD (Agarwal et al., 2024) NLG/NLU/IF XSum+WMT14 en-de+GSM8K+FLAN2021 T5-XL T5 Feature + Feedback D&S + RL QUILL (Srinivasan et al., 2022) IR IR Datasets T5 4-layer Trans...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里是文本处理后的版本：\n",
      "\n",
      "al., 2021a) NLU NLU Tasks GPT3 BERT Expansion SFT InheritSumm (Xu et al., 2023c) NLG Pile + ArXiv + CNN/DM + WikiHow GPT3.5 ZCode++ Label SFT DIMSUM+ (Jung et al., 2023) NLG None GPT2 + CTRL + BioGPT T5 Curation + Self-Knowledge SFT Genie (Yehudai et al., 2024) ELI5 + ASQA + NQ + CNN/DM Falcon + LLaMA FLAN + LLaMA Label SFT GKD (Agarwal et al., 2024) NLG/NLU/IF XSum+WMT14 en-de+GSM8K+FLAN2021 T5-XL T5 Feature + Feedback D&S + RL QUILL (Srinivasan et al., 2022) IR IR Datasets T5 Ran...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "PandaLM (Wang et al., 2023b) Evaluation Alpaca Data ChatGPT LLaMA Labeling SFT Prometheus (Kim et al., 2024) Evaluation 50 Seed Rubrics GPT4 LLaMA Labeling SFT InstructScore (Xu et al., 2023d) Evaluation Mixed Dataset GPT4 LLaMA Labeling SFT WizardMath (Luo et al., 2023b) Math GSM8k + MATH ChatGPT LLaMA Expansion + Feedback SFT + RL Mammoth (Yue et al., 2023a) Math/TP Mixed Math Dataset GPT4 LLaMA Labeling SFT Mixed Distill (Chenglin et al., 2023) Math/TP SVAMP + GSM8K + ASDIV + StrategyQA ChatG...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Clean (Jain et al., 2023) Code Human-written Instructions LLaMA LLaMA Expansion + Self-Knowledge SFT Code...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023) Code Code Datasets ChatGPT LLaMA Labeling SFT Multi-Modality LLaVA (Liu et al., 2023e) Vision-Language COCO GPT4 LLaMA Labeling SFT SVIT (Zhao et al., 2023b) Vision-Language Visual Genome + COCO GPT4 LLaMA Labeling SFT LVIS-Instruct4V (Wang et al., 2023e) Vision-Language LVIS GPT4V LLaMA Labeling SFT LLaVAR (Zhang et al., 2023d) Vision-Language LAION GPT4 LLaMA Labeling SFT Macaw-LLM (Lyu et al., 2023) Multiple Modalities Image/Video with Caption ChatGPT LLaMA Labeling SFT MIMIC-IT (L...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "这里是处理后的文本：\n",
      "2023年，代码代码数据集 ChatGPT LLaMA Labeling SFT Multi-Modality LLaVA (Liu et al., 2023) Vision-Language COCO GPT4 LLaMA Labeling SFT SVIT (Zhao et al., 2023b) Vision-Language Visual Genome + COCO GPT4 LLaMA Labeling SFT LVIS-Instruct4V (Wang et al., 2023e) Vision-Language LVIS GPT4V LLaMA Labeling SFT LLaVAR (Zhang et al., 2023d) Vision-Language LAION GPT4 LLaMA Labeling SFT Macaw-LLM (Lyu et al., 2023) 多模态 数据集 ChatGPT LLaMA Labeling SFT MIMIC-IT (Li et al., 2023f) 多模态 数据集 ChatGPT LLaMA Labe...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Divergence and Similarity, RL: Reinforcement Learning, RO: Ranking Optimization. formats with templates, such as prefacing machine transla- tion data with ”Translate this sentence to Spanish:” . However, these approaches have limitations. Manual data creation is labor-intensive, while template-based transformation lacks diversity in instructions and may not align well with natural human input. LLMs like GPT-4 offer an efficient alternative for creating diverse and controlled SFT data by their ca...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Divergence and Similarity, RL: Reinforcement Learning, RO: Ranking Optimization. 这样的格式化方式有模板，比如给机器翻译数据预前缀为“Translate this sentence to Spanish:”。然而，这些方法存在局限性。手动数据生成工作量大，而基于模板的转换缺乏多样性的指令，并且可能无法很好地与自然的人类输入相匹配。像GPT-4这样的LLM提供了一个高效的方法来生成多样化的受控SFT数据，通过其上下文学习能力和指令遵循能力。目前，大多数相关工作使用OpenAI的GPT系列模型生成提示-响应数据对，然后通过监督微调来训练学生LLM（Wang et al., 2022a; Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Mukherjee et al., 2023; Mitra et al., 2023; Luo et al., 2023b; Peng et al., 2023a）。Self-...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "GPT-3 to expand 15 a seed pool of 175 tasks to 52K task-agnostic instructions, ensuring a broad spectrum of general instructions. Addi- tionally, a filtering and post-processing stage is introduced to eliminate redundant or similar instructions. Notably, through training with this enriched dataset, GPT-3 acquires the ability to follow instructions, enabling it to perform comparably to InstructGPT in zero-shot instruction tasks and when provided with expert-written instructions for novel tasks. B...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "GPT-3 将种子池扩展了 15 个任务，将其扩展为 52,000 个任务agnostic 指令，确保涵盖广泛的一系列通用指令。此外，引入了过滤和后处理阶段以消除冗余或相似的指令。通过使用此丰富数据集进行训练，GPT-3 获得了遵循指令的能力，使其在零样本指令任务中与 InstructGPT 相当，并在提供专家编写的新任务指令时也能表现良好。基于自指导方法，Taori 等人（2023）使用 Llama 7B 模型训练了一个 Alpaca 模型，其生成的 52,000 个指令遵循数据集模仿自指导方法，但利用了更强大的 text-davinci-003 模型。为了提高指令数据的多样性，Wu 等人（2023c）引入了一种称为 Topic-Guided Instruction Generation 的技术。该方法涉及从维基百科中收集 3.5K 个常见主题，用作生成过程中的指导。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Complex Instructions. Some works promote students to solve more complex instructions (Xu et al., 2023a; Luo et al., 2023b,a; Guo et al., 2023c). According to Xu et al. (2023a), in- struction datasets derived from human-written seeds often exhibit low to moderate complexity. To enhance the com- plex instruction-following capabilities of smaller models, WizardLM (Xu et al., 2023a) introduces Evol-Instruct . This method gradually transforms instructions into more com- plex forms through a multi-ste...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Complex Instructions. Some works promote students to solve more complex instructions (Xu et al., 2023a; Luo et al., 2023b,a; Guo et al., 2023c). According to Xu et al. (2023a), instruction datasets derived from human-written seeds often exhibit low to moderate complexity. To enhance the complex instruction-following capabilities of smaller models, WizardLM (Xu et al., 2023a) introduces Evol-Instruct. This method gradually transforms instructions into more complex forms through a multi-step evolu...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "preliminary studies revealing the effectiveness of increasing instruction complexity. Instruction Fusion (Guo et al., 2023c) further uses teacher LLMs to increase the complexity by fusing two distinct evolved instructions. Furthermore, this concept of “evolving” instructions has been extended to distill specific skills such as coding (Luo et al., 2023a) and mathematics (Luo et al., 2023b). Human Instructions. In contrast to works that rely on gener- ating instructions from ChatGPT, which may lac...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "初步研究揭示了增加教学复杂度的有效性。Instruction Fusion（Guo等人，2023c）通过使用教师语言模型（Teacher LLMs）来增加复杂度，通过融合两个独立进化的指令。此外，将“进化”指令的概念扩展到了特定技能的提炼，例如编程（Luo等人，2023a）和数学（Luo等人，2023b）。相比之下，一些工作依赖于从ChatGPT生成指令，这些指令可能缺乏多样性，难以覆盖实际的人类指令。而Vicuna（Chiang等人，2023）和Koala（Geng等人，2023）则通过使用人类对话和社区贡献的自然指令展示了出色的性能。这些对话可以在像ShareGPT这样的平台上找到，用户可以在上面分享他们与ChatGPT的互动。值得注意的是，基于这些自然对话训练的模型可能会模仿风格，但未必能完全覆盖。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "capture the reasoning process of the original teacher (Gudibande et al., 2023; Mukherjee et al., 2023). System Instructions. To encourage student models to learn the reasoning process, Orca and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) enhance the prompt, response data pairs by introducing a system message (e.g., ”explain like I’m five, think step-by-step”) to encourage student mod- els to grasp the reasoning process. This system messageprompts GPT-4 to provide explanation traces that ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "capture the reasoning process of the original teacher (Gudibande et al., 2023; Mukherjee et al., 2023). System Instructions. To encourage student models to learn the reasoning process, Orca and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) enhance the prompt and response data pairs by introducing a system message (e.g., \"explain like I’m five, think step-by-step\") to encourage student models to grasp the reasoning process. This system message prompts GPT-4 to provide explanation traces tha...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023b) distills large-scale data with high-quality and di- verse instructions from teacher LLMs by various meta- information. The UltraLLaMA model, fine-tuned on this data, consistently surpasses other open-source models. The Phi series models (Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023) prioritize data quality and employ synthetic methods to generate data of “textbook quality” to enhance the learning experience for smaller models. Notably, Phi exhibits the ability to follow instruction...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2023年，UltraLLaMA 模型通过高质多样化的指令从教师大语言模型中提取大量数据，并通过各种元信息进行微调。该模型在各项评估中表现优于其他开源模型。Phi 系列模型（Gunasekar 等人，2023；Li 等人，2023a；Mar，2023）则优先保证数据质量，并使用合成方法生成“教科书质量”的数据以提升较小模型的学习体验。值得注意的是，即使不进行特定的指令微调，Phi 也表现出有效遵循指令的能力。特别值得一提的是，仅拥有 2.7 亿参数的 Phi-2 在多个基准评估中超越了 Mistral 和 Llama-2 模型（分别拥有 7B 和 13B 参数）。另一项工作致力于提升现有指令数据的质量，包括指令和相应响应的提升。SelFee（Ye 等人，2023）利用 ChatGPT 迭代地提升指令质量。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "the quality of responses. ExpertLLaMA (Xu et al., 2023f) improves the quality of responses by augment- ing vanilla instructions with specialized Expert Identity descriptions. Reflection-Tuning (Li et al., 2023e) improves both the instruction and response sequentially by reflecting on specific criteria. DEITA (Liu et al., 2023h) proposes to enhance and score instructions in three directions includ- ing complexity, quality, and diversity to get high-quality distillation data. MUFFIN (Lou et al., 2...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "专家LLaMA (Xu et al., 2023f) 通过在普通指令中添加专门的身份描述来提高回应质量。反思调优 (Li et al., 2023e) 通过逐步改进指令和回应来提升两者。DEITA (Liu et al., 2023h) 提出要从三个方面提升指令：复杂性、质量与多样性，从而获得高质量的浓缩数据。MUFFIN (Lou et al., 2023) 提出要根据输入多样化任务，以扩大指令范围。选择性反思调优 (Li et al., 2024d) 通过引入一个新模块让学生模型参与到数据改进流程中，让学生模型决定学习哪些数据。总之，从老师那里浓缩指令数据为训练廉价且可重复的指令遵循语言模型提供了一个有希望的途径。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Cur- rent small models have made strides in enhancing var- ious aspects of instruction-following ability, like diver- sity, complexity and explanation. However, student mod- els trained on instruction data expanded by ChatGPT of- ten mimic ChatGPT’s style without replicating its factual accuracy (Gudibande et al., 2023). Achieving a more ca- pable instruction-following capability requires a stronger teacher LLM (Gudibande et al., 2023) and access to di- verse, high-quality instruction data, such...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "当前的小模型在提升指令跟随能力方面取得了进步，包括多样性和复杂性等。然而，由ChatGPT扩展的数据训练的学生模型往往模仿ChatGPT的风格，而不复制其事实准确性（Gudibande等人，2023年）。要提升指令跟随能力，需要更强的教师LLM以及多样且高质量的指令数据，如Orca项目使用的数据（Mukherjee等人，2023年和Mitra等人，2023年），该数据包含了来自Flan 2022集合的广泛任务指令（Longpre等人，2023年）。4.1.2 多轮对话 单轮指令跟随关注单一实例命令的执行，而多轮对话则通过持续互动保持和理解上下文。这一技能对模型进行有意义的人类对话交流以及在后续交互中保持连贯响应至关重要。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "dialogue turns. Some works have been dedicated to train to small chat models by distilling multi-turn knowl- edge from teacher LLMs (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023). ShareGPT serves as a platform for users to share their conversations with ChatGPT, offering a vast repository of multi-turn conversations readily available. Some small chat models are trained using this data to acquire the capability for engaging...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "一些工作致力于通过从老师语言模型 (LLMs) 中提取多轮知识来训练小型对话模型 (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023)。ShareGPT 作为用户分享与 ChatGPT 对话的平台，提供了一个丰富的多轮对话资源库。一些小型对话模型使用该数据进行训练，以获得参与多轮对话的能力（Chiang et al., 2023; Ye et al., 2023; Wang et al., 2023c）。例如，Vicuna（Chiang et al., 2023）是一个仅使用 ShareGPT 数据进行训练的聊天模型。尽管其唯一的训练来源是 ShareGPT，但 Vicuna 获得了由 GPT-43 分配的高 MT-Bench（Zheng et al., 2023a）评分。在 Wang 等人（2023c）的研究中，GPT-3.5 和 GPT-4 通过使用 ShareGPT 数据生成混合响应。他们给由 ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "aiming to incentivize student models to produce high-quality responses. Addi- tionally, Ye et al. (2023) enhance the quality of multi-turn data from ShareGPT by generating self-feedback on model responses and iteratively refining the responses based on the received feedback. To enhance the multi-turn capabilities of student models, another line of research focuses on expanding conversa- tional datasets through self-chat and using them to train smaller models (Xu et al., 2023b; Ding et al., 2023b...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "为了激励学生模型生成高质量回复，Y\n",
      "等人（2023）通过在ShareGPT生成的多轮数据中添加自我反馈来提升回复质量，并基于收到的反馈迭代改进回复。为增强学生模型的多轮对话能力，另一条研究路径通过自我对话扩展对话数据集，并使用这些数据集训练较小的模型（Xu等人，2023b；Ding等人，2023b；Tunstall等人，2023）。例如，Xu等人（2023b）开始工作时，使用来自Quora和Stack Overflow的问题作为种子，收集了111.5k对话。随后，他们通过参数效率调优训练了一个名为Baize的聊天模型。Ding等人（2023b）首先构建了一个名为UltraChat的较大数据集，包含150,000个高质量的多轮对话，通过指令蒸馏实现。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "dialogues from ChatGPT. Notably, UltraChat encom- passes a wide range of topics and instructions. Building upon the UltraChat dataset, they fine-tune a LLaMA model, resulting in the creation of a powerful chat model known as UltraLLaMA. UltraLLaMA consistently outperforms other open-source chat models, including Vicuna and Baize. Fur- thermore, UltraChat is employed in conjunction with an AI preference-aligned chat model named Zephyr (Tunstall et al., 2023). Zephyr enhances intent alignment thro...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "UltraChat 和 UltraLLaMA 通过 fine-tuning LLaMA 模型，在处理广泛主题和指令方面表现出色。UltraLLaMA 持续优于其他开源模型，如 Vicuna 和 Baize。此外，UltraChat 与 AI 偏好对齐的聊天模型 Zephyr（Tunstall 等人，2023 年）结合使用。Zephyr 通过应用浓缩直接偏好优化（dDPO）来增强意图对齐。4.1.3 集成了检索的生成式（RAG）能力的 LLMs 缺乏使用最新知识的能力，经常因为依赖于参数知识而产生事实不准确的响应。RAG 是一种有前途的技术，能够减少这一问题。处理增强上下文。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "of retrieved information is also a non- trivial skill of LLMs. Several approaches to distill RAG capabilities have been proposed (Kang et al., 2023a; Luo et al., 2023c; Asai et al., 2023). SAIL (Luo et al., 2023c) starts by retrieving search results for each training case using search APIs, creating search- augmented instructions that include both the instruction and grounding information. To encourage the language model to prioritize informative retrieval results, they input each retrieved pass...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "通过检索获取信息也是一个LLM的非 trivial技能。已提出几种方法来提取RAG的能力（Kang et al., 2023a；Luo et al., 2023c；Asai et al., 2023）。SAIL（Luo et al., 2023c）通过使用搜索API为每个训练案例检索结果，并创建包含指令和支撑信息的搜索增强式指令开始。为了鼓励语言模型优先考虑有信息性的检索结果，他们将每个检索到的段落以及真值响应输入到 entailment模型中，以对每个检索结果进行相关性标记。随后，搜索增强式指令和相关性标签被输入到教师LLM（如GPT-4）中生成响应。经过此训练集上的微调后，学生模型能够有效去除搜索结果中的噪音并生成准确的响应。KARD（Kang et al., 2023b）将教师LLM对问题x的解答提取为支撑信息。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "rationales are then utilized to train two models: a student LM and a Reranker. For training the student LM, the rationales serve as a means to retrieve relevant knowledge d, and the student LM is subsequently fine-tuned using the rationales along- side questions and knowledge. However, during inference, only questions are available. To address this, the Reranker is trained to mimic how the retriever scores passages with the rationale by minimizing the KL divergence between Retriever (d|r)andRera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "rationales被用于训练两个模型：学生语言模型（Student LM）和重排器（Reranker）。为了训练学生语言模型，rationales被用作检索相关知识d的手段，学生语言模型随后使用rationales和问题一起进行微调。然而，在推理过程中，只有问题可用。为了解决这个问题，重排器被训练来模仿检索器根据rationales评估段落的方式，通过使Retriever（d|r）和Reranker（d|x）之间的KL散度最小化。然而，语言模型中集成固定数量的段落，而不考虑其必要性和相关性，可能会降低灵活性，并导致生成无帮助的响应。为了赋予学生语言模型基于适应性RAG的能力，Self-Rag（Asai等，2023）将这种适应性能力从教师语言模型中提取到一个小判别模型中。这个判别模型确定检索是否必要，并通过生成来评估检索结果的质量。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "‘reflection to- kens.’ For instance, Self-Rag initiates the retrieval operation when generating the reflection token Retrieve . To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations I, the task input x, and output yto predict a reflection token ras follows: p(r|I, x, y ). 4.2 Alignment 4.2.1 Thinking Pattern Most existing methods mainly focus on directly aligning the direct responses of the student models to the responses of teacher models ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "初始响应中直接提取的关键信息为：“For instance, Self-Rag initiates the retrieval operation when generating the reflection token Retrieve. To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations, I, the task input x, and output y to predict a reflection token r as follows: p(r|I, x, y).”其他部分均为无关信息，已去除。处理后的文本如下：...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "by the effectiveness of LLMs in generat- ing their own feedback without relying on external mod- els (Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022), SelFee (Ye et al., 2023) proposes to train a 17 model that has been fine-tuned to continuously revise its own answer until it provides a high-quality response in a single inference. During training, it utilizes both the final response and feedback chain as the fitting target. This pat- tern, response with the revision process, sho...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "通过LLM自行生成反馈的能力（Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022），SelFee（Ye et al., 2023）提出了一种训练模型的方法，该模型经过微调以持续修订自己的答案，直到生成高质量的单一响应。在训练过程中，它将最终响应和反馈链作为拟合目标。这种响应与修订过程的模式表现出良好的性能提升。随后，Reflection-Tuning（Li et al., 2023e, 2024d）也利用了反馈过程作为学习模式。注意到之前的模型缺乏推理模仿，Orca（Mukherjee et al., 2023）首先提出了Explanation tuning，旨在从教师模型中学习推理步骤，包括解释痕迹、步骤思考过程及其他复杂指令，而非仅模仿基本风格。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to equip the student models with the ability to utilize different solution strategies for different tasks, mo- tivated by the capability discrepancies between the smaller and larger models. By employing this training pattern, the student models are able to gain a better reasoning ability. Be- sides learning with the corresponding revision or reflection process,...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "实验验证了这种思考模式在提取知识方面的有效性。随后，Orca2（Mitra等，2023年）进一步为学生模型配备了根据不同任务使用不同解决方案策略的能力，动机来源于较小模型和较大模型之间存在的能力差异。通过采用这种训练模式，学生模型能够获得更好的推理能力。除了学习相应的修订或反思过程之外，最近出现的另一种思考模式是生成响应和偏好。Zhang等人（2023a）提出通过学习特定领域的知识及其相应的偏好来训练LLMs。最近，DEBATunE（Li等，2024e）提出通过在结构化的多轮辩论中让两名代理参与有争议话题的生成，来提高LLMs生成陈述的可控性。通过这种结构化的多轮辩论，可以获取到突出且深入的陈述。...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distilled into the student models. 4.2.2 Preference The previously mentioned methods primarily focus on the basic capability of student models to produce outcomes that are strictly accurate but may not align with human preferences, reaching alignment at this level enables these models to aid in various tasks without meeting higher-level demands. Early methods mainly utilize human feed...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "早期方法主要依赖于人工反馈。这些方法主要关注学生模型在生成结果时的准确性，但这些结果可能并不符合人类的偏好。达到这种水平的对齐，这些模型可以在不满足更高层次需求的情况下，协助完成各种任务。...\n",
      "==========================================================================================\n",
      "\n"
     ]
    }
   ],
   "execution_count": 30
  },
  {
   "cell_type": "markdown",
   "id": "31cffe8d",
   "metadata": {},
   "source": "让我们看看最后处理的效果吧～🍻"
  },
  {
   "cell_type": "code",
   "id": "89ef51a7-f13f-49a4-8f73-9ac8ce75319d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:12:11.074075Z",
     "start_time": "2024-10-31T16:12:11.071563Z"
    }
   },
   "source": [
    "print(f\"\\n处理完成！\")\n",
    "print(f\"输入文件: {INPUT_FILE}\")\n",
    "print(f\"输出文件: {output_file}\")\n",
    "print(f\"已处理总文本块: {num_chunks}\")\n",
    "\n",
    "# 预览处理后的文本的开头和结尾。\n",
    "print(\"\\n预览处理后的文本：\")\n",
    "print(\"\\n开头:\")\n",
    "print(processed_text[:1000])\n",
    "print(\"\\n...\\n\\n结尾:\")\n",
    "print(processed_text[-1000:])"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "处理完成！\n",
      "输入文件: ./resources/extracted_text.txt\n",
      "输出文件: ./resources/clean_extracted_text.txt\n",
      "已处理总文本块: 101\n",
      "\n",
      "预览处理后的文本：\n",
      "\n",
      "开头:\n",
      "在大规模语言模型（LLMs）的时代，知识蒸馏（KD）作为将顶级专有LLM如GPT-4等先进能力转移给开源模型如LLaMa和Mistral的关键方法出现了。随着开源模型的蓬勃发展，KD在压缩这些模型和通过自身进行自我提升方面扮演了重要角色。本文全面回顾了KD在LLM领域中的作用，强调了其在赋予模型知识方面的重要功能。\n",
      "通过将高级知识传授给较小的模型，并将其应用于模型压缩和自我提升方面具有实用价值。我们的调查围绕三个基础支柱展开：算法、技能和垂直化，提供了一个全面的评估知识传授机制、特定认知能力的提升及其在多个领域中的实际影响。至关重要的是，调查探讨了数据增强（DA）与知识传授（KD）之间的相互作用，展示了如何将DA作为KD框架中的强大范式来增强模型性能。通过利用DA生成富含上下文且特定技能的训练数据，知识传授超越了传统界限，使得开源模型能够模仿其专用版本的上下文敏感性、伦理一致性以及深层语义洞察。本文旨在为研究人员和实践者提供有价值的指南，详细介绍当前在知识传授方面的研究方法。\n",
      "本文通过将专有和开源大型语言模型（LLMs）之间的差距进行桥梁连接，强调了更具访问性、效率和强大性的AI解决方案的潜在价值。我们强烈倡导遵守用于LLMs的法律条款，确保知识蒸馏（KD）的伦理和合法性应用。有关知识蒸馏的GitHub存储库可在https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs处获取。关键术语——大型语言模型，知识蒸馏，数据增强，技能蒸馏，监督微调。1 引言 在人工智能（AI）不断发展的背景下，GPT-3.5（Ouyang et al., 2022）、GPT-4（OpenAI et al., 2023）、Gemini（Team et al., 2023）和Claude2等专有大型语言模型（LLMs）已经成为颠覆性的技术，极大地改变了我们对自然语言处理（NLP）的理解。这些模型以其庞大的规模而著称。\n",
      "这些大型语言模型（LLMs）在解锁新的可能性方面具有核心意义。它们不仅能够生成类似人类的文本，还能够提供高级问题解决能力。这些模型的核心能力在于其涌现能力（Wei et al., 2022a,b; Xu et al., 2024a），即模型在超出其明确训练目标的情况下展现出超出预期的能力，从而能够高效地解决多样\n",
      "\n",
      "...\n",
      "\n",
      "结尾:\n",
      "ieve. To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations, I, the task input x, and output y to predict a reflection token r as follows: p(r|I, x, y).”其他部分均为无关信息，已去除。处理后的文本如下：\n",
      "通过LLM自行生成反馈的能力（Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022），SelFee（Ye et al., 2023）提出了一种训练模型的方法，该模型经过微调以持续修订自己的答案，直到生成高质量的单一响应。在训练过程中，它将最终响应和反馈链作为拟合目标。这种响应与修订过程的模式表现出良好的性能提升。随后，Reflection-Tuning（Li et al., 2023e, 2024d）也利用了反馈过程作为学习模式。注意到之前的模型缺乏推理模仿，Orca（Mukherjee et al., 2023）首先提出了Explanation tuning，旨在从教师模型中学习推理步骤，包括解释痕迹、步骤思考过程及其他复杂指令，而非仅模仿基本风格。\n",
      "实验验证了这种思考模式在提取知识方面的有效性。随后，Orca2（Mitra等，2023年）进一步为学生模型配备了根据不同任务使用不同解决方案策略的能力，动机来源于较小模型和较大模型之间存在的能力差异。通过采用这种训练模式，学生模型能够获得更好的推理能力。除了学习相应的修订或反思过程之外，最近出现的另一种思考模式是生成响应和偏好。Zhang等人（2023a）提出通过学习特定领域的知识及其相应的偏好来训练LLMs。最近，DEBATunE（Li等，2024e）提出通过在结构化的多轮辩论中让两名代理参与有争议话题的生成，来提高LLMs生成陈述的可控性。通过这种结构化的多轮辩论，可以获取到突出且深入的陈述。\n",
      "早期方法主要依赖于人工反馈。这些方法主要关注学生模型在生成结果时的准确性，但这些结果可能并不符合人类的偏好。达到这种水平的对齐，这些模型可以在不满足更高层次需求的情况下，协助完成各种任务。\n",
      "\n"
     ]
    }
   ],
   "execution_count": 31
  },
  {
   "cell_type": "markdown",
   "id": "3d996ac5",
   "metadata": {},
   "source": [
    "### 下一个 Notebook: 转录员\n",
    "\n",
    "现在我们已经预处理好文本，在下一个 Notebook 中将其转换为讲稿"
   ]
  },
  {
   "cell_type": "code",
   "id": "1b16ae0e-04cf-4eb9-a369-dee1728b89ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T16:12:11.092996Z",
     "start_time": "2024-10-31T16:12:11.091314Z"
    }
   },
   "source": [
    "#fin"
   ],
   "outputs": [],
   "execution_count": 32
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}