理解常见陷阱 —— 遗忘
我的理解
ChatGPT 的“遗忘”有两层来源：一是产品层面主动裁剪早期对话（为控制延迟和成本），二是 GPT 模型在上下文接近上限时真实的性能退化。有趣的是，即便早期要求被丢弃，GPT 仍可能通过观察近期对话中隐含的行为模式来推断并遵循约定——说明上下文窗口不仅是限制，也是可被利用的推理载体。作者通过构建娱乐机器人的真实经历印证了这套理论，同时引出“在构建中学习”的第二个核心心态：实操练习比阅读理解更能建立对 AI 行为的直觉认知，因为我们的身份已是构建者而非用户。
相关链接
- Ch02-L02 研究 LLM内部机制 记忆知识上下文 — 上下文窗口机制是理解遗忘现象的理论基础
- Ch02-L03 理解常见陷阱 偷懒 — 同系列陷阱，对比两者有助于全面理解上下文窗口的限制表现
- Ch02-L06 应用 使用 ChatGPT 的新方式 编辑 而非对话 — 主动管理上下文窗口是解决遗忘问题的直接实践方法
- Ch06-L06 重访上下文窗口管理 自动管理 — 后续模块提出的更系统化的上下文自动管理方案
原文
Lesson 11 of 68 理解常见陷阱 —— 遗忘 / Make sense of common pitfalls - Forgetting 遗忘：上下文窗口的限制
有时我们会遇到这样的情况：在对话开始时，我们要求 ChatGPT 遵守某条规则，比如不要使用某个库函数，或在每句话末尾加上一个表情符号。一开始它执行得很好，但随着对话推进，ChatGPT 似乎会忘记这些要求。作为回应，我们会反问它是否做对了，作为一种提醒。有时它能自我修正，但有时它甚至意识不到自己的问题。
造成这种现象的原因有几个，核心在于“上下文窗口（context window）”这一概念。正如此前课程中所提到的，在幕后，ChatGPT 会把所有历史对话拼接成一个 prompt，再喂给底层的 GPT 模型。当 prompt 过长时——哪怕还没有完全填满上下文窗口——ChatGPT 就会有意丢弃最早的对话内容，以将 prompt 控制在合理的长度内。这其实是一种聪明的产品设计，因为它有助于控制延迟和成本。如果对话早期提出的某个要求被丢弃了，就可能导致 ChatGPT 不再遵守该要求。
但故事还没结束。GPT 其实非常聪明，它可能仍然能通过查看最近的对话来推断出该要求。例如，如果我们要求它在每句话末尾加上一个表情符号，即便该要求本身因 ChatGPT 的设计被丢弃，GPT 在查看最近的对话后，仍可能推理出最好遵循这一惯例，并在每句话末尾加上表情符号。因此，在这种情况下你可能仍会观察到它在遵守要求。
你可能好奇：我们能否验证上面这套理论是否正确？答案是可以的。我们之前曾使用 OpenAI 的 GPT API 构建过一个类似 ChatGPT 的娱乐机器人。为了节省成本，我们采用了非常短的记忆，只保留最近 3 到 5 轮对话。在这种设置下，常常可以观察到机器人忘记之前的要求，这让用户颇为不满。后来出现了一些试图“黑掉”机器人、突破这一限制的用户，他们发现 GPT 足够聪明，可以从对话中进行推理，于是他们利用这一点，让 GPT 即便在初始要求已不在历史对话窗口内的情况下，也能遵循他们的简单要求。
这其实非常有趣，也正是我们想分享的第二个心态：在动手构建中学习（learning by building）。我们身处一个非常注重实操的领域。阅读和理解固然有用，但做哪怕是很小的练习、动手构建一些东西，可能效果更好。归根结底，我们是构建者，而不再只是使用者。
另一种可能导致“遗忘”和“漏掉细节”这种可观察行为的场景，是当 prompt 真的非常长，逼近上下文窗口约 3/4 的时候。在较老的 GPT-3.5（非 turbo 版本）上更容易观察到性能下降，因为它的上下文窗口只有 4k token。在这种情况下，真正“忘记”要求并缺乏能力关注细节的，其实是 GPT 模型本身，而不是 ChatGPT 这款产品。再次说明，由于 GPT-3.5-Turbo 的上下文窗口接近 16k token，GPT-4-Turbo 接近 128k token，这种情况就更难被触发了。
English Original Forgetting: Limit of Context Window
Sometimes we also got into a situation where, in the beginning of the conversation, we asked ChatGPT to stick to a rule, such as not using a certain library function or putting an emoji at the end of every sentence. It does the job well at first, but as the conversation progresses, ChatGPT seems to forget about those requirements. As a reaction, we question it whether it did the correct thing as a reminder. Sometimes it is able to correct itself, but sometimes it doesn’t even realize its problems.
There are a few reasons that might cause this, and the core lies in the concept of the context window. As also mentioned in previous lessons, behind the scenes, ChatGPT puts all the historical conversations into a prompt and feeds it to the underlying GPT model. When the prompt is too long, even not to the point of filling up the context window, ChatGPT begins to purposely drop earliest conversations to keep the prompt within a reasonable length. This is actually a clever product design because it helps control the latency and cost. If a request raised in the early conversation gets dropped, this may cause ChatGPT to not follow the request.
But that’s still not the whole story. GPT is actually smart. It may still be able to reason the request by looking at the latest conversations. For example, if we ask it to end each sentence with an emoji, even when the request itself gets dropped because of ChatGPT’s design, after checking the latest conversations, GPT may still infer that it’d better to follow this convention and put an emoji at the end of each sentence. Therefore, you might still observe it following the request in this case.
You may be curious, is it possible for us to verify if our previous theory is true? Actually, yes. We built a ChatGPT-like bot for entertainment using OpenAI’s GPT APIs before. To save costs, we also used a very short memory, keeping only the last 3-5 conversations. In this case, it’s common to observe the bot forget about requests, which upsets our users. And then there appear users trying to hack the bot to go beyond this limit, and they’ve managed to figure out that GPT is smart enough to reason from the conversation, and they took advantage of that to enable GPT to follow their simple requests, even though the initial request may be outside the historical conversation window.
This is pretty interesting and actually the second mindset we wanted to share: learning by building. We are in a very hands-on field. While reading and understanding is useful, doing even small exercises and building stuff is potentially more effective. At the end of the day, we are builders, not users anymore.
Another potential scenario for the observable behavior of forgetting and missing details is when the prompt gets really long, approaching roughly 3/4 of the context window. It is easier for the older GPT-3.5 (the non-turbo version) to observe degraded behavior because it has only a 4k token context window. In this case, it is actually the GPT model that forgets about the requests and lacks the capabilities to note the details, rather than the ChatGPT product. Again, as the GPT-3.5-Turbo has a context window approaching 16k tokens and the GPT-4-Turbo approaching 128k tokens, it is harder to trigger this.