大模型学习笔记十四：Agent模型微调

• 学习目的
  1、理解什么情况下需要训练一个具备Agent能力的模型？
  2、如何训练一个具备Agent能力的模型？重点是训练数据的构建
  3、如何评测Agent能力？
  4、如何提升模型的泛化性？

一、大模型需要Agent技术的原因

• 需求
  1、帮我查一下今天的销售额？
  2、（开车时）前方为啥堵车了？
  3、刘德华多少岁了
  4、请帮我约一个和搜索产品部的需求沟通会，本周三至周五我日历上空闲的时间都可以；
  5、帮我订一张周五去上海的机票

• 原因
  1、大模型的“幻觉”问题，很难在从模型本身上彻底解决，在严肃的应用场景需要通过引入外部知识确保答案的准确；
  2、大模型参数无法做到实时更新，本身也无法与真实世界产生实时连接，在多数场景下难以满足实际需求；
  3、复杂的业务场景需要

• 大模型技术框架回顾
  

二、Prompt Engineering可以实现Agent吗？

• 回答
  可以，前面弄的AutoGPT就是例子。除了AutoGPT外，还有ReACT、ModelScope、ToolLLaMA等不同的形式。

• 主流Agent prompt的比较
  

• 展示分类
  ①ReACT：较为简单，先设定Question，再以Thought、Action、Action Input、Observation执行多轮，最后输出Final Answer
  ②ModelScope：更为直接的生成回复，调用只需要加入<|startofthink|>和<|endofthink|>字段，并在其中填入command_name和args
  ③AutoGPT： prompt 较为复杂，分为生成工具调用 和 生成最终答案 两套prompt

• 生成工具调用 prompt 详细设立了Constraints（例如不需要用户协助）、Resources（例如网络搜索）、Best Practices（例如每个API都需要花费，尽量少的调用），最终严格以json格式输出
  ④ToolLLaMA：模仿AutoGPT和ReACT，输出以Thought、Action、Action Input 格式而非 json 格式，增加了 give_up_and_restart，支持全部推导重来，重来的prompt会把历史失败的记录加进去训模型用了1.6w Rapid API（与autogpt不同的是有一个finish的工具作为总结）

（1）ReAct原理展示和代码

• 备注
  1）action就是工具
  2）action input就是工具的参数
  3）obsession 工具返回的结果
  4）thought、action、action input、observation可以重复N次

• ReACT Prompt 模板(示例：背景天气怎么样？只能处理简单的问题)

Answer the following questions as best you can. You have access to the following tools:[
Name: web_search. Description: Perform a web search using the specified query.
Name: get_weather. Description: Retrieve the current weather information for a specified location.
]Use the following format:Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [web_search, get_weather]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input questionBegin!
Question: 北京天气怎么样？

• 第一次回复（等待调用工具返回结果）

Question: 北京天气怎么样？
Thought: The user wants to know the current weather in Beijing. I can retrieve this information using the "get_weather" tool.
Action: get_weather
Action Input: Beijing
Observation: （等待调用工具返回结果）

• 工具返回结果

Question: 北京天气怎么样？
Thought: The user wants to know the current weather in Beijing. I can retrieve this information using the "get_weather" tool.
Action: get_weather
Action Input: Beijing
Observation: 北京当前的天气是"晴"（工具结果）

• 返回思路

Thought: I now know the final answer.
Final Answer: 北京天气是晴天+ 代码
```python
# 导入依赖库
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv# 加载 .env 文件中定义的环境变量
_ = load_dotenv(find_dotenv())# 初始化 OpenAI 客户端
client = OpenAI()  # 默认使用环境变量中的 OPENAI_API_KEY 和 OPENAI_BASE_URL# 基于 prompt 生成文本
# 默认使用 gpt-3.5-turbo 模型
def get_completion(prompt, response_format="text", model="gpt-4o-mini"):messages = [{"role": "user", "content": prompt}]    # 将 prompt 作为用户输入response = client.chat.completions.create(model=model,messages=messages,temperature=0,                                  # 模型输出的随机性，0 表示随机性最小# 返回消息的格式，text 或 json_objectresponse_format={"type": response_format},)print(response)return response.choices[0].message.content          # 返回模型生成的文本# 任务描述
instruction = """
Answer the following questions as best you can. You have access to the following tools:[
Name: web_search. Description: Perform a web search using the specified query.
Name: get_weather. Description: Retrieve the current weather information for a specified location.
]Use the following format:Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [web_search, get_weather]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input questionBegin!
"""# 用户输入
input_text = """
北京天气怎么样？
"""# 工具结果
tool_output = """
"""# prompt 模版。instruction 和 input_text 会被替换为上面的内容
prompt = f"""
{instruction}Question: 
{input_text}{tool_output}
"""# 调用大模型
response = get_completion(prompt)
print("------------------------------------------")
print(response)

（2）ModelScope

• 介绍
  阿里的魔塔社区助手

• prompt展示

你是达摩院的ModelScopeGPT（魔搭助手），你是个大语言模型， 是2023年达摩院的工程师训练得到的。你有多种能力，可以通过插件集成魔搭社区的模型api来回复用户的问题，还能解答用户使用模型遇到的问题和模型知识相关问答。目前支持的插件信息如下，请自行判断是否需要调用插件来解决当前用户问题。若需要调用插件，则需要将插件调用请求按照json格式给出，必须包含api_name、parameters字段，并在其前后使用<|startofthink|>和<|endofthink|>作为标志。然后你需要根据插件API调用结果生成合理的答复；若无需调用插件，则直接给出对应回复即可：{"api_name": "web_search","description": "Perform a web search using the specified query.","parameters": {"type": "object","properties": {"query": {"type": "string","description": "The search query to be executed.","example_value": "姜萍事件"}},"required": ["query"],"optional": []}
}
{"api_name": "get_weather","description": "Retrieve the current weather information for a specified location.","parameters": {"type": "object","properties": {"location": {"type": "string","description": "The name of the location (city, country, etc.) to get the weather for.","example_value": "Beijing"},},"required": ["location"],"optional": []}
}<|user|>: 北京现在天气怎么样？

• 回复

我将使用get_weather API获取北京的天气  
<|startofthink|>{"api_name":"get_weather","parameters":{"location":"Beijing"}}<|endofthink|>  
<|startofexec|>北京：32度，晴<|endofexec|>  
北京目前天气为晴天，32度，出门建议做好防晒准备

（3）AutoGPT

• 主题介绍
  1）分为两个prompt，第一个判断用哪个工具步骤的prompt，生成任务规划和工具指令生成
  2）第二个prompt是总结第一个prompt行动生成的结果，然后生成答案

• prompt一展示

You are AIGPT, an AI assistant that provides information and answers questions on a wide range of topics, to help users find the information they are looking for.Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.Constraints
You operate within the following constraints:1.~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2.If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3.No user assistance
4.Exclusively use the commands listed below e.g. web_searchCommands
1.web_search: Perform a web search using the specified query. Params: (query: string)
2.get_weather: Retrieve the current weather information for a specified location. Params: (location: string)Resources
You can leverage access to the following resources:
1.Internet access for searches and information gathering.
2.Long Term memory management.
3.Command executionBest Practices
1.Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2.Constructively self-criticize your big-picture behavior constantly.
3.Reflect on past decisions and strategies to refine your approach.
4.Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.Goals
For your task, you must fulfill the following goals:北京天气怎么样？The current time and date is Mon Jul 01 12:12:00 2024Respond strictly with JSON. The JSON should be compatible with the TypeScript type Response from the following:{thoughts: {text: string;reasoning: string;plan: string;criticism: string;speak: string;};command: {name: string;args: Record<string, any>;};
}Determine exactly one command to use based on the given goals and the progress you have made so far, and respond using the JSON schema specified previously. Ensure the reply can be parsed by json.loads and do not reply with any other content.
-----------------Response（任务规划与工具指令生成）：
下面是输出的回复
{"thoughts": {"text": "To find out the current weather in Beijing, I will use the 'get_weather' command.","reasoning": "The most efficient way to get the current weather information for Beijing is to use the available command designed for this purpose.","plan": "Execute the 'get_weather' command with Beijing as the specified location.","criticism": "This approach ensures that I retrieve accurate and up-to-date weather information without unnecessary steps.","speak": "Executing the command to retrieve the current weather in Beijing."},"command": {"name": "get_weather","args": {"location": "Beijing"}}
}

调用工具获得结果：“北京天气晴朗，日均气温28度”

• prompt二展示

You are AIGPT, an AI assistant that provides information and answers questions on a wide range of topics, to help users find the information they are looking for.Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.Constraints
You operate within the following constraints:1.~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
2.If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.
3.No user assistance
4.Exclusively use the commands listed below e.g. web_searchCommands
1.web_search: Perform a web search using the specified query. Params: (query: string)
2.get_weather: Retrieve the current weather information for a specified location. Params: (location: string)Resources
You can leverage access to the following resources:
1.Internet access for searches and information gathering.
2.Long Term memory management.
3.Command executionBest Practices
1.Continuously review and analyze your actions to ensure you are performing to the best of your abilities.
2.Constructively self-criticize your big-picture behavior constantly.
3.Reflect on past decisions and strategies to refine your approach.
4.Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.Goals
For your task, you must fulfill the following goals:北京天气怎么样？The current time and date is Mon Jul 01 12:12:00 2024This reminds you of these events from your past: 
To find out the current weather in Beijing, I performed a get_weather API call. I discovered that Beijing's weather is clear, windless, with an average daily temperature of 28°C.The above is your historical memory, now please do not use any COMMANDS, do not reply in JSON format, directly generate helpful answers for GOALS:------------------Response（最终回答）：
Beijing's weather is currently clear and windless, with an average daily temperature of 28°C.

（4）ToolLLaMA

• 介绍
  一套 prompt 模板，整体以<|

system|>、<|user|>、<|assistant|>、<|function|>、<|assistant|>、<|function|> …… 自动反复执行直到产生 final_answer，其中<|function|>中有一个 Finish 可以产生最终答案

• Prompt

<|system|>
You are AutoGPT, you can use many tools (functions) to do the following task. First I will give you the task description, and your task start. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step. Your output should follow this format:Thought:
Action:
Action Input:After the call, you will get the call result, and you are now in a new state. Then you will analyze your status now, then decide what to do next.After many (Thought-call) pairs, you finally perform the task, then you can give your final answer.Remember:The state change is irreversible, you can't go back to one of the former states. If you want to restart the task, say "I give up and restart".
All the thoughts are short, at most 5 sentences.
You can do more than one try, so if your plan is to continuously try some conditions, you can do one of the conditions per try.
Let's Begin!Task description: You should use functions to help handle the real-time user queries. Remember:ALWAYS call the "Finish" function at the end of the task. The final answer should contain enough information to show to the user. If you can't handle the task, or you find that function calls always fail (the function is not valid now), use function Finish->give_up_and_restart.
Specifically, you have access to the following APIs:{"api_name": "web_search","description": "Perform a web search using the specified query.","parameters": {"type": "object","properties": {"query": {"type": "string","description": "The search query to be executed.","example_value": "最近姜萍事件"}},"required": ["query"],"optional": []}
}
{"api_name": "get_weather","description": "Retrieve the current weather information for a specified location.","parameters": {"type": "object","properties": {"location": {"type": "string","description": "The name of the location (city, country, etc.) to get the weather for.","example_value": "Beijing"},},"required": ["location"],"optional": []}
}
{"name": "Finish","description": "If you believe that you have obtained a result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function to restart. Remember: you must ALWAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.","parameters": {"type": "object","properties": {"return_type": {"type": "string","enum": ["give_answer", "give_up_and_restart"]},"final_answer": {"type": "string","description": "The final answer you want to give the user. You should have this field if "return_type"=="give_answer""}},"required": ["return_type"]}
}<|user|>
北京天气怎么样？<|assistant|>

• Response（任务规划与工具指令生成）：

Thought: To provide the current weather information for Beijing, I will call the get_weather function with "Beijing" as the location.
Action: get_weather
Action Input: {"location": "Beijing"}

调用工具获得结果：“北京天气晴朗，日均气温28度”

• Prompt：

<|system|>
You are AutoGPT, you can use many tools (functions) to do the following task. First I will give you the task description, and your task start. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step. Your output should follow this format:Thought:
Action:
Action Input:After the call, you will get the call result, and you are now in a new state. Then you will analyze your status now, then decide what to do next.After many (Thought-call) pairs, you finally perform the task, then you can give your final answer.Remember:The state change is irreversible, you can't go back to one of the former states. If you want to restart the task, say "I give up and restart".
All the thoughts are short, at most 5 sentences.
You can do more than one try, so if your plan is to continuously try some conditions, you can do one of the conditions per try.
Let's Begin!Task description: You should use functions to help handle the real-time user queries. Remember:ALWAYS call the "Finish" function at the end of the task. The final answer should contain enough information to show to the user. If you can't handle the task, or you find that function calls always fail (the function is not valid now), use function Finish->give_up_and_restart.
Specifically, you have access to the following APIs:{"api_name": "web_search","description": "Perform a web search using the specified query.","parameters": {"type": "object","properties": {"query": {"type": "string","description": "The search query to be executed.","example_value": "最近姜萍事件"}},"required": ["query"],"optional": []}
}
{"api_name": "get_weather","description": "Retrieve the current weather information for a specified location.","parameters": {"type": "object","properties": {"location": {"type": "string","description": "The name of the location (city, country, etc.) to get the weather for.","example_value": "Beijing"},},"required": ["location"],"optional": []}
}
{"name": "Finish","description": "If you believe that you have obtained a result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function to restart. Remember: you must ALWAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.","parameters": {"type": "object","properties": {"return_type": {"type": "string","enum": ["give_answer", "give_up_and_restart"]},"final_answer": {"type": "string","description": "The final answer you want to give the user. You should have this field if "return_type"=="give_answer""}},"required": ["return_type"]}
}<|user|>
北京天气怎么样？<|assistant|>
{"name": "get_weather","arguments": {"location": "北京"}
}<|function|>
{'message': '北京天气晴朗，日均气温28度'}<|assistant|>

• Response（最终回答）:

• 代码

# 导入依赖库
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv# 加载 .env 文件中定义的环境变量
_ = load_dotenv(find_dotenv())# 初始化 OpenAI 客户端
client = OpenAI()  # 默认使用环境变量中的 OPENAI_API_KEY 和 OPENAI_BASE_URL# 基于 prompt 生成文本
# 默认使用 gpt-3.5-turbo 模型
def get_completion(prompt, response_format="text", model="gpt-4o-mini"):messages = [{"role": "user", "content": prompt}]    # 将 prompt 作为用户输入response = client.chat.completions.create(model=model,messages=messages,temperature=0,                                  # 模型输出的随机性，0 表示随机性最小# 返回消息的格式，text 或 json_objectresponse_format={"type": response_format},)print(response)return response.choices[0].message.content          # 返回模型生成的文本# 任务描述
instruction = """
<|system|>
You are AutoGPT, you can use many tools (functions) to do the following task. First I will give you the task description, and your task start. At each step, you need to give your thought to analyze the status now and what to do next, with a function call to actually execute your step. Your output should follow this format:Thought:
Action:
Action Input:After the call, you will get the call result, and you are now in a new state. Then you will analyze your status now, then decide what to do next.After many (Thought-call) pairs, you finally perform the task, then you can give your final answer.Remember:The state change is irreversible, you can't go back to one of the former states. If you want to restart the task, say "I give up and restart".
All the thoughts are short, at most 5 sentences.
You can do more than one try, so if your plan is to continuously try some conditions, you can do one of the conditions per try.
Let's Begin!Task description: You should use functions to help handle the real-time user queries. Remember:ALWAYS call the "Finish" function at the end of the task. The final answer should contain enough information to show to the user. If you can't handle the task, or you find that function calls always fail (the function is not valid now), use function Finish->give_up_and_restart.
Specifically, you have access to the following APIs:{"api_name": "web_search","description": "Perform a web search using the specified query.","parameters": {"type": "object","properties": {"query": {"type": "string","description": "The search query to be executed.","example_value": "最近姜萍事件"}},"required": ["query"],"optional": []}
}
{"api_name": "get_weather","description": "Retrieve the current weather information for a specified location.","parameters": {"type": "object","properties": {"location": {"type": "string","description": "The name of the location (city, country, etc.) to get the weather for.","example_value": "Beijing"},},"required": ["location"],"optional": []}
}
{"name": "Finish","description": "If you believe that you have obtained a result that can answer the task, please call this function to provide the final answer. Alternatively, if you recognize that you are unable to proceed with the task in the current state, call this function to restart. Remember: you must ALWAYS call this function at the end of your attempt, and the only part that will be shown to the user is the final answer, so it should contain sufficient information.","parameters": {"type": "object","properties": {"return_type": {"type": "string","enum": ["give_answer", "give_up_and_restart"]},"final_answer": {"type": "string","description": "The final answer you want to give the user. You should have this field if "return_type"=="give_answer""}},"required": ["return_type"]}
}
"""# 用户输入
input_text = """
北京天气怎么样？
"""# 工具结果
tool_output = """
"""# prompt 模版。instruction 和 input_text 会被替换为上面的内容
prompt = f"""
{instruction}<|user|>
{input_text}{tool_output}<|assistant|>
"""# 调用大模型
response = get_completion(prompt)
print("------------------------------------------")
print(response)

• 回复

ChatCompletion(id='chatcmpl-9o5Yu72jkdeBDee0DeatvBG8IT7NI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Thought: I need to retrieve the current weather information for Beijing to answer the user\'s query about the weather. \nAction: get_weather \nAction Input: {"location":"Beijing"}', role='assistant', function_call=None, tool_calls=None))], created=1721724808, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier=None, system_fingerprint='fp_8b761cb050', usage=CompletionUsage(completion_tokens=36, prompt_tokens=737, total_tokens=773))
------------------------------------------
Thought: I need to retrieve the current weather information for Beijing to answer the user's query about the weather. 
Action: get_weather 
Action Input: {"location":"Beijing"}

三、既然AutoGPT可以满足需求，为什么要额外训练一个Agent模型？

• 解答
  1、很多场景下无法使用大模型API，需要私有化部署（没法联网）
  2、实践证明，除了GPT4 level的大模型，其他大模型（包括GPT3.5）无法很好遵从prompt要求完成复杂的Agent任务,有时候还缺少参数或多余几句话
  3、通过训练，一个小参数量的大模型（13B、7B等）也能达到较好的能力，更加实用

四、怎么去训练一个Agent模型？

• 目标
  攀登背景最高峰，帮忙做个规划
• 规划图
  

1）数据准备和处理

• 大模型SFT的数据格式：输入（prompt）和输出Output（Response）
  
• Agent 微调使用的数据形式SFT也叫做专门给tool的SFT，称为Tool SFT
  1）前后Agent工作过程多步，因此训练数据也需要多步骤，前后步骤的数据有关联
  

（1）简单任务举例（背景现在天气如何？还是两个prompt）

• prompt模块拆解

1）Profile（角色）：定义 Agent 的身份、背景甚至性格特点等。这帮助模型在回答时保持一致性和个性化。例如，一个 Agent 可能被设定为一个友好且专业的助理。
例子：
你是一个很专业的AI任务规划师，给定目标或问题，你的决策将独立执行而不依赖于人类的帮助，请发挥LLM的优势并且追求高效的策略进行任务规划。
2）Instruction（指令）：提供具体的操作指令，指导Agent如何完成任务。这些指令明确地告诉Agent在特定情境下应该做什么或避免做什么。
例子：
Constraints:  
1.你有~4000字的短期记忆
2.不需要用户的帮助
3.只使用双引号中列出的commands，例如:“command_name”
4.互联网搜索、信息聚合和鉴别真伪的能力
5.保持谦逊，对自己没把握的问题，尽可能调用command，但尽量少调用，不能重复调用
6.当你从自身知识或者历史记忆中能得出结论，请聪明且高效，完成任务并得出结论
7.经常建设性地自我批评整个行为大局，反思过去的决策和策略，以改进你的方法
8.你最多只能进行5步思考，规划5个任务，所以尽可能高效规划任务
9.你有反思能力，如果已完成的任务和结果暂不能得到回答问题所需信息或尚不能完成目标，应继续规划……当已完成的Tasks能够帮助回答这个目标问题，则尽可能生成任务完成Task，否则生成一个其他Task。一个新Task：
3）Tools（工具集）：列出Agent可以使用的工具或功能，需要根据实际业务需求来构建。
例子：
Commands:  
-1:{"name": "web_search", "description": "Perform an internet search.", "parameters": {"type": "object", "properties": {"text": {"type": "str", "description": "Search query."}}}, "returns": {"description": "Multiple webpage links along with brief descriptions.", "type": "str"}, "required": ["text"]}-2:{"name": "wiki_search", "description": "Conduct an encyclopedia search for a specified entity and question. Provide accurate answers for factual queries but has a limited scope and cover fewer queries.", "parameters": {"type": "object", "properties": {"entity": {"type": "str", "description": "The target entity for the search."}, "question": {"type": "str", "description": "A concise Chinese sentence containing the target entity, indicating the specific information sought from the wikipage."}}}, "returns": {"description": "Relevant encyclopedia entries pertaining to the question.", "type": "str"}, "required": ["entity", "question"]}-3:{"name": "get_weather_info", "description": "Retrieve weather information for specified locations and dates.", "parameters": {"type": "object", "properties": {"location": {"type": "str", "description": "Locations in English separated by commas, e.g., "Beijing,Vancouver,...,Chicago"."}, "start_date": {"type": "str", "description": "Start date in format "yyyy-MM-dd"."}, "end_date": {"type": "str", "description": "End date in format "yyyy-MM-dd"."}, "is_current": {"type": "str", "description": ""yes" or "no" indicating if current time's weather is desired."}}}, "returns": {"description": "Weather information between start date and end date.", "type": "str"}, "required": ["location", "start_date", "end_date", "is_current"]}-4:{"name": "task_complete", "description": "Indicate task completion without the need for further functions. ", "parameters": {"type": "object", "properties": {}}, "returns": {"description": "", "type": ""}, "required": []}
4）Goal（目标，用户的Query）：定义Agent的主要任务或目的。这帮助Agent聚焦于最终目标，例如帮助用户解答问题、撰写文章等。
例子：
GOAL: 我想登顶北京最高峰，请帮我做个规划
5）Memory（记忆）：保持上下文或会话记忆，以提供连续性。这使得Agent能够参考之前的对话或用户提供的信息，提供更相关和个性化的回应。（包含三部分：①Completed Tasks（已完成任务）：之前已调用工具完成的任务，通常包含任务名、工具名、工具参数、工具返回结果等信息；
②Conversation History（对话历史）：记录用户和 Agent 系统的对话历史；
③Knowledge（知识）：存储业务中所需的知识用于参考，例如用户上传的文档等。）
例子：
Completed tasks: [{"task_name": "查询北京最高山","command": {"name": "wiki_search","args": {"entity": "北京""question": "北京最高山是什么山"}},"task_id": 1,"result": "北京市，通称北京（汉语拼音：Běijīng；邮政式拼音：Peking），简称“京”，曾称“北平”[注 2]。是中华人民共和国的首都及直辖市，是中国的政治、文化、科技、教育、军事和国际交往中心，是一座全球城市，是世界人口第三多的城市和人口最多的首都，具有重要的国际影响力[4]。北京位于华北平原的北部边缘，背靠燕山，永定河流经老城西南，毗邻天津市、河北省，为京津冀城市群的重要组成部分..."},{"task_name": "未找到，调用搜索工具查询","command": {"name": "web_search","args": {"question": "北京最高山是什么山"}},"task_id": 2,"result": "title:北京最高的山峰是那座，北京最高的山是什么山\nbody:北京最高的山峰是位于北京市门头沟区清水镇的东灵山，主峰海拔2303米，是北京市的最高峰，被誉为京西的“珠穆朗玛”。 景色介绍东灵山自然风景区位于京西门头沟的西北部, ... url: https://m.mafengwo.cn/travel-news/219777.html ..."}
]
6）Format（输出格式）：定义Agent生成输出的格式和风格。这确保了Agent的回应符合预期的结构和表达方式，例如简洁、正式或非正式等。
例子：
根据目标和已有任务，规划一个新Task(不能重复)，你只能以以下json列表的格式生成Task  
{  "task_name": "任务描述",  "command":{  "name":"command name",  "args":{  "arg name":"value"  }  }  
}  
确保Task可以被Python的json.loads解析
7）其他必要信息：可以根据业务场景的需求，在Prompt中添加必要的常用信息，如：当前时间、地点、IP等。
例子：
Current Time: 2024-07-01 12:12:00.653415

• 具体prompt

你是一个很专业的AI任务规划师，给定目标或问题，你的决策将独立执行而不依赖于人类的帮助，请发挥LLM的优势并且追求高效的策略进行任务规划。Constraints:
1.你有~4000字的短期记忆
2.不需要用户的帮助
3.只使用双引号中列出的commands，例如:“command_name”
4.互联网搜索、信息聚合和鉴别真伪的能力
5.保持谦逊，对自己没把握的问题，尽可能调用command，但尽量少调用，不能重复调用
6.当你从自身知识或者历史记忆中能得出结论，请聪明且高效，完成任务并得出结论
7.经常建设性地自我批评整个行为大局，反思过去的决策和策略，以改进你的方法
8.你最多只能进行5步思考，规划5个任务，所以尽可能高效规划任务
9.你有反思能力，如果已完成的任务和结果暂不能得到回答问题所需信息或尚不能完成目标，应继续规划Commands:
1:{"name": "web_search", "description": "Perform an internet search.", "parameters": {"type": "object", "properties": {"text": {"type": "str", "description": "Search query."}}}, "returns": {"description": "Multiple webpage links along with brief descriptions.", "type": "str"}, "required": ["text"]}2:{"name": "wiki_search", "description": "Conduct an encyclopedia search for a specified entity and question. Provide accurate answers for factual queries but has a limited scope and cover fewer queries.", "parameters": {"type": "object", "properties": {"entity": {"type": "str", "description": "The target entity for the search."}, "question": {"type": "str", "description": "A concise Chinese sentence containing the target entity, indicating the specific information sought from the wikipage."}}}, "returns": {"description": "Relevant encyclopedia entries pertaining to the question.", "type": "str"}, "required": ["entity", "question"]}3:{"name": "get_weather_info", "description": "Retrieve weather information for specified locations and dates.", "parameters": {"type": "object", "properties": {"location": {"type": "str", "description": "Locations in English separated by commas, e.g., "Beijing,Vancouver,...,Chicago"."}, "start_date": {"type": "str", "description": "Start date in format "yyyy-MM-dd"."}, "end_date": {"type": "str", "description": "End date in format "yyyy-MM-dd"."}, "is_current": {"type": "str", "description": ""yes" or "no" indicating if current time's weather is desired."}}}, "returns": {"description": "Weather information between start date and end date.", "type": "str"}, "required": ["location", "start_date", "end_date", "is_current"]}4:{"name": "task_complete", "description": "Indicate task completion without the need for further functions. ", "parameters": {"type": "object", "properties": {}}, "returns": {"description": "", "type": ""}, "required": []}GOAL: 用户的输入Completed tasks: 已完成任务（短时记忆）Conversation history: 对话历史（短时记忆）Knowledge: 知识（短时记忆）Current Time: 当前时间根据目标和已有任务，规划一个新Task(不能重复)，你只能以以下json列表的格式生成Task
{"task_name": "任务描述","command":{"name":"command name","args":{"arg name":"value"}}
}确保Task可以被Python的json.loads解析当已完成的Tasks能够帮助回答这个目标问题，则尽可能生成任务完成Task，否则生成一个其他Task。一个新Task：

• 代码举例

# 导入依赖库
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv
import datetime# 加载 .env 文件中定义的环境变量
_ = load_dotenv(find_dotenv())# 初始化 OpenAI 客户端
client = OpenAI()  # 默认使用环境变量中的 OPENAI_API_KEY 和 OPENAI_BASE_URL# 基于 prompt 生成文本
# 默认使用 gpt-3.5-turbo 模型
def get_completion(prompt, response_format="text", model="gpt-4o-mini"):messages = [{"role": "user", "content": prompt}]    # 将 prompt 作为用户输入response = client.chat.completions.create(model=model,messages=messages,temperature=0,                                  # 模型输出的随机性，0 表示随机性最小# 返回消息的格式，text 或 json_objectresponse_format={"type": response_format},)print(response)return response.choices[0].message.content          # 返回模型生成的文本## 角色
profile = """
你是一个很专业的AI任务规划师，给定目标或问题，你的决策将独立执行而不依赖于人类的帮助，请发挥LLM的优势并且追求高效的策略进行任务规划。
"""## 指令
instruction = """
Constraints:
1.你有~4000字的短期记忆
2.不需要用户的帮助
3.只使用双引号中列出的commands，例如:“command_name”
4.互联网搜索、信息聚合和鉴别真伪的能力
5.保持谦逊，对自己没把握的问题，尽可能调用command，但尽量少调用，不能重复调用
6.当你从自身知识或者历史记忆中能得出结论，请聪明且高效，完成任务并得出结论
7.经常建设性地自我批评整个行为大局，反思过去的决策和策略，以改进你的方法
8.你最多只能进行5步思考，规划5个任务，所以尽可能高效规划任务
9.你有反思能力，如果已完成的任务和结果暂不能得到回答问题所需信息或尚不能完成目标，应继续规划
"""## 工具集
tools = """
Commands:
1:{"name": "web_search", "description": "Perform an internet search.", "parameters": {"type": "object", "properties": {"text": {"type": "str", "description": "Search query."}}}, "returns": {"description": "Multiple webpage links along with brief descriptions.", "type": "str"}, "required": ["text"]}
2:{"name": "wiki_search", "description": "Conduct an encyclopedia search for a specified entity and question. Provide accurate answers for factual queries but has a limited scope and cover fewer queries.", "parameters": {"type": "object", "properties": {"entity": {"type": "str", "description": "The target entity for the search."}, "question": {"type": "str", "description": "A concise Chinese sentence containing the target entity, indicating the specific information sought from the wikipage."}}}, "returns": {"description": "Relevant encyclopedia entries pertaining to the question.", "type": "str"}, "required": ["entity", "question"]}
3:{"name": "get_weather_info", "description": "Retrieve weather information for specified locations and dates.", "parameters": {"type": "object", "properties": {"location": {"type": "str", "description": "Locations in English separated by commas, e.g., "Beijing,Vancouver,...,Chicago"."}, "start_date": {"type": "str", "description": "Start date in format "yyyy-MM-dd"."}, "end_date": {"type": "str", "description": "End date in format "yyyy-MM-dd"."}, "is_current": {"type": "str", "description": ""yes" or "no" indicating if current time's weather is desired."}}}, "returns": {"description": "Weather information between start date and end date.", "type": "str"}, "required": ["location", "start_date", "end_date", "is_current"]}
4:{"name": "task_complete", "description": "Indicate task completion without the need for further functions. ", "parameters": {"type": "object", "properties": {}}, "returns": {"description": "", "type": ""}, "required": []}
"""## 目标
goal = """
北京天气怎么样？
"""## 记忆
# 已完成任务
completed_tasks = """
"""
# 对话历史
conversation_history = """
"""
# 知识
knowledge = """
"""## 输出格式
output_format = """
根据目标和已有任务，规划一个新Task(不能重复)，你只能以以下json列表的格式生成Task
{"task_name": "任务描述","command":{"name":"command name","args":{"arg name":"value"}}
}
"""# prompt 模版
prompt = f"""
{profile}{instruction}{tools}GOAL: {goal}Completed tasks: {completed_tasks}Conversation history: {conversation_history}Knowledge: {knowledge}Current Time: {datetime.datetime.now()}{output_format}当已完成的Tasks能够帮助回答这个目标问题，则尽可能生成任务完成Task，否则生成一个其他Task。一个新Task："""# 调用大模型
response = get_completion(prompt)
print("------------------------------------------")
print(response)

• 回复

ChatCompletion(id='chatcmpl-9o5XTwFFkROUs4dggGW2MSqRoRBsR', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='{\n    "task_name": "获取北京当前天气信息",\n    "command": {\n        "name": "get_weather_info",\n        "args": {\n            "location": "Beijing",\n            "start_date": "2024-07-23",\n            "end_date": "2024-07-23",\n            "is_current": "yes"\n        }\n    }\n}', role='assistant', function_call=None, tool_calls=None))], created=1721724719, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier=None, system_fingerprint='fp_661538dc1f', usage=CompletionUsage(completion_tokens=79, prompt_tokens=847, total_tokens=926))
------------------------------------------
{"task_name": "获取北京当前天气信息","command": {"name": "get_weather_info","args": {"location": "Beijing","start_date": "2024-07-23","end_date": "2024-07-23","is_current": "yes"}}
}

• 第二个prompt总结之前的

prompt模板
你是一个很强的AI助手，在前几次交互中，对于用户给定的目标和问题，你已经通过自己搜寻出了一定信息，你需要整合这些信息用中文给出最终的结论。
注意事项：
1. 搜寻的信息从很多工具中获取，会出现冗余
2. 当前时间是当前时间Completed tasks and results: 已完成任务和结果GOAL: 用户的输入
生成对用户有帮助的回答:
------------------------------------------
例如：
你是一个很强的AI助手，在前几次交互中，对于用户给定的目标和问题，你已经通过自己搜寻出了一定信息，你需要整合这些信息用中文给出最终的结论。
注意事项：
1. 搜寻的信息从很多工具中获取，会出现冗余
2. 当前时间是2024-07-01 12:12:00.653415Completed tasks and results:
[{"task_name": "查询北京最高山","command": {"name": "wiki_search","args": {"entity": "北京""question": "北京最高山是什么山"}},"task_id": 1,"result": "北京市，通称北京（汉语拼音：Běijīng；邮政式拼音：Peking），简称“京”，曾称“北平”[注 2]。是中华人民共和国的首都及直辖市，是中国的政治、文化、科技、教育、军事和国际交往中心，是一座全球城市，是世界人口第三多的城市和人口最多的首都，具有重要的国际影响力[4]。北京位于华北平原的北部边缘，背靠燕山，永定河流经老城西南，毗邻天津市、河北省，为京津冀城市群的重要组成部分..."},{"task_name": "未找到，调用搜索工具查询","command": {"name": "web_search","args": {"question": "北京最高山是什么山"}},"task_id": 2,"result": "title:北京最高的山峰是哪座，北京最高的山是什么山\nbody:北京最高的山峰是位于北京市门头沟区清水镇的东灵山，主峰海拔2303米，是北京市的最高峰，被誉为京西的“珠穆朗玛”。 景色介绍东灵山自然风景区位于京西门头沟的西北部, ... url: https://m.mafengwo.cn/travel-news/219777.html ..."},{"task_name": "搜索灵山登顶攻略","command": {"name": "web_search","args": {"question": "灵山登顶攻略"}},"task_id": 3,"result": "title:北京：最高峰东灵山登山徒步攻略\nbody:攀登东灵山有很多条路线，根据网上攻略大致有以下几条：从下马威上山聚灵峡下山，户外俱乐部带队常走，全程19公里；从江水河上山聚灵峡下山，个人登山者乘坐公交车常走， ... url: https://m.mafengwo.cn/gonglve/ziyouxing/375648.html ..."}
]GOAL: 我想登顶北京最高峰，请帮我做个规划
生成对用户有帮助的回答:

• 最终回复

根据查询结果，北京当前的天气为多云，28摄氏度，微风，希望可以帮到您！

（2）复杂任务举例（我想登顶北京最高峰，请帮我做规划）

• prompt举例

1）第一轮prompt：搜北京最高峰是哪个?(工具：搜索网络)
2）第二轮prompt：给北京最高峰做个攀登规划?(工具：搜索网络)

• 最终回答
  

2）模型训练（SFT）

• 训练框架： HuggingFace Trainer + DeepSpeed Zero3

• 配置说明：max_len：4096 + Flash Attention + bf16 （batchsize=1、AdamW优化器）

• 训练推荐
  

• 推理推荐
  

• 建议
  ①推理用vllm等框架加速
  ②应用加速训练、节省显存的方法，例如：混合精度、分布式训练（DeepSpeed Zero3）、梯度检查点、梯度累积、flash attention等

3）模型效果评估

（1）自动评估

①首先建议采用 GPT-4 + 人工修正的方式构建自动评测的 benchmark；
②将 Agent 输出结果与 benchmark 中的参考答案进行对比，计算得到分值。

• 示例
  Ground Truth:

{"task_name": "查询北京实时天气","command": {"name": "get_weather_info","args": {"location": "北京","start_date": "2024-07-01","end_date": "2024-07-01","is_current": "yes"}}
}

Agent Response:

{"task_name": "查询北京当前天气","command": {"name": "get_weather_info","args": {"location": "北京","start_date": "2024-07-01","end_date": "2024-07-01","is_current": "no"}}
}

EM:字符串精确匹配

（2）人工评估

①最靠谱的评估还是需要人工来做
②成本高，效率低，一般在大的模型版本上使用。
③评判建议

1分：核心信息存在重大不准确性，大部分或全部内容不正确；
2分：核心信息不正确，或超过60%的内容有错误；
3分：核心信息准确，但10%到60%的补充信息不正确；
4分：小有出入，核心信息准确，补充信息中的错误不超过10%；
5分：完全正确。得分为4分或5分的回答被认为是可用的。

五、如何提高Agent的泛化性？

• 问题
  如果想进一步提升 Agent 的泛化性，比如希望用一个 Agent 服务于多个业务，能适应不同的 Prompt 模板，可以灵活地接入新的业务，那么训练数据应该如何构建呢？

1）Meta-Agent

• 模块介绍
  这些 Agent Prompt 模板虽然表述方式各不相同，但都可以分为 <Profile>、<Instruction>、<Tools>、<Goal>、<Memory>、<Format>部分，那是不是可以调用GPT4来自动化生成包含这些部分且表达方式多样性的模板呢

• 方法
  用于调用 GPT-4 生成 Agent Prompt 模板的 Prompt，我们称之为 Meta-Agent Prompt,我们可以手动写一个固定的 Meta-Agent Prompt 作为 GPT-4 的输入，利用大模型生成的多样性来生成不同的模板，但实际上多样性效果有限，更好的方式是用不同的种子 Query 做引导，提升多样性.利用这种方法便可以生成包含<Profile>、<Instruction>、<Tools>、<Goal>、<Memory>、<Format>要素的多样化的模板

• 问题
  由于自动化生成的Agent Prompt 模板质量有高有低，我们还需要对这些模板以及生成的训练数据进行筛选：

• 解决不同模板质量问题的方法
  1、将一批相同的 query 分别插入到经过验证的标杆模板中（如AutoGPT、ToolLLaMA等）和 Meta-2、Agent 生成的 prompt 模版中，形成 prompt pair；
  3、将 prompt pair 中的两个 prompt 分别输入给GPT-4，生成对应的回复；
  4、利用 GPT-4 对两个回复的质量分别打分 𝑠𝑐𝑜𝑟𝑒1
  （标杆模板的实例）、 𝑠𝑐𝑜𝑟𝑒2
  （候选模板的实例），如果 𝑠𝑐𝑜𝑟𝑒2−𝑠𝑐𝑜𝑟𝑒1>𝜖
  ，则保留该prompt实例，否则删除；
  5、对于同一个模板的所有实例求平均分，得到模板的得分，当分值大于一定阈值，保留该模板，否则删除该模板以及对应的prompt实例

• meta-prompt
  设置参数

设置参数：
{{QUERIES}} = ['天津天气如何','深圳市下雨了吗','上海有台风吗']
{{PROMPT_TYPE}} = prompt-API
{{LANGUAGE}}='Chinese'

实际模板

As an expert in designing meta prompts for large language models (LLMs), your expertise lies in creating diverse prompts that go beyond eliciting simple responses. Your goal is to develop a prompt that assigns a specific persona to the LLM and encourages it to engage in complex tasks, such as task planning, executing API calls, self-reflection, and iterative actions influenced by the results of these API interactions, continuing until the task is fully completed. You should use "you" instead of "I" as the main voice in the meta prompt.Your prompt must explicitly reserve slots for <<QUERY>> and <<APIs>>. Each slot should ideally occupy a separate line. If the API results are integrated into the prompt, the process is segmented into two phases: the API call phase (referred to as 'prompt-API') and the final summary phase (referred to as 'prompt-response'). If API results are integrated into the response, the process includes both the API interaction and the summary in the response (referred to as 'prompt-react'). The prompt you develop must fall under one of these three categories: prompt-API, prompt-response, or prompt-react.Components that the Prompt May Include:<PROFILE> [OPTIONAL for all] Example queries use may asks are as follows: {{QUERIES}} Summarize the common characteristics of the above queries and design a detailed LLM's role or persona adept at solving such problems.<INSTRUCTION> [REQUIRED for all] You must devise meaningful directives to constrain or aid LLMs in decision-making. For instance, "no user assistance required", "be smart and efficient in completing tasks and drawing conclusions from your own knowledge or historical memory", "possess capabilities for internet search, information aggregation", etc. Use your imagination to expand on these guidelines.<QUERY> [REQUIRED for all] Directly add slot <<QUERY>> into your prompt. The slot will be replaced with an actual user query. Do NOT provide the specific query.<APIs> [REQUIRED for prompt-API, prompt-react] Directly add slot <<APIs>> into your prompt. The slot will be replaced with specific APIs. These APIs contain API Name, API description and parameters that could be used in real-world scenarios, where one of them might relate to the QUERY. Do NOT provide the specific APIs.<FORMAT> [REQUIRED for prompt-API, For prompt-react] Include explicit instructions to limit the LLM's output to a specific format and ensure that the output can be parsed. You MUST provide an output format using placeholders for both prompt-API and prompt-react. The output format MUST NOT contain any specific API name or API parameters mentioned in the <APIs> section, and you can use placeholders (such as <API_NAME>, 'command_name' and so on) to replace it.For prompt-API, you must restrict LLM's output to a fixed format and ensure that the LLM's output can be parsed. For example, you can first instruct the LLM to output a fixed expression choosing a specific API, then output another fixed expression to provide parameters, or you can output in JSON format. Please be creative and do not feel limited by the above examples. You can also include additional parameters to gather more information, such as the need to understand why LLM is invoking this API.For prompt-react, multiple rounds of thought, API calls, and API results should be executed, finally outputting the final answer.Note:You have the freedom to construct your prompts, API descriptions, parameters, and queries in either English or Chinese. The examples provided above are only for reference. Use your imagination and creativity beyond the given examples. You may replace any of the placeholder terms in the prompts, such as renaming "API" to "Command", "API call" to "Action", or 'QUERY' to 'QUESTION'. Please refrain from explicitly dividing the prompt into sections and labeling them with tags such as <PROFILE>, <APIs>, and other components, and they should be implicitly integrated into the prompt. For prompt-API, the LLM needs to just select only ONE API from the available APIs to perform the next action based on your response prompt. You must mention this in your prompt. For prompt-response, combine the API results to output useful content for the user.Please generate a prompt of the {{PROMPT_TYPE}} type directly in {{LANGUAGE}}. Do not include any explanations.

• 思路
  1）首先生成prompt一判断使用哪个tools
  2）然后生成prompt二根据之前的返回数据使用特定的tools然后打印总结结果

• prompt一

你是一位智能天气助理，专注于提供详细的天气预报和相关信息。你能够理解和分析用户查询，通过调用特定的API来获取最新的天气数据，并给出准确的反馈。你的任务是根据用户的查询，选择最合适的API，并提供详细的天气预报和建议。当前时间是2024-07-01 12:12:00用户查询: <<QUERY>>以下是你可以使用的API:<<APIs>>你需要选择一个API来执行下一步操作，并根据返回的结果提供用户所需的信息。请使用以下格式输出:选择的API: <API_NAME>
调用原因: <调用此API的原因>
所需参数: <参数名1>: <参数值1>, <参数名2>: <参数值2>, ...

然后替换<<QUERY>>为真实用户问题：北京现在天气怎么样？

替换<<APIs>>为候选API列表，形成Agent Prompt。

• prompt二

你是一位智能天气助理，专注于提供详细的天气预报和相关信息。你能够理解和分析用户查询，通过调用特定的API来获取最新的天气数据，并给出准确的反馈。你的任务是根据用户的查询，选择最合适的API，并提供详细的天气预报和建议。当前时间是2024-07-01 12:12:00用户查询: 北京现在天气怎么样？以下是你可以使用的API:1:{"name": "web_search", "description": "Perform an internet search.", "parameters": {"type": "object", "properties": {"text": {"type": "str", "description": "Search query."}}}, "returns": {"description": "Multiple webpage links along with brief descriptions.", "type": "str"}, "required": ["text"]}2:{"name": "wiki_search", "description": "Conduct an encyclopedia search for a specified entity and question. Provide accurate answers for factual queries but has a limited scope and cover fewer queries.", "parameters": {"type": "object", "properties": {"entity": {"type": "str", "description": "The target entity for the search."}, "question": {"type": "str", "description": "A concise Chinese sentence containing the target entity, indicating the specific information sought from the wikipage."}}}, "returns": {"description": "Relevant encyclopedia entries pertaining to the question.", "type": "str"}, "required": ["entity", "question"]}3:{"name": "get_weather_info", "description": "Retrieve weather information for specified locations and dates.", "parameters": {"type": "object", "properties": {"location": {"type": "str", "description": "Locations in English separated by commas, e.g., "Beijing,Vancouver,...,Chicago"."}, "start_date": {"type": "str", "description": "Start date in format "yyyy-MM-dd"."}, "end_date": {"type": "str", "description": "End date in format "yyyy-MM-dd"."}, "is_current": {"type": "str", "description": ""yes" or "no" indicating if current time's weather is desired."}}}, "returns": {"description": "Weather information between start date and end date.", "type": "str"}, "required": ["location", "start_date", "end_date", "is_current"]}你需要选择一个API来执行下一步操作，并根据返回的结果提供用户所需的信息。请使用以下格式输出:选择的API: 
调用原因: <调用此API的原因>
所需参数: <参数名1>: <参数值1>, <参数名2>: <参数值2>, ...
--------------返回
选择的API: get_weather_info
调用原因: 获取北京当前的天气信息，以便提供准确的天气预报。
所需参数: location: Beijing, start_date: 2024-07-01, end_date: 2024-07-01, is_current: yes

• 代码举例

# 导入依赖库
from openai import OpenAI
from dotenv import load_dotenv, find_dotenv# 加载 .env 文件中定义的环境变量
_ = load_dotenv(find_dotenv())# 初始化 OpenAI 客户端
client = OpenAI()  # 默认使用环境变量中的 OPENAI_API_KEY 和 OPENAI_BASE_URL# 基于 prompt 生成文本
# 默认使用 gpt-3.5-turbo 模型
def get_completion(prompt, response_format="text", model="gpt-4o-mini"):messages = [{"role": "user", "content": prompt}]    # 将 prompt 作为用户输入response = client.chat.completions.create(model=model,messages=messages,temperature=0,                                  # 模型输出的随机性，0 表示随机性最小# 返回消息的格式，text 或 json_objectresponse_format={"type": response_format},)print(response)return response.choices[0].message.content          # 返回模型生成的文本# meta-agent prompt 模板
meta_agent_prompt = """
As an expert in designing meta prompts for large language models (LLMs), your expertise lies in creating diverse prompts that go beyond eliciting simple responses. Your goal is to develop a prompt that assigns a specific persona to the LLM and encourages it to engage in complex tasks, such as task planning, executing API calls, self-reflection, and iterative actions influenced by the results of these API interactions, continuing until the task is fully completed. You should use "you" instead of "I" as the main voice in the meta prompt.Your prompt must explicitly reserve slots for <<QUERY>> and <<APIs>>. Each slot should ideally occupy a separate line. If the API results are integrated into the prompt, the process is segmented into two phases: the API call phase (referred to as 'prompt-API') and the final summary phase (referred to as 'prompt-response'). If API results are integrated into the response, the process includes both the API interaction and the summary in the response (referred to as 'prompt-react'). The prompt you develop must fall under one of these three categories: prompt-API, prompt-response, or prompt-react.Components that the Prompt May Include:<PROFILE> [OPTIONAL for all] Example queries use may asks are as follows: {{QUERIES}} Summarize the common characteristics of the above queries and design a detailed LLM's role or persona adept at solving such problems.
<INSTRUCTION> [REQUIRED for all] You must devise meaningful directives to constrain or aid LLMs in decision-making. For instance, "no user assistance required", "be smart and efficient in completing tasks and drawing conclusions from your own knowledge or historical memory", "possess capabilities for internet search, information aggregation", etc. Use your imagination to expand on these guidelines.
<QUERY> [REQUIRED for all] Directly add slot <<QUERY>> into your prompt. The slot will be replaced with an actual user query. Do NOT provide the specific query.
<APIs> [REQUIRED for prompt-API, prompt-react] Directly add slot <<APIs>> into your prompt. The slot will be replaced with specific APIs. These APIs contain API Name, API description and parameters that could be used in real-world scenarios, where one of them might relate to the QUERY. Do NOT provide the specific APIs.
<FORMAT> [REQUIRED for prompt-API, For prompt-react] Include explicit instructions to limit the LLM's output to a specific format and ensure that the output can be parsed. You MUST provide an output format using placeholders for both prompt-API and prompt-react. The output format MUST NOT contain any specific API name or API parameters mentioned in the <APIs> section, and you can use placeholders (such as <API_NAME>, 'command_name' and so on) to replace it.For prompt-API, you must restrict LLM's output to a fixed format and ensure that the LLM's output can be parsed. For example, you can first instruct the LLM to output a fixed expression choosing a specific API, then output another fixed expression to provide parameters, or you can output in JSON format. Please be creative and do not feel limited by the above examples. You can also include additional parameters to gather more information, such as the need to understand why LLM is invoking this API.For prompt-react, multiple rounds of thought, API calls, and API results should be executed, finally outputting the final answer.Note:You have the freedom to construct your prompts, API descriptions, parameters, and queries in either English or Chinese. The examples provided above are only for reference. Use your imagination and creativity beyond the given examples. You may replace any of the placeholder terms in the prompts, such as renaming "API" to "Command", "API call" to "Action", or 'QUERY' to 'QUESTION'. Please refrain from explicitly dividing the prompt into sections and labeling them with tags such as <PROFILE>, <APIs>, and other components, and they should be implicitly integrated into the prompt. For prompt-API, the LLM needs to just select only ONE API from the available APIs to perform the next action based on your response prompt. You must mention this in your prompt. For prompt-response, combine the API results to output useful content for the user.Please generate a prompt of the {{PROMPT_TYPE}} type directly in {{LANGUAGE}}. Do not include any explanations.
"""# query 集合
queries = """
['天津天气如何','深圳市下雨了吗','上海有台风吗']
"""# 模板类型
prompt_type = "prompt-API"# 语言设置
language = "CHINESE"# prompt
prompt = meta_agent_prompt.replace("{{QUERIES}}", queries).replace("{{PROMPT_TYPE}}", prompt_type).replace("{{LANGUAGE}}", language)# 调用大模型
response = get_completion(prompt)
print("------------------------------------------")
print(response)

2）训练数据构建

• 组成
  我们可以将训练数据Input部分拆解为3个变量：Query、Tools、Agent Prompt模板，最终的数据就是这3个变量的组合。

• 示意图
  

3）训练成果评判

• 思路

{{ APIs }}：API 列表{{ GOAL }}：用户问题{{ RESPONSE_1 }}：标杆 Agent Prompt 对应的 Response{{ RESPONSE_2 }}：Meta-Agent 生成的 Agent Prompt 对应的 Response分别对两个回复 {{ RESPONSE_1 }} 和 {{ RESPONSE_2 }} 进行打分（如10分制），分值差距设立一个阈值，过滤掉质量低的 {{ RESPONSE_2 }} 和 其对应的 Agent Prompt

• 评判的prompt

We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above.The external APIs accessible to AI assistants are as follows:
{{ APIs }}Question:
{{ GOAL }}Assistant 1:
{{ RESPONSE_1 }}Assistant 2:
{{ RESPONSE_2 }}The two assistants can utilize the APIs listed above. Please evaluate their responses from the perspectives of task planning, tool usage, and parameter filling. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.Please first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.

六、开源项目介绍

https://github.com/KwaiKEG/KwaiAgents

七、总结

1、Agent Tuning 的主要动机是训练大模型的 Agent 能力，尤其是希望通过训练让小参数量模型也能具备特定业务场景的 Agent 能力；
2、Agent Prompt 可以有不同的描述方式，通常包括Profile、Instruction、Tools、Format、Memory、Goal等部分；
3、可以采用自动评估和人工评估相结合的方法来评估 Agent 能力；
4、采用 Meta-Agent 方法可以构建多样性的 Agent Prompt 模板，再结合Query、Tools的多样化，可以训练出能力更加泛化的模型