CodexQA

行业与实践研究与基准测试

BizBench:长程商业智能体的参考文献与附录

CodexQA 团队阅读约 25 分钟

承接正文,按原文顺序收录四十一条参考文献,以及附录 A 至 D:客户与智能体提示、模拟器协议、两条执行轨迹,以及任务、文档和工具使用统计。

本文目录

译注:本篇承接《BizBench:长程商业智能体的隐藏挑战》,从参考文献起收录附录 A 至 D。正文篇为 bizbench-long-horizon-business-zh。

参考文献

下面四十一条按原文顺序排列。每条先给引用键和中文题名,再保留作者、年份、出处和链接的原文,不增删条目,文献条目一条不漏。

  • Barres et al. [2025]。Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.《tau 平方基准:在双控制环境中评测对话智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. \tau^{2}-bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506.07982. arXiv preprint arXiv:2506.07982.
  • Boisvert et al. [2024]。Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault《WorkArena++:面向组合规划与基于推理的常识办公任务》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. WorkArena++: Towards compositional planning and reasoning-based common knowledge work tasks. In Advances in Neural Information Processing Systems, volume 37, pages 5996–6051. Curran Associates, Inc., 2024. doi: 10.52202/079017-0195. URL https://doi.org/10.52202/079017-0195.
  • Budzianowski et al. [2018]。Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva,《MultiWOZ:面向任务型对话建模的大规模多领域巫师数据集》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 5016–5026, 2018.
  • DeepSeek-AI [2026a]。DeepSeek-AI.《DeepSeek-V4.1-Flash:推进键值缓存压缩的极限》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:DeepSeek-AI. Deepseek-v4.1-flash: Pushing the limits of kv cache compression, 2026a. URL https://arxiv.org/abs/2609.19969.
  • DeepSeek-AI [2026b]。DeepSeek-AI.《DeepSeek-V4:走向高效的百万词元上下文智能》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026b. URL https://arxiv.org/abs/2606.19348.
  • Deng et al. [2026]。Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang,《SWE-Milestone:在持续软件演化上评测人工智能智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Gangda Deng, Zhaoling Chen, Zhongming Yu, Haoyang Fan, Yuhong Liu, Yuxin Yang, Dhruv Parikh, Rajgopal Kannan, Le Cong, Mengdi Wang, Qian Zhang, Viktor Prasanna, Xiangru Tang, and Xingyao Wang. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution, 2026. URL https://arxiv.org/abs/2603.13428. ICML 2026; arXiv:2603.13428.
  • Deng et al. [2023]。Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan《Mind2Web:走向面向网页的通用智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems, volume 36, pages 28091–28114. Curran Associates, Inc., 2023. doi: 10.52202/075280-1220. URL https://doi.org/10.52202/075280-1220.
  • Drouin et al. [2024]。Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel《WorkArena:网页智能体解决常识办公任务的能力如何》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11642–11662. PMLR, 2024. URL https://proceedings.mlr.press/v235/drouin24a.html.
  • Gulati et al. [2026]。Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen, and Vamse Kumar Subbiah.《早问、晚问、问对:澄清时机何时对长程智能体重要》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Anmol Gulati, Hariom Gupta, Elias Lumer, Sahil Sen, and Vamse Kumar Subbiah. Ask early, ask late, ask right: When does clarification timing matter for long-horizon agents?, 2026. URL https://arxiv.org/abs/2605.07937. arXiv preprint arXiv:2605.07937.
  • Huang et al. [2025]。Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang,《CRMArena:理解大语言模型智能体在真实环境中完成专业客户关系管理任务的能力》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. CRMArena: Understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3830–3850, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.naacl-long.194. URL https://aclanthology.org/2025.naacl-long.194/.
  • Huang et al. [2026]。Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal,《CRMArena-Pro:跨多样业务场景与交互对大语言模型智能体的整体评估》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. CRMArena-Pro: Holistic assessment of LLM agents across diverse business scenarios and interactions. Transactions on Machine Learning Research, 2026. URL https://arxiv.org/abs/2505.18878.
  • Jimenez et al. [2024]。Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir《SWE-bench:语言模型能否解决真实的 GitHub 问题》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, pages 54107–54157, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.
  • Koh et al. [2024]。Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu《VisualWebArena:在真实的视觉落地网页任务上评测多模态智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visually grounded web tasks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.50. URL https://aclanthology.org/2024.acl-long.50/.
  • Li et al. [2017a]。Feng-Lin Li, Minghui Qiu, Haiqing Chen, Xiongwei Wang, Xing Gao, Jun Huang,《AliMe Assist:用于创造创新电子商务体验的智能助手》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Feng-Lin Li, Minghui Qiu, Haiqing Chen, Xiongwei Wang, Xing Gao, Jun Huang, Juwei Ren, Zhongzhou Zhao, Weipeng Zhao, Lei Wang, Guwei Jin, and Wei Chu. AliMe Assist: An intelligent assistant for creating an innovative e-commerce experience. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 2495–2498. ACM, 2017a. doi: 10.1145/3132847.3133169. URL https://doi.org/10.1145/3132847.3133169.
  • Li et al. [2017b]。Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz.《端到端的任务完成神经对话系统》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. End-to-end task-completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 733–743, 2017b.
  • Li et al. [2026]。Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu,《Long-Horizon-Terminal-Bench:用基于稠密奖励的子任务检查测试智能体在长程终端任务上的极限》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, and Leowei Liang. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading, 2026. URL https://arxiv.org/abs/2607.08964. arXiv preprint arXiv:2607.08964.
  • Liu et al. [2024]。Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu,《AgentBench:把大语言模型作为智能体来评测》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. In International Conference on Learning Representations, pages 52989–53046, 2024. URL https://openreview.net/forum?id=zAdUB0aCTQ.
  • Lu et al. [2025]。Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping《ToolSandbox:面向大语言模型工具使用能力的有状态、对话式、交互式评测基准》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-naacl.65. URL https://aclanthology.org/2025.findings-naacl.65/.
  • Merrill et al. [2026]。Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj,《Terminal-Bench:在命令行界面的困难而真实的任务上评测智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, 2026. URL https://arxiv.org/abs/2601.11868. arXiv preprint arXiv:2601.11868.
  • Mialon et al. [2024]。Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann《GAIA:通用人工智能助手基准》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In International Conference on Learning Representations, pages 9025–9049, 2024. URL https://openreview.net/forum?id=fibxvahvs3.
  • Nam et al. [2026]。Youngeun Nam, Joeun Kim, Hwanjun Song, Susik Yoon, Jae-Gil Lee, and Byung Suk《TimelyRAG:面向重叠演化文档中时间关键问答的语义与时间混合检索》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Youngeun Nam, Joeun Kim, Hwanjun Song, Susik Yoon, Jae-Gil Lee, and Byung Suk Lee. TimelyRAG: Semantic-temporal hybrid retrieval for time-critical question answering in overlapping-evolving documents, 2026. URL https://arxiv.org/abs/2609.11572. arXiv preprint arXiv:2609.11572.
  • OpenAI [2026]。OpenAI.《GPT-6 Astra:新一代智能》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:OpenAI. GPT-6 Astra: A new generation of intelligence. OpenAI, September 2026. URL https://openai.com/index/gpt-6-astra/.
  • Ouyang et al. [2025]。Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin,《HoH:评测过时信息对检索增强生成之影响的动态基准》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Jie Ouyang, Tingyue Pan, Mingyue Cheng, Ruiran Yan, Yucong Luo, Jiaying Lin, and Qi Liu. HoH: A dynamic benchmark for evaluating the impact of outdated information on retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6036–6063, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.301. URL https://aclanthology.org/2025.acl-long.301/.
  • Raghu et al. [2021]。Dinesh Raghu, Shantanu Agarwal, Sachindra Joshi, and Mausam.《以流程图为据的任务型对话的端到端学习》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Dinesh Raghu, Shantanu Agarwal, Sachindra Joshi, and Mausam. End-to-end learning of flowchart grounded task-oriented dialogs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4348–4366, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.357. URL https://aclanthology.org/2021.emnlp-main.357/.
  • Ruan et al. [2024]。Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba,《用语言模型模拟的沙箱识别语言模型智能体的风险》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2309.15817.
  • Sakata et al. [2019]。Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi.《用查询与问题的相似度以及基于 BERT 的查询与答案相关性检索常见问题》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. FAQ retrieval using query-question similarity and BERT-based query-answer relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1113–1116. ACM, 2019. doi: 10.1145/3331184.3331326. URL https://doi.org/10.1145/3331184.3331326.
  • Schick et al. [2023]。Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria《Toolformer:语言模型可以教会自己使用工具》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-2997. URL https://doi.org/10.52202/075280-2997.
  • Shepard and Salimans [2026]。Daniel Shepard and Robin Salimans.《AutomationBench》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Daniel Shepard and Robin Salimans. AutomationBench, 2026. URL https://arxiv.org/abs/2604.18934. arXiv preprint arXiv:2604.18934.
  • Shi et al. [2026]。Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor《tau-Knowledge:在非结构化知识上评测对话智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, and Victor Barres. \tau-Knowledge: Evaluating conversational agents over unstructured knowledge, 2026. URL https://arxiv.org/abs/2603.04370. arXiv preprint arXiv:2603.04370.
  • Shinn et al. [2023]。Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu《Reflexion:带有言语强化学习的语言智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.52202/075280-0377. URL https://doi.org/10.52202/075280-0377.
  • Tong et al. [2026]。Yongqi Tong, Xiaoyun Feng, Lyuxin Xue, Jianshe Li, Xin Zhang, and Jiang-Ming《如何训练真实世界的硅基礼宾:把复杂业务流程内化到单一模型》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Yongqi Tong, Xiaoyun Feng, Lyuxin Xue, Jianshe Li, Xin Zhang, and Jiang-Ming Yang. How to train a real-world silicon concierge? internalizing complex business workflow to only OneModel. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 1303–1316, San Diego, California, USA, July 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-industry.91. URL https://aclanthology.org/2026.acl-industry.91/.
  • Wang et al. [2023]。Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu,《Voyager:使用大语言模型的开放式具身智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL https://arxiv.org/abs/2305.16291. arXiv preprint arXiv:2305.16291.
  • Xie et al. [2024]。Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng《OSWorld:为开放式计算机任务评测多模态智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, volume 37, pages 52040–52094. Curran Associates, Inc., 2024. doi: 10.52202/079017-1650. URL https://doi.org/10.52202/079017-1650.
  • Xu et al. [2025]。Frank (Fangzheng) Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain,《TheAgentCompany:在有后果的真实世界任务上评测大语言模型智能体》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Frank (Fangzheng) Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks. In Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc., 2025. doi: 10.52202/085713-0315. URL https://doi.org/10.52202/085713-0315.
  • Yang et al. [2025]。An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen《Qwen3 技术报告》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388.
  • Yao et al. [2022]。Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.《WebShop:用落地的语言智能体走向可扩展的真实网页交互》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, pages 20744–20757. Curran Associates, Inc., 2022. doi: 10.52202/068431-1508. URL https://doi.org/10.52202/068431-1508.
  • Yao et al. [2023a]。Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan《思维树:用大语言模型做深思熟虑的问题求解》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, 2023a. doi: 10.52202/075280-0517. URL https://doi.org/10.52202/075280-0517.
  • Yao et al. [2023b]。Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,《ReAct:协同语言模型中的推理与行动》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=WE_vluYUL-X.
  • Yao et al. [2025]。Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.《tau-bench:真实领域中工具、智能体与用户交互的基准》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. \tau-bench: A benchmark for tool-agent-user interaction in real-world domains. In International Conference on Learning Representations, pages 9965–10017, 2025. URL https://openreview.net/forum?id=roNSXZpUDN.
  • Zhou et al. [2024a]。Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang.《语言智能体树搜索统一语言模型中的推理、行动与规划》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62138–62160. PMLR, 2024a. URL https://proceedings.mlr.press/v235/zhou24r.html.
  • Zhou et al. [2024b]。Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar,《WebArena:用于构建自主智能体的真实网页环境》。这条文献按原文顺序著录,中文题名写在书名号内,作者、年份、场合和网址保持原文。其余著录按原文保留:Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, pages 15585–15606, 2024b. URL https://openreview.net/forum?id=oKn9c6ytLx.

附录 A 一个完整任务例子

图介绍了 Q03,它把回单核对与一项账户申请连接起来。面板(a)呈现工作流依赖,面板(b)说明相应的智能体交互。下面我们解释资金决策如何连接这些工作流。

表 5 中的三条路线,在 Acacia 的档案关闭之后借记同一笔 60 美元储备。在黄金轨迹中,客户明确把总费用上限从 50 美元提高到 60 美元,并批准路线 B,该路线在 \(t=8\) 为单次 50 美元的审核提交提供资金;申请在 \(t=18\) 获批,处于 \(t=19\) 的截止之内。从同一释放时间起,路线 A 在 \(t=18\) 入账,使十分钟的审核超出截止,而路线 C 比提交费少一分钱。这些比较由释放条款导出,而不是观察到的模型失败。客户既不允许充值,也不允许从无关账户转账。

表 5:同一笔 60 美元储备的三条释放路线。50 美元审核费是额外的。

释放路线净入账费用延迟
A:标准$60$010 分钟
B:同一周期$50$10立即
C:降低净额$49.99$10.01立即

每项任务包含:一个客户目标和开场请求;一个管辖私人事实与合作的客户场景;一个记录资源、历史、义务和预定事件的初始状态;对共享文档和可执行工具的访问;以及针对最终结果和执行历史的确定性接受标准。一条可回放的黄金轨迹和验证证据确立可行性,并检查评测器如何对待有效替代和违规。所展示的对话由黄金轨迹构造;标准作业程序图、黄金轨迹和可执行检查不提供给被评测的智能体。

附录 B 数据集统计

全部统计针对已发布的 50 项任务。黄金轨迹统计每项任务使用一条已验证轨迹。表 7 概括任务和共享环境。表 7 报告文档长度和黄金轨迹统计。

译注:原文先说表 7 概括任务和环境,紧接着的表题是表 6。下面按原文表号保留。

表 6:BizBench 中的任务与环境计数。

统计计数
任务50
始终可用的智能体工具9
可发现的业务能力601
用户模拟器工具10
知识文档789
文档正文词元总数598,827
可查询的数据库集合398
数据库记录2,994

表 7:文档长度与黄金轨迹统计。

指标均值中位数范围
知识库:每份文档
文档长度(词元)759.045244–10,755
黄金轨迹:每项任务
助手工具调用111.8100.551–282
用户模拟器工具调用28.621.510–74
阅读的不同文档5.452–15
阅读的不同业务记录29.923.54–96
已执行的业务请求14.2116–36

文档、工具和记录。 每份文档按其标识符计一次。词元计数使用文档正文和 cl100k_base 分词器,排除标题和元数据。工具计数区分始终可用的智能体工具、可发现的业务能力,以及跨产品门户的用户模拟器工具。数据库计数描述任务专属变更之前的共享初始数据库,包括背景记录和空集合;其 398 个可查询集合中有 227 个非空。私人客户文件和隐藏的事件载荷被排除。集合按查询资源名计数。

黄金轨迹。 助手和用户模拟器的工具调用分开计数,批量调用逐个计数。轨迹平均每项任务 111.8 次助手工具调用和 28.6 次用户模拟器工具调用。

黄金轨迹展示可行策略,而不是最短解。

每次分发调用对其目标能力计一次;解锁一项能力计为单独的工具调用。不同文档按文档标识符去重。不同记录统计成功的全记录读取,按资源名和记录标识符去重。已执行的业务请求统计不同的、成功执行的请求标识符。

附录 C 说明性业务案例

下面的例子聚焦 Q24 和 Q34 中的付款与交付决策。全部金额为美元;时间为逻辑分钟。

C.1 有限的净额交付保证(Q24)

两家供应商必须在现行银行指令下收到其发票的精确剩余余额。客户允许的银行费用总额至多 75 美元,包括早先转账的退回费用。该转账解决之后,剩余余额为 300 美元和 400 美元。一次性保证以固定费用提供精确净额交付;标准路线使收款人面临中介扣款,随后是单独收费的补足。

考虑把保证分配给 300 美元余额,并通过标准路线发送 400 美元余额。两家供应商最终都收到正确金额。然而,20 美元退回费、20 美元保证费、35 美元中介扣款和 5 美元补足费合计 80 美元。尽管最终付款正确,费用预算仍被超出。该案例要求在义务之间分配有限服务,并计入结算不同阶段发生的费用。

C.2 付款核对与交付窗口(Q34)

商户必须核对一笔已经支付的订单且不再收费,并交付三件兼容单元:一件在第 2 分钟前评级至少为 15,两件在第 5 分钟前评级至少为 10。全部需要同一插口。优先交付服务需要一分钟,但必须在第 0 分钟前预订;普通交付需要三分钟。

考虑修复商户映射,并传播两笔合计 10 美元的扣款和 1 美元退款,得到订单 9 美元的净付款,然后在第 1 分钟等待仓库结果。优先预订随后被拒绝,因为其窗口已经关闭。兼容库存被识别出来,较晚的两件按时到达,但较早的一件无法通过剩余服务满足截止。这一序列说明,为什么智能体必须把信息收集与即将到期的服务权利协调起来:正确的核对和合适的库存本身并不能完成商业目标。

附录 D 执行历史分析

D.1 模型执行长度与工具使用

我们分析默认条件下 DeepSeek-V4.1-Flash 的执行轨迹。表 8 把批量调用逐个计数,并分开助手与用户模拟器的工具调用。数据库查询使用 search_records 或 get_record。一次相同的重复数据库读取,与同一次运行中早先的读取具有相同的名称、参数和响应,排除观察已变化状态的查询。

表 8:默认条件下 DeepSeek-V4.1-Flash 每次运行的工具使用。

指标均值中位数第 90 百分位
助手工具调用191.3182287
用户模拟器工具调用28.22641
数据库查询111.1101180
阅读的不同文档20.12234
相同的重复数据库读取16.41432

成功运行平均有 13.7 次相同的重复数据库读取,失败运行为 18.5 次。复查未变化的记录是可测量的冗余,尽管有些复查可能是有意的验证。

D.2 用户模拟器的轮次可靠性

我们考察用户模拟器在客户侧操作之后是否把控制交还智能体。在一次成功的写入或等待之后,它必须先报告结果,然后才能使用另一个工具。我们把批量进行多次写入或等待,或在报告结果并交还控制之前继续使用工具,计为协议违规。

该比较使用同样的 50 项任务,并由 DeepSeek-V4.1-Flash 担任双方,对修订提示复用一次既有的完整评测。原始提示已经要求一次只写一次;修订澄清如何排列混合的问题和动作请求的优先级,以及何时交还控制。原始提示下有四项任务(8%)发生违规:两项涉及批量客户批准,两项涉及工具响应之后继续使用门户。修订提示下没有此类违规(0%;成对精确 McNemar \(p=0.125\))。这些观察促使我们在基准评测中使用修订提示。

D.3 达成目标但有过程违规的例子

默认条件下的单次运行审计包含两次 \(G=1\) 且 \(P=0\) 的执行。

被拒绝的过早执行(Q13)。 智能体协调保险结算和一笔储值订单付款。它在门户批准之前对付款调用 execute_request;工具返回 CUSTOMER_APPROVAL_REQUIRED,没有付款或余额变化。客户随后批准,智能体完成目标。这次过早尝试违反批准纪律:\(G=1\),\(P=0\),\(F=1\)。没有发生未经授权的转账。

实际的储备违反(Q18)。 客户需要利息更正和会计付款:第 0 分钟一笔 60 美元预聘费,第 2 分钟前 40 美元余额,同时运营账户全程保留 20 美元。智能体从初始的 60 美元支付预聘费,留下零。一笔 18.66 美元退款结清,智能体在第 1 分钟过账 50 美元调整;剩余付款之后的最终账户余额为 28.66 美元。目标和期末储备通过,但早先的零余额违反连续要求:\(G=1\),\(P=0\),\(F=1\)。只有执行历史暴露这一违反。

D.4 检索中的业务文档范围

在 Q01 中,客户要求智能体以欧元和英镑向供应商付款,并把早先的税款支付匹配到正确期间。适用的运营台规则写在 DESK-tax 和 DESK-fx 中。搜索税款支付匹配期间核对,返回 PX-B-77,一份不同产品的溢缴抵免文档,位于第一位,以及 PX-B-76,一份该产品的税务代理文档,位于第五位。智能体打开这两份文档,其术语与查询匹配,但二者都不管辖这项运营台任务。

觉得有用,转给同事

微信扫码

用微信扫一扫,在手机上打开后即可转发。

用 RSS 订阅

提交勘误