Hands-on AI testingGenerating test cases with Dify
Generating test cases with Dify
Lesson 1 · Yoyo本慢 · 搭一个小应用，把需求变成用例
Free lessons
Free series you watch on this site, no sign-in. Video is served by Bilibili; lessons are in Chinese.
Not translated yet — showing the Chinese originals.
Hands-on AI testingGenerating test cases with Dify
16 lessons
Hands-on AI testing
Public Bilibili videos you watch on this site. Twelve authors, each on one concrete job: writing cases, driving automation, and load-testing a model.
For：Test engineers who already use a model to write cases or scripts and want to see how other people do it.
All lessons
16 lessons
- 3:44
- 14:45Hands-on AI testingDesigning cases with TRAE
Designing cases with TRAE
Lesson 2 · 任我御风而行 · 从安装到导出一份用例表
- 10:33Hands-on AI testingWhy generated cases often cannot be used as they are
Why generated cases often cannot be used as they are
Lesson 3 · 清风说测试开发 · 生成之后，人还要看的地方
- 25:52Hands-on AI testingAsking DeepSeek for MCDC cases
Asking DeepSeek for MCDC cases
Lesson 4 · Hyowinner校长 · 拿一条逻辑，看它覆盖得怎么样
- 18:33Hands-on AI testingWriting tests from a Cursor command
Writing tests from a Cursor command
Lesson 5 · 严师兄聊AI · 把写测试收成一条命令
- 4:45Hands-on AI testingCursor rules for a test engineer
Cursor rules for a test engineer
Lesson 6 · Raina测试 · 规则写对，生成的结果才稳
- 8:05Hands-on AI testingFront-end automation with Playwright MCP
Front-end automation with Playwright MCP
Lesson 7 · 向度之桥 · 让 Claude Code 自己点页面
- 1:21:41Hands-on AI testingWeb automation with a skill and Playwright
Web automation with a skill and Playwright
Lesson 8 · 测试避坑君 · 从安装到把一条流程跑起来
- 4:05Hands-on AI testingApp automation with Maestro
App automation with Maestro
Lesson 9 · Zhu的AI日记 · 在 Cursor 里测一个 App
- 5:02Hands-on AI testingTesting how a model handles concurrency
Testing how a model handles concurrency
Lesson 10 · 西瓜讲大模型 · 并发上去之后先看什么
- 3:01Hands-on AI testingLoad-testing a model with GuideLLM
Load-testing a model with GuideLLM
Lesson 11 · 小工蚁创始人 · 一个开源的模型压测工具
- 19:33Hands-on AI testingA concurrency run against vLLM
A concurrency run against vLLM
Lesson 12 · AIGC科技官 · 压测的同时看监控
- 15:48Hands-on AI testing用 EvalScope 做压力测试
用 EvalScope 做压力测试
Lesson 13 · 每日学编程 · 单集标题是 EvalScope 压力测试，讲的是压测，不是评测指标概述。
- 12:20Hands-on AI testing用 GuideLLM 做部署基准测试
用 GuideLLM 做部署基准测试
Lesson 14 · swanmsg · 简介指向 vLLM 项目下的 GuideLLM 仓库，讲部署后的基准测试。
- 52:43Hands-on AI testingConnecting test automation with Agents, MCP, and Skills
Connecting test automation with Agents, MCP, and Skills
Lesson 15 · 但问智能 · Feed requirements and schemas to an Agent that generates, diagnoses, and repairs API, performance, and UI tests.
- 52:47Hands-on AI testingQoder 驱动的测试提效实战
Qoder 驱动的测试提效实战
Lesson 16 · Qoder · 通过 Qoder 的实际演示，了解 AI 编程工具如何辅助测试工作提效。
Evaluating LLMs and AgentsWhat AI evaluation actually measures
45 lessons · 学掌门
Evaluating LLMs and Agents
How AI evaluation differs from software testing, how to score and grade defects, then how to run batch evals with EvalScope and Langfuse.
For：Testers and engineers who need to accept a model or an agent, not just demo it.
All lessons
45 lessons
- 30:19Evaluating LLMs and AgentsWhat AI evaluation actually measures
What AI evaluation actually measures
Lesson 1 · 学掌门 · Separate the thing you measure from a pass-fail feature check.
- 30:10Evaluating LLMs and AgentsHow this differs from software testing
How this differs from software testing
Lesson 2 · 学掌门 · Software tests want a fixed result. Model evals want a stable, repeatable one.
- 24:18Evaluating LLMs and AgentsWhy accuracy is not enough
Why accuracy is not enough
Lesson 3 · 学掌门 · Look at which class the errors fall into before trusting one score.
- 58:44Evaluating LLMs and AgentsA standard evaluation workflow
A standard evaluation workflow
Lesson 4 · 学掌门 · Data, metric, then a conclusion you can rerun.
- 46:18Evaluating LLMs and AgentsFour objects, four evaluations
Four objects, four evaluations
Lesson 5 · 学掌门 · Name the four objects before comparing scores.
- 34:33Evaluating LLMs and AgentsRisks that invalidate the score
Risks that invalidate the score
Lesson 6 · 学掌门 · Find what would invalidate the whole run before comparing scores.
- 23:52Evaluating LLMs and AgentsFinding and grading defects
Finding and grading defects
Lesson 7 · 学掌门 · A finding needs a grade before it can become a defect.
- 26:34Evaluating LLMs and AgentsWhy batch evaluation exists
Why batch evaluation exists
Lesson 8 · 学掌门 · One manual try hides regressions. A batch does not.
- 12:11Evaluating LLMs and AgentsOptional: prepare a VM
Optional: prepare a VM
Lesson 9 · 学掌门 · Only for the lab environment. Skip it if Docker already runs.
- 23:33Evaluating LLMs and AgentsDeploy Docker
Deploy Docker
Lesson 10 · 学掌门 · Both tools in this course run from this container setup.
- 18:05Evaluating LLMs and AgentsDeploy Langfuse
Deploy Langfuse
Lesson 11 · 学掌门 · Keep inputs, outputs, and scores. A terminal log is not a record.
- 16:08Evaluating LLMs and AgentsDeploy EvalScope
Deploy EvalScope
Lesson 12 · 学掌门 · Connect datasets and metrics instead of a one-off script.
- 10:22Evaluating LLMs and AgentsConnect EvalScope to Langfuse
Connect EvalScope to Langfuse
Lesson 13 · 学掌门 · Review each row, not only the aggregate.
- 41:13Evaluating LLMs and AgentsRun a batch evaluation
Run a batch evaluation
Lesson 14 · 学掌门 · Run the batch and read the metric.
- 30:51Evaluating LLMs and AgentsClose with a full test set
Close with a full test set
Lesson 15 · 学掌门 · Rerun it, compare it, and point at the failing rows.
- 10:23Evaluating LLMs and AgentsEvalScope 概述
EvalScope 概述
Lesson 16 · 每日学编程 · 单集标题是 EvalScope 概述，只讲这个工具在测什么，不讲后面的压力测试。
- 10:56Evaluating LLMs and AgentsLangfuse 的 Trace 和 Evals
Langfuse 的 Trace 和 Evals
Lesson 17 · 前端屎不了 · 简介写了从 Observation、Trace、Session 讲到 Evals、Datasets 和 Metrics。
- 4:08Evaluating LLMs and AgentsDeepEval 和评测卡点
DeepEval 和评测卡点
Lesson 18 · 科罗廖夫的雪松 · 简介里作者用它和 RAGAS 对比，想把上一轮结果拿来做卡点。
- 29:37Evaluating LLMs and Agents用语义熵检测幻觉
用语义熵检测幻觉
Lesson 19 · THUAIR_人机交互研究组 · 简介写的是 Nature 论文：用语义熵检测大语言模型幻觉。
- 10:47Evaluating LLMs and AgentsEvaluating cross-application GUI Agent tasks
Evaluating cross-application GUI Agent tasks
Lesson 20 · 熊二等兵 · Combine final success rate with intermediate checkpoints to locate failures in long-horizon agent tasks.
- 8:41Evaluating LLMs and Agents从 Agent Trace 定位评测对象
从 Agent Trace 定位评测对象
Lesson 21 · 卡比Prover · 先沿执行轨迹看清 Agent 做了什么，再决定在哪些节点设置评测。
- 9:57Evaluating LLMs and Agents设计 Agent 评分器 Grader
设计 Agent 评分器 Grader
Lesson 22 · 慢学AI · 围绕 Agent 评估设计评分器，把结果判断从主观感觉变成明确规则。
- 2:04:18Evaluating LLMs and Agents搭建生产级 Agent Eval 流水线
搭建生产级 Agent Eval 流水线
Lesson 23 · 量子菠萝_ · 从 Trace、失败归因和评分器一路做到自动化实验，验证提示词改动是否真的有效。
- 17:11Evaluating LLMs and AgentsTie Agent evaluation to real task outcomes
Tie Agent evaluation to real task outcomes
Lesson 24 · 源语AI-摸鱼字幕君 · Build a production eval loop from offline simulation and quality gates to online monitoring, grounded in whether the task was actually completed.
- 1:28:41Evaluating LLMs and AgentsWalk through an end-to-end Agent evaluation
Walk through an end-to-end Agent evaluation
Lesson 25 · 西瓜讲大模型 · Follow a complete demonstration to understand the Agent evaluation workflow end to end.
- 5:33Evaluating LLMs and AgentsGraders and pitfalls in long-horizon Agent evaluation
Graders and pitfalls in long-horizon Agent evaluation
Lesson 26 · wanzij177 · Compare code, model, and human graders, then examine noise, gaming, and premature completion in long tasks.
- 41:21Evaluating LLMs and AgentsAgent evaluation across the development lifecycle
Agent evaluation across the development lifecycle
Lesson 27 · 不转到大模型不改名 · Follow Agent evaluation from model selection and integration testing to pipelines and production monitoring.
- 18:32Evaluating LLMs and AgentsBuilding an automated Agent evaluation system
Building an automated Agent evaluation system
Lesson 28 · AI老马啊 · Organize Agent targets, datasets, and execution into an automated evaluation system.
- 4:38Evaluating LLMs and AgentsWhat should an Eval actually measure
What should an Eval actually measure
Lesson 29 · AIOps智能运维 · Separate model quality, system usability, user adoption, and business value, then ground them with a Golden Set and tool correctness.
- 6:14Evaluating LLMs and AgentsWhy Agents need an evaluation system
Why Agents need an evaluation system
Lesson 30 · 慢学AI · Understand why evaluation is the foundation for stable iteration and scale.
- 9:57Evaluating LLMs and AgentsDesigning graders for Agent evaluations
Designing graders for Agent evaluations
Lesson 31 · 慢学AI · Use explicit graders instead of relying on subjective impressions.
- 5:07Evaluating LLMs and AgentsMeasuring randomness with Pass@k and Pass^k
Measuring randomness with Pass@k and Pass^k
Lesson 32 · 慢学AI · Separate occasional success from consistent success across repeated runs.
- 5:15Evaluating LLMs and AgentsChoosing grading priorities by Agent type
Choosing grading priorities by Agent type
Lesson 33 · 慢学AI · Match grader priorities to the type of Agent and its task.
- 7:08Evaluating LLMs and AgentsBuilding an Agent evaluation system from scratch
Building an Agent evaluation system from scratch
Lesson 34 · 慢学AI · Organize targets, data, and grading into a repeatable evaluation system.
- 4:09Evaluating LLMs and AgentsMoving from offline evals to production monitoring
Moving from offline evals to production monitoring
Lesson 35 · 慢学AI · Extend offline evaluation with continuous production monitoring.
- 6:38Evaluating LLMs and Agents快速认识 EvalScope 大模型评测工具
快速认识 EvalScope 大模型评测工具
Lesson 36 · 高楼_Zee · 从工具定位入手，建立 EvalScope 与大模型评测流程的基础认识。
- 3:44Evaluating LLMs and AgentsChoosing between custom and framework-based Agent evals
Choosing between custom and framework-based Agent evals
Lesson 37 · 慢学AI · Choose between a custom eval stack and an existing framework based on isolation, monitoring, and data requirements.
- 1:02:12Evaluating LLMs and AgentsEvaluating and iterating LLM Agents with TruLens
Evaluating and iterating LLM Agents with TruLens
Lesson 38 · GPT中英字幕课程资源 · Evaluate effectiveness, hallucination, and bias, then use the results to iterate and monitor an Agent in production.
- 20:00Evaluating LLMs and AgentsConnecting Agent runs to Evals and LLM Ops
Connecting Agent runs to Evals and LLM Ops
Lesson 39 · 肖恩君Sean · Connect run traces, Langfuse tracking, and LLM-as-judge evaluation into a repair loop.
- 3:24:31Evaluating LLMs and AgentsReading the Claw-Eval Agent evaluation framework
Reading the Claw-Eval Agent evaluation framework
Lesson 40 · 丸一口 · Read the paper and code behind a framework designed to evaluate LLM Agent capabilities.
- 12:01Evaluating LLMs and AgentsRagas workflow, single-sample analysis, and visualization
Ragas workflow, single-sample analysis, and visualization
Lesson 41 · ShiianAI · Run a RAG evaluation from data preparation to visualization and inspect individual samples.
- 15:48Evaluating LLMs and AgentsLoad testing LLMs with EvalScope
Load testing LLMs with EvalScope
Lesson 42 · 丹哥二号 · Apply an evaluation framework to load-test a model
- 5:52Evaluating LLMs and AgentsChecking LLM output quality with DeepEval
Checking LLM output quality with DeepEval
Lesson 43 · poyhh · Replace subjective acceptance with metrics and tests
- 3:52Evaluating LLMs and AgentsDesigning evals for AI Agents
Designing evals for AI Agents
Lesson 44 · TreeDy1 · Understand Agent evaluation through Task, Grader, and Harness
- 34:23Evaluating LLMs and AgentsEvaluating LLMs in Python with DeepEval
Evaluating LLMs in Python with DeepEval
Lesson 45 · David鲍曼 · Bring LLM evaluation into a Python implementation
RAG evaluation metrics with RAGAS评测先看哪些指标
11 lessons · 骰子AI
RAG evaluation metrics with RAGAS
A public multi-part video that walks through RAGAS metrics in order: similarity, relevance, context precision, recall, faithfulness, correctness, and aspect critique.
For：People running retrieval-augmented apps who need to see which part of an answer is unsupported.
All lessons
11 lessons
- 6:19RAG evaluation metrics with RAGAS评测先看哪些指标
评测先看哪些指标
Lesson 1 · 骰子AI · 这一集标题是概览，先把后面要拆的指标摆出来。
- 4:45RAG evaluation metrics with RAGAS答案相似度
答案相似度
Lesson 2 · 骰子AI · 分 P 标题是 Answer Similarity，讲答案和参考答案像不像。
- 3:11RAG evaluation metrics with RAGAS答案相关性
答案相关性
Lesson 3 · 骰子AI · 分 P 标题是 Answer Relevance，讲回答是否扣住问题。
- 1:56RAG evaluation metrics with RAGAS上下文精确率
上下文精确率
Lesson 4 · 骰子AI · 分 P 标题是 Context Precision。
- 3:49RAG evaluation metrics with RAGAS上下文相关性
上下文相关性
Lesson 5 · 骰子AI · 分 P 标题是 Context Relevance。
- 4:03RAG evaluation metrics with RAGAS上下文召回
上下文召回
Lesson 6 · 骰子AI · 分 P 标题是 context recall。
- 5:44RAG evaluation metrics with RAGAS上下文召回在 RAGAS 里的坑
上下文召回在 RAGAS 里的坑
Lesson 7 · 骰子AI · 分 P 标题写的就是 context recall 在 RAGAS 里的坑。
- 5:01RAG evaluation metrics with RAGAS忠实度
忠实度
Lesson 8 · 骰子AI · 分 P 标题是 Faithfulness。
- 6:50RAG evaluation metrics with RAGAS答案正确性
答案正确性
Lesson 9 · 骰子AI · 分 P 标题是 Answer Correctness。
- 5:59RAG evaluation metrics with RAGAS方面评价
方面评价
Lesson 10 · 骰子AI · 分 P 标题是 Aspect Critique。
- 6:27RAG evaluation metrics with RAGAS指标收口
指标收口
Lesson 11 · 骰子AI · 最后一集标题是总结。
Evaluating LLMs with OpenCompassOpenCompass 怎么组织评测
2 lessons · OpenMMLab
Evaluating LLMs with OpenCompass
Two separate public lessons from OpenMMLab: how OpenCompass organizes an evaluation, then a hands-on run. Not a model leaderboard.
For：Testers and engineers who need to run a benchmark, not read a ranking.
All lessons
2 lessons
- 51:31Evaluating LLMs with OpenCompassOpenCompass 怎么组织评测
OpenCompass 怎么组织评测
Lesson 1 · OpenMMLab · 官方教程这一集讲 OpenCompass 大模型评测，简介里给了教程文档和仓库。
- 45:36Evaluating LLMs with OpenCompassOpenCompass 评测实战
OpenCompass 评测实战
Lesson 2 · OpenMMLab · 贡献者讲的实战课，简介里给了课程文档和作业仓库。
Automating test cases with vision LLMsVision LLM-driven test-case automation
5 lessons · 但问智能
Automating test cases with vision LLMs
Learn a reusable AI test-case workflow: use a vision LLM to read mind maps, flowcharts, and UI screens, generate cases, and export the result.
For：Test engineers who want to apply vision LLMs to test-case design or build an AI testing agent.
All lessons
5 lessons
- 52:28Automating test cases with vision LLMsVision LLM-driven test-case automation
Vision LLM-driven test-case automation
Lesson 1 · 但问智能 · Follow the full path from mind maps, flowcharts, and UI screens to generated and exported test cases.
- 1:08:39Automating test cases with vision LLMsGrounding test-case generation with a knowledge base
Grounding test-case generation with a knowledge base
Lesson 2 · 但问智能 · Ground the model in business knowledge instead of relying on a one-off prompt.
- 1:49:32Automating test cases with vision LLMsBuilding a test-case generation agent
Building a test-case generation agent
Lesson 3 · 但问智能 · Implement the generation logic and agent workflow as a complete test-case automation application.
- 57:05Automating test cases with vision LLMsBatch-generating test cases with PydanticAI
Batch-generating test cases with PydanticAI
Lesson 4 · 但问智能 · Use a structured agent to turn large-scale test-case generation into a controlled engineering workflow.
- 55:07Automating test cases with vision LLMsFrom generated cases to AI-driven UI automation
From generated cases to AI-driven UI automation
Lesson 5 · 但问智能 · Connect natural language, UI understanding, Playwright, and visual reports to execute and debug UI tests.
Evaluating AI applications with LangSmithAI application evaluation
9 lessons · 大云哥讲AI
Evaluating AI applications with LangSmith
A nine-part path through AI application evaluation: metrics, datasets, evaluators, Agent and RAG evaluation, and a practical LangSmith workflow.
For：Testers and engineers building AI applications, Agents, or RAG systems who need repeatable evaluations.
All lessons
9 lessons
- 10:26Evaluating AI applications with LangSmithAI application evaluation
AI application evaluation
Lesson 1 · 大云哥讲AI · Establish the objects, workflow, and conclusions of an evaluation.
- Evaluating AI applications with LangSmithDefining evaluation metrics
Defining evaluation metrics
Lesson 2 · 大云哥讲AI · Turn the capability you want to observe into measurable metrics.
- Evaluating AI applications with LangSmithEvaluation datasets
Evaluation datasets
Lesson 3 · 大云哥讲AI · Prepare data that supports repeatable runs and comparisons.
- Evaluating AI applications with LangSmithEvaluators and evaluation execution
Evaluators and evaluation execution
Lesson 4 · 大云哥讲AI · Move from choosing evaluators to running an actual evaluation.
- Evaluating AI applications with LangSmithAgent evaluation methods
Agent evaluation methods
Lesson 5 · 大云哥讲AI · Evaluate an Agent through its behavior and outcomes.
- Evaluating AI applications with LangSmithRAG evaluation methods
RAG evaluation methods
Lesson 6 · 大云哥讲AI · Break RAG answer quality into evaluable questions.
- Evaluating AI applications with LangSmithRunning AI evaluations with LangSmith
Running AI evaluations with LangSmith
Lesson 7 · 大云哥讲AI · Connect the metrics, dataset, and evaluator in LangSmith.
- Evaluating AI applications with LangSmithEvaluating an LLM application with LangSmith
Evaluating an LLM application with LangSmith
Lesson 8 · 大云哥讲AI · Run a complete evaluation on an LLM application.
- Evaluating AI applications with LangSmithEvaluating a RAG application with LangSmith
Evaluating a RAG application with LangSmith
Lesson 9 · 大云哥讲AI · Apply the evaluation workflow to a running RAG application.
Agent Evaluation Metrics and Practical MeasurementEvaluating Agent projects with five metric groups
2 lessons · 森_MS
Agent Evaluation Metrics and Practical Measurement
Build a practical Agent evaluation line across hallucination, empty-answer, satisfaction, token cost, TTFB, and escalation metrics, using reproducible log-based measurement.
For：Testers, engineers, and product teams who need measurable Agent quality signals for acceptance and production monitoring.
All lessons
2 lessons
- 52:46Agent Evaluation Metrics and Practical MeasurementEvaluating Agent projects with five metric groups
Evaluating Agent projects with five metric groups
Lesson 1 · 森_MS · Break Agent quality into measurable signals across hallucination, satisfaction, latency, cost, and escalation.
- 0:54Agent Evaluation Metrics and Practical MeasurementSupplement: closing the Agent metric walkthrough
Supplement: closing the Agent metric walkthrough
Lesson 2 · 森_MS · Close the practical walkthrough with a short supplement on the same Agent metrics.