Skip to content
GitLab
Commit 3dbf401d authored by liguangyu06's avatar liguangyu06
Browse files

清理不应入库的本地产物与一次性脚本，使仓库与 .gitignore 一致。

parent 82116292
Hide whitespace changes
Inline Side-by-side
.cursor/skills/pytest-patterns/references/enterprise-project-structure.md deleted 100644 → 0
View file @ 82116292
# 企业级 pytest 自动化标准目录结构（大型项目）
**适用**：接口自动化 + UI 自动化 + 多模块 + 多环境（dev / test / staging / prod）。
## 推荐目录树
```text
your-pytest-project/
├── core/ # 核心封装（基类、工具、驱动）
│ ├── __init__.py
│ ├── base_api.py # API 基类（统一 session、鉴权、重试、断言封装）
│ ├── base_page.py # UI 基类（Page Object 基类：封装 find、wait、截图）
│ └── browser_engine.py # 浏览器驱动管理（Playwright/Selenium 生命周期）
├── pages/ # UI 页面封装（PO 模式，仅 UI）
│ ├── __init__.py
│ ├── login_page.py
│ └── home_page.py
├── api/ # 接口封装（按域拆分模块）
│ ├── __init__.py
│ ├── user_api.py
│ └── order_api.py
├── common/ # 公共工具（与业务无关）
│ ├── __init__.py
│ ├── read_config.py # 读取 config（yaml/env）
│ ├── logger.py # 日志
│ └── utils.py # 通用方法（时间、加解密、路径等）
├── config/ # 配置文件（不放密钥进仓库）
│ ├── config.yaml # 环境、账号占位、base_url、超时
│ └── settings.py # 可选：将 yaml 转为对象 / pydantic Settings
├── data/ # 测试数据（参数化、边界值）
│ ├── test_data.json
│ └── params.yaml
├── reports/ # 测试报告输出（HTML/allure，加入 .gitignore）
├── tests/ # 测试用例（薄用例：调 api/pages + 断言）
│ ├── __init__.py
│ ├── conftest.py # 全局夹具：环境、浏览器、api client
│ ├── test_api/ # 接口用例（按模块子目录继续拆分）
│ └── test_ui/ # UI 用例
├── conftest.py # 可选：与 tests/conftest 二选一或分层（根目录全局）
├── pytest.ini # pytest 配置（markers、pythonpath、addopts）
└── requirements.txt
```
## 分层原则
| 层级 | 职责 | pytest 关系 |
|------|------|----------------|
| `core/` | 技术底座，不含业务用例 | 可被 `pages/`、`api/` import；一般不被 `tests` 直接滥用 |
| `api/` / `pages/` | 业务操作封装 | 用例通过 fixture 注入 client/page 实例 |
| `tests/` | 场景与断言 | `test_*.py`；保持简短，复杂逻辑下沉到 api/pages |
| `config/` + `common/read_config.py` | 多环境切换 | fixture 按 `PYTEST_ENV` 或 marker 加载不同 `config.yaml` |
| `data/` | 数据驱动 | `@pytest.mark.parametrize` 从 json/yaml 加载 |
## 多环境
- **推荐**：`config/config.dev.yaml`、`config.test.yaml` 或单文件多 key；通过环境变量 **`ENV=test`** 选择。
- **markers**：`@pytest.mark.api`、`@pytest.mark.ui`、`@pytest.mark.smoke`，在 `pytest.ini` 注册，CI 分 job 执行。
## UI + API 同仓
- **UI**：`browser_engine` 提供 session-scoped `browser`/`page` fixture；`pages/*` 封装选择器。
- **接口**：`api/*` 封装路径与 body；与 UI 共用 `config` 中的 `base_url`（注意 API 网关与前端域名可能不同，可分字段）。
## 与「库项目」tests/ 结构的区别
库项目常用 `tests/unit`、`tests/integration` 对照源码包名；**企业自动化仓**更强调 **api / pages / core 与 tests 分离**，便于多人按模块并行开发与按层跑用例（`pytest tests/test_api`、`pytest tests/test_ui`）。
## .gitignore 建议
```gitignore
reports/
.pytest_cache/
__pycache__/
*.pyc
.env
config/config.local.yaml
```
.cursor/skills/pytest/SKILL.md deleted 100644 → 0
View file @ 82116292
---
name: pytest
description: pytest - Python's most powerful testing framework with fixtures, parametrization, plugins, and framework integration for FastAPI, Django, Flask. Includes enterprise monorepo layout for API+UI automation (core, api, pages, tests).
version: 1.0.0
category: toolchain
author: Claude MPM Team
license: MIT
progressive_disclosure:
entry_point:
summary: "Professional Python testing: fixtures, parametrize, markers, async support, FastAPI/Django/Flask integration, coverage, mocking"
when_to_use: "Writing unit tests, integration tests, API testing, TDD workflow, testing async code, database testing, mocking dependencies"
quick_start: "1. pip install pytest 2. Create test_*.py files 3. Use fixtures with @pytest.fixture 4. Parametrize with @pytest.mark.parametrize 5. Run: pytest -v"
context_limit: 700
tags:
- pytest
- testing
- python
- tdd
- unit-testing
- fixtures
- mocking
- async
- fastapi
- django
requires_tools: []
---
# pytest - Professional Python Testing
## Overview
pytest is the industry-standard Python testing framework, offering powerful features like fixtures, parametrization, markers, plugins, and seamless integration with FastAPI, Django, and Flask. It provides a simple, scalable approach to testing from unit tests to complex integration scenarios.
**Key Features**:
- Fixture system for dependency injection
- Parametrization for data-driven tests
- Rich assertion introspection (no need for `self.assertEqual`)
- Plugin ecosystem (pytest-cov, pytest-asyncio, pytest-mock, pytest-django)
- Async/await support
- Parallel test execution with pytest-xdist
- Test discovery and organization
- Detailed failure reporting
**Installation**:
```bash
# Basic pytest
pip install pytest
# With common plugins
pip install pytest pytest-cov pytest-asyncio pytest-mock
# For FastAPI testing
pip install pytest httpx pytest-asyncio
# For Django testing
pip install pytest pytest-django
# For async databases
pip install pytest-asyncio aiosqlite
```
## Enterprise test project layout (large automation)
**适用**：接口自动化 + UI 自动化 + **多模块** + **多环境**。推荐将 **封装**（`core/`、`api/`、`pages/`）与 **用例**（`tests/test_api`、`tests/test_ui`）分离；配置与测试数据集中在 `config/`、`data/`。
完整目录树、分层职责、多环境与 `.gitignore` 建议见 **[references/enterprise-project-structure.md](references/enterprise-project-structure.md)**。
## Basic Testing Patterns
### 1. Simple Test Functions
```python
# test_math.py
def add(a, b):
return a + b
def test_add():
assert add(2, 3) == 5
assert add(-1, 1) == 0
assert add(0, 0) == 0
def test_add_negative():
assert add(-2, -3) == -5
```
**Run tests:**
```bash
# Discover and run all tests
pytest
# Verbose output
pytest -v
# Show print statements
pytest -s
# Run specific test file
pytest test_math.py
# Run specific test function
pytest test_math.py::test_add
```
### 2. Test Classes for Organization
```python
# test_calculator.py
class Calculator:
def add(self, a, b):
return a + b
def multiply(self, a, b):
return a * b
class TestCalculator:
def test_add(self):
calc = Calculator()
assert calc.add(2, 3) == 5
def test_multiply(self):
calc = Calculator()
assert calc.multiply(4, 5) == 20
def test_add_negative(self):
calc = Calculator()
assert calc.add(-1, -1) == -2
```
### 3. Assertions and Expected Failures
```python
import pytest
# Test exception raising
def divide(a, b):
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def test_divide_by_zero():
with pytest.raises(ValueError, match="Cannot divide by zero"):
divide(10, 0)
def test_divide_success():
assert divide(10, 2) == 5.0
# Test approximate equality
def test_float_comparison():
assert 0.1 + 0.2 == pytest.approx(0.3)
# Test containment
def test_list_contains():
result = [1, 2, 3, 4]
assert 3 in result
assert len(result) == 4
```
## Fixtures - Dependency Injection
### Basic Fixtures
```python
# conftest.py
import pytest
@pytest.fixture
def sample_data():
"""Provide sample data for tests."""
return {"name": "Alice", "age": 30, "email": "alice@example.com"}
@pytest.fixture
def empty_list():
"""Provide an empty list."""
return []
# test_fixtures.py
def test_sample_data(sample_data):
assert sample_data["name"] == "Alice"
assert sample_data["age"] == 30
def test_empty_list(empty_list):
empty_list.append(1)
assert len(empty_list) == 1
```
### Fixture Scopes
```python
import pytest
# Function scope (default) - runs for each test
@pytest.fixture(scope="function")
def user():
return {"id": 1, "name": "Alice"}
# Class scope - runs once per test class
@pytest.fixture(scope="class")
def database():
db = setup_database()
yield db
db.close()
# Module scope - runs once per test module
@pytest.fixture(scope="module")
def api_client():
client = APIClient()
yield client
client.shutdown()
# Session scope - runs once for entire test session
@pytest.fixture(scope="session")
def app_config():
return load_config()
```
### Fixture Setup and Teardown
```python
import pytest
import tempfile
import shutil
@pytest.fixture
def temp_directory():
"""Create a temporary directory for test."""
temp_dir = tempfile.mkdtemp()
print(f"
Setup: Created {temp_dir}")
yield temp_dir # Provide directory to test
# Teardown: cleanup after test
shutil.rmtree(temp_dir)
print(f"
Teardown: Removed {temp_dir}")
def test_file_creation(temp_directory):
file_path = f"{temp_directory}/test.txt"
with open(file_path, "w") as f:
f.write("test content")
assert os.path.exists(file_path)
```
### Fixture Dependencies
```python
import pytest
@pytest.fixture
def database_connection():
"""Database connection."""
conn = connect_to_db()
yield conn
conn.close()
@pytest.fixture
def database_session(database_connection):
"""Database session depends on connection."""
session = create_session(database_connection)
yield session
session.rollback()
session.close()
@pytest.fixture
def user_repository(database_session):
"""User repository depends on session."""
return UserRepository(database_session)
def test_create_user(user_repository):
user = user_repository.create(name="Alice", email="alice@example.com")
assert user.name == "Alice"
```
## Parametrization - Data-Driven Testing
### Basic Parametrization
```python
import pytest
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 5),
(5, 7, 12),
(-1, 1, 0),
(0, 0, 0),
(100, 200, 300),
])
def test_add_parametrized(a, b, expected):
assert add(a, b) == expected
```
### Multiple Parameters
```python
@pytest.mark.parametrize("operation,a,b,expected", [
("add", 2, 3, 5),
("subtract", 10, 5, 5),
("multiply", 4, 5, 20),
("divide", 10, 2, 5),
])
def test_calculator_operations(operation, a, b, expected):
calc = Calculator()
result = getattr(calc, operation)(a, b)
assert result == expected
```
### Parametrize with IDs
```python
@pytest.mark.parametrize("input_data,expected", [
pytest.param({"name": "Alice"}, "Alice", id="valid_name"),
pytest.param({"name": ""}, None, id="empty_name"),
pytest.param({}, None, id="missing_name"),
], ids=lambda x: x if isinstance(x, str) else None)
def test_extract_name(input_data, expected):
result = extract_name(input_data)
assert result == expected
```
### Indirect Parametrization (Fixtures)
```python
@pytest.fixture
def user_data(request):
"""Create user based on parameter."""
return {"name": request.param, "email": f"{request.param}@example.com"}
@pytest.mark.parametrize("user_data", ["Alice", "Bob", "Charlie"], indirect=True)
def test_user_creation(user_data):
assert "@example.com" in user_data["email"]
```
## Test Markers
### Built-in Markers
```python
import pytest
# Skip test
@pytest.mark.skip(reason="Not implemented yet")
def test_future_feature():
pass
# Skip conditionally
@pytest.mark.skipif(sys.platform == "win32", reason="Unix-only test")
def test_unix_specific():
pass
# Expected failure
@pytest.mark.xfail(reason="Known bug #123")
def test_known_bug():
assert False
# Slow test marker
@pytest.mark.slow
def test_expensive_operation():
time.sleep(5)
assert True
```
### Custom Markers
```python
# pytest.ini
[pytest]
markers =
slow: marks tests as slow (deselect with '-m "not slow"')
integration: marks tests as integration tests
unit: marks tests as unit tests
smoke: marks tests as smoke tests
# test_custom_markers.py
import pytest
@pytest.mark.unit
def test_fast_unit():
assert True
@pytest.mark.integration
@pytest.mark.slow
def test_slow_integration():
# Integration test with database
pass
@pytest.mark.smoke
def test_critical_path():
# Smoke test for critical functionality
pass
```
**Run tests by marker:**
```bash
# Run only unit tests
pytest -m unit
# Run all except slow tests
pytest -m "not slow"
# Run integration tests
pytest -m integration
# Run unit AND integration
pytest -m "unit or integration"
# Run smoke tests only
pytest -m smoke
```
## FastAPI Testing
### Basic FastAPI Test Setup
```python
# app/main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
class Item(BaseModel):
name: str
price: float
@app.get("/")
def read_root():
return {"message": "Hello World"}
@app.get("/items/{item_id}")
def read_item(item_id: int):
if item_id == 0:
raise HTTPException(status_code=404, detail="Item not found")
return {"item_id": item_id, "name": f"Item {item_id}"}
@app.post("/items")
def create_item(item: Item):
return {"name": item.name, "price": item.price, "id": 123}
```
### FastAPI Test Client
```python
# conftest.py
import pytest
from fastapi.testclient import TestClient
from app.main import app
@pytest.fixture
def client():
"""FastAPI test client."""
return TestClient(app)
# test_api.py
def test_read_root(client):
response = client.get("/")
assert response.status_code == 200
assert response.json() == {"message": "Hello World"}
def test_read_item(client):
response = client.get("/items/1")
assert response.status_code == 200
assert response.json() == {"item_id": 1, "name": "Item 1"}
def test_read_item_not_found(client):
response = client.get("/items/0")
assert response.status_code == 404
assert response.json() == {"detail": "Item not found"}
def test_create_item(client):
response = client.post(
"/items",
json={"name": "Widget", "price": 9.99}
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Widget"
assert data["price"] == 9.99
assert "id" in data
```
### Async FastAPI Testing
```python
# conftest.py
import pytest
from httpx import AsyncClient
from app.main import app
@pytest.fixture
async def async_client():
"""Async test client for FastAPI."""
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
# test_async_api.py
import pytest
@pytest.mark.asyncio
async def test_read_root_async(async_client):
response = await async_client.get("/")
assert response.status_code == 200
assert response.json() == {"message": "Hello World"}
@pytest.mark.asyncio
async def test_create_item_async(async_client):
response = await async_client.post(
"/items",
json={"name": "Gadget", "price": 19.99}
)
assert response.status_code == 200
assert response.json()["name"] == "Gadget"
```
### FastAPI with Database Testing
```python
# conftest.py
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.database import Base, get_db
from app.main import app
# Test database
SQLALCHEMY_DATABASE_URL = "sqlite:///./test.db"
engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
TestingSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
@pytest.fixture(scope="function")
def test_db():
"""Create test database."""
Base.metadata.create_all(bind=engine)
yield
Base.metadata.drop_all(bind=engine)
@pytest.fixture
def client(test_db):
"""Override database dependency."""
def override_get_db():
try:
db = TestingSessionLocal()
yield db
finally:
db.close()
app.dependency_overrides[get_db] = override_get_db
with TestClient(app) as test_client:
yield test_client
app.dependency_overrides.clear()
# test_users.py
def test_create_user(client):
response = client.post(
"/users",
json={"email": "test@example.com", "password": "secret"}
)
assert response.status_code == 200
assert response.json()["email"] == "test@example.com"
def test_read_users(client):
# Create user first
client.post("/users", json={"email": "user1@example.com", "password": "pass1"})
client.post("/users", json={"email": "user2@example.com", "password": "pass2"})
# Read users
response = client.get("/users")
assert response.status_code == 200
assert len(response.json()) == 2
```
## Django Testing
### Django pytest Configuration
```python
# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings
python_files = tests.py test_*.py *_tests.py
# conftest.py
import pytest
from django.conf import settings
@pytest.fixture(scope='session')
def django_db_setup():
settings.DATABASES['default'] = {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
}
```
### Django Model Testing
```python
# models.py
from django.db import models
class User(models.Model):
email = models.EmailField(unique=True)
name = models.CharField(max_length=100)
is_active = models.BooleanField(default=True)
# test_models.py
import pytest
from myapp.models import User
@pytest.mark.django_db
def test_create_user():
user = User.objects.create(
email="test@example.com",
name="Test User"
)
assert user.email == "test@example.com"
assert user.is_active is True
@pytest.mark.django_db
def test_user_unique_email():
User.objects.create(email="test@example.com", name="User 1")
with pytest.raises(Exception): # IntegrityError
User.objects.create(email="test@example.com", name="User 2")
```
### Django View Testing
```python
# views.py
from django.http import JsonResponse
from django.views import View
class UserListView(View):
def get(self, request):
users = User.objects.all()
return JsonResponse({
"users": list(users.values("id", "email", "name"))
})
# test_views.py
import pytest
from django.test import Client
from myapp.models import User
@pytest.fixture
def client():
return Client()
@pytest.mark.django_db
def test_user_list_view(client):
# Create test data
User.objects.create(email="user1@example.com", name="User 1")
User.objects.create(email="user2@example.com", name="User 2")
# Test view
response = client.get("/users/")
assert response.status_code == 200
data = response.json()
assert len(data["users"]) == 2
```
### Django REST Framework Testing
```python
# serializers.py
from rest_framework import serializers
from myapp.models import User
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = User
fields = ['id', 'email', 'name', 'is_active']
# views.py
from rest_framework import viewsets
from myapp.models import User
from myapp.serializers import UserSerializer
class UserViewSet(viewsets.ModelViewSet):
queryset = User.objects.all()
serializer_class = UserSerializer
# test_api.py
import pytest
from rest_framework.test import APIClient
from myapp.models import User
@pytest.fixture
def api_client():
return APIClient()
@pytest.mark.django_db
def test_list_users(api_client):
User.objects.create(email="user1@example.com", name="User 1")
User.objects.create(email="user2@example.com", name="User 2")
response = api_client.get("/api/users/")
assert response.status_code == 200
assert len(response.data) == 2
@pytest.mark.django_db
def test_create_user(api_client):
data = {"email": "new@example.com", "name": "New User"}
response = api_client.post("/api/users/", data)
assert response.status_code == 201
assert User.objects.filter(email="new@example.com").exists()
```
## Mocking and Patching
### pytest-mock (pytest.fixture.mocker)
```python
# Install: pip install pytest-mock
# service.py
import requests
def get_user_data(user_id):
response = requests.get(f"https://api.example.com/users/{user_id}")
return response.json()
# test_service.py
def test_get_user_data(mocker):
# Mock requests.get
mock_response = mocker.Mock()
mock_response.json.return_value = {"id": 1, "name": "Alice"}
mocker.patch("requests.get", return_value=mock_response)
result = get_user_data(1)
assert result["name"] == "Alice"
```
### Mocking Class Methods
```python
class UserService:
def get_user(self, user_id):
# Database call
return database.fetch_user(user_id)
def get_user_name(self, user_id):
user = self.get_user(user_id)
return user["name"]
def test_get_user_name(mocker):
service = UserService()
# Mock the get_user method
mocker.patch.object(
service,
"get_user",
return_value={"id": 1, "name": "Alice"}
)
result = service.get_user_name(1)
assert result == "Alice"
```
### Mocking with Side Effects
```python
def test_retry_on_failure(mocker):
# First call fails, second succeeds
mock_api = mocker.patch("requests.get")
mock_api.side_effect = [
requests.exceptions.Timeout(), # First call
mocker.Mock(json=lambda: {"status": "ok"}) # Second call
]
result = api_call_with_retry()
assert result["status"] == "ok"
assert mock_api.call_count == 2
```
### Spy on Calls
```python
def test_function_called_correctly(mocker):
spy = mocker.spy(module, "function_name")
# Call code that uses the function
module.run_workflow()
# Verify it was called
assert spy.call_count == 1
spy.assert_called_once_with(arg1="value", arg2=42)
```
## Coverage and Reporting
### pytest-cov Configuration
```bash
# Install
pip install pytest-cov
# Run with coverage
pytest --cov=app --cov-report=html --cov-report=term
# Generate coverage report
pytest --cov=app --cov-report=term-missing
# Coverage with minimum threshold
pytest --cov=app --cov-fail-under=80
```
### pytest.ini Coverage Configuration
```ini
# pytest.ini
[pytest]
addopts =
--cov=app
--cov-report=html
--cov-report=term-missing
--cov-fail-under=80
-v
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
```
### Coverage Reports
```bash
# HTML report (opens in browser)
pytest --cov=app --cov-report=html
open htmlcov/index.html
# Terminal report with missing lines
pytest --cov=app --cov-report=term-missing
# XML report (for CI/CD)
pytest --cov=app --cov-report=xml
# JSON report
pytest --cov=app --cov-report=json
```
## Async Testing
### pytest-asyncio
```python
# Install: pip install pytest-asyncio
# conftest.py
import pytest
# Enable asyncio mode
pytest_plugins = ('pytest_asyncio',)
# async_service.py
import asyncio
import aiohttp
async def fetch_data(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
# test_async_service.py
import pytest
@pytest.mark.asyncio
async def test_fetch_data(mocker):
# Mock aiohttp response
mock_response = mocker.AsyncMock()
mock_response.json.return_value = {"data": "test"}
mock_session = mocker.AsyncMock()
mock_session.__aenter__.return_value.get.return_value.__aenter__.return_value = mock_response
mocker.patch("aiohttp.ClientSession", return_value=mock_session)
result = await fetch_data("https://api.example.com/data")
assert result["data"] == "test"
```
### Async Fixtures
```python
@pytest.fixture
async def async_db_session():
"""Async database session."""
async with async_engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
async with AsyncSession(async_engine) as session:
yield session
async with async_engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
@pytest.mark.asyncio
async def test_create_user_async(async_db_session):
user = User(email="test@example.com", name="Test")
async_db_session.add(user)
await async_db_session.commit()
result = await async_db_session.execute(
select(User).where(User.email == "test@example.com")
)
assert result.scalar_one().name == "Test"
```
## Local pytest Profiles (Your Repos)
Common settings from your projects' `pyproject.toml`:
- `asyncio_mode = "auto"` (default in mcp-browser, mcp-memory, claude-mpm, edgar)
- `addopts` includes `--strict-markers` and `--strict-config` for CI consistency
- Coverage flags: `--cov=<package>`, `--cov-report=term-missing`, `--cov-report=xml`
- Selective ignores (mcp-vector-search): `--ignore=tests/manual`, `--ignore=tests/e2e`
- `pythonpath = ["src"]` for editable import resolution (mcp-ticketer)
Typical markers:
- `unit`, `integration`, `e2e`
- `slow`, `benchmark`, `performance`
- `requires_api` (edgar)
Reference: see `pyproject.toml` in `claude-mpm`, `edgar`, `mcp-vector-search`, `mcp-ticketer`, and `kuzu-memory` for full lists.
## Best Practices
### 1. Test Organization
```
project/
├── app/
│ ├── __init__.py
│ ├── main.py
│ ├── models.py
│ └── services.py
├── tests/
│ ├── __init__.py
│ ├── conftest.py # Shared fixtures
│ ├── test_models.py # Model tests
│ ├── test_services.py # Service tests
│ ├── test_api.py # API tests
│ └── integration/
│ ├── __init__.py
│ └── test_workflows.py
└── pytest.ini
```
### 2. Naming Conventions
```python
# ✅ GOOD: Clear test names
def test_user_creation_with_valid_email():
pass
def test_user_creation_raises_error_for_duplicate_email():
pass
# ❌ BAD: Vague names
def test_user1():
pass
def test_case2():
pass
```
### 3. Arrange-Act-Assert Pattern
```python
def test_user_service_creates_user():
# Arrange: Setup test data and dependencies
service = UserService(database=mock_db)
user_data = {"email": "test@example.com", "name": "Test"}
# Act: Perform the action being tested
result = service.create_user(user_data)
# Assert: Verify the outcome
assert result.email == "test@example.com"
assert result.id is not None
```
### 4. Use Fixtures for Common Setup
```python
# ❌ BAD: Repeated setup
def test_user_creation():
db = setup_database()
user = create_user(db)
assert user.id is not None
db.close()
def test_user_deletion():
db = setup_database()
user = create_user(db)
delete_user(db, user.id)
db.close()
# ✅ GOOD: Fixture-based setup
@pytest.fixture
def db():
database = setup_database()
yield database
database.close()
@pytest.fixture
def user(db):
return create_user(db)
def test_user_creation(user):
assert user.id is not None
def test_user_deletion(db, user):
delete_user(db, user.id)
assert not user_exists(db, user.id)
```
### 5. Parametrize Similar Tests
```python
# ❌ BAD: Duplicate test code
def test_add_positive():
assert add(2, 3) == 5
def test_add_negative():
assert add(-2, -3) == -5
def test_add_zero():
assert add(0, 0) == 0
# ✅ GOOD: Parametrized tests
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 5),
(-2, -3, -5),
(0, 0, 0),
])
def test_add(a, b, expected):
assert add(a, b) == expected
```
### 6. Test One Thing Per Test
```python
# ❌ BAD: Testing multiple things
def test_user_workflow():
user = create_user()
assert user.id is not None
updated = update_user(user.id, name="New Name")
assert updated.name == "New Name"
deleted = delete_user(user.id)
assert deleted is True
# ✅ GOOD: Separate tests
def test_user_creation():
user = create_user()
assert user.id is not None
def test_user_update():
user = create_user()
updated = update_user(user.id, name="New Name")
assert updated.name == "New Name"
def test_user_deletion():
user = create_user()
result = delete_user(user.id)
assert result is True
```
### 7. Use Markers for Test Organization
```python
@pytest.mark.unit
def test_pure_function():
pass
@pytest.mark.integration
@pytest.mark.slow
def test_database_integration():
pass
@pytest.mark.smoke
def test_critical_path():
pass
```
### 8. Mock External Dependencies
```python
# ✅ GOOD: Mock external API
def test_fetch_user_data(mocker):
mocker.patch("requests.get", return_value=mock_response)
result = fetch_user_data(user_id=1)
assert result["name"] == "Alice"
# ❌ BAD: Real API call in test
def test_fetch_user_data():
result = fetch_user_data(user_id=1) # Real HTTP request!
assert result["name"] == "Alice"
```
## Common Pitfalls
### ❌ Anti-Pattern 1: Test Depends on Execution Order
```python
# WRONG: Tests should be independent
class TestUserWorkflow:
user_id = None
def test_create_user(self):
user = create_user()
TestUserWorkflow.user_id = user.id
def test_update_user(self):
# Fails if test_create_user didn't run first!
update_user(TestUserWorkflow.user_id, name="New")
```
**Correct:**
```python
@pytest.fixture
def created_user():
return create_user()
def test_create_user(created_user):
assert created_user.id is not None
def test_update_user(created_user):
update_user(created_user.id, name="New")
```
### ❌ Anti-Pattern 2: Not Cleaning Up Resources
```python
# WRONG: Database not cleaned up
def test_user_creation():
db = setup_database()
user = create_user(db)
assert user.id is not None
# Database connection not closed!
```
**Correct:**
```python
@pytest.fixture
def db():
database = setup_database()
yield database
database.close() # Cleanup
```
### ❌ Anti-Pattern 3: Testing Implementation Details
```python
# WRONG: Testing internal implementation
def test_user_service_uses_cache():
service = UserService()
service.get_user(1)
assert service._cache.has_key(1) # Testing internal cache!
```
**Correct:**
```python
# Test behavior, not implementation
def test_user_service_returns_user():
service = UserService()
user = service.get_user(1)
assert user.id == 1
```
### ❌ Anti-Pattern 4: Not Using pytest Features
```python
# WRONG: Using unittest assertions
import unittest
def test_addition():
result = add(2, 3)
unittest.TestCase().assertEqual(result, 5)
```
**Correct:**
```python
# Use pytest's rich assertions
def test_addition():
assert add(2, 3) == 5
```
### ❌ Anti-Pattern 5: Overly Complex Fixtures
```python
# WRONG: Fixture does too much
@pytest.fixture
def everything():
db = setup_db()
user = create_user(db)
session = login(user)
cache = setup_cache()
# ... too many things!
return {"db": db, "user": user, "session": session, "cache": cache}
```
**Correct:**
```python
# Separate, composable fixtures
@pytest.fixture
def db():
return setup_db()
@pytest.fixture
def user(db):
return create_user(db)
@pytest.fixture
def session(user):
return login(user)
```
## Quick Reference
### Common Commands
```bash
# Run all tests
pytest
# Verbose output
pytest -v
# Show print statements
pytest -s
# Run specific file
pytest tests/test_api.py
# Run specific test
pytest tests/test_api.py::test_create_user
# Run by marker
pytest -m unit
pytest -m "not slow"
# Run with coverage
pytest --cov=app --cov-report=html
# Parallel execution
pytest -n auto # Requires pytest-xdist
# Stop on first failure
pytest -x
# Show local variables on failure
pytest -l
# Run last failed tests
pytest --lf
# Run failed tests first
pytest --ff
```
### pytest.ini Template
```ini
[pytest]
# Minimum pytest version
minversion = 7.0
# Test discovery patterns
python_files = test_*.py *_test.py
python_classes = Test*
python_functions = test_*
# Test paths
testpaths = tests
# Command line options
addopts =
-v
--strict-markers
--cov=app
--cov-report=html
--cov-report=term-missing
--cov-fail-under=80
# Markers
markers =
unit: Unit tests
integration: Integration tests
slow: Slow-running tests
smoke: Smoke tests for critical paths
# Django settings (if using Django)
DJANGO_SETTINGS_MODULE = myproject.settings
# Asyncio mode
asyncio_mode = auto
```
### conftest.py Template
```python
# conftest.py
import pytest
from fastapi.testclient import TestClient
from app.main import app
# FastAPI client fixture
@pytest.fixture
def client():
return TestClient(app)
# Database fixture
@pytest.fixture(scope="function")
def db():
database = setup_test_database()
yield database
database.close()
# Mock user fixture
@pytest.fixture
def mock_user():
return {"id": 1, "email": "test@example.com", "name": "Test User"}
# Custom pytest configuration
def pytest_configure(config):
config.addinivalue_line("markers", "api: API tests")
config.addinivalue_line("markers", "db: Database tests")
```
## Resources
- **Enterprise project layout (API + UI)**: [references/enterprise-project-structure.md](references/enterprise-project-structure.md)
- **Official Documentation**: https://docs.pytest.org/
- **pytest-asyncio**: https://pytest-asyncio.readthedocs.io/
- **pytest-cov**: https://pytest-cov.readthedocs.io/
- **pytest-mock**: https://pytest-mock.readthedocs.io/
- **pytest-django**: https://pytest-django.readthedocs.io/
- **FastAPI Testing**: https://fastapi.tiangolo.com/tutorial/testing/
## Related Skills
When using pytest, consider these complementary skills:
- **fastapi-local-dev**: FastAPI development server patterns and test fixtures
- **test-driven-development**: Complete TDD workflow (RED/GREEN/REFACTOR cycle)
- **systematic-debugging**: Root cause investigation for failing tests
### Quick TDD Workflow Reference (Inlined for Standalone Use)
**RED → GREEN → REFACTOR Cycle:**
1. **RED Phase: Write Failing Test**
```python
def test_should_authenticate_user_when_credentials_valid():
# Test that describes desired behavior
user = User(username='alice', password='secret123')
result = authenticate(user)
assert result.is_authenticated is True
# This test will fail because authenticate() doesn't exist yet
```
2. **GREEN Phase: Make It Pass**
```python
def authenticate(user):
# Minimum code to pass the test
if user.username == 'alice' and user.password == 'secret123':
return AuthResult(is_authenticated=True)
return AuthResult(is_authenticated=False)
```
3. **REFACTOR Phase: Improve Code**
```python
def authenticate(user):
# Clean up while keeping tests green
hashed_password = hash_password(user.password)
stored_user = database.get_user(user.username)
return AuthResult(
is_authenticated=(stored_user.password_hash == hashed_password)
)
```
**Test Structure: Arrange-Act-Assert (AAA)**
```python
def test_user_creation():
# Arrange: Set up test data
user_data = {'username': 'alice', 'email': 'alice@example.com'}
# Act: Perform the action
user = create_user(user_data)
# Assert: Verify outcome
assert user.username == 'alice'
assert user.email == 'alice@example.com'
```
### Quick Debugging Reference (Inlined for Standalone Use)
**Phase 1: Root Cause Investigation**
- Read error messages completely (stack traces, line numbers)
- Reproduce consistently (document exact steps)
- Check recent changes (git log, git diff)
- Understand what changed and why it might cause failure
**Phase 2: Isolate the Problem**
```python
# Use pytest's built-in debugging
pytest tests/test_auth.py -vv --pdb # Drop into debugger on failure
pytest tests/test_auth.py -x # Stop on first failure
pytest tests/test_auth.py -k "auth" # Run only auth-related tests
# Add strategic print/logging
def test_complex_workflow():
user = create_user({'username': 'test'})
print(f"DEBUG: Created user {user.id}") # Visible with pytest -s
result = process_user(user)
print(f"DEBUG: Result status {result.status}")
assert result.success
```
**Phase 3: Fix Root Cause**
- Fix the underlying problem, not symptoms
- Add regression test to prevent recurrence
- Verify fix doesn't break other tests
**Phase 4: Verify Solution**
```bash
# Run full test suite
pytest
# Run with coverage
pytest --cov=src --cov-report=html
# Verify specific test patterns
pytest -k "auth or login" -v
```
[Full TDD and debugging workflows available in respective skills if deployed together]
---
**pytest Version Compatibility:** This skill covers pytest 7.0+ and reflects current best practices for Python testing in 2025.
.cursor/skills/pytest/metadata.json deleted 100644 → 0
View file @ 82116292
{
"name": "pytest",
"version": "1.0.0",
"category": "toolchain",
"toolchain": "python",
"framework": "pytest",
"tags": [
"pytest",
"testing",
"python",
"tdd",
"unit-testing",
"integration-testing",
"fixtures",
"parametrize",
"mocking",
"async",
"fastapi",
"django",
"flask",
"coverage"
],
"entry_point_tokens": 163,
"full_tokens": 8084,
"related_skills": [
"../../frameworks/fastapi-local-dev",
"../../../../universal/testing/test-driven-development",
"../../../../universal/debugging/systematic-debugging"
],
"author": "Claude MPM Team",
"license": "MIT",
"requires": [],
"sub_skills": [],
"created": "2025-11-30",
"updated": "2025-12-31",
"source_path": "toolchains/python/testing/pytest/SKILL.md",
"repository": "https://github.com/bobmatnyc/claude-mpm-skills",
"description": "pytest - Python's most powerful testing framework with fixtures, parametrization, plugins, and framework integration for FastAPI, Django, Flask"
}
.cursor/skills/pytest/references/enterprise-project-structure.md deleted 100644 → 0
View file @ 82116292
# 企业级 pytest 自动化标准目录结构（大型项目）
**适用**：接口自动化 + UI 自动化 + 多模块 + 多环境（dev / test / staging / prod）。
## 推荐目录树
```text
your-pytest-project/
├── core/ # 核心封装（基类、工具、驱动）
│ ├── __init__.py
│ ├── base_api.py # API 基类（统一 session、鉴权、重试、断言封装）
│ ├── base_page.py # UI 基类（Page Object 基类：封装 find、wait、截图）
│ └── browser_engine.py # 浏览器驱动管理（Playwright/Selenium 生命周期）
├── pages/ # UI 页面封装（PO 模式，仅 UI）
│ ├── __init__.py
│ ├── login_page.py
│ └── home_page.py
├── api/ # 接口封装（按域拆分模块）
│ ├── __init__.py
│ ├── user_api.py
│ └── order_api.py
├── common/ # 公共工具（与业务无关）
│ ├── __init__.py
│ ├── read_config.py # 读取 config（yaml/env）
│ ├── logger.py # 日志
│ └── utils.py # 通用方法（时间、加解密、路径等）
├── config/ # 配置文件（不放密钥进仓库）
│ ├── config.yaml # 环境、账号占位、base_url、超时
│ └── settings.py # 可选：将 yaml 转为对象 / pydantic Settings
├── data/ # 测试数据（参数化、边界值）
│ ├── test_data.json
│ └── params.yaml
├── reports/ # 测试报告输出（HTML/allure，加入 .gitignore）
├── tests/ # 测试用例（薄用例：调 api/pages + 断言）
│ ├── __init__.py
│ ├── conftest.py # 全局夹具：环境、浏览器、api client
│ ├── test_api/ # 接口用例（按模块子目录继续拆分）
│ └── test_ui/ # UI 用例
├── conftest.py # 可选：与 tests/conftest 二选一或分层（根目录全局）
├── pytest.ini # pytest 配置（markers、pythonpath、addopts）
└── requirements.txt
```
## 分层原则
| 层级 | 职责 | pytest 关系 |
|------|------|----------------|
| `core/` | 技术底座，不含业务用例 | 可被 `pages/`、`api/` import；一般不被 `tests` 直接滥用 |
| `api/` / `pages/` | 业务操作封装 | 用例通过 fixture 注入 client/page 实例 |
| `tests/` | 场景与断言 | `test_*.py`；保持简短，复杂逻辑下沉到 api/pages |
| `config/` + `common/read_config.py` | 多环境切换 | fixture 按 `PYTEST_ENV` 或 marker 加载不同 `config.yaml` |
| `data/` | 数据驱动 | `@pytest.mark.parametrize` 从 json/yaml 加载 |
## 多环境
- **推荐**：`config/config.dev.yaml`、`config.test.yaml` 或单文件多 key；通过环境变量 **`ENV=test`** 选择。
- **markers**：`@pytest.mark.api`、`@pytest.mark.ui`、`@pytest.mark.smoke`，在 `pytest.ini` 注册，CI 分 job 执行。
## UI + API 同仓
- **UI**：`browser_engine` 提供 session-scoped `browser`/`page` fixture；`pages/*` 封装选择器。
- **接口**：`api/*` 封装路径与 body；与 UI 共用 `config` 中的 `base_url`（注意 API 网关与前端域名可能不同，可分字段）。
## 与「库项目」tests/ 结构的区别
库项目常用 `tests/unit`、`tests/integration` 对照源码包名；**企业自动化仓**更强调 **api / pages / core 与 tests 分离**，便于多人按模块并行开发与按层跑用例（`pytest tests/test_api`、`pytest tests/test_ui`）。
## .gitignore 建议
```gitignore
reports/
.pytest_cache/
__pycache__/
*.pyc
.env
config/config.local.yaml
```
.cursor/skills/self-improving-agent/README.md deleted 100644 → 0
View file @ 82116292
# Self-Improving Agent
A universal self-improvement system that learns from ALL skill experiences and continuously updates the codebase.
## Overview
This agent learns from **every skill interaction** to achieve true lifelong learning. It implements a complete feedback loop with multi-memory architecture, self-correction, and evolution markers.
## Key Features
- **Multi-Memory Architecture**: Semantic + Episodic + Working memory
- **Universal Learning**: Learns from ALL skills, not just PRDs
- **Pattern Extraction**: Converts experiences into reusable patterns
- **Self-Correction**: Fixes skill guidance when errors occur
- **Self-Validation**: Periodically verifies skill accuracy
- **Automatic Updates**: Updates related skills based on learned patterns
- **Confidence Tracking**: Measures pattern reliability over time
- **Human-in-the-Loop**: Collects feedback to validate improvements
## Memory System
Canonical paths live **inside this skill**:
```
.cursor/skills/self-improving-agent/memory/
├── semantic-patterns.json
├── episodic/
└── working/ # includes optional last_error.json from hooks
```
Optional `~/.claude/memory/` mirror is documented in `references/appendix.md`.
## How It Works
```
Any Skill Completes
↓
Extract Experience → Identify Patterns → Update Skills → Consolidate Memory
↓ ↓ ↓ ↓
What happened? What can we reuse? Which skills? Track metrics
```
## Installation
```bash
ln -s ~/path/to/agent-playbook/skills/self-improving-agent ~/.claude/skills/self-improving-agent
```
## Hooks (Optional)
Wire hooks to capture errors and session-end signals. On Windows without Bash, point PostToolUse at `hooks/post-bash.ps1` instead of `post-bash.sh`.
```json
{
"hooks": {
"PreToolUse": [
{
"matcher": "Bash|Write|Edit",
"hooks": [
{ "type": "command", "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/pre-tool.sh \"$TOOL_NAME\" \"$TOOL_INPUT\"" }
]
}
],
"PostToolUse": [
{
"matcher": "Bash",
"hooks": [
{ "type": "command", "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/post-bash.sh \"$TOOL_OUTPUT\" \"$EXIT_CODE\"" }
]
}
],
"Stop": [
{
"matcher": "",
"hooks": [
{ "type": "command", "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/session-end.sh" }
]
}
]
}
}
```
## Triggering
### Automatic
After ANY skill completes:
- prd-planner
- code-reviewer
- debugger
- refactoring-specialist
- etc.
### Manual
```
"自我进化"
"self-improve"
"分析今天的经验"
"总结这次教训"
```
## Example Learning
### Episode
```yaml
Skill: debugger
Situation: Form submission doesn't refresh data
Root Cause: Empty callback function
Pattern: Always verify callbacks have implementations
Confidence: 0.95 → Updates: debugger, prd-implementation-precheck
```
### Skill Update
```markdown
## Auto-Update (2025-01-11)
### Pattern Added
**Callback Verification**: Always verify that callback functions
passed as props are not empty and actually execute logic.
**Source**: Episode ep-2025-01-11-003 (3 occurrences)
**Action**: Added to debugger checklist
```
## Research Basis
- [SimpleMem: Efficient Lifelong Memory](https://arxiv.org/html/2601.02553v1)
- [ACM Memory Mechanisms Survey](https://dl.acm.org/doi/10.1145/3748302)
- [Lifelong Learning of LLM Agents](https://arxiv.org/html/2501.07278v1)
## Templates
Reusable templates live in `skills/self-improving-agent/templates`:
- `pattern-template.md`
- `correction-template.md`
- `validation-template.md`
## License
MIT
.cursor/skills/self-improving-agent/SKILL.md deleted 100644 → 0
View file @ 82116292
---
name: self-improving-agent
description: Universal self-improvement from skill sessions—extract lessons, update patterns, and evolve skills with traceable markers. Use when the user says 自我进化、经验总结、复盘、教训、改进技能、skill retro、lessons learned、self-improve、after a failed task、after correcting wrong guidance、or when consolidating reusable patterns into skills or memory. Also use after any substantive skill-run when patterns should be captured for next time.
allowed-tools: Read, Write, StrReplace, Shell, Grep, Glob, WebSearch, SemanticSearch
metadata:
hooks:
before_start:
- trigger: session-logger
mode: auto
context: "Start {skill_name}"
after_complete:
- trigger: create-pr
mode: ask_first
condition: skills_modified
reason: "Submit improvements to repository"
- trigger: session-logger
mode: auto
context: "Self-improvement cycle complete"
# on_error: log only — avoid recursive self-invocation; agent reads memory/working/last_error.json when relevant
on_error:
- trigger: session-logger
mode: auto
context: "Error captured in {skill_name}"
---
# Self-Improving Agent
> An agent workflow that turns concrete sessions into reusable patterns and **bounded** skill updates — with traceability and guardrails.
## When to use this skill
Use proactively (not only when asked) when:
- A task **failed**, was **rolled back**, or the user **disputed** the approach.
- The same mistake or ambiguity already appeared **twice** in the conversation or repo history.
- The user asks for **复盘 / 总结 / 自我进化 / self-improve / retro**.
- You are about to close a long multi-step task and **non-obvious** conventions were discovered.
Hook names (`session-logger`, `create-pr`) are **examples** — wire equivalents in Cursor hooks or team automation as needed.
## Canonical locations (this skill package)
All paths are relative to the skill root: `.cursor/skills/self-improving-agent/`.
| Asset | Path |
|--------|------|
| Semantic memory | `memory/semantic-patterns.json` |
| Episodic episodes | `memory/episodic/` (by year / dated files) |
| Working / errors | `memory/working/` (`last_error.json` when hooks or agent record failures) |
| Pattern / correction / validation templates | `templates/` |
| Deep dive | `references/appendix.md` |
Optional global mirror (e.g. Claude Code): `~/.claude/memory/` — keep **one** source of truth per workspace to avoid drift.
## Operating protocol (follow in order)
1. **Capture**: Summarize what happened (skill used, task, outcome, user stance). If `memory/working/last_error.json` exists, read it for shell/tool failures.
2. **Classify**: Decide whether this is a one-off mistake, a **repeatable** pattern, or a **wrong** piece of skill guidance.
3. **Prefer memory first**: Append or update `memory/semantic-patterns.json` and add an episodic file under `memory/episodic/` before editing other skills.
4. **Edit other skills only with evidence**: Update another skill’s `SKILL.md` only if at least one holds — user explicit correction, repeated failure (≥2), clear contradiction with repo reality, or user rating ≤4 when ratings exist.
5. **Mark every skill edit**: Use HTML comments for evolution or correction markers (date + source episode/pattern id). Keep edits **minimal** and reversible.
6. **Validate before spreading**: For broad checklist changes, run through `templates/validation-template.md` or the checklist in `references/appendix.md`.
7. **Report**: Give the user a short summary — what was stored, what was **not** promoted (and why), and what files changed.
## Guardrails
- **No overfitting**: Do not rewrite skills from a single anecdote; store low-confidence episodic data instead.
- **No contradiction**: If a new pattern conflicts with an existing one, resolve explicitly (deprecate old, merge, or defer).
- **No recursion**: Do not chain-invoke this skill solely because `on_error` fired; consume `last_error.json` and fix the **underlying** task first.
- **Workspace scope**: Patterns belong to **this** workspace unless the user asks for global skill propagation.
## Multi-memory architecture (short)
- **Semantic** (`memory/semantic-patterns.json`): reusable rules, confidence, application counts, `target_skills`.
- **Episodic** (`memory/episodic/`): dated records — situation, root cause, lesson, link to semantic id if any.
- **Working** (`memory/working/`): session scratch + latest failure snapshot for corrections.
Schema samples remain in the JSON file and episodic examples under `memory/episodic/`; mirror structure from existing entries.
## Evolution priority matrix
When promoting experience to a **target skill**, map by domain. If `target_skills` names do not exist locally, update the closest skill under `.cursor/skills/` or only update semantic memory until a skill exists.
| Trigger example | Domain | Typical target | Priority |
|-----------------|--------|----------------|----------|
| PRD / scope patterns | Product | planning / requirements skills | High |
| API or data-flow insight | Backend | api-design / architecture skills | High |
| Debug root cause | Quality | debugger / test skills | High |
| Review checklist gap | Process | review / QA skills | High |
| Perf / security note | Ops | performance / security skills | High |
| Minor stylistic preference | Style | defer or episodic only | Low |
## Self-improvement phases
### Phase 1 — Experience extraction
Extract: skill used, task, outcome (`success` | `partial` | `failure`), what worked, what failed, root cause, user feedback if any.
### Phase 2 — Pattern abstraction
Promote to semantic pattern when abstraction helps future tasks; tie to `target_skills` and set conservative `confidence` (e.g. 0.55–0.75 for first promotion).
Heuristic:
- Repeated ≥3 times → consider **critical** checklist item in the relevant skill.
- Single effective fix → episodic + semantic entry with low confidence until reinforced.
### Phase 3 — Skill updates
Use evolution markers:
```markdown
<!-- Evolution: YYYY-MM-DD | source: ep-... | skill: {name} -->
```
Use correction markers when prior guidance was wrong:
```markdown
<!-- Correction: YYYY-MM-DD | was: "..." | reason: ... -->
```
Full snippets live in `templates/pattern-template.md` and `templates/correction-template.md`.
### Phase 4 — Consolidation
Update semantic counts (`applications`, `confidence`), prune stale low-value patterns, and ensure episodic files reference pattern ids when linked.
## Self-correction
**Inputs**: User report, failing tests, or `memory/working/last_error.json` (written when `hooks/post-bash.sh` runs with non-zero exit and Python is available).
**Steps**:
1. Attribute — which guidance or assumption led here?
2. Diagnose — wrong skill text vs misapplication vs missing context?
3. Patch — minimal correction + correction marker + semantic update.
4. Verify — rerun the failing step or ask the user to confirm.
Avoid automatic mass rewrites across many skills in one pass.
## Hooks integration
Bundled scripts under `hooks/` are **optional** telemetry stubs; `post-bash.sh` additionally tries to write `memory/working/last_error.json` on non-zero exit when `python` / `python3` / `py -3` exists.
**Windows**: If `bash` is not on `PATH`, use the bundled `hooks/post-bash.ps1` (same `last_error.json` format). Example: `powershell -NoProfile -ExecutionPolicy Bypass -File ".../hooks/post-bash.ps1" \"$TOOL_OUTPUT\" \"$EXIT_CODE\"`. Git Bash / WSL remain supported for `.sh` scripts.
Example shape for Claude Code–style settings (replace `${SKILLS_DIR}`):
```json
{
"hooks": {
"PreToolUse": [
{
"matcher": "Bash|Write|Edit",
"hooks": [
{
"type": "command",
"command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/pre-tool.sh \"$TOOL_NAME\" \"$TOOL_INPUT\""
}
]
}
],
"PostToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/post-bash.sh \"$TOOL_OUTPUT\" \"$EXIT_CODE\""
}
]
}
],
"Stop": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/session-end.sh"
}
]
}
]
}
}
```
## Research-informed design (optional reading)
Recent work on agent memory and lifelong learning informs the **separation** of semantic vs episodic stores and confidence tracking — not mandatory to cite in every run.
| Direction | Takeaway for this skill |
|-----------|-------------------------|
| Efficient lifelong memory | Accumulate patterns with decay/confidence, not infinite prose |
| Multi-memory surveys | Keep facts/rules separate from one-off stories |
| Continuous adaptation | Prefer small verified deltas over bulk rewrites |
Links: see **References** below and `references/appendix.md`.
## Best practices
**Do**: Learn from substantive sessions; abstract at the right level; trace with markers; validate risky edits; tell the user what changed.
**Don’t**: Generalize once; hide conflicting advice; spam unrelated skills; chase perfection without user value.
## References
- [SimpleMem: Efficient Lifelong Memory for LLM Agents](https://arxiv.org/html/2601.02553v1)
- [A Survey on the Memory Mechanism of Large Language Model Agents](https://dl.acm.org/doi/10.1145/3748302)
- [Lifelong Learning of LLM based Agents](https://arxiv.org/html/2501.07278v1)
- [Let's Build a Self-Improving AI Agent](https://medium.com/@nomannayeem/lets-build-a-self-improving-ai-agent-that-learns-from-your-feedback-722d2ce9c2d9)
.cursor/skills/self-improving-agent/hooks/post-bash.ps1 deleted 100644 → 0
View file @ 82116292
param(
[Parameter(Position = 0)]
[string] $ToolOutput = "",
[Parameter(Position = 1)]
[string] $ExitCode = "0"
)
Write-Host "[self-improving-agent] PostToolUse: exit=$ExitCode"
if ($ToolOutput) {
Write-Host "[self-improving-agent] Output: $ToolOutput"
}
if ($ExitCode -eq "0") {
exit 0
}
$skillRoot = Split-Path $PSScriptRoot -Parent
$memoryWorking = Join-Path $skillRoot "memory\working"
New-Item -ItemType Directory -Force -Path $memoryWorking | Out-Null
$snippet = $ToolOutput
if ($snippet.Length -gt 8000) {
$snippet = $snippet.Substring(0, 8000)
}
[int] $code = -1
try {
$code = [int]::Parse($ExitCode)
}
catch {
$code = -1
}
$rec = [ordered]@{
timestamp = [DateTime]::UtcNow.ToString("o")
exit_code = $code
tool_output_snippet = $snippet
}
$path = Join-Path $memoryWorking "last_error.json"
($rec | ConvertTo-Json -Depth 5) | Set-Content -Path $path -Encoding utf8
.cursor/skills/self-improving-agent/hooks/post-bash.sh deleted 100644 → 0
View file @ 82116292
#!/usr/bin/env bash
set -euo pipefail
tool_output="${1:-}"
exit_code="${2:-0}"
echo "[self-improving-agent] PostToolUse: exit=${exit_code}" >&2
if [[ -n "${tool_output}" ]]; then
echo "[self-improving-agent] Output: ${tool_output}" >&2
fi
# Persist non-zero exits for self-correction / episodic logging (optional; requires Python).
if [[ "${exit_code}" != "0" ]]; then
SKILL_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MEMORY_WORKING="${SKILL_ROOT}/memory/working"
mkdir -p "${MEMORY_WORKING}"
export SIA_MEMORY_WORKING="${MEMORY_WORKING}"
export SIA_TOOL_OUTPUT="${tool_output}"
export SIA_EXIT_CODE="${exit_code}"
_run_py() {
if command -v python3 >/dev/null 2>&1; then
python3 -c "$1"
elif command -v python >/dev/null 2>&1; then
python -c "$1"
elif command -v py >/dev/null 2>&1; then
py -3 -c "$1"
else
return 1
fi
}
_run_py 'import json, os
from datetime import datetime, timezone
base = os.environ["SIA_MEMORY_WORKING"]
out = os.environ.get("SIA_TOOL_OUTPUT", "")
try:
code = int(os.environ.get("SIA_EXIT_CODE", "-1"))
except ValueError:
code = -1
rec = {"timestamp": datetime.now(timezone.utc).isoformat(), "exit_code": code, "tool_output_snippet": out[:8000]}
path = os.path.join(base, "last_error.json")
with open(path, "w", encoding="utf-8") as f:
json.dump(rec, f, indent=2)
' || true
fi
.cursor/skills/self-improving-agent/hooks/pre-tool.sh deleted 100644 → 0
View file @ 82116292
#!/usr/bin/env bash
set -euo pipefail
tool_name="${1:-unknown}"
tool_input="${2:-}"
echo "[self-improving-agent] PreToolUse: ${tool_name}" >&2
if [[ -n "${tool_input}" ]]; then
echo "[self-improving-agent] Input: ${tool_input}" >&2
fi
.cursor/skills/self-improving-agent/hooks/session-end.sh deleted 100644 → 0
View file @ 82116292
#!/usr/bin/env bash
set -euo pipefail
echo "[self-improving-agent] Session ended" >&2
.cursor/skills/self-improving-agent/memory/semantic-patterns.json deleted 100644 → 0
View file @ 82116292
{
"patterns": {},
"meta": {
"version": "1.1.0",
"last_updated": "2026-05-09",
"total_patterns": 0,
"categories": []
}
}
\ No newline at end of file
.cursor/skills/self-improving-agent/memory/working/last_error.json deleted 100644 → 0
View file @ 82116292
{
"timestamp": "2026-05-08T23:42:54.7953085Z",
"exit_code": 1,
"tool_output_snippet": "sample stderr"
}
.cursor/skills/self-improving-agent/references/appendix.md deleted 100644 → 0
View file @ 82116292
# Appendix
## Self-Validation
### Validation Report Template
```markdown
## Validation Report Template
**Date**: [YYYY-MM-DD]
**Scope**: [skill(s) validated]
### Checks
- [ ] Examples compile or run
- [ ] Checklists match current repo conventions
- [ ] External references still valid
- [ ] No duplicated or conflicting guidance
### Findings
- [Finding 1]
- [Finding 2]
### Actions
- [Action 1]
- [Action 2]
```
## Memory File Structure
**Default (bundled with this skill — preferred for Cursor workspaces)**
```
.cursor/skills/self-improving-agent/
├── memory/
│ ├── semantic-patterns.json
│ ├── episodic/
│ │ └── YYYY/
│ │ └── YYYY-MM-DD-{topic}.json
│ └── working/
│ ├── last_error.json # optional; hooks/post-bash.sh may write on shell failure
│ └── current_session.json
```
**Optional global mirror** (e.g. Claude Code — only if you deliberately sync)
```
~/.claude/memory/
├── semantic/
├── episodic/
├── working/
└── index.json
```
Pick **one** canonical tree per project to avoid conflicting patterns.
## Automatic Workflow Integration
```
Any Skill Run
-> workflow-orchestrator
-> self-improving-agent (background)
-> create-pr (ask_first)
-> session-logger (auto)
```
## Continuous Learning Metrics
```json
{
"metrics": {
"patterns_learned": 47,
"patterns_applied": 238,
"skills_updated": 12,
"avg_confidence": 0.87,
"user_satisfaction_trend": "improving",
"error_rate_reduction": "-35%",
"self_corrections": 8
}
}
```
## Human-in-the-Loop
### Feedback Collection
```markdown
## Self-Improvement Summary
I've learned from our session and updated:
### Updated Skills
- `debugger`: Added callback verification pattern
- `prd-planner`: Enhanced UI/UX specification requirements
### Patterns Extracted
1. **state_monitoring_over_callbacks**: Use usePrevious for state-driven side effects
2. **ui_ux_specification_granularity**: Explicit visual specs prevent rework
### Confidence Levels
- New patterns: 0.85 (needs validation)
- Reinforced patterns: 0.95 (well-established)
### Your Feedback
Rate these improvements (1-10):
- Were the updates helpful?
- Should I apply this pattern more broadly?
- Any corrections needed?
```
### Feedback Integration
```yaml
User Feedback:
positive (rating >= 7):
action: Increase pattern confidence
scope: Expand to related skills
neutral (rating 4-6):
action: Keep pattern, gather more data
scope: Current skill only
negative (rating <= 3):
action: Decrease confidence, revise pattern
scope: Remove from active patterns
```
## Templates
| Template | Purpose |
|----------|---------|
| `templates/pattern-template.md` | Adding new patterns |
| `templates/correction-template.md` | Fixing incorrect guidance |
| `templates/validation-template.md` | Validating skill accuracy |
## References
- [SimpleMem: Efficient Lifelong Memory for LLM Agents](https://arxiv.org/html/2601.02553v1)
- [A Survey on the Memory Mechanism of Large Language Model Agents](https://dl.acm.org/doi/10.1145/3748302)
- [Lifelong Learning of LLM based Agents](https://arxiv.org/html/2501.07278v1)
- [Evo-Memory: DeepMind's Benchmark](https://shothota.medium.com/evo-memory-deepminds-new-benchmark)
- [Let's Build a Self-Improving AI Agent](https://medium.com/@nomannayeem/lets-build-a-self-improving-ai-agent-that-learns-from-your-feedback-722d2ce9c2d9)
.cursor/skills/self-improving-agent/templates/correction-template.md deleted 100644 → 0
View file @ 82116292
# Correction Template
## Issue Summary
## Previous Guidance
## Corrected Guidance
## Root Cause
## Follow-up Actions
.cursor/skills/self-improving-agent/templates/pattern-template.md deleted 100644 → 0
View file @ 82116292
# Pattern Template
## Pattern Name
## Context
- Source skill:
- Situation:
## Guidance
## Examples
## Confidence
- Initial confidence:
- Validation notes:
.cursor/skills/self-improving-agent/templates/validation-template.md deleted 100644 → 0
View file @ 82116292
# Validation Template
## Date
## Scope
## Checks
- [ ] Examples compile or run
- [ ] Guidance matches current repo conventions
- [ ] External references still valid
## Findings
## Actions
.cursor/skills/skill-creator/LICENSE.txt deleted 100644 → 0
View file @ 82116292
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
.cursor/skills/skill-creator/SKILL.md deleted 100644 → 0
View file @ 82116292
---
name: skill-creator
description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
---
# Skill Creator
A skill for creating new skills and iteratively improving them.
At a high level, the process of creating a skill goes like this:
- Decide what you want the skill to do and roughly how it should do it
- Write a draft of the skill
- Create a few test prompts and run claude-with-access-to-the-skill on them
- Help the user evaluate the results both qualitatively and quantitatively
- While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)
- Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics
- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)
- Repeat until you're satisfied
- Expand the test set and try again at larger scale
Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.
On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.
Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead.
Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.
Cool? Cool.
## Communicating with the user
The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate.
So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:
- "evaluation" and "benchmark" are borderline, but OK
- for "JSON" and "assertion" you want to see serious cues from the user that they know what those things are before using them without explaining them
It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.
---
## Creating a skill
### Capture Intent
Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.
1. What should this skill enable Claude to do?
2. When should this skill trigger? (what user phrases/contexts)
3. What's the expected output format?
4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.
### Interview and Research
Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.
Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.
### Write the SKILL.md
Based on the user interview, fill in these components:
- **name**: Skill identifier
- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All "when to use" info goes here, not in the body. Note: currently Claude has a tendency to "undertrigger" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit "pushy". So for instance, instead of "How to build a simple fast dashboard to display internal Anthropic data.", you might write "How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'"
- **compatibility**: Required tools, dependencies (optional, rarely needed)
- **the rest of the skill :)**
### Skill Writing Guide
#### Anatomy of a Skill
```
skill-name/
├── SKILL.md (required)
│ ├── YAML frontmatter (name, description required)
│ └── Markdown instructions
└── Bundled Resources (optional)
├── scripts/ - Executable code for deterministic/repetitive tasks
├── references/ - Docs loaded into context as needed
└── assets/ - Files used in output (templates, icons, fonts)
```
#### Progressive Disclosure
Skills use a three-level loading system:
1. **Metadata** (name + description) - Always in context (~100 words)
2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)
3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)
These word counts are approximate and you can feel free to go longer if needed.
**Key patterns:**
- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.
- Reference files clearly from SKILL.md with guidance on when to read them
- For large reference files (>300 lines), include a table of contents
**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:
```
cloud-deploy/
├── SKILL.md (workflow + selection)
└── references/
├── aws.md
├── gcp.md
└── azure.md
```
Claude reads only the relevant reference file.
#### Principle of Lack of Surprise
This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.
#### Writing Patterns
Prefer using the imperative form in instructions.
**Defining output formats** - You can do it like this:
```markdown
## Report structure
ALWAYS use this exact template:
# [Title]
## Executive summary
## Key findings
## Recommendations
```
**Examples pattern** - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little):
```markdown
## Commit message format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication
```
### Writing Style
Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.
### Test Cases
After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them.
Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.
```json
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "User's task prompt",
"expected_output": "Description of expected result",
"files": []
}
]
}
```
See `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).
## Running and evaluating test cases
This section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.
Put results in `<skill-name>-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.
### Step 1: Spawn all runs (with-skill AND baseline) in the same turn
For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.
**With-skill run:**
```
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">
```
**Baseline run** (same prompt, but the baseline depends on context):
- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.
- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r <skill-path> <workspace>/skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.
Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.
```json
{
"eval_id": 0,
"eval_name": "descriptive-name-here",
"prompt": "The user's task prompt",
"assertions": []
}
```
### Step 2: While runs are in progress, draft assertions
Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.
Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.
Update the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.
### Step 3: As runs complete, capture timing data
When each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:
```json
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3
}
```
This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.
### Step 4: Grade, aggregate, and launch the viewer
Once all runs are done:
1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.
2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:
```bash
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
```
This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.
Put each with_skill version before its baseline counterpart.
3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.
4. **Launch the viewer** with both qualitative outputs and quantitative data:
```bash
nohup python <skill-creator-path>/eval-viewer/generate_review.py \
<workspace>/iteration-N \
--skill-name "my-skill" \
--benchmark <workspace>/iteration-N/benchmark.json \
> /dev/null 2>&1 &
VIEWER_PID=$!
```
For iteration 2+, also pass `--previous-workspace <workspace>/iteration-<N-1>`.
**Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks "Submit All Reviews". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.
Note: please use generate_review.py to create the viewer; there's no need to write custom HTML.
5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know."
### What the user sees in the viewer
The "Outputs" tab shows one test case at a time:
- **Prompt**: the task that was given
- **Output**: the files the skill produced, rendered inline where possible
- **Previous Output** (iteration 2+): collapsed section showing last iteration's output
- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail
- **Feedback**: a textbox that auto-saves as they type
- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox
The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.
Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to `feedback.json`.
### Step 5: Read the feedback
When the user tells you they're done, read `feedback.json`:
```json
{
"reviews": [
{"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
{"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},
{"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."}
],
"status": "complete"
}
```
Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.
Kill the viewer server when you're done with it:
```bash
kill $VIEWER_PID 2>/dev/null
```
---
## Improving the skill
This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.
### How to think about improvements
1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.
2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.
3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.
4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.
This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.
### The iteration loop
After improving the skill:
1. Apply your improvements to the skill
2. Rerun all test cases into a new `iteration-<N+1>/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.
3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration
4. Wait for the user to review and tell you they're done
5. Read the new feedback, improve again, repeat
Keep going until:
- The user says they're happy
- The feedback is all empty (everything looks good)
- You're not making meaningful progress
---
## Advanced: Blind comparison
For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.
This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.
---
## Description Optimization
The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.
### Step 1: Generate trigger eval queries
Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:
```json
[
{"query": "the user prompt", "should_trigger": true},
{"query": "another prompt", "should_trigger": false}
]
```
The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).
Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"`
Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"`
For the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.
For the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.
The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.
### Step 2: Review with user
Present the eval set to the user for review using the HTML template:
1. Read the template from `assets/eval_review.html`
2. Replace the placeholders:
- `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)
- `__SKILL_NAME_PLACEHOLDER__` → the skill's name
- `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description
3. Write to a temp file (e.g., `/tmp/eval_review_<skill-name>.html`) and open it: `open /tmp/eval_review_<skill-name>.html`
4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set"
5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)
This step matters — bad eval queries lead to bad descriptions.
### Step 3: Run the optimization loop
Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."
Save the eval set to the workspace, then run in the background:
```bash
python -m scripts.run_loop \
--eval-set <path-to-trigger-eval.json> \
--skill-path <path-to-skill> \
--model <model-id-powering-this-session> \
--max-iterations 5 \
--verbose
```
Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.
While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.
This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.
### How skill triggering works
Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.
This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.
### Step 4: Apply the result
Take `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.
---
### Package and Present (only if `present_files` tool is available)
Check whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:
```bash
python -m scripts.package_skill <path/to/skill-folder>
```
After packaging, direct the user to the resulting `.skill` file path so they can install it.
---
## Claude.ai-specific instructions
In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:
**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.
**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"
**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.
**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.
**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.
**Blind comparison**: Requires subagents. Skip it.
**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.
**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:
- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).
- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.
- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.
---
## Cowork-Specific Instructions
If you're in Cowork, the main things to know are:
- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)
- You don't have a browser or display, so when generating the eval viewer, use `--static <output_path>` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.
- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!
- Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).
- Packaging works — `package_skill.py` just needs Python and a filesystem.
- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.
- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.
---
## Reference files
The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.
- `agents/grader.md` — How to evaluate assertions against outputs
- `agents/comparator.md` — How to do blind A/B comparison between two outputs
- `agents/analyzer.md` — How to analyze why one version beat another
The references/ directory has additional documentation:
- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.
---
Repeating one more time the core loop here for emphasis:
- Figure out what the skill is about
- Draft or edit the skill
- Run claude-with-access-to-the-skill on test prompts
- With the user, evaluate the outputs:
- Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them
- Run quantitative evals
- Repeat until you and the user are satisfied
- Package the final skill and return it to the user.
Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" in your TodoList to make sure it happens.
Good luck!
.cursor/skills/skill-creator/agents/analyzer.md deleted 100644 → 0
View file @ 82116292
# Post-hoc Analyzer Agent
Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.
## Role
After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?
## Inputs
You receive these parameters in your prompt:
- **winner**: "A" or "B" (from blind comparison)
- **winner_skill_path**: Path to the skill that produced the winning output
- **winner_transcript_path**: Path to the execution transcript for the winner
- **loser_skill_path**: Path to the skill that produced the losing output
- **loser_transcript_path**: Path to the execution transcript for the loser
- **comparison_result_path**: Path to the blind comparator's output JSON
- **output_path**: Where to save the analysis results
## Process
### Step 1: Read Comparison Result
1. Read the blind comparator's output at comparison_result_path
2. Note the winning side (A or B), the reasoning, and any scores
3. Understand what the comparator valued in the winning output
### Step 2: Read Both Skills
1. Read the winner skill's SKILL.md and key referenced files
2. Read the loser skill's SKILL.md and key referenced files
3. Identify structural differences:
- Instructions clarity and specificity
- Script/tool usage patterns
- Example coverage
- Edge case handling
### Step 3: Read Both Transcripts
1. Read the winner's transcript
2. Read the loser's transcript
3. Compare execution patterns:
- How closely did each follow their skill's instructions?
- What tools were used differently?
- Where did the loser diverge from optimal behavior?
- Did either encounter errors or make recovery attempts?
### Step 4: Analyze Instruction Following
For each transcript, evaluate:
- Did the agent follow the skill's explicit instructions?
- Did the agent use the skill's provided tools/scripts?
- Were there missed opportunities to leverage skill content?
- Did the agent add unnecessary steps not in the skill?
Score instruction following 1-10 and note specific issues.
### Step 5: Identify Winner Strengths
Determine what made the winner better:
- Clearer instructions that led to better behavior?
- Better scripts/tools that produced better output?
- More comprehensive examples that guided edge cases?
- Better error handling guidance?
Be specific. Quote from skills/transcripts where relevant.
### Step 6: Identify Loser Weaknesses
Determine what held the loser back:
- Ambiguous instructions that led to suboptimal choices?
- Missing tools/scripts that forced workarounds?
- Gaps in edge case coverage?
- Poor error handling that caused failures?
### Step 7: Generate Improvement Suggestions
Based on the analysis, produce actionable suggestions for improving the loser skill:
- Specific instruction changes to make
- Tools/scripts to add or modify
- Examples to include
- Edge cases to address
Prioritize by impact. Focus on changes that would have changed the outcome.
### Step 8: Write Analysis Results
Save structured analysis to `{output_path}`.
## Output Format
Write a JSON file with this structure:
```json
{
"comparison_summary": {
"winner": "A",
"winner_skill": "path/to/winner/skill",
"loser_skill": "path/to/loser/skill",
"comparator_reasoning": "Brief summary of why comparator chose winner"
},
"winner_strengths": [
"Clear step-by-step instructions for handling multi-page documents",
"Included validation script that caught formatting errors",
"Explicit guidance on fallback behavior when OCR fails"
],
"loser_weaknesses": [
"Vague instruction 'process the document appropriately' led to inconsistent behavior",
"No script for validation, agent had to improvise and made errors",
"No guidance on OCR failure, agent gave up instead of trying alternatives"
],
"instruction_following": {
"winner": {
"score": 9,
"issues": [
"Minor: skipped optional logging step"
]
},
"loser": {
"score": 6,
"issues": [
"Did not use the skill's formatting template",
"Invented own approach instead of following step 3",
"Missed the 'always validate output' instruction"
]
}
},
"improvement_suggestions": [
{
"priority": "high",
"category": "instructions",
"suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
"expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
},
{
"priority": "high",
"category": "tools",
"suggestion": "Add validate_output.py script similar to winner skill's validation approach",
"expected_impact": "Would catch formatting errors before final output"
},
{
"priority": "medium",
"category": "error_handling",
"suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
"expected_impact": "Would prevent early failure on difficult documents"
}
],
"transcript_insights": {
"winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
"loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
}
}
```
## Guidelines
- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear"
- **Be actionable**: Suggestions should be concrete changes, not vague advice
- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent
- **Prioritize by impact**: Which changes would most likely have changed the outcome?
- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?
- **Stay objective**: Analyze what happened, don't editorialize
- **Think about generalization**: Would this improvement help on other evals too?
## Categories for Suggestions
Use these categories to organize improvement suggestions:
| Category | Description |
|----------|-------------|
| `instructions` | Changes to the skill's prose instructions |
| `tools` | Scripts, templates, or utilities to add/modify |
| `examples` | Example inputs/outputs to include |
| `error_handling` | Guidance for handling failures |
| `structure` | Reorganization of skill content |
| `references` | External docs or resources to add |
## Priority Levels
- **high**: Would likely change the outcome of this comparison
- **medium**: Would improve quality but may not change win/loss
- **low**: Nice to have, marginal improvement
---
# Analyzing Benchmark Results
When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.
## Role
Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.
## Inputs
You receive these parameters in your prompt:
- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results
- **skill_path**: Path to the skill being benchmarked
- **output_path**: Where to save the notes (as JSON array of strings)
## Process
### Step 1: Read Benchmark Data
1. Read the benchmark.json containing all run results
2. Note the configurations tested (with_skill, without_skill)
3. Understand the run_summary aggregates already calculated
### Step 2: Analyze Per-Assertion Patterns
For each expectation across all runs:
- Does it **always pass** in both configurations? (may not differentiate skill value)
- Does it **always fail** in both configurations? (may be broken or beyond capability)
- Does it **always pass with skill but fail without**? (skill clearly adds value here)
- Does it **always fail with skill but pass without**? (skill may be hurting)
- Is it **highly variable**? (flaky expectation or non-deterministic behavior)
### Step 3: Analyze Cross-Eval Patterns
Look for patterns across evals:
- Are certain eval types consistently harder/easier?
- Do some evals show high variance while others are stable?
- Are there surprising results that contradict expectations?
### Step 4: Analyze Metrics Patterns
Look at time_seconds, tokens, tool_calls:
- Does the skill significantly increase execution time?
- Is there high variance in resource usage?
- Are there outlier runs that skew the aggregates?
### Step 5: Generate Notes
Write freeform observations as a list of strings. Each note should:
- State a specific observation
- Be grounded in the data (not speculation)
- Help the user understand something the aggregate metrics don't show
Examples:
- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"
- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
- "Skill adds 13s average execution time but improves pass rate by 50%"
- "Token usage is 80% higher with skill, primarily due to script output parsing"
- "All 3 without-skill runs for eval 1 produced empty output"
### Step 6: Write Notes
Save notes to `{output_path}` as a JSON array of strings:
```json
[
"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
"Without-skill runs consistently fail on table extraction expectations",
"Skill adds 13s average execution time but improves pass rate by 50%"
]
```
## Guidelines
**DO:**
- Report what you observe in the data
- Be specific about which evals, expectations, or runs you're referring to
- Note patterns that aggregate metrics would hide
- Provide context that helps interpret the numbers
**DO NOT:**
- Suggest improvements to the skill (that's for the improvement step, not benchmarking)
- Make subjective quality judgments ("the output was good/bad")
- Speculate about causes without evidence
- Repeat information already in the run_summary aggregates
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment