Skip to content
GitLab
Commit 2eef70bd authored by liguangyu06's avatar liguangyu06
Browse files

补传知识库、Cursor 规则与扫描快照到 temp，补全项目备份。 

仍不入库密钥、登录态、依赖包和本地执行报告。
parent 6a181bea
Hide whitespace changes
Inline Side-by-side
.cursor/skills/mysql/references/primary-keys.md 0 → 100644
View file @ 2eef70bd
---
title: Primary Key Design
description: Primary key patterns
tags: mysql, primary-keys, auto-increment, uuid, innodb
---
# Primary Keys
InnoDB stores rows in primary key order (clustered index). This means:
- **Sequential keys = optimal inserts**: new rows append, minimizing page splits and fragmentation.
- **Random keys = fragmentation**: random inserts cause page splits to maintain PK order, wasting space and slowing inserts.
- **Secondary index lookups**: secondary indexes store the PK value and use it to fetch the full row from the clustered index.
## INT vs BIGINT for Primary Keys
- **INT UNSIGNED**: 4 bytes, max ~4.3B rows.
- **BIGINT UNSIGNED**: 8 bytes, max ~18.4 quintillion rows.
Guideline: default to **BIGINT UNSIGNED** unless you're certain the table will never approach the INT limit. The extra 4 bytes is usually cheaper than the risk of exhausting INT.
## Avoid Random UUID as Clustered PK
- UUID PK stored as `BINARY(16)`: 16 bytes (vs 8 for BIGINT). Random inserts cause page splits, and every secondary index entry carries the PK.
- UUID stored as `CHAR(36)`/`VARCHAR(36)`: 36 bytes (+ overhead) and is generally worse for storage and index size.
- If external identifiers are required, store UUID as `BINARY(16)` in a secondary unique column:
```sql
CREATE TABLE users (
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
public_id BINARY(16) NOT NULL,
UNIQUE KEY idx_public_id (public_id)
);
-- UUID_TO_BIN(uuid, 1) reorders UUIDv1 bytes to be roughly time-sorted (reduces fragmentation)
-- MySQL's UUID() returns UUIDv4 (random). For time-ordered IDs, use app-generated UUIDv7/ULID/Snowflake.
INSERT INTO users (public_id) VALUES (UUID_TO_BIN(?, 1)); -- app provides UUID string
```
If UUIDs are required, prefer time-ordered variants such as UUIDv7 (app-generated) to reduce index fragmentation.
## Secondary Indexes Include the Primary Key
InnoDB secondary indexes store the primary key value with each index entry. Implications:
- **Larger secondary indexes**: a secondary index entry includes (indexed columns + PK bytes).
- **Covering reads**: `SELECT id FROM users WHERE email = ?` can often be satisfied from `INDEX(email)` because `id` (PK) is already present in the index entry.
- **UUID penalty**: a `BINARY(16)` PK makes every secondary index entry 8 bytes larger than a BIGINT PK.
## Auto-Increment Considerations
- **Hot spot**: inserts target the end of the clustered index (usually fine; can bottleneck at extreme insert rates).
- **Gaps are normal**: rollbacks or failed inserts can leave gaps.
- **Locking**: auto-increment allocation can introduce contention under very high concurrency.
## Alternative Ordered IDs (Snowflake / ULID / UUIDv7)
If you need globally unique IDs generated outside the database:
- **Snowflake-style**: 64-bit integers (fits in BIGINT), time-ordered, compact.
- **ULID / UUIDv7**: 128-bit (store as `BINARY(16)`), time-ordered, better insert locality than random UUIDv4.
Recommendation: prefer `BIGINT AUTO_INCREMENT` unless you need distributed ID generation or externally meaningful identifiers.
## Replication Considerations
- Random-key insert patterns (UUIDv4) can amplify page splits and I/O on replicas too, increasing lag.
- Time-ordered IDs reduce fragmentation and tend to replicate more smoothly under heavy insert workloads.
## Composite Primary Keys
Use for join/many-to-many tables. Most-queried column first:
```sql
CREATE TABLE user_roles (
user_id BIGINT UNSIGNED NOT NULL,
role_id BIGINT UNSIGNED NOT NULL,
PRIMARY KEY (user_id, role_id)
);
```
.cursor/skills/mysql/references/query-optimization-pitfalls.md 0 → 100644
View file @ 2eef70bd
---
title: Query Optimization Pitfalls
description: Common anti-patterns that silently kill performance
tags: mysql, query-optimization, anti-patterns, performance, indexes
---
# Query Optimization Pitfalls
These patterns look correct but bypass indexes or cause full scans.
## Non-Sargable Predicates
A **sargable** predicate can use an index. Common non-sargable patterns:
- functions/arithmetic on indexed columns
- implicit type conversions
- leading wildcards (`LIKE '%x'`)
- some negations (`!=`, `NOT IN`, `NOT LIKE`) depending on shape/data
## Functions on Indexed Columns
```sql
-- BAD: function prevents index use on created_at
WHERE YEAR(created_at) = 2024
-- GOOD: sargable range
WHERE created_at >= '2024-01-01' AND created_at < '2025-01-01'
```
MySQL 8.0+ can use expression (functional) indexes for some cases:
```sql
CREATE INDEX idx_users_upper_name ON users ((UPPER(name)));
-- Now this can use idx_users_upper_name:
WHERE UPPER(name) = 'SMITH'
```
## Implicit Type Conversions
Implicit casts can make indexes unusable:
```sql
-- If phone is VARCHAR, this may force CAST(phone AS UNSIGNED) and scan
WHERE phone = 1234567890
-- Better: match the column type
WHERE phone = '1234567890'
```
## LIKE Patterns
```sql
-- BAD: leading wildcard cannot use a B-Tree index
WHERE name LIKE '%smith'
WHERE name LIKE '%smith%'
-- GOOD: prefix match can use an index
WHERE name LIKE 'smith%'
```
For suffix search, consider storing a reversed generated column + prefix search:
```sql
ALTER TABLE users
ADD COLUMN name_reversed VARCHAR(255) AS (REVERSE(name)) STORED,
ADD INDEX idx_users_name_reversed (name_reversed);
WHERE name_reversed LIKE CONCAT(REVERSE('smith'), '%');
```
For infix search at scale, use `FULLTEXT` (when appropriate) or a dedicated search engine.
## `OR` Across Different Columns
`OR` across different columns often prevents efficient index use.
```sql
-- Often suboptimal
WHERE status = 'active' OR region = 'us-east'
-- Often better: two indexed queries
SELECT * FROM orders WHERE status = 'active'
UNION ALL
SELECT * FROM orders WHERE region = 'us-east';
```
MySQL can sometimes use `index_merge`, but it's frequently slower than a purpose-built composite index or a UNION rewrite.
## ORDER BY + LIMIT Without an Index
`LIMIT` does not automatically make sorting cheap. If no index supports the order, MySQL may sort many rows (`Using filesort`) and then apply LIMIT.
```sql
-- Needs an index on created_at (or it will filesort)
SELECT * FROM orders ORDER BY created_at DESC LIMIT 10;
-- For WHERE + ORDER BY, you usually need a composite index:
-- (status, created_at DESC)
SELECT * FROM orders
WHERE status = 'pending'
ORDER BY created_at DESC
LIMIT 10;
```
## DISTINCT / GROUP BY
`DISTINCT` and `GROUP BY` can trigger temp tables and sorts (`Using temporary`, `Using filesort`) when indexes don't match.
```sql
-- Often improved by an index on (status)
SELECT DISTINCT status FROM orders;
-- Often improved by an index on (status)
SELECT status, COUNT(*) FROM orders GROUP BY status;
```
## Derived Tables / CTE Materialization
Derived tables and CTEs may be materialized into temporary tables, which can be slower than a flattened query. If performance is surprising, check `EXPLAIN` and consider rewriting the query or adding supporting indexes.
## Other Quick Rules
- **`OFFSET` pagination**: `OFFSET N` scans and discards N rows. Use cursor-based pagination.
- **`SELECT *`** defeats covering indexes. Select only needed columns.
- **`NOT IN` with NULLs**: `NOT IN (subquery)` returns no rows if subquery contains any NULL. Use `NOT EXISTS`.
- **`COUNT(*)` vs `COUNT(col)`**: `COUNT(*)` counts all rows; `COUNT(col)` skips NULLs.
- **Arithmetic on indexed columns**: `WHERE price * 1.1 > 100` prevents index use. Rewrite to keep the column bare: `WHERE price > 100 / 1.1`.
.cursor/skills/mysql/references/replication-lag.md 0 → 100644
View file @ 2eef70bd
---
title: Replication Lag Awareness
description: Read-replica consistency pitfalls and mitigations
tags: mysql, replication, lag, read-replicas, consistency, gtid
---
# Replication Lag
MySQL replication is asynchronous by default. Reads from a replica may return stale data.
## The Core Problem
1. App writes to primary: `INSERT INTO orders ...`
2. App immediately reads from replica: `SELECT * FROM orders WHERE id = ?`
3. Replica hasn't applied the write yet — returns empty or stale data.
## Detecting Lag
```sql
-- On the replica
SHOW REPLICA STATUS\G
-- Key field: Seconds_Behind_Source (0 = caught up, NULL = not replicating)
```
**Warning**: `Seconds_Behind_Source` measures relay-log lag, not true wall-clock staleness. It can underreport during long-running transactions because it only updates when transactions commit.
**GTID-based lag**: for more accurate tracking, compare `@@global.gtid_executed` (replica) to primary GTID position, or use `WAIT_FOR_EXECUTED_GTID_SET()` to wait for a specific transaction.
**Note**: parallel replication with `replica_parallel_type=LOGICAL_CLOCK` requires `binlog_format=ROW`. Statement-based replication (`binlog_format=STATEMENT`) is more limited for parallel apply.
## Mitigation Strategies
| Strategy | How | Trade-off |
|---|---|---|
| **Read from primary** | Route critical reads to primary after writes | Increases primary load |
| **Sticky sessions** | Pin user to primary for N seconds after a write | Adds session affinity complexity |
| **GTID wait** | `SELECT WAIT_FOR_EXECUTED_GTID_SET('gtid', timeout)` on replica | Adds latency equal to lag |
| **Semi-sync replication** | Primary waits for >=1 replica ACK before committing | Higher write latency |
## Common Pitfalls
- **Large transactions cause lag spikes**: A single `INSERT ... SELECT` of 1M rows replays as one big transaction on the replica. Break into batches.
- **DDL blocks replication**: `ALTER TABLE` with `ALGORITHM=COPY` on primary replays on replica, blocking other relay-log events during execution. `INSTANT` and `INPLACE` DDL are less blocking but still require brief metadata locks.
- **Long queries on replica**: A slow `SELECT` on the replica can block relay-log application. Use `replica_parallel_workers` (8.0+) with `replica_parallel_type=LOGICAL_CLOCK` for parallel apply. Note: LOGICAL_CLOCK requires `binlog_format=ROW` and `slave_preserve_commit_order=ON` (or `replica_preserve_commit_order=ON`) to preserve commit order.
- **IO thread bottlenecks**: Network latency, disk I/O, or `relay_log_space_limit` exhaustion can cause lag even when the SQL apply thread isn't saturated. Monitor `Relay_Log_Space` and connectivity.
## Guidelines
- Assume replicas are always slightly behind. Design reads accordingly.
- Use GTID-based replication for reliable failover and lag tracking.
- Monitor `Seconds_Behind_Source` with alerting (>5s warrants investigation).
.cursor/skills/mysql/references/row-locking-gotchas.md 0 → 100644
View file @ 2eef70bd
---
title: InnoDB Row Locking Gotchas
description: Gap locks, next-key locks, and surprise escalation
tags: mysql, innodb, locking, gap-locks, next-key-locks, concurrency
---
# Row Locking Gotchas
InnoDB uses row-level locking, but the actual locked range is often wider than expected.
## Next-Key Locks (REPEATABLE READ)
InnoDB's default isolation level uses next-key locks for **locking reads** (`SELECT ... FOR UPDATE`, `SELECT ... FOR SHARE`, `UPDATE`, `DELETE`) to prevent phantom reads. A range scan locks every gap in that range. Plain `SELECT` statements use consistent reads (MVCC) and don't acquire locks.
**Exception**: a unique index search with a unique search condition (e.g., `WHERE id = 5` on a unique `id`) locks only the index record, not the gap. Gap/next-key locks still apply for range scans and non-unique searches.
```sql
-- Locks rows with id 5..10 AND the gaps between them and after the range
SELECT * FROM orders WHERE id BETWEEN 5 AND 10 FOR UPDATE;
-- Another session inserting id=7 blocks until the lock is released.
```
## Gap Locks on Non-Existent Rows
`SELECT ... FOR UPDATE` on a row that doesn't exist still places a gap lock:
```sql
-- No row with id=999 exists, but this locks the gap around where 999 would be
SELECT * FROM orders WHERE id = 999 FOR UPDATE;
-- Concurrent INSERTs into that gap are blocked.
```
## Index-Less UPDATE/DELETE = Full Scan and Broad Locking
If the WHERE column has no index, InnoDB must scan all rows and locks every row examined (often effectively all rows in the table). This is not table-level locking—InnoDB doesn't escalate locks—but rather row-level locks on all rows:
```sql
-- No index on status → locks all rows (not a table lock, but all row locks)
UPDATE orders SET processed = 1 WHERE status = 'pending';
-- Fix: CREATE INDEX idx_status ON orders (status);
```
## SELECT ... FOR SHARE (Shared Locks)
`SELECT ... FOR SHARE` acquires shared (S) locks instead of exclusive (X) locks. Multiple sessions can hold shared locks simultaneously, but exclusive locks are blocked:
```sql
-- Session 1: shared lock
SELECT * FROM orders WHERE id = 5 FOR SHARE;
-- Session 2: also allowed (shared lock)
SELECT * FROM orders WHERE id = 5 FOR SHARE;
-- Session 3: blocked until shared locks are released
UPDATE orders SET status = 'processed' WHERE id = 5;
```
Gap/next-key locks can still apply in REPEATABLE READ, so inserts into locked gaps may be blocked even with shared locks.
## INSERT ... ON DUPLICATE KEY UPDATE
Takes an exclusive next-key lock on the index entry. If multiple sessions do this concurrently on nearby key values, gap-lock deadlocks are common.
## Lock Escalation Misconception
InnoDB does **not** automatically escalate row locks to table locks. When a missing index causes "table-wide" locking, it's because InnoDB scans and locks all rows individually—not because locks were escalated.
## Mitigation Strategies
- **Use READ COMMITTED** when gap locks cause excessive blocking (gap locks disabled in RC except for FK/duplicate-key checks).
- **Keep transactions short** — hold locks for milliseconds, not seconds.
- **Ensure WHERE columns are indexed** to avoid full-table lock scans.
.cursor/skills/ocr-document-processor/SKILL.md 0 → 100644
View file @ 2eef70bd
---
name: ocr-document-processor
description: Extract text and structure from scans, images, and scanned PDFs. Use for OCR, searchable PDFs, table extraction, receipt parsing, and business card parsing.
---
# OCR Document Processor
Handle OCR-heavy inputs where text must be recovered from images or scanned pages.
## Use This For
- OCR on images and scanned PDFs
- Searchable PDF export
- Structured extraction to text, markdown, JSON, or HTML
- Table extraction from scanned material
- Receipt parsing and business card parsing
## Workflow
1. Decide whether plain OCR, structured extraction, or document-specific parsing is needed.
2. Preprocess noisy inputs before extraction when skew, blur, or shadows are present.
3. Use `scripts/ocr_processor.py` for core OCR tasks.
4. Use the focused helpers when the input is specialized:
- `scripts/business_card_scanner.py`
- `scripts/receipt_scanner.py`
5. Return confidence caveats when the source is low quality, rotated, handwritten, or multilingual.
## Guardrails
- Prefer explicit language selection when accuracy matters.
- Do not claim fields are exact when OCR confidence is weak.
- Route non-scanned digital PDFs to `document-converter-suite` instead of OCR by default.
.cursor/skills/ocr-document-processor/agents/openai.yaml 0 → 100644
View file @ 2eef70bd
display_name: 'OCR Document Processor'
short_description: 'Extract text and structure from scans, receipts, and cards.'
default_prompt: 'Help me OCR this scanned document or image.'
.cursor/skills/ocr-document-processor/scripts/business_card_scanner.py 0 → 100644
View file @ 2eef70bd
#!/usr/bin/env python3
"""
Business Card Scanner - Extract contact info from cards.
"""
import argparse
import json
import re
import pytesseract
import cv2
import numpy as np
from PIL import Image
class BusinessCardScanner:
"""Scan business cards."""
def __init__(self):
"""Initialize scanner."""
self.raw_text = ""
self.data = {}
def scan(self, filepath: str) -> 'BusinessCardScanner':
"""Scan business card image."""
img = cv2.imread(filepath)
# Preprocess
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
_, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
# OCR
self.raw_text = pytesseract.image_to_string(thresh)
# Extract data
self.extract_contact_info()
return self
def extract_contact_info(self):
"""Extract contact information."""
lines = [line.strip() for line in self.raw_text.split('\n') if line.strip()]
# Extract email
email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
emails = re.findall(email_pattern, self.raw_text)
self.data['email'] = emails[0] if emails else None
# Extract phone
phone_patterns = [
r'\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}',
r'\+\d{1,3}[-.\s]?\d{1,4}[-.\s]?\d{1,4}[-.\s]?\d{1,9}'
]
for pattern in phone_patterns:
phones = re.findall(pattern, self.raw_text)
if phones:
self.data['phone'] = phones[0]
break
# Extract name (usually first line)
self.data['name'] = lines[0] if lines else None
# Extract company (heuristic: look for Inc, LLC, Ltd)
company_keywords = ['Inc', 'LLC', 'Ltd', 'Corp', 'Company']
for line in lines:
if any(kw in line for kw in company_keywords):
self.data['company'] = line
break
# Extract website
url_pattern = r'www\.[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}'
urls = re.findall(url_pattern, self.raw_text, re.IGNORECASE)
self.data['website'] = urls[0] if urls else None
def get_data(self) -> dict:
"""Get extracted data."""
return self.data
def to_json(self, output: str) -> str:
"""Export to JSON."""
with open(output, 'w') as f:
json.dump(self.data, f, indent=2)
return output
def main():
parser = argparse.ArgumentParser(description="Business Card Scanner")
parser.add_argument("--input", "-i", required=True, help="Business card image")
parser.add_argument("--output", "-o", required=True, help="Output JSON file")
args = parser.parse_args()
scanner = BusinessCardScanner()
scanner.scan(args.input)
data = scanner.get_data()
print("Extracted Contact Information:")
print(f" Name: {data.get('name', 'Not found')}")
print(f" Company: {data.get('company', 'Not found')}")
print(f" Email: {data.get('email', 'Not found')}")
print(f" Phone: {data.get('phone', 'Not found')}")
print(f" Website: {data.get('website', 'Not found')}")
scanner.to_json(args.output)
print(f"\nData saved: {args.output}")
if __name__ == "__main__":
main()
.cursor/skills/ocr-document-processor/scripts/ocr_processor.py 0 → 100644
View file @ 2eef70bd
#!/usr/bin/env python3
"""
OCR Document Processor - Extract text from images and PDFs
Supports multiple languages, structured output, and batch processing.
"""
import io
import json
import re
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union
import numpy as np
from PIL import Image
# Try to import optional dependencies
try:
import pytesseract
HAS_TESSERACT = True
except ImportError:
HAS_TESSERACT = False
try:
import fitz # PyMuPDF
HAS_PYMUPDF = True
except ImportError:
HAS_PYMUPDF = False
try:
import cv2
HAS_CV2 = True
except ImportError:
HAS_CV2 = False
class OCRError(Exception):
"""Custom exception for OCR errors."""
pass
@dataclass
class OCRConfig:
"""Configuration for OCR processing."""
psm: int = 3 # Page segmentation mode
oem: int = 3 # OCR engine mode
dpi: int = 300
timeout: int = 30
min_confidence: int = 60
preserve_layout: bool = True
def update(self, settings: Dict[str, Any]) -> None:
for key, value in settings.items():
if hasattr(self, key):
setattr(self, key, value)
class OCRProcessor:
"""
Main class for OCR processing.
Extracts text from images and PDFs using Tesseract OCR.
"""
def __init__(
self,
source: Union[str, Path, Image.Image, bytes],
lang: str = 'eng'
):
"""
Initialize OCR processor.
Args:
source: Image path, PDF path, PIL Image, or bytes
lang: OCR language code (e.g., 'eng', 'deu', 'eng+fra')
"""
if not HAS_TESSERACT:
raise ImportError("pytesseract is required. Install with: pip install pytesseract")
self.config = OCRConfig()
self.lang = lang
self._images: List[Image.Image] = []
self._preprocessed = False
self._source_path: Optional[Path] = None
# Load source
if isinstance(source, (str, Path)):
self._source_path = Path(source)
self._load_file(self._source_path)
elif isinstance(source, Image.Image):
self._images = [source.copy()]
elif isinstance(source, bytes):
self._images = [Image.open(io.BytesIO(source))]
else:
raise OCRError(f"Unsupported source type: {type(source)}")
def _load_file(self, path: Path) -> None:
"""Load image or PDF file."""
if not path.exists():
raise FileNotFoundError(f"File not found: {path}")
suffix = path.suffix.lower()
if suffix == '.pdf':
self._load_pdf(path)
elif suffix in ['.png', '.jpg', '.jpeg', '.tiff', '.tif', '.bmp', '.gif']:
self._images = [Image.open(path)]
else:
raise OCRError(f"Unsupported file format: {suffix}")
def _load_pdf(self, path: Path) -> None:
"""Load PDF and convert to images."""
if not HAS_PYMUPDF:
raise ImportError("PyMuPDF is required for PDF processing. Install with: pip install PyMuPDF")
doc = fitz.open(path)
self._images = []
for page_num in range(len(doc)):
page = doc[page_num]
# Render at higher DPI for better OCR
mat = fitz.Matrix(self.config.dpi / 72, self.config.dpi / 72)
pix = page.get_pixmap(matrix=mat)
img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)
self._images.append(img)
doc.close()
def preprocess(
self,
deskew: bool = False,
denoise: bool = False,
threshold: bool = False,
threshold_method: str = 'otsu',
contrast: float = 1.0,
sharpen: float = 0,
scale: float = 1.0,
remove_shadows: bool = False
) -> 'OCRProcessor':
"""
Preprocess images for better OCR accuracy.
Args:
deskew: Correct rotation
denoise: Remove noise
threshold: Convert to binary
threshold_method: 'otsu', 'adaptive', 'simple'
contrast: Contrast adjustment (1.0 = no change)
sharpen: Sharpening factor
scale: Upscale factor for small text
remove_shadows: Remove shadow artifacts
Returns:
self for chaining
"""
if not HAS_CV2:
raise ImportError("OpenCV is required for preprocessing. Install with: pip install opencv-python")
processed = []
for img in self._images:
# Convert to numpy array
img_array = np.array(img)
# Convert to grayscale if color
if len(img_array.shape) == 3:
gray = cv2.cvtColor(img_array, cv2.COLOR_RGB2GRAY)
else:
gray = img_array
# Scale up if needed
if scale > 1.0:
gray = cv2.resize(gray, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)
# Denoise
if denoise:
gray = cv2.fastNlMeansDenoising(gray, None, 10, 7, 21)
# Contrast adjustment
if contrast != 1.0:
gray = cv2.convertScaleAbs(gray, alpha=contrast, beta=0)
# Remove shadows
if remove_shadows:
dilated = cv2.dilate(gray, np.ones((7, 7), np.uint8))
bg = cv2.medianBlur(dilated, 21)
gray = 255 - cv2.absdiff(gray, bg)
# Threshold
if threshold:
if threshold_method == 'otsu':
_, gray = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
elif threshold_method == 'adaptive':
gray = cv2.adaptiveThreshold(
gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2
)
else: # simple
_, gray = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
# Sharpen
if sharpen > 0:
kernel = np.array([[-1, -1, -1], [-1, 9 + sharpen, -1], [-1, -1, -1]])
gray = cv2.filter2D(gray, -1, kernel)
# Deskew
if deskew:
gray = self._deskew_image(gray)
# Convert back to PIL Image
processed.append(Image.fromarray(gray))
self._images = processed
self._preprocessed = True
return self
def _deskew_image(self, img: np.ndarray) -> np.ndarray:
"""Deskew a grayscale image."""
# Find all non-zero points
coords = np.column_stack(np.where(img > 0))
if len(coords) == 0:
return img
# Get rotation angle
angle = cv2.minAreaRect(coords)[-1]
if angle < -45:
angle = 90 + angle
elif angle > 45:
angle = angle - 90
# Rotate image
(h, w) = img.shape[:2]
center = (w // 2, h // 2)
M = cv2.getRotationMatrix2D(center, angle, 1.0)
rotated = cv2.warpAffine(
img, M, (w, h),
flags=cv2.INTER_CUBIC,
borderMode=cv2.BORDER_REPLICATE
)
return rotated
def extract_text(self, pages: Optional[List[int]] = None) -> str:
"""
Extract text from all pages/images.
Args:
pages: Specific pages to extract (1-indexed), None for all
Returns:
Extracted text
"""
texts = []
images_to_process = self._images
if pages:
images_to_process = [self._images[p - 1] for p in pages if 0 < p <= len(self._images)]
for img in images_to_process:
custom_config = f'--psm {self.config.psm} --oem {self.config.oem}'
text = pytesseract.image_to_string(
img,
lang=self.lang,
config=custom_config,
timeout=self.config.timeout
)
texts.append(text)
return '\n\n'.join(texts)
def extract_structured(self) -> Dict[str, Any]:
"""
Extract text with structure information.
Returns:
Dict containing text, blocks, lines, words, confidence
"""
all_blocks = []
all_lines = []
all_words = []
full_text = []
total_conf = []
for page_idx, img in enumerate(self._images):
custom_config = f'--psm {self.config.psm} --oem {self.config.oem}'
data = pytesseract.image_to_data(
img,
lang=self.lang,
config=custom_config,
output_type=pytesseract.Output.DICT,
timeout=self.config.timeout
)
# Process data
current_block = None
current_line = None
for i in range(len(data['text'])):
text = data['text'][i].strip()
conf = int(data['conf'][i])
if not text:
continue
word = {
'text': text,
'confidence': conf,
'bbox': [
data['left'][i],
data['top'][i],
data['width'][i],
data['height'][i]
],
'page': page_idx + 1,
'block_num': data['block_num'][i],
'line_num': data['line_num'][i],
'word_num': data['word_num'][i]
}
all_words.append(word)
if conf >= 0:
total_conf.append(conf)
# Get full text for this page
page_text = pytesseract.image_to_string(
img,
lang=self.lang,
config=custom_config,
timeout=self.config.timeout
)
full_text.append(page_text)
# Calculate average confidence
avg_conf = sum(total_conf) / len(total_conf) if total_conf else 0
# Group words into blocks and lines
blocks = self._group_into_blocks(all_words)
lines = self._group_into_lines(all_words)
return {
'text': '\n\n'.join(full_text),
'blocks': blocks,
'lines': lines,
'words': all_words,
'confidence': round(avg_conf, 1),
'language': self.lang,
'pages': len(self._images),
'source': str(self._source_path) if self._source_path else None,
'preprocessed': self._preprocessed
}
def _group_into_blocks(self, words: List[Dict]) -> List[Dict]:
"""Group words into text blocks."""
blocks = {}
for word in words:
key = (word['page'], word['block_num'])
if key not in blocks:
blocks[key] = {
'page': word['page'],
'block_num': word['block_num'],
'words': [],
'text': ''
}
blocks[key]['words'].append(word)
# Build block text
for block in blocks.values():
block['text'] = ' '.join(w['text'] for w in block['words'])
# Calculate bounding box
if block['words']:
min_x = min(w['bbox'][0] for w in block['words'])
min_y = min(w['bbox'][1] for w in block['words'])
max_x = max(w['bbox'][0] + w['bbox'][2] for w in block['words'])
max_y = max(w['bbox'][1] + w['bbox'][3] for w in block['words'])
block['bbox'] = [min_x, min_y, max_x - min_x, max_y - min_y]
return list(blocks.values())
def _group_into_lines(self, words: List[Dict]) -> List[Dict]:
"""Group words into lines."""
lines = {}
for word in words:
key = (word['page'], word['block_num'], word['line_num'])
if key not in lines:
lines[key] = {
'page': word['page'],
'block_num': word['block_num'],
'line_num': word['line_num'],
'words': [],
'text': ''
}
lines[key]['words'].append(word)
for line in lines.values():
line['text'] = ' '.join(w['text'] for w in line['words'])
return list(lines.values())
def extract_by_page(self) -> Dict[int, str]:
"""Extract text page by page."""
results = {}
for idx, img in enumerate(self._images):
custom_config = f'--psm {self.config.psm} --oem {self.config.oem}'
text = pytesseract.image_to_string(
img,
lang=self.lang,
config=custom_config,
timeout=self.config.timeout
)
results[idx + 1] = text
return results
def extract_tables(self) -> List[List[List[str]]]:
"""
Attempt to extract tabular data.
Returns:
List of tables, each table is a list of rows
"""
tables = []
for img in self._images:
# Use table detection mode
custom_config = f'--psm 6 --oem {self.config.oem}'
data = pytesseract.image_to_data(
img,
lang=self.lang,
config=custom_config,
output_type=pytesseract.Output.DICT,
timeout=self.config.timeout
)
# Simple heuristic: group by Y position for rows
lines_by_y = {}
for i in range(len(data['text'])):
text = data['text'][i].strip()
if not text:
continue
y = data['top'][i]
# Round Y to group nearby items
y_bucket = (y // 20) * 20
if y_bucket not in lines_by_y:
lines_by_y[y_bucket] = []
lines_by_y[y_bucket].append({
'text': text,
'x': data['left'][i]
})
# Sort each row by X position
if lines_by_y:
table = []
for y in sorted(lines_by_y.keys()):
row = sorted(lines_by_y[y], key=lambda x: x['x'])
table.append([item['text'] for item in row])
tables.append(table)
return tables
def export_markdown(self, filepath: Union[str, Path]) -> str:
"""Export extracted text as Markdown."""
result = self.extract_structured()
filepath = Path(filepath)
lines = [
f"# OCR Extracted Document",
f"",
f"**Source:** {result['source'] or 'Unknown'}",
f"**Pages:** {result['pages']}",
f"**Language:** {result['language']}",
f"**Confidence:** {result['confidence']}%",
f"**Extracted:** {datetime.now().isoformat()}",
f"",
"---",
"",
]
# Add text content
lines.append(result['text'])
content = '\n'.join(lines)
filepath.write_text(content, encoding='utf-8')
return str(filepath)
def export_json(self, filepath: Union[str, Path]) -> str:
"""Export structured data as JSON."""
result = self.extract_structured()
result['extracted_at'] = datetime.now().isoformat()
filepath = Path(filepath)
filepath.write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding='utf-8')
return str(filepath)
def export_html(self, filepath: Union[str, Path]) -> str:
"""Export as formatted HTML."""
result = self.extract_structured()
filepath = Path(filepath)
html = [
"<!DOCTYPE html>",
"<html><head>",
"<meta charset='utf-8'>",
"<title>OCR Extracted Document</title>",
"<style>",
"body { font-family: sans-serif; max-width: 800px; margin: 40px auto; padding: 20px; }",
"h1 { color: #333; }",
".meta { background: #f5f5f5; padding: 15px; margin: 20px 0; border-radius: 5px; }",
".meta span { display: inline-block; margin-right: 20px; }",
".content { white-space: pre-wrap; line-height: 1.6; }",
".low-conf { background: #fff3cd; }",
"</style>",
"</head><body>",
"<h1>OCR Extracted Document</h1>",
"<div class='meta'>",
f"<span><strong>Source:</strong> {result['source'] or 'Unknown'}</span>",
f"<span><strong>Pages:</strong> {result['pages']}</span>",
f"<span><strong>Confidence:</strong> {result['confidence']}%</span>",
"</div>",
"<div class='content'>",
result['text'].replace('<', '&lt;').replace('>', '&gt;'),
"</div>",
"</body></html>"
]
content = '\n'.join(html)
filepath.write_text(content, encoding='utf-8')
return str(filepath)
def export_searchable_pdf(self, filepath: Union[str, Path]) -> str:
"""Create searchable PDF with OCR text layer."""
if not HAS_PYMUPDF:
raise ImportError("PyMuPDF required for PDF export")
filepath = Path(filepath)
doc = fitz.open()
for img in self._images:
# Get text and positions
custom_config = f'--psm {self.config.psm} --oem {self.config.oem}'
data = pytesseract.image_to_data(
img,
lang=self.lang,
config=custom_config,
output_type=pytesseract.Output.DICT,
timeout=self.config.timeout
)
# Create PDF page from image
img_bytes = io.BytesIO()
img.save(img_bytes, format='PNG')
img_bytes.seek(0)
img_doc = fitz.open(stream=img_bytes, filetype="png")
rect = img_doc[0].rect
page = doc.new_page(width=rect.width, height=rect.height)
page.insert_image(rect, stream=img_bytes.getvalue())
# Add invisible text layer
for i in range(len(data['text'])):
text = data['text'][i].strip()
if not text:
continue
x = data['left'][i]
y = data['top'][i]
w = data['width'][i]
h = data['height'][i]
# Add text annotation
text_rect = fitz.Rect(x, y, x + w, y + h)
page.insert_text(
(x, y + h * 0.8), # Baseline position
text,
fontsize=max(6, h * 0.7),
render_mode=3 # Invisible
)
img_doc.close()
doc.save(filepath)
doc.close()
return str(filepath)
def export_tables_csv(self, output_dir: Union[str, Path]) -> List[str]:
"""Export detected tables as CSV files."""
import csv
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
tables = self.extract_tables()
exported = []
for idx, table in enumerate(tables):
filepath = output_dir / f"table_{idx + 1}.csv"
with open(filepath, 'w', newline='', encoding='utf-8') as f:
writer = csv.writer(f)
writer.writerows(table)
exported.append(str(filepath))
return exported
def export_tables_json(self, filepath: Union[str, Path]) -> str:
"""Export detected tables as JSON."""
tables = self.extract_tables()
filepath = Path(filepath)
filepath.write_text(json.dumps(tables, indent=2, ensure_ascii=False), encoding='utf-8')
return str(filepath)
def parse_receipt(self) -> Dict[str, Any]:
"""Parse receipt/invoice structure."""
text = self.extract_text()
receipt = {
'vendor': None,
'date': None,
'items': [],
'subtotal': None,
'tax': None,
'total': None,
'raw_text': text
}
lines = text.split('\n')
# Try to find vendor (usually first non-empty line)
for line in lines[:5]:
if line.strip():
receipt['vendor'] = line.strip()
break
# Find date patterns
date_patterns = [
r'\d{1,2}/\d{1,2}/\d{2,4}',
r'\d{1,2}-\d{1,2}-\d{2,4}',
r'\d{4}-\d{2}-\d{2}',
]
for pattern in date_patterns:
match = re.search(pattern, text)
if match:
receipt['date'] = match.group()
break
# Find total (look for common patterns)
total_patterns = [
r'total[:\s]*\$?([\d,]+\.?\d*)',
r'amount[:\s]*\$?([\d,]+\.?\d*)',
r'grand total[:\s]*\$?([\d,]+\.?\d*)',
]
for pattern in total_patterns:
match = re.search(pattern, text.lower())
if match:
receipt['total'] = match.group(1)
break
# Find tax
tax_patterns = [
r'tax[:\s]*\$?([\d,]+\.?\d*)',
r'vat[:\s]*\$?([\d,]+\.?\d*)',
]
for pattern in tax_patterns:
match = re.search(pattern, text.lower())
if match:
receipt['tax'] = match.group(1)
break
return receipt
def parse_business_card(self) -> Dict[str, Any]:
"""Parse business card information."""
text = self.extract_text()
contact = {
'name': None,
'title': None,
'company': None,
'email': [],
'phone': [],
'address': None,
'website': [],
'raw_text': text
}
# Find emails
email_pattern = r'[\w\.-]+@[\w\.-]+\.\w+'
contact['email'] = re.findall(email_pattern, text)
# Find phone numbers
phone_pattern = r'[\+\d][\d\s\-\(\)]{8,}'
contact['phone'] = re.findall(phone_pattern, text)
# Find websites
url_pattern = r'(?:www\.)?[\w\-]+\.(?:com|org|net|io|co)(?:/[\w\-]*)?'
contact['website'] = re.findall(url_pattern, text.lower())
# First line often contains name
lines = [l.strip() for l in text.split('\n') if l.strip()]
if lines:
# Name is usually the first or second line
for line in lines[:3]:
# Skip if it looks like email/phone/url
if '@' in line or 'www' in line.lower() or re.search(r'\d{5,}', line):
continue
# Likely a name if it's 2-4 words
words = line.split()
if 1 < len(words) <= 4 and all(w[0].isupper() for w in words if w):
contact['name'] = line
break
return contact
# ==================== BATCH PROCESSING ====================
def batch_ocr(
input_dir: Union[str, Path],
output_dir: Union[str, Path],
output_format: str = 'text',
lang: str = 'eng',
recursive: bool = False,
preprocess: bool = False
) -> Dict[str, Any]:
"""
Process multiple documents with OCR.
Args:
input_dir: Input directory
output_dir: Output directory
output_format: 'text', 'markdown', 'json', 'html'
lang: OCR language
recursive: Include subdirectories
preprocess: Apply preprocessing
Returns:
Results with success/failed counts
"""
input_dir = Path(input_dir)
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
# Find files
extensions = ['*.png', '*.jpg', '*.jpeg', '*.tiff', '*.tif', '*.bmp', '*.pdf']
files = []
for ext in extensions:
if recursive:
files.extend(input_dir.rglob(ext))
else:
files.extend(input_dir.glob(ext))
results = {'success': 0, 'failed': 0, 'errors': []}
for filepath in files:
try:
processor = OCRProcessor(filepath, lang=lang)
if preprocess:
processor.preprocess(denoise=True, threshold=True)
# Determine output path
rel_path = filepath.relative_to(input_dir) if recursive else filepath.name
out_name = Path(rel_path).stem
if output_format == 'text':
out_path = output_dir / f"{out_name}.txt"
out_path.write_text(processor.extract_text(), encoding='utf-8')
elif output_format == 'markdown':
out_path = output_dir / f"{out_name}.md"
processor.export_markdown(out_path)
elif output_format == 'json':
out_path = output_dir / f"{out_name}.json"
processor.export_json(out_path)
elif output_format == 'html':
out_path = output_dir / f"{out_name}.html"
processor.export_html(out_path)
results['success'] += 1
except Exception as e:
results['failed'] += 1
results['errors'].append({'file': str(filepath), 'error': str(e)})
return results
# ==================== CLI ====================
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description='OCR Document Processor')
parser.add_argument('input', help='Input image, PDF, or directory')
parser.add_argument('-o', '--output', help='Output path')
parser.add_argument('--format', choices=['text', 'markdown', 'json', 'html'],
default='text', help='Output format')
parser.add_argument('--lang', default='eng', help='OCR language code')
parser.add_argument('--batch', action='store_true', help='Batch processing mode')
parser.add_argument('--preprocess', action='store_true', help='Apply preprocessing')
parser.add_argument('--deskew', action='store_true', help='Deskew images')
parser.add_argument('--denoise', action='store_true', help='Denoise images')
args = parser.parse_args()
input_path = Path(args.input)
if args.batch or input_path.is_dir():
# Batch mode
output_dir = args.output or 'ocr_output'
results = batch_ocr(
input_path,
output_dir,
output_format=args.format,
lang=args.lang,
preprocess=args.preprocess
)
print(f"Processed: {results['success']} files")
print(f"Failed: {results['failed']} files")
else:
# Single file mode
processor = OCRProcessor(input_path, lang=args.lang)
if args.preprocess or args.deskew or args.denoise:
processor.preprocess(deskew=args.deskew, denoise=args.denoise)
if args.format == 'text':
text = processor.extract_text()
if args.output:
Path(args.output).write_text(text, encoding='utf-8')
else:
print(text)
elif args.format == 'markdown':
output = args.output or input_path.with_suffix('.md')
processor.export_markdown(output)
print(f"Saved to: {output}")
elif args.format == 'json':
output = args.output or input_path.with_suffix('.json')
processor.export_json(output)
print(f"Saved to: {output}")
elif args.format == 'html':
output = args.output or input_path.with_suffix('.html')
processor.export_html(output)
print(f"Saved to: {output}")
.cursor/skills/ocr-document-processor/scripts/receipt_scanner.py 0 → 100644
View file @ 2eef70bd
#!/usr/bin/env python3
"""
Receipt Scanner - Extract data from receipt images using OCR.
"""
import argparse
import json
import re
from pathlib import Path
from typing import Dict, List, Optional
import pytesseract
import cv2
import numpy as np
from PIL import Image
import pandas as pd
class ReceiptScanner:
"""Scan and extract data from receipts."""
def __init__(self):
"""Initialize scanner."""
self.raw_text = ""
self.data = {}
def scan(self, filepath: str, preprocess: bool = True) -> 'ReceiptScanner':
"""Scan receipt image."""
img = cv2.imread(filepath)
if preprocess:
# Convert to grayscale
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Apply thresholding
_, thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
# Denoise
denoised = cv2.fastNlMeansDenoising(thresh)
# OCR
self.raw_text = pytesseract.image_to_string(denoised)
else:
self.raw_text = pytesseract.image_to_string(filepath)
# Extract data
self.extract_data()
return self
def extract_data(self):
"""Extract structured data from text."""
lines = self.raw_text.split('\n')
# Extract vendor (usually first line)
self.data['vendor'] = lines[0].strip() if lines else "Unknown"
# Extract date
date_pattern = r'\d{1,2}[/-]\d{1,2}[/-]\d{2,4}'
for line in lines:
match = re.search(date_pattern, line)
if match:
self.data['date'] = match.group()
break
# Extract total
total_patterns = [
r'total[:\s]*\$?\s*(\d+\.\d{2})',
r'amount[:\s]*\$?\s*(\d+\.\d{2})',
r'\$\s*(\d+\.\d{2})'
]
for pattern in total_patterns:
match = re.search(pattern, self.raw_text, re.IGNORECASE)
if match:
self.data['total'] = float(match.group(1))
break
# Extract line items
items = []
item_pattern = r'(.*?)\s+\$?\s*(\d+\.\d{2})'
for line in lines:
match = re.search(item_pattern, line)
if match and len(match.group(1)) > 2:
item_name = match.group(1).strip()
amount = float(match.group(2))
if amount < 1000: # Filter out likely totals
items.append({'name': item_name, 'amount': amount})
self.data['items'] = items
def get_data(self) -> Dict:
"""Get extracted data."""
return self.data
def to_json(self, output: str) -> str:
"""Export to JSON."""
with open(output, 'w') as f:
json.dump(self.data, f, indent=2)
return output
def main():
parser = argparse.ArgumentParser(description="Receipt Scanner")
parser.add_argument("--input", "-i", required=True, help="Receipt image")
parser.add_argument("--output", "-o", required=True, help="Output JSON file")
parser.add_argument("--no-preprocess", action="store_true",
help="Skip image preprocessing")
args = parser.parse_args()
scanner = ReceiptScanner()
scanner.scan(args.input, preprocess=not args.no_preprocess)
data = scanner.get_data()
print(f"Vendor: {data.get('vendor', 'Unknown')}")
print(f"Date: {data.get('date', 'Not found')}")
print(f"Total: ${data.get('total', 0):.2f}")
print(f"Items: {len(data.get('items', []))}")
scanner.to_json(args.output)
print(f"\nData saved: {args.output}")
if __name__ == "__main__":
main()
.cursor/skills/ocr-document-processor/scripts/requirements.txt 0 → 100644
View file @ 2eef70bd
Pillow>=10.0.0
PyMuPDF>=1.23.0
numpy>=1.24.0
opencv-python>=4.8.0
pandas>=2.0.0
pillow>=10.0.0
pytesseract>=0.3.10
.cursor/skills/pdf/LICENSE.txt 0 → 100644
View file @ 2eef70bd
© 2025 Anthropic, PBC. All rights reserved.
LICENSE: Use of these materials (including all code, prompts, assets, files,
and other components of this Skill) is governed by your agreement with
Anthropic regarding use of Anthropic's services. If no separate agreement
exists, use is governed by Anthropic's Consumer Terms of Service or
Commercial Terms of Service, as applicable:
https://www.anthropic.com/legal/consumer-terms
https://www.anthropic.com/legal/commercial-terms
Your applicable agreement is referred to as the "Agreement." "Services" are
as defined in the Agreement.
ADDITIONAL RESTRICTIONS: Notwithstanding anything in the Agreement to the
contrary, users may not:
- Extract these materials from the Services or retain copies of these
materials outside the Services
- Reproduce or copy these materials, except for temporary copies created
automatically during authorized use of the Services
- Create derivative works based on these materials
- Distribute, sublicense, or transfer these materials to any third party
- Make, offer to sell, sell, or import any inventions embodied in these
materials
- Reverse engineer, decompile, or disassemble these materials
The receipt, viewing, or possession of these materials does not convey or
imply any license or right beyond those expressly granted above.
Anthropic retains all right, title, and interest in these materials,
including all copyrights, patents, and other intellectual property rights.
.cursor/skills/pdf/SKILL.md 0 → 100644
View file @ 2eef70bd
---
name: pdf
description: Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.
license: Proprietary. LICENSE.txt has complete terms
---
# PDF Processing Guide
## Overview
This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.
## Quick Start
```python
from pypdf import PdfReader, PdfWriter
# Read a PDF
reader = PdfReader("document.pdf")
print(f"Pages: {len(reader.pages)}")
# Extract text
text = ""
for page in reader.pages:
text += page.extract_text()
```
## Python Libraries
### pypdf - Basic Operations
#### Merge PDFs
```python
from pypdf import PdfWriter, PdfReader
writer = PdfWriter()
for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
reader = PdfReader(pdf_file)
for page in reader.pages:
writer.add_page(page)
with open("merged.pdf", "wb") as output:
writer.write(output)
```
#### Split PDF
```python
reader = PdfReader("input.pdf")
for i, page in enumerate(reader.pages):
writer = PdfWriter()
writer.add_page(page)
with open(f"page_{i+1}.pdf", "wb") as output:
writer.write(output)
```
#### Extract Metadata
```python
reader = PdfReader("document.pdf")
meta = reader.metadata
print(f"Title: {meta.title}")
print(f"Author: {meta.author}")
print(f"Subject: {meta.subject}")
print(f"Creator: {meta.creator}")
```
#### Rotate Pages
```python
reader = PdfReader("input.pdf")
writer = PdfWriter()
page = reader.pages[0]
page.rotate(90) # Rotate 90 degrees clockwise
writer.add_page(page)
with open("rotated.pdf", "wb") as output:
writer.write(output)
```
### pdfplumber - Text and Table Extraction
#### Extract Text with Layout
```python
import pdfplumber
with pdfplumber.open("document.pdf") as pdf:
for page in pdf.pages:
text = page.extract_text()
print(text)
```
#### Extract Tables
```python
with pdfplumber.open("document.pdf") as pdf:
for i, page in enumerate(pdf.pages):
tables = page.extract_tables()
for j, table in enumerate(tables):
print(f"Table {j+1} on page {i+1}:")
for row in table:
print(row)
```
#### Advanced Table Extraction
```python
import pandas as pd
with pdfplumber.open("document.pdf") as pdf:
all_tables = []
for page in pdf.pages:
tables = page.extract_tables()
for table in tables:
if table: # Check if table is not empty
df = pd.DataFrame(table[1:], columns=table[0])
all_tables.append(df)
# Combine all tables
if all_tables:
combined_df = pd.concat(all_tables, ignore_index=True)
combined_df.to_excel("extracted_tables.xlsx", index=False)
```
### reportlab - Create PDFs
#### Basic PDF Creation
```python
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas
c = canvas.Canvas("hello.pdf", pagesize=letter)
width, height = letter
# Add text
c.drawString(100, height - 100, "Hello World!")
c.drawString(100, height - 120, "This is a PDF created with reportlab")
# Add a line
c.line(100, height - 140, 400, height - 140)
# Save
c.save()
```
#### Create PDF with Multiple Pages
```python
from reportlab.lib.pagesizes import letter
from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer, PageBreak
from reportlab.lib.styles import getSampleStyleSheet
doc = SimpleDocTemplate("report.pdf", pagesize=letter)
styles = getSampleStyleSheet()
story = []
# Add content
title = Paragraph("Report Title", styles['Title'])
story.append(title)
story.append(Spacer(1, 12))
body = Paragraph("This is the body of the report. " * 20, styles['Normal'])
story.append(body)
story.append(PageBreak())
# Page 2
story.append(Paragraph("Page 2", styles['Heading1']))
story.append(Paragraph("Content for page 2", styles['Normal']))
# Build PDF
doc.build(story)
```
#### Subscripts and Superscripts
**IMPORTANT**: Never use Unicode subscript/superscript characters (₀₁₂₃₄₅₆₇₈₉, ⁰¹²³⁴⁵⁶⁷⁸⁹) in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes.
Instead, use ReportLab's XML markup tags in Paragraph objects:
```python
from reportlab.platypus import Paragraph
from reportlab.lib.styles import getSampleStyleSheet
styles = getSampleStyleSheet()
# Subscripts: use <sub> tag
chemical = Paragraph("H<sub>2</sub>O", styles['Normal'])
# Superscripts: use <super> tag
squared = Paragraph("x<super>2</super> + y<super>2</super>", styles['Normal'])
```
For canvas-drawn text (not Paragraph objects), manually adjust font the size and position rather than using Unicode subscripts/superscripts.
## Command-Line Tools
### pdftotext (poppler-utils)
```bash
# Extract text
pdftotext input.pdf output.txt
# Extract text preserving layout
pdftotext -layout input.pdf output.txt
# Extract specific pages
pdftotext -f 1 -l 5 input.pdf output.txt # Pages 1-5
```
### qpdf
```bash
# Merge PDFs
qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf
# Split pages
qpdf input.pdf --pages . 1-5 -- pages1-5.pdf
qpdf input.pdf --pages . 6-10 -- pages6-10.pdf
# Rotate pages
qpdf input.pdf output.pdf --rotate=+90:1 # Rotate page 1 by 90 degrees
# Remove password
qpdf --password=mypassword --decrypt encrypted.pdf decrypted.pdf
```
### pdftk (if available)
```bash
# Merge
pdftk file1.pdf file2.pdf cat output merged.pdf
# Split
pdftk input.pdf burst
# Rotate
pdftk input.pdf rotate 1east output rotated.pdf
```
## Common Tasks
### Extract Text from Scanned PDFs
```python
# Requires: pip install pytesseract pdf2image
import pytesseract
from pdf2image import convert_from_path
# Convert PDF to images
images = convert_from_path('scanned.pdf')
# OCR each page
text = ""
for i, image in enumerate(images):
text += f"Page {i+1}:\n"
text += pytesseract.image_to_string(image)
text += "\n\n"
print(text)
```
### Add Watermark
```python
from pypdf import PdfReader, PdfWriter
# Create watermark (or load existing)
watermark = PdfReader("watermark.pdf").pages[0]
# Apply to all pages
reader = PdfReader("document.pdf")
writer = PdfWriter()
for page in reader.pages:
page.merge_page(watermark)
writer.add_page(page)
with open("watermarked.pdf", "wb") as output:
writer.write(output)
```
### Extract Images
```bash
# Using pdfimages (poppler-utils)
pdfimages -j input.pdf output_prefix
# This extracts all images as output_prefix-000.jpg, output_prefix-001.jpg, etc.
```
### Password Protection
```python
from pypdf import PdfReader, PdfWriter
reader = PdfReader("input.pdf")
writer = PdfWriter()
for page in reader.pages:
writer.add_page(page)
# Add password
writer.encrypt("userpassword", "ownerpassword")
with open("encrypted.pdf", "wb") as output:
writer.write(output)
```
## Quick Reference
| Task | Best Tool | Command/Code |
|------|-----------|--------------|
| Merge PDFs | pypdf | `writer.add_page(page)` |
| Split PDFs | pypdf | One page per file |
| Extract text | pdfplumber | `page.extract_text()` |
| Extract tables | pdfplumber | `page.extract_tables()` |
| Create PDFs | reportlab | Canvas or Platypus |
| Command line merge | qpdf | `qpdf --empty --pages ...` |
| OCR scanned PDFs | pytesseract | Convert to image first |
| Fill PDF forms | pdf-lib or pypdf (see FORMS.md) | See FORMS.md |
## Next Steps
- For advanced pypdfium2 usage, see REFERENCE.md
- For JavaScript libraries (pdf-lib), see REFERENCE.md
- If you need to fill out a PDF form, follow the instructions in FORMS.md
- For troubleshooting guides, see REFERENCE.md
.cursor/skills/pdf/forms.md 0 → 100644
View file @ 2eef70bd
**CRITICAL: You MUST complete these steps in order. Do not skip ahead to writing code.**
If you need to fill out a PDF form, first check to see if the PDF has fillable form fields. Run this script from this file's directory:
`python scripts/check_fillable_fields <file.pdf>`, and depending on the result go to either the "Fillable fields" or "Non-fillable fields" and follow those instructions.
# Fillable fields
If the PDF has fillable form fields:
- Run this script from this file's directory: `python scripts/extract_form_field_info.py <input.pdf> <field_info.json>`. It will create a JSON file with a list of fields in this format:
```
[
{
"field_id": (unique ID for the field),
"page": (page number, 1-based),
"rect": ([left, bottom, right, top] bounding box in PDF coordinates, y=0 is the bottom of the page),
"type": ("text", "checkbox", "radio_group", or "choice"),
},
// Checkboxes have "checked_value" and "unchecked_value" properties:
{
"field_id": (unique ID for the field),
"page": (page number, 1-based),
"type": "checkbox",
"checked_value": (Set the field to this value to check the checkbox),
"unchecked_value": (Set the field to this value to uncheck the checkbox),
},
// Radio groups have a "radio_options" list with the possible choices.
{
"field_id": (unique ID for the field),
"page": (page number, 1-based),
"type": "radio_group",
"radio_options": [
{
"value": (set the field to this value to select this radio option),
"rect": (bounding box for the radio button for this option)
},
// Other radio options
]
},
// Multiple choice fields have a "choice_options" list with the possible choices:
{
"field_id": (unique ID for the field),
"page": (page number, 1-based),
"type": "choice",
"choice_options": [
{
"value": (set the field to this value to select this option),
"text": (display text of the option)
},
// Other choice options
],
}
]
```
- Convert the PDF to PNGs (one image for each page) with this script (run from this file's directory):
`python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>`
Then analyze the images to determine the purpose of each form field (make sure to convert the bounding box PDF coordinates to image coordinates).
- Create a `field_values.json` file in this format with the values to be entered for each field:
```
[
{
"field_id": "last_name", // Must match the field_id from `extract_form_field_info.py`
"description": "The user's last name",
"page": 1, // Must match the "page" value in field_info.json
"value": "Simpson"
},
{
"field_id": "Checkbox12",
"description": "Checkbox to be checked if the user is 18 or over",
"page": 1,
"value": "/On" // If this is a checkbox, use its "checked_value" value to check it. If it's a radio button group, use one of the "value" values in "radio_options".
},
// more fields
]
```
- Run the `fill_fillable_fields.py` script from this file's directory to create a filled-in PDF:
`python scripts/fill_fillable_fields.py <input pdf> <field_values.json> <output pdf>`
This script will verify that the field IDs and values you provide are valid; if it prints error messages, correct the appropriate fields and try again.
# Non-fillable fields
If the PDF doesn't have fillable form fields, you'll add text annotations. First try to extract coordinates from the PDF structure (more accurate), then fall back to visual estimation if needed.
## Step 1: Try Structure Extraction First
Run this script to extract text labels, lines, and checkboxes with their exact PDF coordinates:
`python scripts/extract_form_structure.py <input.pdf> form_structure.json`
This creates a JSON file containing:
- **labels**: Every text element with exact coordinates (x0, top, x1, bottom in PDF points)
- **lines**: Horizontal lines that define row boundaries
- **checkboxes**: Small square rectangles that are checkboxes (with center coordinates)
- **row_boundaries**: Row top/bottom positions calculated from horizontal lines
**Check the results**: If `form_structure.json` has meaningful labels (text elements that correspond to form fields), use **Approach A: Structure-Based Coordinates**. If the PDF is scanned/image-based and has few or no labels, use **Approach B: Visual Estimation**.
---
## Approach A: Structure-Based Coordinates (Preferred)
Use this when `extract_form_structure.py` found text labels in the PDF.
### A.1: Analyze the Structure
Read form_structure.json and identify:
1. **Label groups**: Adjacent text elements that form a single label (e.g., "Last" + "Name")
2. **Row structure**: Labels with similar `top` values are in the same row
3. **Field columns**: Entry areas start after label ends (x0 = label.x1 + gap)
4. **Checkboxes**: Use the checkbox coordinates directly from the structure
**Coordinate system**: PDF coordinates where y=0 is at TOP of page, y increases downward.
### A.2: Check for Missing Elements
The structure extraction may not detect all form elements. Common cases:
- **Circular checkboxes**: Only square rectangles are detected as checkboxes
- **Complex graphics**: Decorative elements or non-standard form controls
- **Faded or light-colored elements**: May not be extracted
If you see form fields in the PDF images that aren't in form_structure.json, you'll need to use **visual analysis** for those specific fields (see "Hybrid Approach" below).
### A.3: Create fields.json with PDF Coordinates
For each field, calculate entry coordinates from the extracted structure:
**Text fields:**
- entry x0 = label x1 + 5 (small gap after label)
- entry x1 = next label's x0, or row boundary
- entry top = same as label top
- entry bottom = row boundary line below, or label bottom + row_height
**Checkboxes:**
- Use the checkbox rectangle coordinates directly from form_structure.json
- entry_bounding_box = [checkbox.x0, checkbox.top, checkbox.x1, checkbox.bottom]
Create fields.json using `pdf_width` and `pdf_height` (signals PDF coordinates):
```json
{
"pages": [
{"page_number": 1, "pdf_width": 612, "pdf_height": 792}
],
"form_fields": [
{
"page_number": 1,
"description": "Last name entry field",
"field_label": "Last Name",
"label_bounding_box": [43, 63, 87, 73],
"entry_bounding_box": [92, 63, 260, 79],
"entry_text": {"text": "Smith", "font_size": 10}
},
{
"page_number": 1,
"description": "US Citizen Yes checkbox",
"field_label": "Yes",
"label_bounding_box": [260, 200, 280, 210],
"entry_bounding_box": [285, 197, 292, 205],
"entry_text": {"text": "X"}
}
]
}
```
**Important**: Use `pdf_width`/`pdf_height` and coordinates directly from form_structure.json.
### A.4: Validate Bounding Boxes
Before filling, check your bounding boxes for errors:
`python scripts/check_bounding_boxes.py fields.json`
This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
---
## Approach B: Visual Estimation (Fallback)
Use this when the PDF is scanned/image-based and structure extraction found no usable text labels (e.g., all text shows as "(cid:X)" patterns).
### B.1: Convert PDF to Images
`python scripts/convert_pdf_to_images.py <input.pdf> <images_dir/>`
### B.2: Initial Field Identification
Examine each page image to identify form sections and get **rough estimates** of field locations:
- Form field labels and their approximate positions
- Entry areas (lines, boxes, or blank spaces for text input)
- Checkboxes and their approximate locations
For each field, note approximate pixel coordinates (they don't need to be precise yet).
### B.3: Zoom Refinement (CRITICAL for accuracy)
For each field, crop a region around the estimated position to refine coordinates precisely.
**Create a zoomed crop using ImageMagick:**
```bash
magick <page_image> -crop <width>x<height>+<x>+<y> +repage <crop_output.png>
```
Where:
- `<x>, <y>` = top-left corner of crop region (use your rough estimate minus padding)
- `<width>, <height>` = size of crop region (field area plus ~50px padding on each side)
**Example:** To refine a "Name" field estimated around (100, 150):
```bash
magick images_dir/page_1.png -crop 300x80+50+120 +repage crops/name_field.png
```
(Note: if the `magick` command isn't available, try `convert` with the same arguments).
**Examine the cropped image** to determine precise coordinates:
1. Identify the exact pixel where the entry area begins (after the label)
2. Identify where the entry area ends (before next field or edge)
3. Identify the top and bottom of the entry line/box
**Convert crop coordinates back to full image coordinates:**
- full_x = crop_x + crop_offset_x
- full_y = crop_y + crop_offset_y
Example: If the crop started at (50, 120) and the entry box starts at (52, 18) within the crop:
- entry_x0 = 52 + 50 = 102
- entry_top = 18 + 120 = 138
**Repeat for each field**, grouping nearby fields into single crops when possible.
### B.4: Create fields.json with Refined Coordinates
Create fields.json using `image_width` and `image_height` (signals image coordinates):
```json
{
"pages": [
{"page_number": 1, "image_width": 1700, "image_height": 2200}
],
"form_fields": [
{
"page_number": 1,
"description": "Last name entry field",
"field_label": "Last Name",
"label_bounding_box": [120, 175, 242, 198],
"entry_bounding_box": [255, 175, 720, 218],
"entry_text": {"text": "Smith", "font_size": 10}
}
]
}
```
**Important**: Use `image_width`/`image_height` and the refined pixel coordinates from the zoom analysis.
### B.5: Validate Bounding Boxes
Before filling, check your bounding boxes for errors:
`python scripts/check_bounding_boxes.py fields.json`
This checks for intersecting bounding boxes and entry boxes that are too small for the font size. Fix any reported errors before filling.
---
## Hybrid Approach: Structure + Visual
Use this when structure extraction works for most fields but misses some elements (e.g., circular checkboxes, unusual form controls).
1. **Use Approach A** for fields that were detected in form_structure.json
2. **Convert PDF to images** for visual analysis of missing fields
3. **Use zoom refinement** (from Approach B) for the missing fields
4. **Combine coordinates**: For fields from structure extraction, use `pdf_width`/`pdf_height`. For visually-estimated fields, you must convert image coordinates to PDF coordinates:
- pdf_x = image_x * (pdf_width / image_width)
- pdf_y = image_y * (pdf_height / image_height)
5. **Use a single coordinate system** in fields.json - convert all to PDF coordinates with `pdf_width`/`pdf_height`
---
## Step 2: Validate Before Filling
**Always validate bounding boxes before filling:**
`python scripts/check_bounding_boxes.py fields.json`
This checks for:
- Intersecting bounding boxes (which would cause overlapping text)
- Entry boxes that are too small for the specified font size
Fix any reported errors in fields.json before proceeding.
## Step 3: Fill the Form
The fill script auto-detects the coordinate system and handles conversion:
`python scripts/fill_pdf_form_with_annotations.py <input.pdf> fields.json <output.pdf>`
## Step 4: Verify Output
Convert the filled PDF to images and verify text placement:
`python scripts/convert_pdf_to_images.py <output.pdf> <verify_images/>`
If text is mispositioned:
- **Approach A**: Check that you're using PDF coordinates from form_structure.json with `pdf_width`/`pdf_height`
- **Approach B**: Check that image dimensions match and coordinates are accurate pixels
- **Hybrid**: Ensure coordinate conversions are correct for visually-estimated fields
.cursor/skills/pdf/reference.md 0 → 100644
View file @ 2eef70bd
# PDF Processing Advanced Reference
This document contains advanced PDF processing features, detailed examples, and additional libraries not covered in the main skill instructions.
## pypdfium2 Library (Apache/BSD License)
### Overview
pypdfium2 is a Python binding for PDFium (Chromium's PDF library). It's excellent for fast PDF rendering, image generation, and serves as a PyMuPDF replacement.
### Render PDF to Images
```python
import pypdfium2 as pdfium
from PIL import Image
# Load PDF
pdf = pdfium.PdfDocument("document.pdf")
# Render page to image
page = pdf[0] # First page
bitmap = page.render(
scale=2.0, # Higher resolution
rotation=0 # No rotation
)
# Convert to PIL Image
img = bitmap.to_pil()
img.save("page_1.png", "PNG")
# Process multiple pages
for i, page in enumerate(pdf):
bitmap = page.render(scale=1.5)
img = bitmap.to_pil()
img.save(f"page_{i+1}.jpg", "JPEG", quality=90)
```
### Extract Text with pypdfium2
```python
import pypdfium2 as pdfium
pdf = pdfium.PdfDocument("document.pdf")
for i, page in enumerate(pdf):
text = page.get_text()
print(f"Page {i+1} text length: {len(text)} chars")
```
## JavaScript Libraries
### pdf-lib (MIT License)
pdf-lib is a powerful JavaScript library for creating and modifying PDF documents in any JavaScript environment.
#### Load and Manipulate Existing PDF
```javascript
import { PDFDocument } from 'pdf-lib';
import fs from 'fs';
async function manipulatePDF() {
// Load existing PDF
const existingPdfBytes = fs.readFileSync('input.pdf');
const pdfDoc = await PDFDocument.load(existingPdfBytes);
// Get page count
const pageCount = pdfDoc.getPageCount();
console.log(`Document has ${pageCount} pages`);
// Add new page
const newPage = pdfDoc.addPage([600, 400]);
newPage.drawText('Added by pdf-lib', {
x: 100,
y: 300,
size: 16
});
// Save modified PDF
const pdfBytes = await pdfDoc.save();
fs.writeFileSync('modified.pdf', pdfBytes);
}
```
#### Create Complex PDFs from Scratch
```javascript
import { PDFDocument, rgb, StandardFonts } from 'pdf-lib';
import fs from 'fs';
async function createPDF() {
const pdfDoc = await PDFDocument.create();
// Add fonts
const helveticaFont = await pdfDoc.embedFont(StandardFonts.Helvetica);
const helveticaBold = await pdfDoc.embedFont(StandardFonts.HelveticaBold);
// Add page
const page = pdfDoc.addPage([595, 842]); // A4 size
const { width, height } = page.getSize();
// Add text with styling
page.drawText('Invoice #12345', {
x: 50,
y: height - 50,
size: 18,
font: helveticaBold,
color: rgb(0.2, 0.2, 0.8)
});
// Add rectangle (header background)
page.drawRectangle({
x: 40,
y: height - 100,
width: width - 80,
height: 30,
color: rgb(0.9, 0.9, 0.9)
});
// Add table-like content
const items = [
['Item', 'Qty', 'Price', 'Total'],
['Widget', '2', '$50', '$100'],
['Gadget', '1', '$75', '$75']
];
let yPos = height - 150;
items.forEach(row => {
let xPos = 50;
row.forEach(cell => {
page.drawText(cell, {
x: xPos,
y: yPos,
size: 12,
font: helveticaFont
});
xPos += 120;
});
yPos -= 25;
});
const pdfBytes = await pdfDoc.save();
fs.writeFileSync('created.pdf', pdfBytes);
}
```
#### Advanced Merge and Split Operations
```javascript
import { PDFDocument } from 'pdf-lib';
import fs from 'fs';
async function mergePDFs() {
// Create new document
const mergedPdf = await PDFDocument.create();
// Load source PDFs
const pdf1Bytes = fs.readFileSync('doc1.pdf');
const pdf2Bytes = fs.readFileSync('doc2.pdf');
const pdf1 = await PDFDocument.load(pdf1Bytes);
const pdf2 = await PDFDocument.load(pdf2Bytes);
// Copy pages from first PDF
const pdf1Pages = await mergedPdf.copyPages(pdf1, pdf1.getPageIndices());
pdf1Pages.forEach(page => mergedPdf.addPage(page));
// Copy specific pages from second PDF (pages 0, 2, 4)
const pdf2Pages = await mergedPdf.copyPages(pdf2, [0, 2, 4]);
pdf2Pages.forEach(page => mergedPdf.addPage(page));
const mergedPdfBytes = await mergedPdf.save();
fs.writeFileSync('merged.pdf', mergedPdfBytes);
}
```
### pdfjs-dist (Apache License)
PDF.js is Mozilla's JavaScript library for rendering PDFs in the browser.
#### Basic PDF Loading and Rendering
```javascript
import * as pdfjsLib from 'pdfjs-dist';
// Configure worker (important for performance)
pdfjsLib.GlobalWorkerOptions.workerSrc = './pdf.worker.js';
async function renderPDF() {
// Load PDF
const loadingTask = pdfjsLib.getDocument('document.pdf');
const pdf = await loadingTask.promise;
console.log(`Loaded PDF with ${pdf.numPages} pages`);
// Get first page
const page = await pdf.getPage(1);
const viewport = page.getViewport({ scale: 1.5 });
// Render to canvas
const canvas = document.createElement('canvas');
const context = canvas.getContext('2d');
canvas.height = viewport.height;
canvas.width = viewport.width;
const renderContext = {
canvasContext: context,
viewport: viewport
};
await page.render(renderContext).promise;
document.body.appendChild(canvas);
}
```
#### Extract Text with Coordinates
```javascript
import * as pdfjsLib from 'pdfjs-dist';
async function extractText() {
const loadingTask = pdfjsLib.getDocument('document.pdf');
const pdf = await loadingTask.promise;
let fullText = '';
// Extract text from all pages
for (let i = 1; i <= pdf.numPages; i++) {
const page = await pdf.getPage(i);
const textContent = await page.getTextContent();
const pageText = textContent.items
.map(item => item.str)
.join(' ');
fullText += `\n--- Page ${i} ---\n${pageText}`;
// Get text with coordinates for advanced processing
const textWithCoords = textContent.items.map(item => ({
text: item.str,
x: item.transform[4],
y: item.transform[5],
width: item.width,
height: item.height
}));
}
console.log(fullText);
return fullText;
}
```
#### Extract Annotations and Forms
```javascript
import * as pdfjsLib from 'pdfjs-dist';
async function extractAnnotations() {
const loadingTask = pdfjsLib.getDocument('annotated.pdf');
const pdf = await loadingTask.promise;
for (let i = 1; i <= pdf.numPages; i++) {
const page = await pdf.getPage(i);
const annotations = await page.getAnnotations();
annotations.forEach(annotation => {
console.log(`Annotation type: ${annotation.subtype}`);
console.log(`Content: ${annotation.contents}`);
console.log(`Coordinates: ${JSON.stringify(annotation.rect)}`);
});
}
}
```
## Advanced Command-Line Operations
### poppler-utils Advanced Features
#### Extract Text with Bounding Box Coordinates
```bash
# Extract text with bounding box coordinates (essential for structured data)
pdftotext -bbox-layout document.pdf output.xml
# The XML output contains precise coordinates for each text element
```
#### Advanced Image Conversion
```bash
# Convert to PNG images with specific resolution
pdftoppm -png -r 300 document.pdf output_prefix
# Convert specific page range with high resolution
pdftoppm -png -r 600 -f 1 -l 3 document.pdf high_res_pages
# Convert to JPEG with quality setting
pdftoppm -jpeg -jpegopt quality=85 -r 200 document.pdf jpeg_output
```
#### Extract Embedded Images
```bash
# Extract all embedded images with metadata
pdfimages -j -p document.pdf page_images
# List image info without extracting
pdfimages -list document.pdf
# Extract images in their original format
pdfimages -all document.pdf images/img
```
### qpdf Advanced Features
#### Complex Page Manipulation
```bash
# Split PDF into groups of pages
qpdf --split-pages=3 input.pdf output_group_%02d.pdf
# Extract specific pages with complex ranges
qpdf input.pdf --pages input.pdf 1,3-5,8,10-end -- extracted.pdf
# Merge specific pages from multiple PDFs
qpdf --empty --pages doc1.pdf 1-3 doc2.pdf 5-7 doc3.pdf 2,4 -- combined.pdf
```
#### PDF Optimization and Repair
```bash
# Optimize PDF for web (linearize for streaming)
qpdf --linearize input.pdf optimized.pdf
# Remove unused objects and compress
qpdf --optimize-level=all input.pdf compressed.pdf
# Attempt to repair corrupted PDF structure
qpdf --check input.pdf
qpdf --fix-qdf damaged.pdf repaired.pdf
# Show detailed PDF structure for debugging
qpdf --show-all-pages input.pdf > structure.txt
```
#### Advanced Encryption
```bash
# Add password protection with specific permissions
qpdf --encrypt user_pass owner_pass 256 --print=none --modify=none -- input.pdf encrypted.pdf
# Check encryption status
qpdf --show-encryption encrypted.pdf
# Remove password protection (requires password)
qpdf --password=secret123 --decrypt encrypted.pdf decrypted.pdf
```
## Advanced Python Techniques
### pdfplumber Advanced Features
#### Extract Text with Precise Coordinates
```python
import pdfplumber
with pdfplumber.open("document.pdf") as pdf:
page = pdf.pages[0]
# Extract all text with coordinates
chars = page.chars
for char in chars[:10]: # First 10 characters
print(f"Char: '{char['text']}' at x:{char['x0']:.1f} y:{char['y0']:.1f}")
# Extract text by bounding box (left, top, right, bottom)
bbox_text = page.within_bbox((100, 100, 400, 200)).extract_text()
```
#### Advanced Table Extraction with Custom Settings
```python
import pdfplumber
import pandas as pd
with pdfplumber.open("complex_table.pdf") as pdf:
page = pdf.pages[0]
# Extract tables with custom settings for complex layouts
table_settings = {
"vertical_strategy": "lines",
"horizontal_strategy": "lines",
"snap_tolerance": 3,
"intersection_tolerance": 15
}
tables = page.extract_tables(table_settings)
# Visual debugging for table extraction
img = page.to_image(resolution=150)
img.save("debug_layout.png")
```
### reportlab Advanced Features
#### Create Professional Reports with Tables
```python
from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.lib import colors
# Sample data
data = [
['Product', 'Q1', 'Q2', 'Q3', 'Q4'],
['Widgets', '120', '135', '142', '158'],
['Gadgets', '85', '92', '98', '105']
]
# Create PDF with table
doc = SimpleDocTemplate("report.pdf")
elements = []
# Add title
styles = getSampleStyleSheet()
title = Paragraph("Quarterly Sales Report", styles['Title'])
elements.append(title)
# Add table with advanced styling
table = Table(data)
table.setStyle(TableStyle([
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
('FONTSIZE', (0, 0), (-1, 0), 14),
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
('BACKGROUND', (0, 1), (-1, -1), colors.beige),
('GRID', (0, 0), (-1, -1), 1, colors.black)
]))
elements.append(table)
doc.build(elements)
```
## Complex Workflows
### Extract Figures/Images from PDF
#### Method 1: Using pdfimages (fastest)
```bash
# Extract all images with original quality
pdfimages -all document.pdf images/img
```
#### Method 2: Using pypdfium2 + Image Processing
```python
import pypdfium2 as pdfium
from PIL import Image
import numpy as np
def extract_figures(pdf_path, output_dir):
pdf = pdfium.PdfDocument(pdf_path)
for page_num, page in enumerate(pdf):
# Render high-resolution page
bitmap = page.render(scale=3.0)
img = bitmap.to_pil()
# Convert to numpy for processing
img_array = np.array(img)
# Simple figure detection (non-white regions)
mask = np.any(img_array != [255, 255, 255], axis=2)
# Find contours and extract bounding boxes
# (This is simplified - real implementation would need more sophisticated detection)
# Save detected figures
# ... implementation depends on specific needs
```
### Batch PDF Processing with Error Handling
```python
import os
import glob
from pypdf import PdfReader, PdfWriter
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def batch_process_pdfs(input_dir, operation='merge'):
pdf_files = glob.glob(os.path.join(input_dir, "*.pdf"))
if operation == 'merge':
writer = PdfWriter()
for pdf_file in pdf_files:
try:
reader = PdfReader(pdf_file)
for page in reader.pages:
writer.add_page(page)
logger.info(f"Processed: {pdf_file}")
except Exception as e:
logger.error(f"Failed to process {pdf_file}: {e}")
continue
with open("batch_merged.pdf", "wb") as output:
writer.write(output)
elif operation == 'extract_text':
for pdf_file in pdf_files:
try:
reader = PdfReader(pdf_file)
text = ""
for page in reader.pages:
text += page.extract_text()
output_file = pdf_file.replace('.pdf', '.txt')
with open(output_file, 'w', encoding='utf-8') as f:
f.write(text)
logger.info(f"Extracted text from: {pdf_file}")
except Exception as e:
logger.error(f"Failed to extract text from {pdf_file}: {e}")
continue
```
### Advanced PDF Cropping
```python
from pypdf import PdfWriter, PdfReader
reader = PdfReader("input.pdf")
writer = PdfWriter()
# Crop page (left, bottom, right, top in points)
page = reader.pages[0]
page.mediabox.left = 50
page.mediabox.bottom = 50
page.mediabox.right = 550
page.mediabox.top = 750
writer.add_page(page)
with open("cropped.pdf", "wb") as output:
writer.write(output)
```
## Performance Optimization Tips
### 1. For Large PDFs
- Use streaming approaches instead of loading entire PDF in memory
- Use `qpdf --split-pages` for splitting large files
- Process pages individually with pypdfium2
### 2. For Text Extraction
- `pdftotext -bbox-layout` is fastest for plain text extraction
- Use pdfplumber for structured data and tables
- Avoid `pypdf.extract_text()` for very large documents
### 3. For Image Extraction
- `pdfimages` is much faster than rendering pages
- Use low resolution for previews, high resolution for final output
### 4. For Form Filling
- pdf-lib maintains form structure better than most alternatives
- Pre-validate form fields before processing
### 5. Memory Management
```python
# Process PDFs in chunks
def process_large_pdf(pdf_path, chunk_size=10):
reader = PdfReader(pdf_path)
total_pages = len(reader.pages)
for start_idx in range(0, total_pages, chunk_size):
end_idx = min(start_idx + chunk_size, total_pages)
writer = PdfWriter()
for i in range(start_idx, end_idx):
writer.add_page(reader.pages[i])
# Process chunk
with open(f"chunk_{start_idx//chunk_size}.pdf", "wb") as output:
writer.write(output)
```
## Troubleshooting Common Issues
### Encrypted PDFs
```python
# Handle password-protected PDFs
from pypdf import PdfReader
try:
reader = PdfReader("encrypted.pdf")
if reader.is_encrypted:
reader.decrypt("password")
except Exception as e:
print(f"Failed to decrypt: {e}")
```
### Corrupted PDFs
```bash
# Use qpdf to repair
qpdf --check corrupted.pdf
qpdf --replace-input corrupted.pdf
```
### Text Extraction Issues
```python
# Fallback to OCR for scanned PDFs
import pytesseract
from pdf2image import convert_from_path
def extract_text_with_ocr(pdf_path):
images = convert_from_path(pdf_path)
text = ""
for i, image in enumerate(images):
text += pytesseract.image_to_string(image)
return text
```
## License Information
- **pypdf**: BSD License
- **pdfplumber**: MIT License
- **pypdfium2**: Apache/BSD License
- **reportlab**: BSD License
- **poppler-utils**: GPL-2 License
- **qpdf**: Apache License
- **pdf-lib**: MIT License
- **pdfjs-dist**: Apache License
\ No newline at end of file
.cursor/skills/pdf/scripts/check_bounding_boxes.py 0 → 100644
View file @ 2eef70bd
from dataclasses import dataclass
import json
import sys
@dataclass
class RectAndField:
rect: list[float]
rect_type: str
field: dict
def get_bounding_box_messages(fields_json_stream) -> list[str]:
messages = []
fields = json.load(fields_json_stream)
messages.append(f"Read {len(fields['form_fields'])} fields")
def rects_intersect(r1, r2):
disjoint_horizontal = r1[0] >= r2[2] or r1[2] <= r2[0]
disjoint_vertical = r1[1] >= r2[3] or r1[3] <= r2[1]
return not (disjoint_horizontal or disjoint_vertical)
rects_and_fields = []
for f in fields["form_fields"]:
rects_and_fields.append(RectAndField(f["label_bounding_box"], "label", f))
rects_and_fields.append(RectAndField(f["entry_bounding_box"], "entry", f))
has_error = False
for i, ri in enumerate(rects_and_fields):
for j in range(i + 1, len(rects_and_fields)):
rj = rects_and_fields[j]
if ri.field["page_number"] == rj.field["page_number"] and rects_intersect(ri.rect, rj.rect):
has_error = True
if ri.field is rj.field:
messages.append(f"FAILURE: intersection between label and entry bounding boxes for `{ri.field['description']}` ({ri.rect}, {rj.rect})")
else:
messages.append(f"FAILURE: intersection between {ri.rect_type} bounding box for `{ri.field['description']}` ({ri.rect}) and {rj.rect_type} bounding box for `{rj.field['description']}` ({rj.rect})")
if len(messages) >= 20:
messages.append("Aborting further checks; fix bounding boxes and try again")
return messages
if ri.rect_type == "entry":
if "entry_text" in ri.field:
font_size = ri.field["entry_text"].get("font_size", 14)
entry_height = ri.rect[3] - ri.rect[1]
if entry_height < font_size:
has_error = True
messages.append(f"FAILURE: entry bounding box height ({entry_height}) for `{ri.field['description']}` is too short for the text content (font size: {font_size}). Increase the box height or decrease the font size.")
if len(messages) >= 20:
messages.append("Aborting further checks; fix bounding boxes and try again")
return messages
if not has_error:
messages.append("SUCCESS: All bounding boxes are valid")
return messages
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: check_bounding_boxes.py [fields.json]")
sys.exit(1)
with open(sys.argv[1]) as f:
messages = get_bounding_box_messages(f)
for msg in messages:
print(msg)
.cursor/skills/pdf/scripts/check_fillable_fields.py 0 → 100644
View file @ 2eef70bd
import sys
from pypdf import PdfReader
reader = PdfReader(sys.argv[1])
if (reader.get_fields()):
print("This PDF has fillable form fields")
else:
print("This PDF does not have fillable form fields; you will need to visually determine where to enter data")
.cursor/skills/pdf/scripts/convert_pdf_to_images.py 0 → 100644
View file @ 2eef70bd
import os
import sys
from pdf2image import convert_from_path
def convert(pdf_path, output_dir, max_dim=1000):
images = convert_from_path(pdf_path, dpi=200)
for i, image in enumerate(images):
width, height = image.size
if width > max_dim or height > max_dim:
scale_factor = min(max_dim / width, max_dim / height)
new_width = int(width * scale_factor)
new_height = int(height * scale_factor)
image = image.resize((new_width, new_height))
image_path = os.path.join(output_dir, f"page_{i+1}.png")
image.save(image_path)
print(f"Saved page {i+1} as {image_path} (size: {image.size})")
print(f"Converted {len(images)} pages to PNG images")
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: convert_pdf_to_images.py [input pdf] [output directory]")
sys.exit(1)
pdf_path = sys.argv[1]
output_directory = sys.argv[2]
convert(pdf_path, output_directory)
.cursor/skills/pdf/scripts/create_validation_image.py 0 → 100644
View file @ 2eef70bd
import json
import sys
from PIL import Image, ImageDraw
def create_validation_image(page_number, fields_json_path, input_path, output_path):
with open(fields_json_path, 'r') as f:
data = json.load(f)
img = Image.open(input_path)
draw = ImageDraw.Draw(img)
num_boxes = 0
for field in data["form_fields"]:
if field["page_number"] == page_number:
entry_box = field['entry_bounding_box']
label_box = field['label_bounding_box']
draw.rectangle(entry_box, outline='red', width=2)
draw.rectangle(label_box, outline='blue', width=2)
num_boxes += 2
img.save(output_path)
print(f"Created validation image at {output_path} with {num_boxes} bounding boxes")
if __name__ == "__main__":
if len(sys.argv) != 5:
print("Usage: create_validation_image.py [page number] [fields.json file] [input image path] [output image path]")
sys.exit(1)
page_number = int(sys.argv[1])
fields_json_path = sys.argv[2]
input_image_path = sys.argv[3]
output_image_path = sys.argv[4]
create_validation_image(page_number, fields_json_path, input_image_path, output_image_path)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment