Pdf Tools

PDF Tools

Full-lifecycle PDF engineering covering extraction, generation, modification, form filling, and security. Prioritizes JavaScript-first solutions (pdf-lib, unpdf, Puppeteer) with Python/CLI utilities for advanced scenarios.

When to use: Extracting structured data from PDFs, generating pixel-perfect PDFs from HTML/React, modifying existing PDFs, filling forms (fillable or non-fillable), or securing documents with encryption.

When NOT to use: Simple text file processing, image-only manipulation without PDF context, or tasks better handled by a word processor.

Quick Reference

Common Mistakes

Delegation

• Inspect PDF structure and diagnose extraction issues: Use Explore agent to examine AcroForm fields, encoding, and metadata
• Build end-to-end document processing pipelines: Use Task agent to implement extraction, transformation, and generation workflows
• Design PDF architecture for a new system: Use Plan agent to select tools and plan extraction, generation, or modification strategies

References

• AI Extraction Patterns -- Vision-based table extraction, recursive summarization, multi-pass verification
• High-Fidelity Generation -- Puppeteer HTML-to-PDF, CSS print tips, React templates, browser pooling
• Legacy Utilities -- pdfplumber, pypdf, qpdf, poppler-utils for batch and forensic tasks
• Form Filling -- Fillable field extraction, non-fillable annotation workflow, validation scripts
• Batch Processing and Accessibility -- Queue-based batch processing, PDF/A compliance, tagged PDFs, digital signatures, comparison, redaction

The Extraction Stack

1. Layer 1 -- Raw Text Parsing (unpdf): Extract text and metadata via extractText and getDocumentProxy 2. Layer 2 -- Vision Analysis (Gemini/GPT-4o): "Look" at the page to identify tables, headers, and signatures 3. Layer 3 -- Schema Mapping (AI SDK): Force the output into a validated Zod/JSON structure

AI-Driven Semantic Extraction

Use LLMs to turn unstructured PDF text into validated schemas:

import { extractText, getDocumentProxy } from 'unpdf';
import { generateObject } from 'ai';

async function extractInvoice(buffer: ArrayBuffer) {
  const pdf = await getDocumentProxy(new Uint8Array(buffer));
  const { text } = await extractText(pdf, { mergePages: true });

  const { object } = await generateObject({
    model: myModel,
    schema: invoiceSchema,
    prompt: `Extract structured data from this PDF text: ${text}`,
  });

  return object;
}

Visual Table Extraction

Tables are the hardest part of PDF extraction. Borders are often missing or purely decorative. Use vision models to handle complex layouts:

import { generateObject } from 'ai';
import { z } from 'zod';

async function extractComplexTable(pdfBuffer: Buffer) {
  const pages = await pdfToImages(pdfBuffer);

  const { object } = await generateObject({
    model: google('gemini-2.0-pro'),
    schema: z.object({
      rows: z.array(z.record(z.string())),
    }),
    messages: [
      {
        role: 'user',
        content: [
          { type: 'text', text: 'Extract this table:' },
          { type: 'image', image: pages[1] },
        ],
      },
    ],
  });

  return object.rows;
}

Prompting Strategy

Use domain-specific prompts for better extraction:

Act as a forensic document analyst. Extract the table from page 2.
Do not just return text; return a JSON array where each object represents a row.
Identify headers even if they are merged cells.

Recursive Document Summarization

For 100+ page documents, use token-efficient forensic scanning:

1. Extract Table of Contents (TOC) 2. Identify "high-value" pages (financial statements, signatures, terms) 3. Direct the AI model to process only those specific pages in high resolution

Multi-Pass Verification

Prevent hallucinations with a verification step:

• LLM-A extracts structured data from the PDF
• LLM-B verifies the extraction against the original document
• Discrepancies are flagged for human review

Common Pitfalls

Queue-Based Batch Processing

Use BullMQ for reliable queue-based PDF processing with retry logic and progress tracking:

import { Queue, Worker } from 'bullmq';
import { extractText, getDocumentProxy } from 'unpdf';

interface PdfJob {
  filePath: string;
  outputPath: string;
}

const pdfQueue = new Queue<PdfJob>('pdf-processing', {
  connection: { host: 'localhost', port: 6379 },
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: 100,
    removeOnFail: 500,
  },
});

const worker = new Worker<PdfJob>(
  'pdf-processing',
  async (job) => {
    const buffer = await fs.readFile(job.data.filePath);
    const pdf = await getDocumentProxy(new Uint8Array(buffer));
    const { text } = await extractText(pdf, { mergePages: true });

    await job.updateProgress(50);
    await fs.writeFile(job.data.outputPath, text);
    await job.updateProgress(100);

    return { pages: pdf.numPages, chars: text.length };
  },
  {
    connection: { host: 'localhost', port: 6379 },
    concurrency: 4,
  },
);

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed: ${err.message}`);
});

Enqueuing Jobs with Progress Tracking

async function processBatch(files: string[]) {
  const jobs = await pdfQueue.addBulk(
    files.map((filePath, i) => ({
      name: `extract-${i}`,
      data: { filePath, outputPath: filePath.replace('.pdf', '.txt') },
    })),
  );

  const results = await Promise.allSettled(
    jobs.map((job) => job.waitUntilFinished(queueEvents)),
  );

  const succeeded = results.filter((r) => r.status === 'fulfilled').length;
  const failed = results.filter((r) => r.status === 'rejected').length;

  return { succeeded, failed, total: files.length };
}

Handling Corrupt Files

Corrupt PDFs crash parsers silently. Validate before processing:

async function validatePdf(buffer: Buffer): Promise<boolean> {
  const header = buffer.subarray(0, 5).toString('ascii');
  if (header !== '%PDF-') return false;

  try {
    await getDocumentProxy(new Uint8Array(buffer));
    return true;
  } catch {
    return false;
  }
}

For files that fail validation, attempt repair with qpdf before retrying:

qpdf --replace-input possibly-corrupt.pdf

PDF/A Compliance

PDF/A is an ISO standard (ISO 19005) for long-term archival. It restricts features that prevent reliable reproduction: no JavaScript, no external font references, no encryption.

Validation

# verapdf is the reference validator
verapdf --flavour 2b input.pdf

# Output includes compliance status and violations
verapdf --format json input.pdf

Generating PDF/A-Compliant Documents

With Puppeteer, embed all fonts and avoid transparency:

const pdf = await page.pdf({
  format: 'A4',
  printBackground: true,
  tagged: true,
});

// Post-process with ghostscript for PDF/A-2b conversion
// gs -dPDFA=2 -dBATCH -dNOPAUSE -sDEVICE=pdfwrite
//    -sColorConversionStrategy=UseDeviceIndependentColor
//    -sOutputFile=output-pdfa.pdf input.pdf

gs -dPDFA=2 -dBATCH -dNOPAUSE -sDEVICE=pdfwrite \
  -sColorConversionStrategy=UseDeviceIndependentColor \
  -sOutputFile=output-pdfa.pdf input.pdf

PDF/A Levels

Use PDF/A-2b for most archival use cases. The "b" suffix means "basic" conformance (visual appearance only).

Tagged PDFs for Accessibility

Tagged PDFs contain a logical structure tree that screen readers use to navigate the document. Without tags, assistive technology reads raw text in drawing order, which often scrambles multi-column layouts.

Structure Tags

Generating Tagged PDFs with Puppeteer

const pdf = await page.pdf({
  format: 'A4',
  tagged: true,
  printBackground: true,
});

The tagged: true option maps HTML semantic elements to PDF structure tags automatically. Ensure the source HTML uses proper semantic markup:

<article>
  <h1>Annual Report</h1>
  <p>Summary of findings.</p>
  <figure>
    <img src="chart.png" alt="Revenue growth: 15% YoY increase" />
    <figcaption>Figure 1: Revenue Growth</figcaption>
  </figure>
  <table>
    <thead>
      <tr>
        <th>Quarter</th>
        <th>Revenue</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>Q1</td>
        <td>$1.2M</td>
      </tr>
    </tbody>
  </table>
</article>

Language Specification

Set the document language for proper screen reader pronunciation:

<html lang="en"></html>

For mixed-language content, use lang attributes on individual elements:

<p>The French term <span lang="fr">mise en place</span> means preparation.</p>

Accessibility Validation

# PAC (PDF Accessibility Checker) for WCAG compliance
# Available at https://pac.pdf-accessibility.org

# pdfua validates PDF/UA (Universal Accessibility) compliance
verapdf --flavour ua1 input.pdf

Digital Signatures

Signing with node-signpdf

import { plainAddPlaceholder } from '@signpdf/placeholder-plain';
import { P12Signer } from '@signpdf/signer-p12';
import signpdf from '@signpdf/signpdf';

async function signDocument(
  pdfBuffer: Buffer,
  p12Buffer: Buffer,
  passphrase: string,
) {
  const pdfWithPlaceholder = plainAddPlaceholder({
    pdfBuffer,
    reason: 'Document approval',
    contactInfo: 'signer@example.com',
    name: 'Authorized Signer',
    location: 'New York, US',
  });

  const signer = new P12Signer(p12Buffer, { passphrase });
  const signedPdf = await signpdf.sign(pdfWithPlaceholder, signer);

  return signedPdf;
}

Signature Verification

import { extractSignature } from '@signpdf/utils';
import * as forge from 'node-forge';

function verifySignature(signedPdfBuffer: Buffer) {
  const { signature, signedData } = extractSignature(signedPdfBuffer);

  const p7 = forge.pkcs7.messageFromAsn1(
    forge.asn1.fromDer(forge.util.createBuffer(signature)),
  );

  const cert = p7.certificates[0];
  const subject = cert.subject.getField('CN')?.value;
  const validFrom = cert.validity.notBefore;
  const validTo = cert.validity.notAfter;

  return {
    signer: subject,
    validFrom,
    validTo,
    isExpired: new Date() > validTo,
  };
}

Certificate Chain Validation

Verify the signer's certificate chains to a trusted root:

function validateCertChain(
  cert: forge.pki.Certificate,
  caStore: forge.pki.CAStore,
) {
  try {
    forge.pki.verifyCertificateChain(caStore, [cert]);
    return { valid: true };
  } catch (err) {
    return { valid: false, reason: (err as Error).message };
  }
}

PDF Comparison

Text Diff Between Versions

import { extractText, getDocumentProxy } from 'unpdf';
import { diffLines } from 'diff';

async function comparePdfs(bufferA: Buffer, bufferB: Buffer) {
  const pdfA = await getDocumentProxy(new Uint8Array(bufferA));
  const pdfB = await getDocumentProxy(new Uint8Array(bufferB));

  const textA = (await extractText(pdfA, { mergePages: true })).text;
  const textB = (await extractText(pdfB, { mergePages: true })).text;

  const changes = diffLines(textA, textB);

  return changes
    .filter((c) => c.added || c.removed)
    .map((c) => ({
      type: c.added ? 'added' : 'removed',
      value: c.value.trim(),
    }));
}

Visual Diff with pixelmatch

Render each page as an image and compare pixel-by-pixel:

import pixelmatch from 'pixelmatch';
import { PNG } from 'pngjs';

function visualDiff(imgA: PNG, imgB: PNG) {
  const { width, height } = imgA;
  const diff = new PNG({ width, height });

  const mismatchedPixels = pixelmatch(
    imgA.data,
    imgB.data,
    diff.data,
    width,
    height,
    { threshold: 0.1 },
  );

  const totalPixels = width * height;
  const diffPercentage = (mismatchedPixels / totalPixels) * 100;

  return { mismatchedPixels, diffPercentage, diffImage: diff };
}

Redaction

Permanent vs Visual Overlay

Visual overlays are the most common redaction mistake. The text remains in the file and can be extracted with pdftotext or copy-paste.

Secure Redaction with qpdf

# Step 1: Linearize and decompress for inspection
qpdf --qdf --object-streams=disable input.pdf decompressed.pdf

# Step 2: Apply redaction with a dedicated tool
# Python's pymupdf (fitz) performs true content removal

import fitz

doc = fitz.open("input.pdf")
page = doc[0]

sensitive_areas = page.search_for("SSN: 123-45-6789")
for area in sensitive_areas:
    page.add_redact_annot(area, fill=(0, 0, 0))

page.apply_redactions()
doc.save("redacted.pdf")

The apply_redactions() call permanently removes the underlying text content, not just the visual layer.

Metadata Stripping

PDFs carry metadata that may contain sensitive information:

# Remove all metadata with qpdf
qpdf --linearize --replace-input \
  --no-original-object-ids input.pdf

# Inspect metadata with exiftool
exiftool input.pdf

# Strip metadata with exiftool
exiftool -all= -overwrite_original input.pdf

import { PDFDocument } from 'pdf-lib';

async function stripMetadata(buffer: Buffer) {
  const pdf = await PDFDocument.load(buffer);

  pdf.setTitle('');
  pdf.setAuthor('');
  pdf.setSubject('');
  pdf.setKeywords([]);
  pdf.setProducer('');
  pdf.setCreator('');

  return Buffer.from(await pdf.save());
}

Tool Selection

Determining Form Type

First check whether the PDF has fillable form fields:

python scripts/check_fillable_fields <file.pdf>

Based on the result, follow either the fillable or non-fillable workflow.

Fillable Forms Workflow

Step 1: Extract Field Information

python scripts/extract_form_field_info.py <input.pdf> <field_info.json>

This produces a JSON array describing each field:

[
  {
    "field_id": "last_name",
    "page": 1,
    "rect": [100, 200, 300, 220],
    "type": "text"
  },
  {
    "field_id": "Checkbox12",
    "page": 1,
    "type": "checkbox",
    "checked_value": "/On",
    "unchecked_value": "/Off"
  },
  {
    "field_id": "gender_group",
    "page": 1,
    "type": "radio_group",
    "radio_options": [
      { "value": "/Male", "rect": [100, 300, 115, 315] },
      { "value": "/Female", "rect": [150, 300, 165, 315] }
    ]
  }
]

Step 2: Visual Analysis

Convert the PDF to images and match fields to their visual purpose:

python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>

Analyze the images to determine what each field represents.

Step 3: Create Field Values

Create a field_values.json mapping each field to its intended value:

[
  {
    "field_id": "last_name",
    "description": "The user's last name",
    "page": 1,
    "value": "Simpson"
  },
  {
    "field_id": "Checkbox12",
    "description": "Checked if user is 18 or over",
    "page": 1,
    "value": "/On"
  }
]

Step 4: Fill the Form

python scripts/fill_fillable_fields.py <input.pdf> <field_values.json> <output.pdf>

The script validates field IDs and values. Fix any errors and retry.

Non-Fillable Forms Workflow

For PDFs without form fields, create text annotations at visual positions.

Step 1: Visual Analysis

Convert to images and identify all form areas:

python scripts/convert_pdf_to_images.py <file.pdf> <output_directory>

For each field, determine bounding boxes for both the label and the entry area. Label and entry bounding boxes must not intersect.

Common form layouts:

Step 2: Create fields.json

{
  "pages": [{ "page_number": 1, "image_width": 1700, "image_height": 2200 }],
  "form_fields": [
    {
      "page_number": 1,
      "description": "Last name entry",
      "field_label": "Last name",
      "label_bounding_box": [30, 125, 95, 142],
      "entry_bounding_box": [100, 125, 280, 142],
      "entry_text": {
        "text": "Johnson",
        "font_size": 14,
        "font_color": "000000"
      }
    },
    {
      "page_number": 1,
      "description": "Age verification checkbox",
      "field_label": "Yes",
      "label_bounding_box": [100, 525, 132, 540],
      "entry_bounding_box": [140, 525, 155, 540],
      "entry_text": { "text": "X" }
    }
  ]
}

Step 3: Generate and Validate

Create validation images with colored overlays:

python scripts/create_validation_image.py <page> <fields.json> <input_image> <output_image>

Red rectangles mark entry areas, blue rectangles mark labels. Run the automated check:

python scripts/check_bounding_boxes.py <fields.json>

Visually inspect the validation images:

• Red rectangles must only cover input areas (no text)
• Blue rectangles should contain label text
• For checkboxes: red rectangle centered on the checkbox square

Iterate until all bounding boxes are correct.

Step 4: Fill the Form

python scripts/fill_pdf_form_with_annotations.py <input.pdf> <fields.json> <output.pdf>

Common Issues

Puppeteer HTML-to-PDF

Generate PDFs from React components for visual consistency with the web app:

import puppeteer from 'puppeteer';

export async function POST(req: Request) {
  const { htmlContent } = await req.json();
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  await page.setContent(htmlContent, { waitUntil: 'networkidle0' });
  const pdfBuffer = await page.pdf({
    format: 'A4',
    printBackground: true,
    margin: { top: '20px', bottom: '20px' },
  });

  await browser.close();
  return new Response(pdfBuffer, {
    headers: { 'Content-Type': 'application/pdf' },
  });
}

React Template Rendering

Render React components to HTML, then convert to PDF:

import puppeteer from 'puppeteer';
import { renderToString } from 'react-dom/server';

export async function createPdfFromReact(Component, props) {
  const html = renderToString(<Component {...props} />);
  const tailwindCss = await fs.readFile('./public/pdf.css', 'utf-8');

  const fullHtml = `
    <html>
      <head><style>${tailwindCss}</style></head>
      <body>${html}</body>
    </html>
  `;

  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.setContent(fullHtml, { waitUntil: 'networkidle0' });

  const pdf = await page.pdf({
    format: 'A4',
    displayHeaderFooter: true,
    footerTemplate:
      '<span style="font-size: 10px; margin-left: 20px;">Page <span class="pageNumber"></span></span>',
  });

  await browser.close();
  return pdf;
}

CSS Print Tips

For professional printing, consider CMYK color profiles in CSS.

Playwright Equivalent

Playwright uses networkidle instead of Puppeteer's networkidle0. Playwright does not support networkidle2. The page.pdf() API is otherwise identical:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle' });
const pdfBuffer = await page.pdf({ format: 'A4', printBackground: true });
await browser.close();

Font Handling in Containers

System fonts may not be available in Docker or serverless environments:

• Embed Google Fonts via <link> or inline CSS
• Bundle WOFF2 files in the project and load via @font-face
• Ensure waitUntil: 'networkidle0' (Puppeteer) or 'networkidle' (Playwright) to allow font loading

Browser Pool Optimization

Launching a browser takes approximately 500ms. In high-traffic APIs:

• Keep a pool of pre-warmed Puppeteer instances
• Use a dedicated PDF-sidecar service
• Reuse browser instances across requests (manage page lifecycle instead)

Troubleshooting

Python Tools

pdfplumber (Table Specialist)

For non-AI table extraction, pdfplumber is the most precise tool for identifying cell boundaries:

import pdfplumber
import pandas as pd

with pdfplumber.open("complex_report.pdf") as pdf:
    table = pdf.pages[0].extract_table()
    df = pd.DataFrame(table[1:], columns=table[0])

Best for: Multi-column layouts, bordered tables, precise cell boundary detection.

pypdf (Fast Merging)

For merging thousands of files, pypdf is significantly lighter than a headless browser:

from pypdf import PdfWriter

writer = PdfWriter()
for pdf in ["a.pdf", "b.pdf"]:
    writer.append(pdf)
writer.write("combined.pdf")

Best for: Bulk merge/split operations, metadata extraction, simple text extraction.

CLI Forensics

qpdf (Repair and Security)

If a PDF is corrupted or has unreadable metadata, qpdf is the go-to tool:

# Decompress a PDF to inspect raw objects (human readable)
qpdf --qdf --object-streams=disable input.pdf inspect.pdf

# Fix a "Premature EOF" error
qpdf input.pdf --replace-input

# AES-256 encryption
qpdf --encrypt user-pass owner-pass 256 -- input.pdf secured.pdf

poppler-utils (Fast Extraction)

When raw text is needed quickly for search indexing:

# Fast text extraction preserving layout
pdftotext -layout input.pdf -

# Extract with UTF-8 encoding for garbled text
pdftotext -enc UTF-8 input.pdf output.txt

pdf-lib Maintenance Note

The original pdf-lib (Hopding/pdf-lib) has not received updates since 2022. For active maintenance, use one of these forks:

• `@pdfme/pdf-lib` -- Adds drawSvg, rounded rectangles, actively maintained
• `@cantoo/pdf-lib` -- Adds encrypted PDF support ({ ignoreEncryption: true })

Both forks are API-compatible with the original. The original package still works for basic use cases (merge, split, form filling on unencrypted PDFs).

Tool Selection Guide

Troubleshooting