MarkItDown: Files to Markdown

MarkItDown handles the PDFs, Word files and Excel sheets locally. It can't OCR scanned pages, so those need a separate local step first.

1. Install (local only)

uv venv --python 3.12 .venv && source .venv/bin/activate
uv pip install "markitdown[pdf,docx,xlsx,xls]==0.1.8"
brew install ocrmypdf          # local Tesseract-based OCR, for the scanned PDFs

Don't install markitdown-ocr, and don't use Azure or LLM image-description options. Those all send content to an outside service.

2. OCR the scanned PDFs locally

MarkItDown's PDF converter only extracts existing text. A scanned PDF will come out nearly empty. ocrmypdf adds a text layer on your machine, and --skip-text leaves pages that already have text alone:

mkdir -p ocr
find contracts -type f -iname '*.pdf' | while read -r f; do
  out="ocr/${f#contracts/}"; mkdir -p "$(dirname "$out")"
  ocrmypdf --skip-text "$f" "$out" || echo "OCR FAILED: $f" >> ocr_failures.txt
done

This OCRs every PDF and skips pages that already have text. For 300 files that's fine. For other languages, install the matching Tesseract language pack and add -l eng+deu (or similar).

3. Convert everything to Markdown

Use a loop that preserves subfolders and avoids name collisions (a.pdf and a.docx both become .md). It writes <filename>.md, for example a.pdf.md:

mkdir -p markdown
find contracts -type f \( -iname '*.docx' -o -iname '*.xlsx' -o -iname '*.xls' \) > list.txt
find ocr       -type f -iname '*.pdf' >> list.txt
while read -r f; do
  rel="${f#*/}"; out="markdown/$rel.md"; mkdir -p "$(dirname "$out")"
  markitdown "$f" -o "$out" || echo "FAILED: $f" >> convert_failures.txt
done < list.txt

The MarkItDown skill also bundles scripts/batch_convert.py, which does the same with a manifest. I haven't checked that it's present on your machine, so the loop above is the safe option. Plugins are off by default, which is what you want.

What to watch out for

• Network: MarkItDown's local converters don't call out, but its extension checks aren't an offline sandbox. Audio or URL handling can reach the network. For confidential files, run the conversion with Wi-Fi off or in a network-isolated container.
• Old .doc files: these aren't supported, only .docx. Convert them first with soffice --headless --convert-to docx.
• Silent bad output: a conversion can succeed and still be poor. Find empty or tiny outputs with find markdown -size -200c, and open a few OCR'd files to check quality. Skewed or low-resolution scans are the usual cause.
• Excel: each sheet becomes a Markdown table. Large sheets turn into huge tables that chunk badly in a search index. Formulas come through as their cached values, not the formulas.
• Tables and layout: multi-column PDFs, signature pages, stamps, handwriting and images inside Word files won't convert well. Spot-check key contracts against the originals.
• Provenance: keep each file's original path in your index metadata, so a search hit points back to the source document. The Markdown is only a derived copy.
• Hidden or injected text: contracts can contain hidden text or instructions. If an LLM reads your index later, treat the Markdown as data, not instructions.
• Storage: the ocr/ and markdown/ folders are now confidential copies. Keep them in the same access-controlled location as the originals, and don't put them in a synced or cloud-backed folder.
• Failures: check ocr_failures.txt and convert_failures.txt afterwards. Password-protected or corrupted PDFs are the usual culprits.

I can run this on the folder and report the failures and suspiciously small outputs. Tell me the path if you want that.