Turn mixed office files into consistent Markdown before chunking, embeddings and retrieval.
No sign-up for the first conversions · Files processed temporarily · Current plan limits apply
A retrieval system cannot recover structure that was lost or polluted during ingestion. Repeated headers, broken reading order and inconsistent extraction can produce weak chunks before embeddings are even created.
TokenPig provides a simple preprocessing layer: convert supported documents into readable Markdown, inspect the output and then pass the cleaned text into your own chunking and indexing pipeline.
Normalize PDFs, Word files, slides and spreadsheets into a shared text representation.
Review the actual Markdown before it enters a vector store or retrieval index.
Use headings and lists as signals instead of relying only on arbitrary character counts.
Remove repeated content that can create duplicate or misleading retrieval results.
Test the hardest examples in your corpus, not only clean demo files.
Normalize the source while preserving headings, lists and useful tables.
Attach source, date, access level, document type and stable identifiers in your ingestion layer.
Create chunks, run retrieval tests and inspect whether the right passages appear for real questions.
Download the sample, run it through TokenPig and inspect the Markdown yourself before trusting it with important documents.
No. RAG systems can ingest many formats, but Markdown is useful because it is readable, structured and easy to process with standard tools.
No. This page focuses on preparing source content before your own chunking, embedding and indexing steps.
Start with semantic boundaries such as headings and sections, then apply size limits and overlap only where needed. Evaluate retrieval rather than assuming one universal chunk size.
At minimum keep a stable source ID, title, document version, date, permissions and a link back to the original file.
The current public flow is designed around document conversion limits. Batch processing and API capabilities should be matched to the plan and features actually available when you deploy the workflow.
Start with a real file, inspect the output, and keep the original source for verification.
Try TokenPig free