CLARISEARCH PIPELINE · 0% CONTEXT TRUNCATION
Industrial Out-of-Core Discovery Ingestion
Court discovery files are massive, messy, and non-uniform. While Claude caps web uploads at 30MB and drops image fidelity after 100 pages, the Clarisearch pipeline handles multi-gigabyte claim files and 4,000-page medical histories without memory exhaustion.
- Dynamic Resolution Compression: Ghostscript and
pdf-lib compress scanned PDFs with strict 15MB soft and 48MB hard chunk ceilings.
- Sliding Token Windows: 8,000-token sliding windows with 500-token overlap snap to word boundaries, eliminating truncation loss.
- Context Caching: Pre-caches reference corpora in Vertex AI for sub-second evidentiary retrieval.
// Dynamic Ghostscript optimization for gigabyte-scale PDFs
const gsCommand = [
"gs", "-sDEVICE=pdfwrite", "-dCompatibilityLevel=1.4",
"-dPDFSETTINGS=/ebook", "-dNOPAUSE", "-dQUIET", "-dBATCH",
"-dColorImageResolution=150",
"-sOutputFile=" + outputPath, inputPath
].join(" ");
// Enforce strict Vertex payload boundary (48MB hard cap)
if (stat.size > HARD_LIMIT_MB * 1024 * 1024) {
return await splitAndChunkPDF(outputPath, options);
}