非结构化文档摄取管道
1 分•作者: moaffaneh•6 个月前
大家好，我正在设计一个基于 AWS 的非结构化文档摄取平台（PDF/DOCX/PPTX/XLSX），用于大规模企业存储库。我将使用视觉-语言模型将页面规范化为具有布局感知能力的 Markdown 格式，然后构建搜索/RAG 索引或提取结构化数据。
对于最近构建过类似平台的人，你们在规范化 Markdown 中可靠地保留文档结构（标题、阅读顺序、嵌套表格、页面边界）时使用了什么方法，尤其是在文档混乱或扫描的情况下？
你们是仅进行页面级别的提取，还是使用了重叠窗口/多页面上下文来处理跨页的表格和章节？
在索引方面，你们是仅存储块 + 嵌入，还是也持久化了每个块的更丰富的元数据（页面范围、标题层级、has_table/contains_image 标志、提取置信度/质量注释、源指针）？如果是，什么被证明是最有价值的？这在代理检索过程中如何提供帮助？
对于布局密集的页面（多列文本、复杂表格、脚注、重复的页眉/页脚），哪些提示模式效果最好，哪些在实践中失败了？
你们如何大规模评估提取质量，除了抽查（黄金集、自动启发式方法、跨运行/模型的差异分析、表格结构指标）之外？
任何经验教训、反模式或“如果再来一次”的建议都将非常有帮助。
查看原文
Hi all, I am designing an AWS-based unstructured document ingestion platform (PDF/DOCX/PPTX/XLSX) for large-scale enterprise repositories, using vision-language models to normalize pages into layout-aware markdown and then building search/RAG indexes or extract structured data.<p>For those who have built something similar recently, what approach did you use to preserve document structure reliably in the normalized markdown (headings, reading order, nested tables, page boundaries), especially when documents are messy or scanned?
Did you do page-level extraction only, or did you use overlapping windows / multi-page context to handle tables and sections spanning pages?<p>On the indexing side, do you store only chunks + embeddings, or do you also persist richer metadata per chunk (page ranges, heading hierarchy, has_table/contains_image flags, extraction confidence/quality notes, source pointers) and if so, what proved most valuable? How does that help in the agent retrieval process?<p>What prompt patterns worked best for layout-heavy pages (multi-column text, complex tables, footnotes, repeated headers/footers), and what failed in practice?<p>How did you evaluate extraction quality at scale beyond spot checks (golden sets, automatic heuristics, diffing across runs/models, table-structure metrics)?<p>Any lessons learned, anti-patterns, or “if I did it again” recommendations would be very helpful.