50 万行数据的惰性迭代 vs. 数组链式操作 – 基准测试结果
1 分•作者: gvsh_maths•5 个月前
我构建了一个 TypeScript 迭代器库 (iterflow)，并想测量惰性管道和急切管道之间的实际堆差异。这是基准测试报告。<p>管道<p>急切 - 标准数组链式操作：<p>const data = Array.from(generateRows(500_000));<p>const results = data
.filter(r => r.active && r.value > threshold)
.map(r => ({ id: r.id, score: r.value * 1.5 }))
.slice(0, 10_000);<p>每个步骤都会生成一个新的中间数组。.filter() 分配一个，.map() 分配另一个，然后 .slice() 丢弃大部分。<p>惰性 - 通过 iterflow 的相同管道：<p>import { iter } from '@mathscapes/iterflow';<p>const results = iter(generateRows(500_000))
.filter(r => r.active && r.value > threshold)
.map(r => ({ id: r.id, score: r.value * 1.5 }))
.take(10_000)
.toArray();
generateRows 是一个生成器，一次产生一行数据。在 .toArray() 通过链拉取值之前，没有任何东西被物化。没有中间数组。<p>结果<p>数据集：500,000 行
管道：filter(active && value > 5000) → map(score) → take(10,000)<p>原生数组 (.filter → .map → .slice) 15.4 MB (最小值 15.2 MB，最大值 16.2 MB)
iterflow (.filter → .map → .take) 5.8 MB (最小值 5.8 MB，最大值 5.8 MB)<p>方法<p>- 指标：管道前后堆使用量的差值，而不是总的进程内存
- 两个管道都从同一个生成器源开始——差值仅衡量管道分配，而不是源数据——在每次运行之间强制使用 --expose-gc 和显式的 gc() 调用
- 在测量之前丢弃一次预热运行
- 报告 5 次运行的中位数
- 原生数组运行在管道运行之前将完整的 50 万数据集物化到 data 中。该分配不包含在差值中——两种方法在相同的条件下进行测量。<p>关于该库的几点说明<p>- iter() 是 ES2015 生成器和迭代器协议的包装器——没有魔法，只是一个流畅的 API，因此调用站点看起来与数组链式操作相同
- .sum() 和 .mean() 仅被类型化为 Iterflow<number>——在非数字迭代器上调用它们是一个编译错误
- 具有一些流式统计操作 (.streamingMean(), .ewma(), .windowedMin())，用于在没有单独累加器的情况下运行聚合
- 零运行时依赖<p>https://www.npmjs.com/package/@mathscapes/iterflow
查看原文
I built a TypeScript iterator library (iterflow) and wanted to measure the actual heap difference between lazy and eager pipelines. This is the benchmark writeup.<p>The pipelines<p>Eager - standard array chaining:<p>const data = Array.from(generateRows(500_000));<p>const results = data
.filter(r => r.active && r.value > threshold)
.map(r => ({ id: r.id, score: r.value * 1.5 }))
.slice(0, 10_000);<p>Each step produces a new intermediate array. .filter() allocates one, .map() allocates another, .slice() then discards most of both.<p>Lazy - same pipeline via iterflow:<p>import { iter } from '@mathscapes/iterflow';<p>const results = iter(generateRows(500_000))
.filter(r => r.active && r.value > threshold)
.map(r => ({ id: r.id, score: r.value * 1.5 }))
.take(10_000)
.toArray();
generateRows is a generator, yields one row at a time. Nothing is materialized until .toArray() pulls values through the chain. No intermediate arrays.<p>Results<p>Dataset: 500,000 rows
Pipeline: filter(active && value > 5000) → map(score) → take(10,000)<p>native array (.filter → .map → .slice) 15.4 MB (min 15.2 MB, max 16.2 MB)
iterflow (.filter → .map → .take) 5.8 MB (min 5.8 MB, max 5.8 MB)<p>Methodology<p>- Metric: heapUsed delta before and after the pipeline, not total process memory
- Both pipelines start from the same generator source — the delta measures pipeline allocations only, not source data --expose-gc with explicit gc() calls forced between every run
- One warm-up run discarded before measurement
- Median of 5 runs reported
- The native array run materializes the full 500k dataset into data before the pipeline runs. That allocation is not included in the delta - both approaches are measured on the same footing.<p>A few notes on the library<p>- iter() is a wrapper around ES2015 generators and the iterator protocol - no magic, just a fluent API so the call site looks identical to array chaining
- .sum() and .mean() are typed to Iterflow<number> only - calling them on a non-numeric iterator is a compile error
- Has some streaming statistical operations (.streamingMean(), .ewma(), .windowedMin()) for running aggregations without a separate accumulator
- Zero runtime dependencies<p>https://www.npmjs.com/package/@mathscapes/iterflow