【今天下午正准备摸鱼网关突然一串504把我从座位上拽了回来。APM上RT像过山车线程池也快打满了。更离谱的是CPU不高大家都在“等”。这味儿不对我开干。**事故现场现象下单列表/结算两个接口同时卡成PPTTomcat 业务线程数逼近上限但CPU不高Redis/MySQL都正常外部三方偶发超时。日志随手一段2026-03-17 15:21:03 WARN RiskClient - feign call timeout code1008611 feign.RetryableException: Read timed out 2026-03-17 15:21:05 WARN o.a.c.u.ConcurrentMessageDigest - Unable to process request in timeJStack抡一刀关键信息http-nio-8080-exec-126 #312 prio5 tid0x... waiting on condition at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:...) at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:...) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:...) at com.xxx.order.OrderService.loadDetail(OrderService.java:143) ForkJoinPool.commonPool-worker-9 #421 runnable at java.net.SocketInputStream.socketRead0(Native Method) at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:...) at feign.okhttp.OkHttpClient.execute(OkHttpClient.java:...)一眼定性一堆业务线程在等ForkJoinTask.join()而commonPool里的worker都堵在外部IOFeign/HTTP上。典型“用parallelStream跑IO任务把commonPool堵死”的翻车现场。排查脑回路踩坑实录先排除常规Redis/DB/GC都健康瓶颈不是CPU也不是锁冲突。用Arthasthread -n 5看Top阻塞清一色ForkJoinPool.commonPool-worker-*OkHttpread。回溯调用链服务里为了“提速”把N个子查询并行了orders.parallelStream().map(x - 调多方接口/查DB).collect(...)。叠加点commonPool大小≈CPU核数-1IO阻塞时线程太少吞吐崩Servlet线程在join等待形成“线程饥饿死锁”倾向所有worker都在阻塞没人干剩下的任务偶发超时Feign默认重试让阻塞时间更长。问题代码反例// 业务里为了“并行优化”直接上parallelStream public ListOrderDetail loadDetail(ListLong orderIds) { return orderIds.parallelStream() // 使用了ForkJoinPool.commonPool .map(id - { // 1) 调三方风控阻塞IO RiskDTO risk riskClient.check(id); // 偶发超时 // 2) 查DB补充 OrderDO order orderMapper.selectById(id); return assemble(order, risk); }) .collect(Collectors.toList()); // 主线程join等待 }这段在本地看着飞快上线一压就露馅parallelStream是面向CPU密集场景的IO阻塞下直接把commonPool当井盖给你焊死。修复方案别再拿commonPool跑IO目标自定义受控线程池 有界队列 超时/降级 减少join等待。方案一换成CompletableFuture 自定义线程池Configuration public class IoPoolConfig { Bean(ioPool) public Executor ioPool() { return new ThreadPoolExecutor( 32, 64, 60, TimeUnit.SECONDS, new ArrayBlockingQueue(1000), new ThreadFactoryBuilder().setNameFormat(io-%d).build(), new ThreadPoolExecutor.CallerRunsPolicy()); // 背压 } }Service public class OrderService { Resource(name ioPool) private Executor ioPool; Autowired private RiskClient riskClient; Autowired private OrderMapper orderMapper; public ListOrderDetail loadDetail(ListLong orderIds) { ListCompletableFutureOrderDetail futures orderIds.stream() .map(id - CompletableFuture.supplyAsync(() - { RiskDTO risk riskClient.checkWithTimeout(id, Duration.ofMillis(800)); OrderDO order orderMapper.selectById(id); return assemble(order, risk); }, ioPool).exceptionally(ex - OrderDetail.degraded(id))) .toList(); // allOf 超时等待避免永久join CompletableFutureVoid all CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); try { all.get(1200, TimeUnit.MILLISECONDS); } catch (Exception ignore) {} return futures.stream().map(f - f.getNow(OrderDetail.degraded(-1L))).toList(); } }要点自建受控线程池核心/最大/队列/拒绝策略避免commonPool被IO拖死每个任务自己限时并有降级不把等待传染给上游allOf整体也加超时主线程不做无限join()。方案二Resilience4j做舱壁隔离限时熔断resilience4j: bulkhead: instances: risk: maxConcurrentCalls: 64 maxWaitDuration: 0 timelimiter: instances: risk: timeoutDuration: 800ms circuitbreaker: instances: risk: slidingWindowSize: 50 failureRateThreshold: 50 waitDurationInOpenState: 10sBulkhead(name risk, type Bulkhead.Type.THREADPOOL) TimeLimiter(name risk) CircuitBreaker(name risk, fallbackMethod riskFallback) public CompletableFutureRiskDTO riskAsync(Long id){ return CompletableFuture.supplyAsync(() - riskClient.check(id), ioPool); }方案三暂时不得不用parallelStream也要换池子不推荐可以用ForkJoinPool custom new ForkJoinPool(64);包装执行ForkJoinPool custom new ForkJoinPool(64); ListR res custom.submit(() - list.parallelStream().map(this::ioCall).toList() ).join();但注意仍是阻塞IO建议优先方案一/二。验证发布后压测Tomcat活动线程下降P99从2.5s回到220msJStack再看commonPool-worker-*基本消失业务线程栈不再卡ForkJoinTask.join()监控自建ioPool的队列和拒绝数有波峰但在阈值内熔断偶发打开但未扩散。踩坑总结parallelStream/commonPool适合CPU密集别拿去跑阻塞IOWeb场景要“舱壁隔离”自建受控线程池 超时 降级看到线程都在join()/get()排队且CPU不高优先怀疑线程饥饿和池子被IO堵死能异步就异步能拆就拆别让一个外部接口把全站线程都拖下水。】
【ParallelStream把我整破防：一次commonPool被IO堵死导致接口集体卡死】
【今天下午正准备摸鱼网关突然一串504把我从座位上拽了回来。APM上RT像过山车线程池也快打满了。更离谱的是CPU不高大家都在“等”。这味儿不对我开干。**事故现场现象下单列表/结算两个接口同时卡成PPTTomcat 业务线程数逼近上限但CPU不高Redis/MySQL都正常外部三方偶发超时。日志随手一段2026-03-17 15:21:03 WARN RiskClient - feign call timeout code1008611 feign.RetryableException: Read timed out 2026-03-17 15:21:05 WARN o.a.c.u.ConcurrentMessageDigest - Unable to process request in timeJStack抡一刀关键信息http-nio-8080-exec-126 #312 prio5 tid0x... waiting on condition at java.util.concurrent.ForkJoinTask.doJoin(ForkJoinTask.java:...) at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:...) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:...) at com.xxx.order.OrderService.loadDetail(OrderService.java:143) ForkJoinPool.commonPool-worker-9 #421 runnable at java.net.SocketInputStream.socketRead0(Native Method) at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:...) at feign.okhttp.OkHttpClient.execute(OkHttpClient.java:...)一眼定性一堆业务线程在等ForkJoinTask.join()而commonPool里的worker都堵在外部IOFeign/HTTP上。典型“用parallelStream跑IO任务把commonPool堵死”的翻车现场。排查脑回路踩坑实录先排除常规Redis/DB/GC都健康瓶颈不是CPU也不是锁冲突。用Arthasthread -n 5看Top阻塞清一色ForkJoinPool.commonPool-worker-*OkHttpread。回溯调用链服务里为了“提速”把N个子查询并行了orders.parallelStream().map(x - 调多方接口/查DB).collect(...)。叠加点commonPool大小≈CPU核数-1IO阻塞时线程太少吞吐崩Servlet线程在join等待形成“线程饥饿死锁”倾向所有worker都在阻塞没人干剩下的任务偶发超时Feign默认重试让阻塞时间更长。问题代码反例// 业务里为了“并行优化”直接上parallelStream public ListOrderDetail loadDetail(ListLong orderIds) { return orderIds.parallelStream() // 使用了ForkJoinPool.commonPool .map(id - { // 1) 调三方风控阻塞IO RiskDTO risk riskClient.check(id); // 偶发超时 // 2) 查DB补充 OrderDO order orderMapper.selectById(id); return assemble(order, risk); }) .collect(Collectors.toList()); // 主线程join等待 }这段在本地看着飞快上线一压就露馅parallelStream是面向CPU密集场景的IO阻塞下直接把commonPool当井盖给你焊死。修复方案别再拿commonPool跑IO目标自定义受控线程池 有界队列 超时/降级 减少join等待。方案一换成CompletableFuture 自定义线程池Configuration public class IoPoolConfig { Bean(ioPool) public Executor ioPool() { return new ThreadPoolExecutor( 32, 64, 60, TimeUnit.SECONDS, new ArrayBlockingQueue(1000), new ThreadFactoryBuilder().setNameFormat(io-%d).build(), new ThreadPoolExecutor.CallerRunsPolicy()); // 背压 } }Service public class OrderService { Resource(name ioPool) private Executor ioPool; Autowired private RiskClient riskClient; Autowired private OrderMapper orderMapper; public ListOrderDetail loadDetail(ListLong orderIds) { ListCompletableFutureOrderDetail futures orderIds.stream() .map(id - CompletableFuture.supplyAsync(() - { RiskDTO risk riskClient.checkWithTimeout(id, Duration.ofMillis(800)); OrderDO order orderMapper.selectById(id); return assemble(order, risk); }, ioPool).exceptionally(ex - OrderDetail.degraded(id))) .toList(); // allOf 超时等待避免永久join CompletableFutureVoid all CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])); try { all.get(1200, TimeUnit.MILLISECONDS); } catch (Exception ignore) {} return futures.stream().map(f - f.getNow(OrderDetail.degraded(-1L))).toList(); } }要点自建受控线程池核心/最大/队列/拒绝策略避免commonPool被IO拖死每个任务自己限时并有降级不把等待传染给上游allOf整体也加超时主线程不做无限join()。方案二Resilience4j做舱壁隔离限时熔断resilience4j: bulkhead: instances: risk: maxConcurrentCalls: 64 maxWaitDuration: 0 timelimiter: instances: risk: timeoutDuration: 800ms circuitbreaker: instances: risk: slidingWindowSize: 50 failureRateThreshold: 50 waitDurationInOpenState: 10sBulkhead(name risk, type Bulkhead.Type.THREADPOOL) TimeLimiter(name risk) CircuitBreaker(name risk, fallbackMethod riskFallback) public CompletableFutureRiskDTO riskAsync(Long id){ return CompletableFuture.supplyAsync(() - riskClient.check(id), ioPool); }方案三暂时不得不用parallelStream也要换池子不推荐可以用ForkJoinPool custom new ForkJoinPool(64);包装执行ForkJoinPool custom new ForkJoinPool(64); ListR res custom.submit(() - list.parallelStream().map(this::ioCall).toList() ).join();但注意仍是阻塞IO建议优先方案一/二。验证发布后压测Tomcat活动线程下降P99从2.5s回到220msJStack再看commonPool-worker-*基本消失业务线程栈不再卡ForkJoinTask.join()监控自建ioPool的队列和拒绝数有波峰但在阈值内熔断偶发打开但未扩散。踩坑总结parallelStream/commonPool适合CPU密集别拿去跑阻塞IOWeb场景要“舱壁隔离”自建受控线程池 超时 降级看到线程都在join()/get()排队且CPU不高优先怀疑线程饥饿和池子被IO堵死能异步就异步能拆就拆别让一个外部接口把全站线程都拖下水。】
