线程池内存溢出

场景

模拟处理 200万的数据，任务处理时间大概 2s

1
2
3
4
5
6
7
8
9
10
11
12
13
ExecutorService executorService = Executors.newFixedThreadPool(50);

for (int i = 0; i < 2000000; i++) {
  executorService.execute(() -> {
    try {
      log.info("thread:{},begin:{}", Thread.currentThread().getName());
      TimeUnit.SECONDS.sleep(2);
      log.info("thread:{},begin:{}", Thread.currentThread().getName());
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
  });
}
1
-Xmx50m -Xms50m -XX:G1HeapRegionSize=2m -XX:MetaspaceSize=128m --PrintGcDetails -XX:MaxMetaspaceSize=128m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError

结果

线程处理任务太慢，任务被添加到无界队列，超出虚拟机的堆的最大值。

1
2
3
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-7"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-33"

处理

  1. 修改线程池大小，加快处理流程 – 可能 cpu 处理不过来，可能随着任务量的增长，还是可能出现内存溢出
  2. 降低任务添加频率 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
        ExecutorService executorService = Executors.newFixedThreadPool(50);

    for (int i = 0; i < 2000000; i++) {
      executorService.execute(() -> {
        try {
          log.info("thread:{},begin:{}", Thread.currentThread().getName());
          TimeUnit.SECONDS.sleep(2);
          log.info("thread:{},begin:{}", Thread.currentThread().getName());
        } catch (InterruptedException e) {
          e.printStackTrace();
        }
      });
      TimeUnit.SECONDS.sleep(1);
    }
    添加任务的时候休眠，这样做的问题还是没办法保证一定不会出现异常，并且还人为限制了 cpu 的处理效率
  3. 修改线程池的创建，自定义线程池 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
      ExecutorService executorService = new ThreadPoolExecutor(50, 100,
        60l, TimeUnit.MINUTES, new ArrayBlockingQueue(1000), (r, executor) -> {
      if (!executor.isShutdown()) {
        try {
          executor.getQueue().put(r);
        } catch (InterruptedException e) {
          Thread.currentThread().interrupt();
        }
      }
    });
    自定义的最大等待队列为 1000，并且修改为自定义拒绝策略为，如果队列满了，就阻塞添加任务。