线上问题排查2则

Zookeeper报java.io.IOException: Len error

问题

ZK出现如下日志

2021-07-15 17:26:24,297 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 1195725856
2021-07-15 17:26:24,297 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxn@1044] - Closed socket connection for client /127.0.0.1:43678 (no session established for client)
2021-07-15 17:26:24,300 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxnFactory@192] - Accepted socket connection from /0:0:0:0:0:0:0:1:51435
2021-07-15 17:26:24,301 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 1195725856
2021-07-15 17:26:24,301 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxn@1044] - Closed socket connection for client /0:0:0:0:0:0:0:1:51435 (no session established for client)
2021-07-15 17:29:27,055 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxnFactory@192] - Accepted socket connection from /XXX.XXX.XXX.149:XX085
2021-07-15 17:29:27,062 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:ZooKeeperServer@942] - Client attempting to establish new session at /XXX.XXX.XX.149:XX085
2021-07-15 17:29:27,069 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x17aa97b944c0000 with negotiated timeout 30000 for client /XXX.XXX.XXX.149:XX085
2021-07-15 17:33:28,592 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51903:NIOServerCnxn@368] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x17aa97b944c0000, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:745)

经过排查发现，项目中的一个时间片生成任务每秒跑一次任务，每一次任务都会在ZK里生成大量的节点，且不会被清除，堆积到一定长度就会导致超出长度，从而导致ZK失联无法与客户端建联，zk之间同步可能也会有问题

解决方案

https://issues.apache.org/jira/browse/ZOOKEEPER-706

https://blog.csdn.net/wenniuwuren/article/details/78377413

ClientAbortException: java.io.IOException: Broken pipe

问题

线上日志里时常会刷新出一些如下日志，查看相关URL没有发现什么可循的规律，但是也没有客户反馈使用平台有问题。经过排查复现，发现是因为前后端的连接被断开导致Broken pipe，response无法返回所以会导致exceptionHandler的切面异常处理方法无法正常执行。这类报错不会影响正常业务的请求。

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:372)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:842)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:319)
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:272)
        at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:157)
        at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.close(OnCommittedResponseWrapper.java:501)
        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241)
        at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:378)
        at org.apache.poi.xssf.streaming.SXSSFWorkbook.injectData(SXSSFWorkbook.java:401)
        at org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:936)
。。。
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:372)
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:842)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:319)
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:272)
        at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:157)
        at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.close(OnCommittedResponseWrapper.java:501)
        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241)
        at java.util.zip.ZipOutputStream.close(ZipOutputStream.java:378)
        at org.apache.poi.xssf.streaming.SXSSFWorkbook.injectData(SXSSFWorkbook.java:401)
        at org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:936)

ExceptionHandlerExceptionResolver] Failed to invoke @ExceptionHandler method: public com.xxxx.xxx.xxx.handler.ErrorResponse com.xxxx.xxx.xxx.handler.BaseExceptionHandler.handleError(java.lang.Throwable)

复现场景

请求耗时较长，未等请求响应直接关闭浏览器，可以复现出该场景。

解决方案

https://mtyurt.net/post/spring-how-to-handle-ioexception-broken-pipe.html

@ExceptionHandler(IOException.class)
@ResponseStatus(HttpStatus.SERVICE_UNAVAILABLE)   //(1)
public Object exceptionHandler(IOException e, HttpServletRequest request) {
    if (StringUtils.containsIgnoreCase(ExceptionUtils.getRootCauseMessage(e), "Broken pipe")) {   //(2)
        return null;        //(2)    socket is closed, cannot return any response    
    } else {
        return new HttpEntity<>(e.getMessage());  //(3)
    }
}