WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@90] - Exception when following the leader java.net.SocketTimeoutException: Read timed out

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@90] - Exception when following the leader java.net.SocketTimeoutException: Read timed out

svanschalkwyk
Could this be a network issue? (I'm also seeing socket timeouts)...

Periodically getting these.
 /WARN  [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@90] - Exception when
following the leader
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
        at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
        at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
        at
org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
        at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
        at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
2018-02-19 22:20:21,181 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:Follower@169] - shutdown called
java.lang.Exception: shutdown Follower
        at
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
        at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:985)
2018-02-19 22:20:21,183 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:FollowerZooKeeperServer@140] - Shutting
down
2018-02-19 22:20:21,183 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@501] - shutting down
2018-02-19 22:20:21,184 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:FollowerRequestProcessor@107] - Shutting
down
2018-02-19 22:20:21,184 [myid:1] - INFO
[FollowerRequestProcessor:1:FollowerRequestProcessor@97] -
FollowerRequestProcessor exited loop!
2018-02-19 22:20:21,184 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:CommitProcessor@184] - Shutting down
2018-02-19 22:20:21,185 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:FinalRequestProcessor@403] - shutdown of
request processor complete
2018-02-19 22:20:21,185 [myid:1] - INFO
[CommitProcessor:1:CommitProcessor@153] - CommitProcessor exited loop!
2018-02-19 22:20:21,185 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:SyncRequestProcessor@208] - Shutting down
2018-02-19 22:20:21,185 [myid:1] - INFO
[SyncThread:1:SyncRequestProcessor@186] - SyncRequestProcessor exited!
2018-02-19 22:20:21,186 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@909] - LOOKING
2018-02-19 22:20:21,189 [myid:1] - INFO
[QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@820] - New election. My
id =  1, proposed zxid=0x38000004e9
2018-02-19 22:20:21,189 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection@602] - Notification: 1 (message
fo/

zoo.cfg has

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=100
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=10
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=200
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=50
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=3
server.1=zk01:2888:3888
server.2=zk02:2888:3888
server.3=zk03:2888:3888

I have increased initLimit and syncLimit.

Is this a network issue?




--
Sent from: http://zookeeper-user.578899.n2.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: WARN [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@90] - Exception when following the leader java.net.SocketTimeoutException: Read timed out

Andor Molnar
Hi svanschalkwyk,

I'm not really sure.
It could be a network issue or a slow leader which is not able to answer
follower request in a timely fashion.
Does it happen on all followers or only one?

Regards,
Andor



On Tue, Feb 20, 2018 at 6:31 PM, svanschalkwyk <[hidden email]>
wrote:

> Could this be a network issue? (I'm also seeing socket timeouts)...
>
> Periodically getting these.
>  /WARN  [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@90] - Exception when
> following the leader
> java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.socketRead(SocketInputStream.
> java:116)
>         at java.net.SocketInputStream.read(SocketInputStream.java:171)
>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
>         at java.io.DataInputStream.readInt(DataInputStream.java:387)
>         at
> org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
>         at
> org.apache.zookeeper.server.quorum.QuorumPacket.
> deserialize(QuorumPacket.java:85)
>         at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
>         at
> org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
>         at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
> 2018-02-19 22:20:21,181 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:Follower@169] - shutdown called
> java.lang.Exception: shutdown Follower
>         at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
>         at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:985)
> 2018-02-19 22:20:21,183 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerZooKeeperServer@140] - Shutting
> down
> 2018-02-19 22:20:21,183 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@501] - shutting down
> 2018-02-19 22:20:21,184 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:FollowerRequestProcessor@107] - Shutting
> down
> 2018-02-19 22:20:21,184 [myid:1] - INFO
> [FollowerRequestProcessor:1:FollowerRequestProcessor@97] -
> FollowerRequestProcessor exited loop!
> 2018-02-19 22:20:21,184 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:CommitProcessor@184] - Shutting down
> 2018-02-19 22:20:21,185 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:FinalRequestProcessor@403] - shutdown of
> request processor complete
> 2018-02-19 22:20:21,185 [myid:1] - INFO
> [CommitProcessor:1:CommitProcessor@153] - CommitProcessor exited loop!
> 2018-02-19 22:20:21,185 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:SyncRequestProcessor@208] - Shutting down
> 2018-02-19 22:20:21,185 [myid:1] - INFO
> [SyncThread:1:SyncRequestProcessor@186] - SyncRequestProcessor exited!
> 2018-02-19 22:20:21,186 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@909] - LOOKING
> 2018-02-19 22:20:21,189 [myid:1] - INFO
> [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@820] - New election.
> My
> id =  1, proposed zxid=0x38000004e9
> 2018-02-19 22:20:21,189 [myid:1] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@602] - Notification: 1 (message
> fo/
>
> zoo.cfg has
>
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=100
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=10
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/data/zookeeper
> # the port at which the clients will connect
> clientPort=2181
> # the maximum number of client connections.
> # increase this if you need to handle more clients
> maxClientCnxns=200
> #
> # Be sure to read the maintenance section of the
> # administrator guide before turning on autopurge.
> #
> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.
> html#sc_maintenance
> #
> # The number of snapshots to retain in dataDir
> autopurge.snapRetainCount=50
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> autopurge.purgeInterval=3
> server.1=zk01:2888:3888
> server.2=zk02:2888:3888
> server.3=zk03:2888:3888
>
> I have increased initLimit and syncLimit.
>
> Is this a network issue?
>
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>