java.io.EOFException

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

java.io.EOFException

I PVP
How to recover from the following error that  started happening after a server crash?
Zookeeper won’t start and the following message is showing repeatedly on the log.

2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=3.10.0-514.16.1.el7.x86_64
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/opt/zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@829] - tickTime set to 2000
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@838] - minSessionTimeout set to -1
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@847] - maxSessionTimeout set to -1
2017-05-27 01:02:08,080 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2017-05-27 01:02:08,385 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,400 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:ZooKeeperServerMain@64] - Unexpected exception, exiting abnormally
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
        at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:283)
        at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:410)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)


any help is appreciate

best
IPVP
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.io.EOFException

Martin Gainty
MG>below

________________________________
From: I PVP <[hidden email]>
Sent: Saturday, May 27, 2017 12:07 AM
To: UserZooKeeper
Subject: java.io.EOFException

How to recover from the following error that  started happening after a server crash?
Zookeeper won’t start and the following message is showing repeatedly on the log.

2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
2017-05-27 01:02:08,072 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=3.10.0-514.16.1.el7.x86_64
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=/opt/zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@829] - tickTime set to 2000
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@838] - minSessionTimeout set to -1
2017-05-27 01:02:08,074 [myid:] - INFO  [main:ZooKeeperServer@847] - maxSessionTimeout set to -1
2017-05-27 01:02:08,080 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2017-05-27 01:02:08,385 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,400 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:ZooKeeperServerMain@64] - Unexpected exception, exiting abnormally
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
        at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)

MG>in the logs you should see
MG>"Created new input stream" NameOfLogFile
MG>once the archive is created you should see
MG>"Created new input archive " logFile

        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:283)
        at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:410)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)


any help is appreciate

best
IPVP
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.io.EOFException

I PVP
Martin Gainty,

Thanks for answering.

I was able to solve the issue by running the following:

sudo rm -rf /tmp/zookeeper
sudo rm -rf /var/lib/zookeeper
sudo rm -rf /var/log/zookeeper
sudo mkdir -p /tmp/zookeeper
sudo mkdir -p /var/lib/zookeeper/data
sudo mkdir -p /var/lib/zookeeper/data-log
sudo mkdir -p /var/log/zookeeper
sudo chown -R zookeeper:zookeeper /var/log/zookeeper/ /var/lib/zookeeper/ /tmp/zookeeper/

best,

IPVP


On May 28, 2017 at 9:56:43 AM, Martin Gainty ([hidden email]<mailto:[hidden email]>) wrote:

MG>below

________________________________
From: I PVP <[hidden email]>
Sent: Saturday, May 27, 2017 12:07 AM
To: UserZooKeeper
Subject: java.io.EOFException

How to recover from the following error that started happening after a server crash?
Zookeeper won’t start and the following message is showing repeatedly on the log.

2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.10.0-514.16.1.el7.x86_64
2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.name=zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/opt/zookeeper
2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/
2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@829] - tickTime set to 2000
2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@838] - minSessionTimeout set to -1
2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@847] - maxSessionTimeout set to -1
2017-05-27 01:02:08,080 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2017-05-27 01:02:08,385 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,400 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
2017-05-27 01:02:08,404 [myid:] - ERROR [main:ZooKeeperServerMain@64] - Unexpected exception, exiting abnormally
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)

MG>in the logs you should see
MG>"Created new input stream" NameOfLogFile
MG>once the archive is created you should see
MG>"Created new input archive " logFile

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:283)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:410)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)


any help is appreciate

best
IPVP
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: java.io.EOFException

Flavio Junqueira-3
It think it was already established that a txn log file was corrupted. I'm guessing it couldn't find a combination of snapshot + txn log file that made it load the database correctly. Which version are you using?

I would have been useful to check the txn log files and snapshots before getting rid of them. In the case you're interested, you can parse them using log formatter and snapshot formatter, respectively. Those are tools we make available with the distribution.

-Flavio

> On 29 May 2017, at 10:02, I PVP <[hidden email]> wrote:
>
> Martin Gainty,
>
> Thanks for answering.
>
> I was able to solve the issue by running the following:
>
> sudo rm -rf /tmp/zookeeper
> sudo rm -rf /var/lib/zookeeper
> sudo rm -rf /var/log/zookeeper
> sudo mkdir -p /tmp/zookeeper
> sudo mkdir -p /var/lib/zookeeper/data
> sudo mkdir -p /var/lib/zookeeper/data-log
> sudo mkdir -p /var/log/zookeeper
> sudo chown -R zookeeper:zookeeper /var/log/zookeeper/ /var/lib/zookeeper/ /tmp/zookeeper/
>
> best,
>
> IPVP
>
>
> On May 28, 2017 at 9:56:43 AM, Martin Gainty ([hidden email]<mailto:[hidden email]>) wrote:
>
> MG>below
>
> ________________________________
> From: I PVP <[hidden email]>
> Sent: Saturday, May 27, 2017 12:07 AM
> To: UserZooKeeper
> Subject: java.io.EOFException
>
> How to recover from the following error that started happening after a server crash?
> Zookeeper won’t start and the following message is showing repeatedly on the log.
>
> 2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
> 2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
> 2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
> 2017-05-27 01:02:08,072 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
> 2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.10.0-514.16.1.el7.x86_64
> 2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.name=zookeeper
> 2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/opt/zookeeper
> 2017-05-27 01:02:08,073 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/
> 2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@829] - tickTime set to 2000
> 2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@838] - minSessionTimeout set to -1
> 2017-05-27 01:02:08,074 [myid:] - INFO [main:ZooKeeperServer@847] - maxSessionTimeout set to -1
> 2017-05-27 01:02:08,080 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
> 2017-05-27 01:02:08,385 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
> 2017-05-27 01:02:08,400 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
> 2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
> 2017-05-27 01:02:08,403 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
> 2017-05-27 01:02:08,404 [myid:] - ERROR [main:Util@239] - Last transaction was partial.
> 2017-05-27 01:02:08,404 [myid:] - ERROR [main:ZooKeeperServerMain@64] - Unexpected exception, exiting abnormally
> java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
> at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:585)
>
> MG>in the logs you should see
> MG>"Created new input stream" NameOfLogFile
> MG>once the archive is created you should see
> MG>"Created new input archive " logFile
>
> at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:604)
> at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:570)
> at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:652)
> at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:283)
> at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:410)
> at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
> at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:119)
> at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
> at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
> at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>
>
> any help is appreciate
>
> best
> IPVP

Loading...