Quantcast

EOFException on snapshot dump

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

EOFException on snapshot dump

Daniel Chan
I am using org.apache.zookeeper.server.SnapshotFormatter to dump out the content of a snapshot (908M) and got this exception:
Exception in thread "main" java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at java.io.DataInputStream.readFully(DataInputStream.java:169)
        at org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.java:82)
        at org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1217)
        at org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:127)
        at org.apache.zookeeper.server.persistence.FileSnap.deserialize(FileSnap.java:127)
        at org.apache.zookeeper.server.SnapshotFormatter.run(SnapshotFormatter.java:65)
        at org.apache.zookeeper.server.SnapshotFormatter.main(SnapshotFormatter.java:51)

Is there any debug mode I can turn on (in SnapshotFormatter) to find out what happened?

Background: a standalone Zookeeper failed on restart with below exception. I tried to see what is being stored from the last snapshot.
2017-04-25 10:47:27,831 [myid:] - ERROR [main:NIOServerCnxnFactory$1@44] - Thread Thread[main,5,main] died
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:97)
    at org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:158)
    at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
    at org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1194)
    at org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:127)
    at org.apache.zookeeper.server.persistence.FileSnap.deserialize(FileSnap.java:127)
    at org.apache.zookeeper.server.persistence.FileSnap.deserialize(FileSnap.java:87)
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:130)
    at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
    at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:272)
    at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:399)
    at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:113)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Thanks,
Daniel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: EOFException on snapshot dump

hanm
No debug mode afaik. I suspect the snapshot was corrupted - it's partial so
the read was expecting more bytes than the file actually has, thus
EOFException. A workaround would be patch SnapshotFormatter so it caught
the exception and print what's already loaded and parsed, instead of bail
out and throw everything away.

On Tue, Apr 25, 2017 at 3:54 PM, Daniel Chan <[hidden email]>
wrote:

> I am using org.apache.zookeeper.server.SnapshotFormatter to dump out the
> content of a snapshot (908M) and got this exception:
> Exception in thread "main" java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>         at java.io.DataInputStream.readFully(DataInputStream.java:169)
>         at org.apache.jute.BinaryInputArchive.readString(BinaryInputArc
> hive.java:82)
>         at org.apache.zookeeper.server.DataTree.deserialize(DataTree.ja
> va:1217)
>         at org.apache.zookeeper.server.util.SerializeUtils.deserializeS
> napshot(SerializeUtils.java:127)
>         at org.apache.zookeeper.server.persistence.FileSnap.deserialize
> (FileSnap.java:127)
>         at org.apache.zookeeper.server.SnapshotFormatter.run(SnapshotFo
> rmatter.java:65)
>         at org.apache.zookeeper.server.SnapshotFormatter.main(SnapshotF
> ormatter.java:51)
>
> Is there any debug mode I can turn on (in SnapshotFormatter) to find out
> what happened?
>
> Background: a standalone Zookeeper failed on restart with below exception.
> I tried to see what is being stored from the last snapshot.
> 2017-04-25 10:47:27,831 [myid:] - ERROR [main:NIOServerCnxnFactory$1@44]
> - Thread Thread[main,5,main] died
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>     at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArc
> hive.java:97)
>     at org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:158)
>     at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArc
> hive.java:103)
>     at org.apache.zookeeper.server.DataTree.deserialize(DataTree.ja
> va:1194)
>     at org.apache.zookeeper.server.util.SerializeUtils.deserializeS
> napshot(SerializeUtils.java:127)
>     at org.apache.zookeeper.server.persistence.FileSnap.deserialize
> (FileSnap.java:127)
>     at org.apache.zookeeper.server.persistence.FileSnap.deserialize
> (FileSnap.java:87)
>     at org.apache.zookeeper.server.persistence.FileTxnSnapLog.resto
> re(FileTxnSnapLog.java:130)
>     at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDataba
> se.java:223)
>     at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeep
> erServer.java:272)
>     at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKee
> perServer.java:399)
>     at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIO
> ServerCnxnFactory.java:122)
>     at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfi
> g(ZooKeeperServerMain.java:113)
>     at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAn
> dRun(ZooKeeperServerMain.java:86)
>     at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeep
> erServerMain.java:52)
>     at org.apache.zookeeper.server.quorum.QuorumPeerMain.initialize
> AndRun(QuorumPeerMain.java:116)
>     at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(Quoru
> mPeerMain.java:78)
>
> Thanks,
> Daniel
>



--
Cheers
Michael.
Loading...