ephemerals handling after restart

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ephemerals handling after restart

Johannes Zillmann
Hi all,

i have a question regarding ephemerals and it behavior on client crash/
restart.
We've a master/node cluster similar to a hadoop hdfs cluster but using  
zk for management.
The nodes creates an ephemeral to announce there existence to the  
master.

Now what i recognized is that after stopping the whole cluster and  
starting only the master again, still some ephemeral nodes may exists  
for some seconds.
That leads me to following questions.
What is if a node starts up again. Do it have to clean up it "old"  
ephemeral node, or can it somehow acquire the old one ?

Just trying to find a best way how to deal with this, since on regular  
restart of the cluster i often recognize something like this
master : node1, node2 connected
master : node2 disconnected
master : node2 connected again

Thanks for any help
Johannes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec GmbH
Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com

Reply | Threaded
Open this post in threaded view
|

RE: ephemerals handling after restart

Benjamin Reed-2
If a application does not close the ZooKeeper session before shutting
down, ZooKeeper will not cleanup the session until it times out. So when
an application crashes and restarts, ZooKeeper doesn't know if the
client is a restart of an old client or a new client.

There is a way to alleviate this problem: you can actually maintain a
session across client application restarts. If you save off the session
id and password, when you restart you can try to reconnect to the
session using the ZooKeeper constructor that takes the old session id
and password. If the reconnect is successful you can then close the
session and get everything to clean up immediately. (Or you could keep
using the recovered session if you want to.)

ben

-----Original Message-----
From: Johannes Zillmann [mailto:[hidden email]]
Sent: Friday, September 12, 2008 2:49 AM
To: [hidden email]
Subject: ephemerals handling after restart

Hi all,

i have a question regarding ephemerals and it behavior on client crash/
restart.
We've a master/node cluster similar to a hadoop hdfs cluster but using  
zk for management.
The nodes creates an ephemeral to announce there existence to the  
master.

Now what i recognized is that after stopping the whole cluster and  
starting only the master again, still some ephemeral nodes may exists  
for some seconds.
That leads me to following questions.
What is if a node starts up again. Do it have to clean up it "old"  
ephemeral node, or can it somehow acquire the old one ?

Just trying to find a best way how to deal with this, since on regular  
restart of the cluster i often recognize something like this
master : node1, node2 connected
master : node2 disconnected
master : node2 connected again

Thanks for any help
Johannes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec GmbH
Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com

Reply | Threaded
Open this post in threaded view
|

Re: ephemerals handling after restart

Johannes Zillmann
Hi Ben,

thanks for your answer!
Is the session recoverable in case the zk server was restarted in  
meantime ?

Johannes

On Sep 12, 2008, at 3:52 PM, Benjamin Reed wrote:

> If a application does not close the ZooKeeper session before shutting
> down, ZooKeeper will not cleanup the session until it times out. So  
> when
> an application crashes and restarts, ZooKeeper doesn't know if the
> client is a restart of an old client or a new client.
>
> There is a way to alleviate this problem: you can actually maintain a
> session across client application restarts. If you save off the  
> session
> id and password, when you restart you can try to reconnect to the
> session using the ZooKeeper constructor that takes the old session id
> and password. If the reconnect is successful you can then close the
> session and get everything to clean up immediately. (Or you could keep
> using the recovered session if you want to.)
>
> ben
>
> -----Original Message-----
> From: Johannes Zillmann [mailto:[hidden email]]
> Sent: Friday, September 12, 2008 2:49 AM
> To: [hidden email]
> Subject: ephemerals handling after restart
>
> Hi all,
>
> i have a question regarding ephemerals and it behavior on client  
> crash/
> restart.
> We've a master/node cluster similar to a hadoop hdfs cluster but using
> zk for management.
> The nodes creates an ephemeral to announce there existence to the
> master.
>
> Now what i recognized is that after stopping the whole cluster and
> starting only the master again, still some ephemeral nodes may exists
> for some seconds.
> That leads me to following questions.
> What is if a node starts up again. Do it have to clean up it "old"
> ephemeral node, or can it somehow acquire the old one ?
>
> Just trying to find a best way how to deal with this, since on regular
> restart of the cluster i often recognize something like this
> master : node1, node2 connected
> master : node2 disconnected
> master : node2 connected again
>
> Thanks for any help
> Johannes
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec GmbH
> Halle (Saale), Saxony-Anhalt, Germany
> http://www.101tec.com
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec GmbH
Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com

Reply | Threaded
Open this post in threaded view
|

RE: Re: ephemerals handling after restart

Benjamin Reed-2
In reply to this post by Johannes Zillmann
Yes

Ben


 -----Original Message-----
From: Johannes Zillmann [mailto:[hidden email]]
Sent: Thursday, September 18, 2008 02:22 AM Pacific Standard Time
To: [hidden email]
Subject: Re: ephemerals handling after restart

Hi Ben,

thanks for your answer!
Is the session recoverable in case the zk server was restarted in  
meantime ?

Johannes

On Sep 12, 2008, at 3:52 PM, Benjamin Reed wrote:

> If a application does not close the ZooKeeper session before shutting
> down, ZooKeeper will not cleanup the session until it times out. So  
> when
> an application crashes and restarts, ZooKeeper doesn't know if the
> client is a restart of an old client or a new client.
>
> There is a way to alleviate this problem: you can actually maintain a
> session across client application restarts. If you save off the  
> session
> id and password, when you restart you can try to reconnect to the
> session using the ZooKeeper constructor that takes the old session id
> and password. If the reconnect is successful you can then close the
> session and get everything to clean up immediately. (Or you could keep
> using the recovered session if you want to.)
>
> ben
>
> -----Original Message-----
> From: Johannes Zillmann [mailto:[hidden email]]
> Sent: Friday, September 12, 2008 2:49 AM
> To: [hidden email]
> Subject: ephemerals handling after restart
>
> Hi all,
>
> i have a question regarding ephemerals and it behavior on client  
> crash/
> restart.
> We've a master/node cluster similar to a hadoop hdfs cluster but using
> zk for management.
> The nodes creates an ephemeral to announce there existence to the
> master.
>
> Now what i recognized is that after stopping the whole cluster and
> starting only the master again, still some ephemeral nodes may exists
> for some seconds.
> That leads me to following questions.
> What is if a node starts up again. Do it have to clean up it "old"
> ephemeral node, or can it somehow acquire the old one ?
>
> Just trying to find a best way how to deal with this, since on regular
> restart of the cluster i often recognize something like this
> master : node1, node2 connected
> master : node2 disconnected
> master : node2 connected again
>
> Thanks for any help
> Johannes
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec GmbH
> Halle (Saale), Saxony-Anhalt, Germany
> http://www.101tec.com
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec GmbH
Halle (Saale), Saxony-Anhalt, Germany
http://www.101tec.com