What happens when a server loses all its state?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

What happens when a server loses all its state?

Thomas Vinod Johnson
What is the expected behavior if a server in a ZooKeeper service
restarts with all its prior state lost? Empirically, everything seems to
work*.  Is this something that one can count on, as part of ZooKeeper
design, or are there known conditions under which this could cause
problems, either liveness or violation of ZooKeeper guarantees?

I'm really most interested in a situation where a single server loses
state, but insights into issues when more than one server loses state
and other interesting failure scenarios are appreciated.

Thanks.

* The restarted server appears to catch up to the latest snapshot (from
the current leader?).
Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Mahadev Konar
Hi Thomas,

If a zookeeper server loses all state and their are enough servers in the
ensemble to continue a zookeeper service ( like 2 servers in the case of
ensemble of 3), then the server will get the latest snapshot from the leader
and continue.


The idea of zookeeper persisting its state on disk is just so that it does
not lose state. All the guarantees that zookeeper makes is based on the
understanding that we do not lose state of the data we store on the disk.


Their might be problems if we lose the state that we stored on the disk.
We might lose transactions that have been committed and the ensemble might
start with some snapshot in the past.

You might want ot read through how zookeeper internals work. This will help
you understand on why the persistence guarantees are required.

http://wiki.apache.org/hadoop-data/attachments/ZooKeeper(2f)ZooKeeperPresent
ations/attachments/zk-talk-upc.pdf

mahadev



On 12/16/08 9:45 AM, "Thomas Vinod Johnson" <[hidden email]> wrote:

> What is the expected behavior if a server in a ZooKeeper service
> restarts with all its prior state lost? Empirically, everything seems to
> work*.  Is this something that one can count on, as part of ZooKeeper
> design, or are there known conditions under which this could cause
> problems, either liveness or violation of ZooKeeper guarantees?
>
> I'm really most interested in a situation where a single server loses
> state, but insights into issues when more than one server loses state
> and other interesting failure scenarios are appreciated.
>
> Thanks.
>
> * The restarted server appears to catch up to the latest snapshot (from
> the current leader?).

Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Thomas Vinod Johnson
Sorry, I should have been a little more explicit. At this point, the
situation I'm considering is this; out of 3 servers, 1 server 'A'
forgets its persistent state (due to a bad disk, say) and it restarts.
My guess from what I could understand/reason about the internals was
that the server 'A' will re-synchronize correctly on restart, by getting
the entire snapshot.

I just wanted to make sure that this was a good assumption to make - or
find out if I was missing corner cases where the fact that A has lost
all memory could lead to inconsistencies (to take an example, in plain
Paxos, no acceptor can forget the highest number prepare request to
which it has responded).

More generally, is it a safe assumption to make that the ZooKeeper
service will maintain all its guarantees if a minority of servers lose
persistent state (due to bad disks, etc) and restart at some point in
the future?

Thanks.
Mahadev Konar wrote:

> Hi Thomas,
>
> If a zookeeper server loses all state and their are enough servers in the
> ensemble to continue a zookeeper service ( like 2 servers in the case of
> ensemble of 3), then the server will get the latest snapshot from the leader
> and continue.
>
>
> The idea of zookeeper persisting its state on disk is just so that it does
> not lose state. All the guarantees that zookeeper makes is based on the
> understanding that we do not lose state of the data we store on the disk.
>
>
> Their might be problems if we lose the state that we stored on the disk.
> We might lose transactions that have been committed and the ensemble might
> start with some snapshot in the past.
>
> You might want ot read through how zookeeper internals work. This will help
> you understand on why the persistence guarantees are required.
>
> http://wiki.apache.org/hadoop-data/attachments/ZooKeeper(2f)ZooKeeperPresent
> ations/attachments/zk-talk-upc.pdf
>
> mahadev
>
>
>
> On 12/16/08 9:45 AM, "Thomas Vinod Johnson" <[hidden email]> wrote:
>
>  
>> What is the expected behavior if a server in a ZooKeeper service
>> restarts with all its prior state lost? Empirically, everything seems to
>> work*.  Is this something that one can count on, as part of ZooKeeper
>> design, or are there known conditions under which this could cause
>> problems, either liveness or violation of ZooKeeper guarantees?
>>
>> I'm really most interested in a situation where a single server loses
>> state, but insights into issues when more than one server loses state
>> and other interesting failure scenarios are appreciated.
>>
>> Thanks.
>>
>> * The restarted server appears to catch up to the latest snapshot (from
>> the current leader?).
>>    
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Mahadev Konar
Hi Thomas,



> More generally, is it a safe assumption to make that the ZooKeeper
> service will maintain all its guarantees if a minority of servers lose
> persistent state (due to bad disks, etc) and restart at some point in
> the future?
Yes that is true.

mahadev

>
> Thanks.
> Mahadev Konar wrote:
>> Hi Thomas,
>>
>> If a zookeeper server loses all state and their are enough servers in the
>> ensemble to continue a zookeeper service ( like 2 servers in the case of
>> ensemble of 3), then the server will get the latest snapshot from the leader
>> and continue.
>>
>>
>> The idea of zookeeper persisting its state on disk is just so that it does
>> not lose state. All the guarantees that zookeeper makes is based on the
>> understanding that we do not lose state of the data we store on the disk.
>>
>>
>> Their might be problems if we lose the state that we stored on the disk.
>> We might lose transactions that have been committed and the ensemble might
>> start with some snapshot in the past.
>>
>> You might want ot read through how zookeeper internals work. This will help
>> you understand on why the persistence guarantees are required.
>>
>> http://wiki.apache.org/hadoop-data/attachments/ZooKeeper(2f)ZooKeeperPresent
>> ations/attachments/zk-talk-upc.pdf
>>
>> mahadev
>>
>>
>>
>> On 12/16/08 9:45 AM, "Thomas Vinod Johnson" <[hidden email]> wrote:
>>
>>  
>>> What is the expected behavior if a server in a ZooKeeper service
>>> restarts with all its prior state lost? Empirically, everything seems to
>>> work*.  Is this something that one can count on, as part of ZooKeeper
>>> design, or are there known conditions under which this could cause
>>> problems, either liveness or violation of ZooKeeper guarantees?
>>>
>>> I'm really most interested in a situation where a single server loses
>>> state, but insights into issues when more than one server loses state
>>> and other interesting failure scenarios are appreciated.
>>>
>>> Thanks.
>>>
>>> * The restarted server appears to catch up to the latest snapshot (from
>>> the current leader?).
>>>    
>>
>>  
>

Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Thomas Vinod Johnson
Mahadev Konar wrote:

> Hi Thomas,
>
>
>
>  
>> More generally, is it a safe assumption to make that the ZooKeeper
>> service will maintain all its guarantees if a minority of servers lose
>> persistent state (due to bad disks, etc) and restart at some point in
>> the future?
>>    
> Yes that is true.
>
>  
Great - thanks Mahadev.

Not to drag this on more than necessary, please bear with me for one
more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
servers A, B, C.
- C is currently not running, A is the leader, B is the follower.
- A proposes zxid1 to A and B, both acknowledge.
- A asks A to commit (which it persists), but before the same commit
request reaches B, all servers go down (say a power failure).
- Later, B and C come up (A is slow to reboot), but B has lost all state
due to disk failure.
- C becomes the new leader and perhaps continues with some more new
transactions.

Likely I'm misunderstanding the protocol, but have I effectively lost
zxid1 at this point? What would happen when A comes back up?

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Mahadev Konar
Hi Thomas,
 Here is what would happen in the scenario you mentioned.

> Great - thanks Mahadev.
>
> Not to drag this on more than necessary, please bear with me for one
> more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
> servers A, B, C.
> - C is currently not running, A is the leader, B is the follower.
> - A proposes zxid1 to A and B, both acknowledge.
> - A asks A to commit (which it persists), but before the same commit
> request reaches B, all servers go down (say a power failure).
In this case, the zookeeper protocol says that zxid1 would be available only
if the client gets a success. So zxid1 may or may not get committed if A and
B come up later. ( this is a different scenario then what you mention
later).

> - Later, B and C come up (A is slow to reboot), but B has lost all state
> due to disk failure.
This is how zookeeper would work in this scenario ---

Now since we have B and C come up and B has the most recent state but loses
it, then zookeeper is clueless about this. So C would say I have the some
zxid say zxid-n and B would say that I have zxid = 0 (since its stateless)
and C would become a leader (since it has the highest zxid).

This would lead to loss of data and loss of state in zookeeper. That's what
I meant when I mentioned that zookeeper relies heavily on the state being
persisted on disk.

> - C becomes the new leader and perhaps continues with some more new
> transactions.
>
Now if A comes back again, C would say that its the leader and ask A to
truncate all the transactions that A had to come to sync with C.

Again, you can see that how persistence loss can trigger state loss in
zookeeper. If its just minority of servers failing then this can be taken
care of by zookeeper but in this scenario is C failing and then being
brought up with an inconsisten state with another failure of A and data loss
of B -- which zookeeper cannot handle.

I hope this helps.

mahadev


On 12/16/08 4:02 PM, "Thomas Vinod Johnson" <[hidden email]> wrote:

> Mahadev Konar wrote:
>> Hi Thomas,
>>
>>
>>
>>  
>>> More generally, is it a safe assumption to make that the ZooKeeper
>>> service will maintain all its guarantees if a minority of servers lose
>>> persistent state (due to bad disks, etc) and restart at some point in
>>> the future?
>>>    
>> Yes that is true.
>>
>>  
> Likely I'm misunderstanding the protocol, but have I effectively lost
> zxid1 at this point? What would happen when A comes back up?
>
> Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Thomas Vinod Johnson
Mahadev Konar wrote:

> Hi Thomas,
>  Here is what would happen in the scenario you mentioned.
>
>  
>> Great - thanks Mahadev.
>>
>> Not to drag this on more than necessary, please bear with me for one
>> more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
>> servers A, B, C.
>> - C is currently not running, A is the leader, B is the follower.
>> - A proposes zxid1 to A and B, both acknowledge.
>> - A asks A to commit (which it persists), but before the same commit
>> request reaches B, all servers go down (say a power failure).
>>    
> In this case, the zookeeper protocol says that zxid1 would be available only
> if the client gets a success. So zxid1 may or may not get committed if A and
> B come up later. ( this is a different scenario then what you mention
> later).
>
>  
The general scenario I was interested in was a minority of servers
losing state, and trying to understand what other correlated events
could cause issues. Just to be clear, since A has sent the commit to B
(or is it when A has got its own commit), it *could have* sent a success
back to the client before everything went down, correct?

>> - Later, B and C come up (A is slow to reboot), but B has lost all state
>> due to disk failure.
>>    
> This is how zookeeper would work in this scenario ---
>
> Now since we have B and C come up and B has the most recent state but loses
> it, then zookeeper is clueless about this. So C would say I have the some
> zxid say zxid-n and B would say that I have zxid = 0 (since its stateless)
> and C would become a leader (since it has the highest zxid).
>
> This would lead to loss of data and loss of state in zookeeper. That's what
> I meant when I mentioned that zookeeper relies heavily on the state being
> persisted on disk.
>
>  
OK good, my understanding was correct then.
>> - C becomes the new leader and perhaps continues with some more new
>> transactions.
>>
>>    
> Now if A comes back again, C would say that its the leader and ask A to
> truncate all the transactions that A had to come to sync with C.
>
>  
I wasn't aware that C would ask A to truncate even committed
transactions (the zookeeper internals doc/slides talks about proposals -
I suspect I may have some terminology confusion here). Another
possibility is C is now at zxid2 >= zxid1, in which case A could
possibly *not* get rid of the committed transaction?
> Again, you can see that how persistence loss can trigger state loss in
> zookeeper. If its just minority of servers failing then this can be taken
> care of by zookeeper but in this scenario is C failing and then being
> brought up with an inconsisten state with another failure of A and data loss
> of B -- which zookeeper cannot handle.
>
> I hope this helps.
>
>  
Yes thanks. Not sure if this makes sense, but is it worthwhile to have a
'safe' mode when a server comes up with no state (I think it should be
simple to distinguish between having a clean disk 'no state'/corrupt
state and 'empty state')? In this case, I think it could simply wait
till it sees a successful propose/commit cycle to know that it is safe
for it to take a snapshot and start participating in the ensemble.

In the scenario I previously described, when B and C comes up, B would
not respond to C, but just watch - C would not be able to establish
quorum until A came up; at which point B has witnessed a successful
leader activation, and can join. If one is willing to sacrifice liveness
for safety in situations where 1 or more nodes have amnesia, would this
be a viable option?
Reply | Threaded
Open this post in threaded view
|

RE: What happens when a server loses all its state?

Benjamin Reed-2
In reply to this post by Thomas Vinod Johnson
Thomas,

in the scenario you give you have two simultaneous failures with 3 nodes, so it will not recover correctly. A is failed because it is not up. B has failed because it lost all its data.

it would be good for ZooKeeper to not come up in that scenario. perhaps what we need is something similar to your safe state proposal. basically a server that has forgotten everything should not be allowed to vote in the leader election. that would avoid your scenario. we just need to put a flag file in the data directory to say that the data is valid and thus can vote.

ben
________________________________________
From: [hidden email] [[hidden email]]
Sent: Tuesday, December 16, 2008 4:02 PM
To: [hidden email]
Subject: Re: What happens when a server loses all its state?

Mahadev Konar wrote:

> Hi Thomas,
>
>
>
>
>> More generally, is it a safe assumption to make that the ZooKeeper
>> service will maintain all its guarantees if a minority of servers lose
>> persistent state (due to bad disks, etc) and restart at some point in
>> the future?
>>
> Yes that is true.
>
>
Great - thanks Mahadev.

Not to drag this on more than necessary, please bear with me for one
more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
servers A, B, C.
- C is currently not running, A is the leader, B is the follower.
- A proposes zxid1 to A and B, both acknowledge.
- A asks A to commit (which it persists), but before the same commit
request reaches B, all servers go down (say a power failure).
- Later, B and C come up (A is slow to reboot), but B has lost all state
due to disk failure.
- C becomes the new leader and perhaps continues with some more new
transactions.

Likely I'm misunderstanding the protocol, but have I effectively lost
zxid1 at this point? What would happen when A comes back up?

Thanks.
Reply | Threaded
Open this post in threaded view
|

RE: What happens when a server loses all its state?

Krishna Sankar (ksankar)
Just as a supporting note, from what I read, to support n simultaneous
failures we need 2n+1 nodes. In this case, we need 5 nodes to operate
correctly. Might be a good idea to capture this formula and if more than
n failures occur, write the appropriate flags which can then be used for
the right recovery state.

Cheers
<k/>  

|-----Original Message-----
|From: Benjamin Reed [mailto:[hidden email]]
|Sent: Wednesday, December 17, 2008 11:48 AM
|To: [hidden email]
|Subject: RE: What happens when a server loses all its state?
|
|Thomas,
|
|in the scenario you give you have two simultaneous failures with 3
|nodes, so it will not recover correctly. A is failed because it is not
|up. B has failed because it lost all its data.
|
|it would be good for ZooKeeper to not come up in that scenario. perhaps
|what we need is something similar to your safe state proposal.
basically
|a server that has forgotten everything should not be allowed to vote in
|the leader election. that would avoid your scenario. we just need to
put
|a flag file in the data directory to say that the data is valid and
thus
|can vote.
|
|ben
|________________________________________
|From: [hidden email] [[hidden email]]
|Sent: Tuesday, December 16, 2008 4:02 PM
|To: [hidden email]
|Subject: Re: What happens when a server loses all its state?
|
|Mahadev Konar wrote:
|> Hi Thomas,
|>
|>
|>
|>
|>> More generally, is it a safe assumption to make that the ZooKeeper
|>> service will maintain all its guarantees if a minority of servers
|lose
|>> persistent state (due to bad disks, etc) and restart at some point
in
|>> the future?
|>>
|> Yes that is true.
|>
|>
|Great - thanks Mahadev.
|
|Not to drag this on more than necessary, please bear with me for one
|more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
|servers A, B, C.
|- C is currently not running, A is the leader, B is the follower.
|- A proposes zxid1 to A and B, both acknowledge.
|- A asks A to commit (which it persists), but before the same commit
|request reaches B, all servers go down (say a power failure).
|- Later, B and C come up (A is slow to reboot), but B has lost all
state
|due to disk failure.
|- C becomes the new leader and perhaps continues with some more new
|transactions.
|
|Likely I'm misunderstanding the protocol, but have I effectively lost
|zxid1 at this point? What would happen when A comes back up?
|
|Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: What happens when a server loses all its state?

Thomas Vinod Johnson
In reply to this post by Benjamin Reed-2
Thanks for all the responses.
Benjamin Reed wrote:
> Thomas,
>
> in the scenario you give you have two simultaneous failures with 3 nodes, so it will not recover correctly. A is failed because it is not up. B has failed because it lost all its data.
>
> it would be good for ZooKeeper to not come up in that scenario. perhaps what we need is something similar to your safe state proposal. basically a server that has forgotten everything should not be allowed to vote in the leader election. that would avoid your scenario. we just need to put a flag file in the data directory to say that the data is valid and thus can vote.
>
> ben
> ________________________________________
>  
Would this feature be something you'd consider implementing in the short
to medium term?
Reply | Threaded
Open this post in threaded view
|

RE: What happens when a server loses all its state?

Benjamin Reed-2
I have opened ZOOKEEPER-261 for this issue. it shouldn't be too hard to fix and it would be nice to target for 3.1.

ben

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: Wednesday, December 17, 2008 2:52 PM
To: [hidden email]
Subject: Re: What happens when a server loses all its state?

Thanks for all the responses.
Benjamin Reed wrote:
> Thomas,
>
> in the scenario you give you have two simultaneous failures with 3 nodes, so it will not recover correctly. A is failed because it is not up. B has failed because it lost all its data.
>
> it would be good for ZooKeeper to not come up in that scenario. perhaps what we need is something similar to your safe state proposal. basically a server that has forgotten everything should not be allowed to vote in the leader election. that would avoid your scenario. we just need to put a flag file in the data directory to say that the data is valid and thus can vote.
>
> ben
> ________________________________________
>  
Would this feature be something you'd consider implementing in the short
to medium term?