Rolling restart zookeeper

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Rolling restart zookeeper

upendar devu
We have 3 aws instances of Kafka n zookeeper each of them.can we do rolling
restart of zookeeper one instance at a time. Are there any impacts doing
this? Please let me know.
Thanks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Rolling restart zookeeper

Washko, Daniel
This is what I would recommend. Identify the leader and restart that one last. Zookeeper should be able to remain functional and server requests during the restart of a single node. Of course you always take a risk if that node does not come back for some reason and then you will be left with and ensemble that cannot achieve quorum. After each restart verify the state of the node.

echo stat | nc IP_ADDRESS_OF_NODE 2181

Will provide that information – Whether it is the leader or follower.

We run zookeeper with Solr and do not see an impact with rolling restarts of zookeeper.

The only time I see problems is if we are adding new nodes to a live system or there are problems with your ensemble. If for some reason there is a zookeeper node that is not in sync with the others and that node becomes the leader, there is a potential for the data stored in zookeeper to be incomplete or wiped out.


Daniel S Washko
Solutions Architect



[hidden email] <http://www.gannett.com/>

On 7/11/17, 7:06 PM, "upendar devu" <[hidden email]> wrote:

    We have 3 aws instances of Kafka n zookeeper each of them.can we do rolling
    restart of zookeeper one instance at a time. Are there any impacts doing
    this? Please let me know.
    Thanks
   

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Rolling restart zookeeper

upendar devu
Thank you for detailed answer.i will follow the same.

On Jul 12, 2017 10:37 AM, "Washko, Daniel" <[hidden email]> wrote:

This is what I would recommend. Identify the leader and restart that one
last. Zookeeper should be able to remain functional and server requests
during the restart of a single node. Of course you always take a risk if
that node does not come back for some reason and then you will be left with
and ensemble that cannot achieve quorum. After each restart verify the
state of the node.

echo stat | nc IP_ADDRESS_OF_NODE 2181

Will provide that information – Whether it is the leader or follower.

We run zookeeper with Solr and do not see an impact with rolling restarts
of zookeeper.

The only time I see problems is if we are adding new nodes to a live system
or there are problems with your ensemble. If for some reason there is a
zookeeper node that is not in sync with the others and that node becomes
the leader, there is a potential for the data stored in zookeeper to be
incomplete or wiped out.


Daniel S Washko
Solutions Architect



[hidden email] <http://www.gannett.com/>

On 7/11/17, 7:06 PM, "upendar devu" <[hidden email]> wrote:

    We have 3 aws instances of Kafka n zookeeper each of them.can we do
rolling
    restart of zookeeper one instance at a time. Are there any impacts doing
    this? Please let me know.
    Thanks
Loading...