replacing one zookeeper machine with brand new machine

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

replacing one zookeeper machine with brand new machine

Check Peck
I have a zookeeper ensemble of 5 servers and I am using exhibitor on top of
it. And I installed exhibitor and setup zookeeper by following this link:

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Below is how all my zookeeper machines are setup in exhibitor

    S:1:machineA,
    S:2:machineB,
    S:3:machineC,
    S:4:machineD,
    S:5:machineE

Now for some reasons, I need to replace "machineE" with brand new
"machineF". What is the best way by which I can safely remove one machine
and replace it with new machine?
Reply | Threaded
Open this post in threaded view
|

Re: replacing one zookeeper machine with brand new machine

Check Peck
 I have a zookeeper ensemble of 5 servers and I am using exhibitor on top
of it. And I installed exhibitor and setup zookeeper by following this link:

https://github.com/soabase/exhibitor/wiki/Building-Exhibitor

Below is how all my zookeeper machines are setup in exhibitor

    S:1:machineA,
    S:2:machineB,
    S:3:machineC,
    S:4:machineD,
    S:5:machineE

Now for some reasons, I need to replace "machineE" with brand new
"machineF". What is the best way by which I can safely remove one machine
and replace it with new machine?
Reply | Threaded
Open this post in threaded view
|

Re: replacing one zookeeper machine with brand new machine

Washko, Daniel
In reply to this post by Check Peck
The steps are the same whether Exhibitor is in the mix or not. Exhibitor will take care of management, though. I would recommend backing up the data in your Zookeeper ensemble just to be safe.

1) Spin up a new zookeeper and configure it to use exhibitor.
2) Let exhibitor bring it into the ensemble.
3) Use exhibitor to remove the old node.
4) Terminate the old node when exhibitor says it is no longer in the ensemble; or it is down.

It has been a few years since I have worked with Exhibitor. It should automatically pull the new node into the ensemble. I believe there is an option to remove a node. You will be presented with a choice on how you want to initiate the changes - a rolling restart of restart all at once. I would recommend a rolling restart if you want to keep the ensemble live while you make the changes.

If you have a problem with removing one of the nodes, you can edit the node list in exhibitor, remove that node, and save the configuration. Again, this will prompt for a rolling restart or parallel restart.

Without exhibitor these are the steps I follow:

1) Backup the data
2) Spin up a new zookeeper
3) Identify the master
4) Alter the configuration on each zookeeper to add the new node and to add the other nodes to the new zookeeper. Be aware of the zookeeper ID, it has to be unique.
5) Perform a rolling restart of each node with the master last.
6) Verify the new master and that the data stored in zookeeper has migrated successfully to the new node.
7) Remove the old node from each config.
8) Stop zookeeper on the old node and do a rolling restart of the remaining zookeepr nodes with the master last.
9) Terminate the old node.

--
Daniel S Washko
Solutions Architect



Phone: 757 667 1463
[hidden email] <http://www.gannett.com/>

On 2/2/18, 3:20 PM, "Check Peck" <[hidden email]> wrote:

    I have a zookeeper ensemble of 5 servers and I am using exhibitor on top of
    it. And I installed exhibitor and setup zookeeper by following this link:
   
    https://github.com/soabase/exhibitor/wiki/Building-Exhibitor
   
    Below is how all my zookeeper machines are setup in exhibitor
   
        S:1:machineA,
        S:2:machineB,
        S:3:machineC,
        S:4:machineD,
        S:5:machineE
   
    Now for some reasons, I need to replace "machineE" with brand new
    "machineF". What is the best way by which I can safely remove one machine
    and replace it with new machine?
   

Reply | Threaded
Open this post in threaded view
|

Re: replacing one zookeeper machine with brand new machine

Check Peck
Is there an option to remove a zookeeper node from exhibitor? I am not sure
it is there I guess.

On Mon, Feb 5, 2018 at 10:21 AM, Washko, Daniel <[hidden email]> wrote:

> The steps are the same whether Exhibitor is in the mix or not. Exhibitor
> will take care of management, though. I would recommend backing up the data
> in your Zookeeper ensemble just to be safe.
>
> 1) Spin up a new zookeeper and configure it to use exhibitor.
> 2) Let exhibitor bring it into the ensemble.
> 3) Use exhibitor to remove the old node.
> 4) Terminate the old node when exhibitor says it is no longer in the
> ensemble; or it is down.
>
> It has been a few years since I have worked with Exhibitor. It should
> automatically pull the new node into the ensemble. I believe there is an
> option to remove a node. You will be presented with a choice on how you
> want to initiate the changes - a rolling restart of restart all at once. I
> would recommend a rolling restart if you want to keep the ensemble live
> while you make the changes.
>
> If you have a problem with removing one of the nodes, you can edit the
> node list in exhibitor, remove that node, and save the configuration.
> Again, this will prompt for a rolling restart or parallel restart.
>
> Without exhibitor these are the steps I follow:
>
> 1) Backup the data
> 2) Spin up a new zookeeper
> 3) Identify the master
> 4) Alter the configuration on each zookeeper to add the new node and to
> add the other nodes to the new zookeeper. Be aware of the zookeeper ID, it
> has to be unique.
> 5) Perform a rolling restart of each node with the master last.
> 6) Verify the new master and that the data stored in zookeeper has
> migrated successfully to the new node.
> 7) Remove the old node from each config.
> 8) Stop zookeeper on the old node and do a rolling restart of the
> remaining zookeepr nodes with the master last.
> 9) Terminate the old node.
>
> --
> Daniel S Washko
> Solutions Architect
>
>
>
> Phone: 757 667 1463
> [hidden email] <http://www.gannett.com/>
>
> On 2/2/18, 3:20 PM, "Check Peck" <[hidden email]> wrote:
>
>     I have a zookeeper ensemble of 5 servers and I am using exhibitor on
> top of
>     it. And I installed exhibitor and setup zookeeper by following this
> link:
>
>     https://github.com/soabase/exhibitor/wiki/Building-Exhibitor
>
>     Below is how all my zookeeper machines are setup in exhibitor
>
>         S:1:machineA,
>         S:2:machineB,
>         S:3:machineC,
>         S:4:machineD,
>         S:5:machineE
>
>     Now for some reasons, I need to replace "machineE" with brand new
>     "machineF". What is the best way by which I can safely remove one
> machine
>     and replace it with new machine?
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: replacing one zookeeper machine with brand new machine

Washko, Daniel
There is a restart/stop button on the explorer page. If you turn off automatic restarts you can stop an instance. To remove the instance I believe there is a list of zookeeper nodes on the Config page. If you edit that and remove the zookeeper instances you are terminating and commit the change it should push it out. Once the restarts complete and all is green, you should be able to safely terminate the removed nodes.

--
Daniel S Washko
Solutions Architect



Phone: 757 667 1463
[hidden email] <http://www.gannett.com/>
On 2/5/18, 3:27 PM, "Check Peck" <[hidden email]> wrote:

    Is there an option to remove a zookeeper node from exhibitor? I am not sure
    it is there I guess.
   
    On Mon, Feb 5, 2018 at 10:21 AM, Washko, Daniel <[hidden email]> wrote:
   
    > The steps are the same whether Exhibitor is in the mix or not. Exhibitor
    > will take care of management, though. I would recommend backing up the data
    > in your Zookeeper ensemble just to be safe.
    >
    > 1) Spin up a new zookeeper and configure it to use exhibitor.
    > 2) Let exhibitor bring it into the ensemble.
    > 3) Use exhibitor to remove the old node.
    > 4) Terminate the old node when exhibitor says it is no longer in the
    > ensemble; or it is down.
    >
    > It has been a few years since I have worked with Exhibitor. It should
    > automatically pull the new node into the ensemble. I believe there is an
    > option to remove a node. You will be presented with a choice on how you
    > want to initiate the changes - a rolling restart of restart all at once. I
    > would recommend a rolling restart if you want to keep the ensemble live
    > while you make the changes.
    >
    > If you have a problem with removing one of the nodes, you can edit the
    > node list in exhibitor, remove that node, and save the configuration.
    > Again, this will prompt for a rolling restart or parallel restart.
    >
    > Without exhibitor these are the steps I follow:
    >
    > 1) Backup the data
    > 2) Spin up a new zookeeper
    > 3) Identify the master
    > 4) Alter the configuration on each zookeeper to add the new node and to
    > add the other nodes to the new zookeeper. Be aware of the zookeeper ID, it
    > has to be unique.
    > 5) Perform a rolling restart of each node with the master last.
    > 6) Verify the new master and that the data stored in zookeeper has
    > migrated successfully to the new node.
    > 7) Remove the old node from each config.
    > 8) Stop zookeeper on the old node and do a rolling restart of the
    > remaining zookeepr nodes with the master last.
    > 9) Terminate the old node.
    >
    > --
    > Daniel S Washko
    > Solutions Architect
    >
    >
    >
    > Phone: 757 667 1463
    > [hidden email] <http://www.gannett.com/>
    >
    > On 2/2/18, 3:20 PM, "Check Peck" <[hidden email]> wrote:
    >
    >     I have a zookeeper ensemble of 5 servers and I am using exhibitor on
    > top of
    >     it. And I installed exhibitor and setup zookeeper by following this
    > link:
    >
    >     https://github.com/soabase/exhibitor/wiki/Building-Exhibitor
    >
    >     Below is how all my zookeeper machines are setup in exhibitor
    >
    >         S:1:machineA,
    >         S:2:machineB,
    >         S:3:machineC,
    >         S:4:machineD,
    >         S:5:machineE
    >
    >     Now for some reasons, I need to replace "machineE" with brand new
    >     "machineF". What is the best way by which I can safely remove one
    > machine
    >     and replace it with new machine?
    >
    >
    >