Quantcast

Growing a cluster

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Growing a cluster

Jared Cantwell
Hello,

Say I was going to grow a cluster from 1 node to 3 nodes.  Is this possible,
and what would be the recommended way?

At first I was thinking that I could go from 1 to 2 and then 2 to 3, and it
seems to be working actually.  But I'm not sure if this is supported, mostly
because in standalone mode the on-disk files are different than they are in
a multi-node configurations.  Mutli-node configurations embed the quorum
incarnation into the filename, which standalone does not.  Should a quorum
node be able to startup using snapshots and logs that a standalone node
wrote out?  Is there a way around this?

Thanks,
Jared
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Growing a cluster

Alexander Shraer
Hi Jared,

Currently there is no support for adding and removing zookeeper nodes dynamically. See:
https://issues.apache.org/jira/browse/ZOOKEEPER-107

We're currently working to add this feature. However, AFAIK there is no intention to support
transformation between standalone and multi-node modes, only membership changes in multi-node mode.

Regards,
Alex


> -----Original Message-----
> From: Jared Cantwell [mailto:[hidden email]]
> Sent: Wednesday, May 04, 2011 7:17 PM
> To: [hidden email]
> Subject: Growing a cluster
>
> Hello,
>
> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> possible,
> and what would be the recommended way?
>
> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> and it
> seems to be working actually.  But I'm not sure if this is supported,
> mostly
> because in standalone mode the on-disk files are different than they
> are in
> a multi-node configurations.  Mutli-node configurations embed the
> quorum
> incarnation into the filename, which standalone does not.  Should a
> quorum
> node be able to startup using snapshots and logs that a standalone node
> wrote out?  Is there a way around this?
>
> Thanks,
> Jared
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Jared Cantwell
It would be acceptable to me to do this non-dynamically and non-rolling as
well. For example, I can shut down all nodes, make necessary modifications
to config files, and then restart all nodes.  If I do this, should switching
from standalone mode to multi-node mode work?  Has anyone done this before?
 Preliminary tests seem to work, but I haven't looked into all the race
conditions and such yet.

~Jared

On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <[hidden email]>wrote:

> Hi Jared,
>
> Currently there is no support for adding and removing zookeeper nodes
> dynamically. See:
> https://issues.apache.org/jira/browse/ZOOKEEPER-107
>
> We're currently working to add this feature. However, AFAIK there is no
> intention to support
> transformation between standalone and multi-node modes, only membership
> changes in multi-node mode.
>
> Regards,
> Alex
>
>
> > -----Original Message-----
> > From: Jared Cantwell [mailto:[hidden email]]
> > Sent: Wednesday, May 04, 2011 7:17 PM
> > To: [hidden email]
> > Subject: Growing a cluster
> >
> > Hello,
> >
> > Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> > possible,
> > and what would be the recommended way?
> >
> > At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> > and it
> > seems to be working actually.  But I'm not sure if this is supported,
> > mostly
> > because in standalone mode the on-disk files are different than they
> > are in
> > a multi-node configurations.  Mutli-node configurations embed the
> > quorum
> > incarnation into the filename, which standalone does not.  Should a
> > quorum
> > node be able to startup using snapshots and logs that a standalone node
> > wrote out?  Is there a way around this?
> >
> > Thanks,
> > Jared
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Ted Dunning
Jared,

Doing it in a rolling fashion is fine.  Just reconfigure one node at a time.

There are interesting failure scenarios that make that kind of dangerous to
do automatically, but as a manual process it works just fine and only takes
a few minutes (assuming you don't have GB and GB of data in ZK).

On Thu, May 5, 2011 at 5:57 AM, Jared Cantwell <[hidden email]>wrote:

> It would be acceptable to me to do this non-dynamically and non-rolling as
> well. For example, I can shut down all nodes, make necessary modifications
> to config files, and then restart all nodes.  If I do this, should
> switching
> from standalone mode to multi-node mode work?  Has anyone done this before?
>  Preliminary tests seem to work, but I haven't looked into all the race
> conditions and such yet.
>
> ~Jared
>
> On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <[hidden email]
> >wrote:
>
> > Hi Jared,
> >
> > Currently there is no support for adding and removing zookeeper nodes
> > dynamically. See:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-107
> >
> > We're currently working to add this feature. However, AFAIK there is no
> > intention to support
> > transformation between standalone and multi-node modes, only membership
> > changes in multi-node mode.
> >
> > Regards,
> > Alex
> >
> >
> > > -----Original Message-----
> > > From: Jared Cantwell [mailto:[hidden email]]
> > > Sent: Wednesday, May 04, 2011 7:17 PM
> > > To: [hidden email]
> > > Subject: Growing a cluster
> > >
> > > Hello,
> > >
> > > Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> > > possible,
> > > and what would be the recommended way?
> > >
> > > At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> > > and it
> > > seems to be working actually.  But I'm not sure if this is supported,
> > > mostly
> > > because in standalone mode the on-disk files are different than they
> > > are in
> > > a multi-node configurations.  Mutli-node configurations embed the
> > > quorum
> > > incarnation into the filename, which standalone does not.  Should a
> > > quorum
> > > node be able to startup using snapshots and logs that a standalone node
> > > wrote out?  Is there a way around this?
> > >
> > > Thanks,
> > > Jared
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Chang Song
In reply to this post by Jared Cantwell

We are in a bit similar situation.

3 node -> 5 node ensemble.

The only way to do this is the following.

Assumption is that we have one DNS hostname for three zookeeper ensemble IP.
Since five node ensemble allows 2 node failure for quorum, we can do


0. First all two new ensemble IPs

1. change all three existing node config (zoo.cfg) and add two new node information
    Restart all three existing nodes in a sequence.

2. Replicate the new existing zoo.cfg to two new ensemble
    Start Zookeeper on two new server


You can do this in backward sequence (0 -> 2 -> 1)
In your case, you can do 0 -> 2 -> 1, I think.

Chang



2011. 5. 5., 오후 9:57, Jared Cantwell 작성:

> It would be acceptable to me to do this non-dynamically and non-rolling as
> well. For example, I can shut down all nodes, make necessary modifications
> to config files, and then restart all nodes.  If I do this, should switching
> from standalone mode to multi-node mode work?  Has anyone done this before?
> Preliminary tests seem to work, but I haven't looked into all the race
> conditions and such yet.
>
> ~Jared
>
> On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <[hidden email]>wrote:
>
>> Hi Jared,
>>
>> Currently there is no support for adding and removing zookeeper nodes
>> dynamically. See:
>> https://issues.apache.org/jira/browse/ZOOKEEPER-107
>>
>> We're currently working to add this feature. However, AFAIK there is no
>> intention to support
>> transformation between standalone and multi-node modes, only membership
>> changes in multi-node mode.
>>
>> Regards,
>> Alex
>>
>>
>>> -----Original Message-----
>>> From: Jared Cantwell [mailto:[hidden email]]
>>> Sent: Wednesday, May 04, 2011 7:17 PM
>>> To: [hidden email]
>>> Subject: Growing a cluster
>>>
>>> Hello,
>>>
>>> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
>>> possible,
>>> and what would be the recommended way?
>>>
>>> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
>>> and it
>>> seems to be working actually.  But I'm not sure if this is supported,
>>> mostly
>>> because in standalone mode the on-disk files are different than they
>>> are in
>>> a multi-node configurations.  Mutli-node configurations embed the
>>> quorum
>>> incarnation into the filename, which standalone does not.  Should a
>>> quorum
>>> node be able to startup using snapshots and logs that a standalone node
>>> wrote out?  Is there a way around this?
>>>
>>> Thanks,
>>> Jared
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Jared Cantwell
Great information guys-- this helps me understand what needs done when
expanding from 2 nodes to X nodes.

Does anyone have insight on going from 1 node to 3 nodes?

~Jared

2011/5/5 Chang Song <[hidden email]>

>
> We are in a bit similar situation.
>
> 3 node -> 5 node ensemble.
>
> The only way to do this is the following.
>
> Assumption is that we have one DNS hostname for three zookeeper ensemble
> IP.
> Since five node ensemble allows 2 node failure for quorum, we can do
>
>
> 0. First all two new ensemble IPs
>
> 1. change all three existing node config (zoo.cfg) and add two new node
> information
>    Restart all three existing nodes in a sequence.
>
> 2. Replicate the new existing zoo.cfg to two new ensemble
>    Start Zookeeper on two new server
>
>
> You can do this in backward sequence (0 -> 2 -> 1)
> In your case, you can do 0 -> 2 -> 1, I think.
>
> Chang
>
>
>
> 2011. 5. 5., 오후 9:57, Jared Cantwell 작성:
>
> > It would be acceptable to me to do this non-dynamically and non-rolling
> as
> > well. For example, I can shut down all nodes, make necessary
> modifications
> > to config files, and then restart all nodes.  If I do this, should
> switching
> > from standalone mode to multi-node mode work?  Has anyone done this
> before?
> > Preliminary tests seem to work, but I haven't looked into all the race
> > conditions and such yet.
> >
> > ~Jared
> >
> > On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <[hidden email]
> >wrote:
> >
> >> Hi Jared,
> >>
> >> Currently there is no support for adding and removing zookeeper nodes
> >> dynamically. See:
> >> https://issues.apache.org/jira/browse/ZOOKEEPER-107
> >>
> >> We're currently working to add this feature. However, AFAIK there is no
> >> intention to support
> >> transformation between standalone and multi-node modes, only membership
> >> changes in multi-node mode.
> >>
> >> Regards,
> >> Alex
> >>
> >>
> >>> -----Original Message-----
> >>> From: Jared Cantwell [mailto:[hidden email]]
> >>> Sent: Wednesday, May 04, 2011 7:17 PM
> >>> To: [hidden email]
> >>> Subject: Growing a cluster
> >>>
> >>> Hello,
> >>>
> >>> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> >>> possible,
> >>> and what would be the recommended way?
> >>>
> >>> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> >>> and it
> >>> seems to be working actually.  But I'm not sure if this is supported,
> >>> mostly
> >>> because in standalone mode the on-disk files are different than they
> >>> are in
> >>> a multi-node configurations.  Mutli-node configurations embed the
> >>> quorum
> >>> incarnation into the filename, which standalone does not.  Should a
> >>> quorum
> >>> node be able to startup using snapshots and logs that a standalone node
> >>> wrote out?  Is there a way around this?
> >>>
> >>> Thanks,
> >>> Jared
> >>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Patrick Hunt
Hi Jared, you can go from 1 to 2+ nodes just fine:

1) change the configuration of server 1 from standalone to quorum
based (be sure to list the new members of the ensemble)
2) add a myid file for server 1
3) setup new servers 2+
4) restart server 1
5) start servers 2+

At this point the quorum should just come up. I just tried this
example and it worked fine for me going from 1 to 3 (the data I
created in standalone was available once I started the ensemble).

Regards,

Patrick

2011/5/5 Jared Cantwell <[hidden email]>:

> Great information guys-- this helps me understand what needs done when
> expanding from 2 nodes to X nodes.
>
> Does anyone have insight on going from 1 node to 3 nodes?
>
> ~Jared
>
> 2011/5/5 Chang Song <[hidden email]>
>
>>
>> We are in a bit similar situation.
>>
>> 3 node -> 5 node ensemble.
>>
>> The only way to do this is the following.
>>
>> Assumption is that we have one DNS hostname for three zookeeper ensemble
>> IP.
>> Since five node ensemble allows 2 node failure for quorum, we can do
>>
>>
>> 0. First all two new ensemble IPs
>>
>> 1. change all three existing node config (zoo.cfg) and add two new node
>> information
>>    Restart all three existing nodes in a sequence.
>>
>> 2. Replicate the new existing zoo.cfg to two new ensemble
>>    Start Zookeeper on two new server
>>
>>
>> You can do this in backward sequence (0 -> 2 -> 1)
>> In your case, you can do 0 -> 2 -> 1, I think.
>>
>> Chang
>>
>>
>>
>> 2011. 5. 5., 오후 9:57, Jared Cantwell 작성:
>>
>> > It would be acceptable to me to do this non-dynamically and non-rolling
>> as
>> > well. For example, I can shut down all nodes, make necessary
>> modifications
>> > to config files, and then restart all nodes.  If I do this, should
>> switching
>> > from standalone mode to multi-node mode work?  Has anyone done this
>> before?
>> > Preliminary tests seem to work, but I haven't looked into all the race
>> > conditions and such yet.
>> >
>> > ~Jared
>> >
>> > On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <[hidden email]
>> >wrote:
>> >
>> >> Hi Jared,
>> >>
>> >> Currently there is no support for adding and removing zookeeper nodes
>> >> dynamically. See:
>> >> https://issues.apache.org/jira/browse/ZOOKEEPER-107
>> >>
>> >> We're currently working to add this feature. However, AFAIK there is no
>> >> intention to support
>> >> transformation between standalone and multi-node modes, only membership
>> >> changes in multi-node mode.
>> >>
>> >> Regards,
>> >> Alex
>> >>
>> >>
>> >>> -----Original Message-----
>> >>> From: Jared Cantwell [mailto:[hidden email]]
>> >>> Sent: Wednesday, May 04, 2011 7:17 PM
>> >>> To: [hidden email]
>> >>> Subject: Growing a cluster
>> >>>
>> >>> Hello,
>> >>>
>> >>> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
>> >>> possible,
>> >>> and what would be the recommended way?
>> >>>
>> >>> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
>> >>> and it
>> >>> seems to be working actually.  But I'm not sure if this is supported,
>> >>> mostly
>> >>> because in standalone mode the on-disk files are different than they
>> >>> are in
>> >>> a multi-node configurations.  Mutli-node configurations embed the
>> >>> quorum
>> >>> incarnation into the filename, which standalone does not.  Should a
>> >>> quorum
>> >>> node be able to startup using snapshots and logs that a standalone node
>> >>> wrote out?  Is there a way around this?
>> >>>
>> >>> Thanks,
>> >>> Jared
>> >>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Jeremy Stribling
In reply to this post by Alexander Shraer
On 05/04/2011 09:02 PM, Alexander Shraer wrote:
> Hi Jared,
>
> Currently there is no support for adding and removing zookeeper nodes dynamically. See:
> https://issues.apache.org/jira/browse/ZOOKEEPER-107
>
> We're currently working to add this feature. However, AFAIK there is no intention to support
> transformation between standalone and multi-node modes, only membership changes in multi-node mode.
>    

This feature is going to be great to have, thanks for working on it!  
However, can someone help me understand why we have standalone mode at
all?  It seems like, in theory, starting a quorum with just a single
node should work just fine and simply act as if it were in standalone
mode.  (I've used Paxos implementations in the past that work this
way.)  What's the catch that prevents Zookeeper from doing this?

Thanks,

Jeremy

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Jared Cantwell
In reply to this post by Patrick Hunt
Thats Patrick, this is roughly the procedure we have are testing.

In your procedure, can (4) and (5) be performed simultaneously?

Also, am I correct in thinking that going from 1 to 3 server could be
dangerous for this reason:
1. start server 1 in standalone mode and write some data to it
2. follow you procedure with 3 servers
3. say during leader election, servers 2 and 3 form the quorum first and
elect server 2 as the leader.
4. server 2 believes the last txn to be 0, since it is a fresh cluster
5. now server 1 joins the quorum and knows about txn 100, but the other 2
servers do not, uh oh

Is this a real problem, or is there something preventing this case?  It
seems like you can only grow a cluster such that at least on member of the
old quorum MUST be present in the new quorum.  Is this correct?

~Jared

2011/5/5 Patrick Hunt <[hidden email]>

> Hi Jared, you can go from 1 to 2+ nodes just fine:
>
> 1) change the configuration of server 1 from standalone to quorum
> based (be sure to list the new members of the ensemble)
> 2) add a myid file for server 1
> 3) setup new servers 2+
> 4) restart server 1
> 5) start servers 2+
>
> At this point the quorum should just come up. I just tried this
> example and it worked fine for me going from 1 to 3 (the data I
> created in standalone was available once I started the ensemble).
>
> Regards,
>
> Patrick
>
> 2011/5/5 Jared Cantwell <[hidden email]>:
> > Great information guys-- this helps me understand what needs done when
> > expanding from 2 nodes to X nodes.
> >
> > Does anyone have insight on going from 1 node to 3 nodes?
> >
> > ~Jared
> >
> > 2011/5/5 Chang Song <[hidden email]>
> >
> >>
> >> We are in a bit similar situation.
> >>
> >> 3 node -> 5 node ensemble.
> >>
> >> The only way to do this is the following.
> >>
> >> Assumption is that we have one DNS hostname for three zookeeper ensemble
> >> IP.
> >> Since five node ensemble allows 2 node failure for quorum, we can do
> >>
> >>
> >> 0. First all two new ensemble IPs
> >>
> >> 1. change all three existing node config (zoo.cfg) and add two new node
> >> information
> >>    Restart all three existing nodes in a sequence.
> >>
> >> 2. Replicate the new existing zoo.cfg to two new ensemble
> >>    Start Zookeeper on two new server
> >>
> >>
> >> You can do this in backward sequence (0 -> 2 -> 1)
> >> In your case, you can do 0 -> 2 -> 1, I think.
> >>
> >> Chang
> >>
> >>
> >>
> >> 2011. 5. 5., 오후 9:57, Jared Cantwell 작성:
> >>
> >> > It would be acceptable to me to do this non-dynamically and
> non-rolling
> >> as
> >> > well. For example, I can shut down all nodes, make necessary
> >> modifications
> >> > to config files, and then restart all nodes.  If I do this, should
> >> switching
> >> > from standalone mode to multi-node mode work?  Has anyone done this
> >> before?
> >> > Preliminary tests seem to work, but I haven't looked into all the race
> >> > conditions and such yet.
> >> >
> >> > ~Jared
> >> >
> >> > On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <
> [hidden email]
> >> >wrote:
> >> >
> >> >> Hi Jared,
> >> >>
> >> >> Currently there is no support for adding and removing zookeeper nodes
> >> >> dynamically. See:
> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-107
> >> >>
> >> >> We're currently working to add this feature. However, AFAIK there is
> no
> >> >> intention to support
> >> >> transformation between standalone and multi-node modes, only
> membership
> >> >> changes in multi-node mode.
> >> >>
> >> >> Regards,
> >> >> Alex
> >> >>
> >> >>
> >> >>> -----Original Message-----
> >> >>> From: Jared Cantwell [mailto:[hidden email]]
> >> >>> Sent: Wednesday, May 04, 2011 7:17 PM
> >> >>> To: [hidden email]
> >> >>> Subject: Growing a cluster
> >> >>>
> >> >>> Hello,
> >> >>>
> >> >>> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> >> >>> possible,
> >> >>> and what would be the recommended way?
> >> >>>
> >> >>> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> >> >>> and it
> >> >>> seems to be working actually.  But I'm not sure if this is
> supported,
> >> >>> mostly
> >> >>> because in standalone mode the on-disk files are different than they
> >> >>> are in
> >> >>> a multi-node configurations.  Mutli-node configurations embed the
> >> >>> quorum
> >> >>> incarnation into the filename, which standalone does not.  Should a
> >> >>> quorum
> >> >>> node be able to startup using snapshots and logs that a standalone
> node
> >> >>> wrote out?  Is there a way around this?
> >> >>>
> >> >>> Thanks,
> >> >>> Jared
> >> >>
> >>
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Ted Dunning
In reply to this post by Patrick Hunt
To be explicit on a (probably) obvious point, you will have a few tens of
seconds of downtime during the restart
of server 1 until the second server comes on-line and participates in the
election.

It there is a difference between 10 seconds of downtime and 2 minutes of
downtime in your application, I recommend
going from 1 to 3 nodes first then to 3+.  It is easier to have a very short
downtime when only a single additional node
needs to come up.  You might also be able to start server 2 before you
bounce server 1, but you have to be very careful
not to start enough machines to have a quorum before server 1 joins in.

2011/5/5 Patrick Hunt <[hidden email]>

> Hi Jared, you can go from 1 to 2+ nodes just fine:
>
> 1) change the configuration of server 1 from standalone to quorum
> based (be sure to list the new members of the ensemble)
> 2) add a myid file for server 1
> 3) setup new servers 2+
> 4) restart server 1
> 5) start servers 2+
>
> At this point the quorum should just come up. I just tried this
> example and it worked fine for me going from 1 to 3 (the data I
> created in standalone was available once I started the ensemble).
>
> Regards,
>
> Patrick
>
> 2011/5/5 Jared Cantwell <[hidden email]>:
> > Great information guys-- this helps me understand what needs done when
> > expanding from 2 nodes to X nodes.
> >
> > Does anyone have insight on going from 1 node to 3 nodes?
> >
> > ~Jared
> >
> > 2011/5/5 Chang Song <[hidden email]>
> >
> >>
> >> We are in a bit similar situation.
> >>
> >> 3 node -> 5 node ensemble.
> >>
> >> The only way to do this is the following.
> >>
> >> Assumption is that we have one DNS hostname for three zookeeper ensemble
> >> IP.
> >> Since five node ensemble allows 2 node failure for quorum, we can do
> >>
> >>
> >> 0. First all two new ensemble IPs
> >>
> >> 1. change all three existing node config (zoo.cfg) and add two new node
> >> information
> >>    Restart all three existing nodes in a sequence.
> >>
> >> 2. Replicate the new existing zoo.cfg to two new ensemble
> >>    Start Zookeeper on two new server
> >>
> >>
> >> You can do this in backward sequence (0 -> 2 -> 1)
> >> In your case, you can do 0 -> 2 -> 1, I think.
> >>
> >> Chang
> >>
> >>
> >>
> >> 2011. 5. 5., 오후 9:57, Jared Cantwell 작성:
> >>
> >> > It would be acceptable to me to do this non-dynamically and
> non-rolling
> >> as
> >> > well. For example, I can shut down all nodes, make necessary
> >> modifications
> >> > to config files, and then restart all nodes.  If I do this, should
> >> switching
> >> > from standalone mode to multi-node mode work?  Has anyone done this
> >> before?
> >> > Preliminary tests seem to work, but I haven't looked into all the race
> >> > conditions and such yet.
> >> >
> >> > ~Jared
> >> >
> >> > On Thu, May 5, 2011 at 12:02 AM, Alexander Shraer <
> [hidden email]
> >> >wrote:
> >> >
> >> >> Hi Jared,
> >> >>
> >> >> Currently there is no support for adding and removing zookeeper nodes
> >> >> dynamically. See:
> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-107
> >> >>
> >> >> We're currently working to add this feature. However, AFAIK there is
> no
> >> >> intention to support
> >> >> transformation between standalone and multi-node modes, only
> membership
> >> >> changes in multi-node mode.
> >> >>
> >> >> Regards,
> >> >> Alex
> >> >>
> >> >>
> >> >>> -----Original Message-----
> >> >>> From: Jared Cantwell [mailto:[hidden email]]
> >> >>> Sent: Wednesday, May 04, 2011 7:17 PM
> >> >>> To: [hidden email]
> >> >>> Subject: Growing a cluster
> >> >>>
> >> >>> Hello,
> >> >>>
> >> >>> Say I was going to grow a cluster from 1 node to 3 nodes.  Is this
> >> >>> possible,
> >> >>> and what would be the recommended way?
> >> >>>
> >> >>> At first I was thinking that I could go from 1 to 2 and then 2 to 3,
> >> >>> and it
> >> >>> seems to be working actually.  But I'm not sure if this is
> supported,
> >> >>> mostly
> >> >>> because in standalone mode the on-disk files are different than they
> >> >>> are in
> >> >>> a multi-node configurations.  Mutli-node configurations embed the
> >> >>> quorum
> >> >>> incarnation into the filename, which standalone does not.  Should a
> >> >>> quorum
> >> >>> node be able to startup using snapshots and logs that a standalone
> node
> >> >>> wrote out?  Is there a way around this?
> >> >>>
> >> >>> Thanks,
> >> >>> Jared
> >> >>
> >>
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Ted Dunning
In reply to this post by Jared Cantwell
yes.  it is  a real problem.

and yes, the old node(s) must have representation in the new quorum.

2011/5/5 Jared Cantwell <[hidden email]>

> Is this a real problem, or is there something preventing this case?  It
> seems like you can only grow a cluster such that at least on member of the
> old quorum MUST be present in the new quorum.  Is this correct?
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Jared Cantwell
Thanks Ted.  So based on this it seems like you'd have to go from 1 to 2
servers, and then from 2 to 3 servers, 3 to 5, 5 to 9, etc., right?

This is what we have designed and are testing, although it doesn't seem to
always be so smooth.  Sometimes the servers seem to get "stuck" on the
restart-- we're trying to collect more details.

~Jared

On Thu, May 5, 2011 at 1:07 PM, Ted Dunning <[hidden email]> wrote:

> yes.  it is  a real problem.
>
> and yes, the old node(s) must have representation in the new quorum.
>
> 2011/5/5 Jared Cantwell <[hidden email]>
>
> > Is this a real problem, or is there something preventing this case?  It
> > seems like you can only grow a cluster such that at least on member of
> the
> > old quorum MUST be present in the new quorum.  Is this correct?
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Ted Dunning
On Thu, May 5, 2011 at 10:12 AM, Jared Cantwell <[hidden email]>wrote:

> Thanks Ted.  So based on this it seems like you'd have to go from 1 to 2
> servers, and then from 2 to 3 servers, 3 to 5, 5 to 9, etc., right?
>

No.  I would avoid even numbers.  I would go 1 to 3 (but one is missing
initially).  Counting
active servers would give the sequence you say, but the configurations would
say first 1, then
3 and conceptually, you are moving from one server to three with no
intermediate configuraiton.


>
> This is what we have designed and are testing, although it doesn't seem to
> always be so smooth.  Sometimes the servers seem to get "stuck" on the
> restart-- we're trying to collect more details.
>

Even number cluster sizes are evil.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

michael.boom
Is there any update on this thread at this moment? I saw that the reffered JIRA issue was added already in 3.5

I have a single zookeeper instance and i would like to move it to another server and in the same time adding 2 more instances, in a quorum.
Can this be done dynamically?

Thank you!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Camille Fournier
Are you running a snapshot version of 3.5? Or is this for some future use
case?

Here is the documentation for using dynamic reconfig:
https://docs.google.com/document/d/1AF8pIfQbN5cKxe0c4cQ4_DW6ZjBJqSkyANcTGUwkzjc/edit

Unless you're using a snapshot of 3.5, I think you might be better just
doing a rolling restart.

C
On Tue, Nov 5, 2013 at 9:20 AM, michael.boom <[hidden email]> wrote:

> Is there any update on this thread at this moment? I saw that the reffered
> JIRA issue was added already in 3.5
>
> I have a single zookeeper instance and i would like to move it to another
> server and in the same time adding 2 more instances, in a quorum.
> Can this be done dynamically?
>
> Thank you!
>
>
>
> --
> View this message in context:
> http://zookeeper-user.578899.n2.nabble.com/Growing-a-cluster-tp6332959p7579238.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Growing a cluster

Alexander Shraer-2
Dynamic reconfiguration from/to an ensemble containing just one server
is being developed in this JIRA:
https://issues.apache.org/jira/browse/ZOOKEEPER-1691
but its not completely finished yet.

Alex

On Tue, Nov 5, 2013 at 7:44 AM, Camille Fournier <[hidden email]> wrote:

> Are you running a snapshot version of 3.5? Or is this for some future use
> case?
>
> Here is the documentation for using dynamic reconfig:
> https://docs.google.com/document/d/1AF8pIfQbN5cKxe0c4cQ4_DW6ZjBJqSkyANcTGUwkzjc/edit
>
> Unless you're using a snapshot of 3.5, I think you might be better just
> doing a rolling restart.
>
> C
> On Tue, Nov 5, 2013 at 9:20 AM, michael.boom <[hidden email]> wrote:
>
>> Is there any update on this thread at this moment? I saw that the reffered
>> JIRA issue was added already in 3.5
>>
>> I have a single zookeeper instance and i would like to move it to another
>> server and in the same time adding 2 more instances, in a quorum.
>> Can this be done dynamically?
>>
>> Thank you!
>>
>>
>>
>> --
>> View this message in context:
>> http://zookeeper-user.578899.n2.nabble.com/Growing-a-cluster-tp6332959p7579238.html
>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>
Loading...