Is the value of $MYID allowed to change across runs in an HA ZK deployment?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Is the value of $MYID allowed to change across runs in an HA ZK deployment?

jay.taylor
Greetings Zookeepers,

I'm investigating possible ways for Zookeeper to run safely on top of
Kubernetes clusters.

When the zookeeper containers come online, the value for $MYID is
initially derived from the Kubernetes pod name.  All active pod names
are guaranteed to be unique within the cluster at any given point in time.

Example values:

    - zookeeper-0
    - zookeeper-1
    - zookeeper-2

and the formula for $MYID is ((the trailing number of the pod name) + 1):

    - zookeeper-0 => $MYID=1
    - zookeeper-1 => $MYID=2
    - zookeeper-2 => $MYID=3

The part I'm uncertain of is the relationship between $MYID and ensuring
each zookeeper data set stays in sync with the rest of the cluster,
particularly across container restarts.  Restarts can lead to Zookeeper
data set being launched with a different value of $MYID compared with
the previous run.  I.e., Zookeeper may have already run on any given
data set in the past when the myid file contained a different value.

Is it part of the mechanism used to ensure all follower members are in
sync with the current leader?  It seems to me that if the leader (or
followers) keep track of their peers via myid and it gets changed, there
could be problems.

Initial testing (without much load) has gone fine and things seem to
work fine when launched with updated $MYID values.  I've also been
perusing the ZK source code and inspecting how myid is used, and nothing
stood out to indicate that this will lead to future problems.  However,
experience dictates that with distributed systems the devil is often in
nuanced details, so I'm hoping the experts out there may be able to shed
light about the internal dependencies on the value of myid.

Specific questions:

    - Is myid relied on to never change, or does it only need to be
unique within the cluster at any given time?

    - What are the risks with changing myid in relation to ZK data set
directories across runs?

Your insights will be greatly appreciated!

Kind regards,
Jay Taylor


Reply | Threaded
Open this post in threaded view
|

Re: Is the value of $MYID allowed to change across runs in an HA ZK deployment?

Alexander Shraer-2
Hi Jay,

Perhaps it also depends on the restart? if the restart is done gradually,
for example a leader is in the middle of collecting votes when one of the
voters gets a new id and votes twice instead of once ? If the restart is a
barrier, where all servers are shut down and then restarted, this shouldn't
happen.

In 3.5, cluster membership is written into the ZK database as well as
configuration files, and contains server ids and parameters (ports, IPs,
etc). If ids change, it sounds like the membership
information may be wrong.

Perhaps there are also some implications on the security-related configs ?
Someone else may want to comment on these.

In general, changing ids doesn't feel like a very safe method to me...

Cheers,
Alex




On Mon, Feb 5, 2018 at 10:47 AM, <[hidden email]> wrote:

> Greetings Zookeepers,
>
> I'm investigating possible ways for Zookeeper to run safely on top of
> Kubernetes clusters.
>
> When the zookeeper containers come online, the value for $MYID is
> initially derived from the Kubernetes pod name.  All active pod names
> are guaranteed to be unique within the cluster at any given point in time.
>
> Example values:
>
>     - zookeeper-0
>     - zookeeper-1
>     - zookeeper-2
>
> and the formula for $MYID is ((the trailing number of the pod name) + 1):
>
>     - zookeeper-0 => $MYID=1
>     - zookeeper-1 => $MYID=2
>     - zookeeper-2 => $MYID=3
>
> The part I'm uncertain of is the relationship between $MYID and ensuring
> each zookeeper data set stays in sync with the rest of the cluster,
> particularly across container restarts.  Restarts can lead to Zookeeper
> data set being launched with a different value of $MYID compared with
> the previous run.  I.e., Zookeeper may have already run on any given
> data set in the past when the myid file contained a different value.
>
> Is it part of the mechanism used to ensure all follower members are in
> sync with the current leader?  It seems to me that if the leader (or
> followers) keep track of their peers via myid and it gets changed, there
> could be problems.
>
> Initial testing (without much load) has gone fine and things seem to
> work fine when launched with updated $MYID values.  I've also been
> perusing the ZK source code and inspecting how myid is used, and nothing
> stood out to indicate that this will lead to future problems.  However,
> experience dictates that with distributed systems the devil is often in
> nuanced details, so I'm hoping the experts out there may be able to shed
> light about the internal dependencies on the value of myid.
>
> Specific questions:
>
>     - Is myid relied on to never change, or does it only need to be
> unique within the cluster at any given time?
>
>     - What are the risks with changing myid in relation to ZK data set
> directories across runs?
>
> Your insights will be greatly appreciated!
>
> Kind regards,
> Jay Taylor
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Is the value of $MYID allowed to change across runs in an HA ZK deployment?

Dan Benediktson
One concern I had was around session IDs. Uniqueness of session IDs is
predicated on uniqueness of Server IDs, and from personal experience,
accidentally losing uniqueness of session IDs due to having duplicate
server IDs leads to very weird behaviors.

If you can guarantee server IDs are unique for all time, you're definitely
fine on this front, but if your server IDs are only unique at a point in
time, then there is some extra checking you need to do (probably around
system clocks, if I remember correctly?) to ensure it is safe on this
front. It's *probably* always going to be safe even without that check, but
*probably* isn't what you usually want to hear out of your strongly
consistent database...

Dan

On Mon, Feb 5, 2018 at 10:22 PM, Alexander Shraer <[hidden email]> wrote:

> Hi Jay,
>
> Perhaps it also depends on the restart? if the restart is done gradually,
> for example a leader is in the middle of collecting votes when one of the
> voters gets a new id and votes twice instead of once ? If the restart is a
> barrier, where all servers are shut down and then restarted, this shouldn't
> happen.
>
> In 3.5, cluster membership is written into the ZK database as well as
> configuration files, and contains server ids and parameters (ports, IPs,
> etc). If ids change, it sounds like the membership
> information may be wrong.
>
> Perhaps there are also some implications on the security-related configs ?
> Someone else may want to comment on these.
>
> In general, changing ids doesn't feel like a very safe method to me...
>
> Cheers,
> Alex
>
>
>
>
> On Mon, Feb 5, 2018 at 10:47 AM, <[hidden email]> wrote:
>
> > Greetings Zookeepers,
> >
> > I'm investigating possible ways for Zookeeper to run safely on top of
> > Kubernetes clusters.
> >
> > When the zookeeper containers come online, the value for $MYID is
> > initially derived from the Kubernetes pod name.  All active pod names
> > are guaranteed to be unique within the cluster at any given point in
> time.
> >
> > Example values:
> >
> >     - zookeeper-0
> >     - zookeeper-1
> >     - zookeeper-2
> >
> > and the formula for $MYID is ((the trailing number of the pod name) + 1):
> >
> >     - zookeeper-0 => $MYID=1
> >     - zookeeper-1 => $MYID=2
> >     - zookeeper-2 => $MYID=3
> >
> > The part I'm uncertain of is the relationship between $MYID and ensuring
> > each zookeeper data set stays in sync with the rest of the cluster,
> > particularly across container restarts.  Restarts can lead to Zookeeper
> > data set being launched with a different value of $MYID compared with
> > the previous run.  I.e., Zookeeper may have already run on any given
> > data set in the past when the myid file contained a different value.
> >
> > Is it part of the mechanism used to ensure all follower members are in
> > sync with the current leader?  It seems to me that if the leader (or
> > followers) keep track of their peers via myid and it gets changed, there
> > could be problems.
> >
> > Initial testing (without much load) has gone fine and things seem to
> > work fine when launched with updated $MYID values.  I've also been
> > perusing the ZK source code and inspecting how myid is used, and nothing
> > stood out to indicate that this will lead to future problems.  However,
> > experience dictates that with distributed systems the devil is often in
> > nuanced details, so I'm hoping the experts out there may be able to shed
> > light about the internal dependencies on the value of myid.
> >
> > Specific questions:
> >
> >     - Is myid relied on to never change, or does it only need to be
> > unique within the cluster at any given time?
> >
> >     - What are the risks with changing myid in relation to ZK data set
> > directories across runs?
> >
> > Your insights will be greatly appreciated!
> >
> > Kind regards,
> > Jay Taylor
> >
> >
> >
>