Observers taking long time to serve requests

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Observers taking long time to serve requests

rammohan ganapavarapu
Hi,

We have a zookeeper cluster in multi region, we have leader in US EAST  and
observers in US WEST, our data size is 1.3GB,network latency between leader
and observers is around 250ms, upon observer restart it is taking like
close to 30m for observer to serve traffic. Not sure why it's taking so
long to serve request. Does any one see such issue? any idea why it's
taking time?

Thanks,
Ram
Reply | Threaded
Open this post in threaded view
|

Re: Observers taking long time to serve requests

Ben Sherman-2
With latency that high, I'd expect the thruput to be low, is 30m in line
with what it takes to transfer a gig of data across that pipe?

On Mon, May 15, 2017 at 4:19 PM, rammohan ganapavarapu <
[hidden email]> wrote:

> Hi,
>
> We have a zookeeper cluster in multi region, we have leader in US EAST  and
> observers in US WEST, our data size is 1.3GB,network latency between leader
> and observers is around 250ms, upon observer restart it is taking like
> close to 30m for observer to serve traffic. Not sure why it's taking so
> long to serve request. Does any one see such issue? any idea why it's
> taking time?
>
> Thanks,
> Ram
>
Reply | Threaded
Open this post in threaded view
|

Re: Observers taking long time to serve requests

hanm
When an observer (and in general a follower) restarted, it will go through
these stages:

* Look for leader by starting a new leader election round and usually this
is quick as there is already a leader.
* Register with leader and begin synchronize phase - depends on the
observer state the sync might end up with DIFF (quicker) or SNAPSHOT
(longer).
* After sync phase observer will start serving traffic.

Check the observer server log and record how long it took for each stage is
a good starting point..


On Tue, May 16, 2017 at 1:04 PM, Ben Sherman <[hidden email]> wrote:

> With latency that high, I'd expect the thruput to be low, is 30m in line
> with what it takes to transfer a gig of data across that pipe?
>
> On Mon, May 15, 2017 at 4:19 PM, rammohan ganapavarapu <
> [hidden email]> wrote:
>
> > Hi,
> >
> > We have a zookeeper cluster in multi region, we have leader in US EAST
> and
> > observers in US WEST, our data size is 1.3GB,network latency between
> leader
> > and observers is around 250ms, upon observer restart it is taking like
> > close to 30m for observer to serve traffic. Not sure why it's taking so
> > long to serve request. Does any one see such issue? any idea why it's
> > taking time?
> >
> > Thanks,
> > Ram
> >
>



--
Cheers
Michael.
Reply | Threaded
Open this post in threaded view
|

Re: Observers taking long time to serve requests

Mike Richardson
Unsubscribe


Mike Richardson

Senior Software Engineer



MoTuM N.V. | Dellingstraat 34 | B-2800 MECHELEN | Belgium


T +32(0)15 28 16 63
M +41 7943 69538


www.motum.be

On 16 May 2017 at 23:07, Michael Han <[hidden email]> wrote:
When an observer (and in general a follower) restarted, it will go through
these stages:

* Look for leader by starting a new leader election round and usually this
is quick as there is already a leader.
* Register with leader and begin synchronize phase - depends on the
observer state the sync might end up with DIFF (quicker) or SNAPSHOT
(longer).
* After sync phase observer will start serving traffic.

Check the observer server log and record how long it took for each stage is
a good starting point..


On Tue, May 16, 2017 at 1:04 PM, Ben Sherman <[hidden email]> wrote:

> With latency that high, I'd expect the thruput to be low, is 30m in line
> with what it takes to transfer a gig of data across that pipe?
>
> On Mon, May 15, 2017 at 4:19 PM, rammohan ganapavarapu <
> [hidden email]> wrote:
>
> > Hi,
> >
> > We have a zookeeper cluster in multi region, we have leader in US EAST
> and
> > observers in US WEST, our data size is 1.3GB,network latency between
> leader
> > and observers is around 250ms, upon observer restart it is taking like
> > close to 30m for observer to serve traffic. Not sure why it's taking so
> > long to serve request. Does any one see such issue? any idea why it's
> > taking time?
> >
> > Thanks,
> > Ram
> >
>



--
Cheers
Michael.

Reply | Threaded
Open this post in threaded view
|

Re: Observers taking long time to serve requests

rammohan ganapavarapu
In reply to this post by hanm
Michael,

Thanks for your suggestions, so do you know how to find when it is taking
diff or full snapshot? in my case it looks like taking full snapshot. so
from observing leader to snapshotting is taking 30mins.

Ram

On May 16, 2017 2:07 PM, "Michael Han" <[hidden email]> wrote:

> When an observer (and in general a follower) restarted, it will go through
> these stages:
>
> * Look for leader by starting a new leader election round and usually this
> is quick as there is already a leader.
> * Register with leader and begin synchronize phase - depends on the
> observer state the sync might end up with DIFF (quicker) or SNAPSHOT
> (longer).
> * After sync phase observer will start serving traffic.
>
> Check the observer server log and record how long it took for each stage is
> a good starting point..
>
>
> On Tue, May 16, 2017 at 1:04 PM, Ben Sherman <[hidden email]> wrote:
>
> > With latency that high, I'd expect the thruput to be low, is 30m in line
> > with what it takes to transfer a gig of data across that pipe?
> >
> > On Mon, May 15, 2017 at 4:19 PM, rammohan ganapavarapu <
> > [hidden email]> wrote:
> >
> > > Hi,
> > >
> > > We have a zookeeper cluster in multi region, we have leader in US EAST
> > and
> > > observers in US WEST, our data size is 1.3GB,network latency between
> > leader
> > > and observers is around 250ms, upon observer restart it is taking like
> > > close to 30m for observer to serve traffic. Not sure why it's taking so
> > > long to serve request. Does any one see such issue? any idea why it's
> > > taking time?
> > >
> > > Thanks,
> > > Ram
> > >
> >
>
>
>
> --
> Cheers
> Michael.
>
Reply | Threaded
Open this post in threaded view
|

Re: Observers taking long time to serve requests

hanm
Observer server log should contain INFO level logging on whether it's
getting a diff or getting a snapshot from leader. To estimate how long the
data transfer took, look for "UPTODATE" INFO level log on observer. This is
the message logged after data sync between leader and observer finished.

On Wed, May 17, 2017 at 3:59 PM, rammohan ganapavarapu <
[hidden email]> wrote:

> Michael,
>
> Thanks for your suggestions, so do you know how to find when it is taking
> diff or full snapshot? in my case it looks like taking full snapshot. so
> from observing leader to snapshotting is taking 30mins.
>
> Ram
>
> On May 16, 2017 2:07 PM, "Michael Han" <[hidden email]> wrote:
>
> > When an observer (and in general a follower) restarted, it will go
> through
> > these stages:
> >
> > * Look for leader by starting a new leader election round and usually
> this
> > is quick as there is already a leader.
> > * Register with leader and begin synchronize phase - depends on the
> > observer state the sync might end up with DIFF (quicker) or SNAPSHOT
> > (longer).
> > * After sync phase observer will start serving traffic.
> >
> > Check the observer server log and record how long it took for each stage
> is
> > a good starting point..
> >
> >
> > On Tue, May 16, 2017 at 1:04 PM, Ben Sherman <[hidden email]>
> wrote:
> >
> > > With latency that high, I'd expect the thruput to be low, is 30m in
> line
> > > with what it takes to transfer a gig of data across that pipe?
> > >
> > > On Mon, May 15, 2017 at 4:19 PM, rammohan ganapavarapu <
> > > [hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have a zookeeper cluster in multi region, we have leader in US
> EAST
> > > and
> > > > observers in US WEST, our data size is 1.3GB,network latency between
> > > leader
> > > > and observers is around 250ms, upon observer restart it is taking
> like
> > > > close to 30m for observer to serve traffic. Not sure why it's taking
> so
> > > > long to serve request. Does any one see such issue? any idea why it's
> > > > taking time?
> > > >
> > > > Thanks,
> > > > Ram
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>



--
Cheers
Michael.