Quantcast

How to modify Client Connection timer

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to modify Client Connection timer

Ray Chaudhuri, Shirsha (Nokia - IN/Bangalore)
Hi,

We are using ZKFC  as a Zookeeper Client that tries to connect to Zookeeper Server at the time of bringup, when we sometimes encounter the following issue _

1.       Client retrieves the ZK Ensemble address (ensemble consists of 3 nodes)

2.       Client tries to connect to one of the ZK Server nodes

3.       However due to other on-going processing (required at the time of bringup) at the ZK Server Node, it fails to respond in time to the Client. The response goes after 3 seconds

4.       The Client side times out by then, typically seen around 1.6 sec

Is there a way this timer at the Client side can be reconfigured to a higher value?

Regards
Shirsha
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to modify Client Connection timer

Patrick Hunt
You should be able to control that by increasing the session timeout. I'm
not familiar with the client code you are using however. Additionally the
ZK client should retry it's connection (as long as you don't close the ZK
object) continually until it is able to reconnect.

Patrick

On Mon, Mar 20, 2017 at 3:22 PM, Ray Chaudhuri, Shirsha (Nokia -
IN/Bangalore) <[hidden email]> wrote:

> Hi,
>
> We are using ZKFC  as a Zookeeper Client that tries to connect to
> Zookeeper Server at the time of bringup, when we sometimes encounter the
> following issue _
>
> 1.       Client retrieves the ZK Ensemble address (ensemble consists of 3
> nodes)
>
> 2.       Client tries to connect to one of the ZK Server nodes
>
> 3.       However due to other on-going processing (required at the time of
> bringup) at the ZK Server Node, it fails to respond in time to the Client.
> The response goes after 3 seconds
>
> 4.       The Client side times out by then, typically seen around 1.6 sec
>
> Is there a way this timer at the Client side can be reconfigured to a
> higher value?
>
> Regards
> Shirsha
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: How to modify Client Connection timer

Ray Chaudhuri, Shirsha (Nokia - IN/Bangalore)


Thanks Patrick.

Yes, we did eventually figure out that this timeout value is initialised from the Hadoop ZKFC code with the value of "ha.zookeeper.session-timeout.ms" or if this is not set, then the default is 5000. Since the client Cnxn class sets this timer value to a value _  (session timeout) / (number of ZK servers), the timer value for the initial connection to complete is one third the value of 5000. Hence the timeout occurring at around 1.6s each time.

Which brought us to the question that wouldn't this piece of code highlighted from the file zookeeper/ClientCnxn.java _
    public ClientCnxn(String chrootPath, HostProvider hostProvider, int sessionTimeout, ZooKeeper zooKeeper,


            ClientWatchManager watcher, ClientCnxnSocket clientCnxnSocket,


            long sessionId, byte[] sessionPasswd, boolean canBeReadOnly) {


        this.zooKeeper = zooKeeper;


        this.watcher = watcher;


        this.sessionId = sessionId;


        this.sessionPasswd = sessionPasswd;


        this.sessionTimeout = sessionTimeout;


        this.hostProvider = hostProvider;


        this.chrootPath = chrootPath;





        connectTimeout = sessionTimeout / hostProvider.size();


        readTimeout = sessionTimeout * 2 / 3;


        readOnly = canBeReadOnly;





        sendThread = new SendThread(clientCnxnSocket);


        eventThread = new EventThread();


        this.clientConfig=zooKeeper.getClientConfig();


    }




Be leading to trouble if the number of ZK Servers is larger?

This timeout value is initially used when the Client is waiting to connect with the server, even before it has negotiated a sessiontimeout value.

So larger the number of ZK Servers, less will be the timer value. Shouldn’t we have a lower guard for it and ensure that atleast a min value is used for this timer?



Regards

Shirsha



-----Original Message-----
From: Patrick Hunt [mailto:[hidden email]]
Sent: Wednesday, March 22, 2017 11:55 PM
To: UserZooKeeper <[hidden email]>
Subject: Re: How to modify Client Connection timer



You should be able to control that by increasing the session timeout. I'm not familiar with the client code you are using however. Additionally the ZK client should retry it's connection (as long as you don't close the ZK

object) continually until it is able to reconnect.



Patrick



On Mon, Mar 20, 2017 at 3:22 PM, Ray Chaudhuri, Shirsha (Nokia -

IN/Bangalore) <[hidden email]<mailto:[hidden email]>> wrote:



> Hi,

>

> We are using ZKFC  as a Zookeeper Client that tries to connect to

> Zookeeper Server at the time of bringup, when we sometimes encounter

> the following issue _

>

> 1.       Client retrieves the ZK Ensemble address (ensemble consists of 3

> nodes)

>

> 2.       Client tries to connect to one of the ZK Server nodes

>

> 3.       However due to other on-going processing (required at the time of

> bringup) at the ZK Server Node, it fails to respond in time to the Client.

> The response goes after 3 seconds

>

> 4.       The Client side times out by then, typically seen around 1.6 sec

>

> Is there a way this timer at the Client side can be reconfigured to a

> higher value?

>

> Regards

> Shirsha

>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to modify Client Connection timer

Patrick Hunt
On Wed, Mar 22, 2017 at 2:19 PM, Ray Chaudhuri, Shirsha (Nokia -
IN/Bangalore) <[hidden email]> wrote:

>
>
> Thanks Patrick.
>
> Yes, we did eventually figure out that this timeout value is initialised
> from the Hadoop ZKFC code with the value of "ha.zookeeper.session-timeout.
> ms" or if this is not set, then the default is 5000. Since the client
> Cnxn class sets this timer value to a value _  (session timeout) / (number
> of ZK servers), the timer value for the initial connection to complete is
> one third the value of 5000. Hence the timeout occurring at around 1.6s
> each time.
>
> Which brought us to the question that wouldn't this piece of code
> highlighted from the file zookeeper/ClientCnxn.java _
>     public ClientCnxn(String chrootPath, HostProvider hostProvider, int
> sessionTimeout, ZooKeeper zooKeeper,
>
>
>             ClientWatchManager watcher, ClientCnxnSocket clientCnxnSocket,
>
>
>             long sessionId, byte[] sessionPasswd, boolean canBeReadOnly) {
>
>
>         this.zooKeeper = zooKeeper;
>
>
>         this.watcher = watcher;
>
>
>         this.sessionId = sessionId;
>
>
>         this.sessionPasswd = sessionPasswd;
>
>
>         this.sessionTimeout = sessionTimeout;
>
>
>         this.hostProvider = hostProvider;
>
>
>         this.chrootPath = chrootPath;
>
>
>
>
>
>         connectTimeout = sessionTimeout / hostProvider.size();
>
>
>         readTimeout = sessionTimeout * 2 / 3;
>
>
>         readOnly = canBeReadOnly;
>
>
>
>
>
>         sendThread = new SendThread(clientCnxnSocket);
>
>
>         eventThread = new EventThread();
>
>
>         this.clientConfig=zooKeeper.getClientConfig();
>
>
>     }
>
>
>
>
> Be leading to trouble if the number of ZK Servers is larger?
>
> This timeout value is initially used when the Client is waiting to connect
> with the server, even before it has negotiated a sessiontimeout value.
>
> So larger the number of ZK Servers, less will be the timer value.
> Shouldn’t we have a lower guard for it and ensure that atleast a min value
> is used for this timer?
>
>
Probably a good idea. I've seen some terrible thundering herd problems that
result from this very basic logic. Some form of exponential backoff would
probably even be better. (configurable?) That said I've never seen the size
of the ensemble be that much of an issue given the other issue.

You might also check the latest trunk, can't remember if someone's been
trying to look at this recently...

Regards,

Patrick


>
>
> Regards
>
> Shirsha
>
>
>
> -----Original Message-----
> From: Patrick Hunt [mailto:[hidden email]]
> Sent: Wednesday, March 22, 2017 11:55 PM
> To: UserZooKeeper <[hidden email]>
> Subject: Re: How to modify Client Connection timer
>
>
>
> You should be able to control that by increasing the session timeout. I'm
> not familiar with the client code you are using however. Additionally the
> ZK client should retry it's connection (as long as you don't close the ZK
>
> object) continually until it is able to reconnect.
>
>
>
> Patrick
>
>
>
> On Mon, Mar 20, 2017 at 3:22 PM, Ray Chaudhuri, Shirsha (Nokia -
>
> IN/Bangalore) <[hidden email]<mailto:shirsha.ray_
> [hidden email]>> wrote:
>
>
>
> > Hi,
>
> >
>
> > We are using ZKFC  as a Zookeeper Client that tries to connect to
>
> > Zookeeper Server at the time of bringup, when we sometimes encounter
>
> > the following issue _
>
> >
>
> > 1.       Client retrieves the ZK Ensemble address (ensemble consists of 3
>
> > nodes)
>
> >
>
> > 2.       Client tries to connect to one of the ZK Server nodes
>
> >
>
> > 3.       However due to other on-going processing (required at the time
> of
>
> > bringup) at the ZK Server Node, it fails to respond in time to the
> Client.
>
> > The response goes after 3 seconds
>
> >
>
> > 4.       The Client side times out by then, typically seen around 1.6 sec
>
> >
>
> > Is there a way this timer at the Client side can be reconfigured to a
>
> > higher value?
>
> >
>
> > Regards
>
> > Shirsha
>
> >
>
Loading...