Configuration management based on load

[Mohan]

As a new user I was trying to understand how a farm of threaded listeners can have their threads bumped up based on load using zookeeper. Is this a conventional usecase for zookeeper ?

I understand that it involves programming zookeeper. I read that zookeeper is designed for reads rather than writes. Can I not stream load information - bytes or messages received by listeners- to the zookeeper server continuosly ?

Thanks.

[Ted Dunning]

I am completely clueless about what you are asking here.

Can you be a bit more explicit?

- do you mean that you want to increase the number of threads in the
listeners?

- do you mean that you want to increase the number of listeners?

- what do you mean by load?  Load on Zookeeper?  Load on each of the
listeners?  Load on one of the listeners?  Load across all of the
listeners?  Maximum load on any one?

Control and central configuration of a server farm is not an unusual use
case.

On Mon, Jan 30, 2012 at 9:13 PM, Mohan <[hidden email]>wrote:

> As a new user I was trying to understand how a farm of threaded listeners
> can
> have their threads bumped up based on load using zookeeper. Is this a
> conventional usecase for zookeeper ?
>
> I understand that it involves programming zookeeper. I read that zookeeper
> is designed for reads rather than writes. Can I not stream load information
> - bytes or messages received by listeners- to the zookeeper server
> continuosly ?
>
> Thanks.
>
> --
> View this message in context:
> http://zookeeper-user.578899.n2.nabble.com/Configuration-management-based-on-load-tp7238974p7238974.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
>

[Mohan]

It is like a load balancer for socket listeners.  I want to send some information to Zookeeper from each of the socket listeners. This could be the number of bytes waiting to be processed. Zookeeper could then decide based on some configuration data that one or more of the listeners is getting overloaded.

It could send back an event to increase the threads in one or more of the listeners.

Can I post and receive custom events from zookeeper ?

Thanks.

[Camille Fournier]

No. ZooKeeper merely reports on state of things, it does not execute
custom logic. You need to have your own servers that translate the
information provided by ZooKeeper and cause system state to change.

You could do what you want via a set of two types of processes and client logic.
Process type 1 is the servers you're talking about, that report via
writing to zk nodes their current busy level, and watch a set of data
nodes that contain configuration information directed at them.

Process type 2 watches the nodes written by type 1, and will update
configuration nodes based on that information and its belief about the
state of the whole system.

Note a few things about this:
Everything here is a few beats behind. If type 1 is totally
overloaded, it could possibly lack the ability to update zk to
indicate this overload. Type 2 may see temporary overloads that
resolve themselves quickly but still reconfigure systems based on
temporary information. So to do this you'll have to be careful to
identify intelligently long-term bottlenecks, and ensure that you can
always update ZK even when you are very busy.

C

On Tue, Jan 31, 2012 at 1:59 AM, Mohan <[hidden email]> wrote:

> It is like a load balancer for socket listeners using Zookeeper.  I want to
> send some information to Zookeeper from each of the socket listeners. This
> could be the number of bytes waiting to be processed. Zookeeper could then
> decide based on some configuration data that one or more of the listeners is
> getting overloaded.
>
> It could send back an event to increase the threads in one or more of the
> listeners.
>
> Can I post and receive custom events from zookeeper ?
>
> Thanks.
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/Configuration-management-based-on-load-tp7238974p7239127.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.