Identify ephemeral node owner

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Identify ephemeral node owner

Travis Crawford
Hey zookeepers -

Is there a way to map the ephemeral node owner session back to the
process holding that node open? For example, given the following znode
stat output from the python client I'm interested in finding who has
this open.

{'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
1301279980630L, 'cversion': 0, 'czxid': 17302254734L}


Grep'ing through zookeeper.log I found lines like follows referencing
the ephemeralOwner number, however, nothing useful in tracking down
the process holding it open.

2011-03-29 13:12:50,083 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
client: 301083794498035291


Thanks!
Travis
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Mahadev Konar-3
Travis,
 the only way to track an ephemeral node is the session id, in your case:

301083794498035291. You can find which ip has this session by mining
the logs for zookeeper servers, but for pin pointing which process
unfortunately you'll have to look at your process logs and see which
one created that session.

thanks
mahadev

The session id is the session which the znode belongs to. For finding
the process that m

On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
<[hidden email]> wrote:

> Hey zookeepers -
>
> Is there a way to map the ephemeral node owner session back to the
> process holding that node open? For example, given the following znode
> stat output from the python client I'm interested in finding who has
> this open.
>
> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>
>
> Grep'ing through zookeeper.log I found lines like follows referencing
> the ephemeralOwner number, however, nothing useful in tracking down
> the process holding it open.
>
> 2011-03-29 13:12:50,083 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
> client: 301083794498035291
>
>
> Thanks!
> Travis
>



--
thanks
mahadev
@mahadevkonar
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Patrick Hunt
Hi Travis,

as Mahadev mentioned, when a client establishes a session with a
server the host id is logged to the server log:
2011-03-29 14:41:59,741 - INFO  [SyncThread:0:NIOServerCnxn@1580] -
Established session 0x12f0390e7520000 with negotiated timeout 30000
for client /127.0.0.1:47789

You could examine the ephemeral owner and grep the logs for this
(that's what I typically do).

A second more complex option; the client could note it's session id,
along with some other identifying information
(host/process/user/etc...) as a ZNode as part of it's joining the
cluster. This would obv require changes on the client side though.

There's also this jira, might be a good idea for us to capture some
client specific information here (although we'd have to watch out for
security issues with this, exposing this information only to
appropriate parties...)
https://issues.apache.org/jira/browse/ZOOKEEPER-829

Patrick

On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <[hidden email]> wrote:

> Travis,
>  the only way to track an ephemeral node is the session id, in your case:
>
> 301083794498035291. You can find which ip has this session by mining
> the logs for zookeeper servers, but for pin pointing which process
> unfortunately you'll have to look at your process logs and see which
> one created that session.
>
> thanks
> mahadev
>
> The session id is the session which the znode belongs to. For finding
> the process that m
>
> On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
> <[hidden email]> wrote:
>> Hey zookeepers -
>>
>> Is there a way to map the ephemeral node owner session back to the
>> process holding that node open? For example, given the following znode
>> stat output from the python client I'm interested in finding who has
>> this open.
>>
>> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>>
>>
>> Grep'ing through zookeeper.log I found lines like follows referencing
>> the ephemeralOwner number, however, nothing useful in tracking down
>> the process holding it open.
>>
>> 2011-03-29 13:12:50,083 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>> client: 301083794498035291
>>
>>
>> Thanks!
>> Travis
>>
>
>
>
> --
> thanks
> mahadev
> @mahadevkonar
>
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Travis Crawford
Thanks for the info! That initial session creation log line would be
quite handy if it hadn't been rotated away in this case :) I'll
probably change the registration code to include info about who
created the node - in this case I'll track the process down via some
other route.

ZOOKEEPER-829 is interesting btw, sort of like /proc - nice!

--travis



On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <[hidden email]> wrote:

> Hi Travis,
>
> as Mahadev mentioned, when a client establishes a session with a
> server the host id is logged to the server log:
> 2011-03-29 14:41:59,741 - INFO  [SyncThread:0:NIOServerCnxn@1580] -
> Established session 0x12f0390e7520000 with negotiated timeout 30000
> for client /127.0.0.1:47789
>
> You could examine the ephemeral owner and grep the logs for this
> (that's what I typically do).
>
> A second more complex option; the client could note it's session id,
> along with some other identifying information
> (host/process/user/etc...) as a ZNode as part of it's joining the
> cluster. This would obv require changes on the client side though.
>
> There's also this jira, might be a good idea for us to capture some
> client specific information here (although we'd have to watch out for
> security issues with this, exposing this information only to
> appropriate parties...)
> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>
> Patrick
>
> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <[hidden email]> wrote:
>> Travis,
>>  the only way to track an ephemeral node is the session id, in your case:
>>
>> 301083794498035291. You can find which ip has this session by mining
>> the logs for zookeeper servers, but for pin pointing which process
>> unfortunately you'll have to look at your process logs and see which
>> one created that session.
>>
>> thanks
>> mahadev
>>
>> The session id is the session which the znode belongs to. For finding
>> the process that m
>>
>> On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
>> <[hidden email]> wrote:
>>> Hey zookeepers -
>>>
>>> Is there a way to map the ephemeral node owner session back to the
>>> process holding that node open? For example, given the following znode
>>> stat output from the python client I'm interested in finding who has
>>> this open.
>>>
>>> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>>> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>>> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>>> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>>>
>>>
>>> Grep'ing through zookeeper.log I found lines like follows referencing
>>> the ephemeralOwner number, however, nothing useful in tracking down
>>> the process holding it open.
>>>
>>> 2011-03-29 13:12:50,083 - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>>> client: 301083794498035291
>>>
>>>
>>> Thanks!
>>> Travis
>>>
>>
>>
>>
>> --
>> thanks
>> mahadev
>> @mahadevkonar
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Ted Dunning
In reply to this post by Patrick Hunt
The simplest (though probably least acceptable in most cases) solution is to
put the process host and pid into the ephemeral file in some form.

On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <[hidden email]> wrote:

> Hi Travis,
>
> as Mahadev mentioned, when a client establishes a session with a
> server the host id is logged to the server log:
> 2011-03-29 14:41:59,741 - INFO  [SyncThread:0:NIOServerCnxn@1580] -
> Established session 0x12f0390e7520000 with negotiated timeout 30000
> for client /127.0.0.1:47789
>
> You could examine the ephemeral owner and grep the logs for this
> (that's what I typically do).
>
> A second more complex option; the client could note it's session id,
> along with some other identifying information
> (host/process/user/etc...) as a ZNode as part of it's joining the
> cluster. This would obv require changes on the client side though.
>
> There's also this jira, might be a good idea for us to capture some
> client specific information here (although we'd have to watch out for
> security issues with this, exposing this information only to
> appropriate parties...)
> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>
> Patrick
>
> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <[hidden email]> wrote:
> > Travis,
> >  the only way to track an ephemeral node is the session id, in your case:
> >
> > 301083794498035291. You can find which ip has this session by mining
> > the logs for zookeeper servers, but for pin pointing which process
> > unfortunately you'll have to look at your process logs and see which
> > one created that session.
> >
> > thanks
> > mahadev
> >
> > The session id is the session which the znode belongs to. For finding
> > the process that m
> >
> > On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
> > <[hidden email]> wrote:
> >> Hey zookeepers -
> >>
> >> Is there a way to map the ephemeral node owner session back to the
> >> process holding that node open? For example, given the following znode
> >> stat output from the python client I'm interested in finding who has
> >> this open.
> >>
> >> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
> >> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
> >> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
> >> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
> >>
> >>
> >> Grep'ing through zookeeper.log I found lines like follows referencing
> >> the ephemeralOwner number, however, nothing useful in tracking down
> >> the process holding it open.
> >>
> >> 2011-03-29 13:12:50,083 - INFO
> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
> >> client: 301083794498035291
> >>
> >>
> >> Thanks!
> >> Travis
> >>
> >
> >
> >
> > --
> > thanks
> > mahadev
> > @mahadevkonar
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Travis Crawford
On Tue, Mar 29, 2011 at 3:03 PM, Ted Dunning <[hidden email]> wrote:
> The simplest (though probably least acceptable in most cases) solution is to
> put the process host and pid into the ephemeral file in some form.
>

To close the loop... I ended up doing exactly this. The app already
registered itself with:

https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/Group.java#L215

So I wrote a Supplier that puts the hostname + pid into the znode as
json, which lets us add more stuff over time if needed.

  public static JsonSupplier newProcessDescriptionSupplier() {
    RuntimeMXBean bean = ManagementFactory.getRuntimeMXBean();
    String[] parts = bean.getName().split("@");
    JSONObject j = new JSONObject();
    try {
      j.put("pid", parts[0]);
      j.put("hostname", parts[1]);
    } catch (JSONException e) {
      LOG.error("Unable to populate supplier with initial values!", e);
    }
    return new JsonSupplier(j);
  }

Thanks for the pointers, y'all! Not a general solution but works well
in this case.

--travis




> On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <[hidden email]> wrote:
>>
>> Hi Travis,
>>
>> as Mahadev mentioned, when a client establishes a session with a
>> server the host id is logged to the server log:
>> 2011-03-29 14:41:59,741 - INFO  [SyncThread:0:NIOServerCnxn@1580] -
>> Established session 0x12f0390e7520000 with negotiated timeout 30000
>> for client /127.0.0.1:47789
>>
>> You could examine the ephemeral owner and grep the logs for this
>> (that's what I typically do).
>>
>> A second more complex option; the client could note it's session id,
>> along with some other identifying information
>> (host/process/user/etc...) as a ZNode as part of it's joining the
>> cluster. This would obv require changes on the client side though.
>>
>> There's also this jira, might be a good idea for us to capture some
>> client specific information here (although we'd have to watch out for
>> security issues with this, exposing this information only to
>> appropriate parties...)
>> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>>
>> Patrick
>>
>> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <[hidden email]> wrote:
>> > Travis,
>> >  the only way to track an ephemeral node is the session id, in your
>> > case:
>> >
>> > 301083794498035291. You can find which ip has this session by mining
>> > the logs for zookeeper servers, but for pin pointing which process
>> > unfortunately you'll have to look at your process logs and see which
>> > one created that session.
>> >
>> > thanks
>> > mahadev
>> >
>> > The session id is the session which the znode belongs to. For finding
>> > the process that m
>> >
>> > On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
>> > <[hidden email]> wrote:
>> >> Hey zookeepers -
>> >>
>> >> Is there a way to map the ephemeral node owner session back to the
>> >> process holding that node open? For example, given the following znode
>> >> stat output from the python client I'm interested in finding who has
>> >> this open.
>> >>
>> >> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>> >> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>> >> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>> >> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>> >>
>> >>
>> >> Grep'ing through zookeeper.log I found lines like follows referencing
>> >> the ephemeralOwner number, however, nothing useful in tracking down
>> >> the process holding it open.
>> >>
>> >> 2011-03-29 13:12:50,083 - INFO
>> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>> >> client: 301083794498035291
>> >>
>> >>
>> >> Thanks!
>> >> Travis
>> >>
>> >
>> >
>> >
>> > --
>> > thanks
>> > mahadev
>> > @mahadevkonar
>> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Identify ephemeral node owner

Ted Dunning
This sort of thing is exactly the rationale for extensible schemas such as
you get from JSON, Avro or protobufs.

On Wed, Mar 30, 2011 at 4:28 PM, Travis Crawford
<[hidden email]>wrote:

> On Tue, Mar 29, 2011 at 3:03 PM, Ted Dunning <[hidden email]>
> wrote:
> > The simplest (though probably least acceptable in most cases) solution is
> to
> > put the process host and pid into the ephemeral file in some form.
> >
>
> To close the loop... I ended up doing exactly this. The app already
> ...

Thanks for the pointers, y'all! Not a general solution but works well
> in this case.


Another way to look at this is to say that the solution is incredibly
general (you can have your bytes, your way).