Exists Watch Triggered by Delete

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Exists Watch Triggered by Delete

Stu Hood
I'm running the 3.0.0 release, and I'm receiving a warning thrown by this block of code:

> case NodeDeleted:
>     synchronized (dataWatches) {
>         addTo(dataWatches.remove(path), result);
>     }
>     // XXX This shouldn't be needed, but just in case
>     synchronized (existWatches) {
>         addTo(existWatches.remove(path), result);
>         LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");
>     }
>     synchronized (childWatches) {
>         addTo(childWatches.remove(path), result);
>     }
>     break;

--

Why shouldn't an exists watch be triggered by a node being deleted? That is a really common use case in my code, so I want to rule it out as the cause of a bug I'm hunting for.

Thanks,

Stu Hood
Architecture Software Developer
Mailtrust, a Division of Rackspace

Reply | Threaded
Open this post in threaded view
|

Re: Exists Watch Triggered by Delete

Patrick Hunt
Hi Stu,

The zk server maintains 2 lists of watches, data and child watches:
http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperProgrammers.html#ch_zkWatches
(after reviewing this doc I've entered a jira to clarify that the server
is maintaining 2 lists being referenced). From the server perspective if
you register a watch on a node by calling getData & exists, only a
single watch, a data watch, is stored by the server.

The client is maintaining lists of watches as well. This is essentially
to enable the auto watch reset and multi-watcher features added in v3.

Take a look at class ExistsWatchRegistration, it will register client
side dataWatches for exists calls -- unless the result code is not 0 (ie
NONODE), in which case it will register using existsWatches (again,
client side).

The comment you referenced in your original email is true - that code
should never execute as the existsWatches list only contains watches for
NONODE watch registrations (which obv couldn't be deleted since it
doesn't exist).

Hope this helps,

Patrick

Stu Hood wrote:

> I'm running the 3.0.0 release, and I'm receiving a warning thrown by this block of code:
>> case NodeDeleted:
>>     synchronized (dataWatches) {
>>         addTo(dataWatches.remove(path), result);
>>     }
>>     // XXX This shouldn't be needed, but just in case
>>     synchronized (existWatches) {
>>         addTo(existWatches.remove(path), result);
>>         LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");
>>     }
>>     synchronized (childWatches) {
>>         addTo(childWatches.remove(path), result);
>>     }
>>     break;
>
> --
>
> Why shouldn't an exists watch be triggered by a node being deleted? That is a really common use case in my code, so I want to rule it out as the cause of a bug I'm hunting for.
>
> Thanks,
>
> Stu Hood
> Architecture Software Developer
> Mailtrust, a Division of Rackspace
>
Reply | Threaded
Open this post in threaded view
|

Re: Exists Watch Triggered by Delete

Stu Hood
> The comment you referenced in your original email is true - that code
> should never execute as the existsWatches list only contains watches for
> NONODE watch registrations (which obv couldn't be deleted since it
> doesn't exist).
So I am experiencing a bug then?

It may be better not to let the programmer know about those two lists, and to see if the abstraction can be improved instead. Sometimes I feel that I have to know too much about the internal working of ZooKeeper to use its API.

Thanks Patrick,
Stu


-----Original Message-----
From: "Patrick Hunt" <[hidden email]>
Sent: Wednesday, November 12, 2008 2:11pm
To: [hidden email]
Subject: Re: Exists Watch Triggered by Delete

Hi Stu,

The zk server maintains 2 lists of watches, data and child watches:
http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperProgrammers.html#ch_zkWatches
(after reviewing this doc I've entered a jira to clarify that the server
is maintaining 2 lists being referenced). From the server perspective if
you register a watch on a node by calling getData & exists, only a
single watch, a data watch, is stored by the server.

The client is maintaining lists of watches as well. This is essentially
to enable the auto watch reset and multi-watcher features added in v3.

Take a look at class ExistsWatchRegistration, it will register client
side dataWatches for exists calls -- unless the result code is not 0 (ie
NONODE), in which case it will register using existsWatches (again,
client side).

The comment you referenced in your original email is true - that code
should never execute as the existsWatches list only contains watches for
NONODE watch registrations (which obv couldn't be deleted since it
doesn't exist).

Hope this helps,

Patrick

Stu Hood wrote:

> I'm running the 3.0.0 release, and I'm receiving a warning thrown by this block of code:
>> case NodeDeleted:
>>     synchronized (dataWatches) {
>>         addTo(dataWatches.remove(path), result);
>>     }
>>     // XXX This shouldn't be needed, but just in case
>>     synchronized (existWatches) {
>>         addTo(existWatches.remove(path), result);
>>         LOG.warn("We are triggering an exists watch for delete! Shouldn't happen!");
>>     }
>>     synchronized (childWatches) {
>>         addTo(childWatches.remove(path), result);
>>     }
>>     break;
>
> --
>
> Why shouldn't an exists watch be triggered by a node being deleted? That is a really common use case in my code, so I want to rule it out as the cause of a bug I'm hunting for.
>
> Thanks,
>
> Stu Hood
> Architecture Software Developer
> Mailtrust, a Division of Rackspace
>


Reply | Threaded
Open this post in threaded view
|

Re: Exists Watch Triggered by Delete

Patrick Hunt
Stu Hood wrote:
>> The comment you referenced in your original email is true - that
>> code should never execute as the existsWatches list only contains
>> watches for NONODE watch registrations (which obv couldn't be
>> deleted since it doesn't exist).
> So I am experiencing a bug then?

Wow, this is a dumb mistake. The log message you pointed to is being
output _every time_ this code is called. There needs to be a conditional
on the log message to check the result of the remove call.

I have entered a JIRA and have already submitted a patch:
https://issues.apache.org/jira/browse/ZOOKEEPER-221

So unfortunately we can't tell if this is the source of the issue you
are seeing. Can you run with the patch attached to ZOOKEEPER-221?

> It may be better not to let the programmer know about those two
> lists, and to see if the abstraction can be improved instead.
> Sometimes I feel that I have to know too much about the internal
> working of ZooKeeper to use its API.

That's a good point. We're working to improve the docs & code, I've made
a note of it. Perhaps we should move that to "internals" doc and rework
this section of the guide...
https://issues.apache.org/jira/browse/ZOOKEEPER-220

Patrick

>
>
> -----Original Message----- From: "Patrick Hunt" <[hidden email]>
> Sent: Wednesday, November 12, 2008 2:11pm To:
> [hidden email] Subject: Re: Exists Watch Triggered
> by Delete
>
> Hi Stu,
>
> The zk server maintains 2 lists of watches, data and child watches:
> http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperProgrammers.html#ch_zkWatches
>  (after reviewing this doc I've entered a jira to clarify that the
> server is maintaining 2 lists being referenced). From the server
> perspective if you register a watch on a node by calling getData &
> exists, only a single watch, a data watch, is stored by the server.
>
> The client is maintaining lists of watches as well. This is
> essentially to enable the auto watch reset and multi-watcher features
> added in v3.
>
> Take a look at class ExistsWatchRegistration, it will register client
>  side dataWatches for exists calls -- unless the result code is not 0
> (ie NONODE), in which case it will register using existsWatches
> (again, client side).
>
> The comment you referenced in your original email is true - that code
>  should never execute as the existsWatches list only contains watches
> for NONODE watch registrations (which obv couldn't be deleted since
> it doesn't exist).
>
> Hope this helps,
>
> Patrick
>
> Stu Hood wrote:
>> I'm running the 3.0.0 release, and I'm receiving a warning thrown
>> by this block of code:
>>> case NodeDeleted: synchronized (dataWatches) {
>>> addTo(dataWatches.remove(path), result); } // XXX This shouldn't
>>> be needed, but just in case synchronized (existWatches) {
>>> addTo(existWatches.remove(path), result); LOG.warn("We are
>>> triggering an exists watch for delete! Shouldn't happen!"); }
>>> synchronized (childWatches) { addTo(childWatches.remove(path),
>>> result); } break;
>> --
>>
>> Why shouldn't an exists watch be triggered by a node being deleted?
>> That is a really common use case in my code, so I want to rule it
>> out as the cause of a bug I'm hunting for.
>>
>> Thanks,
>>
>> Stu Hood Architecture Software Developer Mailtrust, a Division of
>> Rackspace
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Exists Watch Triggered by Delete

Stu Hood
Thanks a bunch Patrick! I really appreciate your work on ZooKeeper... you have made a tremendous impact on the project.

I will try the ZOOKEEPER-221 patch.

Thanks,
Stu


-----Original Message-----
From: "Patrick Hunt" <[hidden email]>
Sent: Wednesday, November 12, 2008 6:37pm
To: [hidden email]
Subject: Re: Exists Watch Triggered by Delete

Stu Hood wrote:
>> The comment you referenced in your original email is true - that
>> code should never execute as the existsWatches list only contains
>> watches for NONODE watch registrations (which obv couldn't be
>> deleted since it doesn't exist).
> So I am experiencing a bug then?

Wow, this is a dumb mistake. The log message you pointed to is being
output _every time_ this code is called. There needs to be a conditional
on the log message to check the result of the remove call.

I have entered a JIRA and have already submitted a patch:
https://issues.apache.org/jira/browse/ZOOKEEPER-221

So unfortunately we can't tell if this is the source of the issue you
are seeing. Can you run with the patch attached to ZOOKEEPER-221?

> It may be better not to let the programmer know about those two
> lists, and to see if the abstraction can be improved instead.
> Sometimes I feel that I have to know too much about the internal
> working of ZooKeeper to use its API.

That's a good point. We're working to improve the docs & code, I've made
a note of it. Perhaps we should move that to "internals" doc and rework
this section of the guide...
https://issues.apache.org/jira/browse/ZOOKEEPER-220

Patrick

>
>
> -----Original Message----- From: "Patrick Hunt" <[hidden email]>
> Sent: Wednesday, November 12, 2008 2:11pm To:
> [hidden email] Subject: Re: Exists Watch Triggered
> by Delete
>
> Hi Stu,
>
> The zk server maintains 2 lists of watches, data and child watches:
> http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperProgrammers.html#ch_zkWatches
>  (after reviewing this doc I've entered a jira to clarify that the
> server is maintaining 2 lists being referenced). From the server
> perspective if you register a watch on a node by calling getData &
> exists, only a single watch, a data watch, is stored by the server.
>
> The client is maintaining lists of watches as well. This is
> essentially to enable the auto watch reset and multi-watcher features
> added in v3.
>
> Take a look at class ExistsWatchRegistration, it will register client
>  side dataWatches for exists calls -- unless the result code is not 0
> (ie NONODE), in which case it will register using existsWatches
> (again, client side).
>
> The comment you referenced in your original email is true - that code
>  should never execute as the existsWatches list only contains watches
> for NONODE watch registrations (which obv couldn't be deleted since
> it doesn't exist).
>
> Hope this helps,
>
> Patrick
>
> Stu Hood wrote:
>> I'm running the 3.0.0 release, and I'm receiving a warning thrown
>> by this block of code:
>>> case NodeDeleted: synchronized (dataWatches) {
>>> addTo(dataWatches.remove(path), result); } // XXX This shouldn't
>>> be needed, but just in case synchronized (existWatches) {
>>> addTo(existWatches.remove(path), result); LOG.warn("We are
>>> triggering an exists watch for delete! Shouldn't happen!"); }
>>> synchronized (childWatches) { addTo(childWatches.remove(path),
>>> result); } break;
>> --
>>
>> Why shouldn't an exists watch be triggered by a node being deleted?
>> That is a really common use case in my code, so I want to rule it
>> out as the cause of a bug I'm hunting for.
>>
>> Thanks,
>>
>> Stu Hood Architecture Software Developer Mailtrust, a Division of
>> Rackspace
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Exists Watch Triggered by Delete

Patrick Hunt
No problem, we really appreciate your feedback -- keep it coming.

Patrick

Stu Hood wrote:

> Thanks a bunch Patrick! I really appreciate your work on ZooKeeper... you have made a tremendous impact on the project.
>
> I will try the ZOOKEEPER-221 patch.
>
> Thanks,
> Stu
>
>
> -----Original Message-----
> From: "Patrick Hunt" <[hidden email]>
> Sent: Wednesday, November 12, 2008 6:37pm
> To: [hidden email]
> Subject: Re: Exists Watch Triggered by Delete
>
> Stu Hood wrote:
>>> The comment you referenced in your original email is true - that
>>> code should never execute as the existsWatches list only contains
>>> watches for NONODE watch registrations (which obv couldn't be
>>> deleted since it doesn't exist).
>> So I am experiencing a bug then?
>
> Wow, this is a dumb mistake. The log message you pointed to is being
> output _every time_ this code is called. There needs to be a conditional
> on the log message to check the result of the remove call.
>
> I have entered a JIRA and have already submitted a patch:
> https://issues.apache.org/jira/browse/ZOOKEEPER-221
>
> So unfortunately we can't tell if this is the source of the issue you
> are seeing. Can you run with the patch attached to ZOOKEEPER-221?
>
>> It may be better not to let the programmer know about those two
>> lists, and to see if the abstraction can be improved instead.
>> Sometimes I feel that I have to know too much about the internal
>> working of ZooKeeper to use its API.
>
> That's a good point. We're working to improve the docs & code, I've made
> a note of it. Perhaps we should move that to "internals" doc and rework
> this section of the guide...
> https://issues.apache.org/jira/browse/ZOOKEEPER-220
>
> Patrick
>
>>
>> -----Original Message----- From: "Patrick Hunt" <[hidden email]>
>> Sent: Wednesday, November 12, 2008 2:11pm To:
>> [hidden email] Subject: Re: Exists Watch Triggered
>> by Delete
>>
>> Hi Stu,
>>
>> The zk server maintains 2 lists of watches, data and child watches:
>> http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperProgrammers.html#ch_zkWatches
>>  (after reviewing this doc I've entered a jira to clarify that the
>> server is maintaining 2 lists being referenced). From the server
>> perspective if you register a watch on a node by calling getData &
>> exists, only a single watch, a data watch, is stored by the server.
>>
>> The client is maintaining lists of watches as well. This is
>> essentially to enable the auto watch reset and multi-watcher features
>> added in v3.
>>
>> Take a look at class ExistsWatchRegistration, it will register client
>>  side dataWatches for exists calls -- unless the result code is not 0
>> (ie NONODE), in which case it will register using existsWatches
>> (again, client side).
>>
>> The comment you referenced in your original email is true - that code
>>  should never execute as the existsWatches list only contains watches
>> for NONODE watch registrations (which obv couldn't be deleted since
>> it doesn't exist).
>>
>> Hope this helps,
>>
>> Patrick
>>
>> Stu Hood wrote:
>>> I'm running the 3.0.0 release, and I'm receiving a warning thrown
>>> by this block of code:
>>>> case NodeDeleted: synchronized (dataWatches) {
>>>> addTo(dataWatches.remove(path), result); } // XXX This shouldn't
>>>> be needed, but just in case synchronized (existWatches) {
>>>> addTo(existWatches.remove(path), result); LOG.warn("We are
>>>> triggering an exists watch for delete! Shouldn't happen!"); }
>>>> synchronized (childWatches) { addTo(childWatches.remove(path),
>>>> result); } break;
>>> --
>>>
>>> Why shouldn't an exists watch be triggered by a node being deleted?
>>> That is a really common use case in my code, so I want to rule it
>>> out as the cause of a bug I'm hunting for.
>>>
>>> Thanks,
>>>
>>> Stu Hood Architecture Software Developer Mailtrust, a Division of
>>> Rackspace
>>>
>>
>
>