Write Ahead Log

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Write Ahead Log

tj5527
I heard that ZooKeeper uses Write Ahead Log for preserving operation information, and am interested in how it's implemented. But when checking ZooKeeper confluence wiki and searching the internet, I don't find much related documents. Only a few source files such as FileTxnLog.java that seems to be related with some lines written directly about write ahead log string.

https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java

Is this the right place to start? What other resources available out there as reference?

Thanks
Reply | Threaded
Open this post in threaded view
|

RE: Write Ahead Log

Ibrahim
Hi,

You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope these will be helpful.

[1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for Internet-scale Systems." USENIX annual technical conference. Vol. 8. 2010.
[2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab: High-performance broadcast for primary-backup systems." Dependable Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference on. IEEE, 2011.

Ibrahim

-----Original Message-----
From: tj5527 [mailto:[hidden email]]
Sent: 08 January 2018 13:14
To: [hidden email]
Subject: Write Ahead Log

I heard that ZooKeeper uses Write Ahead Log for preserving operation information, and am interested in how it's implemented. But when checking ZooKeeper confluence wiki and searching the internet, I don't find much related documents. Only a few source files such as FileTxnLog.java that seems to be related with some lines written directly about write ahead log string.

https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java

Is this the right place to start? What other resources available out there as reference?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Write Ahead Log

Abraham Fine
In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira,‎ Benjamin Reed for getting a high level understanding of what's going on.

Thanks,
Abe

On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:

> Hi,
>
> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
> these will be helpful.
>
> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
> 2010.
> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
> High-performance broadcast for primary-backup systems." Dependable
> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
> on. IEEE, 2011.
>
> Ibrahim
>
> -----Original Message-----
> From: tj5527 [mailto:[hidden email]]
> Sent: 08 January 2018 13:14
> To: [hidden email]
> Subject: Write Ahead Log
>
> I heard that ZooKeeper uses Write Ahead Log for preserving operation
> information, and am interested in how it's implemented. But when
> checking ZooKeeper confluence wiki and searching the internet, I don't
> find much related documents. Only a few source files such as
> FileTxnLog.java that seems to be related with some lines written
> directly about write ahead log string.
>
> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>
> Is this the right place to start? What other resources available out
> there as reference?
>
> Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Write Ahead Log

tj5527
In addition to those great resources, what files can be checked for such mechanism (only write ahead log not other zk service components), particularly snapshot and log?

Thanks for all the input. Much appreciated!

> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 1:04 AM
> UTC Time: January 9, 2018 12:04 AM
> From: [hidden email]
> To: [hidden email]
>
> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira,‎ Benjamin Reed for getting a high level understanding of what's going on.
>
> Thanks,
> Abe
>
> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>
>> Hi,
>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>> these will be helpful.
>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>> 2010.
>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>> High-performance broadcast for primary-backup systems." Dependable
>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>> on. IEEE, 2011.
>> Ibrahim
>> -----Original Message-----
>> From: tj5527 [mailto:[hidden email]]
>> Sent: 08 January 2018 13:14
>> To: [hidden email]
>> Subject: Write Ahead Log
>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>> information, and am interested in how it's implemented. But when
>> checking ZooKeeper confluence wiki and searching the internet, I don't
>> find much related documents. Only a few source files such as
>> FileTxnLog.java that seems to be related with some lines written
>> directly about write ahead log string.
>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>> Is this the right place to start? What other resources available out
>> there as reference?
>> Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Write Ahead Log

Abraham Fine
I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.

First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".

The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.

Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.

Thanks,
Abe

On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:

> In addition to those great resources, what files can be checked for such
> mechanism (only write ahead log not other zk service components),
> particularly snapshot and log?
>
> Thanks for all the input. Much appreciated!
>
> > -------- Original Message --------
> > Subject: Re: Write Ahead Log
> > Local Time: January 9, 2018 1:04 AM
> > UTC Time: January 9, 2018 12:04 AM
> > From: [hidden email]
> > To: [hidden email]
> >
> > In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira,‎ Benjamin Reed for getting a high level understanding of what's going on.
> >
> > Thanks,
> > Abe
> >
> > On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
> >
> >> Hi,
> >> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
> >> these will be helpful.
> >> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
> >> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
> >> 2010.
> >> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
> >> High-performance broadcast for primary-backup systems." Dependable
> >> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
> >> on. IEEE, 2011.
> >> Ibrahim
> >> -----Original Message-----
> >> From: tj5527 [mailto:[hidden email]]
> >> Sent: 08 January 2018 13:14
> >> To: [hidden email]
> >> Subject: Write Ahead Log
> >> I heard that ZooKeeper uses Write Ahead Log for preserving operation
> >> information, and am interested in how it's implemented. But when
> >> checking ZooKeeper confluence wiki and searching the internet, I don't
> >> find much related documents. Only a few source files such as
> >> FileTxnLog.java that seems to be related with some lines written
> >> directly about write ahead log string.
> >> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
> >> Is this the right place to start? What other resources available out
> >> there as reference?
> >> Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Write Ahead Log

tj5527
Files are source files e.g. FileTxnLog.java that is related to Write Ahead Log and snapshot mechanism. And thanks for the explanation!

> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 7:07 PM
> UTC Time: January 9, 2018 6:07 PM
> From: [hidden email]
> To: [hidden email]
>
> I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.
>
> First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".
>
> The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.
>
> Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.
>
> Thanks,
> Abe
>
> On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:
>
>> In addition to those great resources, what files can be checked for such
>> mechanism (only write ahead log not other zk service components),
>> particularly snapshot and log?
>> Thanks for all the input. Much appreciated!
>>
>>> -------- Original Message --------
>>> Subject: Re: Write Ahead Log
>>> Local Time: January 9, 2018 1:04 AM
>>> UTC Time: January 9, 2018 12:04 AM
>>> From: [hidden email]
>>> To: [hidden email]
>>> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira,‎ Benjamin Reed for getting a high level understanding of what's going on.
>>> Thanks,
>>> Abe
>>> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>>>
>>>> Hi,
>>>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>>>> these will be helpful.
>>>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>>>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>>>> 2010.
>>>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>>>> High-performance broadcast for primary-backup systems." Dependable
>>>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>>>> on. IEEE, 2011.
>>>> Ibrahim
>>>> -----Original Message-----
>>>> From: tj5527 [mailto:[hidden email]]
>>>> Sent: 08 January 2018 13:14
>>>> To: [hidden email]
>>>> Subject: Write Ahead Log
>>>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>>>> information, and am interested in how it's implemented. But when
>>>> checking ZooKeeper confluence wiki and searching the internet, I don't
>>>> find much related documents. Only a few source files such as
>>>> FileTxnLog.java that seems to be related with some lines written
>>>> directly about write ahead log string.
>>>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>>>> Is this the right place to start? What other resources available out
>>>> there as reference?
>>>> Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Write Ahead Log

tj5527
In reply to this post by Abraham Fine
Files are source files e.g. FileTxnLog.java that is related to Write Ahead Log and snapshot mechanism. And thanks for the explanation!

> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 7:07 PM
> UTC Time: January 9, 2018 6:07 PM
> From: [hidden email]
> To: [hidden email]
>
> I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.
>
> First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".
>
> The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.
>
> Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.
>
> Thanks,
> Abe
>
> On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:
>
>> In addition to those great resources, what files can be checked for such
>> mechanism (only write ahead log not other zk service components),
>> particularly snapshot and log?
>> Thanks for all the input. Much appreciated!
>>
>>> -------- Original Message --------
>>> Subject: Re: Write Ahead Log
>>> Local Time: January 9, 2018 1:04 AM
>>> UTC Time: January 9, 2018 12:04 AM
>>> From: [hidden email]
>>> To: [hidden email]
>>> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira,‎ Benjamin Reed for getting a high level understanding of what's going on.
>>> Thanks,
>>> Abe
>>> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>>>
>>>> Hi,
>>>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>>>> these will be helpful.
>>>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>>>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>>>> 2010.
>>>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>>>> High-performance broadcast for primary-backup systems." Dependable
>>>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>>>> on. IEEE, 2011.
>>>> Ibrahim
>>>> -----Original Message-----
>>>> From: tj5527 [mailto:[hidden email]]
>>>> Sent: 08 January 2018 13:14
>>>> To: [hidden email]
>>>> Subject: Write Ahead Log
>>>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>>>> information, and am interested in how it's implemented. But when
>>>> checking ZooKeeper confluence wiki and searching the internet, I don't
>>>> find much related documents. Only a few source files such as
>>>> FileTxnLog.java that seems to be related with some lines written
>>>> directly about write ahead log string.
>>>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>>>> Is this the right place to start? What other resources available out
>>>> there as reference?
>>>> Thanks