Better support for other filesystems

[pforhan]

From the Javadoc in QueueFile.java:

The current implementation is built for file systems that support atomic segment writes (like YAFFS). Most conventional file systems don't support this; if the power goes out while writing a segment, the segment will contain garbage and the file will be corrupt. We'll add journaling support so this class can be used with more file systems later.

[mkurutin]

Hi,
Could you suggest if there is a way to detected if the file corruption happened? I assume that the fix for this case will be to delete the queue file and recreate it again. I'm just wondering are there any distinct syndromes of corruption that you might observed?

[AshwinJay]

I have some follow up questions. The JavaDoc for QueueFile says -

Writes are synchronous; data will be written
to disk before an operation returns. The underlying file is structured to
survive process and even system crashes
.. .. ..
Most conventional file systems don't support this; if the power goes out while
writing a segment, the segment will contain garbage and the file will be
corrupt. We'll add journaling support

• Isn't QueueFile itself like an append-only transaction log?! Without CRC checks and recovery..
• If the last segment is corrupt because of a previous crash, will we be able to read all the entries up to that segment?

Thanks.

[pforhan]

I'd love to implement a directory-based queue, with simple rename-based "AtomicFiles". Should have broad filesystem support. See JakeWharton/DiskLruCache for such a scheme.

[nilskp]

Found this ticket because I was confused about what guarantees QueueFile provides. The description on the main page states

The underlying file is structured to survive process and even system crashes

But as pointed out in the javadocs, this appears to be conditional based on what FS is used. Is that correct interpretation?

[JakeWharton]

If your filesystem has atomic writes of block segments then tape provides
guaranteed durability, yes.

On Wed, Sep 28, 2016, 12:07 PM Nils Kilden-Pedersen <
[email protected]> wrote:

Found this ticket because I was confused about what guarantees QueueFile
provides. The description on the main page states

The underlying file is structured to survive process and even system
crashes

But as pointed out in the javadocs, this appears to be conditional based
on what FS is used. Is that correct interpretation?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#5 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEES2I_BD__zCVOReiXDj3nEfW61y0ks5qupC3gaJpZM4ALALU
.

[nilskp]

It might be nice to update the website to reflect this rather important detail, as it currently doesn't specify this caveat.
The Javadocs are probably a less read source of information.

[f2prateek]

Adding here #123

[JakeWharton]

In practice it shouldn't matter. We get the underlying file descriptor and
tell it to sync which is the best that we can do. If you can't trust your
storage device to respect the semantics of fsync then nothing you do is
going to give you durability without ridiculous overhead.

On Wed, Sep 28, 2016 at 1:10 PM Nils Kilden-Pedersen <
[email protected]> wrote:

It might be nice to update the website to reflect this rather important
detail, as it currently doesn't specify this caveat.
The Javadocs are probably a less read source of information.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#5 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEETBvLWVTGRYWEOA5O2lNr_wURDgvks5qup98gaJpZM4ALALU
.

[ValleZ]

Tape queues were corrupted for me several times on NTFS for no apparent reason. Last time it happened when close() method was called but after that I killed process manually. Is there fsync in close()?

[LouisCAD]

As far as I know, SQLite has been proven to be really reliable across all the file systems available on the variety of Android devices.

SQLite can be used for persisted queues where reliability is critical, regardless of the file system you use (on an Android CTS certified device) I guess, although it's not what it was initially designed for.

Tape is about persisted queues by design, but it seems it's not as reliable as SQLite if the data partition is not on a YAFFS file system, right?

If I'm right about this, could a contributor try to explain why there is such a reliability gap between Tape and SQLite, and if it can be changed?

FYI, I'm working on an app where a failure could be life threatening since it'd fail to alert rescue. I currently use SnappyDB plus custom code to make it be a queue, and I'd like to move away from it since it's not being developed anymore, and it's made of code I don't understand and could not fix if an issue appeared. Tape seems like a perfect replacement, but I'd like to have high reliability, and would use SQLite instead if it fits better this requirement.

[JakeWharton]

SQLite uses a journal to buffer and keep writes, and that journal is very nearly the same thing as this library.

…

On Fri, May 19, 2017, 1:36 AM Louis CAD ***@***.***> wrote: As far as I know, SQLite has been proven to be really reliable across all the file systems available on the variety of Android devices. SQLite can be used for persisted queues where reliability is critical, regardless of the file system you use (on an Android CTS certified device) I guess, although it's not what it was initially designed for. Tape is about persisted queues by design, but it seems it's not as reliable as SQLite if the data partition is not on a YAFFS file system, right? If I'm right about this, could a contributor try to explain why there is such a reliability gap between Tape and SQLite, and if it can be changed? *FYI, I'm working on an app where a failure could be life threatening since it'd fail to alert rescue. I currently use SnappyDB <https://github.com/nhachicha/SnappyDB> plus custom code to make it be a queue, and I'd like to move away from it since it's not being developer anymore, and it's made of code I don't understand and could not fix if an issue appeared. Tape seems like a perfect replacement, but I'd like to have high reliability, and would use SQLite instead if it fits better this requirement.* — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEEEQjY99jsSJ9EvTz6bHc4YhRbfBLLks5r7VSmgaJpZM4ALALU> .

[LouisCAD]

Is this (journal buffer + keeping writes) the only thing that makes SQLite so reliable across the ~2 Billions Android devices, where most of them are using ext4 filesystems?
If yes, it means Tape should be roughly as reliable as SQLite then?

[JakeWharton]

Yes. fsync has to be trusted.

…

On Fri, May 19, 2017 at 8:34 AM Louis CAD ***@***.***> wrote: Is this (journal buffer + keeping writes) the only thing that makes SQLite so reliable across the ~2 Billions Android devices, where most of them are using ext4 filesystems? If yes, it means Tape should be roughly as reliable as SQLite then? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEEEWTuA3nIUgh1373k5dzfUsv5imjFks5r7YxqgaJpZM4ALALU> .

[LouisCAD]

Maybe adding a statement in the README about the fact that this library should be as reliable as SQLite would reassure developers like me, where a reliable persisted queue is critical to the app they are working on?

[E3V3A]

It would be great to clarify which exact FS are supported (i.e. have atomic segment writes.)
Nowadays, most Android's are using:

• F2FS
• EXT4
• ex/FAT (on external SD card)

How does this library affect these?