call fsyncdata on flush

[trinity-1686a]

fix #48

[fulmicoton]

Can you check the performance? I would like to get an idea of the impact of fsync on SSD. In particular, do we want to make it optional in the options?

[guilload]

I think we want a manual mode for Quickwit because we do:

for subrequest in request.subrequests:
  mrecordlog.write(subrequest.records)

It's unfortunate to have to flush or fsync in the middle of the loop currently.

[trinity-1686a]

I think we want a manual mode for Quickwit because we do

hum, currently we offer to flush/fsync either at every write, or at most once every X seconds (which quickwit uses). We may want a 3rd "only flush when asked for", but that seems orthogonal to this pr

[trinity-1686a]

Can you check the performance? I would like to get an idea of the impact of fsync on SSD. In particular, do we want to make it optional in the options?

we definitely wants this optional. When flushing every 16k/256k/4m: bench time is +13600%/+3132.9%/+422.11% (i did not run benches with smaller blocks, they were too slow).
if syncing every 100ms instead of every block of record, the results are less terrible: +92%/+87%/+49%

[fulmicoton]

The code has become a tiny bit too flexible and a bit too confusing. (sync and flush are interverted in places)

Can we call it persist as the generic word, and drop the separate SyncPolicy?

#[derive(Copy, Clone, Debug)]
pub enum PersistAction {
    /// The buffer will be flushed to the OS, but not necessarily to the disk.
    Flush,
    /// The buffer will be flushed to the OS, and the OS will be asked to flush
    /// it to the disk.
    FlushAndFsync,
}

/// We have two type of operations on the mrecordlog.
///
/// Critical records are relatively rare and really need to be persisted:
/// - RecordPosition { queue: &'a str, position: u64 },
/// - DeleteQueue.
///
/// For these operations, we want to always flush and fsync.
///
/// On the other hand,
/// - Truncate
/// - AppendRecords
/// are considered are more frequent and one might want to sacrifice
/// persistence guarantees for performance.
///
/// The `PersistPolicy` defines the trade-off applied for the second kind of
/// operations.
pub enum PersistPolicy {
    DoNothing,
    Always(PersistAction),
    OnDelay(PersistAction),
}

#[derive(Debug)]
enum PersistState {
    OnAppend,
    OnDelay {
        next_sync: Instant,
        interval: Duration,
    },
    OnRequest,
}

Also, we are supposed to fsync the parent directory (yeah it is weird) when we create a new file.

Suggestion: we could always fsync a file when we close it.

[trinity-1686a]

superseded by #54 (conflict with main was easier to fix by starting anew)