Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read/Write File Lock #380

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
bb25b9b
WIP
Yard1 Nov 23, 2024
fe38354
Docstrings
Yard1 Nov 23, 2024
e0e4911
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
c669bdc
Fix inner/outer for write
Yard1 Nov 23, 2024
90a38bf
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
75d7456
Lint
Yard1 Nov 23, 2024
3442e6b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
efd8de1
Lint
Yard1 Nov 23, 2024
b31ce04
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
8a84b54
Lint
Yard1 Nov 23, 2024
a0cba46
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
9a3b7cb
Nit
Yard1 Nov 23, 2024
bad2bd1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
8661be0
Tweak
Yard1 Nov 23, 2024
eed9492
Update _wrapper.py
Yard1 Nov 23, 2024
3ab83c3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
67b53eb
Fixes
Yard1 Nov 23, 2024
c7f8488
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
b5268f9
Fixes
Yard1 Nov 23, 2024
1787fb7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
02f7dd7
Lint
Yard1 Nov 23, 2024
978b128
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
a0c9e9a
Lint
Yard1 Nov 23, 2024
55b2020
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 23, 2024
1dd52b1
Fix
Yard1 Nov 28, 2024
a6d65c6
Asyncio
Yard1 Dec 1, 2024
c12b68b
Remove thread-local
Yard1 Dec 1, 2024
46fba64
Tests, fixes
Yard1 Dec 1, 2024
ab6f64d
Lint
Yard1 Dec 2, 2024
2d08ebc
Docs
Yard1 Dec 2, 2024
eac9062
Docs
Yard1 Dec 2, 2024
da41d44
Fix
Yard1 Dec 2, 2024
e9be639
Fix
Yard1 Dec 2, 2024
321a524
Example
Yard1 Dec 2, 2024
34b209a
Use protocol
Yard1 Dec 3, 2024
d1196f6
Protocols
Yard1 Dec 3, 2024
9caea5e
Default poll interval constant
Yard1 Dec 3, 2024
6dd9736
Remove extra backtics
Yard1 Dec 3, 2024
ec5f49c
Merge branch 'main' into read_write_lock
gaborbernat Jan 3, 2025
d9afe63
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 3, 2025
1afe716
Update _api.py
Yard1 Jan 12, 2025
913f496
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 12, 2025
1c0680b
Update _api.py
Yard1 Jan 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,7 @@ API
.. automodule:: filelock
:members:
:show-inheritance:

.. automodule:: filelock.read_write
:members:
:show-inheritance:
15 changes: 13 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,19 @@ cases.
Asyncio support
---------------

This library currently does not support asyncio. We'd recommend adding an asyncio variant though if someone can make a
pull request for it, `see here <https://github.com/tox-dev/py-filelock/issues/99>`_.
This library supports asyncio. See :class:`AsyncFileLock <filelock.AsyncFileLock>`.

Read/write FileLock
-------------------

An implementation of a read/write FileLock is also available: and :class:`ReadWriteFileLockWrapper <filelock.read_write.ReadWriteFileLockWrapper>` and
:class:`AsyncReadWriteFileLockWrapper <filelock.read_write.AsyncReadWriteFileLockWrapper>`.

Multiple readers can hold the lock at the same time, but a writer is guaranteed to hold the lock exclusively across both readers and writers.

The lock is writer-preferring on a best effort basis (there are no guarantees).

Currently, this FileLock type is implemented only for Unix.

FileLocks and threads
---------------------
Expand Down
5 changes: 4 additions & 1 deletion src/filelock/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,14 @@
import warnings
from typing import TYPE_CHECKING

from ._api import AcquireReturnProxy, BaseFileLock
from ._api import AcquireReturnProxy, BaseFileLock, LockProtocol
from ._error import Timeout
from ._soft import SoftFileLock
from ._unix import UnixFileLock, has_fcntl
from ._windows import WindowsFileLock
from .asyncio import (
AsyncAcquireReturnProxy,
AsyncLockProtocol,
AsyncSoftFileLock,
AsyncUnixFileLock,
AsyncWindowsFileLock,
Expand Down Expand Up @@ -56,12 +57,14 @@
"AcquireReturnProxy",
"AsyncAcquireReturnProxy",
"AsyncFileLock",
"AsyncLockProtocol",
"AsyncSoftFileLock",
"AsyncUnixFileLock",
"AsyncWindowsFileLock",
"BaseAsyncFileLock",
"BaseFileLock",
"FileLock",
"LockProtocol",
"SoftFileLock",
"Timeout",
"UnixFileLock",
Expand Down
28 changes: 24 additions & 4 deletions src/filelock/_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from abc import ABCMeta, abstractmethod
from dataclasses import dataclass
from threading import local
from typing import TYPE_CHECKING, Any, cast
from typing import TYPE_CHECKING, Any, Protocol, cast
from weakref import WeakValueDictionary

from ._error import Timeout
Expand All @@ -26,17 +26,37 @@

_LOGGER = logging.getLogger("filelock")

DEFAULT_POLL_INTERVAL = 0.05


class LockProtocol(Protocol):
"""Protocol for objects implementing ``acquire`` and ``release`` methods."""

@abstractmethod
def acquire(
self,
timeout: float | None = None,
poll_interval: float = DEFAULT_POLL_INTERVAL,
*,
poll_intervall: float | None = None,
blocking: bool | None = None,
) -> AcquireReturnProxy: ...

@abstractmethod
def release(self, force: bool = False) -> None: # noqa: FBT001, FBT002
...


# This is a helper class which is returned by :meth:`BaseFileLock.acquire` and wraps the lock to make sure __enter__
# is not called twice when entering the with statement. If we would simply return *self*, the lock would be acquired
# again in the *__enter__* method of the BaseFileLock, but not released again automatically. issue #37 (memory leak)
class AcquireReturnProxy:
"""A context-aware object that will release the lock file when exiting."""

def __init__(self, lock: BaseFileLock) -> None:
def __init__(self, lock: LockProtocol) -> None:
self.lock = lock

def __enter__(self) -> BaseFileLock:
def __enter__(self) -> LockProtocol:
return self.lock

def __exit__(
Expand Down Expand Up @@ -271,7 +291,7 @@ def lock_counter(self) -> int:
def acquire(
self,
timeout: float | None = None,
poll_interval: float = 0.05,
poll_interval: float = DEFAULT_POLL_INTERVAL,
*,
poll_intervall: float | None = None,
blocking: bool | None = None,
Expand Down
14 changes: 13 additions & 1 deletion src/filelock/_unix.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ def _acquire(self) -> None:
def _release(self) -> None:
raise NotImplementedError

class NonExclusiveUnixFileLock(UnixFileLock):
"""Uses the :func:`fcntl.flock` to non-exclusively lock the lock file on unix systems."""


else: # pragma: win32 no cover
try:
import fcntl
Expand All @@ -34,6 +38,8 @@ def _release(self) -> None:
class UnixFileLock(BaseFileLock):
"""Uses the :func:`fcntl.flock` to hard lock the lock file on unix systems."""

_fcntl_mode: int = fcntl.LOCK_EX

def _acquire(self) -> None:
ensure_directory_exists(self.lock_file)
open_flags = os.O_RDWR | os.O_TRUNC
Expand All @@ -43,7 +49,7 @@ def _acquire(self) -> None:
with suppress(PermissionError): # This locked is not owned by this UID
os.fchmod(fd, self._context.mode)
try:
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
fcntl.flock(fd, self._fcntl_mode | fcntl.LOCK_NB)
except OSError as exception:
os.close(fd)
if exception.errno == ENOSYS: # NotImplemented error
Expand All @@ -61,8 +67,14 @@ def _release(self) -> None:
fcntl.flock(fd, fcntl.LOCK_UN)
os.close(fd)

class NonExclusiveUnixFileLock(UnixFileLock):
"""Uses the :func:`fcntl.flock` to non-exclusively lock the lock file on unix systems."""

_fcntl_mode = fcntl.LOCK_SH


__all__ = [
"NonExclusiveUnixFileLock",
"UnixFileLock",
"has_fcntl",
]
35 changes: 29 additions & 6 deletions src/filelock/asyncio.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@
import logging
import os
import time
from abc import abstractmethod
from dataclasses import dataclass
from threading import local
from typing import TYPE_CHECKING, Any, Callable, NoReturn, cast
from typing import TYPE_CHECKING, Any, Callable, NoReturn, Protocol, cast

from ._api import BaseFileLock, FileLockContext, FileLockMeta
from ._api import DEFAULT_POLL_INTERVAL, BaseFileLock, FileLockContext, FileLockMeta
from ._error import Timeout
from ._soft import SoftFileLock
from ._unix import UnixFileLock
from ._unix import NonExclusiveUnixFileLock, UnixFileLock
from ._windows import WindowsFileLock

if TYPE_CHECKING:
Expand All @@ -31,6 +32,23 @@
_LOGGER = logging.getLogger("filelock")


class AsyncLockProtocol(Protocol):
"""Protocol for async objects implementing ``acquire`` and ``release`` methods."""

@abstractmethod
async def acquire(
self,
timeout: float | None = None,
poll_interval: float = DEFAULT_POLL_INTERVAL,
*,
blocking: bool | None = None,
) -> AsyncAcquireReturnProxy: ...

@abstractmethod
async def release(self, force: bool = False) -> None: # noqa: FBT001, FBT002
...


@dataclass
class AsyncFileLockContext(FileLockContext):
"""A dataclass which holds the context for a ``BaseAsyncFileLock`` object."""
Expand All @@ -52,10 +70,10 @@ class AsyncThreadLocalFileContext(AsyncFileLockContext, local):
class AsyncAcquireReturnProxy:
"""A context-aware object that will release the lock file when exiting."""

def __init__(self, lock: BaseAsyncFileLock) -> None: # noqa: D107
def __init__(self, lock: AsyncLockProtocol) -> None: # noqa: D107
self.lock = lock

async def __aenter__(self) -> BaseAsyncFileLock: # noqa: D105
async def __aenter__(self) -> AsyncLockProtocol: # noqa: D105
return self.lock

async def __aexit__( # noqa: D105
Expand Down Expand Up @@ -180,7 +198,7 @@ def loop(self) -> asyncio.AbstractEventLoop | None:
async def acquire( # type: ignore[override]
self,
timeout: float | None = None,
poll_interval: float = 0.05,
poll_interval: float = DEFAULT_POLL_INTERVAL,
*,
blocking: bool | None = None,
) -> AsyncAcquireReturnProxy:
Expand Down Expand Up @@ -329,12 +347,17 @@ class AsyncUnixFileLock(UnixFileLock, BaseAsyncFileLock):
"""Uses the :func:`fcntl.flock` to hard lock the lock file on unix systems."""


class AsyncNonExclusiveUnixFileLock(NonExclusiveUnixFileLock, BaseAsyncFileLock):
"""Uses the :func:`fcntl.flock` to non-exclusively lock the lock file on unix systems."""


class AsyncWindowsFileLock(WindowsFileLock, BaseAsyncFileLock):
"""Uses the :func:`msvcrt.locking` to hard lock the lock file on windows systems."""


__all__ = [
"AsyncAcquireReturnProxy",
"AsyncNonExclusiveUnixFileLock",
"AsyncSoftFileLock",
"AsyncUnixFileLock",
"AsyncWindowsFileLock",
Expand Down
64 changes: 64 additions & 0 deletions src/filelock/read_write/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""Read/write file lock."""

from __future__ import annotations

from typing import TYPE_CHECKING

from filelock._unix import NonExclusiveUnixFileLock, UnixFileLock, has_fcntl
from filelock.read_write._api import BaseReadWriteFileLock, ReadWriteMode, _DisabledReadWriteFileLock
from filelock.read_write._wrapper import BaseReadWriteFileLockWrapper, _DisabledReadWriteFileLockWrapper
from filelock.read_write.asyncio import (
AsyncReadWriteFileLock,
AsyncReadWriteFileLockWrapper,
BaseAsyncReadWriteFileLock,
BaseAsyncReadWriteFileLockWrapper,
UnixAsyncReadWriteFileLock,
UnixAsyncReadWriteFileLockWrapper,
)

if TYPE_CHECKING:
from filelock._api import BaseFileLock

ReadWriteFileLock: type[BaseReadWriteFileLock]
ReadWriteFileLockWrapper: type[BaseReadWriteFileLockWrapper]


class UnixReadWriteFileLock(BaseReadWriteFileLock):
"""Unix implementation of a read/write FileLock."""

_shared_file_lock_cls: type[BaseFileLock] = NonExclusiveUnixFileLock
_exclusive_file_lock_cls: type[BaseFileLock] = UnixFileLock


class UnixReadWriteFileLockWrapper(BaseReadWriteFileLockWrapper):
"""Wrapper for a Unix implementation of a read/write FileLock."""

_read_write_file_lock_cls = UnixReadWriteFileLock


if has_fcntl: # pragma: win32 no cover
ReadWriteFileLock = UnixReadWriteFileLock
ReadWriteFileLockWrapper = UnixReadWriteFileLockWrapper
has_read_write_file_lock = True
else: # pragma: win32 cover
ReadWriteFileLock = _DisabledReadWriteFileLock
ReadWriteFileLockWrapper = _DisabledReadWriteFileLockWrapper
has_read_write_file_lock = False


__all__ = [
"AsyncReadWriteFileLock",
"AsyncReadWriteFileLockWrapper",
"BaseAsyncReadWriteFileLock",
"BaseAsyncReadWriteFileLockWrapper",
"BaseReadWriteFileLock",
"BaseReadWriteFileLockWrapper",
"ReadWriteFileLock",
"ReadWriteFileLockWrapper",
"ReadWriteMode",
"UnixAsyncReadWriteFileLock",
"UnixAsyncReadWriteFileLockWrapper",
"UnixReadWriteFileLock",
"UnixReadWriteFileLockWrapper",
"has_read_write_file_lock",
]
Loading
Loading