[bug] API row locking (#13274)

* row locking on patch

* row-level locking on action endpoints

* apply row locking to barcode actions

* additional row-locking for build actions

* rename exception class

* updated docs

* Add specific error code

* Fix translated text

* Handle AttributeError
This commit is contained in:
Oliver authored and GitHub committed 2026-11-15 15:26:47 +11:00
1 parent 3572714f5f
commit b2a9218cfb
16 files changed
+1125 -41

No files matched your search

+1
View File
@@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- [#13274](https://github.com/inventree/InvenTree/pull/13274) API requests which modify an existing object (updates, and "action" endpoints such as issuing or completing an order) now lock the database row before reading it, and run within a single database transaction. This prevents a concurrent edit from silently overwriting changes made by a background task (e.g. the "completed" quantity of a build order being reset while its outputs were being completed). If the object is locked by another long-running process, the request now returns a `409 Conflict` response and can be retried. A failed action no longer leaves partial changes behind.
- [#13241](https://github.com/inventree/InvenTree/pull/13241) completing a password reset now verifies the user's email address (when the account has a single address matching the account email), allowing users created by an administrator to set up MFA while it is enforced.
### Removed
Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

+7
View File
@@ -128,6 +128,13 @@ An InvenTree process (web server or background worker) reloaded its plugins, but
The affected process stops reloading its plugins to prevent an endless reload loop. Restart all InvenTree server and background worker processes to resolve this error. See [reloading plugins](../plugins/install.md#reloading-plugins) for further information.
#### INVE-E21
**Resource locked by another process - Backend**
An API request attempted to modify an object that is currently locked by another long-running process. This typically occurs when multiple processes are trying to update the same object concurrently.
The request will return a `409 Conflict` response. Retry the request after the other process has completed its operation.
## Warning Codes
### INVE-W (InvenTree Warning)
+42
View File
@@ -312,6 +312,48 @@ If running with a SQLite database backend, the following additional options are
{{ configsetting("INVENTREE_DB_TIMEOUT", default="10") }} Database connection timeout (s) |
| `INVENTREE_DB_WAL_MODE` | database.wal_mode | True | Enable Write-Ahead Logging (WAL) mode for SQLite databases |
### Concurrent Updates
InvenTree is a multi-user system, and it also runs [background tasks](./processes.md#background-worker) which modify the database. This means that the same database record can be changed by two processes at (almost) the same time - for example, a user edits a build order while a background task is completing the outputs of that same build order.
If this is not managed, one process can silently overwrite the changes made by the other (a "lost update"), as each process saves the data it read *before* the other process finished.
#### How Contention is Managed
InvenTree relies on *row-level locking* in the database to prevent lost updates:
- **Editing an object:** When an existing object is updated via the API, the database row is locked *before* the object is read. The object is then validated and saved within a single database transaction, and the lock is released when the request completes.
- **Performing an action:** Actions against an existing object (e.g. issuing, completing or cancelling an order, completing build outputs, receiving items, assigning a barcode) are handled in the same way: the target object is locked for the duration of the request, and if the action fails then none of its changes are saved.
- **Background tasks:** Tasks which adjust quantities (e.g. completing build outputs, consuming allocated stock, receiving items) lock the rows they work on, and re-read the current values before changing them.
A request waits for a short time (around two seconds) for a lock to be released, which is long enough for most operations to complete without the user noticing. If the object is still locked after this time (e.g. a background task is processing a large number of items), the request is rejected with a `409 Conflict` response, and no changes are made. The request can safely be submitted again once the other process has finished.
In the user interface, this is displayed as an error message at the top of the form:
{{ image("start/contention_error.png", "Error message displayed when an item is locked by another process") }}
!!! tip "Try Again"
This message does not indicate a problem with the data which was entered. Wait a few moments for the other process to complete, and then submit the form again.
#### Where Failures Can Still Occur
Row-level locking prevents the most common lost updates, but it cannot cover every case:
- **Stale forms:** A lock is only held while a request is being processed, not while a form is open. If an object is changed by someone else after a form has been opened, submitting the form overwrites those changes with the (older) values displayed in the form.
- **Operations across many objects:** Operations which act on many objects at once (e.g. bulk stock adjustments) are not locked as a single unit. Two such operations which overlap may have to wait for each other, and one of them may fail.
- **Deadlocks:** If two processes each wait for a lock held by the other, the database aborts one of them. The aborted request (or background task) fails with an error, and its changes are not saved.
- **Plugins and custom code:** Code which loads an object, modifies it and saves it without locking the row (e.g. a third-party plugin, or the admin interface) is not protected, and can overwrite a concurrent change.
- **Serializable isolation:** If the `INVENTREE_DB_ISOLATION_SERIALIZABLE` option is enabled, the database itself rejects any transaction which conflicts with another. Data is never silently overwritten in this mode, but InvenTree does not automatically retry a rejected transaction: a request or background task which conflicts with another fails with a server error, and must be retried manually.
#### SQLite Limitations
SQLite does not support row-level locking - instead, the entire database is locked while a process is writing to it. As a result:
- The protections described above are **not available**. A concurrent edit can still overwrite changes made by another process.
- Only one process can write to the database at a time. A process which cannot obtain the database lock within the configured `INVENTREE_DB_TIMEOUT` fails with a "database is locked" error.
This is one of the reasons that SQLite is not recommended for production use, particularly where multiple users (or a background worker) are active at the same time. Use PostgreSQL or MySQL for any installation where data integrity is important.
## Caching
InvenTree can be configured to use [redis](https://redis.io) as a global cache backend.
+17 -1
View File
@@ -12,13 +12,29 @@ from django.core.exceptions import ValidationError as DjangoValidationError
from django.utils.translation import gettext_lazy as _
import structlog
from rest_framework import serializers
from rest_framework import serializers, status
from rest_framework.exceptions import APIException
from rest_framework.exceptions import ValidationError as DRFValidationError
from rest_framework.response import Response
logger = structlog.get_logger('inventree')
class ResourceLockedError(APIException):
"""The requested object is locked by another operation, and cannot be modified right now.
Returned (as HTTP 409) when a database row could not be locked within a reasonable
time, e.g. because a background task is updating the same object.
The request has not been applied, and can safely be retried.
"""
status_code = status.HTTP_409_CONFLICT
default_detail = _(
'INVE-E21: This item is currently being updated by another process - please try again'
)
default_code = 'resource_locked'
def log_error(
path: Optional[str] = None,
error_name: Optional[str] = None,
@@ -5,11 +5,13 @@ import contextvars
import io
import ipaddress
import socket
import time
from typing import Optional, cast
from urllib.parse import urljoin, urlparse
from django.conf import settings
from django.core.validators import URLValidator
from django.db import connection, transaction
from django.db.utils import OperationalError, ProgrammingError
from django.utils.translation import gettext_lazy as _
@@ -433,3 +435,95 @@ def notify_users(
target_exclude=[exclude],
context=context,
)
class RowLockTimeoutError(Exception):
"""A database row lock could not be acquired within the allowed time."""
# Default time (in seconds) to wait for a contended row lock
ROW_LOCK_TIMEOUT = 2.0
# Time (in seconds) between attempts to acquire a contended row lock
ROW_LOCK_INTERVAL = 0.1
def _is_lock_unavailable_error(exc: OperationalError) -> bool:
"""Return True if the database error indicates that a row lock could not be acquired."""
cause = exc.__cause__
# PostgreSQL: 'lock_not_available' (psycopg 3 / psycopg2)
sqlstate = getattr(cause, 'sqlstate', None) or getattr(cause, 'pgcode', None)
if sqlstate:
return sqlstate == '55P03'
# MySQL / MariaDB: NOWAIT lock failure, or lock wait timeout
code = exc.args[0] if exc.args else None
return code in (3572, 1205)
def lock_row(model, pk, timeout: float | None = None) -> bool:
"""Acquire a row-level lock on a single database row, with a bounded wait.
A plain select_for_update() blocks for as long as another transaction holds the row
(e.g. a long-running background task). This helper gives up after a short time instead,
so that the caller can report the conflict rather than hanging.
Note: Must be called from within a database transaction (transaction.atomic),
and the lock is held until that transaction completes. Any instance of this row
which was loaded *before* the lock was acquired may be stale, and must be reloaded.
Arguments:
model: The model class
pk: The primary key of the row to lock
timeout: Maximum time (in seconds) to wait for the lock (default = ROW_LOCK_TIMEOUT)
Returns:
True if the row exists (and is now locked), False if the row does not exist.
If the database does not support row-level locking (e.g. SQLite), no lock is taken.
Raises:
RowLockTimeoutError: If the lock could not be acquired in time
"""
queryset = model._base_manager.filter(pk=pk)
features = connection.features
if not features.has_select_for_update:
return queryset.exists()
if not features.has_select_for_update_nowait:
# No way to bound the wait on this database - block until the lock is available
return (
queryset.select_for_update().values_list('pk', flat=True).first()
is not None
)
if timeout is None:
timeout = ROW_LOCK_TIMEOUT
deadline = time.monotonic() + timeout
while True:
try:
# Each attempt runs in a savepoint: a failed attempt must not
# invalidate the surrounding transaction
with transaction.atomic():
return (
queryset
.select_for_update(nowait=True)
.values_list('pk', flat=True)
.first()
is not None
)
except OperationalError as exc:
if not _is_lock_unavailable_error(exc):
raise
if time.monotonic() >= deadline:
raise RowLockTimeoutError(
f'Could not lock {model._meta.label} <{pk}>'
) from exc
time.sleep(ROW_LOCK_INTERVAL)
+193 -7
View File
@@ -1,12 +1,20 @@
"""Mixins for (API) views in the whole project."""
import contextlib
from django.core.exceptions import ValidationError as DjangoValidationError
from django.db import connection, models, transaction
from rest_framework import generics, mixins, status
from rest_framework.exceptions import NotFound
from rest_framework.response import Response
import data_exporter.mixins
import importer.mixins
from InvenTree.exceptions import ResourceLockedError
from InvenTree.fields import OutputConfiguration
from InvenTree.helpers import remove_non_printable_characters, strip_html_tags
from InvenTree.helpers_model import RowLockTimeoutError, lock_row
from InvenTree.schema import schema_for_view_output_options
from InvenTree.serializers import FilterableSerializerMixin
@@ -23,16 +31,147 @@ class CleanCreate: # noqa: D101
)
class CleanUpdate: # noqa: D101
class RowLockMixin:
"""View mixin which provides helpers for locking the database row a request acts upon.
A request which reads an object, and later writes it back, must lock the row *before*
reading it (and hold the lock until it has finished writing). Otherwise, the write is
based on a snapshot which may be stale - silently reverting any change committed in
the meantime (e.g. by a background task, or another request).
If the row is locked by another (long-running) transaction, the request is rejected
with a 409 (see InvenTree.exceptions.ResourceLockedError) rather than blocking.
"""
def lock_row_or_conflict(self, model, pk) -> bool:
"""Lock the specified database row (until the current transaction completes).
Must be called from within a database transaction.
Returns:
True if the row exists (and is now locked), otherwise False.
An invalid primary key value is treated as a row which does not exist.
Raises:
ResourceLockedError: If the row is locked by another transaction
"""
try:
pk = model._meta.pk.to_python(pk)
except AttributeError:
# This class may not have an associated model or _meta attribute
return False
except DjangoValidationError:
# The primary key value could not be converted to the correct type
return False
if pk is None:
return False
try:
return lock_row(model, pk)
except RowLockTimeoutError:
raise ResourceLockedError()
def lock_instance(self, instance):
"""Lock the database row of an instance which has already been loaded, and reload it.
To be used where the instance is resolved before the row can be locked
(e.g. by a serializer field): any data which was read before the lock was
acquired may be stale, so the instance is refreshed from the database.
Must be called from within a database transaction.
Raises:
ResourceLockedError: If the row is locked by another transaction
NotFound: If the row no longer exists
"""
if not self.lock_row_or_conflict(type(instance), instance.pk):
raise NotFound()
instance.refresh_from_db()
return instance
@contextlib.contextmanager
def locked_action(self, model, pk):
"""Run the enclosed block in a transaction, with the specified database row locked.
Any object which is read from the database within this block is up to date with
respect to the locked row, which cannot be changed by another process until
the block exits.
Raises:
ResourceLockedError: If the row is locked by another transaction
"""
with transaction.atomic():
self.lock_row_or_conflict(model, pk)
yield
class ActionLockMixin(RowLockMixin):
"""Mixin for an "action" endpoint (POST) which acts upon an existing object.
The entire request is handled within a database transaction, with the row of the
target object (as identified by the URL) locked before anything is read.
Attributes:
lock_model: The model class of the object which the action is performed against
lock_url_kwarg: The URL keyword argument which provides the primary key of that object
"""
lock_model = None
lock_url_kwarg = 'pk'
def is_locked_action(self, request) -> bool:
"""Return True if the provided request should be handled with the target object locked."""
return self.lock_model is not None and request.method == 'POST'
def dispatch(self, request, *args, **kwargs):
"""Handle the entire action within a single database transaction.
Note: This is implemented here (rather than in the 'post' method) as many
action endpoints provide a custom 'post' or 'create' method.
If the action fails (e.g. validation error), the DRF exception handler marks
the transaction for rollback, so a failed action does not write partial changes.
"""
if not self.is_locked_action(request):
return super().dispatch(request, *args, **kwargs)
with transaction.atomic():
return super().dispatch(request, *args, **kwargs)
def initial(self, request, *args, **kwargs):
"""Lock the target object, once the request has been authenticated and authorized."""
super().initial(request, *args, **kwargs)
if self.is_locked_action(request) and connection.in_atomic_block:
self.lock_row_or_conflict(
self.lock_model, self.kwargs.get(self.lock_url_kwarg)
)
class CleanUpdate(RowLockMixin):
"""Mixin which cleans the provided data before updating an object.
The update is performed within a single database transaction, with the target row
locked *before* it is read (see RowLockMixin).
"""
# Set to False to skip row locking for a particular view
lock_on_update: bool = True
def update(self, request, *args, **kwargs): # noqa: D102
# Override to clean data before processing it
partial = kwargs.pop('partial', False)
instance = self.get_object()
serializer = self.get_serializer(
instance, data=self.clean_data(request.data), partial=partial
)
serializer.is_valid(raise_exception=True)
self.perform_update(serializer)
with transaction.atomic():
instance = self.get_object_for_update()
serializer = self.get_serializer(
instance, data=self.clean_data(request.data), partial=partial
)
serializer.is_valid(raise_exception=True)
self.perform_update(serializer)
if getattr(instance, '_prefetched_objects_cache', None):
# If 'prefetch_related' has been applied to a queryset, we need to
@@ -41,6 +180,53 @@ class CleanUpdate: # noqa: D101
return Response(serializer.data)
def get_object_for_update(self):
"""Return the object to be updated, with its database row locked.
Must be called from within a database transaction.
Raises:
ResourceLockedError: If the row is locked by another transaction
"""
if not self.lock_on_update or not connection.features.has_select_for_update:
return self.get_object()
# Preferably, lock the row *before* reading the object.
# This is possible if the primary key is provided directly by the URL
locked = None
target = self._get_lock_target()
if target is not None:
locked = target
self.lock_row_or_conflict(*target)
instance = self.get_object()
if not isinstance(instance, models.Model) or instance.pk is None:
return instance
if locked == (type(instance), instance.pk):
return instance
# The object was resolved some other way (e.g. a custom lookup):
# lock its row now, and then read it again as it may have changed
self.lock_row_or_conflict(type(instance), instance.pk)
return self.get_object()
def _get_lock_target(self):
"""Return the (model, pk) which the URL of this request refers to, if known."""
if self.lookup_field != 'pk':
return None
value = self.kwargs.get(self.lookup_url_kwarg or self.lookup_field)
try:
model = self.get_queryset().model
return (model, model._meta.pk.to_python(value))
except (AssertionError, AttributeError, DjangoValidationError):
return None
class CleanBase:
"""Model mixin class which cleans inputs using nh3."""
@@ -29,8 +29,11 @@ def sentry_ignore_errors(): # pragma: no cover
These error types will *not* be reported to sentry.io.
"""
from InvenTree.exceptions import ResourceLockedError
return [
Http404,
ResourceLockedError,
MissingRate,
TemplateSyntaxError,
ValidationError,
+9 -2
View File
@@ -45,6 +45,7 @@ from InvenTree.filters import (
)
from InvenTree.helpers import str2bool
from InvenTree.mixins import (
ActionLockMixin,
CreateAPI,
ListCreateAPI,
OutputOptionsMixin,
@@ -739,8 +740,14 @@ class BuildLineDetail(BuildLineMixin, OutputOptionsMixin, RetrieveUpdateDestroyA
return None
class BuildOrderContextMixin:
"""Mixin class which adds build order as serializer context variable."""
class BuildOrderContextMixin(ActionLockMixin):
"""Mixin class which adds build order as serializer context variable.
The build order row is locked for the duration of the action (see ActionLockMixin),
so the action operates on (and saves) up-to-date build order data.
"""
lock_model = Build
def get_build(self):
"""Return the Build object associated with this API endpoint."""
+38 -15
View File
@@ -121,6 +121,11 @@ def delete_build_outputs(build_id: int, output_ids: list, **kwargs):
build = Build.objects.get(pk=build_id)
with transaction.atomic():
# Lock the build order row first: every operation against this build order
# (including the API endpoints) locks the build before any of its outputs,
# so that concurrent operations cannot deadlock against each other
Build.objects.select_for_update().filter(pk=build_id).first()
for output_id in output_ids:
# Lock the output row, and re-check that it is still "in production" -
# it may have been processed already (e.g. by a duplicated task)
@@ -167,6 +172,11 @@ def scrap_build_outputs(
user = User.objects.filter(pk=user_id).first() if user_id else None
with transaction.atomic():
# Lock the build order row first: every operation against this build order
# (including the API endpoints) locks the build before any of its outputs,
# so that concurrent operations cannot deadlock against each other
Build.objects.select_for_update().filter(pk=build_id).first()
for item in outputs:
# Lock the output row, and re-check that it is still "in production" -
# it may have been processed already (e.g. by a duplicated task)
@@ -229,6 +239,11 @@ def complete_build_outputs(
required_tests = build.part.getRequiredTests()
with transaction.atomic():
# Lock the build order row first: every operation against this build order
# (including the API endpoints) locks the build before any of its outputs,
# so that concurrent operations cannot deadlock against each other
Build.objects.select_for_update().filter(pk=build_id).first()
for item in outputs:
# Lock the output row, and re-check that it is still "in production" -
# it may have been processed already (e.g. by a duplicated task)
@@ -426,24 +441,32 @@ def update_build_order_lines(bom_item_pk: int):
# Iterate through each build, and update the relevant line items
for bo in builds:
# Try to find a matching build order line
line = BuildLine.objects.filter(build=bo, bom_item=bom_item).first()
q = bom_item.get_required_quantity(bo.quantity)
if line:
# If the BOM item points to a "virtual" part, delete the BuildLine instance
if bom_item.sub_part.virtual:
line.delete()
continue
with transaction.atomic():
# Try to find a matching build order line
# The row is locked before it is read, as the line is saved in full below:
# a stale copy would revert a concurrent update (e.g. the 'consumed' quantity)
line = (
BuildLine.objects
.select_for_update()
.filter(build=bo, bom_item=bom_item)
.first()
)
# Ensure quantity is correct
if line.quantity != q:
line.quantity = q
line.save()
elif not bom_item.sub_part.virtual:
# Create a new line item (for non-virtual parts)
BuildLine.objects.create(build=bo, bom_item=bom_item, quantity=q)
if line:
# If the BOM item points to a "virtual" part, delete the BuildLine instance
if bom_item.sub_part.virtual:
line.delete()
continue
# Ensure quantity is correct
if line.quantity != q:
line.quantity = q
line.save()
elif not bom_item.sub_part.virtual:
# Create a new line item (for non-virtual parts)
BuildLine.objects.create(build=bo, bom_item=bom_item, quantity=q)
if builds.count() > 0:
logger.info(
+656
View File
@@ -0,0 +1,656 @@
"""Regression tests for concurrent updates of build orders (and similar "counter" fields).
Background: the 'completed' count of a build order was reset to zero when a user
edited the build order while a background task was completing its outputs. The edit
loaded the build order before the task committed, and then wrote *every* column back
(including the stale 'completed' value) after the task had committed.
"""
import threading
import time
from unittest import mock
from django.contrib.auth import get_user_model
from django.db import connection, transaction
from django.test import TransactionTestCase, skipUnlessDBFeature
from django.urls import reverse
from rest_framework.test import APIClient
import build.tasks
import InvenTree.helpers_model
from build.models import Build, generate_next_build_reference
from build.status_codes import BuildStatus
from company.models import Company
from InvenTree.helpers_model import RowLockTimeoutError, lock_row
from order.models import (
PurchaseOrder,
ReturnOrder,
SalesOrder,
SalesOrderShipment,
TransferOrder,
)
from part.models import Part
from stock.models import StockItem
from stock.status_codes import StockStatus
def create_build(quantity: int = 100, n_outputs: int = 4, output_quantity: int = 5):
"""Create an issued build order, with a number of (incomplete) build outputs."""
assembly = Part.objects.create(
name=f'Locking assembly {generate_next_build_reference()}',
description='Assembly for build order locking tests',
assembly=True,
)
bo = Build.objects.create(
reference=generate_next_build_reference(),
part=assembly,
quantity=quantity,
title='Original title',
)
bo.issue_build()
outputs = [
StockItem.objects.create(
part=assembly, quantity=output_quantity, is_building=True, build=bo
)
for _ in range(n_outputs)
]
return bo, outputs
def run_complete_outputs_task(bo, outputs):
"""Run the background task which completes the provided build outputs."""
build.tasks.complete_build_outputs(
bo.pk, [{'output_id': output.pk} for output in outputs], None, StockStatus.OK
)
class ThreadedTestMixin:
"""Helpers for tests which run code in multiple threads (each with its own database connection)."""
def run_thread(self, func, *args) -> threading.Thread:
"""Run the provided function in a new thread, recording any exception."""
def target():
try:
func(*args)
except Exception as exc: # pragma: no cover - surfaced via errors list
self.errors.append(exc)
finally:
connection.close()
thread = threading.Thread(target=target)
thread.start()
self.threads.append(thread)
return thread
def join_threads(self):
"""Wait for all threads to complete, and check that no errors occurred."""
for thread in self.threads:
thread.join(timeout=20)
self.assertFalse(thread.is_alive())
self.assertEqual(self.errors, [])
def setUp(self):
"""Reset the thread state."""
super().setUp()
self.threads = []
self.errors = []
def hold_row_lock(self, model, pk, hold: threading.Event, **changes):
"""Lock a database row in another thread, until the 'hold' event is set.
Any 'changes' are written to the row (and committed) when the lock is released.
Returns once the lock has been acquired.
"""
locked = threading.Event()
def holder():
with transaction.atomic():
model.objects.select_for_update().get(pk=pk)
locked.set()
hold.wait(timeout=15)
if changes:
model.objects.filter(pk=pk).update(**changes)
self.run_thread(holder)
self.assertTrue(locked.wait(timeout=5))
@skipUnlessDBFeature('has_select_for_update')
class BuildCompletedConcurrencyTest(ThreadedTestMixin, TransactionTestCase):
"""Cross-transaction tests for the 'completed' count of a build order.
These tests use real threads (each with its own database connection),
to reproduce the interleaving of a background task and a concurrent edit.
"""
def pause_task_before_commit(self):
"""Patch the output completion task so that it pauses after completing an output.
At this point the task has updated (and so holds a lock on) the build order row,
but has not yet committed.
Returns:
A tuple of (context manager, 'paused' event, 'resume' event)
"""
paused = threading.Event()
resume = threading.Event()
original = Build.complete_build_output
def complete_and_pause(bo, *args, **kwargs):
result = original(bo, *args, **kwargs)
paused.set()
resume.wait(timeout=15)
return result
patcher = mock.patch.object(Build, 'complete_build_output', complete_and_pause)
return patcher, paused, resume
def test_concurrent_output_completion(self):
"""Tasks which complete different outputs of the same build order at the same time are all counted."""
bo, outputs = create_build(n_outputs=6)
barrier = threading.Barrier(3, timeout=10)
def run(task_outputs):
barrier.wait()
run_complete_outputs_task(bo, task_outputs)
for idx in range(3):
self.run_thread(run, outputs[idx * 2 : idx * 2 + 2])
self.join_threads()
bo.refresh_from_db()
self.assertEqual(bo.completed, 30)
def test_duplicate_output_completion(self):
"""The same task delivered twice (and running concurrently) only counts each output once."""
bo, outputs = create_build()
barrier = threading.Barrier(2, timeout=10)
def run():
barrier.wait()
run_complete_outputs_task(bo, outputs)
self.run_thread(run)
self.run_thread(run)
self.join_threads()
bo.refresh_from_db()
self.assertEqual(bo.completed, 20)
def test_complete_build_during_output_completion(self):
"""The task which completes the build order must not revert the count of a concurrent output completion."""
bo, outputs = create_build(quantity=20)
user = get_user_model().objects.create(username='locking-user')
patcher, paused, resume = self.pause_task_before_commit()
with patcher:
self.run_thread(run_complete_outputs_task, bo, outputs)
self.assertTrue(paused.wait(timeout=10))
# This task waits for the build order row, which is locked by the task above
self.run_thread(build.tasks.complete_build, bo.pk, user.pk)
time.sleep(0.5)
resume.set()
self.join_threads()
bo.refresh_from_db()
self.assertEqual(bo.status, 40)
self.assertEqual(bo.completed, 20)
def test_lock_row(self):
"""The lock_row() helper gives up if the row is locked by another transaction."""
bo, _outputs = create_build()
# No such row
with transaction.atomic():
self.assertFalse(lock_row(Build, bo.pk + 1000))
release = threading.Event()
self.hold_row_lock(Build, bo.pk, release)
try:
start = time.monotonic()
with transaction.atomic():
with self.assertRaises(RowLockTimeoutError):
lock_row(Build, bo.pk, timeout=0.3)
# The transaction is still usable after the failed attempt
self.assertTrue(Build.objects.filter(pk=bo.pk).exists())
self.assertGreaterEqual(time.monotonic() - start, 0.3)
finally:
release.set()
self.join_threads()
# The lock is available again
with transaction.atomic():
self.assertTrue(lock_row(Build, bo.pk, timeout=0.3))
@skipUnlessDBFeature('has_select_for_update')
class BuildUpdateLockingAPITest(ThreadedTestMixin, TransactionTestCase):
"""Cross-transaction tests for updating a build order via the API."""
def setUp(self):
"""Create an authenticated API client."""
super().setUp()
user = get_user_model().objects.create(
username='locking-admin', is_superuser=True, is_staff=True
)
self.client = APIClient()
self.client.force_authenticate(user=user)
self.bo, self.outputs = create_build()
self.url = reverse('api-build-detail', kwargs={'pk': self.bo.pk})
def test_update_locked_build(self):
"""An update is rejected (409) if the build order is locked by a long-running process."""
release = threading.Event()
self.hold_row_lock(Build, self.bo.pk, release)
try:
with mock.patch.object(InvenTree.helpers_model, 'ROW_LOCK_TIMEOUT', 0.3):
response = self.client.patch(
self.url, {'title': 'Edited title'}, format='json'
)
self.assertEqual(response.status_code, 409)
self.assertIn('currently being updated', response.data['detail'])
# Reading the build order is not affected
self.assertEqual(self.client.get(self.url).status_code, 200)
finally:
release.set()
self.join_threads()
# Nothing was written
self.bo.refresh_from_db()
self.assertEqual(self.bo.title, 'Original title')
# The update succeeds once the lock has been released
response = self.client.patch(self.url, {'title': 'Edited title'}, format='json')
self.assertEqual(response.status_code, 200)
self.bo.refresh_from_db()
self.assertEqual(self.bo.title, 'Edited title')
def test_update_waits_for_lock(self):
"""An update which waits (briefly) for a lock is applied to the *latest* data.
The other process changes fields which are not otherwise protected: if the
build order had been read before the lock was acquired, the update
would write the stale values back.
"""
release = threading.Event()
self.hold_row_lock(
Build, self.bo.pk, release, priority=5, batch='ABC', completed=15
)
# Release the lock while the request is waiting for it
timer = threading.Timer(0.5, release.set)
timer.start()
response = self.client.patch(self.url, {'title': 'Edited title'}, format='json')
timer.join()
self.assertEqual(response.status_code, 200)
self.join_threads()
# The response reflects the latest data
self.assertEqual(response.data['priority'], 5)
self.assertEqual(response.data['completed'], 15)
self.bo.refresh_from_db()
self.assertEqual(self.bo.title, 'Edited title')
self.assertEqual(self.bo.priority, 5)
self.assertEqual(self.bo.batch, 'ABC')
self.assertEqual(self.bo.completed, 15)
def test_update_during_output_completion(self):
"""Editing a build order (via the API) while its outputs are being completed.
The request is rejected while the task holds the build order row,
and the 'completed' count is intact afterwards.
"""
paused = threading.Event()
resume = threading.Event()
original = Build.complete_build_output
def complete_and_pause(bo, *args, **kwargs):
result = original(bo, *args, **kwargs)
paused.set()
resume.wait(timeout=15)
return result
try:
with mock.patch.object(Build, 'complete_build_output', complete_and_pause):
self.run_thread(run_complete_outputs_task, self.bo, self.outputs)
self.assertTrue(paused.wait(timeout=10))
with mock.patch.object(
InvenTree.helpers_model, 'ROW_LOCK_TIMEOUT', 0.3
):
response = self.client.patch(
self.url, {'title': 'Edited title'}, format='json'
)
self.assertEqual(response.status_code, 409)
resume.set()
self.join_threads()
finally:
resume.set()
response = self.client.patch(self.url, {'title': 'Edited title'}, format='json')
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data['completed'], 20)
self.bo.refresh_from_db()
self.assertEqual(self.bo.title, 'Edited title')
self.assertEqual(self.bo.completed, 20)
def test_update_not_found(self):
"""Updating a build order which does not exist returns a 404."""
url = reverse('api-build-detail', kwargs={'pk': self.bo.pk + 1000})
response = self.client.patch(url, {'title': 'Edited title'}, format='json')
self.assertEqual(response.status_code, 404)
@skipUnlessDBFeature('has_select_for_update')
class ActionLockingAPITest(ThreadedTestMixin, TransactionTestCase):
"""Cross-transaction tests for "action" endpoints, which act upon an existing object.
These endpoints (e.g. "hold" a build order) load the object, and then save it again
in full. The row must be locked *before* the object is loaded, otherwise the save
reverts any change which was committed in the meantime.
"""
def setUp(self):
"""Create an authenticated API client."""
super().setUp()
user = get_user_model().objects.create(
username='locking-admin', is_superuser=True, is_staff=True
)
self.client = APIClient()
self.client.force_authenticate(user=user)
def post(self, url_name: str, pk: int, data=None):
"""POST to the named action endpoint."""
url = reverse(url_name, kwargs={'pk': pk})
return self.client.post(url, data or {}, format='json')
def assert_locked(self, model, pk, url_name: str, url_pk=None, data=None):
"""Assert that the action is rejected (409) while the specified row is locked."""
release = threading.Event()
self.hold_row_lock(model, pk, release)
try:
with mock.patch.object(InvenTree.helpers_model, 'ROW_LOCK_TIMEOUT', 0.3):
response = self.post(url_name, url_pk or pk, data)
finally:
release.set()
self.join_threads()
self.threads.clear()
self.assertEqual(response.status_code, 409, f'{url_name}: {response.data}')
self.assertIn('currently being updated', response.data['detail'])
def test_build_hold_locked(self):
"""A build order cannot be placed on hold while it is locked by another process."""
bo, _outputs = create_build()
self.assert_locked(Build, bo.pk, 'api-build-hold')
# Nothing was changed
bo.refresh_from_db()
self.assertEqual(bo.status, BuildStatus.PRODUCTION.value)
# The action succeeds once the lock has been released
response = self.post('api-build-hold', bo.pk)
self.assertEqual(response.status_code, 201)
bo.refresh_from_db()
self.assertEqual(bo.status, BuildStatus.ON_HOLD.value)
def test_build_hold_waits_for_lock(self):
"""An action which waits (briefly) for a lock is applied to the *latest* data.
The other process changes fields which are not otherwise protected: if the
build order had been read before the lock was acquired, the action
(which saves the entire build order) would write the stale values back.
"""
bo, _outputs = create_build()
release = threading.Event()
self.hold_row_lock(Build, bo.pk, release, priority=5, batch='ABC', completed=15)
# Release the lock while the request is waiting for it
timer = threading.Timer(0.5, release.set)
timer.start()
response = self.post('api-build-hold', bo.pk)
timer.join()
self.assertEqual(response.status_code, 201)
self.join_threads()
bo.refresh_from_db()
self.assertEqual(bo.status, BuildStatus.ON_HOLD.value)
self.assertEqual(bo.priority, 5)
self.assertEqual(bo.batch, 'ABC')
self.assertEqual(bo.completed, 15)
def test_build_hold_during_output_completion(self):
"""Placing a build order on hold while its outputs are being completed.
The request is rejected while the task holds the build order row,
and the 'completed' count is intact afterwards.
"""
bo, outputs = create_build()
paused = threading.Event()
resume = threading.Event()
original = Build.complete_build_output
def complete_and_pause(build_order, *args, **kwargs):
result = original(build_order, *args, **kwargs)
paused.set()
resume.wait(timeout=15)
return result
try:
with mock.patch.object(Build, 'complete_build_output', complete_and_pause):
self.run_thread(run_complete_outputs_task, bo, outputs)
self.assertTrue(paused.wait(timeout=10))
with mock.patch.object(
InvenTree.helpers_model, 'ROW_LOCK_TIMEOUT', 0.3
):
response = self.post('api-build-hold', bo.pk)
self.assertEqual(response.status_code, 409)
resume.set()
self.join_threads()
finally:
resume.set()
response = self.post('api-build-hold', bo.pk)
self.assertEqual(response.status_code, 201)
bo.refresh_from_db()
self.assertEqual(bo.status, BuildStatus.ON_HOLD.value)
self.assertEqual(bo.completed, 20)
def test_build_actions_locked(self):
"""Every build order action is rejected while the build order is locked."""
bo, _outputs = create_build()
for url_name in [
'api-build-issue',
'api-build-hold',
'api-build-cancel',
'api-build-finish',
'api-build-allocate',
'api-build-auto-allocate',
'api-build-consume',
'api-build-output-create',
'api-build-output-complete',
'api-build-output-scrap',
'api-build-output-delete',
]:
with self.subTest(url=url_name):
self.assert_locked(Build, bo.pk, url_name)
def test_barcode_assign(self):
"""Assigning (or removing) a barcode saves the entire object, so the row must be locked."""
bo, _outputs = create_build()
url = reverse('api-barcode-link')
data = {'barcode': 'locking-test-barcode', 'build': bo.pk}
release = threading.Event()
self.hold_row_lock(Build, bo.pk, release)
try:
with mock.patch.object(InvenTree.helpers_model, 'ROW_LOCK_TIMEOUT', 0.3):
response = self.client.post(url, data, format='json')
finally:
release.set()
self.join_threads()
self.threads.clear()
self.assertEqual(response.status_code, 409)
bo.refresh_from_db()
self.assertEqual(bo.barcode_data, '')
# Once the lock is available, the barcode is assigned to the *latest* data
for url_name, expected in [
('api-barcode-link', 'locking-test-barcode'),
('api-barcode-unlink', ''),
]:
completed = bo.completed + 5
release = threading.Event()
self.hold_row_lock(Build, bo.pk, release, completed=completed)
timer = threading.Timer(0.5, release.set)
timer.start()
response = self.client.post(reverse(url_name), data, format='json')
timer.join()
self.join_threads()
self.threads.clear()
self.assertEqual(response.status_code, 200, response.data)
bo.refresh_from_db()
self.assertEqual(bo.barcode_data, expected)
self.assertEqual(bo.completed, completed)
def test_build_action_not_found(self):
"""An action against a build order which does not exist returns a 404."""
response = self.post('api-build-hold', 9999)
self.assertEqual(response.status_code, 404)
def test_order_actions_locked(self):
"""Order actions are rejected while the order is locked."""
company = Company.objects.create(
name='Locking company', is_customer=True, is_supplier=True
)
orders = {
SalesOrder: (
SalesOrder.objects.create(reference='SO-9990', customer=company),
['api-so-hold', 'api-so-issue', 'api-so-cancel', 'api-so-complete'],
),
ReturnOrder: (
ReturnOrder.objects.create(reference='RMA-9990', customer=company),
['api-ro-hold', 'api-return-order-issue', 'api-return-order-cancel'],
),
TransferOrder: (
TransferOrder.objects.create(reference='TO-9990'),
['api-transfer-order-hold', 'api-transfer-order-issue'],
),
PurchaseOrder: (
PurchaseOrder.objects.create(reference='PO-9990', supplier=company),
['api-po-hold', 'api-po-issue', 'api-po-cancel', 'api-po-receive'],
),
}
for model, (order, url_names) in orders.items():
for url_name in url_names:
with self.subTest(url=url_name):
self.assert_locked(model, order.pk, url_name)
# A transition is applied to the latest data once the lock is available
po, _url_names = orders[PurchaseOrder]
release = threading.Event()
self.hold_row_lock(PurchaseOrder, po.pk, release, description='Changed')
timer = threading.Timer(0.5, release.set)
timer.start()
response = self.post('api-po-issue', po.pk)
timer.join()
self.join_threads()
self.assertEqual(response.status_code, 201)
po.refresh_from_db()
self.assertEqual(po.description, 'Changed')
self.assertEqual(po.status, 20)
def test_shipment_complete_locked(self):
"""A shipment cannot be completed while it is locked."""
company = Company.objects.create(name='Locking customer', is_customer=True)
order = SalesOrder.objects.create(reference='SO-9991', customer=company)
shipment = SalesOrderShipment.objects.create(order=order, reference='1')
self.assert_locked(SalesOrderShipment, shipment.pk, 'api-so-shipment-ship')
def test_stock_item_actions_locked(self):
"""Stock item actions are rejected while the stock item is locked."""
part = Part.objects.create(
name='Locking part', description='Part for locking tests'
)
item = StockItem.objects.create(part=part, quantity=10)
for url_name in [
'api-stock-item-serialize',
'api-stock-item-install',
'api-stock-item-uninstall',
'api-stock-item-convert',
]:
with self.subTest(url=url_name):
self.assert_locked(StockItem, item.pk, url_name)
@@ -7,6 +7,7 @@ from typing import Any
from django.apps import apps
from django.core.exceptions import ValidationError as DjangoValidationError
from django.db import transaction
from django.utils.translation import gettext_lazy as _
from drf_spectacular.utils import extend_schema
@@ -134,7 +135,15 @@ def transition_action(
input_serializer_class = serializer_class
def endpoint(self, request: Request, pk: str | None = None) -> Response:
instance = self.get_object()
# Lock the row before reading the instance (where the viewset supports it),
# as the transition saves the entire instance back to the database
with transaction.atomic():
get_object = getattr(self, 'get_object_for_update', self.get_object)
instance = get_object()
return run_transition(self, request, instance)
def run_transition(self, request: Request, instance) -> Response:
kwargs: dict[str, Any] = {}
context = self.get_serializer_context()
+24 -8
View File
@@ -52,6 +52,7 @@ from InvenTree.helpers_api import (
)
from InvenTree.helpers_model import construct_absolute_url, get_base_url
from InvenTree.mixins import (
ActionLockMixin,
CreateAPI,
ListAPI,
ListCreateAPI,
@@ -499,11 +500,14 @@ class PurchaseOrderViewSet(
)
def receive(self, request, pk=None):
"""API endpoint to receive stock items against a PurchaseOrder."""
self.get_order()
# The order row is locked while the items are received
with self.locked_action(models.PurchaseOrder, pk):
self.get_order()
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
items = serializer.save()
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
items = serializer.save()
queryset = stock_serializers.StockItemSerializer.annotate_queryset(items)
response = stock_serializers.StockItemSerializer(
@@ -1175,11 +1179,14 @@ class SalesOrderExtraLineDetail(RetrieveUpdateDestroyAPI):
serializer_class = serializers.SalesOrderExtraLineSerializer
class SalesOrderContextMixin:
class SalesOrderContextMixin(ActionLockMixin):
"""Mixin to add sales order object as serializer context variable."""
queryset = models.SalesOrder.objects.all()
# The order row is locked for the duration of the action
lock_model = models.SalesOrder
def get_order(self):
"""Return the SalesOrder object associated with this API endpoint."""
pk = self.kwargs.get('pk', None)
@@ -1584,10 +1591,13 @@ class SalesOrderShipmentDetail(SalesOrderShipmentMixin, RetrieveUpdateDestroyAPI
"""API detail endpoint for SalesOrderShipment model."""
class SalesOrderShipmentComplete(CreateAPI):
class SalesOrderShipmentComplete(ActionLockMixin, CreateAPI):
"""API endpoint for completing (shipping) a SalesOrderShipment."""
queryset = models.SalesOrderShipment.objects.all()
# The shipment row is locked for the duration of the action
lock_model = models.SalesOrderShipment
serializer_class = serializers.SalesOrderShipmentCompleteSerializer
def get_shipment(self):
@@ -1766,11 +1776,14 @@ class ReturnOrderDetail(ReturnOrderMixin, OutputOptionsMixin, RetrieveUpdateDest
output_options = ReturnOrderOutputOptions
class ReturnOrderContextMixin:
class ReturnOrderContextMixin(ActionLockMixin):
"""Simple mixin class to add a ReturnOrder to the serializer context."""
queryset = models.ReturnOrder.objects.all()
# The order row is locked for the duration of the action
lock_model = models.ReturnOrder
def get_order(self):
"""Return the ReturnOrder object associated with this API endpoint."""
pk = self.kwargs.get('pk', None)
@@ -2075,11 +2088,14 @@ class TransferOrderDetail(
# output_options = TransferOrderOutputOptions
class TransferOrderContextMixin:
class TransferOrderContextMixin(ActionLockMixin):
"""Simple mixin class to add a TransferOrder to the serializer context."""
queryset = models.TransferOrder.objects.all()
# The order row is locked for the duration of the action
lock_model = models.TransferOrder
def get_order(self):
"""Return the TransferOrder object associated with this API endpoint."""
pk = self.kwargs.get('pk', None)
@@ -1,5 +1,6 @@
"""API endpoints for barcode plugins."""
from django.db import transaction
from django.db.models import F
from django.urls import include, path
from django.utils.translation import gettext_lazy as _
@@ -22,7 +23,7 @@ from InvenTree.api import BulkDeleteMixin
from InvenTree.exceptions import log_error
from InvenTree.filters import SEARCH_ORDER_FILTER
from InvenTree.helpers import hash_barcode
from InvenTree.mixins import ListAPI, RetrieveDestroyAPI
from InvenTree.mixins import ListAPI, RetrieveDestroyAPI, RowLockMixin
from plugin import PluginMixinEnum, registry
from users.permissions import check_user_permission
@@ -285,7 +286,7 @@ class BarcodeGenerate(CreateAPIView):
return Response({'barcode': barcode_data}, status=status.HTTP_200_OK)
class BarcodeAssign(BarcodeView):
class BarcodeAssign(RowLockMixin, BarcodeView):
"""Endpoint for assigning a barcode to a stock item.
- This only works if the barcode is not already associated with an object in the database
@@ -330,7 +331,12 @@ class BarcodeAssign(BarcodeView):
'error': f'You do not have the required permissions for {model}'
})
instance.assign_barcode(barcode_data=barcode, barcode_hash=barcode_hash)
# Lock (and reload) the instance, as assigning the barcode saves it in full
with transaction.atomic():
instance = self.lock_instance(instance)
instance.assign_barcode(
barcode_data=barcode, barcode_hash=barcode_hash
)
return Response({
'success': f'Assigned barcode to {label} instance',
@@ -345,7 +351,7 @@ class BarcodeAssign(BarcodeView):
})
class BarcodeUnassign(BarcodeView):
class BarcodeUnassign(RowLockMixin, BarcodeView):
"""Endpoint for unlinking / unassigning a custom barcode from a database object."""
serializer_class = barcode_serializers.BarcodeUnassignSerializer
@@ -389,7 +395,10 @@ class BarcodeUnassign(BarcodeView):
})
# Unassign the barcode data from the model instance
instance.unassign_barcode()
# Lock (and reload) the instance, as this saves it in full
with transaction.atomic():
instance = self.lock_instance(instance)
instance.unassign_barcode()
return Response({
'success': f'Barcode unassigned from {label} instance'
+8 -2
View File
@@ -45,6 +45,7 @@ from InvenTree.filters import (
)
from InvenTree.helpers import extract_serial_numbers, generateTestKey, str2bool
from InvenTree.mixins import (
ActionLockMixin,
CreateAPI,
CustomRetrieveUpdateDestroyAPI,
ListAPI,
@@ -110,11 +111,16 @@ class GenerateSerialNumber(GenericAPIView):
return Response(data, status=status.HTTP_201_CREATED)
class StockItemContextMixin:
"""Mixin class for adding StockItem object to serializer context."""
class StockItemContextMixin(ActionLockMixin):
"""Mixin class for adding StockItem object to serializer context.
The stock item row is locked for the duration of the action (see ActionLockMixin).
"""
role_required = 'stock.change'
lock_model = StockItem
queryset = StockItem.objects.none()
def get_serializer_context(self):
@@ -763,6 +763,15 @@ export function ApiForm({
invalidResponse(error.response.status);
props.onFormError?.(error, form);
break;
case 409:
// The item is locked by another process (e.g. a background task),
// and the request was not applied - the user can retry
setNonFieldErrors([
error.response.data?.detail ??
t`This item is being updated by another process. Please try again.`
]);
props.onFormError?.(error, form);
break;
default:
// Unexpected state on form error
invalidResponse(error.response.status);