diff --git a/CHANGELOG.md b/CHANGELOG.md index 17d0b40e45..8db8b4469e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Breaking Changes +- [#13034](https://github.com/inventree/InvenTree/pull/13034) moves pricing models into a new dedicated "pricing" model, and provides new API endpoints for managing pricing separately from other models. + ### Added +- [#13069](https://github.com/inventree/InvenTree/pull/13069) Adds an audit log which tracks changes made to internal data and exposes read-only API endpoints (/api/audit/) for accessing the audit log. + ### Changed ### Removed diff --git a/docs/docs/api/metadata.md b/docs/docs/api/metadata.md index 792bc547e7..e7abcf9717 100644 --- a/docs/docs/api/metadata.md +++ b/docs/docs/api/metadata.md @@ -28,6 +28,7 @@ The `OPTIONS` endpoint provides the following information: | name | The human-readable name of the API endpoint | | description | Descriptive detail for the endpoint, extracted from the python docstring | | actions | Contains the available HTTP actions and field information (see below) | +| filters | For list endpoints, the available choices for any choice-based query filters (e.g. `model_type` for the audit log). Only filters with a fixed set of choices are listed - filters on related models are not | Specific details are provided on the available attributes of each field: diff --git a/docs/docs/assets/images/concepts/audit_admin_center.png b/docs/docs/assets/images/concepts/audit_admin_center.png new file mode 100644 index 0000000000..17410d9860 Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_admin_center.png differ diff --git a/docs/docs/assets/images/concepts/audit_admin_settings.png b/docs/docs/assets/images/concepts/audit_admin_settings.png new file mode 100644 index 0000000000..3ffbbd16ae Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_admin_settings.png differ diff --git a/docs/docs/assets/images/concepts/audit_filter.png b/docs/docs/assets/images/concepts/audit_filter.png new file mode 100644 index 0000000000..115cbb9122 Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_filter.png differ diff --git a/docs/docs/assets/images/concepts/audit_part_history.png b/docs/docs/assets/images/concepts/audit_part_history.png new file mode 100644 index 0000000000..7ce5d67d91 Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_part_history.png differ diff --git a/docs/docs/assets/images/concepts/audit_po_history.png b/docs/docs/assets/images/concepts/audit_po_history.png new file mode 100644 index 0000000000..c5609a9fab Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_po_history.png differ diff --git a/docs/docs/assets/images/concepts/audit_user_setting.png b/docs/docs/assets/images/concepts/audit_user_setting.png new file mode 100644 index 0000000000..cdcae45276 Binary files /dev/null and b/docs/docs/assets/images/concepts/audit_user_setting.png differ diff --git a/docs/docs/concepts/audit.md b/docs/docs/concepts/audit.md new file mode 100644 index 0000000000..1a8f409237 --- /dev/null +++ b/docs/docs/concepts/audit.md @@ -0,0 +1,112 @@ +--- +title: Audit Log +--- + +## Audit Log + +InvenTree keeps an *audit log* (change history) of the changes made to key database models. Each time a supported item is created, updated or deleted, an entry is written to the audit log, recording: + +- **When** the change was made +- **Who** made the change (the user account associated with the request) +- **What** was changed (the item, and the old and new values of each modified field) +- **How** the change was made (the source of the change, e.g. an API request or a background task) + +The audit log is useful for tracing the history of an item, understanding *why* a value is what it is, and identifying which user made a particular change. + +!!! info "Version Information" + Audit history is not recorded prior to InvenTree version 1.7.0. Any changes made before upgrading to version 1.7.0 will not appear in the audit log. + +### How Auditing Works + +Auditing in InvenTree is implemented using the [django-auditlog](https://github.com/jazzband/django-auditlog) library. Changes are captured at the database model level, so they are recorded regardless of how the change was made - whether via the user interface, the [REST API](../api/index.md), the Python bindings, or a plugin. + +Some notes on what is recorded: + +- **Field changes** - For updates, only the fields which have actually changed are recorded. Saving an item without changing any values does not create an audit entry. +- **Parent linkage** - Many audited models are linked to a *parent* item. For example, a purchase order line item is linked to its purchase order, and a BOM item is linked to its assembly part. Changes to child items are displayed in the history of the parent item. +- **Files** - For file fields (such as attachments), only the filename is recorded, not the file contents. +- **Request information** - Each entry records the user who made the change, along with the remote IP address and a correlation ID which links together all changes made in a single request. The username is retained, even if the user account is later deleted. +- **Background tasks** - Changes made by background tasks are recorded against the name of the task which made the change. +- **System settings** - Changes to [global settings](../settings/global.md) are recorded (and are only visible to staff users). Internal settings which are updated automatically by the system are not recorded, and the values of protected settings are never stored. +- **Audit settings** - Changes to the audit log settings are *always* recorded, even if the audit log is disabled. This ensures that disabling the audit log (or adjusting the retention period) cannot go unnoticed. + +#### What Is Not Recorded + +To keep the audit log focused and performant, some changes are intentionally *not* recorded: + +- **Derived fields** - Internal or calculated fields (such as tree structure fields, cached pricing values and metadata) are excluded. +- **Data import** - Changes made using the [data import](./data_import.md) tool are not recorded. +- **Stock items** - Changes to stock items are tracked separately using [stock tracking](../stock/tracking.md). +- **Unsupported models** - Only the models listed below are audited. + +### Audited Models + +The following models are currently audited. Where a model has a *parent*, changes to that model are also displayed in the history of the parent item. + +{{ audited_models() }} + +### Viewing Change History + +#### Detail Pages + +When enabled, a *History* panel is displayed on the detail page for supported items (such as parts, companies and orders). This panel shows all changes made to the item, including changes made to any linked child items. + +{{ image("concepts/audit_part_history.png", "Part change history") }} + +For example, the history panel for a purchase order shows changes made to the order itself, as well as changes to its line items: + +{{ image("concepts/audit_po_history.png", "Purchase order change history") }} + +The *History* panel is only displayed if **all** of the following conditions are met: + +- The audit log is enabled (via the `Enable Audit Log` system setting) +- The user is a *staff* user +- The user has enabled the *Show Change History* user setting + +{{ image("concepts/audit_user_setting.png", "Show change history setting") }} + +#### Admin Center + +Staff users can view the complete audit log in the *Audit Log* panel of the [Admin Center](../settings/admin.md). This panel shows all recorded changes across all audited models, and provides access to the audit log settings. + +{{ image("concepts/audit_admin_center.png", "Audit log in the admin center") }} + +The audit log table can be filtered by action, user, date range and model type: + +{{ image("concepts/audit_filter.png", "Filter by model type") }} + +### Permissions + +Access to the audit log is restricted: + +- The audit log (including the `/api/audit/` API endpoint) is only available to *staff* users. +- Audit entries are additionally filtered by the user's [role permissions](../settings/permissions.md). A user can only view audit entries for models which they have permission to view. +- Some models (e.g. attachments and parameters) require the user to also have permission to view the *parent* model. +- The remote IP address associated with each entry is only visible to staff users. + +### Data Retention + +By default, audit log entries are kept indefinitely. Over time, the audit log may grow to a significant size. InvenTree can be configured to automatically delete old audit log entries, via a scheduled task which runs daily. + +When deleting old entries, the *Retain Audit Log Summary* setting can be used to keep a minimal history for each item - the creation entry, and the most recent change - while still removing older intermediate entries. + +### Settings + +The following settings are available to control the behavior of the audit log, and can be adjusted in the *Audit Log* panel of the Admin Center: + +{{ image("concepts/audit_admin_settings.png", "Audit log settings") }} + +#### System Settings + +| Name | Description | Default | Units | +| ---- | ----------- | ------- | ----- | +{{ globalsetting("AUDIT_LOG_ENABLED") }} +{{ globalsetting("AUDIT_LOG_DELETE_OLD_ENTRIES") }} +{{ globalsetting("AUDIT_LOG_DELETE_DAYS") }} +{{ globalsetting("AUDIT_LOG_DELETE_KEEP_SUMMARY") }} + +#### User Settings + +| Name | Description | Default | Units | +| ---- | ----------- | ------- | ----- | +{{ usersetting("DISPLAY_HISTORY_TAB") }} diff --git a/docs/docs/settings/error_codes.md b/docs/docs/settings/error_codes.md index fbd4d8f9ec..d26bd1cbd6 100644 --- a/docs/docs/settings/error_codes.md +++ b/docs/docs/settings/error_codes.md @@ -255,6 +255,13 @@ CORS settings only affect user browsers that respect them. Setting them correctl Use the INVENTREE_CORS_ORIGIN_ALLOW_ALL and INVENTREE_CORS_ORIGIN_WHITELIST settings - see the [Server Access settings](../start/config.md#server-access). +#### INVE-W17 +**Audit log middleware missing from configuration - Backend** + +The `middleware` list provided in the configuration file does not include the audit log middleware (`audit.middleware.InvenTreeAuditlogMiddleware`). This middleware is required for audit log entries to record which user made each change. + +The middleware has been added automatically. To remove this warning, add it to the `middleware` list in your configuration file (after `django_structlog.middlewares.RequestMiddleware`). + ### INVE-I (InvenTree Information) Information — These are not errors but information messages. They might point out potential issues or just provide information. diff --git a/docs/main.py b/docs/main.py index b7aed84775..24450cb288 100644 --- a/docs/main.py +++ b/docs/main.py @@ -91,6 +91,9 @@ with open(gen_base.joinpath('inventree_report_context.json'), encoding='utf-8') # User permission roles with open(gen_base.joinpath('inventree_roles.json'), encoding='utf-8') as f: ROLES = json.load(f) +# Audited models +with open(gen_base.joinpath('inventree_audited_models.json'), encoding='utf-8') as f: + AUDITED_MODELS = json.load(f) def get_repo_url(raw=False): @@ -495,6 +498,32 @@ def define_env(env): return ret_data + @env.macro + def audited_models(): + """Render a markdown table of the models which are recorded in the audit log. + + The table is built directly from `docs/generated/inventree_audited_models.json` + (produced by the `export_audited_models` management command, sourced from + the models which implement `InvenTreeAuditMixin`), so it can never drift out + of sync with the models actually audited in the source code. + """ + global AUDITED_MODELS + + ret_data = '| Category | Model | Model Type | Parent | Notes |\n' + ret_data += '| --- | --- | --- | --- | --- |\n' + + for model in AUDITED_MODELS: + parent = model['parent'] + notes = model['parent_note'] + + if fields := model['include_fields']: + fields = ', '.join(f'`{field}`' for field in fields) + notes = f'Only the following fields are recorded: {fields}' + + ret_data += f'| {model["app"]} | {model["name"]} | `{model["model_type"]}` | {parent} | {notes} |\n' + + return ret_data + @env.macro def report_context(type_: Literal['models', 'base'], model: str): """Extract information on a particular report context.""" diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 2dd93db915..d89068ca31 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -109,6 +109,7 @@ nav: - Parameters: concepts/parameters.md - Notes: concepts/notes.md - Tags: concepts/tags.md + - Audit Log: concepts/audit.md - Barcodes: - Barcode Support: barcodes/index.md - Internal Barcodes: barcodes/internal.md diff --git a/src/backend/InvenTree/InvenTree/api.py b/src/backend/InvenTree/InvenTree/api.py index c00f2292b3..d68a4ed2a9 100644 --- a/src/backend/InvenTree/InvenTree/api.py +++ b/src/backend/InvenTree/InvenTree/api.py @@ -29,6 +29,7 @@ import InvenTree.config import InvenTree.filters import InvenTree.permissions import InvenTree.version +from audit.helpers import write_batched from common.settings import get_global_setting from InvenTree import helpers, ready from InvenTree.auth_overrides import registration_enabled @@ -534,7 +535,12 @@ class BulkCreateMixin: if has_unique_errors: raise ValidationError(unique_errors) - with transaction.atomic(), batch_events(), batch_offload_tasks(): + with ( + transaction.atomic(), + batch_events(), + batch_offload_tasks(), + write_batched(), + ): for item in data: serializer = self.get_serializer(data=item) if serializer.is_valid(): diff --git a/src/backend/InvenTree/InvenTree/api_version.py b/src/backend/InvenTree/InvenTree/api_version.py index 241a3a3ac9..be5ad33454 100644 --- a/src/backend/InvenTree/InvenTree/api_version.py +++ b/src/backend/InvenTree/InvenTree/api_version.py @@ -1,12 +1,16 @@ """InvenTree API version information.""" # InvenTree API version -INVENTREE_API_VERSION = 564 +INVENTREE_API_VERSION = 565 """Increment this API version number whenever there is a significant change to the API that any clients need to know about.""" INVENTREE_API_TEXT = """ -v564 -> 2026-10-24 : https://github.com/inventree/InvenTree/pull/xxxxx +v565 -> 2026-10-24 : https://github.com/inventree/InvenTree/pull/13069 + - Adds read-only audit log API endpoints (/api/audit/) + - OPTIONS metadata includes a "filters" entry, listing the available choices for choice-based list filters + +v564 -> 2026-10-24 : https://github.com/inventree/InvenTree/pull/13090 - BomItem "can_build", "pricing_min_total" and "pricing_max_total" fields now account for "piece_count" - BuildLine "unit_quantity" ordering now accounts for the BOM "piece_count" diff --git a/src/backend/InvenTree/InvenTree/config.py b/src/backend/InvenTree/InvenTree/config.py index 7cf2ee5191..7ac1fe50db 100644 --- a/src/backend/InvenTree/InvenTree/config.py +++ b/src/backend/InvenTree/InvenTree/config.py @@ -58,6 +58,41 @@ def to_dict(value): return {} +def ensure_middleware( + middleware: list[str], required: str, after: Optional[str] = None +) -> list[str]: + """Ensure that a required middleware class is present in the provided MIDDLEWARE list. + + The MIDDLEWARE setting can be overridden via the configuration file, which may omit + middleware which is required for correct operation. Any missing middleware is added + (and a warning logged), rather than preventing the server from starting. + + Arguments: + middleware: The configured list of middleware classes + required: The middleware class which must be present + after: Insert after this middleware class (if present), otherwise append + + Returns: + A list of middleware classes, including the required middleware + """ + middleware = list(middleware) + + if required in middleware: + return middleware + + logger.warning( + 'INVE-W17: %s missing from MIDDLEWARE configuration - adding automatically', + required, + ) + + if after and after in middleware: + middleware.insert(middleware.index(after) + 1, required) + else: + middleware.append(required) + + return middleware + + def is_true(x): """Shortcut function to determine if a value "looks" like a boolean.""" return str(x).strip().lower() in ['1', 'y', 'yes', 't', 'true', 'on'] diff --git a/src/backend/InvenTree/InvenTree/management/commands/export_audited_models.py b/src/backend/InvenTree/InvenTree/management/commands/export_audited_models.py new file mode 100644 index 0000000000..3ea70841ab --- /dev/null +++ b/src/backend/InvenTree/InvenTree/management/commands/export_audited_models.py @@ -0,0 +1,83 @@ +"""Custom management command to export the models which are recorded in the audit log. + +This is used to generate a JSON file which contains all of the audited models +(those which implement the InvenTreeAuditMixin), so that they can be introspected +by the InvenTree documentation system. This allows the audited models table to be +documented without having to manually duplicate the information (which otherwise +silently drifts out of sync with the source code). +""" + +import inspect +import json + +from django.apps import apps +from django.core.management.base import BaseCommand +from django.utils.text import capfirst + + +class Command(BaseCommand): + """Extract audited model information, and export to a JSON file.""" + + def add_arguments(self, parser): + """Add custom arguments for this command.""" + parser.add_argument( + 'filename', type=str, help='Output filename for audited model definitions' + ) + + def handle(self, *args, **kwargs): + """Export audited model information to a JSON file.""" + models = discover_audited_models() + + filename = kwargs.get('filename', 'inventree_audited_models.json') + + with open(filename, 'w', encoding='utf-8') as f: + json.dump(models, f, indent=4) + + print(f"Exported InvenTree audited model definitions to '{filename}'") + + +def audit_parent(model) -> tuple[str, str]: + """Return a (parent, note) description of the "parent" object for audit entries against a model. + + - If the parent is a foreign key field, the name of the related model is returned + - If the parent reference is resolved by a custom method, the method docstring summary is returned as a note + """ + from InvenTree.models import InvenTreeAuditMixin + + if model.AUDIT_PARENT_FIELD: + field = model._meta.get_field(model.AUDIT_PARENT_FIELD) + + if field.related_model: + return capfirst(str(field.related_model._meta.verbose_name)), '' + + method = model.get_audit_parent_reference + + if method is not InvenTreeAuditMixin.get_audit_parent_reference: + doc = inspect.getdoc(method) or '' + return '', doc.split('\n\n')[0].replace('\n', ' ').strip() + + return '', '' + + +def discover_audited_models(): + """Discover all models which are recorded in the audit log. + + Returns a list of models, sorted by app and then by model name. + """ + from audit.registry import audited_models + + models = [] + + for model in audited_models(): + parent, parent_note = audit_parent(model) + + models.append({ + 'model_type': model._meta.label_lower, + 'app': str(apps.get_app_config(model._meta.app_label).verbose_name), + 'name': capfirst(str(model._meta.verbose_name)), + 'parent': parent, + 'parent_note': parent_note, + 'include_fields': sorted(getattr(model, 'AUDIT_INCLUDE_FIELDS', ())), + }) + + return sorted(models, key=lambda m: (m['app'], m['name'])) diff --git a/src/backend/InvenTree/InvenTree/metadata.py b/src/backend/InvenTree/InvenTree/metadata.py index a7a2768984..576c849150 100644 --- a/src/backend/InvenTree/InvenTree/metadata.py +++ b/src/backend/InvenTree/InvenTree/metadata.py @@ -168,8 +168,74 @@ class InvenTreeMetadata(SimpleMetadata): # then we don't need a permission pass + if filters := self.get_filter_metadata(request, view): + metadata['filters'] = filters + return metadata + def get_filter_metadata(self, request, view) -> dict: + """Return information on the choice filters available for an API endpoint. + + For each ChoiceFilter (or MultipleChoiceFilter) defined by the view's filterset, + the available choices are provided, e.g. so that a client can render a filter selection. + Model-backed choice filters (ModelChoiceFilter etc) are excluded. + + The filterset is instantiated with the current request, so that the + choices can be specific to the requesting user. + + Returns: + dict: A mapping of filter name -> {'type', 'label', 'choices'} + """ + from django_filters import ( + ChoiceFilter, + ModelChoiceFilter, + ModelMultipleChoiceFilter, + MultipleChoiceFilter, + ) + + filterset_class = getattr(view, 'filterset_class', None) + + if filterset_class is None: + return {} + + try: + filterset = filterset_class( + data={}, queryset=view.get_queryset(), request=request + ) + except Exception: + # Filter information is optional - never prevent the OPTIONS response + logger.exception( + 'Could not construct filterset for %s', view.__class__.__name__ + ) + return {} + + filters = {} + + for name, filter_instance in filterset.filters.items(): + if not isinstance(filter_instance, (ChoiceFilter, MultipleChoiceFilter)): + continue + + # Model-backed choices (e.g. foreign key filters) are not exposed: + # these may be unbounded, and would expose related objects to the client + if isinstance( + filter_instance, (ModelChoiceFilter, ModelMultipleChoiceFilter) + ): + continue + + choices = [ + {'value': value, 'display_name': str(label)} + for value, label in filter_instance.field.choices + if value not in ('', None) + ] + + filters[name] = { + 'type': 'choice', + 'label': str(filter_instance.label or name), + 'choices': choices, + } + + return filters + def override_value(self, field_name: str, field_key: str, field_value, model_value): """Override a value on the serializer with a matching value for the model. diff --git a/src/backend/InvenTree/InvenTree/models.py b/src/backend/InvenTree/InvenTree/models.py index bc67bbde7c..7f880c053b 100644 --- a/src/backend/InvenTree/InvenTree/models.py +++ b/src/backend/InvenTree/InvenTree/models.py @@ -532,6 +532,93 @@ class InvenTreePermissionCheckMixin: return user.has_perm(perm) +class InvenTreeAuditMixin: + """Opt a model in to the audit log (field-level change history). + + Any concrete model which inherits from this mixin is registered with django-auditlog + when the 'audit' app is loaded (see audit.registry). + + Attributes: + AUDIT_INCLUDE_FIELDS: If specified, *only* these fields are recorded (e.g. user-editable fields of a cache model) + AUDIT_EXCLUDE_FIELDS: Fields which are never recorded (merged with a base set, see audit.registry) + AUDIT_MASK_FIELDS: Fields whose values are masked (the change is recorded, the value is not) + AUDIT_M2M_FIELDS: Many-to-many fields which are tracked + AUDIT_PARENT_FIELD: Name of the field which links to the "parent" object (e.g. 'order' for a line item) + AUDIT_VIEW_RESTRICTION: Restrict reading audit history to 'staff' or 'superuser' users (or None) + AUDIT_REQUIRE_PARENT_PERMISSION: Entries are only visible to users who can view the *parent* model + (for models whose access is determined by the linked object, e.g. attachments) + """ + + AUDIT_INCLUDE_FIELDS: tuple[str, ...] = () + AUDIT_EXCLUDE_FIELDS: tuple[str, ...] = () + AUDIT_MASK_FIELDS: tuple[str, ...] = () + AUDIT_M2M_FIELDS: tuple[str, ...] = () + AUDIT_PARENT_FIELD: Optional[str] = None + AUDIT_VIEW_RESTRICTION: Optional[str] = None + AUDIT_REQUIRE_PARENT_PERMISSION: bool = False + + def get_audit_parent(self): + """Return the "parent" object for audit log entries against this instance. + + Audit entries linked to a parent are included in the history of that parent + (e.g. changes to line items are shown against the order). + """ + if self.AUDIT_PARENT_FIELD: + return getattr(self, self.AUDIT_PARENT_FIELD, None) + + return None + + def get_audit_parent_reference(self) -> Optional[tuple[Any, str]]: + """Return a (ContentType, pk) reference to the "parent" object (or None). + + Override this (rather than get_audit_parent) where the parent can be identified + without fetching it from the database (e.g. generic model_type / model_id links). + """ + from django.contrib.contenttypes.models import ContentType + + # Resolve a foreign key parent from its ID, without fetching the related object + if self.AUDIT_PARENT_FIELD: + field = self._meta.get_field(self.AUDIT_PARENT_FIELD) + + if field.many_to_one or field.one_to_one: + parent_id = getattr(self, field.attname) + + if parent_id is None: + return None + + return ContentType.objects.get_for_model(field.related_model), str( + parent_id + ) + + if parent := self.get_audit_parent(): + return ContentType.objects.get_for_model(parent), str(parent.pk) + + return None + + def should_audit(self) -> bool: + """Return False to skip recording audit entries for this instance. + + Used where only some rows of a model are of interest + (e.g. settings which are written automatically by the system). + """ + return True + + def audit_always(self) -> bool: + """Return True to record audit entries for this instance, even if the audit log is disabled. + + Used for changes which must always be recorded (e.g. disabling the audit log itself). + """ + return False + + def is_audit_masked(self) -> bool: + """Return True if *all* values recorded against this instance must be masked. + + Used where sensitivity depends on the row rather than the field + (e.g. a 'protected' setting value). + """ + return False + + class InvenTreeParameterMixin(InvenTreePermissionCheckMixin, models.Model): """Provides an abstracted class for managing parameters. @@ -629,7 +716,9 @@ class InvenTreeParameterMixin(InvenTreePermissionCheckMixin, models.Model): parameters.append(parameter) if len(parameters) > 0: - common.models.Parameter.objects.bulk_create(parameters, batch_size=250) + from audit.helpers import audited_bulk_create + + audited_bulk_create(common.models.Parameter, parameters) def get_parameter(self, name: str): """Return a Parameter instance for the given parameter name. @@ -1000,7 +1089,12 @@ class InvenTreeTree(ContentTypeMixin, MPTTModel): B) delete_children = True and delete_items = False C) delete_children = False and delete_items = True D) delete_children = False and delete_items = False + + Note: re-parenting uses audited_update(), so that the change is recorded + for audited models (it falls back to a plain update() otherwise). """ + from audit.helpers import audited_update + child_nodes = self.get_descendants(include_self=False) # Case A: Delete all child items, and all child nodes. @@ -1016,7 +1110,7 @@ class InvenTreeTree(ContentTypeMixin, MPTTModel): elif delete_children and not delete_items: if items := self.get_items(cascade=True): parent = getattr(self, self.NODE_PARENT_KEY, None) - items.update(**{self.ITEM_PARENT_KEY: parent}) + audited_update(items, **{self.ITEM_PARENT_KEY: parent}) self.delete_nodes(child_nodes) # Case C: Delete all child items, but keep all child nodes @@ -1025,7 +1119,7 @@ class InvenTreeTree(ContentTypeMixin, MPTTModel): elif not delete_children and delete_items: self.delete_items(cascade=False) parent = getattr(self, self.NODE_PARENT_KEY, None) - self.get_children().update(**{self.NODE_PARENT_KEY: parent}) + audited_update(self.get_children(), **{self.NODE_PARENT_KEY: parent}) # Case D: Keep all child items, and keep all child nodes # - Move all items directly associated with this node up one level @@ -1033,8 +1127,8 @@ class InvenTreeTree(ContentTypeMixin, MPTTModel): elif not delete_children and not delete_items: parent = getattr(self, self.NODE_PARENT_KEY, None) if items := self.get_items(cascade=False): - items.update(**{self.ITEM_PARENT_KEY: parent}) - self.get_children().update(**{self.NODE_PARENT_KEY: parent}) + audited_update(items, **{self.ITEM_PARENT_KEY: parent}) + audited_update(self.get_children(), **{self.NODE_PARENT_KEY: parent}) def delete_nodes(self, nodes): """Delete a set of nodes from the tree. diff --git a/src/backend/InvenTree/InvenTree/settings.py b/src/backend/InvenTree/InvenTree/settings.py index 030411ca66..0dedd57c3e 100644 --- a/src/backend/InvenTree/InvenTree/settings.py +++ b/src/backend/InvenTree/InvenTree/settings.py @@ -25,7 +25,12 @@ from corsheaders.defaults import default_headers as default_cors_headers import InvenTree.backup from InvenTree.cache import get_cache_config, is_global_cache_enabled -from InvenTree.config import get_boolean_setting, get_oidc_private_key, get_setting +from InvenTree.config import ( + ensure_middleware, + get_boolean_setting, + get_oidc_private_key, + get_setting, +) from InvenTree.ready import isInMainThread, isRunningBackup from InvenTree.sentry import default_sentry_dsn, init_sentry from InvenTree.validators import invalid_site_url_hint @@ -369,6 +374,10 @@ INSTALLED_APPS = [ 'django_mailbox', # For email import 'anymail', # For email sending/receiving via ESPs 'storages', + 'auditlog', # Audit log / model change history (see 'audit' app) + # New InvenTree apps are appended here, so that existing ContentType IDs + # match between existing databases and freshly migrated (test) databases + 'audit.apps.AuditConfig', ] MIDDLEWARE = CONFIG.get( @@ -396,10 +405,33 @@ MIDDLEWARE = CONFIG.get( 'InvenTree.middleware.InvenTreeRequestCacheMiddleware', # Request caching 'InvenTree.middleware.InvenTreeHostSettingsMiddleware', # Ensuring correct hosting/security settings 'django_structlog.middlewares.RequestMiddleware', # Structured logging + 'audit.middleware.InvenTreeAuditlogMiddleware', # Audit log context (after maintenance + request ID) 'InvenTree.middleware.InvenTreeVersionHeaderMiddleware', ], ) +# The audit log middleware is required for audit entries to record the user (and request context). +# MIDDLEWARE may be overridden via the config file - ensure that it is always present +MIDDLEWARE = ensure_middleware( + MIDDLEWARE, + 'audit.middleware.InvenTreeAuditlogMiddleware', + after='django_structlog.middlewares.RequestMiddleware', +) + +# Audit log configuration (django-auditlog) +# Ref: https://django-auditlog.readthedocs.io/en/latest/usage.html#settings +# Note: AUDITLOG_LOGENTRY_MODEL is a swappable model, and must not be changed once migrations have been applied +AUDITLOG_LOGENTRY_MODEL = 'audit.AuditLogEntry' +AUDITLOG_STORE_JSON_CHANGES = True # Store values as JSON types, rather than strings +AUDITLOG_DISABLE_ON_RAW_SAVE = True # Do not log fixture loading / raw saves +AUDITLOG_CID_HEADER = None # Correlation ID is provided by AUDITLOG_CID_GETTER (below) +AUDITLOG_CID_GETTER = 'audit.context.get_cid' # Matches the structlog request ID + +# The upstream auditlog migrations only define the (swapped out) LogEntry model, +# and do not mark it as swappable - so would create an unused 'auditlog_logentry' table. +# Disable them entirely - the 'audit' app provides the migrations for AuditLogEntry. +MIGRATION_MODULES = {'auditlog': None} + # In DEBUG mode, add support for django-silk # Ref: https://silk.readthedocs.io/en/latest/ DJANGO_SILK_ENABLED = ( diff --git a/src/backend/InvenTree/InvenTree/tasks.py b/src/backend/InvenTree/InvenTree/tasks.py index 1f6bf19f7e..c21b538d67 100644 --- a/src/backend/InvenTree/InvenTree/tasks.py +++ b/src/backend/InvenTree/InvenTree/tasks.py @@ -515,9 +515,14 @@ def offload_task( ) return False + from audit.context import audit_task + # Workers are not running: run it as synchronous task try: - with tracer.start_as_current_span(f'sync worker: {taskname}'): + with ( + tracer.start_as_current_span(f'sync worker: {taskname}'), + audit_task(taskname), + ): _func(*args, **kwargs) except Exception as exc: log_error('offload_task', scope='worker') diff --git a/src/backend/InvenTree/InvenTree/urls.py b/src/backend/InvenTree/InvenTree/urls.py index 3b583f4dea..4b79f50110 100644 --- a/src/backend/InvenTree/InvenTree/urls.py +++ b/src/backend/InvenTree/InvenTree/urls.py @@ -16,6 +16,7 @@ from flags.urls import flagged_path from oauth2_provider import urls as oauth2_urls from sesame.views import LoginView +import audit.api import build.api import common.api import company.api @@ -55,6 +56,7 @@ admin.site.site_header = get_setting( apipatterns = [ # Global search path('admin/', include(common.api.admin_api_urls)), + path('audit/', include(audit.api.audit_api_urls)), path('bom/', include(part.api.bom_api_urls)), path('build/', include(build.api.build_api_urls)), path('company/', include(company.api.company_api_urls)), diff --git a/src/backend/InvenTree/audit/__init__.py b/src/backend/InvenTree/audit/__init__.py new file mode 100644 index 0000000000..2374b9cece --- /dev/null +++ b/src/backend/InvenTree/audit/__init__.py @@ -0,0 +1,5 @@ +"""The audit app provides a read-only change history for InvenTree models. + +Built on top of django-auditlog, with an InvenTree specific LogEntry model +(see audit.models.AuditLogEntry) which links entries to a "parent" object. +""" diff --git a/src/backend/InvenTree/audit/admin.py b/src/backend/InvenTree/audit/admin.py new file mode 100644 index 0000000000..c225c82713 --- /dev/null +++ b/src/backend/InvenTree/audit/admin.py @@ -0,0 +1,29 @@ +"""Admin site configuration for the audit app.""" + +from django.contrib import admin + +# Ensure the upstream admin class has been registered, so that it can be replaced below +import auditlog.admin +from auditlog.filters import CIDFilter, ResourceTypeFilter + +from audit.models import AuditLogEntry + +if admin.site.is_registered(AuditLogEntry): + admin.site.unregister(AuditLogEntry) + + +@admin.register(AuditLogEntry) +class AuditLogEntryAdmin(auditlog.admin.LogEntryAdmin): + """Admin class for the AuditLogEntry model. + + Extends the (read-only) upstream LogEntry admin, adding: + - Filtering by the user who made the change + """ + + list_filter = [ + 'action', + # Only list users which have recorded entries + ('actor', admin.RelatedOnlyFieldListFilter), + ResourceTypeFilter, + CIDFilter, + ] diff --git a/src/backend/InvenTree/audit/api.py b/src/backend/InvenTree/audit/api.py new file mode 100644 index 0000000000..c45aef7bdd --- /dev/null +++ b/src/backend/InvenTree/audit/api.py @@ -0,0 +1,217 @@ +"""API endpoints for the audit app.""" + +from django import forms +from django.db.models import Q +from django.urls import include, path +from django.utils.text import capfirst +from django.utils.translation import gettext_lazy as _ + +from django_filters import rest_framework as rest_filters +from drf_spectacular.types import OpenApiTypes +from drf_spectacular.utils import extend_schema_field +from rest_framework import viewsets +from rest_framework.exceptions import ValidationError + +import audit.serializers +from audit.models import AuditLogEntry +from audit.permissions import ( + get_viewable_audit_content_types, + parent_permission_content_types, + user_can_view_audit_model, +) +from common.filters import determine_content_type +from InvenTree.filters import SEARCH_ORDER_FILTER +from InvenTree.helpers_api import InvenTreeApiRouter +from InvenTree.permissions import IsAdminOrAdminScope + + +def audited_model_type_choices(user=None) -> list[tuple[str, str]]: + """Return (model_type, label) choices for the audited models. + + If a user is provided, only the models which the user may view are returned. + """ + from django.contrib.contenttypes.models import ContentType + + from audit.registry import audited_models + + allowed = get_viewable_audit_content_types(user) if user else None + + choices = [] + + for model in audited_models(): + ct = ContentType.objects.get_for_model(model) + + if allowed is not None and ct.pk not in allowed: + continue + + choices.append(( + f'{ct.app_label}.{ct.model}', + capfirst(str(model._meta.verbose_name)), + )) + + return sorted(choices, key=lambda choice: choice[1]) + + +class AnyValueChoiceField(forms.ChoiceField): + """A ChoiceField which provides a list of choices, but accepts any value. + + The choices are used to describe the common values (e.g. for display in a client), + while other supported formats are still accepted by the filter. + """ + + def valid_value(self, value): + """Accept any value.""" + return True + + +# Note: the choices are dynamic (and any value is accepted), so the schema type is declared explicitly +@extend_schema_field(OpenApiTypes.STR) +class ModelTypeChoiceFilter(rest_filters.ChoiceFilter): + """Choice filter for the model type of an audit entry. + + The choices are the audited model types ('app_label.model'), but any value supported + by common.filters.determine_content_type is accepted (e.g. a model name, or a ContentType ID). + """ + + field_class = AnyValueChoiceField + + +class AuditLogFilter(rest_filters.FilterSet): + """Custom filters for the AuditLogList API endpoint.""" + + class Meta: + """Metaclass options.""" + + model = AuditLogEntry + fields = ['action', 'actor', 'cid'] + + model_type = ModelTypeChoiceFilter( + choices=audited_model_type_choices, + method='filter_noop', + label=_('Model Type'), + help_text=_('Model type (e.g. "order.purchaseorder" or "purchaseorder")'), + ) + + model_id = rest_filters.CharFilter( + method='filter_noop', + label=_('Model ID'), + help_text=_('Primary key of the object (requires model_type)'), + ) + + include_children = rest_filters.BooleanFilter( + method='filter_noop', + label=_('Include Children'), + help_text=_('Include entries for child objects (default = true)'), + ) + + min_date = rest_filters.DateFilter( + field_name='timestamp', lookup_expr='date__gte', label=_('Min Date') + ) + + max_date = rest_filters.DateFilter( + field_name='timestamp', lookup_expr='date__lte', label=_('Max Date') + ) + + def __init__(self, *args, **kwargs): + """Limit the model type choices to those which the requesting user may view.""" + super().__init__(*args, **kwargs) + + user = getattr(self.request, 'user', None) + + if user and user.is_authenticated: + self.filters['model_type'].extra['choices'] = audited_model_type_choices( + user + ) + + def filter_noop(self, queryset, name, value): + """Combined filtering is performed in filter_queryset().""" + return queryset + + def filter_queryset(self, queryset): + """Filter by (model_type, model_id), optionally including child entries.""" + queryset = super().filter_queryset(queryset) + + model_type = self.form.cleaned_data.get('model_type') + model_id = self.form.cleaned_data.get('model_id') + include_children = self.form.cleaned_data.get('include_children') + + if include_children is None: + include_children = True + + if model_id and not model_type: + raise ValidationError({'model_id': _('model_type must also be provided')}) + + if not model_type: + return queryset + + ct = determine_content_type(model_type) + + if ct is None: + raise ValidationError({'model_type': _('Invalid model type')}) + + own = Q(content_type=ct) + children = Q(parent_content_type=ct) + + if model_id: + own &= Q(object_pk=str(model_id)) + children &= Q(parent_object_pk=str(model_id)) + + # Querying *through* a parent requires permission to view the parent model, + # otherwise the parent linkage of child objects would be exposed + user = getattr(self.request, 'user', None) + + if include_children and user_can_view_audit_model(user, ct.model_class()): + return queryset.filter(own | children) + + return queryset.filter(own) + + +class AuditLogViewSet(viewsets.ReadOnlyModelViewSet): + """API endpoint for audit log entries (read-only). + + - GET (list): Return a list of audit log entries + - GET (detail): Return a single audit log entry + """ + + queryset = AuditLogEntry.objects.all().select_related( + 'content_type', 'parent_content_type' + ) + serializer_class = audit.serializers.AuditLogEntrySerializer + + # The audit log is only available to staff users. + # Entries are *additionally* filtered by the user's permissions on each audited model. + permission_classes = [IsAdminOrAdminScope] + + filterset_class = AuditLogFilter + filter_backends = SEARCH_ORDER_FILTER + + ordering_fields = ['timestamp', 'action', 'actor'] + ordering = '-timestamp' + + search_fields = ['object_repr', 'actor__username'] + + def get_queryset(self): + """Restrict entries to those the requesting user is allowed to view.""" + queryset = super().get_queryset() + + allowed = get_viewable_audit_content_types(self.request.user) + + if allowed is None: + return queryset + + queryset = queryset.filter(content_type__in=allowed) + + # Some models (e.g. attachments) are only visible with permission on the parent model + if restricted := parent_permission_content_types(): + queryset = queryset.filter( + ~Q(content_type__in=restricted) | Q(parent_content_type__in=allowed) + ) + + # Note: otherwise parent permissions are handled by the filter (and serializer) + return queryset + + +audit_router = InvenTreeApiRouter() +audit_router.register('', AuditLogViewSet, basename='api-audit-log') + +audit_api_urls = [path('', include(audit_router.urls))] diff --git a/src/backend/InvenTree/audit/apps.py b/src/backend/InvenTree/audit/apps.py new file mode 100644 index 0000000000..c0b0b404d2 --- /dev/null +++ b/src/backend/InvenTree/audit/apps.py @@ -0,0 +1,21 @@ +"""AppConfig for the 'audit' app.""" + +from django.apps import AppConfig + + +class AuditConfig(AppConfig): + """AppConfig class for the 'audit' app.""" + + default_auto_field = 'django.db.models.BigAutoField' + name = 'audit' + + def ready(self): + """Register audited models with django-auditlog. + + Registration only connects model signals (no database access), + and must run in every process which can save models (server, worker, shell). + """ + import audit.signals # noqa: F401 - connect signal receivers + from audit.registry import register_audited_models + + register_audited_models() diff --git a/src/backend/InvenTree/audit/context.py b/src/backend/InvenTree/audit/context.py new file mode 100644 index 0000000000..2426d082aa --- /dev/null +++ b/src/backend/InvenTree/audit/context.py @@ -0,0 +1,165 @@ +"""Context tracking for audit log entries. + +django-auditlog resolves the "actor" eagerly in its middleware, which misses users +authenticated later by DRF (e.g. HTTP Basic auth). Instead, the current request is +stored here, and the actor is resolved lazily when a log entry is written. +DRF writes the authenticated user back to the underlying HttpRequest, so +request.user is correct by the time any model is saved. +""" + +import contextlib +import uuid +from contextvars import ContextVar +from typing import Optional + +import structlog + +_audit_request = ContextVar('inventree_audit_request', default=None) +_audit_cid = ContextVar('inventree_audit_cid', default=None) +_audit_source = ContextVar('inventree_audit_source', default=None) +_audit_batch = ContextVar('inventree_audit_batch', default=None) + + +@contextlib.contextmanager +def audit_batch(): + """Collect (unsaved) audit entries created within this context, rather than saving each one. + + Nested contexts share the outermost list. + + Yields: + The list of collected entries, which the caller is responsible for writing + (or None for a nested context, as there is nothing to write). + """ + if _audit_batch.get() is not None: + yield None + return + + entries = [] + token = _audit_batch.set(entries) + + try: + yield entries + finally: + _audit_batch.reset(token) + + +def get_batch() -> Optional[list]: + """Return the list of audit entries being collected, or None if not batching.""" + return _audit_batch.get() + + +@contextlib.contextmanager +def audit_request(request): + """Track the provided request (and a new correlation ID) for the duration of the context.""" + request_token = _audit_request.set(request) + cid_token = _audit_cid.set(uuid.uuid4().hex) + + try: + yield + finally: + _audit_request.reset(request_token) + _audit_cid.reset(cid_token) + + +@contextlib.contextmanager +def audit_source(source: str): + """Mark audit entries created within this context as originating from 'source'. + + Intended for changes made outside of a request (e.g. 'task:', 'command:'). + A new correlation ID is generated if one is not already active. + """ + source_token = _audit_source.set(source) + cid_token = _audit_cid.set(_audit_cid.get() or uuid.uuid4().hex) + + try: + yield + finally: + _audit_source.reset(source_token) + _audit_cid.reset(cid_token) + + +def task_name(func) -> str: + """Return the dotted name of a task function (which may be provided as a string).""" + if isinstance(func, str): + return func + + module = getattr(func, '__module__', None) + name = getattr(func, '__qualname__', None) or getattr(func, '__name__', None) + + if module and name: + return f'{module}.{name}' + + return repr(func) + + +@contextlib.contextmanager +def audit_task(func): + """Mark audit entries created within this context as originating from a background task. + + If a request is active (e.g. a task run synchronously as part of a request), + the changes are attributed to the request instead. + """ + if _audit_request.get() is not None: + yield + return + + with audit_source(f'task:{task_name(func)}'): + yield + + +# Context managers for tasks which are executed by the background worker +_worker_tasks: list = [] + + +def begin_worker_task(func) -> None: + """Enter the audit_task context for a task executed by the background worker. + + The worker provides separate signals before and after a task is executed, + so the context is entered (and exited) manually. + """ + context = audit_task(func) + context.__enter__() + _worker_tasks.append(context) + + +def end_worker_task() -> None: + """Exit the audit_task context for the most recent worker task.""" + if _worker_tasks: + _worker_tasks.pop().__exit__(None, None, None) + + +def get_cid() -> Optional[str]: + """Return the current correlation ID (used as AUDITLOG_CID_GETTER). + + - Within a request: the request ID bound by django-structlog, so that audit entries + can be matched against log entries (and the X-Request-ID header) for the request. + Note: django-structlog uses the incoming 'X-Request-ID' header if provided + (e.g. by a reverse proxy), otherwise generates a new UUID. + - Otherwise (e.g. background tasks, or if the structlog middleware is not installed): + a server-generated ID (see audit_request / audit_source). + """ + if _audit_source.get() is None and _audit_request.get() is not None: + if request_id := structlog.contextvars.get_contextvars().get('request_id'): + return str(request_id) + + return _audit_cid.get() + + +def get_source() -> Optional[str]: + """Return the current audit source, if set.""" + return _audit_source.get() + + +def get_request_actor(): + """Return the authenticated user for the current request, or None.""" + request = _audit_request.get() + + if request is None: + return None + + user = getattr(request, 'user', None) + + if user is not None and user.is_authenticated: + return user + + return None diff --git a/src/backend/InvenTree/audit/helpers.py b/src/backend/InvenTree/audit/helpers.py new file mode 100644 index 0000000000..dfcdfe6b03 --- /dev/null +++ b/src/backend/InvenTree/audit/helpers.py @@ -0,0 +1,256 @@ +"""Helper functions for recording audit entries for bulk database operations. + +QuerySet.update() (and bulk_create / bulk_update) bypass model signals, +so changes made this way are not recorded by django-auditlog. +Code which modifies audited models in bulk should use these helpers instead. +""" + +import contextlib + +from django.conf import settings +from django.core.exceptions import FieldDoesNotExist +from django.db import connection, transaction +from django.db.models.signals import pre_save + +from auditlog.context import auditlog_disabled +from auditlog.registry import auditlog + + +def is_audited(model) -> bool: + """Return True if changes to the given model are currently being audited.""" + try: + disabled = auditlog_disabled.get() + except LookupError: + disabled = False + + return not disabled and auditlog.contains(model) + + +@contextlib.contextmanager +def write_batched(): + """Write all audit entries created within this context with a single bulk insert. + + Avoids one INSERT query per modified row for the bulk helpers below. + Must be used within a transaction, so that the entries are written atomically with the changes. + + Note: post_log receivers are passed the (unsaved) log entry. + """ + from audit.context import audit_batch + from audit.models import AuditLogEntry + + with audit_batch() as entries: + yield + + if entries: + # Upstream applies context data (e.g. remote_addr) via a pre_save receiver, + # which bulk_create() does not trigger + for entry in entries: + pre_save.send( + sender=AuditLogEntry, + instance=entry, + raw=False, + using=AuditLogEntry.objects.db, + update_fields=None, + ) + + AuditLogEntry.objects.bulk_create(entries) + + +def copy_related_cache(source, target) -> None: + """Copy cached related objects from one instance of a model to another. + + Avoids re-fetching related objects (e.g. for the audit entry string representation) + for rows which have been re-read from the database. Only relations whose + foreign key value is unchanged between the two instances are copied. + """ + for name, related in source._state.fields_cache.items(): + try: + field = source._meta.get_field(name) + except FieldDoesNotExist: + continue + + attname = getattr(field, 'attname', None) + + if ( + attname + and (field.many_to_one or field.one_to_one) + and getattr(source, attname, None) == getattr(target, attname, None) + ): + target._state.fields_cache.setdefault(name, related) + + +def audited_update(queryset, **values) -> int: + """Perform QuerySet.update(), recording an audit entry for each modified row. + + Supports any value accepted by QuerySet.update() (including F() expressions), + as the new values are re-read from the database after the update. + + Arguments: + queryset: The queryset to update + values: The field values to update + + Returns: + int: The number of rows updated (as per QuerySet.update()) + """ + # Imported here to keep the dependency on this (private) upstream function in one place. + # It provides the same pre_log / diff / log_create / post_log path as the model signals. + # Pinned via requirements: django-auditlog>=3.4.1,<4 + from auditlog.receivers import _create_log_entry + + from audit.models import AuditLogEntry + + model = queryset.model + + if not is_audited(model): + return queryset.update(**values) + + manager = model._base_manager + + with transaction.atomic(): + if connection.features.has_select_for_update: + old_rows = list(queryset.select_for_update()) + else: + old_rows = list(queryset) + + if not old_rows: + return 0 + + pks = [row.pk for row in old_rows] + + # Update by PK - the original queryset may filter on the fields being updated + count = manager.filter(pk__in=pks).update(**values) + + new_rows = manager.in_bulk(pks) + + with write_batched(): + for old in old_rows: + if new := new_rows.get(old.pk): + _create_log_entry( + action=AuditLogEntry.Action.UPDATE, + instance=new, + sender=model, + diff_old=old, + diff_new=new, + fields_to_check=list(values.keys()), + use_json_for_changes=settings.AUDITLOG_STORE_JSON_CHANGES, + ) + + return count + + +def audited_bulk_update(model, objs: list, fields: list[str], **kwargs) -> int: + """Perform Model.objects.bulk_update(), recording an audit entry for each modified row. + + As with audited_update(), the old and new values are read from the database, + so the recorded changes reflect what was actually written (including F() expressions). + + Arguments: + model: The model class + objs: The (saved) model instances to update + fields: The fields to update + kwargs: Additional arguments passed through to bulk_update() (e.g. batch_size) + + Returns: + int: The number of rows updated (as per bulk_update()) + """ + from auditlog.receivers import _create_log_entry + + from audit.models import AuditLogEntry + + if not is_audited(model): + return model.objects.bulk_update(objs, fields, **kwargs) + + manager = model._base_manager + pks = [obj.pk for obj in objs] + + with transaction.atomic(): + if connection.features.has_select_for_update: + old_rows = manager.select_for_update().in_bulk(pks) + else: + old_rows = manager.in_bulk(pks) + + count = model.objects.bulk_update(objs, fields, **kwargs) + + new_rows = manager.in_bulk(pks) + + # Reuse any related objects already loaded by the caller + for obj in objs: + if new := new_rows.get(obj.pk): + copy_related_cache(obj, new) + + with write_batched(): + for pk, old in old_rows.items(): + if new := new_rows.get(pk): + _create_log_entry( + action=AuditLogEntry.Action.UPDATE, + instance=new, + sender=model, + diff_old=old, + diff_new=new, + fields_to_check=list(fields), + use_json_for_changes=settings.AUDITLOG_STORE_JSON_CHANGES, + ) + + return count + + +def audited_bulk_create(model, objs: list) -> list: + """Bulk create model instances, recording a CREATE audit entry for each new row. + + Uses InvenTree.helpers_db.bulk_create_and_fetch() to resolve the created rows, + so that entries are recorded on every database backend - including those which do + not return primary keys from bulk_create() (e.g. MySQL). + + Primary keys are populated on the provided objects (as per bulk_create()). + + Arguments: + model: The model class (must provide a 'metadata' field) + objs: The (unsaved) model instances to create + + Returns: + list: The created objects, as fetched from the database (in insertion order) + """ + from auditlog.receivers import _create_log_entry + + from audit.models import AuditLogEntry + from InvenTree.helpers_db import bulk_create_and_fetch + + # Required by bulk_create_and_fetch() - checked for all models (not only audited ones), + # so that an unsupported model is caught in development rather than at runtime + try: + model._meta.get_field('metadata') + except FieldDoesNotExist: + raise TypeError( + f"audited_bulk_create() requires a 'metadata' field: {model._meta.label}" + ) + + if not objs: + return [] + + with transaction.atomic(): + created = list(bulk_create_and_fetch(model, objs)) + + if len(created) != len(objs): # pragma: no cover + raise ValueError( + f'Expected {len(objs)} {model._meta.label} objects, fetched {len(created)}' + ) + + # Back-populate primary keys onto the provided objects + for obj, instance in zip(objs, created, strict=True): + obj.pk = instance.pk + obj._state.adding = False + obj._state.db = instance._state.db + + if is_audited(model): + with write_batched(): + for instance in created: + _create_log_entry( + action=AuditLogEntry.Action.CREATE, + instance=instance, + sender=model, + diff_old=None, + diff_new=instance, + use_json_for_changes=settings.AUDITLOG_STORE_JSON_CHANGES, + ) + + return created diff --git a/src/backend/InvenTree/audit/middleware.py b/src/backend/InvenTree/audit/middleware.py new file mode 100644 index 0000000000..3412b355fd --- /dev/null +++ b/src/backend/InvenTree/audit/middleware.py @@ -0,0 +1,48 @@ +"""Middleware for the audit app.""" + +import ipaddress + +from auditlog.context import set_extra_data +from auditlog.middleware import AuditlogMiddleware + +from audit.context import audit_request + + +class InvenTreeAuditlogMiddleware(AuditlogMiddleware): + """Bind request context to audit log entries. + + Differences from the upstream AuditlogMiddleware: + - The actor is not captured here, but resolved lazily (see audit.context) + - The remote address is taken from REMOTE_ADDR only, and validated + (upstream trusts the first X-Forwarded-For entry and X-Forwarded-Port as-is) + - The correlation ID matches the structlog request ID (see audit.context.get_cid) + """ + + @staticmethod + def _get_remote_addr(request): + """Return the (validated) remote address for the request. + + An invalid value would otherwise cause every audited save to fail + on databases with a native IP address column type (PostgreSQL 'inet'). + """ + remote_addr = request.META.get('REMOTE_ADDR') + + if not remote_addr: + return None + + try: + return str(ipaddress.ip_address(remote_addr.strip())) + except ValueError: + return None + + def get_extra_data(self, request): + """Return the data to be attached to all log entries created during this request.""" + return {'remote_addr': self._get_remote_addr(request), 'remote_port': None} + + def __call__(self, request): + """Process the request within the audit context.""" + with ( + audit_request(request), + set_extra_data(context_data=self.get_extra_data(request)), + ): + return self.get_response(request) diff --git a/src/backend/InvenTree/audit/migrations/0001_initial.py b/src/backend/InvenTree/audit/migrations/0001_initial.py new file mode 100644 index 0000000000..0572d5aceb --- /dev/null +++ b/src/backend/InvenTree/audit/migrations/0001_initial.py @@ -0,0 +1,171 @@ +# Generated by Django 5.2.17 on 2026-10-20 07:02 + +import django.db.models.deletion +import django.utils.timezone +from django.conf import settings +from django.db import migrations, models + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ("contenttypes", "0002_remove_content_type_name"), + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.CreateModel( + name="AuditLogEntry", + fields=[ + ( + "id", + models.BigAutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ( + "object_pk", + models.CharField( + db_index=True, max_length=255, verbose_name="object pk" + ), + ), + ( + "object_id", + models.BigIntegerField( + blank=True, db_index=True, null=True, verbose_name="object id" + ), + ), + ("object_repr", models.TextField(verbose_name="object representation")), + ("serialized_data", models.JSONField(null=True)), + ( + "action", + models.PositiveSmallIntegerField( + choices=[ + (0, "create"), + (1, "update"), + (2, "delete"), + (3, "access"), + ], + db_index=True, + verbose_name="action", + ), + ), + ( + "changes_text", + models.TextField(blank=True, verbose_name="change message"), + ), + ("changes", models.JSONField(null=True, verbose_name="change message")), + ( + "cid", + models.CharField( + blank=True, + db_index=True, + max_length=255, + null=True, + verbose_name="Correlation ID", + ), + ), + ( + "remote_addr", + models.GenericIPAddressField( + blank=True, null=True, verbose_name="remote address" + ), + ), + ( + "remote_port", + models.PositiveIntegerField( + blank=True, null=True, verbose_name="remote port" + ), + ), + ( + "timestamp", + models.DateTimeField( + db_index=True, + default=django.utils.timezone.now, + verbose_name="timestamp", + ), + ), + ( + "additional_data", + models.JSONField( + blank=True, null=True, verbose_name="additional data" + ), + ), + ( + "actor_email", + models.CharField( + blank=True, + max_length=254, + null=True, + verbose_name="actor email", + ), + ), + ( + "parent_object_pk", + models.CharField( + blank=True, + max_length=255, + null=True, + verbose_name="Parent Object PK", + ), + ), + ( + "actor", + models.ForeignKey( + blank=True, + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="+", + to=settings.AUTH_USER_MODEL, + verbose_name="actor", + ), + ), + ( + "content_type", + models.ForeignKey( + on_delete=django.db.models.deletion.CASCADE, + related_name="+", + to="contenttypes.contenttype", + verbose_name="content type", + ), + ), + ( + "parent_content_type", + models.ForeignKey( + blank=True, + null=True, + on_delete=django.db.models.deletion.CASCADE, + related_name="+", + to="contenttypes.contenttype", + verbose_name="Parent Content Type", + ), + ), + ], + options={ + "verbose_name": "Audit Log Entry", + "verbose_name_plural": "Audit Log Entries", + "ordering": ["-timestamp"], + "get_latest_by": "timestamp", + "abstract": False, + "indexes": [ + models.Index( + fields=["content_type", "object_pk", "-timestamp"], + name="audit_object_idx", + ), + models.Index( + fields=[ + "parent_content_type", + "parent_object_pk", + "-timestamp", + ], + name="audit_parent_idx", + ), + ], + }, + ), + ] diff --git a/src/backend/InvenTree/audit/migrations/0002_actor_username.py b/src/backend/InvenTree/audit/migrations/0002_actor_username.py new file mode 100644 index 0000000000..a20effa869 --- /dev/null +++ b/src/backend/InvenTree/audit/migrations/0002_actor_username.py @@ -0,0 +1,52 @@ +# Generated by Django 5.2.17 on 2026-10-23 23:18 + +import django.db.models.deletion +from django.conf import settings +from django.db import migrations, models +from django.db.models import OuterRef, Subquery + + +def set_actor_username(apps, schema_editor): + """Record the username of the actor against existing entries.""" + AuditLogEntry = apps.get_model('audit', 'AuditLogEntry') + User = apps.get_model(settings.AUTH_USER_MODEL) + + AuditLogEntry.objects.filter(actor__isnull=False).update( + actor_username=Subquery( + User.objects.filter(pk=OuterRef('actor_id')).values('username')[:1] + ) + ) + + +class Migration(migrations.Migration): + + dependencies = [ + ("audit", "0001_initial"), + ("contenttypes", "0002_remove_content_type_name"), + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ] + + operations = [ + migrations.AddField( + model_name="auditlogentry", + name="actor_username", + field=models.CharField( + blank=True, max_length=150, null=True, verbose_name="Actor Username" + ), + ), + migrations.AlterField( + model_name="auditlogentry", + name="parent_content_type", + field=models.ForeignKey( + blank=True, + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="+", + to="contenttypes.contenttype", + verbose_name="Parent Content Type", + ), + ), + migrations.RunPython( + set_actor_username, reverse_code=migrations.RunPython.noop + ), + ] diff --git a/src/backend/InvenTree/audit/migrations/__init__.py b/src/backend/InvenTree/audit/migrations/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/backend/InvenTree/audit/models.py b/src/backend/InvenTree/audit/models.py new file mode 100644 index 0000000000..f17501873e --- /dev/null +++ b/src/backend/InvenTree/audit/models.py @@ -0,0 +1,337 @@ +"""Database models for the audit app.""" + +import os + +from django.contrib.contenttypes.models import ContentType +from django.core.exceptions import FieldDoesNotExist +from django.db import models +from django.utils.functional import Promise +from django.utils.translation import gettext_lazy as _ + +from auditlog.models import AbstractLogEntry, LogEntryManager + +import audit.context + +MASKED_VALUE = '***' + + +def get_audit_parent_reference(instance): + """Return a (ContentType, pk) reference to the audit "parent" of an instance (or None).""" + if callable(func := getattr(instance, 'get_audit_parent_reference', None)): + return func() + + # Fallback for models which provide get_audit_parent() only + if callable(func := getattr(instance, 'get_audit_parent', None)): + if parent := func(): + return ContentType.objects.get_for_model(parent), str(parent.pk) + + return None + + +# Attribute used to defer an audit entry until the instance has been saved +DEFERRED_LOG_ATTR = '_audit_deferred_log' + + +def expression_fields(instance, changes: dict) -> list[str]: + """Return the changed fields whose (unsaved) value is a query expression (e.g. F('x') + 1). + + The value written by such an expression is only known once the instance has been saved. + """ + fields = [] + + for name in changes: + try: + field = type(instance)._meta.get_field(name) + except FieldDoesNotExist: + continue + + # Read the raw attribute value - avoids any related object lookups + value = instance.__dict__.get(getattr(field, 'attname', name)) + + if hasattr(value, 'resolve_expression'): + fields.append(name) + + return fields + + +def file_field_basenames(model, changes: dict) -> dict: + """Reduce recorded file field values to the filename only (not the full storage path).""" + for name, value in changes.items(): + try: + field = model._meta.get_field(name) + except FieldDoesNotExist: + continue + + if isinstance(field, models.FileField) and isinstance(value, (list, tuple)): + changes[name] = [ + os.path.basename(v) if isinstance(v, str) and v else v for v in value + ] + + return changes + + +def resolve_lazy_strings(changes: dict) -> dict: + """Convert lazy translation strings to plain strings, so that the changes are JSON serializable. + + Upstream only performs this conversion when AUDITLOG_STORE_JSON_CHANGES is disabled. + """ + for name, value in changes.items(): + if isinstance(value, (list, tuple)): + changes[name] = [str(v) if isinstance(v, Promise) else v for v in value] + + return changes + + +def drop_orphan_currency_changes(instance, changes: dict) -> dict: + """Remove changes to money currency fields whose amount is not set (and has not changed). + + django-money stores the currency of a MoneyField 'x' in a separate field 'x_currency', + which has a default value (e.g. 'USD') even when the amount is empty. + Without this, creating an object records e.g. {'price_currency': [None, 'USD']}. + """ + from djmoney.models.fields import MoneyField + + result = {} + + for field, value in changes.items(): + if field.endswith('_currency'): + amount = field[: -len('_currency')] + + try: + amount_field = type(instance)._meta.get_field(amount) + except FieldDoesNotExist: + amount_field = None + + # Only applies to the currency field of a MoneyField (e.g. not 'order_currency') + if ( + isinstance(amount_field, MoneyField) + and amount not in changes + and getattr(instance, amount, None) is None + ): + continue + + result[field] = value + + return result + + +def is_concrete_field(model, field_name: str) -> bool: + """Return True if field_name is a concrete field on the model (or is unknown).""" + try: + field = model._meta.get_field(field_name) + except FieldDoesNotExist: + return True + + return getattr(field, 'concrete', False) + + +def mask_changes(changes): + """Replace every recorded value in the provided changes dict.""" + if not changes: + return changes + + masked = {} + + for field, value in changes.items(): + if isinstance(value, dict) and value.get('type') == 'm2m': + masked[field] = { + **value, + 'objects': [MASKED_VALUE for _obj in value.get('objects', [])], + } + else: + masked[field] = [MASKED_VALUE, MASKED_VALUE] + + return masked + + +class AuditLogEntryManager(LogEntryManager): + """Manager for the AuditLogEntry model. + + Extends the upstream manager to attach InvenTree specific context to each entry: + - Parent object reference + - Actor (resolved lazily from the current request) + - Source (for changes made outside of a request) + - Row-level masking of sensitive values + """ + + def get_audit_kwargs(self, instance, kwargs: dict) -> dict: + """Update the provided LogEntry kwargs with InvenTree specific data.""" + if reference := get_audit_parent_reference(instance): + parent_ct, parent_pk = reference + kwargs.setdefault('parent_content_type', parent_ct) + kwargs.setdefault('parent_object_pk', parent_pk) + + if 'actor' not in kwargs and (actor := audit.context.get_request_actor()): + kwargs['actor'] = actor + kwargs['actor_email'] = getattr(actor, 'email', None) + + # Record the username, so that the actor is still identified if the user is deleted + if (actor := kwargs.get('actor')) and not kwargs.get('actor_username'): + kwargs['actor_username'] = getattr(actor, 'username', None) + + if source := audit.context.get_source(): + get_additional_data = getattr(instance, 'get_additional_data', None) + + additional_data = kwargs.get('additional_data') + + if additional_data is None and callable(get_additional_data): + additional_data = get_additional_data() + + kwargs['additional_data'] = {**(additional_data or {}), 'source': source} + + is_masked = getattr(instance, 'is_audit_masked', None) + + if callable(is_masked) and is_masked(): + # Mask at write time - sensitive values must never reach the log table + kwargs['changes'] = mask_changes(kwargs.get('changes')) + kwargs['serialized_data'] = None + + return kwargs + + def log_create(self, instance, force_log: bool = False, **kwargs): + """Create a new log entry for the provided instance. + + For updates, fields whose recorded old and new values are identical are dropped. + These occur when the in-memory value differs from the database value only in type + (e.g. an empty FieldFile vs. an empty string), and would otherwise add noise to every save. + + Updates which assign a query expression (e.g. F('completed') + 1) are deferred until + after the save, so that the value actually written can be recorded + (see audit.signals.resolve_deferred_log). + """ + # Internal flag: this entry has been deferred, and the expression values resolved + resolved = kwargs.pop('_audit_resolved', False) + + changes = kwargs.get('changes') + + if isinstance(changes, dict): + # Upstream includes reverse relations (e.g. 'company.contacts') in create / delete diffs, + # recording the string representation of the related manager - discard these + changes = { + field: value + for field, value in changes.items() + if is_concrete_field(type(instance), field) + } + + # A money currency field (e.g. 'price_currency') is meaningless without an amount + changes = drop_orphan_currency_changes(instance, changes) + + if not changes and not force_log: + return None + + if kwargs.get('action') == AuditLogEntry.Action.UPDATE: + if not resolved and (fields := expression_fields(instance, changes)): + instance.__dict__[DEFERRED_LOG_ATTR] = { + 'force_log': force_log, + 'kwargs': {**kwargs, 'changes': changes}, + 'fields': fields, + } + return None + + changes = { + field: value + for field, value in changes.items() + if not ( + isinstance(value, (list, tuple)) + and len(value) == 2 + and value[0] == value[1] + ) + } + + if not changes and not force_log: + return None + + kwargs['changes'] = file_field_basenames( + type(instance), resolve_lazy_strings(changes) + ) + + kwargs = self.get_audit_kwargs(instance, kwargs) + return super().log_create(instance, force_log=force_log, **kwargs) + + def create(self, **kwargs): + """Create a new log entry, or add it to the current batch (see audit.helpers.write_batched).""" + if (batch := audit.context.get_batch()) is not None: + entry = self.model(**kwargs) + batch.append(entry) + return entry + + return super().create(**kwargs) + + def log_m2m_changes( + self, changed_queryset, instance, operation, field_name, **kwargs + ): + """Create a new log entry for a many-to-many change.""" + is_masked = getattr(instance, 'is_audit_masked', None) + + if callable(is_masked) and is_masked() and changed_queryset: + # Upstream builds the 'changes' dict from str(obj) for each object, + # so substitute placeholders before anything is written + changed_queryset = [MASKED_VALUE] * len(changed_queryset) + + kwargs = self.get_audit_kwargs(instance, kwargs) + + return super().log_m2m_changes( + changed_queryset, instance, operation, field_name, **kwargs + ) + + +class AuditLogEntry(AbstractLogEntry): + """A single audit log entry, recording a change to a model instance. + + Extends the upstream LogEntry with an (optional) reference to a "parent" object, + so that the history of a parent (e.g. a PurchaseOrder) can include changes + made to its children (e.g. line items) - even after the child has been deleted. + + Entries are immutable via the API. + """ + + class Meta(AbstractLogEntry.Meta): + """Metaclass options for the AuditLogEntry model.""" + + verbose_name = _('Audit Log Entry') + verbose_name_plural = _('Audit Log Entries') + + indexes = [ + models.Index( + fields=['content_type', 'object_pk', '-timestamp'], + name='audit_object_idx', + ), + models.Index( + fields=['parent_content_type', 'parent_object_pk', '-timestamp'], + name='audit_parent_idx', + ), + ] + + objects = AuditLogEntryManager() + + # Note: SET_NULL (rather than CASCADE) so that removing a stale content type + # (e.g. an uninstalled plugin) does not delete the history of child objects + parent_content_type = models.ForeignKey( + ContentType, + on_delete=models.SET_NULL, + null=True, + blank=True, + related_name='+', + verbose_name=_('Parent Content Type'), + ) + + parent_object_pk = models.CharField( + max_length=255, null=True, blank=True, verbose_name=_('Parent Object PK') + ) + + # Snapshot of the actor's username (retained if the user is deleted) + actor_username = models.CharField( + max_length=150, null=True, blank=True, verbose_name=_('Actor Username') + ) + + @property + def model_type(self) -> str: + """Return the 'app_label.model' string for the audited model.""" + return f'{self.content_type.app_label}.{self.content_type.model}' + + @property + def parent_model_type(self): + """Return the 'app_label.model' string for the parent model (if any).""" + if ct := self.parent_content_type: + return f'{ct.app_label}.{ct.model}' + return None diff --git a/src/backend/InvenTree/audit/permissions.py b/src/backend/InvenTree/audit/permissions.py new file mode 100644 index 0000000000..a4027a9cc7 --- /dev/null +++ b/src/backend/InvenTree/audit/permissions.py @@ -0,0 +1,73 @@ +"""Permission checks for the audit app. + +Rules: +- An audit entry is visible if the user may view the *audited* model +- Querying the history of a parent object (which includes child entries) requires + permission to view the parent model +- The parent reference of an entry is only exposed if the user may view the parent model +- For models with AUDIT_REQUIRE_PARENT_PERMISSION (e.g. attachments), entries are only + visible if the user may view the parent model +""" + +from typing import Optional + +from audit.registry import get_view_restriction + + +def user_can_view_audit_model(user, model, groups=None) -> bool: + """Determine if the user may view audit history for the given model class.""" + from users.permissions import check_user_permission + + if user is None or model is None: + return False + + restriction = get_view_restriction(model) + + if restriction == 'superuser' and not user.is_superuser: + return False + + if restriction == 'staff' and not user.is_staff: + return False + + return check_user_permission(user, model, 'view', groups=groups) + + +def get_viewable_audit_content_types(user) -> Optional[list[int]]: + """Return the ContentType IDs for which the user may view audit history. + + Returns None if the user may view everything. + """ + from InvenTree.cache import get_cached_content_types + from users.permissions import prefetch_rule_sets + + if user.is_superuser: + return None + + groups = prefetch_rule_sets(user) + + allowed = [] + + for ct in get_cached_content_types(): + model = ct.model_class() + + # Stale content types (model no longer installed) are never visible + if model is None: + continue + + if user_can_view_audit_model(user, model, groups=groups): + allowed.append(ct.pk) + + return allowed + + +def parent_permission_content_types() -> list[int]: + """Return the ContentType IDs of audited models which require parent permission.""" + from django.contrib.contenttypes.models import ContentType + + from audit.registry import audited_models + + return [ + ContentType.objects.get_for_model(model).pk + for model in audited_models() + if getattr(model, 'AUDIT_REQUIRE_PARENT_PERMISSION', False) + ] diff --git a/src/backend/InvenTree/audit/registry.py b/src/backend/InvenTree/audit/registry.py new file mode 100644 index 0000000000..49c1ee114f --- /dev/null +++ b/src/backend/InvenTree/audit/registry.py @@ -0,0 +1,105 @@ +"""Registration of audited models with django-auditlog.""" + +from django.apps import apps + +import structlog +from auditlog.registry import auditlog + +logger = structlog.get_logger('inventree') + +# Fields which are never recorded in the audit log, for any model +BASE_EXCLUDE_FIELDS = ( + 'metadata', + # Derived sorting fields + 'reference_int', + 'line_int', + # MPTT tree fields + 'lft', + 'rght', + 'tree_id', + 'level', + # Automatically updated timestamps + 'updated', + 'updated_at', + 'last_updated', +) + +# Models (by 'app_label.model' label) whose audit history is restricted, +# regardless of the RuleSet permissions of the requesting user. +# These are models which InvenTree does not define (so cannot carry the mixin attribute), +# or whose history is more sensitive than the current value. +AUDIT_VIEW_RESTRICTIONS = { + 'auth.user': 'staff', + 'auth.group': 'staff', + 'users.ruleset': 'staff', + 'common.inventreesetting': 'staff', + 'plugin.pluginsetting': 'superuser', +} + + +def audited_models() -> list: + """Return all installed models which opt in to the audit log. + + Uses the app registry rather than ContentTypes, so that no database access is required. + """ + from InvenTree.models import InvenTreeAuditMixin + + return sorted( + ( + model + for model in apps.get_models() + if issubclass(model, InvenTreeAuditMixin) and not model._meta.abstract + ), + key=lambda model: model._meta.label_lower, + ) + + +def register_model(model, **kwargs) -> None: + """Register a single model with django-auditlog, using the model's audit attributes.""" + exclude_fields = { + *BASE_EXCLUDE_FIELDS, + *getattr(model, 'AUDIT_EXCLUDE_FIELDS', ()), + *kwargs.pop('exclude_fields', ()), + } + + mask_fields = { + *getattr(model, 'AUDIT_MASK_FIELDS', ()), + *kwargs.pop('mask_fields', ()), + } + + m2m_fields = { + *getattr(model, 'AUDIT_M2M_FIELDS', ()), + *kwargs.pop('m2m_fields', ()), + } + + # If specified, *only* these fields are recorded + include_fields = { + *getattr(model, 'AUDIT_INCLUDE_FIELDS', ()), + *kwargs.pop('include_fields', ()), + } + + auditlog.register( + model, + include_fields=sorted(include_fields), + exclude_fields=sorted(exclude_fields), + mask_fields=sorted(mask_fields), + m2m_fields=m2m_fields, + **kwargs, + ) + + +def register_audited_models() -> None: + """Register all models which opt in to the audit log.""" + for model in audited_models(): + if not auditlog.contains(model): + register_model(model) + + logger.debug('Registered %s models with the audit log', len(auditlog.get_models())) + + +def get_view_restriction(model): + """Return the audit view restriction for a model: None, 'staff' or 'superuser'.""" + if restriction := getattr(model, 'AUDIT_VIEW_RESTRICTION', None): + return restriction + + return AUDIT_VIEW_RESTRICTIONS.get(model._meta.label_lower) diff --git a/src/backend/InvenTree/audit/serializers.py b/src/backend/InvenTree/audit/serializers.py new file mode 100644 index 0000000000..9ebbf1ea08 --- /dev/null +++ b/src/backend/InvenTree/audit/serializers.py @@ -0,0 +1,306 @@ +"""API serializers for the audit app.""" + +from typing import Any, Optional + +from django.core.exceptions import ( + FieldDoesNotExist, + ObjectDoesNotExist, + ValidationError, +) +from django.db import models +from django.utils.text import capfirst +from django.utils.translation import gettext_lazy as _ + +from drf_spectacular.utils import extend_schema_field +from rest_framework import serializers + +import InvenTree.serializers +from audit.models import MASKED_VALUE, AuditLogEntry +from InvenTree.serializers import OptionalField +from users.serializers import UserSerializer + + +class AuditChangeDetailSerializer(serializers.Serializer): + """Describes a single field change within an audit log entry.""" + + field = serializers.CharField(label=_('Field')) + label = serializers.CharField(label=_('Label')) + type = serializers.ChoiceField( + choices=['value', 'fk', 'm2m'], label=_('Change Type') + ) + old = serializers.JSONField(allow_null=True, required=False) + new = serializers.JSONField(allow_null=True, required=False) + old_display = serializers.CharField(allow_null=True, required=False) + new_display = serializers.CharField(allow_null=True, required=False) + model_type = serializers.CharField( + allow_null=True, required=False, help_text=_('Related model (for FK fields)') + ) + operation = serializers.CharField( + allow_null=True, required=False, help_text=_('M2M operation (add / delete)') + ) + objects = serializers.ListField( + child=serializers.CharField(), allow_null=True, required=False + ) + numeric = serializers.BooleanField( + required=False, help_text=_('The field holds a numeric value') + ) + + +def is_numeric_field(field) -> bool: + """Return True if the model field holds a plain numeric value. + + Fields with choices are excluded (their values are displayed via the choice label). + Note: decimal values are recorded as strings (e.g. '10.00000'). + """ + if field is None or getattr(field, 'flatchoices', None): + return False + + return isinstance( + field, (models.IntegerField, models.DecimalField, models.FloatField) + ) + + +class AuditLogEntrySerializer( + InvenTree.serializers.FilterableSerializerMixin, + InvenTree.serializers.InvenTreeModelSerializer, +): + """Serializer for the AuditLogEntry model (read-only).""" + + # Fields which are only exposed to staff users + STAFF_ONLY_FIELDS = ['remote_addr', 'remote_port'] + + class Meta: + """Metaclass options.""" + + model = AuditLogEntry + fields = [ + 'pk', + 'timestamp', + 'action', + 'action_label', + 'actor', + 'actor_detail', + 'actor_username', + 'model_type', + 'model_label', + 'object_pk', + 'object_repr', + 'parent_model_type', + 'parent_object_pk', + 'changes', + 'changes_detail', + 'cid', + 'additional_data', + 'remote_addr', + 'remote_port', + ] + read_only_fields = fields + + action_label = serializers.CharField( + source='get_action_display', read_only=True, label=_('Action') + ) + + model_type = serializers.CharField(read_only=True, label=_('Model Type')) + + model_label = serializers.SerializerMethodField(label=_('Model Label')) + + def get_model_label(self, entry: AuditLogEntry) -> str: + """Return the human readable name of the audited model.""" + if model := entry.content_type.model_class(): + return capfirst(str(model._meta.verbose_name)) + + return entry.model_type + + parent_model_type = serializers.CharField( + read_only=True, allow_null=True, label=_('Parent Model Type') + ) + + changes = serializers.SerializerMethodField(label=_('Changes')) + + changes_detail = OptionalField( + serializer_class=serializers.SerializerMethodField, + serializer_kwargs={'read_only': True, 'label': _('Change Details')}, + default_include=False, + ) + + def get_request_user(self): + """Return the requesting user (if any).""" + request = self.context.get('request') + return getattr(request, 'user', None) + + def can_view_model(self, model) -> bool: + """Return True if the requesting user may view the given model (cached).""" + from audit.permissions import user_can_view_audit_model + + if model is None: + return False + + cache = self.__dict__.setdefault('_audit_perm_cache', {}) + + if model not in cache: + cache[model] = user_can_view_audit_model(self.get_request_user(), model) + + return cache[model] + + def get_entry_fields(self, entry: AuditLogEntry) -> dict: + """Return a mapping of field name -> model field for the audited model.""" + model = entry.content_type.model_class() + fields = {} + + # The audited model may no longer exist (e.g. an uninstalled plugin) + if model is None: + return fields + + for name in entry.changes_dict or {}: + try: + fields[name] = model._meta.get_field(name) + except FieldDoesNotExist: + continue + + return fields + + @extend_schema_field(serializers.JSONField()) + def get_changes(self, entry: AuditLogEntry) -> dict: + """Return the raw changes recorded against this entry. + + The string representation of related objects in M2M changes is + removed where the user cannot view the related model. + """ + changes = dict(entry.changes_dict or {}) + fields = self.get_entry_fields(entry) + + for name, value in changes.items(): + if isinstance(value, dict) and value.get('type') == 'm2m': + field = fields.get(name) + related = getattr(field, 'related_model', None) + + if not self.can_view_model(related): + changes[name] = {**value, 'objects': None} + + return changes + + def get_related_display(self, model, value) -> Optional[str]: + """Return the string representation of a related object (if it can be viewed).""" + if value in (None, '', 'None', MASKED_VALUE) or not self.can_view_model(model): + return None + + cache = self.__dict__.setdefault('_audit_fk_cache', {}) + key = (model, str(value)) + + if key not in cache: + try: + pk = model._meta.pk.to_python(value) + cache[key] = str(model._base_manager.get(pk=pk)) + except ObjectDoesNotExist: + cache[key] = _('Deleted') + except (ValidationError, ValueError, TypeError): + cache[key] = None + + return cache[key] + + @extend_schema_field(AuditChangeDetailSerializer(many=True)) + def get_changes_detail(self, entry: AuditLogEntry) -> list[dict[str, Any]]: + """Return a list of per-field changes, with display values. + + - Field labels are resolved from the model (if it still exists) + - Choice values are translated to their labels + - Foreign key values are resolved to the related object, *only* if the + requesting user has permission to view the related model + """ + fields = self.get_entry_fields(entry) + rows = [] + + for name, value in self.get_changes(entry).items(): + field = fields.get(name) + + # The primary key is recorded on creation, but is not useful to display + if getattr(field, 'primary_key', False): + continue + + label = capfirst(str(getattr(field, 'verbose_name', name))) + + row = { + 'field': name, + 'label': label, + 'type': 'value', + 'old': None, + 'new': None, + 'old_display': None, + 'new_display': None, + 'model_type': None, + 'operation': None, + 'objects': None, + 'numeric': is_numeric_field(field), + } + + if isinstance(value, dict) and value.get('type') == 'm2m': + row['type'] = 'm2m' + row['operation'] = value.get('operation') + row['objects'] = value.get('objects') + rows.append(row) + continue + + if isinstance(value, (list, tuple)) and len(value) == 2: + row['old'], row['new'] = value + else: + row['new'] = value + + related = getattr(field, 'related_model', None) + + if ( + field is not None + and related + and (field.many_to_one or field.one_to_one) + ): + row['type'] = 'fk' + row['model_type'] = ( + f'{related._meta.app_label}.{related._meta.model_name}' + ) + row['old_display'] = self.get_related_display(related, row['old']) + row['new_display'] = self.get_related_display(related, row['new']) + + elif field is not None and getattr(field, 'flatchoices', None): + choices = dict(field.flatchoices) + + for key in ['old', 'new']: + if row[key] in choices: + row[f'{key}_display'] = str(choices[row[key]]) + + rows.append(row) + + # Order by the model field definition order (unknown fields last) + model = entry.content_type.model_class() + order = ( + {f.name: idx for idx, f in enumerate(model._meta.get_fields())} + if model + else {} + ) + + rows.sort(key=lambda row: order.get(row['field'], len(order))) + + return rows + + actor_detail = OptionalField( + serializer_class=UserSerializer, + serializer_kwargs={'source': 'actor', 'read_only': True, 'allow_null': True}, + default_include=True, + prefetch_fields=['actor'], + ) + + def to_representation(self, instance): + """Remove staff-only fields for non-staff users.""" + data = super().to_representation(instance) + + user = self.get_request_user() + + if not (user and user.is_staff): + for field in self.STAFF_ONLY_FIELDS: + data.pop(field, None) + + # Only expose the parent reference if the user may view the parent model + if parent_ct := instance.parent_content_type: + if not self.can_view_model(parent_ct.model_class()): + data['parent_model_type'] = None + data['parent_object_pk'] = None + + return data diff --git a/src/backend/InvenTree/audit/signals.py b/src/backend/InvenTree/audit/signals.py new file mode 100644 index 0000000000..44c05982fe --- /dev/null +++ b/src/backend/InvenTree/audit/signals.py @@ -0,0 +1,115 @@ +"""Signal receivers for the audit app.""" + +from django.conf import settings +from django.db.models.signals import post_save, pre_save +from django.dispatch import receiver + +from auditlog.signals import pre_log +from django_q.signals import post_execute_in_worker, pre_execute + +import audit.context +from audit.models import DEFERRED_LOG_ATTR + + +@receiver(pre_log, dispatch_uid='inventree_audit_pre_log') +def check_audit_enabled(sender, instance, action, **kwargs): + """Skip writing audit entries if the audit log is disabled via global setting. + + - Instances can opt out of being recorded (should_audit) + - Instances can be recorded even if the audit log is disabled (audit_always) + + Returning False from a pre_log receiver prevents the entry being created. + """ + from common.settings import get_global_setting + + should_audit = getattr(instance, 'should_audit', None) + + if callable(should_audit) and not should_audit(): + return False + + audit_always = getattr(instance, 'audit_always', None) + + if callable(audit_always) and audit_always(): + return None + + # create=False: never write to the settings table from within the audit hook + if not get_global_setting('AUDIT_LOG_ENABLED', True, create=False): + return False + + return None + + +def recorded_field_value(model, instance, field_name: str): + """Return the value of a field, in the form recorded by auditlog. + + Mirrors the conversion applied by auditlog.diff.model_instance_diff(), + so that deferred entries are identical to those recorded directly. + """ + from django.utils.encoding import smart_str + + from auditlog.diff import get_field_value, get_mask_function, is_primitive + from auditlog.registry import auditlog + + use_json = settings.AUDITLOG_STORE_JSON_CHANGES + value = get_field_value(instance, model._meta.get_field(field_name), use_json) + + model_fields = auditlog.get_model_fields(model) + + if field_name in model_fields['mask_fields']: + mask_func = get_mask_function(model_fields.get('mask_callable')) + return mask_func(smart_str(value)) + + if not use_json or not is_primitive(value): + return smart_str(value) + + return value + + +@receiver(pre_save, dispatch_uid='inventree_audit_clear_deferred_log') +def clear_deferred_log(sender, instance, **kwargs): + """Discard any deferred entry left over from a previous (failed) save of this instance. + + Connected before the auditlog receivers (see AuditConfig.ready), so runs first. + """ + instance.__dict__.pop(DEFERRED_LOG_ATTR, None) + + +@receiver(post_save, dispatch_uid='inventree_audit_resolve_deferred_log') +def resolve_deferred_log(sender, instance, created, **kwargs): + """Write a deferred audit entry, with query expression values resolved from the database.""" + from audit.models import AuditLogEntry + + if not (pending := instance.__dict__.pop(DEFERRED_LOG_ATTR, None)): + return + + saved = sender._base_manager.filter(pk=instance.pk).first() + + if saved is None: # pragma: no cover + return + + log_kwargs = pending['kwargs'] + changes = dict(log_kwargs['changes']) + + for name in pending['fields']: + old = changes[name][0] + new = recorded_field_value(sender, saved, name) + changes[name] = [old, new] + + AuditLogEntry.objects.log_create( + instance, + force_log=pending['force_log'], + _audit_resolved=True, + **{**log_kwargs, 'changes': changes}, + ) + + +@receiver(pre_execute, dispatch_uid='inventree_audit_pre_execute') +def begin_worker_task(sender, func, task, **kwargs): + """Tag audit entries created by a background worker task with the task name.""" + audit.context.begin_worker_task(task.get('func') or func) + + +@receiver(post_execute_in_worker, dispatch_uid='inventree_audit_post_execute') +def end_worker_task(sender, func, task, **kwargs): + """Clear the audit context once a background worker task has finished.""" + audit.context.end_worker_task() diff --git a/src/backend/InvenTree/audit/tasks.py b/src/backend/InvenTree/audit/tasks.py new file mode 100644 index 0000000000..8b6e7dfe46 --- /dev/null +++ b/src/backend/InvenTree/audit/tasks.py @@ -0,0 +1,65 @@ +"""Background tasks for the audit app.""" + +from datetime import timedelta + +from django.db.models import Exists, OuterRef +from django.utils import timezone + +import structlog +from opentelemetry import trace + +from common.settings import get_global_setting +from InvenTree.tasks import ScheduledTask, scheduled_task + +tracer = trace.get_tracer(__name__) +logger = structlog.get_logger('inventree') + +# Number of entries to delete per query +DELETE_BATCH_SIZE = 1000 + + +@tracer.start_as_current_span('delete_old_audit_entries') +@scheduled_task(ScheduledTask.DAILY) +def delete_old_audit_entries(): + """Remove audit log entries older than the configured retention period. + + Entries are deleted in small batches, so that no single long-running + transaction holds locks on the (potentially very large) audit table. + + If AUDIT_LOG_DELETE_KEEP_SUMMARY is enabled, the following entries are retained + for each object (regardless of age), so that it is always known who created + and who last modified an object: + - The CREATE entry + - The most recent entry (for a deleted object, this is the DELETE entry) + """ + from audit.models import AuditLogEntry + + if not get_global_setting('AUDIT_LOG_DELETE_OLD_ENTRIES', False): + return + + delete_n_days = int(get_global_setting('AUDIT_LOG_DELETE_DAYS', 365)) + + threshold = timezone.now() - timedelta(days=delete_n_days) + + old_entries = AuditLogEntry.objects.filter(timestamp__lte=threshold) + + if get_global_setting('AUDIT_LOG_DELETE_KEEP_SUMMARY', True): + # Only delete entries which have a newer entry for the same object + newer_entries = AuditLogEntry.objects.filter( + content_type=OuterRef('content_type'), + object_pk=OuterRef('object_pk'), + pk__gt=OuterRef('pk'), + ) + + old_entries = old_entries.exclude(action=AuditLogEntry.Action.CREATE).filter( + Exists(newer_entries) + ) + + deleted = 0 + + while batch := list(old_entries.values_list('pk', flat=True)[:DELETE_BATCH_SIZE]): + n, _ = AuditLogEntry.objects.filter(pk__in=batch).delete() + deleted += n + + if deleted: + logger.info('Deleted old audit log entries', count=deleted, threshold=threshold) diff --git a/src/backend/InvenTree/audit/tests.py b/src/backend/InvenTree/audit/tests.py new file mode 100644 index 0000000000..631b3defa7 --- /dev/null +++ b/src/backend/InvenTree/audit/tests.py @@ -0,0 +1,1261 @@ +"""Unit tests for the audit app. + +No production models are audited yet, so these tests temporarily register +PurchaseOrder (parent) and PurchaseOrderLineItem (child) with the audit log. +""" + +import contextlib +from base64 import b64encode +from datetime import timedelta +from unittest import mock + +from django.contrib.auth import get_user_model +from django.contrib.contenttypes.models import ContentType +from django.test import TestCase, override_settings +from django.urls import reverse +from django.utils import timezone + +from auditlog.registry import auditlog + +import audit.context +from audit.models import MASKED_VALUE, AuditLogEntry, mask_changes +from audit.registry import BASE_EXCLUDE_FIELDS, audited_models, register_model +from audit.serializers import AuditLogEntrySerializer +from audit.tasks import delete_old_audit_entries +from common.settings import set_global_setting +from company.models import Company, SupplierPart +from InvenTree.unit_test import InvenTreeAPITestCase +from order.models import PurchaseOrder, PurchaseOrderLineItem +from part.models import Part + +TEST_MODELS = [PurchaseOrder, PurchaseOrderLineItem] + + +def setting_content_type(): + """Return the ContentType for global settings.""" + from common.models import InvenTreeSetting + + return ContentType.objects.get_for_model(InvenTreeSetting) + + +def audited_entries(): + """Return all audit entries, except for changes to global settings. + + Changes to the audit log settings are always recorded, + even when the audit log is disabled. + """ + return AuditLogEntry.objects.exclude(content_type=setting_content_type()) + + +class AuditTestMixin: + """Ensure that the test models are registered with the audit log. + + Models which are already audited (in production) are left registered as they are - + only models registered here are unregistered again afterwards. + """ + + @classmethod + def setUpClass(cls): + """Register test models before any test data is created.""" + cls.parent_patch = mock.patch.object( + PurchaseOrderLineItem, + 'get_audit_parent', + lambda self: self.order, + create=True, + ) + cls.parent_patch.start() + + cls.registered_models = [ + model for model in TEST_MODELS if not auditlog.contains(model) + ] + + for model in cls.registered_models: + register_model(model) + + super().setUpClass() + + @classmethod + def tearDownClass(cls): + """Unregister test models.""" + super().tearDownClass() + + for model in cls.registered_models: + auditlog.unregister(model) + + cls.parent_patch.stop() + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.supplier = Company.objects.create(name='Audit Supplier', is_supplier=True) + cls.part = Part.objects.create( + name='Audit Part', description='A part', purchaseable=True + ) + cls.supplier_part = SupplierPart.objects.create( + part=cls.part, supplier=cls.supplier, SKU='AUDIT-SKU' + ) + cls.po = PurchaseOrder.objects.create( + reference='PO-88880001', supplier=cls.supplier, description='Original' + ) + + def entries_for(self, instance, **kwargs): + """Return audit entries recorded directly against the provided instance.""" + return AuditLogEntry.objects.get_for_object(instance).filter(**kwargs) + + +class AuditRegistryTest(TestCase): + """Tests for model registration.""" + + def test_audited_models(self): + """Only the expected production models opt in to the audit log.""" + self.assertEqual( + [model._meta.label_lower for model in audited_models()], + [ + 'build.build', + 'common.attachment', + 'common.customunit', + 'common.inventreesetting', + 'common.parameter', + 'common.parametertemplate', + 'common.projectcode', + 'company.address', + 'company.company', + 'company.contact', + 'company.manufacturerpart', + 'company.supplierpart', + 'company.supplierpricebreak', + 'order.purchaseorder', + 'order.purchaseorderextraline', + 'order.purchaseorderlineitem', + 'order.returnorder', + 'order.returnorderextraline', + 'order.returnorderlineitem', + 'order.salesorder', + 'order.salesorderextraline', + 'order.salesorderlineitem', + 'order.salesordershipment', + 'order.transferorder', + 'order.transferorderlineitem', + 'part.bomitem', + 'part.bomitemsubstitute', + 'part.part', + 'part.partcategory', + 'pricing.partinternalpricebreak', + 'pricing.partpricing', + 'pricing.partsellpricebreak', + ], + ) + + for model in audited_models(): + self.assertTrue(auditlog.contains(model)) + + def test_register_model(self): + """Registration applies the base excluded fields.""" + from common.models import NotificationEntry + + # Note: a model which is *not* audited in production + self.assertFalse(auditlog.contains(NotificationEntry)) + + register_model(NotificationEntry, exclude_fields=['notes']) + + try: + self.assertTrue(auditlog.contains(NotificationEntry)) + exclude = auditlog.get_model_fields(NotificationEntry)['exclude_fields'] + + for field in [*BASE_EXCLUDE_FIELDS, 'notes']: + self.assertIn(field, exclude) + finally: + auditlog.unregister(NotificationEntry) + + def test_mask_changes(self): + """All values are replaced when masking changes.""" + changes = { + 'value': ['old-secret', 'new-secret'], + 'tags': {'type': 'm2m', 'operation': 'add', 'objects': ['a', 'b']}, + } + + masked = mask_changes(changes) + + self.assertEqual(masked['value'], [MASKED_VALUE, MASKED_VALUE]) + self.assertEqual(masked['tags']['objects'], [MASKED_VALUE, MASKED_VALUE]) + self.assertEqual(masked['tags']['operation'], 'add') + self.assertNotIn('secret', str(masked)) + + +class AuditModelTest(AuditTestMixin, TestCase): + """Tests for audit entry creation.""" + + def test_create_update_delete(self): + """Basic create / update / delete operations are recorded.""" + self.assertEqual( + self.entries_for(self.po, action=AuditLogEntry.Action.CREATE).count(), 1 + ) + + self.po.description = 'Updated' + self.po.save() + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + self.assertEqual(entry.changes_dict['description'], ['Original', 'Updated']) + self.assertIsNone(entry.parent_content_type) + + # Excluded fields are not recorded + self.assertNotIn('reference_int', entry.changes_dict) + self.assertNotIn('metadata', entry.changes_dict) + + def test_parent_link(self): + """Child entries are linked to the parent - and are retained after deletion.""" + line = PurchaseOrderLineItem.objects.create( + order=self.po, part=self.supplier_part, quantity=5 + ) + line_pk = line.pk + + line.quantity = 10 + line.save() + line.delete() + + entries = AuditLogEntry.objects.filter( + parent_content_type=self.po.get_content_type(), + parent_object_pk=str(self.po.pk), + ) + + self.assertEqual(entries.count(), 3) + self.assertEqual( + {e.action for e in entries}, + { + AuditLogEntry.Action.CREATE, + AuditLogEntry.Action.UPDATE, + AuditLogEntry.Action.DELETE, + }, + ) + + for entry in entries: + self.assertEqual(entry.object_pk, str(line_pk)) + self.assertEqual(entry.parent_model_type, 'order.purchaseorder') + + def test_masked_instance(self): + """Row-level masking is applied before the entry is written.""" + with mock.patch.object( + PurchaseOrder, 'is_audit_masked', lambda self: True, create=True + ): + self.po.description = 'top-secret-value' + self.po.save() + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + + self.assertEqual(entry.changes_dict['description'], [MASKED_VALUE] * 2) + self.assertIsNone(entry.serialized_data) + + # Ensure the raw value is not stored anywhere in the entry + for field in ['changes', 'changes_text', 'additional_data', 'object_repr']: + self.assertNotIn('top-secret-value', str(getattr(entry, field))) + + def test_audit_source(self): + """Changes made outside of a request are tagged with their source.""" + with audit.context.audit_source('task:test_task'): + cid = audit.context.get_cid() + self.po.description = 'From task' + self.po.save() + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + + self.assertIsNone(entry.actor) + self.assertEqual(entry.additional_data, {'source': 'task:test_task'}) + self.assertIsNotNone(cid) + self.assertEqual(entry.cid, cid) + + # Context is reset afterwards + self.assertIsNone(audit.context.get_source()) + self.assertIsNone(audit.context.get_cid()) + + def test_disabled(self): + """No entries are recorded when the audit log is disabled.""" + set_global_setting('AUDIT_LOG_ENABLED', False) + self.addCleanup(set_global_setting, 'AUDIT_LOG_ENABLED', True) + + self.po.description = 'Not recorded' + self.po.save() + + self.assertFalse( + self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).exists() + ) + + def test_retention(self): + """Old entries are removed by the scheduled task (only when enabled).""" + old = timezone.now() - timedelta(days=100) + + AuditLogEntry.objects.update(timestamp=old) + + self.po.description = 'Recent' + self.po.save() + + n = AuditLogEntry.objects.count() + self.assertGreater(n, 1) + + # Disabled by default + delete_old_audit_entries() + self.assertEqual(AuditLogEntry.objects.count(), n) + + set_global_setting('AUDIT_LOG_DELETE_OLD_ENTRIES', True) + set_global_setting('AUDIT_LOG_DELETE_DAYS', 30) + set_global_setting('AUDIT_LOG_DELETE_KEEP_SUMMARY', False) + self.addCleanup(set_global_setting, 'AUDIT_LOG_DELETE_OLD_ENTRIES', False) + self.addCleanup(set_global_setting, 'AUDIT_LOG_DELETE_DAYS', 365) + self.addCleanup(set_global_setting, 'AUDIT_LOG_DELETE_KEEP_SUMMARY', True) + + with mock.patch('audit.tasks.DELETE_BATCH_SIZE', 1): + delete_old_audit_entries() + + # Only the recent entries remain (including the settings changes above) + self.assertEqual( + AuditLogEntry.objects.exclude(content_type=setting_content_type()).count(), + 1, + ) + + def test_retention_keep_summary(self): + """With 'keep summary' enabled, the creation and latest entry of each object are kept.""" + # An object with multiple updates + for description in ['First', 'Second', 'Third']: + self.po.description = description + self.po.save() + + # A deleted object + line = PurchaseOrderLineItem.objects.create( + order=self.po, part=self.supplier_part, quantity=5 + ) + line_pk = line.pk + line.quantity = 10 + line.save() + line.delete() + + # Everything is "old" + AuditLogEntry.objects.update(timestamp=timezone.now() - timedelta(days=100)) + + set_global_setting('AUDIT_LOG_DELETE_OLD_ENTRIES', True) + set_global_setting('AUDIT_LOG_DELETE_DAYS', 30) + self.addCleanup(set_global_setting, 'AUDIT_LOG_DELETE_OLD_ENTRIES', False) + self.addCleanup(set_global_setting, 'AUDIT_LOG_DELETE_DAYS', 365) + + with mock.patch('audit.tasks.DELETE_BATCH_SIZE', 1): + delete_old_audit_entries() + + po_entries = self.entries_for(self.po).order_by('pk') + self.assertEqual( + [e.action for e in po_entries], + [AuditLogEntry.Action.CREATE, AuditLogEntry.Action.UPDATE], + ) + self.assertEqual(po_entries.last().changes_dict['description'][1], 'Third') + + line_entries = AuditLogEntry.objects.filter( + content_type=PurchaseOrderLineItem.get_content_type(), + object_pk=str(line_pk), + ).order_by('pk') + self.assertEqual( + [e.action for e in line_entries], + [AuditLogEntry.Action.CREATE, AuditLogEntry.Action.DELETE], + ) + + +class AuditAPITest(AuditTestMixin, InvenTreeAPITestCase): + """Tests for the audit log API endpoints.""" + + roles = ['purchase_order.view', 'purchase_order.change'] + is_staff = True + + def list_url(self): + """Return the list endpoint URL.""" + return reverse('api-audit-log-list') + + def po_history(self, expected_code=200, **params): + """Return the audit history for the test purchase order.""" + params = {'model_type': 'order.purchaseorder', 'model_id': self.po.pk, **params} + return self.get(self.list_url(), params, expected_code=expected_code).data + + def patch_po(self, description, **kwargs): + """Edit the test purchase order via the API.""" + url = reverse('api-po-detail', kwargs={'pk': self.po.pk}) + self.patch(url, {'description': description}, **kwargs) + + def test_actor(self): + """The authenticated user is recorded as the actor (session auth).""" + self.patch_po('Session edit') + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.remote_addr, '127.0.0.1') + self.assertIsNotNone(entry.cid) + + data = self.po_history() + self.assertEqual(data[0]['actor'], self.user.pk) + self.assertEqual(data[0]['actor_detail']['username'], self.user.username) + self.assertEqual(data[0]['model_type'], 'order.purchaseorder') + self.assertEqual(data[0]['model_label'], 'Purchase Order') + self.assertEqual(data[0]['changes']['description'][1], 'Session edit') + + def test_correlation_id(self): + """The correlation ID matches the request ID (as used by structlog).""" + url = reverse('api-po-detail', kwargs={'pk': self.po.pk}) + + # Request ID provided by the client / reverse proxy + self.patch(url, {'description': 'One'}, HTTP_X_REQUEST_ID='req-abc-123') + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).get() + self.assertEqual(entry.cid, 'req-abc-123') + + # Otherwise a unique ID is generated per request + self.patch(url, {'description': 'Two'}) + self.patch(url, {'description': 'Three'}) + + cids = list( + self + .entries_for(self.po, action=AuditLogEntry.Action.UPDATE) + .exclude(cid='req-abc-123') + .values_list('cid', flat=True) + ) + + self.assertEqual(len(cids), 2) + self.assertTrue(all(cids)) + self.assertNotEqual(cids[0], cids[1]) + + def test_basic_auth_actor(self): + """Users authenticated by DRF (not middleware) are recorded as the actor.""" + self.logout() + + auth = b64encode(f'{self.username}:{self.password}'.encode()).decode() + self.client.credentials(HTTP_AUTHORIZATION=f'Basic {auth}') + + self.patch_po('Basic auth edit') + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + self.assertEqual(entry.actor, self.user) + + def test_spoofed_remote_addr(self): + """An invalid forwarded address does not break saving, and is not recorded.""" + self.patch_po('Spoofed', HTTP_X_FORWARDED_FOR='not-an-ip-address') + + entry = self.entries_for(self.po, action=AuditLogEntry.Action.UPDATE).first() + self.assertIsNone(entry.remote_addr) + + def test_children(self): + """The history of a parent includes child entries (unless excluded).""" + line = PurchaseOrderLineItem.objects.create( + order=self.po, part=self.supplier_part, quantity=5 + ) + line.delete() + + data = self.po_history() + models = [entry['model_type'] for entry in data] + + self.assertIn('order.purchaseorderlineitem', models) + self.assertEqual(models.count('order.purchaseorderlineitem'), 2) + + data = self.po_history(include_children=False) + self.assertEqual( + {entry['model_type'] for entry in data}, {'order.purchaseorder'} + ) + + # Short model names are also accepted + data = self.get(self.list_url(), {'model_type': 'purchaseorderlineitem'}).data + self.assertEqual(len(data), 2) + + def test_filter_validation(self): + """Invalid filter combinations are rejected.""" + self.get(self.list_url(), {'model_id': 1}, expected_code=400) + self.get(self.list_url(), {'model_type': 'not_a_model'}, expected_code=400) + + def test_read_only(self): + """Audit entries cannot be modified via the API.""" + entry = self.entries_for(self.po).first() + url = reverse('api-audit-log-detail', kwargs={'pk': entry.pk}) + + self.get(url) + self.patch(url, {'object_repr': 'x'}, expected_code=405) + self.delete(url, expected_code=405) + self.post(self.list_url(), {}, expected_code=405) + + def test_permissions(self): + """Entries are only visible with 'view' permission on the audited model.""" + entry = self.entries_for(self.po).first() + detail_url = reverse('api-audit-log-detail', kwargs={'pk': entry.pk}) + + self.assertGreater(len(self.po_history()), 0) + + # Remove all roles + self.clearRoles() + + self.assertEqual(len(self.po_history()), 0) + self.assertEqual(len(self.get(self.list_url()).data), 0) + self.get(detail_url, expected_code=404) + + def test_child_without_parent_permission(self): + """Child entries without 'view' permission on the parent model. + + - The child entry itself is still visible (permission on the child model) + - It is *not* returned when querying the history of the parent + - The parent reference is removed from the response + """ + PurchaseOrderLineItem.objects.create( + order=self.po, part=self.supplier_part, quantity=5 + ) + + def line_entries(params=None): + return [ + e + for e in self.get(self.list_url(), params or {}).data + if e['model_type'] == 'order.purchaseorderlineitem' + ] + + [entry] = line_entries() + self.assertEqual(entry['parent_model_type'], 'order.purchaseorder') + self.assertEqual(entry['parent_object_pk'], str(self.po.pk)) + self.assertEqual(len(line_entries({'model_type': 'purchaseorder'})), 1) + + # Line item visible, but parent (order) not + def can_view(user, model, groups=None): + return model is not PurchaseOrder + + with ( + mock.patch('audit.api.user_can_view_audit_model', can_view), + mock.patch('audit.permissions.user_can_view_audit_model', can_view), + ): + [entry] = line_entries() + self.assertIsNone(entry['parent_model_type']) + self.assertIsNone(entry['parent_object_pk']) + + # Not available when querying through the parent + self.assertEqual( + line_entries({'model_type': 'purchaseorder', 'model_id': self.po.pk}), + [], + ) + self.assertEqual(line_entries({'model_type': 'purchaseorder'}), []) + + # Still available when querying the child model directly + self.assertEqual( + len(line_entries({'model_type': 'purchaseorderlineitem'})), 1 + ) + + def test_staff_required(self): + """The audit log API is only available to staff users.""" + self.patch_po('Edit') + + entry = self.entries_for(self.po).first() + detail_url = reverse('api-audit-log-detail', kwargs={'pk': entry.pk}) + + # Staff users can see entries (including the remote address) + data = self.po_history() + self.assertEqual(data[0]['remote_addr'], '127.0.0.1') + self.get(detail_url) + + # Non-staff users are denied - even with view permission on the audited model + self.user.is_staff = False + self.user.save() + + self.get(self.list_url(), expected_code=403) + self.get(detail_url, expected_code=403) + + def test_view_restriction(self): + """Models with a view restriction are hidden from users without that status.""" + self.assertGreater(len(self.po_history()), 0) + + with mock.patch.object( + PurchaseOrder, 'AUDIT_VIEW_RESTRICTION', 'superuser', create=True + ): + # Staff, but not superuser + self.assertEqual(len(self.po_history(include_children=False)), 0) + + self.user.is_superuser = True + self.user.save() + + self.assertGreater(len(self.po_history(include_children=False)), 0) + + def test_model_type_filter_options(self): + """The model type filter choices are provided via the OPTIONS metadata.""" + response = self.options(self.list_url()) + + filters = response.data['filters'] + + # Model-backed filters (e.g. 'actor') are never exposed + self.assertNotIn('actor', filters) + + choices = { + c['value']: c['display_name'] for c in filters['model_type']['choices'] + } + + self.assertEqual(choices['order.purchaseorder'], 'Purchase Order') + self.assertIn('order.purchaseorderlineitem', choices) + + # Only models which the user can view are provided + self.assertNotIn('part.part', choices) + self.assertNotIn('build.build', choices) + + # The chosen value filters the entries + data = self.get(self.list_url(), {'model_type': 'order.purchaseorder'}).data + self.assertGreater(len(data), 0) + + for entry in data: + self.assertEqual(entry['model_type'], 'order.purchaseorder') + + def test_superuser(self): + """Superusers can view all entries.""" + self.clearRoles() + self.assertEqual(len(self.po_history()), 0) + + get_user_model().objects.filter(pk=self.user.pk).update(is_superuser=True) + + self.assertGreater(len(self.po_history()), 0) + + def get_change_rows(self, field_name: str) -> list: + """Return the 'changes_detail' rows for the given field, across the PO history.""" + data = self.po_history( + changes_detail=True, + include_children=False, + action=AuditLogEntry.Action.UPDATE, + ) + + return [ + row + for entry in data + for row in entry['changes_detail'] + if row['field'] == field_name + ] + + def test_changes_detail_fk(self): + """Foreign key changes are resolved for display, if the related model can be viewed.""" + old_supplier = self.supplier + new_supplier = Company.objects.create(name='Other Supplier', is_supplier=True) + + self.po.supplier = new_supplier + self.po.save() + + [row] = self.get_change_rows('supplier') + + self.assertEqual(row['type'], 'fk') + self.assertEqual(row['model_type'], 'company.company') + self.assertEqual(row['old'], old_supplier.pk) + self.assertEqual(row['new'], new_supplier.pk) + self.assertEqual(row['old_display'], str(old_supplier)) + self.assertEqual(row['new_display'], str(new_supplier)) + + # Deleted related objects are marked as such + new_supplier.delete() + [row] = self.get_change_rows('supplier') + self.assertEqual(row['new_display'], 'Deleted') + + # Without permission on the related model, only the PK is returned + with mock.patch.object( + AuditLogEntrySerializer, + 'can_view_model', + lambda self, model: model is PurchaseOrder, + ): + [row] = self.get_change_rows('supplier') + + self.assertEqual(row['old'], old_supplier.pk) + self.assertIsNone(row['old_display']) + self.assertIsNone(row['new_display']) + + def test_changes_detail_value(self): + """Plain value changes include the field label.""" + self.po.description = 'New description' + self.po.save() + + [row] = self.get_change_rows('description') + + self.assertEqual(row['type'], 'value') + self.assertEqual(row['label'], 'Description') + self.assertFalse(row['numeric']) + + # Primary key and automatically updated fields are not shown + data = self.po_history(changes_detail=True, include_children=False) + fields = {row['field'] for entry in data for row in entry['changes_detail']} + + self.assertNotIn('id', fields) + self.assertNotIn('updated_at', fields) + self.assertEqual(row['old'], 'Original') + self.assertEqual(row['new'], 'New description') + + def test_m2m_objects_hidden(self): + """M2M object names are removed if the related model cannot be viewed.""" + AuditLogEntry.objects.create( + content_type=self.po.get_content_type(), + object_pk=str(self.po.pk), + object_repr=str(self.po), + action=AuditLogEntry.Action.UPDATE, + changes={ + 'tags': {'type': 'm2m', 'operation': 'add', 'objects': ['hidden-tag']} + }, + ) + + with mock.patch.object( + AuditLogEntrySerializer, + 'can_view_model', + lambda self, model: model is PurchaseOrder, + ): + response = self.get( + self.list_url(), + { + 'model_type': 'order.purchaseorder', + 'model_id': self.po.pk, + 'changes_detail': True, + }, + ) + + self.assertNotIn('hidden-tag', str(response.content)) + + with mock.patch.object( + AuditLogEntrySerializer, 'can_view_model', lambda self, model: True + ): + [row] = self.get_change_rows('tags') + + self.assertEqual(row['type'], 'm2m') + self.assertEqual(row['operation'], 'add') + self.assertEqual(row['objects'], ['hidden-tag']) + + def test_stale_model(self): + """Entries for models which no longer exist can still be serialized.""" + get_user_model().objects.filter(pk=self.user.pk).update(is_superuser=True) + + ct = ContentType.objects.create(app_label='removed_app', model='removedmodel') + + AuditLogEntry.objects.create( + content_type=ct, + object_pk='1', + object_repr='Removed object', + action=AuditLogEntry.Action.UPDATE, + changes={'name': ['a', 'b']}, + ) + + data = self.get( + self.list_url(), {'model_type': ct.pk, 'changes_detail': True} + ).data + + self.assertEqual(len(data), 1) + self.assertEqual(data[0]['changes_detail'][0]['label'], 'Name') + self.assertEqual(data[0]['changes_detail'][0]['new'], 'b') + + +class AuditMiddlewareConfigTest(TestCase): + """Tests for ensuring the audit middleware is always configured.""" + + AUDIT = 'audit.middleware.InvenTreeAuditlogMiddleware' + STRUCTLOG = 'django_structlog.middlewares.RequestMiddleware' + + def test_installed(self): + """The audit middleware is present in the active configuration.""" + from django.conf import settings + + self.assertIn(self.AUDIT, settings.MIDDLEWARE) + + # Placed after the structlog middleware (shares the request ID) + self.assertGreater( + settings.MIDDLEWARE.index(self.AUDIT), + settings.MIDDLEWARE.index(self.STRUCTLOG), + ) + + def test_ensure_middleware(self): + """Missing middleware is added (after the specified middleware, if present).""" + from InvenTree.config import ensure_middleware + + # Already present - unchanged + middleware = ['a', self.AUDIT, 'b'] + self.assertEqual(ensure_middleware(middleware, self.AUDIT), middleware) + + # Missing - inserted after the specified middleware (with a warning) + with self.assertLogs('inventree', level='WARNING') as logs: + result = ensure_middleware( + ['a', self.STRUCTLOG, 'b'], self.AUDIT, after=self.STRUCTLOG + ) + + self.assertEqual(result, ['a', self.STRUCTLOG, self.AUDIT, 'b']) + self.assertIn('INVE-W17', logs.output[0]) + + # Missing (and the 'after' middleware is not present) - appended + with self.assertLogs('inventree', level='WARNING'): + result = ensure_middleware(['a', 'b'], self.AUDIT, after=self.STRUCTLOG) + + self.assertEqual(result, ['a', 'b', self.AUDIT]) + + +class AuditAdminTest(AuditTestMixin, InvenTreeAPITestCase): + """Tests for the AuditLogEntry admin class.""" + + superuser = True + + def test_admin_registration(self): + """The custom admin class replaces the upstream one.""" + from django.contrib import admin + + from audit.admin import AuditLogEntryAdmin + + model_admin = admin.site._registry[AuditLogEntry] + + self.assertIsInstance(model_admin, AuditLogEntryAdmin) + + # Remains read-only + self.assertFalse(model_admin.has_add_permission(None)) + self.assertFalse(model_admin.has_change_permission(None)) + + @override_settings( + SITE_URL='http://testserver', CSRF_TRUSTED_ORIGINS=['http://testserver'] + ) + def test_filter_by_user(self): + """The admin changelist can be filtered by the user who made the change.""" + other = get_user_model().objects.create_user(username='other_user') + + with audit.context.audit_source('test'): + from auditlog.context import set_actor + + with set_actor(self.user): + self.po.description = 'By test user' + self.po.save() + + with set_actor(other): + self.po.description = 'By other user' + self.po.save() + + url = reverse('admin:audit_auditlogentry_changelist') + + # The filter lists users with recorded entries + response = self.client.get(url) + self.assertEqual(response.status_code, 200) + self.assertContains(response, 'other_user') + + response = self.client.get(url, {'actor__id__exact': other.pk}) + self.assertEqual(response.status_code, 200) + + entries = list(response.context['cl'].result_list) + + self.assertEqual(len(entries), 1) + self.assertEqual(entries[0].actor, other) + + +def audit_task_helper(pk: int, description: str): + """Background task used to test the audit source of task changes.""" + company = Company.objects.get(pk=pk) + company.description = description + company.save() + + +class AuditSourceTest(InvenTreeAPITestCase): + """Tests for the context recorded against programmatic (non-request) changes.""" + + roles = ['purchase_order.change'] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.company = Company.objects.create(name='Source Company', is_supplier=True) + + def last_update(self): + """Return the most recent UPDATE entry for the test company.""" + return ( + AuditLogEntry.objects + .get_for_object(self.company) + .filter(action=AuditLogEntry.Action.UPDATE) + .order_by('pk') + .last() + ) + + def test_programmatic_update(self): + """A plain ORM change (no request, no task) is recorded without any request context.""" + self.company.description = 'Programmatic' + self.company.save() + + entry = self.last_update() + + self.assertEqual(entry.changes_dict['description'], ['', 'Programmatic']) + self.assertIsNone(entry.actor) + self.assertIsNone(entry.remote_addr) + self.assertIsNone(entry.additional_data) + + def test_programmatic_bulk_create(self): + """Entries for a programmatic bulk create have no actor.""" + from audit.helpers import audited_bulk_create + from common.models import Parameter, ParameterTemplate + + template = ParameterTemplate.objects.create(name='Source Template') + + [parameter] = audited_bulk_create( + Parameter, + [ + Parameter( + model_type=Part.get_content_type(), + model_id=Part.objects.create(name='P', description='P').pk, + template=template, + data='X', + ) + ], + ) + + entry = AuditLogEntry.objects.get_for_object(parameter).get() + + self.assertEqual(entry.action, AuditLogEntry.Action.CREATE) + self.assertIsNone(entry.actor) + self.assertIsNone(entry.remote_addr) + + def test_actor_not_retained(self): + """The actor of a request is not attributed to later programmatic changes.""" + self.patch( + reverse('api-company-detail', kwargs={'pk': self.company.pk}), + {'description': 'Via API'}, + ) + + api_entry = self.last_update() + self.assertEqual(api_entry.actor, self.user) + + self.company.refresh_from_db() + self.company.description = 'Programmatic' + self.company.save() + + entry = self.last_update() + + self.assertNotEqual(entry.pk, api_entry.pk) + self.assertIsNone(entry.actor) + self.assertIsNone(entry.remote_addr) + self.assertNotEqual(entry.cid, api_entry.cid) + + def run_task(self, description: str, **kwargs): + """Run the test task (synchronously) via offload_task.""" + from InvenTree.tasks import offload_task + + offload_task( + 'audit.tests.audit_task_helper', + self.company.pk, + description, + force_sync=True, + **kwargs, + ) + + def test_sync_task(self): + """Changes made by a (synchronous) task are tagged with the task name.""" + self.run_task('From task') + + entry = self.last_update() + + self.assertEqual(entry.changes_dict['description'], ['', 'From task']) + self.assertEqual( + entry.additional_data, {'source': 'task:audit.tests.audit_task_helper'} + ) + self.assertIsNone(entry.actor) + self.assertIsNotNone(entry.cid) + + # Context is reset afterwards + self.assertIsNone(audit.context.get_source()) + self.assertIsNone(audit.context.get_cid()) + + def test_sync_task_in_request(self): + """A task run synchronously within a request is attributed to the request.""" + request = mock.Mock(user=self.user) + + with audit.context.audit_request(request): + self.run_task('From request') + + entry = self.last_update() + + self.assertEqual(entry.actor, self.user) + self.assertIsNone(entry.additional_data) + + def test_worker_task(self): + """Changes made by a task executed by the background worker are tagged with the task name.""" + from django_q.tasks import AsyncTask + + task = AsyncTask( + 'audit.tests.audit_task_helper', self.company.pk, 'From worker', sync=True + ) + task.run() + + entry = self.last_update() + + self.assertEqual(entry.changes_dict['description'], ['', 'From worker']) + self.assertEqual( + entry.additional_data, {'source': 'task:audit.tests.audit_task_helper'} + ) + self.assertIsNone(entry.actor) + self.assertIsNotNone(entry.cid) + + # Context is reset afterwards + self.assertIsNone(audit.context.get_source()) + self.assertIsNone(audit.context.get_cid()) + + def test_worker_task_failure(self): + """The audit context is reset even if a worker task fails.""" + from django_q.tasks import AsyncTask + + task = AsyncTask('audit.tests.audit_task_helper', -1, 'Missing', sync=True) + + with contextlib.suppress(Exception): + task.run() + + self.assertIsNone(audit.context.get_source()) + self.assertIsNone(audit.context.get_cid()) + + +class AuditDisabledTest(InvenTreeAPITestCase): + """No audit entries are recorded (via any write path) when auditing is disabled. + + Each path is tested both with the AUDIT_LOG_ENABLED setting disabled, + and within the django-auditlog disable_auditlog() context. + """ + + roles = ['part.add', 'part.change', 'part.view', 'purchase_order.change'] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + from common.models import ParameterTemplate + + super().setUpTestData() + + cls.company = Company.objects.create(name='Disabled Company', is_supplier=True) + cls.part = Part.objects.create(name='Disabled Part', description='A part') + cls.template = ParameterTemplate.objects.create(name='Disabled Template') + + @contextlib.contextmanager + def setting_disabled(self): + """Disable auditing via the global setting.""" + set_global_setting('AUDIT_LOG_ENABLED', False) + + try: + yield + finally: + set_global_setting('AUDIT_LOG_ENABLED', True) + + def check_disabled(self, operation): + """Check that the operation is recorded when enabled - and not when disabled.""" + from auditlog.context import disable_auditlog + + for name, context in [ + ('setting', self.setting_disabled), + ('context', disable_auditlog), + ]: + with self.subTest(disabled_by=name): + n = audited_entries().count() + + with context(): + operation() + + self.assertEqual(audited_entries().count(), n) + + # Sanity check: the same operation *is* recorded when enabled + n = audited_entries().count() + operation() + self.assertGreater(audited_entries().count(), n) + + def test_create_update_delete(self): + """Model create / update / delete operations.""" + counter = iter(range(100)) + + def create(): + Company.objects.create(name=f'New {next(counter)}') + + def update(): + self.company.description = f'Description {next(counter)}' + self.company.save() + + def delete(): + Company.objects.create(name=f'Delete {next(counter)}').delete() + + for operation in [create, update, delete]: + with self.subTest(operation=operation.__name__): + self.check_disabled(operation) + + def test_child_model(self): + """Changes to child models (linked to a parent).""" + from company.models import Address + + counter = iter(range(100)) + + self.check_disabled( + lambda: Address.objects.create( + company=self.company, title=f'Address {next(counter)}', line1='1' + ) + ) + + def test_expression_update(self): + """Deferred (F() expression) updates.""" + from django.db.models import F + + from build.models import Build + + build = Build.objects.create( + part=Part.objects.create( + name='Assembly', description='An assembly', assembly=True + ), + reference='BO-9901', + quantity=10, + ) + + def update(): + build.priority = F('priority') + 1 + build.save(update_fields=['priority']) + build.refresh_from_db() + + self.check_disabled(update) + + def test_audited_update(self): + """The audited_update() helper.""" + from audit.helpers import audited_update + + counter = iter(range(100)) + + self.check_disabled( + lambda: audited_update( + Company.objects.filter(pk=self.company.pk), + description=f'Update {next(counter)}', + ) + ) + + def test_audited_bulk_update(self): + """The audited_bulk_update() helper.""" + from audit.helpers import audited_bulk_update + + counter = iter(range(100)) + + def update(): + self.company.description = f'Bulk {next(counter)}' + audited_bulk_update(Company, [self.company], ['description']) + + self.check_disabled(update) + + def test_audited_bulk_create(self): + """The audited_bulk_create() helper.""" + from audit.helpers import audited_bulk_create + from common.models import Parameter + + counter = iter(range(100)) + + self.check_disabled( + lambda: audited_bulk_create( + Parameter, + [ + Parameter( + model_type=Part.get_content_type(), + model_id=Part.objects.create( + name=f'P{next(counter)}', description='P' + ).pk, + template=self.template, + data='X', + ) + ], + ) + ) + + def test_api(self): + """API requests (including batched bulk creation).""" + counter = iter(range(100)) + + def update(): + self.patch( + reverse('api-company-detail', kwargs={'pk': self.company.pk}), + {'description': f'API {next(counter)}'}, + ) + + def bulk_create(): + parts = [ + Part.objects.create(name=f'API {next(counter)}', description='P') + for _ in range(2) + ] + + self.post( + reverse('api-parameter-list'), + [ + { + 'model_type': 'part', + 'model_id': part.pk, + 'template': self.template.pk, + 'data': 'Y', + } + for part in parts + ], + format='json', + ) + + for operation in [update, bulk_create]: + with self.subTest(operation=operation.__name__): + # Note: disable_auditlog() is not propagated to the request thread, + # so only the global setting is tested here + n = audited_entries().count() + + with self.setting_disabled(): + operation() + + self.assertEqual(audited_entries().count(), n) + + operation() + self.assertGreater(audited_entries().count(), n) + + +class AuditRetentionTest(InvenTreeAPITestCase): + """Audit entries are retained when related objects are removed.""" + + roles = ['purchase_order.change', 'purchase_order.view'] + + def test_stale_parent_content_type(self): + """Removing a stale (parent) content type does not delete child entries.""" + from django.core.management import call_command + + company = Company.objects.create(name='Child Company') + entry = AuditLogEntry.objects.get_for_object(company).get() + + stale = ContentType.objects.create(app_label='removed_app', model='removed') + + entry.parent_content_type = stale + entry.parent_object_pk = '1' + entry.save() + + call_command( + 'remove_stale_contenttypes', + '--include-stale-apps', + '--no-input', + verbosity=0, + ) + + self.assertFalse(ContentType.objects.filter(pk=stale.pk).exists()) + + entry.refresh_from_db() + self.assertIsNone(entry.parent_content_type) + + def test_deleted_user(self): + """The username of the actor is retained if the user is deleted.""" + user = get_user_model().objects.create_user( + username='leaver', password='password', is_staff=True + ) + self.client.force_login(user) + + company = Company.objects.create(name='Edited Company', is_supplier=True) + + # Grant the user permission to edit the company + for group in self.user.groups.all(): + user.groups.add(group) + + self.patch( + reverse('api-company-detail', kwargs={'pk': company.pk}), + {'description': 'Edited by leaver'}, + ) + + entry = ( + AuditLogEntry.objects + .get_for_object(company) + .filter(action=AuditLogEntry.Action.UPDATE) + .get() + ) + + self.assertEqual(entry.actor, user) + self.assertEqual(entry.actor_username, 'leaver') + + user.delete() + entry.refresh_from_db() + + self.assertIsNone(entry.actor) + self.assertEqual(entry.actor_username, 'leaver') + + # The username is provided via the API + self.client.force_login(self.user) + url = reverse('api-audit-log-detail', kwargs={'pk': entry.pk}) + + data = self.get(url).data + self.assertIsNone(data['actor']) + self.assertEqual(data['actor_username'], 'leaver') diff --git a/src/backend/InvenTree/build/models.py b/src/backend/InvenTree/build/models.py index e8c0bcc4f0..02ee6b91f5 100644 --- a/src/backend/InvenTree/build/models.py +++ b/src/backend/InvenTree/build/models.py @@ -84,6 +84,7 @@ class BuildReportContext(report.mixins.BaseReportContext, TypedDict): class Build( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.PluginValidationMixin, report.mixins.InvenTreeReportMixin, InvenTree.models.InvenTreeParameterMixin, diff --git a/src/backend/InvenTree/build/test_api.py b/src/backend/InvenTree/build/test_api.py index 2bd0b33abc..9d954a4d1b 100644 --- a/src/backend/InvenTree/build/test_api.py +++ b/src/backend/InvenTree/build/test_api.py @@ -2693,7 +2693,7 @@ class BuildConsumeTest(BuildAPITest): expected_code=201, benchmark=True, max_query_count=180, - max_query_time=2.5, + max_query_time=4.5, ) build.refresh_from_db() diff --git a/src/backend/InvenTree/build/test_audit.py b/src/backend/InvenTree/build/test_audit.py new file mode 100644 index 0000000000..d44ed1f269 --- /dev/null +++ b/src/backend/InvenTree/build/test_audit.py @@ -0,0 +1,154 @@ +"""Audit log tests for the Build model.""" + +from django.db.models import F +from django.urls import reverse + +from audit.models import AuditLogEntry +from build.models import Build +from build.status_codes import BuildStatus +from build.test_build import BuildTestBase +from InvenTree.unit_test import InvenTreeAPITestCase + +ACTION = AuditLogEntry.Action + + +def build_entries(build_pk, **filters): + """Return audit entries recorded against a Build (by PK).""" + return AuditLogEntry.objects.filter( + content_type=Build.get_content_type(), object_pk=str(build_pk), **filters + ) + + +class BuildAuditTest(BuildTestBase): + """Tests for audit history of Build objects.""" + + def updates(self): + """Return UPDATE entries for the test build (oldest first).""" + return build_entries(self.build.pk, action=ACTION.UPDATE).order_by('pk') + + def test_create(self): + """Creating a build order is recorded.""" + entry = build_entries(self.build.pk, action=ACTION.CREATE).get() + + self.assertEqual(entry.changes_dict['reference'][1], self.build.reference) + self.assertEqual(entry.object_repr, self.build.reference) + + # Derived / tree fields are not recorded + for field in ['reference_int', 'lft', 'rght', 'tree_id', 'level']: + self.assertNotIn(field, entry.changes_dict) + + def test_status_change(self): + """Status transitions (e.g. issuing the build) are recorded.""" + self.build.issue_build() + + entry = self.updates().last() + + self.assertEqual( + entry.changes_dict['status'], + [BuildStatus.PENDING.value, BuildStatus.PRODUCTION.value], + ) + + def test_complete_output(self): + """The 'completed' quantity (updated via an F() expression) is recorded as a number.""" + self.build.issue_build() + + completed = self.build.completed + + self.build.complete_build_output(self.output_1, None) + self.build.refresh_from_db() + + self.assertEqual(self.build.completed, completed + self.output_1.quantity) + + [entry] = [e for e in self.updates() if 'completed' in e.changes_dict] + + self.assertEqual( + entry.changes_dict['completed'], [completed, self.build.completed] + ) + self.assertNotIn('F(', str(entry.changes)) + + def test_expression_update(self): + """Saving any field as an F() expression records the resolved value.""" + self.build.priority = 5 + self.build.save() + + self.build.priority = F('priority') + 3 + self.build.save(update_fields=['priority']) + + entry = self.updates().last() + self.assertEqual(entry.changes_dict, {'priority': [5, 8]}) + + # An expression which does not change the value is not recorded + n = self.updates().count() + + self.build.priority = F('priority') + 0 + self.build.save(update_fields=['priority']) + + self.assertEqual(self.updates().count(), n) + + def test_failed_save_not_recorded(self): + """A deferred (expression) entry from a failed save is not attached to a later save.""" + from django.core.exceptions import FieldError + from django.db import transaction + + self.build.priority = F('not_a_field') + 1 + + with self.assertRaises(FieldError), transaction.atomic(): + self.build.save(update_fields=['priority']) + + # Subsequent save of an unrelated field + self.build.refresh_from_db() + self.build.title = 'New title' + self.build.save() + + entry = self.updates().last() + self.assertEqual(list(entry.changes_dict.keys()), ['title']) + + def test_delete_reparents_children(self): + """Deleting a build re-parents its child builds - and the change is recorded.""" + child = Build.objects.create( + part=self.assembly, quantity=1, reference='BO-9001', parent=self.build + ) + + grandchild = Build.objects.create( + part=self.assembly, quantity=1, reference='BO-9002', parent=child + ) + + child_pk = child.pk + child.delete() + + grandchild.refresh_from_db() + self.assertEqual(grandchild.parent, self.build) + + entry = build_entries(grandchild.pk, action=ACTION.UPDATE).get() + self.assertEqual(entry.changes_dict['parent'], [child_pk, self.build.pk]) + + +class BuildAuditAPITest(InvenTreeAPITestCase): + """Tests for build audit history via the API.""" + + fixtures = ['category', 'part', 'location', 'bom', 'build', 'build_line', 'stock'] + + roles = ['build.view', 'build.change'] + + def test_edit_via_api(self): + """Edits made via the API are recorded against the user, and shown in the history.""" + build = Build.objects.get(pk=1) + + self.patch( + reverse('api-build-detail', kwargs={'pk': build.pk}), + {'title': 'Updated title', 'priority': 7}, + ) + + entry = build_entries(build.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['title'][1], 'Updated title') + self.assertEqual(entry.changes_dict['priority'][1], 7) + + history = self.get( + reverse('api-audit-log-list'), + {'model_type': 'build.build', 'model_id': build.pk}, + ).data + + self.assertEqual(len(history), 1) + self.assertEqual(history[0]['model_label'], 'Build Order') diff --git a/src/backend/InvenTree/common/filters.py b/src/backend/InvenTree/common/filters.py index 7013d9d303..7f7938d4bd 100644 --- a/src/backend/InvenTree/common/filters.py +++ b/src/backend/InvenTree/common/filters.py @@ -45,10 +45,12 @@ def determine_content_type(content_type: str | int | None) -> ContentType | None # First, try to resolve the content type via a PK value try: content_type_id = int(content_type) - ct = ContentType.objects.get_for_id(content_type_id) + return ContentType.objects.get_for_id(content_type_id) except (ValueError, ContentType.DoesNotExist): ct = None + content_type = str(content_type) + if len(content_type.split('.')) == 2: # Next, try to resolve the content type via app_label.model_name try: diff --git a/src/backend/InvenTree/common/models.py b/src/backend/InvenTree/common/models.py index 9784361e5f..8d60456de3 100644 --- a/src/backend/InvenTree/common/models.py +++ b/src/backend/InvenTree/common/models.py @@ -151,7 +151,9 @@ class UpdatedUserMixin(models.Model): ) -class ProjectCode(InvenTree.models.InvenTreeMetadataModel): +class ProjectCode( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeMetadataModel +): """A ProjectCode is a unique identifier for a project.""" IMPORT_ID_FIELDS = ['code'] @@ -246,7 +248,18 @@ class BaseInvenTreeSetting(UpdatedUserMixin, models.Model): # Execute before_save action self._call_settings_function('before_save', args, kwargs) - super().save() + # Values are stored as strings - convert the (cleaned) value for the database write, + # so that the audit log compares like with like (e.g. 'True' vs True). + # The cleaned (native) value is restored afterwards, as callers may rely on it + value = self.value + + if value is not None and not isinstance(value, str): + self.value = str(value) + + try: + super().save() + finally: + self.value = value # Update this setting in the cache after it was saved so a pk exists if do_cache: @@ -652,6 +665,9 @@ class BaseInvenTreeSetting(UpdatedUserMixin, models.Model): setting = cls(key=key, value=default_value, **extra_fields) + # Creating a setting with its default value is not a user change + setting.audit_skip = True + try: # Wrap this statement in "atomic", so it can be rolled back if it fails with transaction.atomic(): @@ -1258,13 +1274,44 @@ class BaseInvenTreeSetting(UpdatedUserMixin, models.Model): return self.__class__.is_required(self.key, **self.get_filters_for_instance()) -class InvenTreeSetting(BaseInvenTreeSetting): +class InvenTreeSetting(InvenTree.models.InvenTreeAuditMixin, BaseInvenTreeSetting): """An InvenTreeSetting object is a key:value pair used for storing single values (e.g. one-off settings values). The class provides a way of retrieving the value for a particular key, even if that key does not exist. """ + # Only the setting key and value are recorded in the audit log + # (e.g. 'updated_by' is redundant with the audit entry actor) + AUDIT_INCLUDE_FIELDS = ('key', 'value') + + # Settings which are written automatically by the system (not by users) + AUDIT_IGNORE_KEYS = ('SERVER_RESTART_REQUIRED',) + + def __str__(self): + """Return the setting key.""" + return str(self.key) + + def should_audit(self) -> bool: + """Do not record changes to settings which are written by the system. + + - Internal settings (prefixed with '_') are written automatically, + e.g. by every scheduled task run + - Settings which are created automatically (with the default value) when first accessed + """ + if getattr(self, 'audit_skip', False): + return False + + return not (str(self.key).startswith('_') or self.key in self.AUDIT_IGNORE_KEYS) + + def audit_always(self) -> bool: + """Changes to the audit log settings are always recorded (e.g. disabling the audit log).""" + return str(self.key).startswith('AUDIT_LOG_') + + def is_audit_masked(self) -> bool: + """Protected setting values are never recorded.""" + return self.protected + from common.setting.system import SYSTEM_SETTINGS SETTINGS: dict[str, InvenTreeSettingsKeyType] = SYSTEM_SETTINGS @@ -1417,6 +1464,10 @@ class PriceBreak(MetaMixin): help_text=_('Unit price at specified quantity'), ) + def __str__(self): + """Return a string representation of this price break (e.g. 'R_10K - $1.50 @ 10').""" + return f'{self.part} - {self.price} @ {InvenTree.helpers.decimal2string(self.quantity)}' + def convert_to(self, currency_code: str, raise_error: bool = False): """Convert the unit-price at this price break to the specified currency code. @@ -1815,7 +1866,7 @@ class NewsFeedEntry(models.Model): ) -class CustomUnit(models.Model): +class CustomUnit(InvenTree.models.InvenTreeAuditMixin, models.Model): """Model for storing custom physical unit definitions. Model Attributes: @@ -1832,6 +1883,10 @@ class CustomUnit(models.Model): verbose_name = _('Custom Unit') + def __str__(self): + """Return a string representation of this CustomUnit.""" + return self.name + def fmt_string(self): """Construct a unit definition string e.g. 'dog_year = 52 * day = dy'.""" fmt = f'{self.name} = {self.definition}' @@ -1956,6 +2011,7 @@ def rename_attachment(instance, filename: str): class Attachment( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.MetadataMixin, InvenTree.models.InvenTreeTagsMixin, InvenTree.models.InvenTreeModel, @@ -1981,6 +2037,12 @@ class Attachment( THUMBNAIL_SIZE = 256 + # Derived values, updated automatically after upload (and by the thumbnail task) + AUDIT_EXCLUDE_FIELDS = ('thumbnail', 'is_image', 'file_size') + + # Access to an attachment is determined by the object it is linked to + AUDIT_REQUIRE_PARENT_PERMISSION = True + class Meta: """Metaclass options.""" @@ -2060,6 +2122,23 @@ class Attachment( common.tasks.rebuild_attachment, self.pk, group='attachments' ) + def get_audit_parent_reference(self): + """Audit entries for an attachment are included in the history of the linked object. + + The linked object is identified without being fetched, so that the reference + is still recorded when the attachment is deleted along with its object. + """ + from django.contrib.contenttypes.models import ContentType + + try: + model_class = common.validators.attachment_model_class_from_label( + self.model_type + ) + except ValidationError: + return None + + return ContentType.objects.get_for_model(model_class), str(self.model_id) + def clean_svg(self, field): """Sanitize SVG file before saving.""" cleaned = sanitize_svg(field.file.read()) @@ -2632,7 +2711,9 @@ class SelectionListEntry(models.Model): class ParameterTemplate( - InvenTree.models.MetadataMixin, InvenTree.models.InvenTreeModel + InvenTree.models.InvenTreeAuditMixin, + InvenTree.models.MetadataMixin, + InvenTree.models.InvenTreeModel, ): """A ParameterTemplate provides a template for defining parameter values against various models. @@ -2853,7 +2934,10 @@ def post_save_parameter_template(sender, instance, created, **kwargs): class Parameter( - UpdatedUserMixin, InvenTree.models.MetadataMixin, InvenTree.models.InvenTreeModel + InvenTree.models.InvenTreeAuditMixin, + UpdatedUserMixin, + InvenTree.models.MetadataMixin, + InvenTree.models.InvenTreeModel, ): """Class which represents a parameter value assigned to a particular model instance. @@ -2868,6 +2952,16 @@ class Parameter( updated_by: User who last updated this parameter """ + AUDIT_EXCLUDE_FIELDS = ( + # Derived from 'data' + 'data_numeric', + # Redundant with the audit entry actor + 'updated_by', + ) + + # Access to a parameter is determined by the object it is linked to + AUDIT_REQUIRE_PARENT_PERMISSION = True + class Meta: """Meta options for Parameter model.""" @@ -2891,6 +2985,22 @@ class Parameter( """Return the API URL associated with the Parameter model.""" return reverse('api-parameter-list') + def __str__(self): + """Return a string representation of this Parameter (e.g. 'Length: 10 mm').""" + value = f'{self.template.name}: {self.data}' + + if self.template.units: + value += f' {self.template.units}' + + return value + + def get_audit_parent_reference(self): + """Audit entries for a parameter are included in the history of the linked object.""" + if self.model_type_id is None: + return None + + return ContentType.objects.get_for_id(self.model_type_id), str(self.model_id) + def save(self, *args, **kwargs): """Custom save method for Parameter model. diff --git a/src/backend/InvenTree/common/setting/system.py b/src/backend/InvenTree/common/setting/system.py index 6a924fa93d..30c39d68fe 100644 --- a/src/backend/InvenTree/common/setting/system.py +++ b/src/backend/InvenTree/common/setting/system.py @@ -1334,6 +1334,37 @@ SYSTEM_SETTINGS: dict[str, InvenTreeSettingsKeyType] = { 'units': _('days'), 'validator': [int, MinValueValidator(30)], }, + 'AUDIT_LOG_ENABLED': { + 'name': _('Enable Audit Log'), + 'description': _('Record a history of changes made to supported models'), + 'default': True, + 'validator': bool, + }, + 'AUDIT_LOG_DELETE_OLD_ENTRIES': { + 'name': _('Delete Old Audit Log Entries'), + 'description': _( + 'Delete audit log entries older than the specified number of days' + ), + 'default': False, + 'validator': bool, + }, + 'AUDIT_LOG_DELETE_KEEP_SUMMARY': { + 'name': _('Retain Audit Log Summary'), + 'description': _( + 'When deleting old audit log entries, always keep the creation entry and the most recent entry for each item' + ), + 'default': True, + 'validator': bool, + }, + 'AUDIT_LOG_DELETE_DAYS': { + 'name': _('Audit Log Deletion Interval'), + 'description': _( + 'Audit log entries will be deleted after specified number of days' + ), + 'default': 365, + 'units': _('days'), + 'validator': [int, MinValueValidator(7)], + }, 'DISPLAY_FULL_NAMES': { 'name': _('Display Users full names'), 'description': _('Display Users full names instead of usernames'), diff --git a/src/backend/InvenTree/common/setting/user.py b/src/backend/InvenTree/common/setting/user.py index 01e2e9b4af..2cf5c2bff0 100644 --- a/src/backend/InvenTree/common/setting/user.py +++ b/src/backend/InvenTree/common/setting/user.py @@ -249,6 +249,14 @@ USER_SETTINGS: dict[str, InvenTreeSettingsKeyType] = { 'default': True, 'validator': bool, }, + 'DISPLAY_HISTORY_TAB': { + 'name': _('Show Change History'), + 'description': _( + 'Display the change history (audit log) panel on detail pages (staff users only)' + ), + 'default': False, + 'validator': bool, + }, 'ENABLE_LAST_BREADCRUMB': { 'name': _('Show Last Breadcrumb'), 'description': _('Show the current page in breadcrumbs'), diff --git a/src/backend/InvenTree/common/test_audit.py b/src/backend/InvenTree/common/test_audit.py new file mode 100644 index 0000000000..9c4b7ae014 --- /dev/null +++ b/src/backend/InvenTree/common/test_audit.py @@ -0,0 +1,822 @@ +"""Audit log tests for the Attachment model.""" + +import uuid + +from django.core.files.base import ContentFile +from django.urls import reverse + +from audit.models import AuditLogEntry +from common.models import Attachment +from company.models import Company +from InvenTree.unit_test import InvenTreeAPITestCase + +ACTION = AuditLogEntry.Action + + +class AttachmentAuditTest(InvenTreeAPITestCase): + """Tests for audit history of Attachment objects.""" + + roles = [ + 'purchase_order.view', + 'purchase_order.add', + 'purchase_order.change', + 'purchase_order.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.company = Company.objects.create(name='Attachment Co', is_supplier=True) + + def upload(self, company=None, filename=None, **data): + """Upload a file attachment against a company, via the API.""" + company = company or self.company + filename = filename or f'audit_{uuid.uuid4().hex[:8]}.txt' + + response = self.post( + reverse('api-attachment-list'), + data={ + 'model_type': 'company', + 'model_id': company.pk, + 'attachment': ContentFile(b'Hello world', name=filename), + **data, + }, + format='multipart', + expected_code=201, + ) + + attachment = Attachment.objects.get(pk=response.data['pk']) + + # Remove the uploaded file from storage once the test is complete + self.addCleanup( + lambda: ( + attachment.attachment.storage.delete(attachment.attachment.name) + if Attachment.objects.filter(pk=attachment.pk).exists() + else None + ) + ) + + return attachment + + def entries(self, attachment, **filters): + """Return audit entries for an attachment (by PK - which survives deletion).""" + return AuditLogEntry.objects.filter( + content_type=Attachment.get_content_type(), + object_pk=str(attachment if isinstance(attachment, int) else attachment.pk), + **filters, + ) + + def company_history(self, company=None, **params): + """Return the audit history for a company (including attachments).""" + company = company or self.company + + return self.get( + reverse('api-audit-log-list'), + {'model_type': 'company.company', 'model_id': company.pk, **params}, + ).data + + def test_upload(self): + """Uploading an attachment records the filename only, against the linked object.""" + attachment = self.upload(filename='datasheet.txt', comment='Datasheet') + + # Sanity check: the stored path includes the upload directory + self.assertNotEqual(attachment.attachment.name, 'datasheet.txt') + self.assertTrue(attachment.attachment.name.endswith('datasheet.txt')) + + entry = self.entries(attachment, action=ACTION.CREATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['attachment'], [None, 'datasheet.txt']) + self.assertEqual(entry.changes_dict['comment'], [None, 'Datasheet']) + self.assertEqual(entry.object_repr, 'datasheet.txt') + + # Linked to the attached object + self.assertEqual(entry.parent_content_type, Company.get_content_type()) + self.assertEqual(entry.parent_object_pk, str(self.company.pk)) + + # Derived fields are not recorded + for field in ['thumbnail', 'is_image', 'file_size']: + self.assertNotIn(field, entry.changes_dict) + + # The follow-up save (file size) and thumbnail task do not create extra entries + self.assertFalse(self.entries(attachment, action=ACTION.UPDATE).exists()) + + # Shown in the company history + self.assertIn( + ('common.attachment', ACTION.CREATE), + [(e['model_type'], e['action']) for e in self.company_history()], + ) + + def test_lazy_string_value(self): + """Lazy translation strings (e.g. a generated report comment) are recorded as plain strings.""" + from django.utils.translation import gettext_lazy + + self.company.create_attachment( + link='https://example.com/report', comment=gettext_lazy('Generated') + ) + + attachment = Attachment.objects.get(link='https://example.com/report') + + entry = self.entries(attachment, action=ACTION.CREATE).get() + self.assertEqual(entry.changes_dict['comment'], [None, 'Generated']) + + def test_edit_and_rename(self): + """Comment edits and file renames are recorded (filename only).""" + attachment = self.upload(filename='original.txt') + url = reverse('api-attachment-detail', kwargs={'pk': attachment.pk}) + + self.patch(url, {'comment': 'Updated comment'}) + + entry = self.entries(attachment, action=ACTION.UPDATE).get() + self.assertEqual(entry.changes_dict, {'comment': ['', 'Updated comment']}) + + new_name = f'renamed_{uuid.uuid4().hex[:8]}.txt' + self.patch(url, {'filename': new_name}) + + attachment.refresh_from_db() + self.addCleanup( + lambda: attachment.attachment.storage.delete(attachment.attachment.name) + ) + + rename = ( + self + .entries(attachment, action=ACTION.UPDATE) + .filter(changes__has_key='attachment') + .get() + ) + + self.assertEqual(rename.changes_dict['attachment'], ['original.txt', new_name]) + + for value in rename.changes_dict['attachment']: + self.assertNotIn('/', value) + + def test_delete(self): + """Deleted attachments remain in the history of the linked object.""" + attachment = self.upload(filename='obsolete.txt') + pk = attachment.pk + + self.delete(reverse('api-attachment-detail', kwargs={'pk': pk})) + + entry = self.entries(pk, action=ACTION.DELETE).get() + + self.assertEqual(entry.object_repr, 'obsolete.txt') + self.assertEqual(entry.changes_dict['attachment'], ['obsolete.txt', None]) + self.assertEqual(entry.parent_object_pk, str(self.company.pk)) + + self.assertIn( + ('common.attachment', ACTION.DELETE), + [(e['model_type'], e['action']) for e in self.company_history()], + ) + + def test_parent_deleted(self): + """Attachments deleted along with their object retain the parent reference.""" + company = Company.objects.create(name='Temporary Co') + company_pk = company.pk + + attachment = self.upload(company=company, filename='temp.txt') + attachment_pk = attachment.pk + + company.delete() + + entry = self.entries(attachment_pk, action=ACTION.DELETE).get() + self.assertEqual(entry.parent_object_pk, str(company_pk)) + + def test_parent_permission(self): + """Attachment entries require permission to view the linked object.""" + self.upload(filename='private.txt') + + def attachment_entries(): + return [ + e + for e in self.get(reverse('api-audit-log-list')).data + if e['model_type'] == 'common.attachment' + ] + + self.assertEqual(len(attachment_entries()), 1) + + # The Attachment model itself requires no permissions, + # but without access to companies the attachment history is hidden + self.clearRoles() + + self.assertEqual(attachment_entries(), []) + self.assertEqual( + self.get( + reverse('api-audit-log-list'), {'model_type': 'common.attachment'} + ).data, + [], + ) + + entry = self.entries(Attachment.objects.get().pk).first() + self.get( + reverse('api-audit-log-detail', kwargs={'pk': entry.pk}), expected_code=404 + ) + + +class ParameterAuditTest(InvenTreeAPITestCase): + """Tests for audit history of Parameter objects.""" + + roles = ['part.view', 'part.add', 'part.change', 'part.delete'] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + from common.models import ParameterTemplate + from part.models import Part, PartCategory + + super().setUpTestData() + + cls.category = PartCategory.objects.create(name='Resistors') + cls.part = Part.objects.create( + name='R1', description='Resistor', category=cls.category + ) + cls.length = ParameterTemplate.objects.create(name='Length', units='mm') + cls.colour = ParameterTemplate.objects.create(name='Colour') + + def entries(self, **filters): + """Return audit entries for Parameter objects.""" + from common.models import Parameter + + return AuditLogEntry.objects.filter( + content_type=Parameter.get_content_type(), **filters + ) + + def part_history(self, part=None, **params): + """Return the audit history for a part (including parameters).""" + part = part or self.part + + return self.get( + reverse('api-audit-log-list'), + {'model_type': 'part.part', 'model_id': part.pk, **params}, + ).data + + def test_parameter_crud(self): + """Parameter create / update / delete via the API is recorded against the part.""" + from part.models import Part + + response = self.post( + reverse('api-parameter-list'), + { + 'model_type': 'part', + 'model_id': self.part.pk, + 'template': self.length.pk, + 'data': '10', + }, + ) + + pk = response.data['pk'] + url = reverse('api-parameter-detail', kwargs={'pk': pk}) + + self.patch(url, {'data': '12'}) + self.delete(url) + + entries = self.entries(object_pk=str(pk)) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_content_type, Part.get_content_type()) + self.assertEqual(entry.parent_object_pk, str(self.part.pk)) + + # Derived / redundant fields are not recorded + self.assertNotIn('data_numeric', entry.changes_dict) + self.assertNotIn('updated_by', entry.changes_dict) + + update = entries.get(action=ACTION.UPDATE) + self.assertEqual(update.changes_dict, {'data': ['10', '12']}) + self.assertEqual(update.object_repr, 'Length: 12 mm') + + # Shown in the part history (part is not itself audited) + history = self.part_history() + self.assertEqual( + sorted( + e['action'] for e in history if e['model_type'] == 'common.parameter' + ), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + def test_copy_parameters(self): + """Parameters copied in bulk (copy_parameters_from) are recorded.""" + from common.models import Parameter + from part.models import Part + + Parameter.objects.create( + model_type=Part.get_content_type(), + model_id=self.part.pk, + template=self.colour, + data='Blue', + ) + + other = Part.objects.create(name='R2', description='Resistor') + other.copy_parameters_from(self.part) + + copied = Parameter.objects.get(model_id=other.pk, template=self.colour) + + entry = self.entries(object_pk=str(copied.pk), action=ACTION.CREATE).get() + + self.assertEqual(entry.parent_object_pk, str(other.pk)) + self.assertEqual(entry.changes_dict['data'], [None, 'Blue']) + self.assertEqual(entry.object_repr, 'Colour: Blue') + + def test_category_parameters(self): + """Default category parameters (created in bulk) are recorded.""" + from common.models import Parameter + from part.models import Part, PartCategoryParameterTemplate + + PartCategoryParameterTemplate.objects.create( + category=self.category, template=self.length, default_value='5' + ) + + part = Part.objects.create( + name='R3', description='Resistor', category=self.category + ) + part.copy_category_parameters(self.category) + + parameter = Parameter.objects.get(model_id=part.pk, template=self.length) + + entry = self.entries(object_pk=str(parameter.pk), action=ACTION.CREATE).get() + self.assertEqual(entry.parent_object_pk, str(part.pk)) + self.assertEqual(entry.changes_dict['data'], [None, '5']) + + def test_parent_permission(self): + """Parameter entries require permission to view the linked object.""" + from common.models import Parameter + from part.models import Part + + Parameter.objects.create( + model_type=Part.get_content_type(), + model_id=self.part.pk, + template=self.colour, + data='Red', + ) + + def parameter_entries(): + return [ + e + for e in self.get(reverse('api-audit-log-list')).data + if e['model_type'] == 'common.parameter' + ] + + self.assertEqual(len(parameter_entries()), 1) + + self.clearRoles() + + self.assertEqual(parameter_entries(), []) + + def test_bulk_create_without_pk(self): + """Entries are recorded on backends which do not return PKs from bulk_create (e.g. MySQL).""" + from unittest import mock + + from audit.helpers import audited_bulk_create + from common.models import Parameter + from part.models import Part + + original_bulk_create = Parameter.objects.bulk_create + + def bulk_create_without_pk(objs, *args, **kwargs): + """Simulate a database which does not return primary keys.""" + created = original_bulk_create(objs, *args, **kwargs) + + for obj in created: + obj.pk = None + + return created + + objs = [ + Parameter( + model_type=Part.get_content_type(), + model_id=self.part.pk, + template=template, + data=value, + ) + for template, value in [(self.colour, 'Green'), (self.length, '7')] + ] + + with ( + mock.patch( + 'InvenTree.helpers_db.bulk_insert_returns_pks', return_value=False + ), + mock.patch.object( + Parameter.objects, 'bulk_create', side_effect=bulk_create_without_pk + ), + ): + created = audited_bulk_create(Parameter, objs) + + self.assertEqual(len(created), 2) + + for obj, instance in zip(objs, created, strict=True): + # Primary keys are populated on the provided objects + self.assertIsNotNone(obj.pk) + self.assertEqual(obj.pk, instance.pk) + + entry = self.entries(object_pk=str(obj.pk), action=ACTION.CREATE).get() + self.assertEqual(entry.changes_dict['data'], [None, obj.data]) + self.assertEqual(entry.parent_object_pk, str(self.part.pk)) + + def test_copy_preserves_metadata(self): + """Copying parameters (via audited_bulk_create) retains their metadata.""" + from common.models import Parameter + from part.models import Part + + Parameter.objects.create( + model_type=Part.get_content_type(), + model_id=self.part.pk, + template=self.colour, + data='Blue', + metadata={'source': 'datasheet'}, + ) + + other = Part.objects.create(name='R4', description='Resistor') + other.copy_parameters_from(self.part) + + copied = Parameter.objects.get(model_id=other.pk, template=self.colour) + self.assertEqual(copied.metadata, {'source': 'datasheet'}) + + def test_bulk_create_api(self): + """Entries for a bulk API create are written together, with the request context.""" + response = self.post( + reverse('api-parameter-list'), + [ + { + 'model_type': 'part', + 'model_id': self.part.pk, + 'template': template.pk, + 'data': value, + } + for template, value in [(self.length, '10'), (self.colour, 'Red')] + ], + format='json', + ) + + pks = [str(item['pk']) for item in response.data] + entries = self.entries(object_pk__in=pks, action=ACTION.CREATE) + + self.assertEqual(entries.count(), 2) + self.assertEqual(len({entry.cid for entry in entries}), 1) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.remote_addr, '127.0.0.1') + self.assertEqual(entry.parent_object_pk, str(self.part.pk)) + + def test_bulk_create_api_error(self): + """No entries are written if any item in a bulk API create is invalid.""" + self.post( + reverse('api-parameter-list'), + [ + { + 'model_type': 'part', + 'model_id': self.part.pk, + 'template': self.colour.pk, + 'data': 'Red', + }, + {'model_type': 'part', 'model_id': self.part.pk, 'template': 9999}, + ], + format='json', + expected_code=400, + ) + + self.assertFalse(self.entries().exists()) + + def test_bulk_create_single_insert(self): + """audited_bulk_create() writes all audit entries with a single query.""" + from django.db import connection + from django.test.utils import CaptureQueriesContext + + from audit.helpers import audited_bulk_create + from common.models import Parameter + from part.models import Part + + parts = [ + Part.objects.create(name=f'R{idx}', description='Resistor') + for idx in range(5) + ] + + objs = [ + Parameter( + model_type=Part.get_content_type(), + model_id=part.pk, + template=self.colour, + data='Red', + ) + for part in parts + ] + + with CaptureQueriesContext(connection) as context: + audited_bulk_create(Parameter, objs) + + # Table name quoting differs between backends (e.g. backticks for MySQL) + table = AuditLogEntry._meta.db_table + + inserts = [ + q + for q in context.captured_queries + if q['sql'].startswith('INSERT INTO') and table in q['sql'] + ] + + self.assertEqual(len(inserts), 1) + self.assertEqual(self.entries(action=ACTION.CREATE).count(), 5) + + def test_requires_metadata_field(self): + """audited_bulk_create() rejects models without a 'metadata' field.""" + from audit.helpers import audited_bulk_create + from company.models import Address + + with self.assertRaises(TypeError): + audited_bulk_create(Address, []) + + +class ConfigurationAuditTest(InvenTreeAPITestCase): + """Tests for audit history of ParameterTemplate / ProjectCode / CustomUnit objects.""" + + roles = ['part.view', 'part.add', 'part.change', 'part.delete'] + + def entries(self, model, pk, **filters): + """Return audit entries for a given object (by PK - which survives deletion).""" + from django.contrib.contenttypes.models import ContentType + + return AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(model), + object_pk=str(pk), + **filters, + ) + + def history(self, model_type, pk): + """Return the audit history for an object, with change details.""" + return self.get( + reverse('api-audit-log-list'), + {'model_type': model_type, 'model_id': pk, 'changes_detail': True}, + ).data + + def check_crud(self, model, list_url, detail_name, create, update): + """Create, update and delete an object via the API, and check the audit trail. + + Returns the UPDATE entry. + """ + response = self.post(reverse(list_url), create) + pk = response.data['pk'] + + url = reverse(detail_name, kwargs={'pk': pk}) + self.patch(url, update) + self.delete(url) + + entries = self.entries(model, pk) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertIsNone(entry.parent_content_type) + self.assertNotIn('metadata', entry.changes_dict) + + return entries.get(action=ACTION.UPDATE) + + def test_parameter_template(self): + """ParameterTemplate changes are recorded (including choice field labels).""" + from common.models import ParameterTemplate + + update = self.check_crud( + ParameterTemplate, + 'api-parameter-template-list', + 'api-parameter-template-detail', + {'name': 'Resistance', 'units': 'ohm', 'description': 'Resistance'}, + {'units': 'kohm', 'unique': ParameterTemplate.UniqueOptions.GLOBAL}, + ) + + self.assertEqual(update.changes_dict['units'], ['ohm', 'kohm']) + self.assertEqual( + update.changes_dict['unique'], + [ + ParameterTemplate.UniqueOptions.NONE, + ParameterTemplate.UniqueOptions.GLOBAL, + ], + ) + self.assertEqual(update.object_repr, 'Resistance (kohm)') + + def test_parameter_template_display(self): + """Choice field values are shown with their labels in the API.""" + from common.models import ParameterTemplate + + template = ParameterTemplate.objects.create(name='Colour') + template.unique = ParameterTemplate.UniqueOptions.MODEL_TYPE + template.save() + + [entry] = [ + e + for e in self.history('common.parametertemplate', template.pk) + if e['action'] == ACTION.UPDATE + ] + [row] = entry['changes_detail'] + + self.assertEqual(row['field'], 'unique') + self.assertEqual( + row['new_display'], str(ParameterTemplate.UniqueOptions.MODEL_TYPE.label) + ) + + def test_project_code(self): + """ProjectCode changes are recorded (including the responsible owner).""" + from common.models import ProjectCode + from users.models import Owner + + owner = Owner.get_owner(self.user) + + update = self.check_crud( + ProjectCode, + 'api-project-code-list', + 'api-project-code-detail', + {'code': 'PRJ-001', 'description': 'Project one'}, + {'description': 'Project 1', 'responsible': owner.pk, 'active': False}, + ) + + self.assertEqual( + update.changes_dict['description'], ['Project one', 'Project 1'] + ) + self.assertEqual(update.changes_dict['responsible'], [None, owner.pk]) + self.assertEqual(update.changes_dict['active'], [True, False]) + self.assertEqual(update.object_repr, 'PRJ-001') + + def test_custom_unit(self): + """CustomUnit changes are recorded, and the unit registry still reloads.""" + from common.models import CustomUnit + from InvenTree.conversion import convert_physical_value + + update = self.check_crud( + CustomUnit, + 'api-custom-unit-list', + 'api-custom-unit-detail', + {'name': 'furlong', 'definition': '220 yard', 'symbol': 'fur'}, + {'definition': '201.168 meter'}, + ) + + self.assertEqual( + update.changes_dict['definition'], ['220 yard', '201.168 meter'] + ) + self.assertEqual(update.object_repr, 'furlong') + + # Registry reload (post_save receiver) is unaffected by auditing + CustomUnit.objects.create(name='smoot', definition='1.7018 meter') + self.assertAlmostEqual( + float(convert_physical_value('1 smoot', 'meter')), 1.7018, places=4 + ) + + +class GlobalSettingAuditTest(InvenTreeAPITestCase): + """Audit tests for global settings.""" + + roles = 'all' + + def entries(self, **filters): + """Return audit entries for global settings.""" + from django.contrib.contenttypes.models import ContentType + + from common.models import InvenTreeSetting + + return AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(InvenTreeSetting), **filters + ) + + def set_value(self, key, value): + """Change a global setting via the API.""" + self.patch( + reverse('api-global-setting-detail', kwargs={'key': key}), {'value': value} + ) + + def test_setting_change(self): + """Changes to a global setting are recorded (key and value only).""" + self.set_value('INVENTREE_INSTANCE', 'Audit Instance') + + entry = self.entries(object_repr='INVENTREE_INSTANCE').order_by('pk').last() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['value'][1], 'Audit Instance') + + for field in entry.changes_dict: + self.assertIn(field, ['key', 'value']) + + def test_unchanged_value(self): + """Saving a setting with an unchanged value is not recorded.""" + self.set_value('AUDIT_LOG_DELETE_OLD_ENTRIES', True) + + n = self.entries().count() + self.set_value('AUDIT_LOG_DELETE_OLD_ENTRIES', True) + + self.assertEqual(self.entries().count(), n) + + def test_system_settings_not_recorded(self): + """Settings which are written automatically by the system are not recorded.""" + from unittest import mock + + from common.models import InvenTreeSetting + from common.settings import set_global_setting + + n = self.entries().count() + + # Internal settings (e.g. written by every scheduled task run) + set_global_setting('_TEST_TASK_ATTEMPT', 'now', change_user=None) + + # Changing a setting which requires a restart also sets 'SERVER_RESTART_REQUIRED' + # Note: A setting which is not overridden by the environment is used here + # (e.g. PLUGIN_ON_STARTUP is set via environment variable in the docker image) + def requires_restart(setting): + return setting.key == 'INVENTREE_INSTANCE' + + with mock.patch.object( + InvenTreeSetting, + 'requires_restart', + autospec=True, + side_effect=requires_restart, + ): + self.assertFalse(InvenTreeSetting.get_setting('SERVER_RESTART_REQUIRED')) + self.set_value('INVENTREE_INSTANCE', 'Restart Instance') + self.assertTrue(InvenTreeSetting.get_setting('SERVER_RESTART_REQUIRED')) + + keys = list( + self.entries().order_by('pk')[n:].values_list('object_repr', flat=True) + ) + + self.assertEqual(keys, ['INVENTREE_INSTANCE']) + + def test_protected_value(self): + """The values of protected settings are never recorded.""" + from unittest import mock + + from audit.models import MASKED_VALUE + from common.models import InvenTreeSetting + + with mock.patch.object(InvenTreeSetting, 'is_protected', return_value=True): + self.set_value('INVENTREE_INSTANCE', 'secret-value') + + entry = self.entries(object_repr='INVENTREE_INSTANCE').order_by('pk').last() + + self.assertEqual(entry.changes_dict['value'][1], MASKED_VALUE) + + for field in ['changes', 'changes_text', 'additional_data', 'object_repr']: + self.assertNotIn('secret-value', str(getattr(entry, field))) + + def test_disable_audit_log(self): + """Disabling (and re-enabling) the audit log is always recorded.""" + self.set_value('AUDIT_LOG_ENABLED', False) + + entry = self.entries(object_repr='AUDIT_LOG_ENABLED').order_by('pk').last() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['value'], ['True', 'False']) + + # Other changes are no longer recorded + n = AuditLogEntry.objects.count() + self.set_value('INVENTREE_INSTANCE', 'Not recorded') + self.set_value('AUDIT_LOG_DELETE_DAYS', 7) + Company.objects.create(name='Not recorded') + + # ...except for changes to the audit log settings themselves + self.assertEqual(AuditLogEntry.objects.count(), n + 1) + self.assertEqual( + AuditLogEntry.objects.order_by('pk').last().object_repr, + 'AUDIT_LOG_DELETE_DAYS', + ) + + self.set_value('AUDIT_LOG_ENABLED', True) + + entry = self.entries(object_repr='AUDIT_LOG_ENABLED').order_by('pk').last() + self.assertEqual(entry.changes_dict['value'], ['False', 'True']) + + def test_staff_only(self): + """Setting history is only visible to staff users.""" + self.set_value('INVENTREE_INSTANCE', 'Staff only') + + url = reverse('api-audit-log-list') + params = {'model_type': 'common.inventreesetting'} + + self.assertGreater(len(self.get(url, params).data), 0) + + # Staff users without the 'admin' role can still view settings history + self.user.is_superuser = False + self.user.save() + self.clearRoles() + + self.assertGreater(len(self.get(url, params).data), 0) + + def test_default_value_not_recorded(self): + """Settings created automatically (with their default value) are not recorded.""" + from common.models import InvenTreeSetting + from common.settings import get_global_setting + + InvenTreeSetting.objects.filter(key='AUDIT_LOG_DELETE_DAYS').delete() + + n = self.entries().count() + + # Reading the setting creates it (with the default value) + get_global_setting('AUDIT_LOG_DELETE_DAYS') + self.assertTrue( + InvenTreeSetting.objects.filter(key='AUDIT_LOG_DELETE_DAYS').exists() + ) + + self.assertEqual(self.entries().count(), n) diff --git a/src/backend/InvenTree/company/models.py b/src/backend/InvenTree/company/models.py index 3efe6d844a..e9cbc25213 100644 --- a/src/backend/InvenTree/company/models.py +++ b/src/backend/InvenTree/company/models.py @@ -26,6 +26,7 @@ import InvenTree.models import InvenTree.ready import InvenTree.validators import report.mixins +from audit.helpers import audited_update from common.currency import currency_code_default from common.settings import get_global_setting from InvenTree.fields import InvenTreeURLField, RoundingDecimalField @@ -86,6 +87,7 @@ def default_internal_company() -> Optional[int]: class Company( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeAttachmentMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.InvenTreeNoteMixin, @@ -337,7 +339,9 @@ class Company( ).distinct() -class Contact(InvenTree.models.InvenTreeMetadataModel): +class Contact( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeMetadataModel +): """A Contact represents a person who works at a particular company. A Company may have zero or more associated Contact objects. Attributes: @@ -350,6 +354,9 @@ class Contact(InvenTree.models.InvenTreeMetadataModel): IMPORT_ID_FIELDS = ['name', 'email'] + # Changes to a contact are included in the history of the company + AUDIT_PARENT_FIELD = 'company' + class Meta: """Metaclass defines extra model options.""" @@ -372,8 +379,12 @@ class Contact(InvenTree.models.InvenTreeMetadataModel): role = models.CharField(max_length=100, blank=True) + def __str__(self): + """Return a string representation of this Contact.""" + return self.name -class Address(InvenTree.models.InvenTreeModel): + +class Address(InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeModel): """An address represents a physical location where the company is located. It is possible for a company to have multiple locations. Attributes: @@ -391,6 +402,9 @@ class Address(InvenTree.models.InvenTreeModel): link: External link to additional address information """ + # Changes to an address are included in the history of the company + AUDIT_PARENT_FIELD = 'company' + class Meta: """Metaclass defines extra model options.""" @@ -435,9 +449,13 @@ class Address(InvenTree.models.InvenTreeModel): # Once this address is saved, check others if self.primary: - Address.objects.filter(company=self.company).exclude(pk=self.pk).filter( - primary=True - ).update(primary=False) + audited_update( + Address.objects + .filter(company=self.company) + .exclude(pk=self.pk) + .filter(primary=True), + primary=False, + ) @staticmethod def get_api_url(): @@ -543,6 +561,7 @@ class Address(InvenTree.models.InvenTreeModel): class ManufacturerPart( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeAttachmentMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.InvenTreeBarcodeMixin, @@ -562,6 +581,9 @@ class ManufacturerPart( IMPORT_ID_FIELDS = ['MPN'] + # Changes to a manufacturer part are included in the history of the manufacturer + AUDIT_PARENT_FIELD = 'manufacturer' + class Meta: """Metaclass defines extra model options.""" @@ -658,6 +680,7 @@ class ManufacturerPart( class SupplierPart( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeAttachmentMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.MetadataMixin, @@ -689,6 +712,17 @@ class SupplierPart( IMPORT_ID_FIELDS = ['SKU'] + # Changes to a supplier part are included in the history of the supplier + AUDIT_PARENT_FIELD = 'supplier' + + AUDIT_EXCLUDE_FIELDS = ( + # Supplier-provided stock levels (e.g. updated by supplier plugins) are not user edits + 'available', + 'availability_updated', + # Derived from 'pack_quantity' + 'pack_quantity_native', + ) + class Meta: """Metaclass defines extra model options.""" @@ -810,9 +844,13 @@ class SupplierPart( # Once this SupplierPart is saved, check others if self.primary: - SupplierPart.objects.filter(part=self.part).exclude(pk=self.pk).filter( - primary=True - ).update(primary=False) + audited_update( + SupplierPart.objects + .filter(part=self.part) + .exclude(pk=self.pk) + .filter(primary=True), + primary=False, + ) part = models.ForeignKey( 'part.Part', @@ -1055,7 +1093,9 @@ class SupplierPart( return s -class SupplierPriceBreak(common.models.PriceBreak): +class SupplierPriceBreak( + InvenTree.models.InvenTreeAuditMixin, common.models.PriceBreak +): """Represents a quantity price break for a SupplierPart. - Suppliers can offer discounts at larger quantities @@ -1069,6 +1109,9 @@ class SupplierPriceBreak(common.models.PriceBreak): currency: Reference to the currency of this pricebreak (leave empty for base currency) """ + # Changes to a price break are included in the history of the supplier part + AUDIT_PARENT_FIELD = 'part' + class Meta: """Metaclass defines extra model options.""" diff --git a/src/backend/InvenTree/company/test_audit.py b/src/backend/InvenTree/company/test_audit.py new file mode 100644 index 0000000000..bf706f6117 --- /dev/null +++ b/src/backend/InvenTree/company/test_audit.py @@ -0,0 +1,526 @@ +"""Audit log tests for the Company, Contact and Address models.""" + +from django.db.models import F +from django.urls import reverse + +from auditlog.context import disable_auditlog + +from audit.helpers import audited_update +from audit.models import AuditLogEntry +from common.settings import set_global_setting +from company.models import ( + Address, + Company, + Contact, + ManufacturerPart, + SupplierPart, + SupplierPriceBreak, +) +from InvenTree.unit_test import InvenTreeAPITestCase +from part.models import Part + +ACTION = AuditLogEntry.Action + + +class CompanyAuditTest(InvenTreeAPITestCase): + """Tests for audit history of Company / Contact / Address models.""" + + roles = [ + 'purchase_order.view', + 'purchase_order.add', + 'purchase_order.change', + 'purchase_order.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.company = Company.objects.create( + name='Acme Widgets', description='Widget supplier', is_supplier=True + ) + + def history(self, company=None, **params): + """Return the audit history for a company (including contacts / addresses).""" + company = company or self.company + + return self.get( + reverse('api-audit-log-list'), + {'model_type': 'company.company', 'model_id': company.pk, **params}, + ).data + + def entries(self, instance, **filters): + """Return audit entries recorded directly against an instance.""" + return AuditLogEntry.objects.get_for_object(instance).filter(**filters) + + def test_company_edit(self): + """Company edits via the API are recorded against the user.""" + self.patch( + reverse('api-company-detail', kwargs={'pk': self.company.pk}), + {'description': 'Widgets and gadgets', 'is_customer': True}, + ) + + entry = self.entries(self.company, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual( + entry.changes_dict['description'], + ['Widget supplier', 'Widgets and gadgets'], + ) + self.assertEqual(entry.changes_dict['is_customer'], [False, True]) + self.assertNotIn('metadata', entry.changes_dict) + + # Reverse relations are not recorded on creation + created = self.entries(self.company, action=ACTION.CREATE).get() + + self.assertEqual(created.changes_dict['name'], [None, 'Acme Widgets']) + + for field in ['contacts', 'addresses', 'supplied_parts']: + self.assertNotIn(field, created.changes_dict) + + def test_contact_history(self): + """Contact changes are included in the company history.""" + url = reverse('api-contact-list') + + response = self.post(url, {'company': self.company.pk, 'name': 'Jane Doe'}) + contact = Contact.objects.get(pk=response.data['pk']) + + self.patch( + reverse('api-contact-detail', kwargs={'pk': contact.pk}), + {'role': 'Purchasing'}, + ) + self.delete(reverse('api-contact-detail', kwargs={'pk': contact.pk})) + + entries = [e for e in self.history() if e['model_type'] == 'company.contact'] + + self.assertEqual( + sorted(e['action'] for e in entries), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry['parent_model_type'], 'company.company') + self.assertEqual(entry['parent_object_pk'], str(self.company.pk)) + self.assertEqual(entry['actor'], self.user.pk) + + # The contact history is not included when child entries are excluded + self.assertFalse( + any( + e['model_type'] == 'company.contact' + for e in self.history(include_children=False) + ) + ) + + def test_primary_address(self): + """Demoting the previous primary address (a bulk update) is recorded.""" + url = reverse('api-address-list') + + first = self.post( + url, {'company': self.company.pk, 'title': 'Head Office', 'line1': 'A'} + ).data + self.assertTrue(first['primary']) + + self.post( + url, + { + 'company': self.company.pk, + 'title': 'Warehouse', + 'line1': 'B', + 'primary': True, + }, + ) + + first = Address.objects.get(pk=first['pk']) + self.assertFalse(first.primary) + + entry = self.entries(first, action=ACTION.UPDATE).get() + + self.assertEqual(entry.changes_dict, {'primary': [True, False]}) + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(self.company.pk)) + self.assertIsNotNone(entry.cid) + + def test_company_delete(self): + """Deleting a company records the cascaded contact / address deletions.""" + company = Company.objects.create(name='Short Lived Ltd') + contact = Contact.objects.create(company=company, name='John Smith') + address = Address.objects.create(company=company, title='HQ', line1='X') + + # Note: delete() clears the PK of the deleted instance + targets = [(type(obj), str(obj.pk)) for obj in [company, contact, address]] + company_pk = company.pk + + company.delete() + + for model, pk in targets: + self.assertEqual( + AuditLogEntry.objects.filter( + content_type=model.get_content_type(), + object_pk=pk, + action=ACTION.DELETE, + ).count(), + 1, + ) + + # History of the deleted company is still available + actions = [ + (e['model_type'], e['action']) + for e in self.get( + reverse('api-audit-log-list'), + {'model_type': 'company.company', 'model_id': company_pk}, + ).data + ] + + for model_type in ['company.company', 'company.contact', 'company.address']: + self.assertIn((model_type, ACTION.CREATE), actions) + self.assertIn((model_type, ACTION.DELETE), actions) + + +class AuditedUpdateTest(InvenTreeAPITestCase): + """Tests for the audited_update helper.""" + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.companies = [ + Company.objects.create(name=f'Company {idx}', description='Original') + for idx in range(3) + ] + + def update_entries(self): + """Return all UPDATE entries for Company objects.""" + return AuditLogEntry.objects.filter( + content_type=Company.get_content_type(), action=ACTION.UPDATE + ) + + def test_update(self): + """Each updated row gets an entry - matching the signal-based format.""" + n = audited_update( + Company.objects.filter(pk__in=[c.pk for c in self.companies[:2]]), + description='Bulk', + ) + + self.assertEqual(n, 2) + self.assertEqual(self.update_entries().count(), 2) + + for entry in self.update_entries(): + self.assertEqual(entry.changes_dict, {'description': ['Original', 'Bulk']}) + + # Compare with an entry generated by a regular save() + company = self.companies[2] + company.description = 'Bulk' + company.save() + + entry = self.update_entries().get(object_pk=str(company.pk)) + self.assertEqual(entry.changes_dict, {'description': ['Original', 'Bulk']}) + + def test_unchanged_rows(self): + """Rows which are not actually modified do not produce entries.""" + n = audited_update(Company.objects.all(), description='Original') + + self.assertEqual(n, 3) + self.assertEqual(self.update_entries().count(), 0) + + def test_filter_on_updated_field(self): + """Updating the field the queryset is filtered on still records each row.""" + audited_update( + Company.objects.filter(description='Original'), description='Changed' + ) + + self.assertEqual(self.update_entries().count(), 3) + + def test_expression(self): + """F() expressions are supported (new values are re-read from the database).""" + audited_update(Company.objects.all(), name=F('description')) + + for entry in self.update_entries(): + self.assertEqual(entry.changes_dict['name'][1], 'Original') + + def test_disabled(self): + """No entries are created when auditing is disabled.""" + with disable_auditlog(): + n = audited_update(Company.objects.all(), description='A') + + self.assertEqual(n, 3) + self.assertEqual(self.update_entries().count(), 0) + + set_global_setting('AUDIT_LOG_ENABLED', False) + self.addCleanup(set_global_setting, 'AUDIT_LOG_ENABLED', True) + + audited_update(Company.objects.all(), description='B') + + self.assertEqual(self.update_entries().count(), 0) + self.assertEqual(Company.objects.filter(description='B').count(), 3) + + def test_bulk_update(self): + """audited_bulk_update() records an entry for each modified row.""" + from audit.helpers import audited_bulk_update + + a, b, c = self.companies + + a.description = 'Bulk A' + b.description = 'Bulk B' + # 'c' is included, but not modified + + n = audited_bulk_update(Company, [a, b, c], ['description']) + + self.assertEqual(n, 3) + self.assertEqual(self.update_entries().count(), 2) + + for company in [a, b]: + entry = self.update_entries().get(object_pk=str(company.pk)) + self.assertEqual( + entry.changes_dict, {'description': ['Original', company.description]} + ) + + def test_bulk_update_expression(self): + """audited_bulk_update() records the value written by an F() expression.""" + from audit.helpers import audited_bulk_update + + for company in self.companies: + company.name = F('description') + + audited_bulk_update(Company, self.companies, ['name']) + + self.assertEqual(self.update_entries().count(), 3) + + for entry in self.update_entries(): + self.assertEqual(entry.changes_dict['name'][1], 'Original') + + def test_bulk_update_fields_only(self): + """Only the fields passed to bulk_update() are recorded.""" + from audit.helpers import audited_bulk_update + + company = self.companies[0] + company.description = 'Changed' + company.website = 'https://example.com' + + audited_bulk_update(Company, [company], ['description']) + + entry = self.update_entries().get() + self.assertEqual(list(entry.changes_dict.keys()), ['description']) + + def test_bulk_update_disabled(self): + """No entries are recorded by audited_bulk_update() when auditing is disabled.""" + from audit.helpers import audited_bulk_update + + for company in self.companies: + company.description = 'Quiet' + + with disable_auditlog(): + audited_bulk_update(Company, self.companies, ['description']) + + self.assertEqual(self.update_entries().count(), 0) + self.assertEqual(Company.objects.filter(description='Quiet').count(), 3) + + def test_unaudited_model(self): + """Models which are not audited fall back to a plain update.""" + from django.contrib.contenttypes.models import ContentType + + from audit.helpers import is_audited + from stock.models import StockLocationType + + self.assertFalse(is_audited(StockLocationType)) + + StockLocationType.objects.create(name='Type', description='x') + + n = audited_update(StockLocationType.objects.all(), description='y') + + self.assertEqual(n, 1) + self.assertEqual(StockLocationType.objects.get().description, 'y') + self.assertFalse( + AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(StockLocationType) + ).exists() + ) + + +class SupplierPartAuditTest(InvenTreeAPITestCase): + """Tests for audit history of ManufacturerPart / SupplierPart / SupplierPriceBreak.""" + + roles = [ + 'part.view', + 'purchase_order.view', + 'purchase_order.add', + 'purchase_order.change', + 'purchase_order.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.supplier = Company.objects.create(name='Supplier Co', is_supplier=True) + cls.manufacturer = Company.objects.create(name='Maker Co', is_manufacturer=True) + cls.part = Part.objects.create( + name='Widget', description='A widget', purchaseable=True + ) + cls.supplier_part = SupplierPart.objects.create( + part=cls.part, supplier=cls.supplier, SKU='WID-001' + ) + + def entries(self, instance, **filters): + """Return audit entries recorded directly against an instance.""" + return AuditLogEntry.objects.get_for_object(instance).filter(**filters) + + def history(self, model_type, pk, **params): + """Return the audit history for an object (including children).""" + return self.get( + reverse('api-audit-log-list'), + {'model_type': model_type, 'model_id': pk, **params}, + ).data + + def test_supplier_part_edit(self): + """Supplier part edits are recorded, and shown in the supplier history.""" + self.patch( + reverse('api-supplier-part-detail', kwargs={'pk': self.supplier_part.pk}), + {'note': 'Preferred source', 'pack_quantity': '10'}, + ) + + entry = self.entries(self.supplier_part, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(self.supplier.pk)) + self.assertEqual(entry.changes_dict['note'][1], 'Preferred source') + self.assertEqual(entry.changes_dict['pack_quantity'][1], '10') + + # Derived field is not recorded + self.assertNotIn('pack_quantity_native', entry.changes_dict) + + history = self.history('company.company', self.supplier.pk) + self.assertIn( + ('company.supplierpart', ACTION.UPDATE), + [(e['model_type'], e['action']) for e in history], + ) + + def test_availability_not_recorded(self): + """Supplier availability updates are not recorded.""" + self.supplier_part.update_available_quantity(250) + + self.assertFalse( + self.entries(self.supplier_part, action=ACTION.UPDATE).exists() + ) + + # Other changes in the same save are still recorded + self.supplier_part.note = 'Checked' + self.supplier_part.update_available_quantity(300) + + entry = self.entries(self.supplier_part, action=ACTION.UPDATE).get() + self.assertEqual(list(entry.changes_dict.keys()), ['note']) + + def test_primary_supplier_part(self): + """Demoting the previous primary supplier part (a bulk update) is recorded.""" + self.assertTrue(self.supplier_part.primary) + + SupplierPart.objects.create( + part=self.part, supplier=self.supplier, SKU='WID-002', primary=True + ) + + self.supplier_part.refresh_from_db() + self.assertFalse(self.supplier_part.primary) + + entry = self.entries(self.supplier_part, action=ACTION.UPDATE).get() + self.assertEqual(entry.changes_dict, {'primary': [True, False]}) + self.assertEqual(entry.parent_object_pk, str(self.supplier.pk)) + + def test_manufacturer_part(self): + """Manufacturer part changes are shown in the manufacturer history.""" + response = self.post( + reverse('api-manufacturer-part-list'), + {'part': self.part.pk, 'manufacturer': self.manufacturer.pk, 'MPN': 'M-1'}, + ) + + mp = ManufacturerPart.objects.get(pk=response.data['pk']) + + self.patch( + reverse('api-manufacturer-part-detail', kwargs={'pk': mp.pk}), + {'description': 'Rev B'}, + ) + + history = self.history('company.company', self.manufacturer.pk) + entries = [e for e in history if e['model_type'] == 'company.manufacturerpart'] + + self.assertEqual( + sorted(e['action'] for e in entries), [ACTION.CREATE, ACTION.UPDATE] + ) + + for entry in entries: + self.assertEqual(entry['parent_object_pk'], str(self.manufacturer.pk)) + self.assertEqual(entry['actor'], self.user.pk) + + def test_price_breaks(self): + """Price break changes are shown in the supplier part history.""" + url = reverse('api-part-supplier-price-list') + + response = self.post( + url, + { + 'part': self.supplier_part.pk, + 'quantity': 10, + 'price': '1.50', + 'price_currency': 'USD', + }, + ) + + pb = SupplierPriceBreak.objects.get(pk=response.data['pk']) + + self.patch( + reverse('api-part-supplier-price-detail', kwargs={'pk': pb.pk}), + {'price': '1.25'}, + ) + + entry = self.entries(pb, action=ACTION.UPDATE).get() + self.assertIn('price', entry.changes_dict) + self.assertEqual(entry.parent_object_pk, str(self.supplier_part.pk)) + + history = self.history( + 'company.supplierpart', self.supplier_part.pk, changes_detail=True + ) + entries = [ + e for e in history if e['model_type'] == 'company.supplierpricebreak' + ] + + self.assertEqual( + sorted(e['action'] for e in entries), [ACTION.CREATE, ACTION.UPDATE] + ) + + # Price breaks are not included in the supplier (company) history + self.assertFalse( + any( + e['model_type'] == 'company.supplierpricebreak' + for e in self.history('company.company', self.supplier.pk) + ) + ) + + def test_price_break_permission(self): + """Price break history requires permission on the price break model.""" + SupplierPriceBreak.objects.create(part=self.supplier_part, quantity=1, price=2) + + def price_entries(): + return [ + e + for e in self.history('company.supplierpart', self.supplier_part.pk) + if e['model_type'] == 'company.supplierpricebreak' + ] + + self.assertEqual(len(price_entries()), 1) + + # Supplier part is visible via the 'part' role, but companies / price breaks are not + self.clearRoles() + self.assignRole('part.view') + + history = self.history('company.supplierpart', self.supplier_part.pk) + self.assertGreater(len(history), 0) + self.assertEqual(len(price_entries()), 0) + + # Parent (supplier company) reference is hidden + for entry in history: + self.assertIsNone(entry['parent_model_type']) + self.assertIsNone(entry['parent_object_pk']) + + # Supplier part entries cannot be retrieved via the supplier + self.assertEqual(self.history('company.company', self.supplier.pk), []) diff --git a/src/backend/InvenTree/importer/models.py b/src/backend/InvenTree/importer/models.py index 6c6f1f6f34..157fd72128 100644 --- a/src/backend/InvenTree/importer/models.py +++ b/src/backend/InvenTree/importer/models.py @@ -15,6 +15,7 @@ from django.urls import reverse from django.utils.translation import gettext_lazy as _ import structlog +from auditlog.context import disable_auditlog from rest_framework.exceptions import ValidationError as DRFValidationError import importer.operations @@ -1014,7 +1015,9 @@ class DataImportRow(models.Model): if commit: try: - serializer.save() + # Data imports are not recorded in the audit log + with disable_auditlog(): + serializer.save() self.complete = True except ValueError as e: # Exception as e: diff --git a/src/backend/InvenTree/importer/tests.py b/src/backend/InvenTree/importer/tests.py index 706c44b203..f9a4dc271b 100644 --- a/src/backend/InvenTree/importer/tests.py +++ b/src/backend/InvenTree/importer/tests.py @@ -82,6 +82,36 @@ class ImporterTest(ImporterMixin, InvenTreeTestCase): # Check that the new companies have been created self.assertEqual(n + 12, Company.objects.count()) + def test_import_not_audited(self): + """Records created via a data import are not recorded in the audit log.""" + from audit.models import AuditLogEntry + from company.models import Company + + content_type = Company.get_content_type() + n = AuditLogEntry.objects.filter(content_type=content_type).count() + + session = DataImportSession.objects.create( + data_file=self.helper_file('companies.csv'), model_type='company' + ) + session.extract_columns() + session.import_data() + + companies = Company.objects.count() + + for row in session.rows.all(): + self.assertTrue(row.validate(commit=True)) + + self.assertGreater(Company.objects.count(), companies) + self.assertEqual( + AuditLogEntry.objects.filter(content_type=content_type).count(), n + ) + + # Sanity check: companies created outside of the importer *are* recorded + Company.objects.create(name='Not imported') + self.assertEqual( + AuditLogEntry.objects.filter(content_type=content_type).count(), n + 1 + ) + def test_row_data_datetime_serialization(self): """Test that row data containing native datetime values can be saved. diff --git a/src/backend/InvenTree/order/models.py b/src/backend/InvenTree/order/models.py index edd84a6a4f..056adf2daa 100644 --- a/src/backend/InvenTree/order/models.py +++ b/src/backend/InvenTree/order/models.py @@ -32,6 +32,7 @@ import order.validators import report.mixins import stock.models import users.models as UserModels +from audit.helpers import audited_bulk_update from build.status_codes import BuildStatus from common.currency import currency_code_default from common.notifications import InvenTreeNotificationBodies @@ -88,6 +89,32 @@ from stock.status_codes import StockHistoryCode, StockStatus logger = structlog.get_logger('inventree') +class AuditedOrderMixin(InvenTree.models.InvenTreeAuditMixin): + """Audit log configuration for order models.""" + + AUDIT_EXCLUDE_FIELDS = ( + # Recalculated automatically on every save + 'total_price', + 'total_price_currency', + # Derived from 'barcode_data' + 'barcode_hash', + ) + + +class AuditedLineItemMixin(InvenTree.models.InvenTreeAuditMixin): + """Audit log configuration for order line item models. + + Changes to line items are included in the history of the order. + """ + + AUDIT_PARENT_FIELD = 'order' + + AUDIT_EXCLUDE_FIELDS = ( + # Additional (plugin) context data for extra lines + 'context', + ) + + class TotalPriceMixin(models.Model): """Mixin which provides 'total_price' field for an order.""" @@ -642,7 +669,7 @@ class Order( raise NotImplementedError(f'get_status_class() not implemented for {__class__}') -class PurchaseOrder(TotalPriceMixin, Order): +class PurchaseOrder(AuditedOrderMixin, TotalPriceMixin, Order): """A PurchaseOrder represents goods shipped inwards from an external supplier. Attributes: @@ -1366,7 +1393,7 @@ class PurchaseOrder(TotalPriceMixin, Order): ) # Update received quantity for each line item - PurchaseOrderLineItem.objects.bulk_update(line_items_to_update, ['received']) + audited_bulk_update(PurchaseOrderLineItem, line_items_to_update, ['received']) # Trigger an event for any interested plugins trigger_event( @@ -1451,7 +1478,7 @@ SERIALIZED_STOCK_CHOICES = [ SERIALIZED_STOCK_DEFAULT = 'any' -class SalesOrder(TotalPriceMixin, Order): +class SalesOrder(AuditedOrderMixin, TotalPriceMixin, Order): """A SalesOrder represents a list of goods shipped outwards to a customer.""" REFERENCE_PATTERN_SETTING = 'SALESORDER_REFERENCE_PATTERN' @@ -2286,6 +2313,14 @@ class OrderExtraLine(OrderLineItem): abstract = True + def __str__(self): + """Render a string representation of an extra line item.""" + return '{n} x {description} - {order}'.format( + n=decimal2string(self.quantity), + description=self.description or self.reference or _('Extra line'), + order=self.order, + ) + description = models.CharField( max_length=250, blank=True, @@ -2311,7 +2346,7 @@ class OrderExtraLine(OrderLineItem): ) -class PurchaseOrderLineItem(OrderLineItem): +class PurchaseOrderLineItem(AuditedLineItemMixin, OrderLineItem): """Model for a purchase order line item. Attributes: @@ -2507,7 +2542,7 @@ class PurchaseOrderLineItem(OrderLineItem): self.save() -class PurchaseOrderExtraLine(OrderExtraLine): +class PurchaseOrderExtraLine(AuditedLineItemMixin, OrderExtraLine): """Model for a single ExtraLine in a PurchaseOrder. Attributes: @@ -2533,7 +2568,7 @@ class PurchaseOrderExtraLine(OrderExtraLine): ) -class SalesOrderLineItem(OrderLineItem): +class SalesOrderLineItem(AuditedLineItemMixin, OrderLineItem): """Model for a single LineItem in a SalesOrder. Attributes: @@ -2576,6 +2611,14 @@ class SalesOrderLineItem(OrderLineItem): 'part': _('Only salable parts can be assigned to a sales order') }) + def __str__(self): + """Render a string representation of a SalesOrderLineItem instance.""" + return '{n} x {part} - {order}'.format( + n=decimal2string(self.quantity), + part=self.part.full_name if self.part else _('Unknown part'), + order=self.order, + ) + order = models.ForeignKey( SalesOrder, on_delete=models.CASCADE, @@ -2689,6 +2732,7 @@ class SalesOrderShipmentReportContext(report.mixins.BaseReportContext, TypedDict class SalesOrderShipment( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.InvenTreeAttachmentMixin, InvenTree.models.InvenTreeBarcodeMixin, @@ -2713,6 +2757,14 @@ class SalesOrderShipment( reference: Custom reference text for this shipment (e.g. consignment number?) """ + # Changes to a shipment are included in the history of the sales order + AUDIT_PARENT_FIELD = 'order' + + AUDIT_EXCLUDE_FIELDS = ( + # Derived from 'barcode_data' + 'barcode_hash', + ) + @classmethod def barcode_model_type_code(cls): """Return the associated barcode model type code for this model.""" @@ -2762,6 +2814,12 @@ class SalesOrderShipment( 'title': str(self), } + def __str__(self): + """Render a string representation of a SalesOrderShipment instance.""" + return '{order} - {label} {reference}'.format( + order=self.order, label=_('Shipment'), reference=self.reference + ) + order = models.ForeignKey( SalesOrder, on_delete=models.CASCADE, @@ -3074,7 +3132,7 @@ class SalesOrderShipment( stock.models.StockItemTracking.objects.bulk_create(tracking_entries) # Update sales order lines for "seen_lines" - SalesOrderLineItem.objects.bulk_update(seen_lines.values(), ['shipped']) + audited_bulk_update(SalesOrderLineItem, list(seen_lines.values()), ['shipped']) # Repoint allocations onto their (possibly newly split) StockItem if allocations_to_update: @@ -3146,7 +3204,7 @@ class SalesOrderShipment( return task_id -class SalesOrderExtraLine(OrderExtraLine): +class SalesOrderExtraLine(AuditedLineItemMixin, OrderExtraLine): """Model for a single ExtraLine in a SalesOrder. Attributes: @@ -3317,7 +3375,7 @@ class SalesOrderAllocation(models.Model): self.refresh_from_db() -class ReturnOrder(TotalPriceMixin, Order): +class ReturnOrder(AuditedOrderMixin, TotalPriceMixin, Order): """A ReturnOrder represents goods returned from a customer, e.g. an RMA or warranty. Attributes: @@ -3642,7 +3700,7 @@ class ReturnOrder(TotalPriceMixin, Order): ) -class ReturnOrderLineItem(StatusCodeMixin, OrderLineItem): +class ReturnOrderLineItem(AuditedLineItemMixin, StatusCodeMixin, OrderLineItem): """Model for a single LineItem in a ReturnOrder.""" STATUS_CLASS = ReturnOrderLineStatus @@ -3681,6 +3739,12 @@ class ReturnOrderLineItem(StatusCodeMixin, OrderLineItem): 'quantity': _('Invalid quantity for serialized stock item') }) + def __str__(self): + """Render a string representation of a ReturnOrderLineItem instance.""" + return '{item} - {order}'.format( + item=self.item if self.item_id else _('Unknown item'), order=self.order + ) + order = models.ForeignKey( ReturnOrder, on_delete=models.CASCADE, @@ -3734,7 +3798,7 @@ class ReturnOrderLineItem(StatusCodeMixin, OrderLineItem): ) -class ReturnOrderExtraLine(OrderExtraLine): +class ReturnOrderExtraLine(AuditedLineItemMixin, OrderExtraLine): """Model for a single ExtraLine in a ReturnOrder.""" class Meta: @@ -3756,7 +3820,7 @@ class ReturnOrderExtraLine(OrderExtraLine): ) -class TransferOrder(Order): +class TransferOrder(AuditedOrderMixin, Order): """A Transfer Order represents a request to transfer stock from one location to another. It provides a place to queue and review changes before execution. Attributes: @@ -4088,7 +4152,7 @@ class TransferOrder(Order): return TransferOrderAllocation.objects.filter(line__order=self) -class TransferOrderLineItem(OrderLineItem): +class TransferOrderLineItem(AuditedLineItemMixin, OrderLineItem): """Model for a single LineItem in a TransferOrder. Attributes: @@ -4114,6 +4178,14 @@ class TransferOrderLineItem(OrderLineItem): """Return the API URL associated with the TransferOrderLineItem model.""" return reverse('api-transfer-order-line-list') + def __str__(self): + """Render a string representation of a TransferOrderLineItem instance.""" + return '{n} x {part} - {order}'.format( + n=decimal2string(self.quantity), + part=self.part.full_name if self.part else _('Unknown part'), + order=self.order, + ) + order = models.ForeignKey( TransferOrder, on_delete=models.CASCADE, diff --git a/src/backend/InvenTree/order/test_api.py b/src/backend/InvenTree/order/test_api.py index 396c12d72c..be7519e1bf 100644 --- a/src/backend/InvenTree/order/test_api.py +++ b/src/backend/InvenTree/order/test_api.py @@ -2138,6 +2138,7 @@ class PurchaseOrderReceiveTest(OrderTest): N_ITEMS = StockItem.objects.count() # Receive all items in a single request + # Includes a fixed overhead (independent of N_LINES) for recording audit entries response = self.post( url, { @@ -2151,7 +2152,7 @@ class PurchaseOrderReceiveTest(OrderTest): ], 'location': location.pk, }, - max_query_count=104 + 3 * N_LINES, + max_query_count=124 + 3 * N_LINES, ).data # Check for expected response @@ -2273,11 +2274,12 @@ class PurchaseOrderReceiveTest(OrderTest): 'location': location.pk, } + # Includes a fixed overhead (independent of N_LINES) for recording audit entries with self.settings( PLUGIN_TESTING_EVENTS=True, PLUGIN_TESTING_EVENTS_ASYNC=True ): response = self.post( - url, data, max_query_count=400, benchmark=True, format='json' + url, data, max_query_count=420, benchmark=True, format='json' ) self.assertEqual(response.status_code, 201) diff --git a/src/backend/InvenTree/order/test_audit.py b/src/backend/InvenTree/order/test_audit.py new file mode 100644 index 0000000000..99691e7041 --- /dev/null +++ b/src/backend/InvenTree/order/test_audit.py @@ -0,0 +1,568 @@ +"""Audit log tests for the order models.""" + +from decimal import Decimal + +from django.contrib.contenttypes.models import ContentType +from django.urls import reverse + +from audit.models import AuditLogEntry +from order import models +from order.status_codes import PurchaseOrderStatus +from order.test_api import OrderTest +from stock.models import StockLocation + +ACTION = AuditLogEntry.Action + + +class OrderAuditTestBase(OrderTest): + """Base class for order audit tests.""" + + roles = [ + f'{ruleset}.{perm}' + for ruleset in [ + 'purchase_order', + 'sales_order', + 'return_order', + 'transfer_order', + ] + for perm in ['view', 'add', 'change', 'delete'] + ] + ['part.view', 'stock.view', 'stock.add', 'stock_location.view'] + + def entries(self, model, pk, **filters): + """Return audit entries for a given object (by PK - which survives deletion).""" + return AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(model), + object_pk=str(pk), + **filters, + ) + + def history(self, model_type: str, pk, **params): + """Return the audit history for an object (including child entries).""" + return self.get( + reverse('api-audit-log-list'), + {'model_type': model_type, 'model_id': pk, **params}, + ).data + + def check_child_crud( + self, parent, child_model, list_url, detail_url, create, update + ): + """Create / update / delete a child (line) object via the API. + + Checks that each entry is linked to the parent order, and appears in its history. + + Returns: + The UPDATE entry + """ + pk = self.post(reverse(list_url), create).data['pk'] + + url = reverse(detail_url, kwargs={'pk': pk}) + self.patch(url, update) + self.delete(url) + + entries = self.entries(child_model, pk) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(parent.pk)) + self.assertNotIn('context', entry.changes_dict) + self.assertNotIn('line_int', entry.changes_dict) + + # A meaningful string representation (not the Django default) + self.assertNotIn(' object (', entry.object_repr) + + label = f'{child_model._meta.app_label}.{child_model._meta.model_name}' + parent_label = f'{parent._meta.app_label}.{parent._meta.model_name}' + + child_actions = [ + e['action'] + for e in self.history(parent_label, parent.pk) + if e['model_type'] == label + ] + + self.assertEqual( + sorted(child_actions), [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE] + ) + + return entries.get(action=ACTION.UPDATE) + + +class PurchaseOrderAuditTest(OrderAuditTestBase): + """Audit tests for PurchaseOrder, PurchaseOrderLineItem and PurchaseOrderExtraLine.""" + + def test_order_edit(self): + """Order edits are recorded - without the (derived) total price.""" + po = models.PurchaseOrder.objects.get(pk=1) + + self.patch( + reverse('api-po-detail', kwargs={'pk': po.pk}), + {'description': 'Audited description', 'supplier_reference': 'SUP-1'}, + ) + + entry = self.entries(models.PurchaseOrder, po.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['description'][1], 'Audited description') + self.assertEqual(entry.changes_dict['supplier_reference'][1], 'SUP-1') + + for field in ['total_price', 'total_price_currency', 'updated_at']: + self.assertNotIn(field, entry.changes_dict) + + def test_order_currency(self): + """Changes to the order currency (not a MoneyField currency) are recorded.""" + po = models.PurchaseOrder.objects.get(pk=1) + po.order_currency = 'AUD' + po.save() + + entry = ( + self + .entries(models.PurchaseOrder, po.pk, action=ACTION.UPDATE) + .order_by('pk') + .last() + ) + self.assertEqual(entry.changes_dict['order_currency'][1], 'AUD') + + def test_total_price_not_recorded(self): + """Saving an order (which recalculates the total price) records no entry.""" + po = models.PurchaseOrder.objects.get(pk=1) + + # Initial save (fixture data has no creation_date, which is set on save) + po.save() + n = self.entries(models.PurchaseOrder, po.pk, action=ACTION.UPDATE).count() + + po.refresh_from_db() + total = po.total_price + + # Change the price of a line, then re-save the order (total price changes) + line = po.lines.first() + line.purchase_price = 123.45 + line.save() + + po.save() + po.refresh_from_db() + + # Sanity check - the total price has changed + self.assertNotEqual(po.total_price, total) + + self.assertEqual( + self.entries(models.PurchaseOrder, po.pk, action=ACTION.UPDATE).count(), n + ) + + def test_status_change(self): + """Issuing an order (status change) is recorded.""" + po = models.PurchaseOrder.objects.get(pk=1) + self.assertEqual(po.status, PurchaseOrderStatus.PENDING) + + self.post(reverse('api-po-issue', kwargs={'pk': po.pk}), {}, expected_code=201) + + entry = self.entries(models.PurchaseOrder, po.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual( + entry.changes_dict['status'], + [PurchaseOrderStatus.PENDING.value, PurchaseOrderStatus.PLACED.value], + ) + + def test_line_items(self): + """Line item changes are recorded against the order.""" + po = models.PurchaseOrder.objects.get(pk=1) + supplier_part = po.lines.first().part + + update = self.check_child_crud( + po, + models.PurchaseOrderLineItem, + 'api-po-line-list', + 'api-po-line-detail', + {'order': po.pk, 'part': supplier_part.pk, 'quantity': 5}, + # Note: the serializer requires 'order' and 'part', even for PATCH requests + # (pre-existing: validate() does not fall back to the instance values) + { + 'order': po.pk, + 'part': supplier_part.pk, + 'quantity': 8, + 'reference': 'Audit', + }, + ) + + # Note: decimal values are recorded as strings + self.assertEqual([Decimal(v) for v in update.changes_dict['quantity']], [5, 8]) + self.assertEqual(update.changes_dict['reference'], ['', 'Audit']) + + # Numeric fields are flagged for display formatting (decimals are stored as strings) + [entry] = [ + e + for e in self.history( + 'order.purchaseorder', po.pk, changes_detail=True, action=ACTION.UPDATE + ) + if e['pk'] == update.pk + ] + rows = {row['field']: row for row in entry['changes_detail']} + + self.assertTrue(rows['quantity']['numeric']) + self.assertFalse(rows['reference']['numeric']) + + def test_extra_lines(self): + """Extra line changes are recorded against the order.""" + po = models.PurchaseOrder.objects.get(pk=1) + + update = self.check_child_crud( + po, + models.PurchaseOrderExtraLine, + 'api-po-extra-line-list', + 'api-po-extra-line-detail', + { + 'order': po.pk, + 'quantity': 1, + 'description': 'Freight', + 'price': '10.00', + 'price_currency': 'USD', + }, + {'description': 'Express freight', 'price': '25.00'}, + ) + + self.assertEqual( + update.changes_dict['description'], ['Freight', 'Express freight'] + ) + self.assertIn('price', update.changes_dict) + + def test_receive(self): + """Received quantities (updated in bulk) are recorded against the user.""" + po = models.PurchaseOrder.objects.get(pk=1) + po.status = PurchaseOrderStatus.PLACED.value + po.save() + + lines = list(po.lines.all()[:2]) + location = StockLocation.objects.filter(structural=False).first() + + self.post( + reverse('api-po-receive', kwargs={'pk': po.pk}), + { + 'items': [ + {'line_item': line.pk, 'quantity': 3, 'status': 10} + for line in lines + ], + 'location': location.pk, + }, + expected_code=201, + ) + + for line in lines: + entry = self.entries( + models.PurchaseOrderLineItem, + line.pk, + action=ACTION.UPDATE, + changes__has_key='received', + ).get() + + self.assertEqual(entry.actor, self.user) + old, new = (Decimal(v) for v in entry.changes_dict['received']) + self.assertEqual(old, line.received) + self.assertEqual(new, line.received + 3) + self.assertEqual(entry.parent_object_pk, str(po.pk)) + + +class SalesOrderAuditTest(OrderAuditTestBase): + """Audit tests for SalesOrder, line items, extra lines and shipments.""" + + @classmethod + def setUpTestData(cls): + """Create a simple sales order to test against.""" + from company.models import Company + from part.models import Part + from stock.models import StockItem + + super().setUpTestData() + + cls.customer = Company.objects.create(name='Audit Customer', is_customer=True) + cls.part = Part.objects.create( + name='Audit Widget', description='A widget', salable=True + ) + cls.stock = StockItem.objects.create(part=cls.part, quantity=100) + + cls.order = models.SalesOrder.objects.create( + customer=cls.customer, reference='SO-9001' + ) + + def test_order_edit(self): + """Order edits are recorded - without the (derived) total price.""" + self.patch( + reverse('api-so-detail', kwargs={'pk': self.order.pk}), + {'description': 'Audited', 'customer_reference': 'CUST-9'}, + ) + + entry = self.entries( + models.SalesOrder, self.order.pk, action=ACTION.UPDATE + ).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['customer_reference'][1], 'CUST-9') + self.assertNotIn('total_price', entry.changes_dict) + + def test_line_items(self): + """Line item changes are recorded against the order.""" + update = self.check_child_crud( + self.order, + models.SalesOrderLineItem, + 'api-so-line-list', + 'api-so-line-detail', + {'order': self.order.pk, 'part': self.part.pk, 'quantity': 5}, + {'quantity': 7, 'sale_price': '12.50', 'sale_price_currency': 'USD'}, + ) + + self.assertEqual([Decimal(v) for v in update.changes_dict['quantity']], [5, 7]) + self.assertIn('sale_price', update.changes_dict) + self.assertEqual( + update.object_repr, f'7 x {self.part.full_name} - {self.order}' + ) + + def test_extra_lines(self): + """Extra line changes are recorded against the order.""" + update = self.check_child_crud( + self.order, + models.SalesOrderExtraLine, + 'api-so-extra-line-list', + 'api-so-extra-line-detail', + {'order': self.order.pk, 'quantity': 1, 'description': 'Packaging'}, + {'description': 'Gift wrapping'}, + ) + + self.assertEqual( + update.changes_dict['description'], ['Packaging', 'Gift wrapping'] + ) + self.assertEqual(update.object_repr, f'1 x Gift wrapping - {self.order}') + + def test_shipments(self): + """Shipment changes are recorded against the order, and the shipment itself.""" + update = self.check_child_crud( + self.order, + models.SalesOrderShipment, + 'api-so-shipment-list', + 'api-so-shipment-detail', + {'order': self.order.pk, 'reference': 'SHIP-1'}, + {'tracking_number': 'TRACK-123', 'invoice_number': 'INV-9'}, + ) + + self.assertEqual(update.changes_dict['tracking_number'], ['', 'TRACK-123']) + self.assertNotIn('barcode_hash', update.changes_dict) + + def test_shipment_history(self): + """The shipment History tab shows the shipment's own entries.""" + shipment = models.SalesOrderShipment.objects.create( + order=self.order, reference='SHIP-2' + ) + + self.patch( + reverse('api-so-shipment-detail', kwargs={'pk': shipment.pk}), + {'tracking_number': 'ABC'}, + ) + + history = self.history('order.salesordershipment', shipment.pk) + + self.assertEqual( + sorted(e['action'] for e in history), [ACTION.CREATE, ACTION.UPDATE] + ) + + for entry in history: + self.assertEqual(entry['model_type'], 'order.salesordershipment') + self.assertEqual(entry['parent_object_pk'], str(self.order.pk)) + + def test_complete_shipment(self): + """Shipped quantities (updated in bulk) are recorded against the user.""" + line = models.SalesOrderLineItem.objects.create( + order=self.order, part=self.part, quantity=10 + ) + shipment = models.SalesOrderShipment.objects.create( + order=self.order, reference='SHIP-3' + ) + + self.order.issue_order() + + models.SalesOrderAllocation.objects.create( + line=line, shipment=shipment, item=self.stock, quantity=4 + ) + + # Run in the request context of the test user (e.g. as via the API) + from auditlog.context import set_actor + + with set_actor(self.user): + shipment.complete_shipment(self.user) + + entry = self.entries( + models.SalesOrderLineItem, + line.pk, + action=ACTION.UPDATE, + changes__has_key='shipped', + ).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual([Decimal(v) for v in entry.changes_dict['shipped']], [0, 4]) + self.assertEqual(entry.parent_object_pk, str(self.order.pk)) + + # Shipment completion is recorded too + shipment_entry = self.entries( + models.SalesOrderShipment, + shipment.pk, + action=ACTION.UPDATE, + changes__has_key='shipment_date', + ).get() + + self.assertIsNone(shipment_entry.changes_dict['shipment_date'][0]) + + +class ReturnOrderAuditTest(OrderAuditTestBase): + """Audit tests for ReturnOrder, line items and extra lines.""" + + @classmethod + def setUpTestData(cls): + """Create a simple return order to test against.""" + from company.models import Company + from part.models import Part + from stock.models import StockItem + + super().setUpTestData() + + cls.customer = Company.objects.create(name='Returning Co', is_customer=True) + cls.part = Part.objects.create( + name='Returned Widget', description='A widget', salable=True + ) + cls.item = StockItem.objects.create( + part=cls.part, quantity=1, serial='RMA-1', customer=cls.customer + ) + + cls.order = models.ReturnOrder.objects.create( + customer=cls.customer, reference='RMA-9001' + ) + + def test_order_edit(self): + """Order edits are recorded.""" + self.patch( + reverse('api-return-order-detail', kwargs={'pk': self.order.pk}), + {'description': 'Faulty units', 'customer_reference': 'C-77'}, + ) + + entry = self.entries( + models.ReturnOrder, self.order.pk, action=ACTION.UPDATE + ).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['description'][1], 'Faulty units') + self.assertNotIn('total_price', entry.changes_dict) + + def test_line_items(self): + """Line item changes (including the outcome status) are recorded against the order.""" + from order.status_codes import ReturnOrderLineStatus + + update = self.check_child_crud( + self.order, + models.ReturnOrderLineItem, + 'api-return-order-line-list', + 'api-return-order-line-detail', + {'order': self.order.pk, 'item': self.item.pk}, + {'outcome': ReturnOrderLineStatus.REFUND.value, 'notes': 'Refund issued'}, + ) + + self.assertEqual( + update.changes_dict['outcome'][1], ReturnOrderLineStatus.REFUND.value + ) + + def test_extra_lines(self): + """Extra line changes are recorded against the order.""" + update = self.check_child_crud( + self.order, + models.ReturnOrderExtraLine, + 'api-return-order-extra-line-list', + 'api-return-order-extra-line-detail', + {'order': self.order.pk, 'quantity': 1, 'description': 'Restocking fee'}, + {'description': 'Inspection fee'}, + ) + + self.assertEqual( + update.changes_dict['description'], ['Restocking fee', 'Inspection fee'] + ) + + +class TransferOrderAuditTest(OrderAuditTestBase): + """Audit tests for TransferOrder and line items.""" + + @classmethod + def setUpTestData(cls): + """Create a simple transfer order to test against.""" + from part.models import Part + from stock.models import StockItem + + super().setUpTestData() + + cls.source = StockLocation.objects.create(name='Audit Source') + cls.target = StockLocation.objects.create(name='Audit Target') + + cls.part = Part.objects.create(name='Moved Widget', description='A widget') + cls.stock = StockItem.objects.create( + part=cls.part, quantity=50, location=cls.source + ) + + cls.order = models.TransferOrder.objects.create( + reference='TO-9001', take_from=cls.source, destination=cls.target + ) + + def test_order_edit(self): + """Order edits are recorded.""" + self.patch( + reverse('api-transfer-order-detail', kwargs={'pk': self.order.pk}), + {'description': 'Restock line side'}, + ) + + entry = self.entries( + models.TransferOrder, self.order.pk, action=ACTION.UPDATE + ).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['description'][1], 'Restock line side') + + def test_line_items(self): + """Line item changes are recorded against the order.""" + update = self.check_child_crud( + self.order, + models.TransferOrderLineItem, + 'api-transfer-order-line-list', + 'api-transfer-order-line-detail', + {'order': self.order.pk, 'part': self.part.pk, 'quantity': 10}, + {'quantity': 12}, + ) + + self.assertEqual( + [Decimal(v) for v in update.changes_dict['quantity']], [10, 12] + ) + + def test_transferred(self): + """The 'transferred' quantity (updated via an F() expression) is recorded as a number.""" + line = models.TransferOrderLineItem.objects.create( + order=self.order, part=self.part, quantity=10 + ) + + self.order.issue_order() + + allocation = models.TransferOrderAllocation.objects.create( + line=line, item=self.stock, quantity=6 + ) + + allocation.complete_allocation(self.user) + + line.refresh_from_db() + self.assertEqual(line.transferred, 6) + + entry = self.entries( + models.TransferOrderLineItem, + line.pk, + action=ACTION.UPDATE, + changes__has_key='transferred', + ).get() + + self.assertEqual( + [Decimal(v) for v in entry.changes_dict['transferred']], [0, 6] + ) + self.assertNotIn('F(', str(entry.changes)) + self.assertEqual(entry.parent_object_pk, str(self.order.pk)) diff --git a/src/backend/InvenTree/order/test_sales_order.py b/src/backend/InvenTree/order/test_sales_order.py index 8ebe43dce9..47ba514bc2 100644 --- a/src/backend/InvenTree/order/test_sales_order.py +++ b/src/backend/InvenTree/order/test_sales_order.py @@ -619,7 +619,7 @@ class SalesOrderTest(InvenTreeAPITestCase): expected_code=200, benchmark=True, max_query_time=5.0, - max_query_count=100, + max_query_count=120, ) self.assertEqual(response.status_code, 200) diff --git a/src/backend/InvenTree/part/models.py b/src/backend/InvenTree/part/models.py index 11eae3b2d2..3a2854f76b 100644 --- a/src/backend/InvenTree/part/models.py +++ b/src/backend/InvenTree/part/models.py @@ -41,6 +41,7 @@ import part.helpers as part_helpers import part.settings as part_settings import report.mixins import users.models +from audit.helpers import audited_bulk_create from build import models as BuildModels from build.status_codes import BuildStatusGroups from common.icons import validate_icon @@ -61,6 +62,7 @@ logger = structlog.get_logger('inventree') class PartCategory( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.PluginValidationMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.MetadataMixin, @@ -80,6 +82,11 @@ class PartCategory( EXTRA_PATH_FIELDS = ['icon'] IMPORT_ID_FIELDS = ['pathstring', 'name'] + AUDIT_EXCLUDE_FIELDS = ( + # Derived - recalculated (in bulk) when a category is renamed or moved + 'pathstring', + ) + class Meta: """Metaclass defines extra model properties.""" @@ -455,6 +462,7 @@ class PartReportContext(report.mixins.BaseReportContext, TypedDict): @cleanup.ignore class Part( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.PluginValidationMixin, InvenTree.models.InvenTreeParameterMixin, InvenTree.models.InvenTreeAttachmentMixin, @@ -516,6 +524,13 @@ class Part( objects = TreeManager() + AUDIT_EXCLUDE_FIELDS = ( + # Derived from the BOM (bom_validated / bom_checked_by / bom_checked_date are recorded) + 'bom_checksum', + # Derived from 'barcode_data' + 'barcode_hash', + ) + class Meta: """Metaclass defines extra model properties.""" @@ -2403,7 +2418,7 @@ class Part( ) ) - Parameter.objects.bulk_create(parameters, batch_size=250) + audited_bulk_create(Parameter, parameters) def getTestTemplates( self, required=None, include_parent: bool = True, enabled=None @@ -2887,7 +2902,11 @@ class PartTestTemplate(InvenTree.models.InvenTreeMetadataModel): return [x.strip() for x in self.choices.split(',') if x.strip()] -class BomItem(InvenTree.models.MetadataMixin, InvenTree.models.InvenTreeModel): +class BomItem( + InvenTree.models.InvenTreeAuditMixin, + InvenTree.models.MetadataMixin, + InvenTree.models.InvenTreeModel, +): """A BomItem links a part to its component items. A part can have a BOM (bill of materials) which defines @@ -2913,6 +2932,15 @@ class BomItem(InvenTree.models.MetadataMixin, InvenTree.models.InvenTreeModel): allow_variants: Stock for part variants can be substituted for this BomItem """ + # Changes to a BOM item are included in the history of the assembly + AUDIT_PARENT_FIELD = 'part' + + AUDIT_EXCLUDE_FIELDS = ( + # Recalculated automatically - validate_bom() re-saves every BOM line + 'checksum', + 'validated', + ) + class Meta: """Metaclass providing extra model definition.""" @@ -3567,7 +3595,9 @@ def update_pricing_after_delete(sender, instance, **kwargs): pass -class BomItemSubstitute(InvenTree.models.InvenTreeMetadataModel): +class BomItemSubstitute( + InvenTree.models.InvenTreeAuditMixin, InvenTree.models.InvenTreeMetadataModel +): """A BomItemSubstitute provides a specification for alternative parts, which can be used in a bill of materials. Attributes: @@ -3583,6 +3613,35 @@ class BomItemSubstitute(InvenTree.models.InvenTreeMetadataModel): # Prevent duplication of substitute parts unique_together = ('part', 'bom_item') + def __str__(self): + """Render a string representation of a BomItemSubstitute instance.""" + return _('{substitute} (substitute for {part})').format( + substitute=self.part.full_name, part=self.bom_item.sub_part.full_name + ) + + def get_audit_parent_reference(self): + """Audit entries for a substitute are included in the history of the *assembly*. + + i.e. the parent of the BomItem (skipping a level), so that all BOM changes + are shown in the history of the assembly part. + """ + from django.contrib.contenttypes.models import ContentType + + if 'bom_item' in self._state.fields_cache: + part_id = self.bom_item.part_id + else: + part_id = ( + BomItem.objects + .filter(pk=self.bom_item_id) + .values_list('part_id', flat=True) + .first() + ) + + if part_id is None: + return None + + return ContentType.objects.get_for_model(Part), str(part_id) + def save(self, *args, **kwargs): """Enforce a full_clean when saving the BomItemSubstitute model.""" self.full_clean() diff --git a/src/backend/InvenTree/part/test_audit.py b/src/backend/InvenTree/part/test_audit.py new file mode 100644 index 0000000000..5b3b302b10 --- /dev/null +++ b/src/backend/InvenTree/part/test_audit.py @@ -0,0 +1,350 @@ +"""Audit log tests for the Part and PartCategory models.""" + +from django.contrib.contenttypes.models import ContentType +from django.urls import reverse + +from audit.models import AuditLogEntry +from InvenTree.unit_test import InvenTreeAPITestCase +from part.models import Part, PartCategory + +ACTION = AuditLogEntry.Action + + +class PartAuditTest(InvenTreeAPITestCase): + """Audit tests for Part and PartCategory objects.""" + + roles = [ + 'part.view', + 'part.add', + 'part.change', + 'part.delete', + 'part_category.view', + 'part_category.add', + 'part_category.change', + 'part_category.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.top = PartCategory.objects.create(name='Electronics') + cls.category = PartCategory.objects.create(name='Resistors', parent=cls.top) + cls.part = Part.objects.create( + name='R_10K', description='10k resistor', category=cls.category + ) + + def entries(self, model, pk, **filters): + """Return audit entries for a given object (by PK - which survives deletion).""" + return AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(model), + object_pk=str(pk), + **filters, + ) + + def history(self, model_type: str, pk, **params): + """Return the audit history for an object (including child entries).""" + return self.get( + reverse('api-audit-log-list'), + {'model_type': model_type, 'model_id': pk, **params}, + ).data + + def test_part_create(self): + """Creating a part is recorded.""" + entry = self.entries(Part, self.part.pk, action=ACTION.CREATE).get() + + self.assertEqual(entry.changes_dict['name'], [None, 'R_10K']) + self.assertEqual(entry.object_repr, str(self.part)) + + # Derived / tree fields are not recorded + for field in ['bom_checksum', 'barcode_hash', 'lft', 'rght', 'tree_id']: + self.assertNotIn(field, entry.changes_dict) + + def test_part_edit(self): + """Part edits via the API are recorded against the user.""" + self.patch( + reverse('api-part-detail', kwargs={'pk': self.part.pk}), + {'description': '10k resistor, 1%', 'IPN': 'RES-10K', 'active': False}, + ) + + entry = self.entries(Part, self.part.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual( + entry.changes_dict['description'], ['10k resistor', '10k resistor, 1%'] + ) + self.assertEqual(entry.changes_dict['IPN'], [None, 'RES-10K']) + self.assertEqual(entry.changes_dict['active'], [True, False]) + + def test_bom_validation(self): + """Validating the BOM is recorded - without the (derived) checksum.""" + self.part.assembly = True + self.part.save() + + self.part.validate_bom(self.user) + + # Note: entries are ordered by timestamp (newest first) by default + entry = ( + self.entries(Part, self.part.pk, action=ACTION.UPDATE).order_by('pk').last() + ) + + self.assertEqual(entry.changes_dict['bom_validated'], [False, True]) + self.assertEqual(entry.changes_dict['bom_checked_by'], [None, self.user.pk]) + self.assertIn('bom_checked_date', entry.changes_dict) + self.assertNotIn('bom_checksum', entry.changes_dict) + + def test_part_history_includes_parameters(self): + """Parameters (already audited) are shown in the part history.""" + from common.models import Parameter, ParameterTemplate + + template = ParameterTemplate.objects.create(name='Resistance', units='ohm') + + Parameter.objects.create( + model_type=Part.get_content_type(), + model_id=self.part.pk, + template=template, + data='10000', + ) + + model_types = {e['model_type'] for e in self.history('part.part', self.part.pk)} + + self.assertIn('part.part', model_types) + self.assertIn('common.parameter', model_types) + + def test_category_edit(self): + """Category edits are recorded - without the (derived) pathstring.""" + child = PartCategory.objects.create(name='SMD', parent=self.category) + + self.patch( + reverse('api-part-category-detail', kwargs={'pk': self.category.pk}), + {'name': 'Resistor', 'description': 'All resistors'}, + ) + + entry = self.entries(PartCategory, self.category.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.changes_dict['name'], ['Resistors', 'Resistor']) + self.assertNotIn('pathstring', entry.changes_dict) + + # The pathstring of the child category is updated (in bulk) - but not recorded + child.refresh_from_db() + self.assertEqual(child.pathstring, 'Electronics/Resistor/SMD') + self.assertFalse( + self.entries(PartCategory, child.pk, action=ACTION.UPDATE).exists() + ) + + def test_category_delete(self): + """Deleting a category re-parents its parts and subcategories - and is recorded.""" + child = PartCategory.objects.create(name='SMD', parent=self.category) + category_pk = self.category.pk + + self.category.delete() + + self.assertEqual( + self.entries(PartCategory, category_pk, action=ACTION.DELETE).count(), 1 + ) + + # The part is moved to the parent category + part_entry = self.entries(Part, self.part.pk, action=ACTION.UPDATE).get() + self.assertEqual( + part_entry.changes_dict['category'], [category_pk, self.top.pk] + ) + + # The subcategory is moved to the parent category + child_entry = self.entries(PartCategory, child.pk, action=ACTION.UPDATE).get() + self.assertEqual(child_entry.changes_dict['parent'], [category_pk, self.top.pk]) + + def test_category_history(self): + """The category history is available via the API.""" + history = self.history('part.partcategory', self.category.pk) + + self.assertEqual(len(history), 1) + self.assertEqual(history[0]['action'], ACTION.CREATE) + self.assertEqual(history[0]['model_label'], 'Part Category') + + +class BomAuditTest(InvenTreeAPITestCase): + """Audit tests for BomItem and BomItemSubstitute objects.""" + + roles = [ + 'part.view', + 'part.change', + 'bom.view', + 'bom.add', + 'bom.change', + 'bom.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.assembly = Part.objects.create( + name='Widget Assembly', description='An assembly', assembly=True + ) + cls.resistor = Part.objects.create( + name='R_10K', description='10k resistor', component=True + ) + cls.alternative = Part.objects.create( + name='R_10K_ALT', description='10k resistor (alt)', component=True + ) + + def entries(self, model, pk, **filters): + """Return audit entries for a given object (by PK - which survives deletion).""" + return AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(model), + object_pk=str(pk), + **filters, + ) + + def assembly_history(self, assembly=None): + """Return the audit history for the assembly (including BOM changes).""" + assembly = assembly or self.assembly + + return self.get( + reverse('api-audit-log-list'), + {'model_type': 'part.part', 'model_id': assembly.pk}, + ).data + + def test_bom_item(self): + """BOM item changes via the API are recorded against the assembly.""" + from part.models import BomItem + + pk = self.post( + reverse('api-bom-list'), + {'part': self.assembly.pk, 'sub_part': self.resistor.pk, 'quantity': 4}, + ).data['pk'] + + url = reverse('api-bom-item-detail', kwargs={'pk': pk}) + self.patch(url, {'quantity': 6, 'reference': 'R1-R6'}) + self.delete(url) + + entries = self.entries(BomItem, pk) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(self.assembly.pk)) + self.assertNotIn('checksum', entry.changes_dict) + self.assertNotIn('validated', entry.changes_dict) + self.assertNotIn(' object (', entry.object_repr) + + update = entries.get(action=ACTION.UPDATE) + self.assertEqual(update.changes_dict['reference'], ['', 'R1-R6']) + + bom_actions = [ + e['action'] + for e in self.assembly_history() + if e['model_type'] == 'part.bomitem' + ] + + self.assertEqual( + sorted(bom_actions), [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE] + ) + + def test_validate_bom(self): + """Validating the BOM (which re-saves every line) records no BOM item entries.""" + from part.models import BomItem + + items = [ + BomItem.objects.create(part=self.assembly, sub_part=part, quantity=1) + for part in [self.resistor, self.alternative] + ] + + self.assembly.validate_bom(self.user) + + for item in items: + item.refresh_from_db() + self.assertTrue(item.validated) + self.assertFalse( + self.entries(BomItem, item.pk, action=ACTION.UPDATE).exists() + ) + + # ...but the validation itself is recorded against the assembly + entry = ( + self + .entries(Part, self.assembly.pk, action=ACTION.UPDATE) + .order_by('pk') + .last() + ) + self.assertEqual(entry.changes_dict['bom_validated'], [False, True]) + + def test_substitutes(self): + """Substitute changes are recorded against the assembly (not the BOM item).""" + from part.models import BomItem, BomItemSubstitute + + bom_item = BomItem.objects.create( + part=self.assembly, sub_part=self.resistor, quantity=1 + ) + + pk = self.post( + reverse('api-bom-substitute-list'), + {'bom_item': bom_item.pk, 'part': self.alternative.pk}, + ).data['pk'] + + self.delete(reverse('api-bom-substitute-detail', kwargs={'pk': pk})) + + entries = self.entries(BomItemSubstitute, pk) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.parent_content_type, Part.get_content_type()) + self.assertEqual(entry.parent_object_pk, str(self.assembly.pk)) + self.assertEqual( + entry.object_repr, + f'{self.alternative.full_name} (substitute for {self.resistor.full_name})', + ) + + self.assertIn( + 'part.bomitemsubstitute', {e['model_type'] for e in self.assembly_history()} + ) + + def test_substitute_cascade_delete(self): + """Substitutes deleted along with their BOM item retain the assembly reference.""" + from part.models import BomItem, BomItemSubstitute + + bom_item = BomItem.objects.create( + part=self.assembly, sub_part=self.resistor, quantity=1 + ) + substitute = BomItemSubstitute.objects.create( + bom_item=bom_item, part=self.alternative + ) + substitute_pk = substitute.pk + + BomItem.objects.get(pk=bom_item.pk).delete() + + entry = self.entries( + BomItemSubstitute, substitute_pk, action=ACTION.DELETE + ).get() + self.assertEqual(entry.parent_object_pk, str(self.assembly.pk)) + + def test_copy_bom(self): + """Copying a BOM is recorded against the new assembly.""" + from part.models import BomItem, BomItemSubstitute + + bom_item = BomItem.objects.create( + part=self.assembly, sub_part=self.resistor, quantity=2 + ) + BomItemSubstitute.objects.create(bom_item=bom_item, part=self.alternative) + + other = Part.objects.create( + name='Widget Assembly v2', description='An assembly', assembly=True + ) + other.copy_bom_from(self.assembly) + + model_types = [ + (e['model_type'], e['action']) for e in self.assembly_history(other) + ] + + self.assertIn(('part.bomitem', ACTION.CREATE), model_types) + self.assertIn(('part.bomitemsubstitute', ACTION.CREATE), model_types) diff --git a/src/backend/InvenTree/pricing/models.py b/src/backend/InvenTree/pricing/models.py index 611c3c4b11..72f41a6b0d 100644 --- a/src/backend/InvenTree/pricing/models.py +++ b/src/backend/InvenTree/pricing/models.py @@ -19,6 +19,7 @@ import common.currency import common.models import InvenTree.fields import InvenTree.helpers +import InvenTree.models import InvenTree.ready import InvenTree.tasks from common.currency import currency_code_default @@ -33,7 +34,7 @@ from part.models import Part logger = structlog.get_logger('inventree') -class PartPricing(common.models.MetaMixin): +class PartPricing(InvenTree.models.InvenTreeAuditMixin, common.models.MetaMixin): """Model for caching min/max pricing information for a particular Part. It is prohibitively expensive to calculate min/max pricing for a part "on the fly". @@ -59,6 +60,18 @@ class PartPricing(common.models.MetaMixin): - Detailed pricing information is very context specific in any case """ + # Pricing is recalculated automatically (in the background) - only the + # user-specified override values are recorded in the audit log + AUDIT_INCLUDE_FIELDS = ( + 'override_min', + 'override_min_currency', + 'override_max', + 'override_max_currency', + ) + + # Changes to the pricing overrides are included in the history of the part + AUDIT_PARENT_FIELD = 'part' + class Meta: """Metaclass providing extra model definition.""" @@ -68,6 +81,10 @@ class PartPricing(common.models.MetaMixin): # When calculating assembly pricing, we limit the depth of the calculation MAX_PRICING_DEPTH = 50 + def __str__(self): + """Return a string representation of this PartPricing instance.""" + return _('{part} pricing').format(part=self.part) + @property def is_valid(self): """Return True if the cached pricing is valid.""" @@ -834,9 +851,14 @@ class PartPricing(common.models.MetaMixin): ) -class PartSellPriceBreak(common.models.PriceBreak): +class PartSellPriceBreak( + InvenTree.models.InvenTreeAuditMixin, common.models.PriceBreak +): """Represents a price break for selling this part.""" + # Changes to a price break are included in the history of the part + AUDIT_PARENT_FIELD = 'part' + class Meta: """Metaclass providing extra model definition.""" @@ -859,9 +881,14 @@ class PartSellPriceBreak(common.models.PriceBreak): ) -class PartInternalPriceBreak(common.models.PriceBreak): +class PartInternalPriceBreak( + InvenTree.models.InvenTreeAuditMixin, common.models.PriceBreak +): """Represents a price break for internally selling this part.""" + # Changes to a price break are included in the history of the part + AUDIT_PARENT_FIELD = 'part' + class Meta: """Metaclass providing extra model definition.""" diff --git a/src/backend/InvenTree/pricing/test_audit.py b/src/backend/InvenTree/pricing/test_audit.py new file mode 100644 index 0000000000..c0a93f8a5e --- /dev/null +++ b/src/backend/InvenTree/pricing/test_audit.py @@ -0,0 +1,161 @@ +"""Audit log tests for the pricing models.""" + +from django.contrib.contenttypes.models import ContentType +from django.urls import reverse + +from audit.models import AuditLogEntry +from InvenTree.unit_test import InvenTreeAPITestCase +from part.models import Part +from pricing.models import PartInternalPriceBreak, PartPricing, PartSellPriceBreak + +ACTION = AuditLogEntry.Action + + +class PricingAuditTest(InvenTreeAPITestCase): + """Audit tests for PartPricing and the part price break models.""" + + roles = [ + 'part.view', + 'pricing.view', + 'pricing.add', + 'pricing.change', + 'pricing.delete', + ] + + @classmethod + def setUpTestData(cls): + """Create test data.""" + super().setUpTestData() + + cls.part = Part.objects.create( + name='Widget', description='A widget', salable=True + ) + + def entries(self, model, pk=None, **filters): + """Return audit entries for a given model (and object).""" + queryset = AuditLogEntry.objects.filter( + content_type=ContentType.objects.get_for_model(model), **filters + ) + + if pk is not None: + queryset = queryset.filter(object_pk=str(pk)) + + return queryset + + def part_history(self): + """Return the audit history for the part (including pricing entries).""" + return self.get( + reverse('api-audit-log-list'), + {'model_type': 'part.part', 'model_id': self.part.pk}, + ).data + + def check_price_breaks(self, model, list_url, detail_url): + """Create / update / delete a price break via the API, and check the audit trail.""" + pk = self.post( + reverse(list_url), + { + 'part': self.part.pk, + 'quantity': 10, + 'price': '1.50', + 'price_currency': 'USD', + }, + ).data['pk'] + + url = reverse(detail_url, kwargs={'pk': pk}) + self.patch(url, {'price': '1.25'}) + + self.assertEqual( + model.objects.get(pk=pk).__str__(), f'{self.part} - $1.25 @ 10' + ) + + self.delete(url) + + entries = self.entries(model, pk) + + self.assertEqual( + sorted(entries.values_list('action', flat=True)), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + for entry in entries: + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(self.part.pk)) + self.assertNotIn(' object (', entry.object_repr) + + update = entries.get(action=ACTION.UPDATE) + self.assertEqual(update.changes_dict['price'], ['$1.50', '$1.25']) + + label = f'{model._meta.app_label}.{model._meta.model_name}' + + self.assertEqual( + sorted( + e['action'] for e in self.part_history() if e['model_type'] == label + ), + [ACTION.CREATE, ACTION.UPDATE, ACTION.DELETE], + ) + + def test_sale_price_breaks(self): + """Sale price break changes are recorded against the part.""" + self.check_price_breaks( + PartSellPriceBreak, 'api-part-sale-price-list', 'api-part-sale-price-detail' + ) + + def test_internal_price_breaks(self): + """Internal price break changes are recorded against the part.""" + self.check_price_breaks( + PartInternalPriceBreak, + 'api-part-internal-price-list', + 'api-part-internal-price-detail', + ) + + def test_pricing_overrides(self): + """Only the user-specified pricing overrides are recorded.""" + url = reverse('api-part-pricing', kwargs={'pk': self.part.pk}) + + # Creating the pricing record (with no overrides) is not recorded + # Note: Part.pricing returns an unsaved instance, which is saved on first update + pricing = self.part.pricing + pricing.save() + self.assertFalse(self.entries(PartPricing).exists()) + + self.patch(url, {'override_max': 25, 'override_max_currency': 'USD'}) + + entry = self.entries(PartPricing, pricing.pk, action=ACTION.UPDATE).get() + + self.assertEqual(entry.actor, self.user) + self.assertEqual(entry.parent_object_pk, str(self.part.pk)) + self.assertEqual(entry.changes_dict['override_max'], [None, '$25.00']) + self.assertEqual(entry.object_repr, f'{self.part} pricing') + + # Only override fields are ever recorded + for field in entry.changes_dict: + self.assertTrue(field.startswith('override_'), field) + + def test_recalculation_not_recorded(self): + """Automatic pricing recalculation (cached values) is not recorded.""" + PartSellPriceBreak.objects.create(part=self.part, quantity=1, price=5) + + pricing = self.part.pricing + pricing.update_pricing() + + pricing.refresh_from_db() + self.assertIsNotNone(pricing.sale_price_min) + + self.assertFalse(self.entries(PartPricing).exists()) + + def test_pricing_permission(self): + """Pricing entries require the 'pricing' role, even via the part history.""" + PartSellPriceBreak.objects.create(part=self.part, quantity=1, price=5) + + def pricing_entries(): + return [ + e for e in self.part_history() if e['model_type'].startswith('pricing.') + ] + + self.assertEqual(len(pricing_entries()), 1) + + # Remove the pricing role (keep 'part.view') + self.clearRoles() + self.assignRole('part.view') + + self.assertEqual(pricing_entries(), []) diff --git a/src/backend/InvenTree/users/ruleset.py b/src/backend/InvenTree/users/ruleset.py index b714a07baa..f1a85b4e13 100644 --- a/src/backend/InvenTree/users/ruleset.py +++ b/src/backend/InvenTree/users/ruleset.py @@ -200,6 +200,8 @@ def get_ruleset_ignore() -> list[str]: return [ # Core django models (not user configurable) 'admin_logentry', + # Audit log - access is determined by the permissions of the audited model + 'audit_auditlogentry', 'contenttypes_contenttype', # Models which currently do not require permissions 'common_attachment', diff --git a/src/backend/requirements-3.14.txt b/src/backend/requirements-3.14.txt index e93199ddc6..1a6b6f1229 100644 --- a/src/backend/requirements-3.14.txt +++ b/src/backend/requirements-3.14.txt @@ -603,6 +603,7 @@ django==5.2.17 \ # -r src/backend/requirements.in # django-allauth # django-anymail + # django-auditlog # django-cors-headers # django-dbbackup # django-error-report-2 @@ -645,6 +646,12 @@ django-anymail[amazon-ses, postal]==15.1 \ # via # -c src/backend/requirements.txt # -r src/backend/requirements.in +django-auditlog==3.4.1 \ + --hash=sha256:29958ecacfee00144127214f3ccef3f0c203c3659bcb6dd404a0f3d5551a10a5 \ + --hash=sha256:ad07b9db452d5fa8303822cccd78cd3fcb2c2863aeb6abe039ec45739b4d7e33 + # via + # -c src/backend/requirements.txt + # -r src/backend/requirements.in django-cleanup==9.0.0 \ --hash=sha256:19f8b0e830233f9f0f683b17181f414672a0f48afe3ea3cc80ba47ae40ad880c \ --hash=sha256:bb9fb560aaf62959c81e31fa40885c36bbd5854d5aa21b90df2c7e4ba633531e @@ -2134,6 +2141,7 @@ python-dateutil==2.9.0.post0 \ # via # -c src/backend/requirements.txt # botocore + # django-auditlog # django-recurrence # icalendar python-dotenv==1.2.4 \ diff --git a/src/backend/requirements.in b/src/backend/requirements.in index 002833d032..9e596f041e 100644 --- a/src/backend/requirements.in +++ b/src/backend/requirements.in @@ -4,6 +4,7 @@ blessed # CLI for Q Monitor cryptography # Core cryptographic functionality django-anymail[amazon_ses,postal] # Email backend for various providers django-allauth[mfa,socialaccount,saml,openid,headless] # SSO for external providers via OpenID +django-auditlog>=3.4.1,<4 # Model change history / audit log django-cleanup # Automated deletion of old / unused uploaded files django-cors-headers # CORS headers extension for DRF django-dbbackup # Backup / restore of database and media files diff --git a/src/backend/requirements.txt b/src/backend/requirements.txt index ac1486da53..361fc69d58 100644 --- a/src/backend/requirements.txt +++ b/src/backend/requirements.txt @@ -577,6 +577,7 @@ django==5.2.17 \ # -r src/backend/requirements.in # django-allauth # django-anymail + # django-auditlog # django-cors-headers # django-dbbackup # django-error-report-2 @@ -615,6 +616,10 @@ django-anymail[amazon-ses, postal]==15.1 \ --hash=sha256:299fb12a21ac4101e9706c2ade461d18deacbb6bc3750cfa6ca08aeea1fc9442 \ --hash=sha256:cb482b0e2b0aea665c83d49e3f13bb0673210aa0de285b1748a3b5e6277fc826 # via -r src/backend/requirements.in +django-auditlog==3.4.1 \ + --hash=sha256:29958ecacfee00144127214f3ccef3f0c203c3659bcb6dd404a0f3d5551a10a5 \ + --hash=sha256:ad07b9db452d5fa8303822cccd78cd3fcb2c2863aeb6abe039ec45739b4d7e33 + # via -r src/backend/requirements.in django-cleanup==9.0.0 \ --hash=sha256:19f8b0e830233f9f0f683b17181f414672a0f48afe3ea3cc80ba47ae40ad880c \ --hash=sha256:bb9fb560aaf62959c81e31fa40885c36bbd5854d5aa21b90df2c7e4ba633531e @@ -1907,6 +1912,7 @@ python-dateutil==2.9.0.post0 \ --hash=sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427 # via # botocore + # django-auditlog # django-recurrence # icalendar python-dotenv==1.2.4 \ diff --git a/src/frontend/lib/enums/ApiEndpoints.tsx b/src/frontend/lib/enums/ApiEndpoints.tsx index 29b6fea2b2..4e6ab23b7b 100644 --- a/src/frontend/lib/enums/ApiEndpoints.tsx +++ b/src/frontend/lib/enums/ApiEndpoints.tsx @@ -253,6 +253,7 @@ export enum ApiEndpoints { // Miscellaneous API endpoints attachment_list = 'attachment/', + audit_log_list = 'audit/', instance_info = 'instance-info/', error_report_list = 'error-report/', project_code_list = 'project-code/', diff --git a/src/frontend/lib/enums/ModelInformation.tsx b/src/frontend/lib/enums/ModelInformation.tsx index cdb6e6692b..93f670402c 100644 --- a/src/frontend/lib/enums/ModelInformation.tsx +++ b/src/frontend/lib/enums/ModelInformation.tsx @@ -83,7 +83,6 @@ export const ModelInformationDict: ModelDict = { parttesttemplate: { label: () => t`Part Test Template`, label_multiple: () => t`Part Test Templates`, - url_detail: '/parttesttemplate/:pk/', api_endpoint: ApiEndpoints.part_test_template_list, icon: 'test' }, @@ -170,7 +169,6 @@ export const ModelInformationDict: ModelDict = { label: () => t`Build Line`, label_multiple: () => t`Build Lines`, url_overview: '/build/line', - url_detail: '/build/line/:pk/', api_endpoint: ApiEndpoints.build_line_list, icon: 'build_order' }, @@ -191,7 +189,6 @@ export const ModelInformationDict: ModelDict = { projectcode: { label: () => t`Project Code`, label_multiple: () => t`Project Codes`, - url_detail: '/project-code/:pk/', api_endpoint: ApiEndpoints.project_code_list, icon: 'list_details' }, @@ -270,21 +267,18 @@ export const ModelInformationDict: ModelDict = { address: { label: () => t`Address`, label_multiple: () => t`Addresses`, - url_detail: '/address/:pk/', api_endpoint: ApiEndpoints.address_list, icon: 'address' }, contact: { label: () => t`Contact`, label_multiple: () => t`Contacts`, - url_detail: '/contact/:pk/', api_endpoint: ApiEndpoints.contact_list, icon: 'group' }, owner: { label: () => t`Owner`, label_multiple: () => t`Owners`, - url_detail: '/owner/:pk/', api_endpoint: ApiEndpoints.owner_list, icon: 'group' }, @@ -307,7 +301,6 @@ export const ModelInformationDict: ModelDict = { label: () => t`Import Session`, label_multiple: () => t`Import Sessions`, url_overview: '/settings/admin/import', - url_detail: '/import/:pk/', api_endpoint: ApiEndpoints.import_session_list, icon: 'import' }, diff --git a/src/frontend/src/components/panels/HistoryPanel.tsx b/src/frontend/src/components/panels/HistoryPanel.tsx new file mode 100644 index 0000000000..b01cb643eb --- /dev/null +++ b/src/frontend/src/components/panels/HistoryPanel.tsx @@ -0,0 +1,60 @@ +import { t } from '@lingui/core/macro'; +import { Skeleton } from '@mantine/core'; +import { IconHistory } from '@tabler/icons-react'; +import { useMemo } from 'react'; + +import type { ModelType } from '@lib/enums/ModelType'; +import type { PanelType } from '@lib/types/Panel'; +import { + useGlobalSettingsState, + useUserSettingsState +} from '../../states/SettingsStates'; +import { useUserState } from '../../states/UserState'; +import { AuditLogTable } from '../../tables/general/AuditLogTable'; + +/** + * Determine whether the history panel should be displayed on detail pages: + * - Only available to staff users + * - Must be enabled via the 'DISPLAY_HISTORY_TAB' user setting + */ +export function useShowHistoryPanel(): boolean { + const user = useUserState(); + const userSettings = useUserSettingsState(); + const globalSettings = useGlobalSettingsState(); + + return useMemo( + () => + user.isStaff() && + globalSettings.isSet('AUDIT_LOG_ENABLED') && + userSettings.isSet('DISPLAY_HISTORY_TAB', false), + [user, userSettings, globalSettings] + ); +} + +/** + * Panel displaying the audit log (change history) for a model instance. + * Only add this panel to detail pages for models which are audited, + * and pass the result of useShowHistoryPanel() as 'visible'. + */ +export default function HistoryPanel({ + model_type, + model_id, + visible +}: { + model_type: ModelType; + model_id: number | undefined; + visible: boolean; +}): PanelType { + return { + name: 'history', + label: t`History`, + icon: , + hidden: !visible, + content: + visible && model_type && model_id ? ( + + ) : ( + + ) + }; +} diff --git a/src/frontend/src/hooks/UseFilter.tsx b/src/frontend/src/hooks/UseFilter.tsx index 88781f9d8d..c36c6346dd 100644 --- a/src/frontend/src/hooks/UseFilter.tsx +++ b/src/frontend/src/hooks/UseFilter.tsx @@ -15,13 +15,15 @@ type UseFilterProps = { params?: any; accessor?: string; transform: (item: any) => TableFilterChoice; + // Set to false to skip fetching the filter choices (default = true) + enabled?: boolean; }; export function useFilters(props: UseFilterProps) { const api = useApi(); const query = useQuery({ - enabled: true, + enabled: props.enabled ?? true, gcTime: 500, queryKey: [props.url, props.method, props.params], queryFn: async () => { diff --git a/src/frontend/src/pages/Index/Settings/AdminCenter/AuditLogManagementPanel.tsx b/src/frontend/src/pages/Index/Settings/AdminCenter/AuditLogManagementPanel.tsx new file mode 100644 index 0000000000..5dfff198e5 --- /dev/null +++ b/src/frontend/src/pages/Index/Settings/AdminCenter/AuditLogManagementPanel.tsx @@ -0,0 +1,51 @@ +import { t } from '@lingui/core/macro'; +import { Accordion, Alert, Stack } from '@mantine/core'; +import { IconInfoCircle } from '@tabler/icons-react'; + +import { StylishText } from '@lib/components/StylishText'; +import { GlobalSettingList } from '../../../../components/settings/SettingList'; +import { useGlobalSettingsState } from '../../../../states/SettingsStates'; +import { AuditLogTable } from '../../../../tables/general/AuditLogTable'; + +export default function AuditLogManagementPanel() { + const globalSettings = useGlobalSettingsState(); + + return ( + + {!globalSettings.isSet('AUDIT_LOG_ENABLED', true) && ( + } + title={t`Audit Log Disabled`} + > + {t`Changes are not currently being recorded in the audit log`} + + )} + + + + {t`Audit Log`} + + + + + + + + {t`Audit Log Settings`} + + + + + + + + ); +} diff --git a/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx b/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx index 232150d2bd..728c7f827c 100644 --- a/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx +++ b/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx @@ -11,6 +11,7 @@ import { IconExclamationCircle, IconFileCode, IconFileDownload, + IconFileSearch, IconFileUpload, IconHome, IconList, @@ -82,6 +83,10 @@ const IdentityManagementPanel = Loadable( lazy(() => import('./IdentityManagementPanel')) ); +const AuditLogManagementPanel = Loadable( + lazy(() => import('./AuditLogManagementPanel')) +); + const ErrorReportTable = Loadable( lazy(() => import('../../../../tables/settings/ErrorTable')) ); @@ -178,6 +183,12 @@ export default function AdminCenter() { icon: , content: }, + { + name: 'audit-log', + label: t`Audit Log`, + icon: , + content: + }, { name: 'companies', label: t`Companies`, @@ -312,6 +323,7 @@ export default function AdminCenter() { 'barcode-history', 'background', 'errors', + 'audit-log', 'companies', 'currencies', 'email' diff --git a/src/frontend/src/pages/Index/Settings/UserSettings.tsx b/src/frontend/src/pages/Index/Settings/UserSettings.tsx index 376f6e9985..d83f7e82cb 100644 --- a/src/frontend/src/pages/Index/Settings/UserSettings.tsx +++ b/src/frontend/src/pages/Index/Settings/UserSettings.tsx @@ -62,6 +62,7 @@ export default function UserSettings() { 'DATE_DISPLAY_FORMAT', 'FORMS_CLOSE_USING_ESCAPE', 'DISPLAY_STOCKTAKE_TAB', + 'DISPLAY_HISTORY_TAB', 'ENABLE_LAST_BREADCRUMB', 'NAVIGATION_TREE_PINNED', 'SHOW_EXTRA_MODEL_INFO', diff --git a/src/frontend/src/pages/build/BuildDetail.tsx b/src/frontend/src/pages/build/BuildDetail.tsx index fe48207322..493b36f1da 100644 --- a/src/frontend/src/pages/build/BuildDetail.tsx +++ b/src/frontend/src/pages/build/BuildDetail.tsx @@ -38,6 +38,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -237,6 +240,8 @@ export default function BuildDetail() { modelId: build?.pk }); + const showHistory = useShowHistoryPanel(); + const buildPanels: PanelType[] = useMemo(() => { return [ { @@ -375,6 +380,11 @@ export default function BuildDetail() { model_type: ModelType.build, model_id: build.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.build, + model_id: build.pk, + visible: showHistory }) ]; }, [ @@ -388,7 +398,8 @@ export default function BuildDetail() { showChildBuilds, buildLineQuery.isFetching, buildLineQuery.isLoading, - buildLineData + buildLineData, + showHistory ]); const editBuildOrderFields = useBuildOrderFields({ diff --git a/src/frontend/src/pages/company/CompanyDetail.tsx b/src/frontend/src/pages/company/CompanyDetail.tsx index 2c7b855f4b..0418273cbd 100644 --- a/src/frontend/src/pages/company/CompanyDetail.tsx +++ b/src/frontend/src/pages/company/CompanyDetail.tsx @@ -31,6 +31,9 @@ import type { Breadcrumb } from '../../components/nav/BreadcrumbList'; import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -96,6 +99,8 @@ export default function CompanyDetail(props: Readonly) { /> ); + const showHistory = useShowHistoryPanel(); + const companyPanels: PanelType[] = useMemo(() => { return [ { @@ -211,9 +216,14 @@ export default function CompanyDetail(props: Readonly) { model_type: ModelType.company, model_id: company.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.company, + model_id: company.pk, + visible: showHistory }) ]; - }, [id, company, user, instanceInfo]); + }, [id, company, user, instanceInfo, showHistory]); const editCompany = useEditApiFormModal({ url: ApiEndpoints.company_list, diff --git a/src/frontend/src/pages/company/ManufacturerPartDetail.tsx b/src/frontend/src/pages/company/ManufacturerPartDetail.tsx index 26e1d6a97b..69216ec7ce 100644 --- a/src/frontend/src/pages/company/ManufacturerPartDetail.tsx +++ b/src/frontend/src/pages/company/ManufacturerPartDetail.tsx @@ -23,6 +23,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -64,6 +67,8 @@ export default function ManufacturerPartDetail() { modelId: manufacturerPart?.pk }); + const showHistory = useShowHistoryPanel(); + const panels: PanelType[] = useMemo(() => { return [ { @@ -120,9 +125,14 @@ export default function ManufacturerPartDetail() { model_type: ModelType.manufacturerpart, model_id: manufacturerPart?.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.manufacturerpart, + model_id: manufacturerPart?.pk, + visible: showHistory }) ]; - }, [user, manufacturerPart, instanceInfo]); + }, [user, manufacturerPart, instanceInfo, showHistory]); const editManufacturerPartFields = useManufacturerPartFields(); diff --git a/src/frontend/src/pages/company/SupplierPartDetail.tsx b/src/frontend/src/pages/company/SupplierPartDetail.tsx index 376a0f08d9..36e43d5c48 100644 --- a/src/frontend/src/pages/company/SupplierPartDetail.tsx +++ b/src/frontend/src/pages/company/SupplierPartDetail.tsx @@ -27,6 +27,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -72,6 +75,8 @@ export default function SupplierPartDetail() { modelId: supplierPart?.pk }); + const showHistory = useShowHistoryPanel(); + const panels: PanelType[] = useMemo(() => { return [ { @@ -137,9 +142,14 @@ export default function SupplierPartDetail() { model_type: ModelType.supplierpart, model_id: supplierPart?.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.supplierpart, + model_id: supplierPart?.pk, + visible: showHistory }) ]; - }, [supplierPart, instanceInfo]); + }, [supplierPart, instanceInfo, showHistory]); const supplierPartActions = useMemo(() => { return [ diff --git a/src/frontend/src/pages/part/CategoryDetail.tsx b/src/frontend/src/pages/part/CategoryDetail.tsx index 8d13aa49d7..3e5bce06a5 100644 --- a/src/frontend/src/pages/part/CategoryDetail.tsx +++ b/src/frontend/src/pages/part/CategoryDetail.tsx @@ -30,6 +30,9 @@ import { ApiIcon } from '../../components/items/ApiIcon'; import InstanceDetail from '../../components/nav/InstanceDetail'; import NavigationTree from '../../components/nav/NavigationTree'; import { PageDetail } from '../../components/nav/PageDetail'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; import SegmentedControlPanel from '../../components/panels/SegmentedControlPanel'; @@ -198,6 +201,8 @@ export default function CategoryDetail() { defaultValue: 'table' }); + const showHistory = useShowHistoryPanel(); + const panels: PanelType[] = useMemo( () => [ { @@ -269,9 +274,15 @@ export default function CategoryDetail() { icon: , hidden: !id || !category.pk, content: - } + }, + HistoryPanel({ + model_type: ModelType.partcategory, + model_id: category?.pk, + // Not shown for the top-level (root) category view + visible: showHistory && !!id && !!category?.pk + }) ], - [category, id, partsView, instanceInfo] + [category, id, partsView, instanceInfo, showHistory] ); const breadcrumbs = useMemo( diff --git a/src/frontend/src/pages/part/PartDetail.tsx b/src/frontend/src/pages/part/PartDetail.tsx index 22636c7068..6e4253df0f 100644 --- a/src/frontend/src/pages/part/PartDetail.tsx +++ b/src/frontend/src/pages/part/PartDetail.tsx @@ -64,6 +64,9 @@ import InstanceDetail from '../../components/nav/InstanceDetail'; import NavigationTree from '../../components/nav/NavigationTree'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import { RenderPart } from '../../components/render/Part'; @@ -284,6 +287,8 @@ export default function PartDetail() { }, [enableRevisionSelection, part, partRevisionOptions]); // Part data panels (recalculate when part data changes) + const showHistory = useShowHistoryPanel(); + const partPanels: PanelType[] = useMemo(() => { return [ { @@ -518,6 +523,11 @@ export default function PartDetail() { model_type: ModelType.part, model_id: part?.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.part, + model_id: part?.pk, + visible: showHistory }) ]; }, [ @@ -529,7 +539,8 @@ export default function PartDetail() { bomInformation, revisionSelector, refreshInstance, - instanceInfo + instanceInfo, + showHistory ]); const breadcrumbs = useMemo(() => { diff --git a/src/frontend/src/pages/purchasing/PurchaseOrderDetail.tsx b/src/frontend/src/pages/purchasing/PurchaseOrderDetail.tsx index 05ad1fa814..ec1c2993cc 100644 --- a/src/frontend/src/pages/purchasing/PurchaseOrderDetail.tsx +++ b/src/frontend/src/pages/purchasing/PurchaseOrderDetail.tsx @@ -24,6 +24,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -133,6 +136,8 @@ export default function PurchaseOrderDetail() { modelType: ModelType.purchaseorder }); + const showHistory = useShowHistoryPanel(); + const orderPanels: PanelType[] = useMemo(() => { return [ { @@ -223,9 +228,14 @@ export default function PurchaseOrderDetail() { !globalSettings.isSet('PURCHASEORDER_EDIT_COMPLETED_ORDERS') ? false : undefined + }), + HistoryPanel({ + model_type: ModelType.purchaseorder, + model_id: order.pk, + visible: showHistory }) ]; - }, [order, id, user, instanceInfo]); + }, [order, id, user, instanceInfo, showHistory]); const issueOrder = useCreateApiFormModal({ url: apiUrl(ApiEndpoints.purchase_order_issue, order.pk), diff --git a/src/frontend/src/pages/sales/ReturnOrderDetail.tsx b/src/frontend/src/pages/sales/ReturnOrderDetail.tsx index 4a7ad11d99..6d6bd192cf 100644 --- a/src/frontend/src/pages/sales/ReturnOrderDetail.tsx +++ b/src/frontend/src/pages/sales/ReturnOrderDetail.tsx @@ -24,6 +24,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -97,6 +100,8 @@ export default function ReturnOrderDetail() { ); }, [order, globalSettings]); + const showHistory = useShowHistoryPanel(); + const orderPanels: PanelType[] = useMemo(() => { return [ { @@ -168,9 +173,14 @@ export default function ReturnOrderDetail() { model_type: ModelType.returnorder, model_id: order.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.returnorder, + model_id: order.pk, + visible: showHistory }) ]; - }, [order, id, user, instanceInfo]); + }, [order, id, user, instanceInfo, showHistory]); const orderBadges: ReactNode[] = useMemo(() => { return instanceQuery.isLoading diff --git a/src/frontend/src/pages/sales/SalesOrderDetail.tsx b/src/frontend/src/pages/sales/SalesOrderDetail.tsx index 6b472b75be..512b11ad5c 100644 --- a/src/frontend/src/pages/sales/SalesOrderDetail.tsx +++ b/src/frontend/src/pages/sales/SalesOrderDetail.tsx @@ -30,6 +30,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -135,6 +138,8 @@ export default function SalesOrderDetail() { modelType: ModelType.salesorder }); + const showHistory = useShowHistoryPanel(); + const orderPanels: PanelType[] = useMemo(() => { return [ { @@ -247,9 +252,14 @@ export default function SalesOrderDetail() { !globalSettings.isSet('SALESORDER_EDIT_COMPLETED_ORDERS') ? false : undefined + }), + HistoryPanel({ + model_type: ModelType.salesorder, + model_id: order.pk, + visible: showHistory }) ]; - }, [order, id, user, soStatus, instanceInfo]); + }, [order, id, user, soStatus, instanceInfo, showHistory]); const issueOrder = useCreateApiFormModal({ url: apiUrl(ApiEndpoints.sales_order_issue, order.pk), diff --git a/src/frontend/src/pages/sales/SalesOrderShipmentDetail.tsx b/src/frontend/src/pages/sales/SalesOrderShipmentDetail.tsx index b9a1d2a308..be1f1d7400 100644 --- a/src/frontend/src/pages/sales/SalesOrderShipmentDetail.tsx +++ b/src/frontend/src/pages/sales/SalesOrderShipmentDetail.tsx @@ -27,6 +27,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -74,6 +77,8 @@ export default function SalesOrderShipmentDetail() { const isPending = useMemo(() => !shipment.shipment_date, [shipment]); const isChecked = useMemo(() => !!shipment.checked_by, [shipment]); + const showHistory = useShowHistoryPanel(); + const shipmentPanels: PanelType[] = useMemo(() => { return [ { @@ -116,9 +121,14 @@ export default function SalesOrderShipmentDetail() { model_type: ModelType.salesordershipment, model_id: shipment.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.salesordershipment, + model_id: shipment.pk, + visible: showHistory }) ]; - }, [isPending, shipment, instanceInfo]); + }, [isPending, shipment, instanceInfo, showHistory]); const editShipmentFields = useSalesOrderShipmentFields({ pending: isPending, diff --git a/src/frontend/src/pages/stock/TransferOrderDetail.tsx b/src/frontend/src/pages/stock/TransferOrderDetail.tsx index b1ccea9a0b..2b709187b7 100644 --- a/src/frontend/src/pages/stock/TransferOrderDetail.tsx +++ b/src/frontend/src/pages/stock/TransferOrderDetail.tsx @@ -27,6 +27,9 @@ import { import InstanceDetail from '../../components/nav/InstanceDetail'; import { PageDetail } from '../../components/nav/PageDetail'; import AttachmentPanel from '../../components/panels/AttachmentPanel'; +import HistoryPanel, { + useShowHistoryPanel +} from '../../components/panels/HistoryPanel'; import NotesPanel from '../../components/panels/NotesPanel'; import { PanelGroup } from '../../components/panels/PanelGroup'; import ParametersPanel from '../../components/panels/ParametersPanel'; @@ -99,6 +102,8 @@ export default function TransferOrderDetail() { ); + const showHistory = useShowHistoryPanel(); + const orderPanels: PanelType[] = useMemo(() => { return [ { @@ -192,9 +197,14 @@ export default function TransferOrderDetail() { model_type: ModelType.transferorder, model_id: order.pk, note_count: instanceInfo.note_count + }), + HistoryPanel({ + model_type: ModelType.transferorder, + model_id: order.pk, + visible: showHistory }) ]; - }, [order, id, user, instanceInfo]); + }, [order, id, user, instanceInfo, showHistory]); const orderBadges: ReactNode[] = useMemo(() => { return instanceQuery.isLoading diff --git a/src/frontend/src/tables/general/AuditLogTable.tsx b/src/frontend/src/tables/general/AuditLogTable.tsx new file mode 100644 index 0000000000..0088b259ab --- /dev/null +++ b/src/frontend/src/tables/general/AuditLogTable.tsx @@ -0,0 +1,569 @@ +import { t } from '@lingui/core/macro'; +import { Badge, Group, Paper, Stack, Table, Text } from '@mantine/core'; +import { IconArrowRight } from '@tabler/icons-react'; +import type { DataTableRowExpansionProps } from 'mantine-datatable'; +import { type ReactNode, useMemo } from 'react'; +import { type NavigateFunction, useNavigate } from 'react-router-dom'; + +import { ApiEndpoints } from '@lib/enums/ApiEndpoints'; +import { ModelInformationDict } from '@lib/enums/ModelInformation'; +import type { ModelType } from '@lib/enums/ModelType'; +import { apiUrl } from '@lib/functions/Api'; +import { formatDecimal } from '@lib/functions/Formatting'; +import { getDetailUrl } from '@lib/functions/Navigation'; +import useTable from '@lib/hooks/UseTable'; +import type { TableFilter } from '@lib/types/Filters'; +import type { TableColumn } from '@lib/types/Tables'; +import { RenderInlineModel } from '../../components/render/Instance'; +import { RenderUser } from '../../components/render/User'; +import { DateColumn } from '../../components/tables/ColumnRenderers'; +import { + MaxDateFilter, + MinDateFilter, + UserFilter +} from '../../components/tables/Filter'; +import { InvenTreeTable } from '../../components/tables/InvenTreeTable'; +import { useFilters } from '../../hooks/UseFilter'; +import { useUserState } from '../../states/UserState'; + +// Action codes, as defined by django-auditlog +enum AuditAction { + CREATE = 0, + UPDATE = 1, + DELETE = 2, + ACCESS = 3 +} + +// A single field change, as returned by the 'changes_detail' API field +type AuditChange = { + field: string; + label: string; + type: 'value' | 'fk' | 'm2m'; + old?: any; + new?: any; + old_display?: string | null; + new_display?: string | null; + model_type?: string | null; + operation?: string | null; + objects?: string[] | null; + numeric?: boolean; +}; + +// Maximum number of changes rendered inline (the remainder are shown when the row is expanded) +const MAX_INLINE_CHANGES = 3; + +// Maximum length of a single rendered value +const MAX_VALUE_LENGTH = 100; + +/** + * Convert a server-side model label ('app_label.model') to a frontend ModelType (if known) + */ +function resolveModelType(label?: string | null): ModelType | undefined { + const name = label?.split('.').pop(); + + if (name && name in ModelInformationDict) { + return name as ModelType; + } + + return undefined; +} + +function actionLabel(action: AuditAction): string { + switch (action) { + case AuditAction.CREATE: + return t`Created`; + case AuditAction.UPDATE: + return t`Updated`; + case AuditAction.DELETE: + return t`Deleted`; + default: + return t`Accessed`; + } +} + +function actionColor(action: AuditAction): string { + switch (action) { + case AuditAction.CREATE: + return 'green'; + case AuditAction.UPDATE: + return 'blue'; + case AuditAction.DELETE: + return 'red'; + default: + return 'gray'; + } +} + +/** + * Render a single recorded value (old or new) for a field change + */ +function AuditValue({ + value, + display, + modelType, + numeric, + navigate +}: Readonly<{ + value: any; + display?: string | null; + modelType?: string | null; + numeric?: boolean; + navigate: NavigateFunction; +}>): ReactNode { + if (value === null || value === undefined || value === '') { + return ( + + {t`None`} + + ); + } + + let text: string; + + if (display) { + text = display; + } else if (numeric && Number.isFinite(Number(value))) { + // Numeric values (decimals are recorded as strings, e.g. '10.00000') + text = String(formatDecimal(Number(value))); + } else if (typeof value === 'boolean') { + text = value ? t`Yes` : t`No`; + } else if (typeof value === 'object') { + text = JSON.stringify(value); + } else { + text = String(value); + } + + if (text.length > MAX_VALUE_LENGTH) { + text = `${text.slice(0, MAX_VALUE_LENGTH)}...`; + } + + // Link to related objects (only if the server resolved a display value) + const relatedType = resolveModelType(modelType); + + if (display && relatedType) { + return ( + + ); + } + + return {text}; +} + +/** + * Render a single field change + */ +function AuditChangeItem({ + change, + action, + navigate +}: Readonly<{ + change: AuditChange; + action: AuditAction; + navigate: NavigateFunction; +}>): ReactNode { + if (change.type === 'm2m') { + const added = change.operation === 'add'; + + return ( + + + {change.label}: + + + {added ? t`Added` : t`Removed`} + + + {change.objects ? change.objects.join(', ') : t`Hidden`} + + + ); + } + + const newValue = ( + + ); + + return ( + + + {change.label}: + + {action == AuditAction.UPDATE && ( + <> + + + + )} + {newValue} + + ); +} + +/** + * Return the list of changes to display for an audit entry + */ +function getChanges(record: any): AuditChange[] { + const changes: AuditChange[] = record?.changes_detail ?? []; + + if (record?.action == AuditAction.CREATE) { + // For new objects, empty fields are not interesting + return changes.filter( + (change) => + change.type == 'm2m' || + !(change.new === null || change.new === undefined || change.new === '') + ); + } + + return changes; +} + +/** + * Render a summary of the changes for an audit entry + */ +function AuditChangesSummary({ + record, + navigate +}: Readonly<{ + record: any; + navigate: NavigateFunction; +}>): ReactNode { + if (record.action == AuditAction.DELETE) { + return ( + + {t`Object was deleted`} + + ); + } + + const changes = getChanges(record); + + if (changes.length == 0) { + return ( + + {t`No changes recorded`} + + ); + } + + return ( + + {changes.slice(0, MAX_INLINE_CHANGES).map((change) => ( + + ))} + {changes.length > MAX_INLINE_CHANGES && ( + + {t`+ ${changes.length - MAX_INLINE_CHANGES} more (expand row to view)`} + + )} + + ); +} + +/** + * Render a table of all changes for an audit entry (shown when a row is expanded) + */ +function AuditChangesDetail({ + record, + navigate +}: Readonly<{ + record: any; + navigate: NavigateFunction; +}>): ReactNode { + const changes = getChanges(record); + const showOld = record.action != AuditAction.CREATE; + const showNew = record.action != AuditAction.DELETE; + + return ( + + + + + {t`Field`} + {showOld && {t`Old Value`}} + {showNew && {t`New Value`}} + + + + {changes.map((change) => ( + + + + {change.label} + + + {change.type == 'm2m' ? ( + + + + ) : ( + <> + {showOld && ( + + + + )} + {showNew && ( + + + + )} + + )} + + ))} + +
+ + ); +} + +/** + * Table for displaying audit log (change history) entries. + * + * - When modelType and modelId are provided, displays the history of a single object + * (including changes to its child objects, unless includeChildren is false) + * - When only modelType is provided, displays the history of all objects of that type + * - Otherwise, displays all audit log entries visible to the user + */ +export function AuditLogTable({ + modelType, + modelId, + includeChildren +}: Readonly<{ + modelType?: ModelType; + modelId?: number | string; + includeChildren?: boolean; +}>): ReactNode { + const navigate = useNavigate(); + const user = useUserState(); + + const table = useTable(modelType ? `audit-log-${modelType}` : 'audit-log'); + + const singleObject = modelType !== undefined && modelId !== undefined; + + // Model type choices are provided via the OPTIONS metadata for the list endpoint + const modelTypeFilters = useFilters({ + url: apiUrl(ApiEndpoints.audit_log_list), + method: 'OPTIONS', + accessor: 'data.filters.model_type.choices', + transform: (item: any) => ({ + value: item.value, + label: item.display_name + }), + // Not required if the table is restricted to a particular model type + enabled: !modelType + }); + + const tableFilters: TableFilter[] = useMemo(() => { + const filters: TableFilter[] = [ + { + name: 'action', + label: t`Action`, + description: t`Filter by action type`, + type: 'choice', + choices: [ + { value: `${AuditAction.CREATE}`, label: t`Created` }, + { value: `${AuditAction.UPDATE}`, label: t`Updated` }, + { value: `${AuditAction.DELETE}`, label: t`Deleted` } + ] + }, + UserFilter({ + name: 'actor', + label: t`User`, + description: t`Filter by the user who made the change` + }), + MinDateFilter(), + MaxDateFilter() + ]; + + if (!modelType) { + filters.push({ + name: 'model_type', + label: t`Model Type`, + description: t`Filter by the type of item which was changed`, + type: 'choice', + choices: modelTypeFilters.choices + }); + } + + if (singleObject && includeChildren === undefined) { + filters.push({ + name: 'include_children', + label: t`Include Related`, + description: t`Include changes to related items (e.g. line items)`, + type: 'boolean' + }); + } + + return filters; + }, [singleObject, includeChildren, modelType, modelTypeFilters.choices]); + + const tableColumns: TableColumn[] = useMemo(() => { + return [ + DateColumn({ + accessor: 'timestamp', + title: t`Date`, + switchable: false, + extra: { showTime: true } + }), + { + accessor: 'action', + title: t`Action`, + sortable: true, + switchable: false, + render: (record: any) => ( + + {actionLabel(record.action)} + + ) + }, + { + accessor: 'object_repr', + title: t`Item`, + sortable: false, + switchable: true, + render: (record: any) => { + const objectType = resolveModelType(record.model_type); + const typeLabel = objectType + ? ModelInformationDict[objectType].label() + : (record.model_label ?? record.model_type); + + // Deleted objects no longer have a detail page + const url = + objectType && record.action != AuditAction.DELETE + ? getDetailUrl(objectType, record.object_pk) + : undefined; + + return ( + + + + {typeLabel} + + + ); + } + }, + { + accessor: 'changes', + title: t`Changes`, + sortable: false, + switchable: false, + render: (record: any) => ( + + ) + }, + { + accessor: 'actor', + title: t`User`, + sortable: true, + switchable: true, + render: (record: any) => { + if (record.actor_detail) { + return RenderUser({ instance: record.actor_detail }); + } + + // The user account has since been deleted + if (record.actor_username) { + return ( + + {t`${record.actor_username} (deleted)`} + + ); + } + + if (record.additional_data?.source) { + return ( + + {record.additional_data.source} + + ); + } + + return {t`System`}; + } + }, + { + accessor: 'remote_addr', + title: t`IP Address`, + sortable: false, + switchable: true, + defaultVisible: false, + // Only returned by the API for staff users + hidden: !user.isStaff() + } + ]; + }, [navigate, user]); + + const rowExpansion: DataTableRowExpansionProps = useMemo(() => { + return { + allowMultiple: true, + expandable: ({ record }: { record: any }) => + record.action != AuditAction.DELETE && getChanges(record).length > 0, + content: ({ record }: { record: any }) => ( + + ) + }; + }, [navigate]); + + return ( + + ); +} diff --git a/src/frontend/tests/helpers.ts b/src/frontend/tests/helpers.ts index f6e5183eaa..d74e8fdfc3 100644 --- a/src/frontend/tests/helpers.ts +++ b/src/frontend/tests/helpers.ts @@ -141,6 +141,18 @@ export const activateCalendarView = async (page: Page) => { await page.waitForLoadState('networkidle'); }; +export const tableSearch = async (page: Page, query: string) => { + await page.getByLabel('table-search-input').fill(query); + await page.waitForTimeout(300); + await page.waitForLoadState('networkidle'); +}; + +export const clearTableSearch = async (page: Page) => { + await page.getByLabel('table-search-input').clear(); + await page.waitForTimeout(300); + await page.waitForLoadState('networkidle'); +}; + /** * Perform a 'global search' on the provided page, for the provided query text */ diff --git a/src/frontend/tests/pages/pui_build.spec.ts b/src/frontend/tests/pages/pui_build.spec.ts index 46d568ff77..e5f9a94228 100644 --- a/src/frontend/tests/pages/pui_build.spec.ts +++ b/src/frontend/tests/pages/pui_build.spec.ts @@ -3,6 +3,7 @@ import { test } from '../baseFixtures.ts'; import { activateCalendarView, clearTableFilters, + clearTableSearch, clickOnRowMenu, getRowFromCell, loadTab, @@ -12,6 +13,7 @@ import { showCalendarView, showParametricView, showTableView, + tableSearch, waitForDataOutput } from '../helpers.ts'; import { doCachedLogin } from '../login.ts'; @@ -275,7 +277,7 @@ test('Build Order - Build Outputs', async ({ browser }) => { await clearTableFilters(page); // We have now loaded the "Build Order" table. Check for some expected texts - await page.getByRole('textbox', { name: 'table-search-input' }).fill('1'); + await tableSearch(page, '1'); await page.getByText('On Hold').first().waitFor(); await page.getByText('Pending').first().waitFor(); @@ -757,11 +759,11 @@ test('Build Order - Filters', async ({ browser }) => { // Toggle 'Outstanding' filter await setTableChoiceFilter(page, 'Outstanding', 'Yes'); - await page.getByRole('textbox', { name: 'table-search-input' }).fill('1'); + await tableSearch(page, '1'); await page.getByRole('cell', { name: 'BO0017' }).waitFor(); await clearTableFilters(page); - await page.getByRole('textbox', { name: 'table-search-input' }).fill(''); + await clearTableSearch(page); await setTableChoiceFilter(page, 'Outstanding', 'No'); await page.getByText(/1 - \d+ \/ \d+/).waitFor(); diff --git a/src/frontend/tests/pages/pui_part.spec.ts b/src/frontend/tests/pages/pui_part.spec.ts index ed18aac434..ff79ff3187 100644 --- a/src/frontend/tests/pages/pui_part.spec.ts +++ b/src/frontend/tests/pages/pui_part.spec.ts @@ -13,7 +13,8 @@ import { openDetailAction, setTableChoiceFilter, showParametricView, - showTableView + showTableView, + tableSearch } from '../helpers'; import { adminuser } from '../defaults'; @@ -406,7 +407,7 @@ test('Parts - BOM Nesting', async ({ browser }) => { await expect(page.getByText('MAX232IDR')).toHaveCount(0); // Searching reloads the table, which discards sub-assembly rows - await page.getByLabel('table-search-input').fill('Doohickey'); + await tableSearch(page, 'Doohickey'); await expect(page.getByText('M3x10 Torx')).toHaveCount(0); await expect( page.getByRole('button', { name: 'nested-row-collapse' }) diff --git a/src/frontend/tests/pages/pui_purchasing.spec.ts b/src/frontend/tests/pages/pui_purchasing.spec.ts index be277b5e8e..3049d9146f 100644 --- a/src/frontend/tests/pages/pui_purchasing.spec.ts +++ b/src/frontend/tests/pages/pui_purchasing.spec.ts @@ -16,6 +16,7 @@ import { showCalendarView, showParametricView, showTableView, + tableSearch, waitForDataOutput } from '../helpers.ts'; import { doCachedLogin } from '../login.ts'; @@ -53,9 +54,7 @@ test('Purchasing - Index', async ({ browser }) => { // Check for expected values await clearTableFilters(page); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('R_100K_0402'); + await tableSearch(page, 'R_100K_0402'); await page.getByText('R_100K_0402_1%').first().waitFor(); await page.getByRole('cell', { name: 'RR05P100KDTR-ND' }).first().waitFor(); @@ -155,9 +154,7 @@ test('Purchasing - Manufacturer Parts', async ({ browser }) => { url: 'purchasing/index/manufacturer-parts' }); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('CPF0402B100KE1'); + await tableSearch(page, 'CPF0402B100KE1'); await page.getByText('R_100K_0402_1%').first().waitFor(); await page.getByRole('cell', { name: 'CPF0402B100KE1' }).waitFor(); @@ -609,9 +606,7 @@ test('Purchase Orders - Receive Items', async ({ browser }) => { await loadTab(page, 'Received Stock'); await clearTableFilters(page); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('my-batch-code'); + await tableSearch(page, 'my-batch-code'); await page.getByRole('cell', { name: 'my-batch-code' }).first().waitFor(); }); @@ -669,9 +664,7 @@ test('Purchase Orders - Custom Location', async ({ browser }) => { await loadTab(page, 'Received Stock'); await clearTableFilters(page); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('po-custom-location-test'); + await tableSearch(page, 'po-custom-location-test'); const receivedRow = page .getByRole('row') diff --git a/src/frontend/tests/pui_audit.spec.ts b/src/frontend/tests/pui_audit.spec.ts new file mode 100644 index 0000000000..1e4c632592 --- /dev/null +++ b/src/frontend/tests/pui_audit.spec.ts @@ -0,0 +1,100 @@ +import type { Page } from '@playwright/test'; +import { expect, test } from './baseFixtures.js'; +import { adminuser, allaccessuser, stevenuser } from './defaults.js'; +import { + clearTableSearch, + loadTab, + navigate, + setTableChoiceFilter, + tableSearch +} from './helpers.js'; +import { doCachedLogin } from './login.js'; + +/** + * Tests for the audit log (change history) functionality. + * + * Note: These tests rely on the audit log entries provided in the demo dataset + */ + +// Return the "History" tab on a detail page (if present) +const historyTab = (page: Page) => + page + .getByLabel(/panel-tabs-/) + .getByRole('tab', { name: 'History', exact: true }); + +test('Audit - Admin Center', async ({ browser }) => { + const page = await doCachedLogin(browser, { + user: adminuser, + url: 'settings/admin/audit-log' + }); + + await page.getByRole('button', { name: 'Audit Log', exact: true }).waitFor(); + + // Historical records are displayed + await tableSearch(page, 'red round table'); + await page.getByRole('cell', { name: 'BOM Item' }).first().waitFor(); + + await tableSearch(page, 'PO0020'); + await page.getByRole('cell', { name: 'Purchase Order' }).first().waitFor(); + await page.getByText('UPDATED').first().waitFor(); + + // Filter by model type + await clearTableSearch(page); + await setTableChoiceFilter(page, 'Model Type', 'Build Order'); + await page.getByRole('cell', { name: 'Build Order' }).first().waitFor(); + await expect(page.getByRole('cell', { name: 'BOM Item' })).toHaveCount(0); + + // Audit log settings + await page.getByRole('button', { name: 'Audit Log Settings' }).click(); + await page.getByText('Enable Audit Log').waitFor(); + await page.getByText('Delete Old Audit Log Entries').waitFor(); + await page.getByText('Audit Log Deletion Interval').waitFor(); + await page.getByText('Retain Audit Log Summary').waitFor(); +}); + +test('Audit - Detail Pages', async ({ browser }) => { + // Part (includes BOM item changes) + const page = await doCachedLogin(browser, { + user: adminuser, + url: 'part/111/details' + }); + + await loadTab(page, 'History', true); + await page + .getByRole('cell', { name: /to make TB2 \| Test Board 2/ }) + .first() + .waitFor(); + await page.getByText('Setup Quantity:').first().waitFor(); + + // Purchase order (includes line item changes) + await navigate(page, 'purchasing/purchase-order/9/detail'); + await loadTab(page, 'History', true); + + await page.getByRole('cell', { name: 'PO0009 - PCBWOY' }).first().waitFor(); + await page.getByText('Purchase Order Extra Line').first().waitFor(); + await page.getByText('Supplier Reference:').first().waitFor(); +}); + +test('Audit - Hidden', async ({ browser }) => { + // Staff user, but the 'DISPLAY_HISTORY_TAB' setting is not enabled + const page = await doCachedLogin(browser, { + user: stevenuser, + url: 'part/111/details' + }); + + await page.getByRole('tab', { name: 'Part Details' }).waitFor(); + await expect(historyTab(page)).toHaveCount(0); + + // Non-staff user + const page2 = await doCachedLogin(browser, { + user: allaccessuser, + url: 'part/111/details' + }); + + await page2.getByRole('tab', { name: 'Part Details' }).waitFor(); + await expect(historyTab(page2)).toHaveCount(0); + + await navigate(page2, 'purchasing/purchase-order/9/detail'); + await page2.getByRole('tab', { name: 'Order Details' }).waitFor(); + await expect(historyTab(page2)).toHaveCount(0); +}); diff --git a/src/frontend/tests/pui_permissions.spec.ts b/src/frontend/tests/pui_permissions.spec.ts index 73d29b0d6c..60f71fec0c 100644 --- a/src/frontend/tests/pui_permissions.spec.ts +++ b/src/frontend/tests/pui_permissions.spec.ts @@ -6,12 +6,18 @@ import { test } from './baseFixtures'; import { adminuser, readeruser } from './defaults'; import { clickOnRowMenu, loadTab } from './helpers'; import { doCachedLogin } from './login'; +import { setSettingState } from './settings'; /** * Test the "admin" account * - This is a superuser account, so should have *all* permissions available */ test('Permissions - Admin', async ({ browser }) => { + await setSettingState({ + setting: 'INVENTREE_SHOW_SUPERUSER_BANNER', + value: true + }); + // Login, and start on the "admin" page const page = await doCachedLogin(browser, { user: adminuser, diff --git a/src/frontend/tests/pui_plugins.spec.ts b/src/frontend/tests/pui_plugins.spec.ts index e00435a0a9..65f4b64763 100644 --- a/src/frontend/tests/pui_plugins.spec.ts +++ b/src/frontend/tests/pui_plugins.spec.ts @@ -6,7 +6,8 @@ import { clickOnRowMenu, loadTab, navigate, - setTableChoiceFilter + setTableChoiceFilter, + tableSearch } from './helpers.js'; import { doCachedLogin } from './login.js'; import { setPluginState, setSettingState } from './settings.js'; @@ -26,7 +27,7 @@ test('Plugins - Settings', async ({ browser }) => { // Navigate and select the plugin await navigate(page, 'settings/admin/plugin/'); await clearTableFilters(page); - await page.getByLabel('table-search-input').fill('integration'); + await tableSearch(page, 'integration'); await page.waitForTimeout(250); await page.waitForLoadState('networkidle'); @@ -305,7 +306,7 @@ test('Plugins - Creator', async ({ browser }) => { // View a custom table await page.getByRole('button', { name: 'Custom Table Example' }).click(); - await page.getByRole('textbox', { name: 'table-search-input' }).fill('red'); + await tableSearch(page, 'red'); await page.waitForLoadState('networkidle'); await page.getByRole('cell', { name: 'Red Square Table' }).first().waitFor(); }); diff --git a/src/frontend/tests/pui_printing.spec.ts b/src/frontend/tests/pui_printing.spec.ts index 47eb686abe..89503b71e8 100644 --- a/src/frontend/tests/pui_printing.spec.ts +++ b/src/frontend/tests/pui_printing.spec.ts @@ -5,6 +5,7 @@ import { activateTableView, loadTab, navigate, + tableSearch, waitForDataOutput } from './helpers.js'; import { doCachedLogin } from './login.js'; @@ -243,9 +244,7 @@ test('Printing - Broken Template', async ({ browser }) => { // Next, check error message from the template editor preview await navigate(page, 'settings/admin/reports'); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('broken'); + await tableSearch(page, 'broken'); await page.getByRole('cell', { name: 'Broken Sales Order Report' }).click(); await page.getByLabel('split-button-preview-options-action').click(); diff --git a/src/frontend/tests/pui_settings.spec.ts b/src/frontend/tests/pui_settings.spec.ts index 56c646213c..a6529a4de0 100644 --- a/src/frontend/tests/pui_settings.spec.ts +++ b/src/frontend/tests/pui_settings.spec.ts @@ -2,7 +2,13 @@ import type { Page } from '@playwright/test'; import { createApi } from './api.js'; import { expect, test } from './baseFixtures.js'; import { adminuser, allaccessuser, stevenuser } from './defaults.js'; -import { getRowFromCell, loadTab, navigate } from './helpers.js'; +import { + clearTableSearch, + getRowFromCell, + loadTab, + navigate, + tableSearch +} from './helpers.js'; import { doCachedLogin } from './login.js'; import { setPluginState, setSettingState } from './settings.js'; @@ -417,10 +423,8 @@ test('Settings - Admin - Barcode History', async ({ browser }) => { await page.waitForTimeout(500); const checkBarcode = async (barcode: string) => { - await page.getByRole('textbox', { name: 'table-search-input' }).clear(); - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill(barcode); + await clearTableSearch(page); + await tableSearch(page, barcode); await page.waitForLoadState('networkidle'); await page.getByText(barcode).first().waitFor(); }; @@ -441,16 +445,14 @@ test('Settings - Admin - Parameter', async ({ browser }) => { await loadTab(page, 'Selection Lists'); // Check for expected entry - await page - .getByRole('textbox', { name: 'table-search-input' }) - .fill('Animals'); + await tableSearch(page, 'Animals'); await page.getByRole('cell', { name: 'Animals', exact: true }).waitFor(); await page.getByText('Various animals and descriptions thereof').waitFor(); await page.waitForLoadState('networkidle'); await page.waitForTimeout(250); - await page.getByRole('textbox', { name: 'table-search-input' }).fill('some'); + await tableSearch(page, 'some'); await page.waitForTimeout(500); await page.waitForLoadState('networkidle'); diff --git a/tasks.py b/tasks.py index 8e018c0ac0..0379cc5056 100644 --- a/tasks.py +++ b/tasks.py @@ -1206,9 +1206,11 @@ def update( 'exclude_plugins': 'Exclude plugin data from the output file (default = False)', 'include_sso': 'Include SSO token data in the output file (default = False)', 'include_session': 'Include user session data in the output file (default = False)', + 'include_remote_addr': 'Include remote IP addresses for audit log entries in the output file (default = False)', 'prettify': 'Pretty-print the output file with indentation (default = False)', 'bulk': 'Use bulkdumpdata for improved performance on large datasets (default = False)', 'verbose': 'Print verbose output from management commands', + 'summary': 'Write a summary of the exported records (record counts and metadata) to the specified JSON file', } ) @state_logger @@ -1222,9 +1224,11 @@ def export_records( exclude_plugins: bool = False, include_sso: bool = False, include_session: bool = False, + include_remote_addr: bool = False, prettify: bool = False, bulk: bool = False, verbose: bool = False, + summary: Optional[str] = None, ): """Export all database records to a file.""" # Get an absolute path to the file @@ -1232,12 +1236,22 @@ def export_records( if not target.is_absolute(): target = local_dir().joinpath(filename).resolve() + summary_target = None + + if summary: + summary_target = Path(summary) + if not summary_target.is_absolute(): + summary_target = local_dir().joinpath(summary).resolve() + info(f"Exporting database records to file '{target}'") wait(c, verbose=verbose) check_file_existence(target, overwrite) + if summary_target: + check_file_existence(summary_target, overwrite) + excludes = content_excludes( allow_email=include_email, allow_tokens=include_tokens, @@ -1298,10 +1312,17 @@ def export_records( if model_name == 'auth.user': entry['fields']['user_permissions'] = [] + # Remove remote IP addresses from audit log entries + if include_remote_addr is False and model_name == 'audit.auditlogentry': + entry['fields']['remote_addr'] = None + yield entry indent = 2 if prettify else None + # Number of exported records, for each model type + record_counts: dict[str, int] = {} + # Write the processed data to file, one entry at a time - avoids ever # holding a second full copy of the (potentially huge) dataset in memory, # and avoids a single json.dumps() call across the entire dataset at once @@ -1313,10 +1334,25 @@ def export_records( if prettify: f_out.write('\n') f_out.write(json.dumps(entry, indent=indent)) + + if model_name := entry.get('model'): + record_counts[model_name] = record_counts.get(model_name, 0) + 1 if prettify: f_out.write('\n') f_out.write(']') + if summary_target: + info(f"Writing export summary to file '{summary_target}'") + + summary_data = { + 'metadata': metadata_entry, + 'total_records': sum(record_counts.values()), + 'records': dict(sorted(record_counts.items())), + } + + with open(summary_target, 'w', encoding='utf-8') as f_out: + json.dump(summary_data, f_out, indent=2) + success('Data export completed') @@ -2126,6 +2162,7 @@ def export_definitions(c, basedir: str = ''): base_path.joinpath('inventree_report_context.json'), base_path.joinpath('inventree_status_codes.json'), base_path.joinpath('inventree_roles.json'), + base_path.joinpath('inventree_audited_models.json'), ] info('Exporting definitions...') @@ -2146,6 +2183,9 @@ def export_definitions(c, basedir: str = ''): check_file_existence(filenames[5], overwrite=True) manage(c, f'export_roles {filenames[5]}', pty=True) + check_file_existence(filenames[6], overwrite=True) + manage(c, f'export_audited_models {filenames[6]}', pty=True) + info('Exporting definitions complete')