diff --git a/docs/docs/api/schema.md b/docs/docs/api/schema.md index 441e977e2e..66830a3a59 100644 --- a/docs/docs/api/schema.md +++ b/docs/docs/api/schema.md @@ -7,7 +7,7 @@ The API schema as documented below is generated using the [drf-spectactular](htt ## API Version -This documentation is for API version: `171` +This documentation is for API version: `315` !!! tip "API Schema History" We track API schema changes, and provide a snapshot of each API schema version in the [API schema repository](https://github.com/inventree/schema/). diff --git a/docs/docs/build/bom.md b/docs/docs/build/bom.md index 281e801d41..e4ee23d29a 100644 --- a/docs/docs/build/bom.md +++ b/docs/docs/build/bom.md @@ -119,7 +119,7 @@ Select a part in the list and click on "Add Substitute" button to confirm. ### Validate BOM -After [adding BOM items manually](#add-bom-item) or [uploading a BOM file](./bom_import.md), you should see the following view: +After [adding BOM items manually](#add-bom-item) or uploading a BOM file, you should see the following view: {% with id="bom_invalid", url="build/bom_invalid.png", description="Invalid BOM View" %} {% include 'img.html' %} {% endwith %} diff --git a/docs/docs/build/bom_export.md b/docs/docs/build/bom_export.md deleted file mode 100644 index 541b481286..0000000000 --- a/docs/docs/build/bom_export.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: BOM Export ---- - -## Exporting BOM Data - - -BOM data can be exported for any given assembly by selecting the *Export BOM* action from the BOM actions menu. - -You will be presented with the *Export BOM* options dialog, shown below: - -{% with id="bom_export", url="build/bom_export.png", description="Export BOM Data" %} -{% include 'img.html' %} -{% endwith %} - -### BOM Export Options - -**Format** - -Select the file format for the exported BOM data - -**Multi Level BOM** - -If selected, BOM data will be included for any subassemblies. If not selected, only top level (flat) BOM data will be exported. - -**Levels** - -Define the maximum level of data to export for subassemblies. If set to zero, all levels of subassembly data will be exported. - -**Include Parameter Data** - -Include part parameter data in the exported dataset. - -**Include Stock Data** - -Include part stock level information in the exported dataset. - -**Include Manufacturer Data** - -Include part manufacturer information in the exported dataset. - -**Include Supplier Data** - -Include part supplier information in the exported dataset. diff --git a/docs/docs/build/bom_import.md b/docs/docs/build/bom_import.md deleted file mode 100644 index 95c4cd572e..0000000000 --- a/docs/docs/build/bom_import.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: BOM Import ---- - -## Importing BOM Data - -Uploading a BOM to InvenTree is a three steps process: - -1. Upload BOM file -0. Select matching InvenTree fields -0. Select matching InvenTree parts. - -To upload a BOM file, navigate to the part/assembly detail page then click on the "BOM" tab. On top of the tab view, click on the icon then, after the page reloads, click on the icon. - -The following view will load: -{% with id="bom_upload_file", url="build/bom_upload_file.png", description="BOM Upload View" %} -{% include 'img.html' %} -{% endwith %} - -#### Upload BOM File - -Click on the "Choose File" button, select your BOM file when prompted then click on the "Upload File" button. - -!!! info "BOM Formats" - The following BOM file formats are supported: CSV, TSV, XLS, XLSX, JSON and YAML - -#### Select Fields - -Once the BOM file is uploaded, the following view will load: -{% with id="bom_select_fields", url="build/bom_select_fields.png", description="Select Fields View" %} -{% include 'img.html' %} -{% endwith %} - -InvenTree will attempt to automatically match the BOM file columns with InvenTree part fields. `Part_Name` is a **required** field for the upload process and moving on to the next step. Specifying the `Part_IPN` field matching is very powerful as it allows to create direct pointers to InvenTree parts. - -Once you have selected the corresponding InvenTree fields, click on the "Submit Selections" button to move on to the next step. - -#### Select Parts - -Once the BOM file columns and InvenTree fields are correctly matched, the following view will load: -{% with id="bom_select_parts", url="build/bom_select_parts.png", description="Select Parts View" %} -{% include 'img.html' %} -{% endwith %} - -InvenTree automatically tries to match parts from the BOM file with parts in its database. For parts that are found in InvenTree's database, the `Select Part` field selection will automatically point to the matching database part. - -In this view, you can also edit the parts `Reference` and `Quantity` fields. - -Once you have selected the corresponding InvenTree parts, click on the "Submit BOM" button to complete the BOM upload process. diff --git a/docs/docs/extend/plugins.md b/docs/docs/extend/plugins.md index c206f5786b..3be254e8de 100644 --- a/docs/docs/extend/plugins.md +++ b/docs/docs/extend/plugins.md @@ -113,6 +113,7 @@ Supported mixin classes are: | [AppMixin](./plugins/app.md) | Integrate additional database tables | | [BarcodeMixin](./plugins/barcode.md) | Support custom barcode actions | | [CurrencyExchangeMixin](./plugins/currency.md) | Custom interfaces for currency exchange rates | +| [DataExport](./plugins/export.md) | Customize data export functionality | | [EventMixin](./plugins/event.md) | Respond to events | | [LabelPrintingMixin](./plugins/label.md) | Custom label printing support | | [LocateMixin](./plugins/locate.md) | Locate and identify stock items | diff --git a/docs/docs/extend/plugins/export.md b/docs/docs/extend/plugins/export.md new file mode 100644 index 0000000000..98c893cfc8 --- /dev/null +++ b/docs/docs/extend/plugins/export.md @@ -0,0 +1,122 @@ +--- +title: Data Export Mixin +--- + +## DataExportMixin + +The `DataExportMixin` class provides a plugin with the ability to customize the data export process. The [InvenTree API](../../api/api.md) provides an integrated method to export a dataset to a tabulated file. The default export process is generic, and simply exports the data presented via the API in a tabulated file format. + +Custom data export plugins allow this process to be adjusted: + +- Data columns can be added or removed +- Rows can be removed or added +- Custom calculations or annotations can be performed. + +### Supported Export Types + +Each plugin can dictate which datasets are supported using the `supports_export` method. This allows a plugin to dynamically specify whether it can be selected by the user for a given export session. + +::: plugin.base.integration.DataExport.DataExportMixin.supports_export + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_sources: True + summary: False + members: [] + +The default implementation returns `True` for all data types. + +### Filename Generation + +The `generate_filename` method constructs a filename for the exported file. + +::: plugin.base.integration.DataExport.DataExportMixin.generate_filename + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_sources: True + summary: False + members: [] + +### Adjust Columns + +The `update_headers` method allows the plugin to adjust the columns selected to be exported to the file. + +::: plugin.base.integration.DataExport.DataExportMixin.update_headers + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_sources: True + summary: False + members: [] + +### Queryset Filtering + +The `filter_queryset` method allows the plugin to provide custom filtering to the database query, before it is exported. + +::: plugin.base.integration.DataExport.DataExportMixin.filter_queryset + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_sources: True + summary: False + members: [] + +### Export Data + +The `export_data` method performs the step of transforming a [Django QuerySet]({% include "django.html" %}/ref/models/querysets/) into a dataset which can be processed by the [tablib](https://tablib.readthedocs.io/en/stable/) library. + +::: plugin.base.integration.DataExport.DataExportMixin.export_data + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_sources: True + summary: False + members: [] + +Note that the default implementation simply uses the builtin tabulation functionality of the provided serializer class. In most cases, this will be sufficient. + +## Custom Export Options + +To provide the user with custom options to control the behavior of the export process *at the time of export*, the plugin can define a custom serializer class. + +To enable this feature, define an `ExportOptionsSerializer` attribute on the plugin class which points to a DRF serializer class. Refer to the examples below for more information. + +### Builtin Exporter Classes + +InvenTree provides the following builtin data exporter classes. + +### InvenTreeExporter + +A generic exporter class which simply serializes the API output into a data file. + +::: plugin.builtin.exporter.inventree_exporter.InvenTreeExporter + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_source: True + members: [] + +### BOM Exporter + +A custom exporter which only supports [bill of materials](../../build/bom.md) exporting. + +::: plugin.builtin.exporter.bom_exporter.BomExporterPlugin + options: + show_bases: False + show_root_heading: False + show_root_toc_entry: False + show_source: True + members: [] + +## Source Code + +The full source code of the `DataExportMixin` class: + +{{ includefile("src/backend/InvenTree/plugin/base/integration/DataExport.py", title="DataExportMixin") }} diff --git a/docs/docs/extend/plugins/report.md b/docs/docs/extend/plugins/report.md index 282c86e4e7..09e6e9d89a 100644 --- a/docs/docs/extend/plugins/report.md +++ b/docs/docs/extend/plugins/report.md @@ -4,7 +4,7 @@ title: Report Mixin ## ReportMixin -The ReportMixin class provides a plugin with the ability to extend the functionality of custom [report templates](../../report/report.md). A plugin which implements the ReportMixin mixin class can add custom context data to a report template for rendering. +The `ReportMixin` class provides a plugin with the ability to extend the functionality of custom [report templates](../../report/report.md). A plugin which implements the ReportMixin mixin class can add custom context data to a report template for rendering. ### Add Report Context diff --git a/docs/main.py b/docs/main.py index 70c6cbb126..659d5af6dc 100644 --- a/docs/main.py +++ b/docs/main.py @@ -102,7 +102,7 @@ def check_link(url) -> bool: return False -def get_build_enviroment() -> str: +def get_build_environment() -> str: """Returns the branch we are currently building on, based on the environment variables of the various CI platforms.""" # Check if we are in ReadTheDocs if os.environ.get('READTHEDOCS') == 'True': @@ -134,7 +134,7 @@ def define_env(env): - FileNotFoundError: If the directory does not exist, or the generated URL is invalid """ if branch == None: - branch = get_build_enviroment() + branch = get_build_environment() if dirname.startswith('/'): dirname = dirname[1:] @@ -173,7 +173,7 @@ def define_env(env): - FileNotFoundError: If the file does not exist, or the generated URL is invalid """ if branch == None: - branch = get_build_enviroment() + branch = get_build_environment() if filename.startswith('/'): filename = filename[1:] diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 4957d45436..d18b94d346 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -130,8 +130,6 @@ nav: - Allocating Stock: build/allocate.md - Example Build Order: build/example.md - Bill of Materials: build/bom.md - - Importing BOM Data: build/bom_import.md - - Exporting BOM Data: build/bom_export.md - Orders: - Companies: order/company.md - Purchase Orders: order/purchase_order.md @@ -208,6 +206,7 @@ nav: - App Mixin: extend/plugins/app.md - Barcode Mixin: extend/plugins/barcode.md - Currency Mixin: extend/plugins/currency.md + - Data Export Mixin: extend/plugins/export.md - Event Mixin: extend/plugins/event.md - Icon Pack Mixin: extend/plugins/icon.md - Label Printing Mixin: extend/plugins/label.md diff --git a/src/backend/InvenTree/InvenTree/api_version.py b/src/backend/InvenTree/InvenTree/api_version.py index 24a13aaa6e..ab243f423d 100644 --- a/src/backend/InvenTree/InvenTree/api_version.py +++ b/src/backend/InvenTree/InvenTree/api_version.py @@ -1,13 +1,18 @@ """InvenTree API version information.""" # InvenTree API version -INVENTREE_API_VERSION = 325 +INVENTREE_API_VERSION = 326 """Increment this API version number whenever there is a significant change to the API that any clients need to know about.""" INVENTREE_API_TEXT = """ +v326 - 2025-03-18 : https://github.com/inventree/InvenTree/pull/9096 + - Overhaul the data-export API functionality + - Allow customization of data exporting via plugins + - Consolidate LabelOutput and ReportOutput API endpoints into single DataOutput endpoint + v325 - 2024-03-17 : https://github.com/inventree/InvenTree/pull/9244 - Adds the option for superusers to list all user tokens - Make list endpoints sortable, filterable and searchable diff --git a/src/backend/InvenTree/InvenTree/apps.py b/src/backend/InvenTree/InvenTree/apps.py index bb24e109e7..59f480577c 100644 --- a/src/backend/InvenTree/InvenTree/apps.py +++ b/src/backend/InvenTree/InvenTree/apps.py @@ -88,6 +88,8 @@ class InvenTreeConfig(AppConfig): 'InvenTree.tasks.delete_expired_sessions', 'stock.tasks.delete_old_stock_items', 'label.tasks.cleanup_old_label_outputs', + 'report.tasks.cleanup_old_report_outputs', + 'data_exporter.tasks.cleanup_old_export_outputs', ] try: diff --git a/src/backend/InvenTree/InvenTree/exchange.py b/src/backend/InvenTree/InvenTree/exchange.py index 6d97bfaabc..29fec64f29 100644 --- a/src/backend/InvenTree/InvenTree/exchange.py +++ b/src/backend/InvenTree/InvenTree/exchange.py @@ -22,7 +22,7 @@ class InvenTreeExchange(SimpleExchangeBackend): def get_rates(self, **kwargs) -> dict: """Set the requested currency codes and get rates.""" - from plugin import registry + from plugin import PluginMixinEnum, registry base_currency = kwargs.get('base_currency', currency_code_default()) symbols = kwargs.get('symbols', currency_codes()) @@ -34,7 +34,9 @@ class InvenTreeExchange(SimpleExchangeBackend): if not plugin: # Find the first active currency exchange plugin - plugins = registry.with_mixin('currencyexchange', active=True) + plugins = registry.with_mixin( + PluginMixinEnum.CURRENCY_EXCHANGE, active=True + ) if len(plugins) > 0: plugin = plugins[0] diff --git a/src/backend/InvenTree/InvenTree/helpers.py b/src/backend/InvenTree/InvenTree/helpers.py index 18b80ec850..c5b329a82f 100644 --- a/src/backend/InvenTree/InvenTree/helpers.py +++ b/src/backend/InvenTree/InvenTree/helpers.py @@ -384,9 +384,14 @@ def WrapWithQuotes(text, quote='"'): return text -def GetExportFormats(): +def GetExportOptions() -> list: + """Return a set of allowable import / export file formats.""" + return [['csv', 'CSV'], ['xlsx', 'Excel'], ['tsv', 'TSV']] + + +def GetExportFormats() -> list: """Return a list of allowable file formats for importing or exporting tabular data.""" - return ['csv', 'xlsx', 'tsv', 'json'] + return [opt[0] for opt in GetExportOptions()] def DownloadFile( @@ -437,14 +442,14 @@ def increment_serial_number(serial, part=None): incremented value, or None if incrementing could not be performed. """ from InvenTree.exceptions import log_error - from plugin.registry import registry + from plugin import PluginMixinEnum, registry # Ensure we start with a string value if serial is not None: serial = str(serial).strip() # First, let any plugins attempt to increment the serial number - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: if not hasattr(plugin, 'increment_serial_number'): continue diff --git a/src/backend/InvenTree/InvenTree/mixins.py b/src/backend/InvenTree/InvenTree/mixins.py index 645a4442c2..b0f22e5dfc 100644 --- a/src/backend/InvenTree/InvenTree/mixins.py +++ b/src/backend/InvenTree/InvenTree/mixins.py @@ -5,6 +5,8 @@ from django.core.exceptions import FieldDoesNotExist from rest_framework import generics, mixins, status from rest_framework.response import Response +import data_exporter.mixins +import importer.mixins from InvenTree.fields import InvenTreeNotesField from InvenTree.helpers import ( clean_markdown, @@ -197,3 +199,10 @@ class RetrieveDestroyAPI(generics.RetrieveDestroyAPIView): class UpdateAPI(CleanMixin, generics.UpdateAPIView): """View for update API.""" + + +class DataImportExportSerializerMixin( + data_exporter.mixins.DataExportSerializerMixin, + importer.mixins.DataImportSerializerMixin, +): + """Mixin class for adding data import/export functionality to a DRF serializer.""" diff --git a/src/backend/InvenTree/InvenTree/models.py b/src/backend/InvenTree/InvenTree/models.py index 92c19a227f..1e0738d802 100644 --- a/src/backend/InvenTree/InvenTree/models.py +++ b/src/backend/InvenTree/InvenTree/models.py @@ -87,11 +87,11 @@ class PluginValidationMixin(DiffMixin): def run_plugin_validation(self): """Throw this model against the plugin validation interface.""" - from plugin.registry import registry + from plugin import PluginMixinEnum, registry deltas = self.get_field_deltas() - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: if plugin.validate_model_instance(self, deltas=deltas) is True: return @@ -130,9 +130,9 @@ class PluginValidationMixin(DiffMixin): Note: Each plugin may raise a ValidationError to prevent deletion. """ from InvenTree.exceptions import log_error - from plugin.registry import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: plugin.validate_model_deletion(self) except ValidationError as e: diff --git a/src/backend/InvenTree/InvenTree/settings.py b/src/backend/InvenTree/InvenTree/settings.py index 7c98d8c5b7..617a39a028 100644 --- a/src/backend/InvenTree/InvenTree/settings.py +++ b/src/backend/InvenTree/InvenTree/settings.py @@ -250,14 +250,15 @@ INSTALLED_APPS = [ # InvenTree apps 'build.apps.BuildConfig', 'common.apps.CommonConfig', - 'company.apps.CompanyConfig', 'plugin.apps.PluginAppConfig', # Plugin app runs before all apps that depend on the isPluginRegistryLoaded function + 'company.apps.CompanyConfig', 'order.apps.OrderConfig', 'part.apps.PartConfig', 'report.apps.ReportConfig', 'stock.apps.StockConfig', 'users.apps.UsersConfig', 'machine.apps.MachineConfig', + 'data_exporter.apps.DataExporterConfig', 'importer.apps.ImporterConfig', 'web', 'generic', diff --git a/src/backend/InvenTree/InvenTree/tasks.py b/src/backend/InvenTree/InvenTree/tasks.py index 6a91e53a3c..dbe91d1974 100644 --- a/src/backend/InvenTree/InvenTree/tasks.py +++ b/src/backend/InvenTree/InvenTree/tasks.py @@ -280,14 +280,15 @@ class ScheduledTask: interval: str minutes: Optional[int] = None - MINUTES = 'I' - HOURLY = 'H' - DAILY = 'D' - WEEKLY = 'W' - MONTHLY = 'M' - QUARTERLY = 'Q' - YEARLY = 'Y' - TYPE = [MINUTES, HOURLY, DAILY, WEEKLY, MONTHLY, QUARTERLY, YEARLY] + MINUTES: str = 'I' + HOURLY: str = 'H' + DAILY: str = 'D' + WEEKLY: str = 'W' + MONTHLY: str = 'M' + QUARTERLY: str = 'Q' + YEARLY: str = 'Y' + + TYPE: tuple[str] = (MINUTES, HOURLY, DAILY, WEEKLY, MONTHLY, QUARTERLY, YEARLY) class TaskRegister: diff --git a/src/backend/InvenTree/InvenTree/unit_test.py b/src/backend/InvenTree/InvenTree/unit_test.py index 6511c384fa..8fee88a85c 100644 --- a/src/backend/InvenTree/InvenTree/unit_test.py +++ b/src/backend/InvenTree/InvenTree/unit_test.py @@ -486,7 +486,13 @@ class InvenTreeAPITestCase(ExchangeRateMixin, UserMixin, APITestCase): ) def download_file( - self, url, data, expected_code=None, expected_fn=None, decode=True, **kwargs + self, + url, + data=None, + expected_code=None, + expected_fn=None, + decode=True, + **kwargs, ): """Download a file from the server, and return an in-memory file.""" response = self.client.get(url, data=data, format='json') @@ -502,9 +508,11 @@ class InvenTreeAPITestCase(ExchangeRateMixin, UserMixin, APITestCase): # Extract filename disposition = response.headers['Content-Disposition'] - result = re.search(r'attachment; filename="([\w\d\-.]+)"', disposition) + result = re.search( + r'(attachment|inline); filename=[\'"]([\w\d\-.]+)[\'"]', disposition + ) - fn = result.groups()[0] + fn = result.groups()[1] if expected_fn is not None: self.assertRegex(fn, expected_fn) @@ -524,6 +532,72 @@ class InvenTreeAPITestCase(ExchangeRateMixin, UserMixin, APITestCase): return file + def export_data( + self, + url, + params=None, + export_format='csv', + export_plugin='inventree-exporter', + **kwargs, + ): + """Perform a data export operation against the provided URL. + + Uses the 'data_exporter' functionality to override the POST response. + + Arguments: + url: URL to perform the export operation against + params: Dictionary of parameters to pass to the export operation + export_format: Export format (default = 'csv') + export_plugin: Export plugin (default = 'inventree-exporter') + + Returns: + A file object containing the exported dataset + """ + # Ensure that the plugin registry is up-to-date + registry.reload_plugins(full_reload=True, force_reload=True, collect=True) + + download = kwargs.pop('download', True) + expected_code = kwargs.pop('expected_code', 200) + + if not params: + params = {} + + params = { + **params, + 'export': True, + 'export_format': export_format, + 'export_plugin': export_plugin, + } + + # Add in any other export specific kwargs + for key, value in kwargs.items(): + if key.startswith('export_'): + params[key] = value + + # Append URL params + url += '?' + '&'.join([f'{key}={value}' for key, value in params.items()]) + + response = self.client.get(url, data=None, format='json') + self.check_response(url, response, expected_code=expected_code) + + # Check that the response is of the correct type + data = response.data + + if expected_code != 200: + # Response failed + return response.data + + self.assertEqual(data['plugin'], export_plugin) + self.assertTrue(data['complete']) + filename = data.get('output') + self.assertIsNotNone(filename) + + if download: + return self.download_file(filename, **kwargs) + + else: + return response.data + def process_csv( self, file_object, diff --git a/src/backend/InvenTree/build/api.py b/src/backend/InvenTree/build/api.py index c902abe12f..9fb43d1c87 100644 --- a/src/backend/InvenTree/build/api.py +++ b/src/backend/InvenTree/build/api.py @@ -16,8 +16,8 @@ import common.models import part.models as part_models from build.models import Build, BuildItem, BuildLine from build.status_codes import BuildStatus, BuildStatusGroups +from data_exporter.mixins import DataExportViewMixin from generic.states.api import StatusView -from importer.mixins import DataExportViewMixin from InvenTree.api import BulkDeleteMixin, MetadataView from InvenTree.filters import SEARCH_ORDER_FILTER_ALIAS, InvenTreeDateFilter from InvenTree.helpers import isNull, str2bool @@ -388,7 +388,7 @@ class BuildList(DataExportViewMixin, BuildMixin, ListCreateAPI): kwargs['part_detail'] = part_detail kwargs['create'] = True - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class BuildDetail(BuildMixin, RetrieveUpdateDestroyAPI): @@ -529,7 +529,7 @@ class BuildLineEndpoint: except AttributeError: pass - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_source_build(self) -> Build: """Return the source Build object for the BuildLine queryset. @@ -838,7 +838,7 @@ class BuildItemList(DataExportViewMixin, BulkDeleteMixin, ListCreateAPI): except AttributeError: pass - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self): """Override the queryset method, to perform custom prefetch.""" diff --git a/src/backend/InvenTree/build/serializers.py b/src/backend/InvenTree/build/serializers.py index afc21240b4..c017f0480c 100644 --- a/src/backend/InvenTree/build/serializers.py +++ b/src/backend/InvenTree/build/serializers.py @@ -31,7 +31,7 @@ import part.serializers as part_serializers from common.serializers import ProjectCodeSerializer from common.settings import get_global_setting from generic.states.fields import InvenTreeCustomStatusSerializerMixin -from importer.mixins import DataImportExportSerializerMixin +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.ready import isGeneratingSchema from InvenTree.serializers import ( InvenTreeDecimalField, diff --git a/src/backend/InvenTree/build/test_api.py b/src/backend/InvenTree/build/test_api.py index 887e743292..a29d64143a 100644 --- a/src/backend/InvenTree/build/test_api.py +++ b/src/backend/InvenTree/build/test_api.py @@ -449,7 +449,6 @@ class BuildTest(BuildAPITest): 'Build Status', 'Completed items', 'Batch Code', - 'Notes', 'Description', 'Part', 'Part Name', @@ -459,9 +458,9 @@ class BuildTest(BuildAPITest): excluded_cols = ['lft', 'rght', 'tree_id', 'level', 'metadata'] - with self.download_file(reverse('api-build-list'), {'export': 'csv'}) as file: + with self.export_data(reverse('api-build-list')) as data_file: data = self.process_csv( - file, + data_file, required_cols=required_cols, excluded_cols=excluded_cols, required_rows=Build.objects.count(), diff --git a/src/backend/InvenTree/common/admin.py b/src/backend/InvenTree/common/admin.py index 1326085082..d9885a5b38 100644 --- a/src/backend/InvenTree/common/admin.py +++ b/src/backend/InvenTree/common/admin.py @@ -33,6 +33,15 @@ class AttachmentAdmin(admin.ModelAdmin): search_fields = ('content_type', 'comment') +@admin.register(common.models.DataOutput) +class DataOutputAdmin(admin.ModelAdmin): + """Admin interface for DataOutput objects.""" + + list_display = ('user', 'created', 'output_type', 'output') + + list_filter = ('user', 'output_type') + + @admin.register(common.models.BarcodeScanResult) class BarcodeScanResultAdmin(admin.ModelAdmin): """Admin interface for BarcodeScanResult objects.""" diff --git a/src/backend/InvenTree/common/api.py b/src/backend/InvenTree/common/api.py index 82c51c9d30..c40542a695 100644 --- a/src/backend/InvenTree/common/api.py +++ b/src/backend/InvenTree/common/api.py @@ -31,8 +31,8 @@ import common.serializers import InvenTree.conversion from common.icons import get_icon_packs from common.settings import get_global_setting +from data_exporter.mixins import DataExportViewMixin from generic.states.api import urlpattern as generic_states_api_urls -from importer.mixins import DataExportViewMixin from InvenTree.api import BulkDeleteMixin, MetadataView from InvenTree.config import CONFIG_LOOKUPS from InvenTree.filters import ORDER_FILTER, SEARCH_ORDER_FILTER @@ -854,6 +854,25 @@ class SelectionEntryDetail(EntryMixin, RetrieveUpdateDestroyAPI): """Detail view for a SelectionEntry object.""" +class DataOutputEndpoint: + """Mixin class for DataOutput endpoints.""" + + queryset = common.models.DataOutput.objects.all() + serializer_class = common.serializers.DataOutputSerializer + permission_classes = [permissions.IsAuthenticated] + + +class DataOutputList(DataOutputEndpoint, BulkDeleteMixin, ListAPI): + """List view for DataOutput objects.""" + + filter_backends = SEARCH_ORDER_FILTER + ordering_fields = ['pk', 'user', 'plugin', 'output_type', 'created'] + + +class DataOutputDetail(DataOutputEndpoint, RetrieveAPI): + """Detail view for a DataOutput object.""" + + selection_urls = [ path( '/', @@ -1096,6 +1115,16 @@ common_api_urls = [ path('icons/', IconList.as_view(), name='api-icon-list'), # Selection lists path('selection/', include(selection_urls)), + # Data output + path( + 'data-output/', + include([ + path( + '/', DataOutputDetail.as_view(), name='api-data-output-detail' + ), + path('', DataOutputList.as_view(), name='api-data-output-list'), + ]), + ), ] admin_api_urls = [ diff --git a/src/backend/InvenTree/common/currency.py b/src/backend/InvenTree/common/currency.py index 6841529e81..b7749db39c 100644 --- a/src/backend/InvenTree/common/currency.py +++ b/src/backend/InvenTree/common/currency.py @@ -132,9 +132,9 @@ def validate_currency_codes(value): def currency_exchange_plugins() -> Optional[list]: """Return a list of plugin choices which can be used for currency exchange.""" try: - from plugin import registry + from plugin import PluginMixinEnum, registry - plugs = registry.with_mixin('currencyexchange', active=True) + plugs = registry.with_mixin(PluginMixinEnum.CURRENCY_EXCHANGE, active=True) except Exception: plugs = [] diff --git a/src/backend/InvenTree/common/icons.py b/src/backend/InvenTree/common/icons.py index 5ad87bea5a..76747b1954 100644 --- a/src/backend/InvenTree/common/icons.py +++ b/src/backend/InvenTree/common/icons.py @@ -73,9 +73,9 @@ def get_icon_packs(): ] from InvenTree.exceptions import log_error - from plugin import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('icon_pack', active=True): + for plugin in registry.with_mixin(PluginMixinEnum.ICON_PACK, active=True): try: icon_packs.extend(plugin.icon_packs()) except Exception as e: diff --git a/src/backend/InvenTree/common/migrations/0037_dataoutput.py b/src/backend/InvenTree/common/migrations/0037_dataoutput.py new file mode 100644 index 0000000000..72bf97eb1b --- /dev/null +++ b/src/backend/InvenTree/common/migrations/0037_dataoutput.py @@ -0,0 +1,58 @@ +# Generated by Django 4.2.19 on 2025-03-11 08:22 + +from django.conf import settings +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + migrations.swappable_dependency(settings.AUTH_USER_MODEL), + ("common", "0036_alter_attachment_link"), + ] + + operations = [ + migrations.CreateModel( + name="DataOutput", + fields=[ + ( + "id", + models.AutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ("created", models.DateField(auto_now_add=True)), + ("total", models.PositiveIntegerField(default=1)), + ("progress", models.PositiveIntegerField(default=0)), + ("complete", models.BooleanField(default=False)), + ( + "output_type", + models.CharField(blank=True, max_length=100, null=True), + ), + ( + "template_name", + models.CharField(blank=True, max_length=100, null=True), + ), + ("plugin", models.CharField(blank=True, max_length=100, null=True)), + ( + "output", + models.FileField(blank=True, null=True, upload_to="data_output"), + ), + ("errors", models.JSONField(blank=True, null=True)), + ( + "user", + models.ForeignKey( + blank=True, + null=True, + on_delete=django.db.models.deletion.SET_NULL, + related_name="+", + to=settings.AUTH_USER_MODEL, + ), + ), + ], + ), + ] diff --git a/src/backend/InvenTree/common/models.py b/src/backend/InvenTree/common/models.py index 61213b5d17..c486fdedfe 100644 --- a/src/backend/InvenTree/common/models.py +++ b/src/backend/InvenTree/common/models.py @@ -4,6 +4,7 @@ These models are 'generic' and do not fit a particular business logic object. """ import base64 +import enum import hashlib import hmac import json @@ -2312,3 +2313,56 @@ class BarcodeScanResult(InvenTree.models.InvenTreeModel): help_text=_('Was the barcode scan successful?'), default=False, ) + + +class DataOutput(models.Model): + """Model for storing generated data output from various processes. + + This model is intended for storing data files which are generated by various processes, + and need to be retained for future use (e.g. download by the user). + + Attributes: + created: Date and time that the data output was created + user: User who created the data output (if applicable) + total: Total number of items / records in the data output + progress: Current progress of the data output generation process + complete: Has the data output generation process completed? + output_type: The type of data output generated (e.g. 'label', 'report', etc) + template_name: Name of the template used to generate the data output (if applicable) + plugin: Key for the plugin which generated the data output (if applicable) + output: File field for storing the generated file + errors: JSON field for storing any errors generated during the data output generation process + """ + + class DataOutputTypes(str, enum.Enum): + """Enum for data output types.""" + + def __str__(self): + """Return the string representation of the data output type.""" + return str(self.value) + + LABEL = 'label' + REPORT = 'report' + EXPORT = 'export' + + created = models.DateField(auto_now_add=True, editable=False) + + user = models.ForeignKey( + User, on_delete=models.SET_NULL, blank=True, null=True, related_name='+' + ) + + total = models.PositiveIntegerField(default=1) + + progress = models.PositiveIntegerField(default=0) + + complete = models.BooleanField(default=False) + + output_type = models.CharField(max_length=100, blank=True, null=True) + + template_name = models.CharField(max_length=100, blank=True, null=True) + + plugin = models.CharField(max_length=100, blank=True, null=True) + + output = models.FileField(upload_to='data_output', blank=True, null=True) + + errors = models.JSONField(blank=True, null=True) diff --git a/src/backend/InvenTree/common/serializers.py b/src/backend/InvenTree/common/serializers.py index 5410437abd..896c75d9e3 100644 --- a/src/backend/InvenTree/common/serializers.py +++ b/src/backend/InvenTree/common/serializers.py @@ -15,10 +15,10 @@ from taggit.serializers import TagListSerializerField import common.models as common_models import common.validators import generic.states.custom -from importer.mixins import DataImportExportSerializerMixin from importer.registry import register_importer from InvenTree.helpers import get_objectreference from InvenTree.helpers_model import construct_absolute_url +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.serializers import ( InvenTreeAttachmentSerializerField, InvenTreeImageSerializerField, @@ -757,7 +757,7 @@ class SelectionListSerializer(InvenTreeModelSerializer): def update(self, instance, validated_data): """Update an existing selection list. Save the choices separately.""" inst_mapping = {inst.id: inst for inst in instance.entries.all()} - exsising_ids = {a.get('id') for a in self._choices_validated} + existing_ids = {a.get('id') for a in self._choices_validated} # Perform creations and updates. ret = [] @@ -772,7 +772,7 @@ class SelectionListSerializer(InvenTreeModelSerializer): ret.append(SelectionEntrySerializer().update(inst, data)) # Perform deletions. - for entry_id in inst_mapping.keys() - exsising_ids: + for entry_id in inst_mapping.keys() - existing_ids: inst_mapping[entry_id].delete() return super().update(instance, validated_data) @@ -783,3 +783,32 @@ class SelectionListSerializer(InvenTreeModelSerializer): if self.instance and self.instance.locked: raise serializers.ValidationError({'locked': _('Selection list is locked')}) return ret + + +class DataOutputSerializer(InvenTreeModelSerializer): + """Serializer for the DataOutput model.""" + + class Meta: + """Meta options for DataOutputSerializer.""" + + model = common_models.DataOutput + fields = [ + 'pk', + 'created', + 'user', + 'user_detail', + 'total', + 'progress', + 'complete', + 'output_type', + 'template_name', + 'plugin', + 'output', + 'errors', + ] + + user_detail = UserSerializer(source='user', read_only=True, many=False) + + output = InvenTreeAttachmentSerializerField( + required=False, allow_null=True, read_only=True + ) diff --git a/src/backend/InvenTree/common/setting/system.py b/src/backend/InvenTree/common/setting/system.py index 1f08450a1e..ff54f94714 100644 --- a/src/backend/InvenTree/common/setting/system.py +++ b/src/backend/InvenTree/common/setting/system.py @@ -110,9 +110,9 @@ def reload_plugin_registry(setting): def barcode_plugins() -> list: """Return a list of plugin choices which can be used for barcode generation.""" try: - from plugin import registry + from plugin import PluginMixinEnum, registry - plugins = registry.with_mixin('barcode', active=True) + plugins = registry.with_mixin(PluginMixinEnum.BARCODE, active=True) except Exception: # pragma: no cover plugins = [] diff --git a/src/backend/InvenTree/common/setting/user.py b/src/backend/InvenTree/common/setting/user.py index 6581cacc22..7f648fe514 100644 --- a/src/backend/InvenTree/common/setting/user.py +++ b/src/backend/InvenTree/common/setting/user.py @@ -4,13 +4,13 @@ from django.core.validators import MinValueValidator from django.utils.translation import gettext_lazy as _ from common.setting.type import InvenTreeSettingsKeyType -from plugin import registry +from plugin import PluginMixinEnum, registry def label_printer_options(): """Build a list of available label printer options.""" printers = [] - label_printer_plugins = registry.with_mixin('labels') + label_printer_plugins = registry.with_mixin(PluginMixinEnum.LABELS) if label_printer_plugins: printers.extend([ (p.slug, p.name + ' - ' + p.human_name) for p in label_printer_plugins diff --git a/src/backend/InvenTree/common/tasks.py b/src/backend/InvenTree/common/tasks.py index 616001cd25..69487d5c6d 100644 --- a/src/backend/InvenTree/common/tasks.py +++ b/src/backend/InvenTree/common/tasks.py @@ -12,6 +12,7 @@ import feedparser import requests import structlog +import common.models import InvenTree.helpers from InvenTree.helpers_model import getModelsWithMixin from InvenTree.models import InvenTreeNotesMixin @@ -20,6 +21,17 @@ from InvenTree.tasks import ScheduledTask, scheduled_task logger = structlog.get_logger('inventree') +@scheduled_task(ScheduledTask.DAILY) +def cleanup_old_data_outputs(): + """Remove old data outputs from the database.""" + # Remove any outputs which are older than 5 days + # Note: Remove them individually, to ensure that the files are removed too + threshold = InvenTree.helpers.current_date() - timedelta(days=5) + + for output in common.models.DataOutput.objects.filter(created__lte=threshold): + output.delete() + + @scheduled_task(ScheduledTask.DAILY) def delete_old_notifications(): """Remove old notifications from the database. diff --git a/src/backend/InvenTree/company/api.py b/src/backend/InvenTree/company/api.py index 9ff22f6e8a..29810e2dca 100644 --- a/src/backend/InvenTree/company/api.py +++ b/src/backend/InvenTree/company/api.py @@ -7,7 +7,7 @@ from django.utils.translation import gettext_lazy as _ from django_filters import rest_framework as rest_filters import part.models -from importer.mixins import DataExportViewMixin +from data_exporter.mixins import DataExportViewMixin from InvenTree.api import ListCreateDestroyAPIView, MetadataView from InvenTree.filters import SEARCH_ORDER_FILTER, SEARCH_ORDER_FILTER_ALIAS from InvenTree.helpers import str2bool @@ -176,7 +176,7 @@ class ManufacturerPartList(DataExportViewMixin, ListCreateDestroyAPIView): kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) filter_backends = SEARCH_ORDER_FILTER @@ -245,7 +245,7 @@ class ManufacturerPartParameterList(ListCreateDestroyAPIView): kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) filter_backends = SEARCH_ORDER_FILTER @@ -355,7 +355,7 @@ class SupplierPartMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class SupplierPartList( @@ -467,7 +467,7 @@ class SupplierPriceBreakList(ListCreateAPI): kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) filter_backends = SEARCH_ORDER_FILTER_ALIAS diff --git a/src/backend/InvenTree/company/serializers.py b/src/backend/InvenTree/company/serializers.py index dd8ecc673e..3214595901 100644 --- a/src/backend/InvenTree/company/serializers.py +++ b/src/backend/InvenTree/company/serializers.py @@ -12,8 +12,8 @@ from taggit.serializers import TagListSerializerField import company.filters import part.filters import part.serializers as part_serializers -from importer.mixins import DataImportExportSerializerMixin from importer.registry import register_importer +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.ready import isGeneratingSchema from InvenTree.serializers import ( InvenTreeCurrencySerializer, diff --git a/src/backend/InvenTree/data_exporter/__init__.py b/src/backend/InvenTree/data_exporter/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/backend/InvenTree/data_exporter/apps.py b/src/backend/InvenTree/data_exporter/apps.py new file mode 100644 index 0000000000..a9eb5d8bbd --- /dev/null +++ b/src/backend/InvenTree/data_exporter/apps.py @@ -0,0 +1,25 @@ +"""Config options for the exporter app.""" + +from django.apps import AppConfig + + +class DataExporterConfig(AppConfig): + """Configuration class for the 'exporter' app.""" + + default_auto_field = 'django.db.models.BigAutoField' + name = 'data_exporter' + + def ready(self): + """Run any code that needs to be executed when the app is loaded.""" + super().ready() + + self.cleanup() + + def cleanup(self): + """Cleanup any old export files.""" + try: + from data_exporter.tasks import cleanup_old_export_outputs + + cleanup_old_export_outputs() + except Exception: + pass diff --git a/src/backend/InvenTree/data_exporter/migrations/__init__.py b/src/backend/InvenTree/data_exporter/migrations/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/backend/InvenTree/data_exporter/mixins.py b/src/backend/InvenTree/data_exporter/mixins.py new file mode 100644 index 0000000000..73df81906d --- /dev/null +++ b/src/backend/InvenTree/data_exporter/mixins.py @@ -0,0 +1,467 @@ +"""Mixin classes for the exporter app.""" + +from collections import OrderedDict + +from django.core.exceptions import ValidationError +from django.core.files.base import ContentFile +from django.utils.translation import gettext_lazy as _ + +import structlog +import tablib +from rest_framework import serializers +from rest_framework.response import Response +from taggit.serializers import TagListSerializerField + +import data_exporter.serializers +import data_exporter.tasks +import InvenTree.exceptions +from common.models import DataOutput +from InvenTree.helpers import str2bool +from InvenTree.tasks import offload_task +from plugin import PluginMixinEnum, registry + +logger = structlog.get_logger('inventree') + + +class DataExportSerializerMixin: + """Mixin class for adding data export functionality to a DRF serializer. + + Provides generic functionality to take the output of a serializer and export it to a file. + + Attributes: + export_only_fields: List of field names which are only used during data export + export_exclude_fields: List of field names which are excluded during data export + export_child_fields: List of child fields which are exported (using dot notation) + """ + + export_only_fields = [] + export_exclude_fields = [] + export_child_fields = [] + + def get_export_only_fields(self, **kwargs) -> list: + """Return the list of field names which are only used during data export.""" + return self.export_only_fields + + def get_export_exclude_fields(self, **kwargs) -> list: + """Return the list of field names which are excluded during data export.""" + return self.export_exclude_fields + + def __init__(self, *args, **kwargs): + """Initialise the DataExportSerializerMixin. + + Determine if the serializer is being used for data export, + and if so, adjust the serializer fields accordingly. + """ + exporting = kwargs.pop('exporting', False) + + super().__init__(*args, **kwargs) + + # Cache the request object + self.request = self.context.get('request') + + if exporting: + # Exclude fields which are not required for data export + for field in self.get_export_exclude_fields(**kwargs): + self.fields.pop(field, None) + else: + # Exclude fields which are only used for data export + for field in self.get_export_only_fields(**kwargs): + self.fields.pop(field, None) + + def get_exportable_fields(self) -> dict: + """Return a dict of fields which can be exported against this serializer instance. + + Note: Any fields which should be excluded from export have already been removed + + Returns: + dict: A dictionary of field names and field objects + """ + fields = {} + + if meta := getattr(self, 'Meta', None): + write_only_fields = getattr(meta, 'write_only_fields', []) + else: + write_only_fields = [] + + for name, field in self.fields.items(): + # Skip write-only fields + if getattr(field, 'write_only', False) or name in write_only_fields: + continue + + # Skip tags fields + # TODO: Implement tag field export support + if issubclass(field.__class__, TagListSerializerField): + continue + + # Top-level serializer fields can be exported with dot notation + if issubclass(field.__class__, serializers.Serializer): + fields.update(self.get_child_fields(name, field)) + continue + + fields[name] = field + + return fields + + def get_child_fields(self, field_name: str, field) -> dict: + """Return a dictionary of child fields for a given field. + + Only child fields which match the 'export_child_fields' list will be returned. + """ + child_fields = {} + + if sub_fields := getattr(field, 'fields', None): + for sub_name, sub_field in sub_fields.items(): + name = f'{field_name}.{sub_name}' + + if name in self.export_child_fields: + sub_field.parent_field = field + child_fields[name] = sub_field + + return child_fields + + @classmethod + def arrange_export_headers(cls, headers: list) -> list: + """Optional method to arrange the export headers. + + By default, the headers are returned in the order they are provided. + """ + return headers + + def get_nested_value(self, row: dict, key: str) -> any: + """Get a nested value from a dictionary. + + This method allows for dot notation to access nested fields. + + Arguments: + row: The dictionary to extract the value from + key: The key to extract + + Returns: + any: The extracted value + """ + keys = key.split('.') + + value = row + + for key in keys: + if not value: + break + + if not key: + continue + + value = value.get(key, None) + + return value + + def generate_headers(self) -> OrderedDict: + """Generate a list of default headers for the exported data. + + Returns an ordered dict of field names and their corresponding human-readable labels. + + e.g. + + { + 'id': 'ID', + 'name': 'Name', + ... + } + + """ + fields = self.get_exportable_fields() + field_names = self.arrange_export_headers(list(fields.keys())) + + headers = OrderedDict() + + for field_name in field_names: + field = fields[field_name] + + label = getattr(field, 'label', field_name) + + if parent := getattr(field, 'parent_field', None): + label = f'{parent.label}.{label}' + + headers[field_name] = label + + return headers + + def export_to_file(self, data, headers: OrderedDict, file_format): + """Export the queryset to a file in the specified format. + + Arguments: + queryset: The queryset to export + data: The serialized dataset to export + headers: The headers to use for the exported data {field: label} + file_format: The file format to export to + + Returns: + File object containing the exported data + """ + field_names = list(headers.keys()) + field_headers = list(headers.values()) + + # Create a new dataset with the provided header labels + dataset = tablib.Dataset(headers=field_headers) + + for row in data: + dataset.append([self.get_nested_value(row, f) for f in field_names]) + + return dataset.export(file_format) + + +class DataExportViewMixin: + """An API view mixin for directly exporting selected data. + + To perform a data export against an API endpoint which inherits from this mixin, + perform a GET request with 'export=True'. + + This will run validation against the DataExportOptionsSerializer. + + Once the export options have been validated, a new DataOutput object will be created, + and this will be returned to the client (including a download link to the exported file). + """ + + def is_exporting(self) -> bool: + """Determine if the view is currently exporting data.""" + if request := getattr(self, 'request', None): + return str2bool( + request.data.get('export') or request.query_params.get('export') + ) + + return False + + def get_plugin(self, plugin_slug=None): + """Return the plugin instance associated with the export request. + + Arguments: + plugin_slug: The slug of the plugin to use for exporting the data (optional) + """ + PLUGIN_KEY = 'export_plugin' + + if not plugin_slug: + if request := getattr(self, 'request', None): + plugin_slug = request.data.get(PLUGIN_KEY) or request.query_params.get( + PLUGIN_KEY + ) + + if plugin_slug: + return registry.get_plugin( + plugin_slug, active=True, with_mixin=PluginMixinEnum.EXPORTER + ) + + return None + + def get_serializer(self, *args, **kwargs): + """Return the serializer instance for the view. + + - Only applies for OPTIONS or GET requests + - OPTIONS requests to determine plugin serializer options + - GET request to perform the data export + - If the view is exporting data, return the DataExportOptionsSerializer. + - Otherwise, return the default serializer. + """ + exporting = kwargs.pop('exporting', None) + + if exporting is None: + exporting = ( + self.request.method.lower() in ['options', 'get'] + and self.is_exporting() + ) + + if exporting: + # Override kwargs when initializing the DataExportOptionsSerializer + export_kwargs = { + 'plugin': self.get_plugin(), + 'request': self.request, + 'data': kwargs.get('data'), + 'context': kwargs.get('context'), + } + + # Get the base model associated with this view + try: + serializer_class = self.get_serializer_class() + export_kwargs['model_class'] = serializer_class.Meta.model + except AttributeError: + export_kwargs['model_class'] = None + + return data_exporter.serializers.DataExportOptionsSerializer( + *args, **export_kwargs + ) + else: + return super().get_serializer(*args, **kwargs) + + def export_data( + self, + export_plugin, + export_format: str, + export_context: dict, + output: DataOutput, + ): + """Export the data in the specified format. + + Arguments: + export_plugin: The plugin instance to use for exporting the data + export_format: The file format to export the data in + export_context: Additional context data to pass to the plugin + output: The DataOutput object to write to + + - By default, uses the provided serializer to generate the data, and return it as a file download. + - If a plugin is specified, the plugin can be used to augment or replace the export functionality. + """ + # Get the base serializer class for the view + serializer_class = self.get_serializer_class() + + if not issubclass(serializer_class, DataExportSerializerMixin): + raise ValidationError( + 'Serializer class must inherit from DataExportSerializerMixin' + ) + + export_error = _('Error occurred during data export') + + context = self.get_serializer_context() + + # Perform initial filtering of the queryset, based on the query parameters + queryset = self.filter_queryset(self.get_queryset()) + + # Perform additional filtering, as per the provided plugin + try: + queryset = export_plugin.filter_queryset(queryset) + except Exception: + InvenTree.exceptions.log_error( + f'plugins.{export_plugin.slug}.filter_queryset' + ) + raise ValidationError(export_error) + + # Update the output instance with the total number of items to export + output.total = queryset.count() + output.save() + + data = None + serializer = serializer_class(context=context, exporting=True) + serializer.initial_data = queryset + + # Construct 'default' headers (note: may be overridden by plugin) + headers = serializer.generate_headers() + + # Generate a filename for the exported data (implemented by the plugin) + try: + filename = export_plugin.generate_filename( + serializer_class.Meta.model, export_format + ) + except Exception: + InvenTree.exceptions.log_error( + f'plugins.{export_plugin.slug}.generate_filename' + ) + raise ValidationError(export_error) + + # The provided plugin is responsible for exporting the data + # The returned data *must* be a list of dict objects + try: + data = export_plugin.export_data( + queryset, serializer_class, headers, export_context, output + ) + + except Exception: + InvenTree.exceptions.log_error(f'plugins.{export_plugin.slug}.export_data') + raise ValidationError(export_error) + + if not isinstance(data, list): + raise ValidationError( + _('Data export plugin returned incorrect data format') + ) + + # Augment / update the headers (if required) + if hasattr(export_plugin, 'update_headers'): + try: + headers = export_plugin.update_headers(headers, export_context) + except Exception: + InvenTree.exceptions.log_error( + f'plugins.{export_plugin.slug}.update_headers' + ) + raise ValidationError(export_error) + + # Now, export the data to file + try: + datafile = serializer.export_to_file(data, headers, export_format) + except Exception: + InvenTree.exceptions.log_error('export_to_file') + raise ValidationError(_('Error occurred during data export')) + + # Update the output object with the exported data + output.progress = 100 + output.complete = True + output.output = ContentFile(datafile, filename) + output.save() + + def get(self, request, *args, **kwargs): + """Override the GET method to determine export options.""" + from common.serializers import DataOutputSerializer + + # If we are not exporting data, return the default response + if self.is_exporting(): + # Determine if the export options are valid + + # Extract the export options from the provided query parameters + export_options = {} + + for key in request.query_params: + if key.startswith('export_'): + export_options[key] = request.query_params.get(key) + + # Construct the options serializer with the provided data + serializer = self.get_serializer(exporting=True, data=export_options) + + serializer.is_valid(raise_exception=True) + serializer_data = serializer.validated_data + + export_format = serializer_data.pop('export_format', 'csv') + plugin_slug = serializer_data.pop('export_plugin', 'inventree-exporter') + export_plugin = self.get_plugin(plugin_slug) + + export_context = {} + + # Also run the data against the plugin serializer + if export_plugin: + if hasattr(export_plugin, 'get_export_options_serializer'): + if plugin_serializer := export_plugin.get_export_options_serializer( + data=export_options + ): + plugin_serializer.is_valid(raise_exception=True) + export_context = plugin_serializer.validated_data + + user = getattr(request, 'user', None) + + # Add in extra context data for the plugin + export_context['user'] = user + + # Create an output object to export against + output = DataOutput.objects.create( + user=user if user and user.is_authenticated else None, + total=0, # Note: this should get updated by the export task + progress=0, + complete=False, + output_type=DataOutput.DataOutputTypes.EXPORT, + plugin=export_plugin.slug, + output=None, + ) + + # Offload the export task to a background worker + # This is to avoid blocking the web server + # Note: The export task will loop back and call the 'export_data' method on this class + offload_task( + data_exporter.tasks.export_data, + self.__class__, + request.user.id, + request.query_params, + plugin_slug, + export_format, + export_context, + output.id, + ) + + output.refresh_from_db() + + # Return a response to the frontend + return Response(DataOutputSerializer(output).data, status=200) + + return super().get(request, *args, **kwargs) diff --git a/src/backend/InvenTree/data_exporter/serializers.py b/src/backend/InvenTree/data_exporter/serializers.py new file mode 100644 index 0000000000..214ab24973 --- /dev/null +++ b/src/backend/InvenTree/data_exporter/serializers.py @@ -0,0 +1,69 @@ +"""Serializers for the exporter app.""" + +from django.utils.translation import gettext_lazy as _ + +from rest_framework import serializers + +import InvenTree.helpers +import InvenTree.serializers +from plugin import PluginMixinEnum, registry + + +class DataExportOptionsSerializer(serializers.Serializer): + """Serializer class for defining a data export session.""" + + class Meta: + """Metaclass options for this serializer.""" + + fields = ['export_format', 'export_plugin'] + + def __init__(self, *args, **kwargs): + """Construct the DataExportOptionsSerializer. + + - The exact nature of the available fields depends on which plugin is selected. + - The selected plugin may 'extend' the fields available in the serializer. + """ + # Reset fields to a known state + self.Meta.fields = ['export_format', 'export_plugin'] + + # Generate a list of plugins to choose from + # If a model type is provided, use this to filter the list of plugins + model_class = kwargs.pop('model_class', None) + request = kwargs.pop('request', None) + + # Is a plugin serializer provided? + if plugin := kwargs.pop('plugin', None): + if hasattr(plugin, 'get_export_options_serializer'): + plugin_serializer = plugin.get_export_options_serializer() + + if plugin_serializer: + for key, field in plugin_serializer.fields.items(): + # Note: Custom fields *must* start with 'export_' prefix + if key.startswith('export_') and key not in self.Meta.fields: + self.Meta.fields.append(key) + setattr(self, key, field) + + plugin_options = [] + + for plugin in registry.with_mixin(PluginMixinEnum.EXPORTER): + if plugin.supports_export(model_class, request.user): + plugin_options.append((plugin.slug, plugin.name)) + + self.fields['export_plugin'].choices = plugin_options + + super().__init__(*args, **kwargs) + + export_format = serializers.ChoiceField( + choices=InvenTree.helpers.GetExportOptions(), + default='csv', + label=_('Export Format'), + help_text=_('Select export file format'), + ) + + # Select plugin for export - the options will be dynamically generated later on + export_plugin = serializers.ChoiceField( + choices=[], + default='inventree-exporter', + label=_('Export Plugin'), + help_text=_('Select export plugin'), + ) diff --git a/src/backend/InvenTree/data_exporter/tasks.py b/src/backend/InvenTree/data_exporter/tasks.py new file mode 100644 index 0000000000..6895895c19 --- /dev/null +++ b/src/backend/InvenTree/data_exporter/tasks.py @@ -0,0 +1,62 @@ +"""Background tasks for the exporting app.""" + +from django.contrib.auth.models import User +from django.test.client import RequestFactory + +import structlog + +from common.models import DataOutput + +logger = structlog.get_logger('inventree') + + +def export_data( + view_class, + user_id: int, + query_params: dict, + plugin_key: str, + export_format: str, + export_context: dict, + output_id: int, +): + """Perform the data export task using the provided parameters. + + Arguments: + view_class: The class of the view to export data from + user_id: The ID of the user who requested the export + query_params: Query parameters for the export + plugin_key: The key for the export plugin + export_format: The output format for the export + export_context: Additional options for the export + output_id: The ID of the DataOutput instance to write to + + This function is designed to be called by the background task, + to avoid blocking the web server. + """ + from plugin import registry + + if (plugin := registry.get_plugin(plugin_key)) is None: + logger.warning("export_data: Plugin '%s' not found", plugin_key) + return + + if (user := User.objects.filter(pk=user_id).first()) is None: + logger.warning('export_data: User not found: %d', user_id) + return + + if (output := DataOutput.objects.filter(pk=output_id).first()) is None: + logger.warning('export_data: Output object not found: %d', output_id) + return + + # Recreate the request object - this is required for the view to function correctly + # Note that the request object cannot be pickled, so we need to recreate it here + request = RequestFactory() + request.user = user + request.query_params = query_params + + view = view_class() + view.request = request + view.args = getattr(view, 'args', ()) + view.kwargs = getattr(view, 'kwargs', {}) + view.format_kwarg = getattr(view, 'format_kwarg', None) + + view.export_data(plugin, export_format, export_context, output) diff --git a/src/backend/InvenTree/generic/events.py b/src/backend/InvenTree/generic/events.py index f1b5235e7c..496cf12ee5 100644 --- a/src/backend/InvenTree/generic/events.py +++ b/src/backend/InvenTree/generic/events.py @@ -1,4 +1,4 @@ -"""Generic event enumerations for InevnTree.""" +"""Generic event enumerations for InvenTree.""" import enum diff --git a/src/backend/InvenTree/generic/states/api.py b/src/backend/InvenTree/generic/states/api.py index 80016152ea..7eaec223a1 100644 --- a/src/backend/InvenTree/generic/states/api.py +++ b/src/backend/InvenTree/generic/states/api.py @@ -11,7 +11,7 @@ from rest_framework.response import Response import common.models import common.serializers -from importer.mixins import DataExportViewMixin +from data_exporter.mixins import DataExportViewMixin from InvenTree.filters import SEARCH_ORDER_FILTER from InvenTree.mixins import ListCreateAPI, RetrieveUpdateDestroyAPI from InvenTree.permissions import IsStaffOrReadOnly diff --git a/src/backend/InvenTree/importer/mixins.py b/src/backend/InvenTree/importer/mixins.py index 23f7b6ce1b..1a36982ed6 100644 --- a/src/backend/InvenTree/importer/mixins.py +++ b/src/backend/InvenTree/importer/mixins.py @@ -1,14 +1,8 @@ """Mixin classes for data import/export functionality.""" -from django.core.exceptions import ValidationError -from django.utils.translation import gettext_lazy as _ - -import tablib from rest_framework import fields, serializers from taggit.serializers import TagListSerializerField -from InvenTree.helpers import DownloadFile, GetExportFormats, current_date - class DataImportSerializerMixin: """Mixin class for adding data import functionality to a DRF serializer.""" @@ -89,242 +83,3 @@ class DataImportSerializerMixin: importable_fields[name] = field return importable_fields - - -class DataExportSerializerMixin: - """Mixin class for adding data export functionality to a DRF serializer. - - Attributes: - export_only_fields: List of field names which are only used during data export - export_exclude_fields: List of field names which are excluded during data export - export_child_fields: List of child fields which are exported (using dot notation) - """ - - export_only_fields = [] - export_exclude_fields = [] - export_child_fields = [] - - def get_export_only_fields(self, **kwargs) -> list: - """Return the list of field names which are only used during data export.""" - return self.export_only_fields - - def get_export_exclude_fields(self, **kwargs) -> list: - """Return the list of field names which are excluded during data export.""" - return self.export_exclude_fields - - def __init__(self, *args, **kwargs): - """Initialise the DataExportSerializerMixin. - - Determine if the serializer is being used for data export, - and if so, adjust the serializer fields accordingly. - """ - exporting = kwargs.pop('exporting', False) - - super().__init__(*args, **kwargs) - - if exporting: - # Exclude fields which are not required for data export - for field in self.get_export_exclude_fields(**kwargs): - self.fields.pop(field, None) - else: - # Exclude fields which are only used for data export - for field in self.get_export_only_fields(**kwargs): - self.fields.pop(field, None) - - def get_exportable_fields(self) -> dict: - """Return a dict of fields which can be exported against this serializer instance. - - Note: Any fields which should be excluded from export have already been removed - - Returns: - dict: A dictionary of field names and field objects - """ - fields = {} - - if meta := getattr(self, 'Meta', None): - write_only_fields = getattr(meta, 'write_only_fields', []) - else: - write_only_fields = [] - - for name, field in self.fields.items(): - # Skip write-only fields - if getattr(field, 'write_only', False) or name in write_only_fields: - continue - - # Skip tags fields - # TODO: Implement tag field export support - if issubclass(field.__class__, TagListSerializerField): - continue - - # Top-level serializer fields can be exported with dot notation - # Skip fields which are themselves serializers - if issubclass(field.__class__, serializers.Serializer): - fields.update(self.get_child_fields(name, field)) - continue - - fields[name] = field - - return fields - - def get_child_fields(self, field_name: str, field) -> dict: - """Return a dictionary of child fields for a given field. - - Only child fields which match the 'export_child_fields' list will be returned. - """ - child_fields = {} - - if sub_fields := getattr(field, 'fields', None): - for sub_name, sub_field in sub_fields.items(): - name = f'{field_name}.{sub_name}' - - if name in self.export_child_fields: - sub_field.parent_field = field - child_fields[name] = sub_field - - return child_fields - - def get_exported_filename(self, export_format) -> str: - """Return the filename for the exported data file. - - An implementing class can override this implementation if required. - - Arguments: - export_format: The file format to be exported - - Returns: - str: The filename for the exported file - """ - model = self.Meta.model - date = current_date().isoformat() - - return f'InvenTree_{model.__name__}_{date}.{export_format}' - - @classmethod - def arrange_export_headers(cls, headers: list) -> list: - """Optional method to arrange the export headers.""" - return headers - - def get_nested_value(self, row: dict, key: str) -> any: - """Get a nested value from a dictionary. - - This method allows for dot notation to access nested fields. - - Arguments: - row: The dictionary to extract the value from - key: The key to extract - - Returns: - any: The extracted value - """ - keys = key.split('.') - - value = row - - for key in keys: - if not value: - break - - if not key: - continue - - value = value.get(key, None) - - return value - - def process_row(self, row): - """Optional method to process a row before exporting it.""" - return row - - def export_to_file(self, data, file_format): - """Export the queryset to a file in the specified format. - - Arguments: - queryset: The queryset to export - data: The serialized dataset to export - file_format: The file format to export to - - Returns: - File object containing the exported data - """ - # Extract all exportable fields from this serializer - fields = self.get_exportable_fields() - - field_names = self.arrange_export_headers(list(fields.keys())) - - # Extract human-readable field names - headers = [] - - for field_name, field in fields.items(): - field = fields[field_name] - - label = getattr(field, 'label', field_name) - - if parent := getattr(field, 'parent_field', None): - label = f'{parent.label}.{label}' - - headers.append(label) - - dataset = tablib.Dataset(headers=headers) - - for row in data: - row = self.process_row(row) - dataset.append([self.get_nested_value(row, f) for f in field_names]) - - return dataset.export(file_format) - - -class DataImportExportSerializerMixin( - DataImportSerializerMixin, DataExportSerializerMixin -): - """Mixin class for adding data import/export functionality to a DRF serializer.""" - - -class DataExportViewMixin: - """Mixin class for exporting a dataset via the API. - - Adding this mixin to an API view allows the user to export the dataset to file in a variety of formats. - - We achieve this by overriding the 'get' method, and checking for the presence of the required query parameter. - """ - - EXPORT_QUERY_PARAMETER = 'export' - - def export_data(self, export_format): - """Export the data in the specified format. - - Use the provided serializer to generate the data, and return it as a file download. - """ - serializer_class = self.get_serializer_class() - - if not issubclass(serializer_class, DataExportSerializerMixin): - raise TypeError( - 'Serializer class must inherit from DataExportSerialierMixin' - ) - - queryset = self.filter_queryset(self.get_queryset()) - - serializer = serializer_class(exporting=True) - serializer.initial_data = queryset - - # Export dataset with a second copy of the serializer - # This is because when we pass many=True, the returned class is a ListSerializer - data = serializer_class(queryset, many=True, exporting=True).data - - filename = serializer.get_exported_filename(export_format) - datafile = serializer.export_to_file(data, export_format) - - return DownloadFile(datafile, filename=filename) - - def get(self, request, *args, **kwargs): - """Override the 'get' method to check for the export query parameter.""" - if export_format := request.query_params.get(self.EXPORT_QUERY_PARAMETER, None): - export_format = str(export_format).strip().lower() - if export_format in GetExportFormats(): - return self.export_data(export_format) - else: - raise ValidationError({ - self.EXPORT_QUERY_PARAMETER: _('Invalid export format') - }) - - # If the export query parameter is not present, return the default response - return super().get(request, *args, **kwargs) diff --git a/src/backend/InvenTree/order/api.py b/src/backend/InvenTree/order/api.py index 0c2130a0d4..11e31cca0d 100644 --- a/src/backend/InvenTree/order/api.py +++ b/src/backend/InvenTree/order/api.py @@ -19,8 +19,8 @@ from rest_framework.response import Response import common.models import common.settings import company.models +from data_exporter.mixins import DataExportViewMixin from generic.states.api import StatusView -from importer.mixins import DataExportViewMixin from InvenTree.api import BulkUpdateMixin, ListCreateDestroyAPIView, MetadataView from InvenTree.filters import ( SEARCH_ORDER_FILTER, @@ -57,7 +57,7 @@ class GeneralExtraLineList(DataExportViewMixin): kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return the annotated queryset for this endpoint.""" @@ -336,7 +336,7 @@ class PurchaseOrderMixin: # Ensure the request context is passed through kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return the annotated queryset for this endpoint.""" @@ -594,7 +594,7 @@ class PurchaseOrderLineItemMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def perform_update(self, serializer): """Override the perform_update method to auto-update pricing if required.""" @@ -786,7 +786,7 @@ class SalesOrderMixin: # Ensure the context is passed through to the serializer kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return annotated queryset for this endpoint.""" @@ -942,7 +942,7 @@ class SalesOrderLineItemMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return annotated queryset for this endpoint.""" @@ -1221,7 +1221,7 @@ class SalesOrderAllocationList(SalesOrderAllocationMixin, BulkUpdateMixin, ListA except AttributeError: pass - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class SalesOrderAllocationDetail(SalesOrderAllocationMixin, RetrieveUpdateDestroyAPI): @@ -1385,7 +1385,7 @@ class ReturnOrderMixin: # Ensure the context is passed through to the serializer kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return annotated queryset for this endpoint.""" @@ -1536,7 +1536,7 @@ class ReturnOrderLineItemMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return annotated queryset for this endpoint.""" diff --git a/src/backend/InvenTree/order/serializers.py b/src/backend/InvenTree/order/serializers.py index 36f689ae89..36cea1e3ee 100644 --- a/src/backend/InvenTree/order/serializers.py +++ b/src/backend/InvenTree/order/serializers.py @@ -35,7 +35,6 @@ from company.serializers import ( SupplierPartSerializer, ) from generic.states.fields import InvenTreeCustomStatusSerializerMixin -from importer.mixins import DataImportExportSerializerMixin from importer.registry import register_importer from InvenTree.helpers import ( current_date, @@ -44,6 +43,7 @@ from InvenTree.helpers import ( normalize, str2bool, ) +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.ready import isGeneratingSchema from InvenTree.serializers import ( InvenTreeCurrencySerializer, diff --git a/src/backend/InvenTree/order/test_api.py b/src/backend/InvenTree/order/test_api.py index e0148b72f1..9057a24aa0 100644 --- a/src/backend/InvenTree/order/test_api.py +++ b/src/backend/InvenTree/order/test_api.py @@ -870,14 +870,13 @@ class PurchaseOrderDownloadTest(OrderTest): """Incorrect format should default raise an error.""" url = reverse('api-po-list') - with self.assertRaises(ValueError): - self.download_file(url, {'export': 'xyz'}) + response = self.export_data(url, export_format='xyz', expected_code=400) + self.assertIn('is not a valid choice', str(response['export_format'])) def test_download_csv(self): """Download PurchaseOrder data as .csv.""" - with self.download_file( + with self.export_data( reverse('api-po-list'), - {'export': 'csv'}, expected_code=200, expected_fn=r'InvenTree_PurchaseOrder_.+\.csv', ) as file: @@ -896,12 +895,12 @@ class PurchaseOrderDownloadTest(OrderTest): def test_download_line_items(self): """Test that the PurchaseOrderLineItems can be downloaded to a file.""" - with self.download_file( + with self.export_data( reverse('api-po-line-list'), - {'export': 'xlsx'}, - decode=False, + export_format='xlsx', expected_code=200, expected_fn=r'InvenTree_PurchaseOrderLineItem.+\.xlsx', + decode=False, ) as file: self.assertIsInstance(file, io.BytesIO) @@ -1602,9 +1601,9 @@ class SalesOrderTest(OrderTest): # Download file, check we get a 200 response for fmt in ['csv', 'xlsx', 'tsv']: - self.download_file( + self.export_data( reverse('api-so-list'), - {'export': fmt}, + export_format=fmt, decode=fmt == 'csv', expected_code=200, expected_fn=r'InvenTree_SalesOrder_.+', @@ -1856,16 +1855,16 @@ class SalesOrderDownloadTest(OrderTest): """Test that downloading without the 'export' option fails.""" url = reverse('api-so-list') - with self.assertRaises(ValueError): - self.download_file(url, {}, expected_code=200) + response = self.export_data(url, export_plugin='no-plugin', expected_code=400) + self.assertIn('is not a valid choice', str(response['export_plugin'])) def test_download_xlsx(self): """Test xlsx file download.""" url = reverse('api-so-list') # Download .xls file - with self.download_file( - url, {'export': 'xlsx'}, expected_code=200, decode=False + with self.export_data( + url, export_format='xlsx', expected_code=200, decode=False ) as file: self.assertIsInstance(file, io.BytesIO) @@ -1888,9 +1887,7 @@ class SalesOrderDownloadTest(OrderTest): excluded_cols = ['metadata'] # Download .xls file - with self.download_file( - url, {'export': 'csv'}, expected_code=200, decode=True - ) as file: + with self.export_data(url, export_format='csv') as file: data = self.process_csv( file, required_cols=required_cols, @@ -1905,9 +1902,7 @@ class SalesOrderDownloadTest(OrderTest): self.assertEqual(line['Order Status'], str(order.status)) # Download only outstanding sales orders - with self.download_file( - url, {'export': 'tsv', 'outstanding': True}, expected_code=200, decode=True - ) as file: + with self.export_data(url, {'outstanding': True}, export_format='tsv') as file: self.process_csv( file, required_cols=required_cols, diff --git a/src/backend/InvenTree/part/api.py b/src/backend/InvenTree/part/api.py index f50a30f8b5..3faaccddfe 100644 --- a/src/backend/InvenTree/part/api.py +++ b/src/backend/InvenTree/part/api.py @@ -19,7 +19,7 @@ import order.models import part.filters from build.models import Build, BuildItem from build.status_codes import BuildStatusGroups -from importer.mixins import DataExportViewMixin +from data_exporter.mixins import DataExportViewMixin from InvenTree.api import BulkUpdateMixin, ListCreateDestroyAPIView, MetadataView from InvenTree.filters import ( ORDER_FILTER, @@ -69,6 +69,17 @@ class CategoryMixin: serializer_class = part_serializers.CategorySerializer queryset = PartCategory.objects.all() + def get_serializer(self, *args, **kwargs): + """Add additional context based on query parameters.""" + try: + params = self.request.query_params + + kwargs['path_detail'] = str2bool(params.get('path_detail', False)) + except AttributeError: + pass + + return super().get_serializer(*args, **kwargs) + def get_queryset(self, *args, **kwargs): """Return an annotated queryset for the CategoryDetail endpoint.""" queryset = super().get_queryset(*args, **kwargs) @@ -248,19 +259,6 @@ class CategoryList(CategoryMixin, BulkUpdateMixin, DataExportViewMixin, ListCrea class CategoryDetail(CategoryMixin, CustomRetrieveUpdateDestroyAPI): """API endpoint for detail view of a single PartCategory object.""" - def get_serializer(self, *args, **kwargs): - """Add additional context based on query parameters.""" - try: - params = self.request.query_params - - kwargs['path_detail'] = str2bool(params.get('path_detail', False)) - except AttributeError: - pass - - kwargs.setdefault('context', self.get_serializer_context()) - - return self.serializer_class(*args, **kwargs) - def update(self, request, *args, **kwargs): """Perform 'update' function and mark this part as 'starred' (or not).""" # Clean up input data @@ -1229,7 +1227,7 @@ class PartMixin: except AttributeError: pass - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_serializer_context(self): """Extend serializer context data.""" @@ -1600,7 +1598,7 @@ class PartParameterAPIMixin: except AttributeError: pass - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class PartParameterFilter(rest_filters.FilterSet): @@ -1861,7 +1859,7 @@ class BomMixin: # Ensure the request context is passed through! kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return the queryset object for this endpoint.""" diff --git a/src/backend/InvenTree/part/bom.py b/src/backend/InvenTree/part/bom.py deleted file mode 100644 index 67bf98bdf8..0000000000 --- a/src/backend/InvenTree/part/bom.py +++ /dev/null @@ -1,303 +0,0 @@ -"""Functionality for Bill of Material (BOM) management. - -Primarily BOM upload tools. -""" - -from typing import Optional - -from .models import Part - -# TODO: 2024-12-17 - This entire file is to be removed -# TODO: Ref: https://github.com/inventree/InvenTree/pull/8685 -# TODO: To be removed as part of https://github.com/inventree/InvenTree/issues/9078 - - -def ExportBom( - part: Part, - fmt='csv', - cascade: bool = False, - max_levels: Optional[int] = None, - **kwargs, -): - """Export a BOM (Bill of Materials) for a given part. - - Args: - part (Part): Part for which the BOM should be exported - fmt (str, optional): file format. Defaults to 'csv'. - cascade (bool, optional): If True, multi-level BOM output is supported. Otherwise, a flat top-level-only BOM is exported.. Defaults to False. - max_levels (int, optional): Levels of items that should be included. None for np sublevels. Defaults to None. - - kwargs: - parameter_data (bool, optional): Additional data that should be added. Defaults to False. - stock_data (bool, optional): Additional data that should be added. Defaults to False. - supplier_data (bool, optional): Additional data that should be added. Defaults to False. - manufacturer_data (bool, optional): Additional data that should be added. Defaults to False. - pricing_data (bool, optional): Include pricing data in exported BOM. Defaults to False - substitute_part_data (bool, optional): Include substitute part numbers in exported BOM. Defaults to False - - Returns: - StreamingHttpResponse: Response that can be passed to the endpoint - """ - # TODO: All this will be pruned!!! - """ - parameter_data = str2bool(kwargs.get('parameter_data', False)) - stock_data = str2bool(kwargs.get('stock_data', False)) - supplier_data = str2bool(kwargs.get('supplier_data', False)) - manufacturer_data = str2bool(kwargs.get('manufacturer_data', False)) - pricing_data = str2bool(kwargs.get('pricing_data', False)) - substitute_part_data = str2bool(kwargs.get('substitute_part_data', False)) - - bom_items = [] - - uids = [] - - def add_items(items, level, cascade=True): - # Add items at a given layer - for item in items: - item.level = str(int(level)) - - # Avoid circular BOM references - if item.pk in uids: - continue - - bom_items.append(item) - - if cascade and item.sub_part.assembly: - if max_levels is None or level < max_levels: - add_items(item.sub_part.bom_items.all().order_by('id'), level + 1) - - top_level_items = part.get_bom_items().order_by('id') - - add_items(top_level_items, 1, cascade) - - dataset = BomItemResource().export( - queryset=bom_items, cascade=cascade, include_pricing=pricing_data - ) - - def add_columns_to_dataset(columns, column_size): - try: - for header, column_dict in columns.items(): - # Construct column tuple - col = tuple(column_dict.get(c_idx, '') for c_idx in range(column_size)) - # Add column to dataset - dataset.append_col(col, header=header) - except AttributeError: - pass - - if substitute_part_data: - # If requested, add extra columns for all substitute part numbers associated with each line item. - - col_index = 0 - substitute_cols = {} - - for bom_item in bom_items: - substitutes = BomItemSubstitute.objects.filter(bom_item=bom_item) - for s_idx, substitute in enumerate(substitutes): - # Create substitute part IPN column. - name = f'{_("Substitute IPN")}{s_idx + 1}' - value = substitute.part.IPN - try: - substitute_cols[name].update({col_index: value}) - except KeyError: - substitute_cols[name] = {col_index: value} - - # Create substitute part name column. - name = f'{_("Substitute Part")}{s_idx + 1}' - value = substitute.part.name - try: - substitute_cols[name].update({col_index: value}) - except KeyError: - substitute_cols[name] = {col_index: value} - - # Create substitute part description column. - name = f'{_("Substitute Description")}{s_idx + 1}' - value = substitute.part.description - try: - substitute_cols[name].update({col_index: value}) - except KeyError: - substitute_cols[name] = {col_index: value} - - col_index = col_index + 1 - - # Add substitute columns to dataset - add_columns_to_dataset(substitute_cols, len(bom_items)) - - if parameter_data: - # If requested, add extra columns for each PartParameter associated with each line item. -3 - parameter_cols = {} - - for b_idx, bom_item in enumerate(bom_items): - # Get part parameters - parameters = bom_item.sub_part.get_parameters() - # Add parameters to columns - if parameters: - for parameter in parameters: - name = parameter.template.name - value = parameter.data - - try: - parameter_cols[name].update({b_idx: value}) - except KeyError: - parameter_cols[name] = {b_idx: value} - - # Add parameter columns to dataset - parameter_cols_ordered = OrderedDict( - sorted(parameter_cols.items(), key=lambda x: x[0]) - ) - add_columns_to_dataset(parameter_cols_ordered, len(bom_items)) - - if stock_data: - # If requested, add extra columns for stock data associated with each line item. - - stock_headers = [ - _('Default Location'), - _('Total Stock'), - _('Available Stock'), - _('On Order'), - ] - - stock_cols = {} - - for b_idx, bom_item in enumerate(bom_items): - stock_data = [] - - sub_part = bom_item.sub_part - - # Get part default location - try: - loc = sub_part.get_default_location() - - if loc is not None: - stock_data.append(str(loc.name)) - else: - stock_data.append('') - except AttributeError: - stock_data.append('') - - # Total "in stock" quantity for this part - stock_data.append(str(normalize(sub_part.total_stock))) - - # Total "available stock" quantity for this part - stock_data.append(str(normalize(sub_part.available_stock))) - - # Total "on order" quantity for this part - stock_data.append(str(normalize(sub_part.on_order))) - - for s_idx, header in enumerate(stock_headers): - try: - stock_cols[header].update({b_idx: stock_data[s_idx]}) - except KeyError: - stock_cols[header] = {b_idx: stock_data[s_idx]} - - # Add stock columns to dataset - add_columns_to_dataset(stock_cols, len(bom_items)) - - if manufacturer_data or supplier_data: - # If requested, add extra columns for each SupplierPart and ManufacturerPart associated with each line item. - - # Keep track of the supplier parts we have already exported - supplier_parts_used = set() - - manufacturer_cols = {} - - for bom_idx, bom_item in enumerate(bom_items): - # Get part instance - b_part = bom_item.sub_part - - # Include manufacturer data for each BOM item - if manufacturer_data: - # Filter manufacturer parts - manufacturer_parts = ManufacturerPart.objects.filter( - part__pk=b_part.pk - ).prefetch_related('supplier_parts') - - for mp_idx, mp_part in enumerate(manufacturer_parts): - # Extract the "name" field of the Manufacturer (Company) - if mp_part and mp_part.manufacturer: - manufacturer_name = mp_part.manufacturer.name - else: - manufacturer_name = '' - - # Extract the "MPN" field from the Manufacturer Part - manufacturer_mpn = mp_part.MPN if mp_part else '' - - # Generate a column name for this manufacturer - k_man = f'{_("Manufacturer")}_{mp_idx}' - k_mpn = f'{_("MPN")}_{mp_idx}' - - try: - manufacturer_cols[k_man].update({bom_idx: manufacturer_name}) - manufacturer_cols[k_mpn].update({bom_idx: manufacturer_mpn}) - except KeyError: - manufacturer_cols[k_man] = {bom_idx: manufacturer_name} - manufacturer_cols[k_mpn] = {bom_idx: manufacturer_mpn} - - # We wish to include supplier data for this manufacturer part - if supplier_data: - for sp_idx, sp_part in enumerate(mp_part.supplier_parts.all()): - supplier_parts_used.add(sp_part) - - if sp_part.supplier: - supplier_name = sp_part.supplier.name - else: - supplier_name = '' - - supplier_sku = sp_part.SKU if sp_part else '' - - # Generate column names for this supplier - k_sup = ( - str(_('Supplier')) - + '_' - + str(mp_idx) - + '_' - + str(sp_idx) - ) - k_sku = ( - str(_('SKU')) + '_' + str(mp_idx) + '_' + str(sp_idx) - ) - - try: - manufacturer_cols[k_sup].update({ - bom_idx: supplier_name - }) - manufacturer_cols[k_sku].update({bom_idx: supplier_sku}) - except KeyError: - manufacturer_cols[k_sup] = {bom_idx: supplier_name} - manufacturer_cols[k_sku] = {bom_idx: supplier_sku} - - if supplier_data: - # Add in any extra supplier parts, which are not associated with a manufacturer part - - for sp_idx, sp_part in enumerate( - SupplierPart.objects.filter(part__pk=b_part.pk) - ): - if sp_part in supplier_parts_used: - continue - - supplier_parts_used.add(sp_part) - - supplier_name = sp_part.supplier.name if sp_part.supplier else '' - - supplier_sku = sp_part.SKU - - # Generate column names for this supplier - k_sup = str(_('Supplier')) + '_' + str(sp_idx) - k_sku = str(_('SKU')) + '_' + str(sp_idx) - - try: - manufacturer_cols[k_sup].update({bom_idx: supplier_name}) - manufacturer_cols[k_sku].update({bom_idx: supplier_sku}) - except KeyError: - manufacturer_cols[k_sup] = {bom_idx: supplier_name} - manufacturer_cols[k_sku] = {bom_idx: supplier_sku} - - # Add supplier columns to dataset - add_columns_to_dataset(manufacturer_cols, len(bom_items)) - - data = dataset.export(fmt) - - filename = f'{part.full_name}_BOM.{fmt}' - - return DownloadFile(data, filename) - """ diff --git a/src/backend/InvenTree/part/models.py b/src/backend/InvenTree/part/models.py index ec3c9b3927..5367944c28 100644 --- a/src/backend/InvenTree/part/models.py +++ b/src/backend/InvenTree/part/models.py @@ -603,9 +603,9 @@ class Part( This function is exposed to any Validation plugins, and thus can be customized. """ - from plugin.registry import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): # Run the name through each custom validator # If the plugin returns 'True' we will skip any subsequent validation @@ -625,9 +625,9 @@ class Part( - Validation is handled by custom plugins - By default, no validation checks are performed """ - from plugin.registry import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: result = plugin.validate_part_ipn(self.IPN, self) @@ -724,10 +724,10 @@ class Part( serial = str(serial).strip() # First, throw the serial number against each of the loaded validation plugins - from plugin.registry import registry + from plugin import PluginMixinEnum, registry try: - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): # Run the serial number through each custom validator # If the plugin returns 'True' we will skip any subsequent validation @@ -844,12 +844,12 @@ class Part( Returns: The latest serial number specified for this part, or None """ - from plugin.registry import registry + from plugin import PluginMixinEnum, registry if allow_plugins: # Check with plugin system # If any plugin returns a non-null result, that takes priority - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: result = plugin.get_latest_serial_number(self) if result is not None: @@ -3922,9 +3922,9 @@ class PartParameter(InvenTree.models.InvenTreeMetadataModel): self.calculate_numeric_value() # Run custom validation checks (via plugins) - from plugin.registry import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): # Note: The validate_part_parameter function may raise a ValidationError try: result = plugin.validate_part_parameter(self, self.data) diff --git a/src/backend/InvenTree/part/serializers.py b/src/backend/InvenTree/part/serializers.py index ad906f6ffc..3a69b4d73f 100644 --- a/src/backend/InvenTree/part/serializers.py +++ b/src/backend/InvenTree/part/serializers.py @@ -33,8 +33,8 @@ import part.tasks import stock.models import users.models from build.status_codes import BuildStatusGroups -from importer.mixins import DataImportExportSerializerMixin from importer.registry import register_importer +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.ready import isGeneratingSchema from InvenTree.tasks import offload_task from users.serializers import UserSerializer @@ -1526,6 +1526,8 @@ class BomItemSerializer( import_exclude_fields = ['validated', 'substitutes'] + export_exclude_fields = ['substitutes'] + export_child_fields = [ 'sub_part_detail.name', 'sub_part_detail.IPN', diff --git a/src/backend/InvenTree/part/test_api.py b/src/backend/InvenTree/part/test_api.py index 22b2344d38..9130478044 100644 --- a/src/backend/InvenTree/part/test_api.py +++ b/src/backend/InvenTree/part/test_api.py @@ -1180,7 +1180,6 @@ class PartAPITest(PartAPITestBase): 'Virtual', 'Trackable', 'Active', - 'Notes', 'Creation Date', 'On Order', 'In Stock', @@ -1189,9 +1188,9 @@ class PartAPITest(PartAPITestBase): excluded_cols = ['lft', 'rght', 'level', 'tree_id', 'metadata'] - with self.download_file(url, {'export': 'csv'}) as file: + with self.export_data(url, export_format='csv') as data_file: data = self.process_csv( - file, + data_file, excluded_cols=excluded_cols, required_cols=required_cols, required_rows=Part.objects.count(), @@ -2628,6 +2627,46 @@ class BomItemTest(InvenTreeAPITestCase): self.assertEqual(response.data['available_variant_stock'], 1000) + def test_bom_export(self): + """Test for exporting BOM data.""" + url = reverse('api-bom-list') + + required_cols = [ + 'Assembly', + 'Component', + 'Reference', + 'Quantity', + 'Component.Name', + 'Component.Description', + 'Available Stock', + ] + + excluded_cols = ['BOM Level', 'Supplier 1'] + + # First, download *all* BOM data, with the default exporter + with self.export_data(url, expected_code=200) as data_file: + self.process_csv( + data_file, + required_cols=required_cols, + excluded_cols=excluded_cols, + required_rows=BomItem.objects.all().count(), + ) + + # Check that the correct exporter plugin has been used + required_cols.extend(['BOM Level']) + + # Next, download BOM data for a specific sub-assembly, and use the BOM exporter + with self.export_data( + url, {'part': 100}, export_plugin='bom-exporter' + ) as data_file: + data = self.process_csv( + data_file, required_cols=required_cols, required_rows=4 + ) + + for row in data: + self.assertEqual(str(row['Assembly']), '100') + self.assertEqual(str(row['BOM Level']), '1') + class PartAttachmentTest(InvenTreeAPITestCase): """Unit tests for the PartAttachment API endpoint.""" diff --git a/src/backend/InvenTree/plugin/__init__.py b/src/backend/InvenTree/plugin/__init__.py index 01dff2124f..9d0fb43a8e 100644 --- a/src/backend/InvenTree/plugin/__init__.py +++ b/src/backend/InvenTree/plugin/__init__.py @@ -1,12 +1,13 @@ """Utility file to enable simper imports.""" from .helpers import MixinImplementationError, MixinNotImplementedError -from .plugin import InvenTreePlugin +from .plugin import InvenTreePlugin, PluginMixinEnum from .registry import registry __all__ = [ 'InvenTreePlugin', 'MixinImplementationError', 'MixinNotImplementedError', + 'PluginMixinEnum', 'registry', ] diff --git a/src/backend/InvenTree/plugin/base/action/api.py b/src/backend/InvenTree/plugin/base/action/api.py index a59eac6384..f41d6a30b3 100644 --- a/src/backend/InvenTree/plugin/base/action/api.py +++ b/src/backend/InvenTree/plugin/base/action/api.py @@ -7,7 +7,7 @@ from rest_framework.generics import GenericAPIView from rest_framework.response import Response from InvenTree.exceptions import log_error -from plugin import registry +from plugin import PluginMixinEnum, registry class ActionPluginSerializer(serializers.Serializer): @@ -32,7 +32,7 @@ class ActionPluginView(GenericAPIView): if action is None: return Response({'error': _('No action specified')}) - action_plugins = registry.with_mixin('action') + action_plugins = registry.with_mixin(PluginMixinEnum.ACTION) for plugin in action_plugins: try: if plugin.action_name() == action: diff --git a/src/backend/InvenTree/plugin/base/action/mixins.py b/src/backend/InvenTree/plugin/base/action/mixins.py index e4931c1012..13568c44a3 100644 --- a/src/backend/InvenTree/plugin/base/action/mixins.py +++ b/src/backend/InvenTree/plugin/base/action/mixins.py @@ -1,5 +1,7 @@ """Plugin mixin classes for action plugin.""" +from plugin import PluginMixinEnum + class ActionMixin: """Mixin that enables custom actions.""" @@ -14,7 +16,7 @@ class ActionMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('action', True, __class__) + self.add_mixin(PluginMixinEnum.ACTION, True, __class__) def action_name(self): """Action name for this plugin. diff --git a/src/backend/InvenTree/plugin/base/barcodes/api.py b/src/backend/InvenTree/plugin/base/barcodes/api.py index 5c88e7baf8..4a140a91cc 100644 --- a/src/backend/InvenTree/plugin/base/barcodes/api.py +++ b/src/backend/InvenTree/plugin/base/barcodes/api.py @@ -23,7 +23,7 @@ from InvenTree.filters import SEARCH_ORDER_FILTER from InvenTree.helpers import hash_barcode from InvenTree.mixins import ListAPI, RetrieveDestroyAPI from InvenTree.permissions import IsStaffOrReadOnly -from plugin import registry +from plugin import PluginMixinEnum, registry from users.models import RuleSet from . import serializers as barcode_serializers @@ -143,7 +143,7 @@ class BarcodeView(CreateAPIView): Check each loaded plugin, and return the first valid match """ - plugins = registry.with_mixin('barcode') + plugins = registry.with_mixin(PluginMixinEnum.BARCODE) # Look for a barcode plugin which knows how to deal with this barcode plugin = None @@ -521,7 +521,7 @@ class BarcodePOReceive(BarcodeView): except Exception: pass - plugins = registry.with_mixin('barcode') + plugins = registry.with_mixin(PluginMixinEnum.BARCODE) # Look for a barcode plugin which knows how to deal with this barcode plugin = None @@ -539,7 +539,7 @@ class BarcodePOReceive(BarcodeView): raise ValidationError(response) # Now, look just for "supplier-barcode" plugins - plugins = registry.with_mixin('supplier-barcode') + plugins = registry.with_mixin(PluginMixinEnum.SUPPLIER_BARCODE) plugin_response = None diff --git a/src/backend/InvenTree/plugin/base/barcodes/mixins.py b/src/backend/InvenTree/plugin/base/barcodes/mixins.py index 69905e1dcc..ff3080dabc 100644 --- a/src/backend/InvenTree/plugin/base/barcodes/mixins.py +++ b/src/backend/InvenTree/plugin/base/barcodes/mixins.py @@ -13,6 +13,7 @@ from InvenTree.exceptions import log_error from InvenTree.models import InvenTreeBarcodeMixin from order.models import PurchaseOrder from part.models import Part +from plugin import PluginMixinEnum from plugin.base.integration.SettingsMixin import SettingsMixin logger = structlog.get_logger('inventree') @@ -34,7 +35,7 @@ class BarcodeMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('barcode', 'has_barcode', __class__) + self.add_mixin(PluginMixinEnum.BARCODE, 'has_barcode', __class__) @property def has_barcode(self): @@ -104,7 +105,7 @@ class SupplierBarcodeMixin(BarcodeMixin): def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('supplier-barcode', True, __class__) + self.add_mixin(PluginMixinEnum.SUPPLIER_BARCODE, True, __class__) def get_field_value(self, key, backup_value=None): """Return the value of a barcode field.""" diff --git a/src/backend/InvenTree/plugin/base/event/mixins.py b/src/backend/InvenTree/plugin/base/event/mixins.py index 51c526ec7a..cba2f659c3 100644 --- a/src/backend/InvenTree/plugin/base/event/mixins.py +++ b/src/backend/InvenTree/plugin/base/event/mixins.py @@ -1,5 +1,6 @@ """Plugin mixin class for events.""" +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError @@ -33,4 +34,4 @@ class EventMixin: def __init__(self): """Register the mixin.""" super().__init__() - self.add_mixin('events', True, __class__) + self.add_mixin(PluginMixinEnum.EVENTS, True, __class__) diff --git a/src/backend/InvenTree/plugin/base/icons/mixins.py b/src/backend/InvenTree/plugin/base/icons/mixins.py index 76e15cf146..801d280296 100644 --- a/src/backend/InvenTree/plugin/base/icons/mixins.py +++ b/src/backend/InvenTree/plugin/base/icons/mixins.py @@ -3,6 +3,7 @@ import structlog from common.icons import IconPack, reload_icon_packs +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError logger = structlog.get_logger('inventree') @@ -19,7 +20,7 @@ class IconPackMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('icon_pack', True, __class__) + self.add_mixin(PluginMixinEnum.ICON_PACK, True, __class__) @classmethod def _activate_mixin(cls, registry, plugins, *args, **kwargs): diff --git a/src/backend/InvenTree/plugin/base/integration/APICallMixin.py b/src/backend/InvenTree/plugin/base/integration/APICallMixin.py index 5aa04c2d96..d67bc64857 100644 --- a/src/backend/InvenTree/plugin/base/integration/APICallMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/APICallMixin.py @@ -7,6 +7,7 @@ from typing import Optional import requests import structlog +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError logger = structlog.get_logger('inventree') @@ -72,7 +73,7 @@ class APICallMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('api_call', 'has_api_call', __class__) + self.add_mixin(PluginMixinEnum.API_CALL, 'has_api_call', __class__) @property def has_api_call(self): diff --git a/src/backend/InvenTree/plugin/base/integration/AppMixin.py b/src/backend/InvenTree/plugin/base/integration/AppMixin.py index d20a754aff..e0819cb30f 100644 --- a/src/backend/InvenTree/plugin/base/integration/AppMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/AppMixin.py @@ -11,6 +11,7 @@ from django.contrib import admin import structlog from InvenTree.config import get_plugin_dir +from plugin import PluginMixinEnum logger = structlog.get_logger('inventree') @@ -26,7 +27,7 @@ class AppMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('app', 'has_app', __class__) + self.add_mixin(PluginMixinEnum.APP, 'has_app', __class__) @classmethod def _activate_mixin( diff --git a/src/backend/InvenTree/plugin/base/integration/CurrencyExchangeMixin.py b/src/backend/InvenTree/plugin/base/integration/CurrencyExchangeMixin.py index 43e22fe856..08f7794de4 100644 --- a/src/backend/InvenTree/plugin/base/integration/CurrencyExchangeMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/CurrencyExchangeMixin.py @@ -1,5 +1,6 @@ """Plugin mixin class for supporting currency exchange data.""" +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError @@ -21,7 +22,7 @@ class CurrencyExchangeMixin: def __init__(self): """Register the mixin.""" super().__init__() - self.add_mixin('currencyexchange', True, __class__) + self.add_mixin(PluginMixinEnum.CURRENCY_EXCHANGE, True, __class__) def update_exchange_rates(self, base_currency: str, symbols: list[str]) -> dict: """Update currency exchange rates. diff --git a/src/backend/InvenTree/plugin/base/integration/DataExport.py b/src/backend/InvenTree/plugin/base/integration/DataExport.py new file mode 100644 index 0000000000..7368a5b0d7 --- /dev/null +++ b/src/backend/InvenTree/plugin/base/integration/DataExport.py @@ -0,0 +1,115 @@ +"""Plugin class for custom data exporting.""" + +from collections import OrderedDict +from typing import Union + +from django.contrib.auth.models import User +from django.db.models import QuerySet + +from rest_framework import serializers + +from common.models import DataOutput +from InvenTree.helpers import current_date +from plugin import PluginMixinEnum + + +class DataExportMixin: + """Mixin which provides ability to customize data exports. + + When exporting data from the API, this mixin can be used to provide + custom data export functionality. + """ + + ExportOptionsSerializer = None + + class MixinMeta: + """Meta options for this mixin.""" + + MIXIN_NAME = 'DataExport' + + def __init__(self): + """Register mixin.""" + super().__init__() + self.add_mixin(PluginMixinEnum.EXPORTER, True, __class__) + + def supports_export(self, model_class: type, user: User, *args, **kwargs) -> bool: + """Return True if this plugin supports exporting data for the given model. + + Args: + model_class: The model class to check + user: The user requesting the export + + Returns: + True if the plugin supports exporting data for the given model + """ + # By default, plugins support all models + return True + + def generate_filename(self, model_class, export_format: str) -> str: + """Generate a filename for the exported data.""" + model = model_class.__name__ + date = current_date().isoformat() + + return f'InvenTree_{model}_{date}.{export_format}' + + def update_headers( + self, headers: OrderedDict, context: dict, **kwargs + ) -> OrderedDict: + """Update the headers for the data export. + + Allows for optional modification of the headers for the data export. + + Arguments: + headers: The current headers for the export + context: The context for the export (provided by the plugin serializer) + + Returns: The updated headers + """ + # The default implementation does nothing + return headers + + def filter_queryset(self, queryset: QuerySet) -> QuerySet: + """Filter the queryset before exporting data.""" + # The default implementation returns the queryset unchanged + return queryset + + def export_data( + self, + queryset: QuerySet, + serializer_class: serializers.Serializer, + headers: OrderedDict, + context: dict, + output: DataOutput, + **kwargs, + ) -> list: + """Export data from the queryset. + + This method should be implemented by the plugin to provide + the actual data export functionality. + + Arguments: + queryset: The queryset to export + serializer_class: The serializer class to use for exporting the data + headers: The headers for the export + context: Any custom context for the export (provided by the plugin serializer) + output: The DataOutput object for the export + + Returns: The exported data (a list of dict objects) + """ + # The default implementation simply serializes the queryset + return serializer_class(queryset, many=True, exporting=True).data + + def get_export_options_serializer( + self, **kwargs + ) -> Union[serializers.Serializer, None]: + """Return a serializer class with dynamic export options for this plugin. + + Returns: + A class instance of a DRF serializer class, by default this an instance of + self.ExportOptionsSerializer using the *args, **kwargs if existing for this plugin + """ + # By default, look for a class level attribute + serializer = getattr(self, 'ExportOptionsSerializer', None) + + if serializer: + return serializer(**kwargs) diff --git a/src/backend/InvenTree/plugin/base/integration/NavigationMixin.py b/src/backend/InvenTree/plugin/base/integration/NavigationMixin.py index 49366edcc3..937f46d298 100644 --- a/src/backend/InvenTree/plugin/base/integration/NavigationMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/NavigationMixin.py @@ -2,6 +2,7 @@ import structlog +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError logger = structlog.get_logger('inventree') @@ -21,7 +22,7 @@ class NavigationMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('navigation', 'has_navigation', __class__) + self.add_mixin(PluginMixinEnum.NAVIGATION, 'has_navigation', __class__) self.navigation = self.setup_navigation() def setup_navigation(self): diff --git a/src/backend/InvenTree/plugin/base/integration/ReportMixin.py b/src/backend/InvenTree/plugin/base/integration/ReportMixin.py index 781a80b9e9..77ee7e3dd2 100644 --- a/src/backend/InvenTree/plugin/base/integration/ReportMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/ReportMixin.py @@ -1,5 +1,7 @@ """Plugin mixin class for ReportContextMixin.""" +from plugin import PluginMixinEnum + class ReportMixin: """Mixin which provides additional context to generated reports. @@ -20,7 +22,7 @@ class ReportMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('report', True, __class__) + self.add_mixin(PluginMixinEnum.REPORT, True, __class__) def add_report_context(self, report_instance, model_instance, request, context): """Add extra context to the provided report instance. diff --git a/src/backend/InvenTree/plugin/base/integration/ScheduleMixin.py b/src/backend/InvenTree/plugin/base/integration/ScheduleMixin.py index e2784939e4..4ffcc41182 100644 --- a/src/backend/InvenTree/plugin/base/integration/ScheduleMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/ScheduleMixin.py @@ -6,6 +6,7 @@ from django.db.utils import OperationalError, ProgrammingError import structlog from common.settings import get_global_setting +from plugin import PluginMixinEnum from plugin.helpers import MixinImplementationError logger = structlog.get_logger('inventree') @@ -54,7 +55,7 @@ class ScheduleMixin: self.scheduled_tasks = [] - self.add_mixin('schedule', 'has_scheduled_tasks', __class__) + self.add_mixin(PluginMixinEnum.SCHEDULE, 'has_scheduled_tasks', __class__) @classmethod def _activate_mixin(cls, registry, plugins, *args, **kwargs): diff --git a/src/backend/InvenTree/plugin/base/integration/SettingsMixin.py b/src/backend/InvenTree/plugin/base/integration/SettingsMixin.py index 03f9b14613..989e1e24b8 100644 --- a/src/backend/InvenTree/plugin/base/integration/SettingsMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/SettingsMixin.py @@ -6,6 +6,8 @@ from django.db.utils import OperationalError, ProgrammingError import structlog +from plugin import PluginMixinEnum + logger = structlog.get_logger('inventree') # import only for typechecking, otherwise this throws a model is unready error @@ -30,7 +32,7 @@ class SettingsMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('settings', 'has_settings', __class__) + self.add_mixin(PluginMixinEnum.SETTINGS, 'has_settings', __class__) self.settings = getattr(self, 'SETTINGS', {}) @classmethod diff --git a/src/backend/InvenTree/plugin/base/integration/UrlsMixin.py b/src/backend/InvenTree/plugin/base/integration/UrlsMixin.py index 904bc8bec0..499b163775 100644 --- a/src/backend/InvenTree/plugin/base/integration/UrlsMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/UrlsMixin.py @@ -6,6 +6,7 @@ from django.urls import include, re_path import structlog from common.settings import get_global_setting +from plugin import PluginMixinEnum from plugin.urls import PLUGIN_BASE logger = structlog.get_logger('inventree') @@ -22,7 +23,7 @@ class UrlsMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('urls', 'has_urls', __class__) + self.add_mixin(PluginMixinEnum.URLS, 'has_urls', __class__) self.urls = self.setup_urls() @classmethod diff --git a/src/backend/InvenTree/plugin/base/integration/ValidationMixin.py b/src/backend/InvenTree/plugin/base/integration/ValidationMixin.py index 6aee6c19ce..341696bf62 100644 --- a/src/backend/InvenTree/plugin/base/integration/ValidationMixin.py +++ b/src/backend/InvenTree/plugin/base/integration/ValidationMixin.py @@ -7,6 +7,7 @@ from django.db.models import Model import part.models import stock.models +from plugin import PluginMixinEnum class ValidationMixin: @@ -46,7 +47,7 @@ class ValidationMixin: def __init__(self): """Register the mixin.""" super().__init__() - self.add_mixin('validation', True, __class__) + self.add_mixin(PluginMixinEnum.VALIDATION, True, __class__) def raise_error(self, message): """Raise a ValidationError with the given message.""" diff --git a/src/backend/InvenTree/plugin/base/label/mixins.py b/src/backend/InvenTree/plugin/base/label/mixins.py index 0561cd4a24..15aed82956 100644 --- a/src/backend/InvenTree/plugin/base/label/mixins.py +++ b/src/backend/InvenTree/plugin/base/label/mixins.py @@ -9,12 +9,13 @@ import pdf2image from rest_framework import serializers from rest_framework.request import Request -from common.models import InvenTreeSetting +from common.models import DataOutput, InvenTreeSetting from InvenTree.exceptions import log_error from InvenTree.tasks import offload_task +from plugin import PluginMixinEnum from plugin.base.label import label as plugin_label from plugin.helpers import MixinNotImplementedError -from report.models import LabelTemplate, TemplateOutput +from report.models import LabelTemplate class LabelPrintingMixin: @@ -35,7 +36,7 @@ class LabelPrintingMixin: def __init__(self): # pragma: no cover """Register mixin.""" super().__init__() - self.add_mixin('labels', True, __class__) + self.add_mixin(PluginMixinEnum.LABELS, True, __class__) BLOCKING_PRINT = True @@ -104,7 +105,7 @@ class LabelPrintingMixin: def print_labels( self, label: LabelTemplate, - output: TemplateOutput, + output: DataOutput, items: list, request: Request, **kwargs, @@ -113,7 +114,7 @@ class LabelPrintingMixin: Arguments: label: The LabelTemplate object to use for printing - output: The TemplateOutput object used to store the results + output: The DataOutput object used to store the results items: The list of database items to print (e.g. StockItem instances) request: The HTTP request object which triggered this print job @@ -121,7 +122,7 @@ class LabelPrintingMixin: printing_options: The printing options set for this print job defined in the PrintingOptionsSerializer Returns: - None. Output data should be stored in the provided TemplateOutput object + None. Output data should be stored in the provided DataOutput object Raises: ValidationError if there is an error during the print process @@ -211,7 +212,7 @@ class LabelPrintingMixin: filename: The filename of this PDF label label_instance: The instance of the label model which triggered the print_label() method item_instance: The instance of the database model against which the label is printed - output: The TemplateOutput object used to store the results of the print job + output: The DataOutput object used to store the results of the print job user: The user who triggered this print job width: The expected width of the label (in mm) height: The expected height of the label (in mm) diff --git a/src/backend/InvenTree/plugin/base/label/test_label_mixin.py b/src/backend/InvenTree/plugin/base/label/test_label_mixin.py index c95fb4036b..73b522450f 100644 --- a/src/backend/InvenTree/plugin/base/label/test_label_mixin.py +++ b/src/backend/InvenTree/plugin/base/label/test_label_mixin.py @@ -13,10 +13,9 @@ from PIL import Image from InvenTree.settings import BASE_DIR from InvenTree.unit_test import InvenTreeAPITestCase from part.models import Part +from plugin import InvenTreePlugin, PluginMixinEnum, registry from plugin.base.label.mixins import LabelPrintingMixin from plugin.helpers import MixinNotImplementedError -from plugin.plugin import InvenTreePlugin -from plugin.registry import registry from report.models import LabelTemplate from report.tests import PrintTestMixins from stock.models import StockItem, StockLocation @@ -48,11 +47,11 @@ class LabelMixinTests(PrintTestMixins, InvenTreeAPITestCase): def test_installed(self): """Test that the sample printing plugin is installed.""" # Get all label plugins - plugins = registry.with_mixin('labels', active=None) + plugins = registry.with_mixin(PluginMixinEnum.LABELS, active=None) self.assertEqual(len(plugins), 4) # But, it is not 'active' - plugins = registry.with_mixin('labels', active=True) + plugins = registry.with_mixin(PluginMixinEnum.LABELS, active=True) self.assertEqual(len(plugins), 3) def test_api(self): @@ -121,7 +120,7 @@ class LabelMixinTests(PrintTestMixins, InvenTreeAPITestCase): self.assertIn('Plugin does not support label printing', str(response.data)) # Find available plugins - plugins = registry.with_mixin('labels') + plugins = registry.with_mixin(PluginMixinEnum.LABELS) self.assertGreater(len(plugins), 0) plugin = registry.get_plugin('samplelabelprinter') diff --git a/src/backend/InvenTree/plugin/base/locate/api.py b/src/backend/InvenTree/plugin/base/locate/api.py index 2d3d64a4ce..08d3c3b1b6 100644 --- a/src/backend/InvenTree/plugin/base/locate/api.py +++ b/src/backend/InvenTree/plugin/base/locate/api.py @@ -7,7 +7,8 @@ from rest_framework.response import Response from InvenTree.exceptions import log_error from InvenTree.tasks import offload_task -from plugin.registry import call_plugin_function, registry +from plugin import PluginMixinEnum, registry +from plugin.registry import call_plugin_function from stock.models import StockItem, StockLocation @@ -39,7 +40,7 @@ class LocatePluginView(GenericAPIView): raise ParseError("'plugin' field must be supplied") # Check that the plugin exists, and supports the 'locate' mixin - plugins = registry.with_mixin('locate') + plugins = registry.with_mixin(PluginMixinEnum.LOCATE) if plugin not in [p.slug for p in plugins]: raise ParseError( diff --git a/src/backend/InvenTree/plugin/base/locate/mixins.py b/src/backend/InvenTree/plugin/base/locate/mixins.py index aaf9f20435..b1986f3cf5 100644 --- a/src/backend/InvenTree/plugin/base/locate/mixins.py +++ b/src/backend/InvenTree/plugin/base/locate/mixins.py @@ -2,6 +2,7 @@ import structlog +from plugin import PluginMixinEnum from plugin.helpers import MixinNotImplementedError logger = structlog.get_logger('inventree') @@ -31,7 +32,7 @@ class LocateMixin: def __init__(self): """Register the mixin.""" super().__init__() - self.add_mixin('locate', True, __class__) + self.add_mixin(PluginMixinEnum.LOCATE, True, __class__) def locate_stock_item(self, item_pk): """Attempt to locate a particular StockItem. diff --git a/src/backend/InvenTree/plugin/base/locate/test_locate.py b/src/backend/InvenTree/plugin/base/locate/test_locate.py index e03070a411..6170edfb8b 100644 --- a/src/backend/InvenTree/plugin/base/locate/test_locate.py +++ b/src/backend/InvenTree/plugin/base/locate/test_locate.py @@ -3,7 +3,7 @@ from django.urls import reverse from InvenTree.unit_test import InvenTreeAPITestCase -from plugin import InvenTreePlugin, MixinNotImplementedError, registry +from plugin import InvenTreePlugin, MixinNotImplementedError, PluginMixinEnum, registry from plugin.base.locate.mixins import LocateMixin from stock.models import StockItem, StockLocation @@ -24,7 +24,7 @@ class LocatePluginTests(InvenTreeAPITestCase): def test_installed(self): """Test that a locate plugin is actually installed.""" - plugins = registry.with_mixin('locate') + plugins = registry.with_mixin(PluginMixinEnum.LOCATE) self.assertGreater(len(plugins), 0) diff --git a/src/backend/InvenTree/plugin/base/ui/api.py b/src/backend/InvenTree/plugin/base/ui/api.py index 63916e0d89..30ff3a1558 100644 --- a/src/backend/InvenTree/plugin/base/ui/api.py +++ b/src/backend/InvenTree/plugin/base/ui/api.py @@ -10,7 +10,7 @@ from rest_framework.views import APIView import plugin.base.ui.serializers as UIPluginSerializers from common.settings import get_global_setting from InvenTree.exceptions import log_error -from plugin import registry +from plugin import PluginMixinEnum, registry class PluginUIFeatureList(APIView): @@ -28,7 +28,9 @@ class PluginUIFeatureList(APIView): if get_global_setting('ENABLE_PLUGINS_INTERFACE'): # Extract all plugins from the registry which provide custom ui features - for _plugin in registry.with_mixin('ui', active=True): + for _plugin in registry.with_mixin( + PluginMixinEnum.USER_INTERFACE, active=True + ): # Allow plugins to fill this data out try: diff --git a/src/backend/InvenTree/plugin/base/ui/mixins.py b/src/backend/InvenTree/plugin/base/ui/mixins.py index ad3990b205..4534efa5f1 100644 --- a/src/backend/InvenTree/plugin/base/ui/mixins.py +++ b/src/backend/InvenTree/plugin/base/ui/mixins.py @@ -8,6 +8,8 @@ from typing import Literal, TypedDict import structlog from rest_framework.request import Request +from plugin import PluginMixinEnum + logger = structlog.get_logger('inventree') @@ -79,7 +81,7 @@ class UserInterfaceMixin: def __init__(self): """Register mixin.""" super().__init__() - self.add_mixin('ui', True, __class__) # type: ignore + self.add_mixin(PluginMixinEnum.USER_INTERFACE, True, __class__) # type: ignore def get_ui_features( self, feature_type: FeatureType, context: dict, request: Request, **kwargs diff --git a/src/backend/InvenTree/plugin/base/ui/tests.py b/src/backend/InvenTree/plugin/base/ui/tests.py index feda515928..2ff09e8960 100644 --- a/src/backend/InvenTree/plugin/base/ui/tests.py +++ b/src/backend/InvenTree/plugin/base/ui/tests.py @@ -4,7 +4,7 @@ from django.urls import reverse from common.models import InvenTreeSetting from InvenTree.unit_test import InvenTreeAPITestCase -from plugin.registry import registry +from plugin import PluginMixinEnum, registry class UserInterfaceMixinTests(InvenTreeAPITestCase): @@ -30,7 +30,7 @@ class UserInterfaceMixinTests(InvenTreeAPITestCase): plugin = registry.get_plugin('sampleui') self.assertTrue(plugin.is_active()) - plugins = registry.with_mixin('ui') + plugins = registry.with_mixin(PluginMixinEnum.USER_INTERFACE) self.assertGreater(len(plugins), 0) def test_ui_dashboard_items(self): diff --git a/src/backend/InvenTree/plugin/builtin/exporter/__init__.py b/src/backend/InvenTree/plugin/builtin/exporter/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/src/backend/InvenTree/plugin/builtin/exporter/bom_exporter.py b/src/backend/InvenTree/plugin/builtin/exporter/bom_exporter.py new file mode 100644 index 0000000000..d5fbc385bd --- /dev/null +++ b/src/backend/InvenTree/plugin/builtin/exporter/bom_exporter.py @@ -0,0 +1,286 @@ +"""Multi-level BOM exporter plugin.""" + +from django.utils.translation import gettext_lazy as _ + +import rest_framework.serializers as serializers + +from part.models import BomItem +from plugin import InvenTreePlugin +from plugin.mixins import DataExportMixin + + +class BomExporterOptionsSerializer(serializers.Serializer): + """Custom export options for the BOM exporter plugin.""" + + export_levels = serializers.IntegerField( + default=1, + label=_('Levels'), + help_text=_('Number of levels to export'), + min_value=0, + ) + + export_stock_data = serializers.BooleanField( + default=True, label=_('Stock Data'), help_text=_('Include part stock data') + ) + + export_pricing_data = serializers.BooleanField( + default=True, label=_('Pricing Data'), help_text=_('Include part pricing data') + ) + + export_supplier_data = serializers.BooleanField( + default=True, label=_('Supplier Data'), help_text=_('Include supplier data') + ) + + export_manufacturer_data = serializers.BooleanField( + default=True, + label=_('Manufacturer Data'), + help_text=_('Include manufacturer data'), + ) + + export_substitute_data = serializers.BooleanField( + default=True, + label=_('Substitute Data'), + help_text=_('Include substitute part data'), + ) + + export_parameter_data = serializers.BooleanField( + default=True, + label=_('Parameter Data'), + help_text=_('Include part parameter data'), + ) + + +class BomExporterPlugin(DataExportMixin, InvenTreePlugin): + """Builtin plugin for performing multi-level BOM exports.""" + + NAME = 'BOM Exporter' + SLUG = 'bom-exporter' + TITLE = _('Multi-Level BOM Exporter') + DESCRIPTION = _('Provides support for exporting multi-level BOMs') + VERSION = '1.0.0' + AUTHOR = _('InvenTree contributors') + + ExportOptionsSerializer = BomExporterOptionsSerializer + + def supports_export(self, model_class: type, user, *args, **kwargs) -> bool: + """This exported only supports the BomItem model.""" + return model_class == BomItem + + def update_headers(self, headers, context, **kwargs): + """Update headers for the BOM export.""" + if not self.export_stock_data: + # Remove stock data from the headers + for field in [ + 'available_stock', + 'available_substitute_stock', + 'available_variant_stock', + 'external_stock', + 'on_order', + 'building', + 'can_build', + ]: + headers.pop(field, None) + + if not self.export_pricing_data: + # Remove pricing data from the headers + for field in [ + 'pricing_min', + 'pricing_max', + 'pricing_min_total', + 'pricing_max_total', + 'pricing_updated', + ]: + headers.pop(field, None) + + # Append a "BOM Level" field + headers['level'] = _('BOM Level') + + # Append variant part columns + if self.export_substitute_data and self.n_substitute_cols > 0: + for idx in range(self.n_substitute_cols): + n = idx + 1 + headers[f'substitute_{idx}'] = _(f'Substitute {n}') + + # Append supplier part columns + if self.export_supplier_data and self.n_supplier_cols > 0: + for idx in range(self.n_supplier_cols): + n = idx + 1 + headers[f'supplier_name_{idx}'] = _(f'Supplier {n}') + headers[f'supplier_sku_{idx}'] = _(f'Supplier {n} SKU') + headers[f'supplier_mpn_{idx}'] = _(f'Supplier {n} MPN') + + # Append manufacturer part columns + if self.export_manufacturer_data and self.n_manufacturer_cols > 0: + for idx in range(self.n_manufacturer_cols): + n = idx + 1 + headers[f'manufacturer_name_{idx}'] = _(f'Manufacturer {n}') + headers[f'manufacturer_mpn_{idx}'] = _(f'Manufacturer {n} MPN') + + # Append part parameter columns + if self.export_parameter_data and len(self.parameters) > 0: + for key, value in self.parameters.items(): + headers[f'parameter_{key}'] = value + + return headers + + def prefetch_queryset(self, queryset): + """Perform pre-fetch on the provided queryset.""" + queryset = queryset.prefetch_related('sub_part') + + if self.export_substitute_data: + queryset = queryset.prefetch_related('substitutes') + + if self.export_supplier_data: + queryset = queryset.prefetch_related('sub_part__supplier_parts') + queryset = queryset.prefetch_related( + 'sub_part__supplier_parts__manufacturer_part' + ) + + if self.export_manufacturer_data: + queryset = queryset.prefetch_related('sub_part__manufacturer_parts') + + if self.export_parameter_data: + queryset = queryset.prefetch_related('sub_part__parameters') + queryset = queryset.prefetch_related('sub_part__parameters__template') + + return queryset + + def export_data( + self, queryset, serializer_class, headers, context, output, **kwargs + ): + """Export BOM data from the queryset.""" + self.serializer_class = serializer_class + + # Track how many extra columns we need + self.n_substitute_cols = 0 + self.n_supplier_cols = 0 + self.n_manufacturer_cols = 0 + + # A dict of "Parameter ID" -> "Parameter Name" + self.parameters = {} + + # Extract the export options from the context (and cache for later) + self.export_levels = context.get('export_levels', 1) + self.export_stock_data = context.get('export_stock_data', True) + self.export_pricing_data = context.get('export_pricing_data', True) + self.export_supplier_data = context.get('export_supplier_data', True) + self.export_manufacturer_data = context.get('export_manufacturer_data', True) + self.export_substitute_data = context.get('export_substitute_data', True) + self.export_parameter_data = context.get('export_parameter_data', True) + + # Pre-fetch related data to reduce database queries + queryset = self.prefetch_queryset(queryset) + + self.bom_data = [] + + # Run through each item in the queryset + for bom_item in queryset: + self.process_bom_row(bom_item, 1, **kwargs) + + return self.bom_data + + def process_bom_row(self, bom_item, level, **kwargs) -> list: + """Process a single BOM row. + + Arguments: + bom_item: The BomItem object to process + level: The current level of export + + """ + # Add this row to the output dataset + row = self.serializer_class(bom_item, exporting=True).data + row['level'] = level + + # Extend with additional data + + if self.export_substitute_data: + row.update(self.get_substitute_data(bom_item)) + + if self.export_supplier_data: + row.update(self.get_supplier_data(bom_item)) + + if self.export_manufacturer_data: + row.update(self.get_manufacturer_data(bom_item)) + + if self.export_parameter_data: + row.update(self.get_parameter_data(bom_item)) + + self.bom_data.append(row) + + # If we have reached the maximum export level, return just this bom item + if bom_item.sub_part.assembly and ( + self.export_levels <= 0 or level < self.export_levels + ): + sub_items = bom_item.sub_part.get_bom_items() + sub_items = self.prefetch_queryset(sub_items) + + for item in sub_items.all(): + self.process_bom_row(item, level + 1, **kwargs) + + def get_substitute_data(self, bom_item: BomItem) -> dict: + """Return substitute part data for a BomItem.""" + substitute_part_data = {} + + idx = 0 + + for substitute in bom_item.substitutes.all(): + substitute_part_data.update({f'substitute_{idx}': substitute.part.name}) + + idx += 1 + + self.n_substitute_cols = max(self.n_substitute_cols, idx) + + return substitute_part_data + + def get_supplier_data(self, bom_item: BomItem) -> dict: + """Return supplier and manufacturer data for a BomItem.""" + supplier_part_data = {} + + idx = 0 + + for supplier_part in bom_item.sub_part.supplier_parts.all(): + manufacturer_part = supplier_part.manufacturer_part + supplier_part_data.update({ + f'supplier_name_{idx}': supplier_part.supplier.name, + f'supplier_sku_{idx}': supplier_part.SKU, + f'supplier_mpn_{idx}': manufacturer_part.MPN + if manufacturer_part + else '', + }) + + idx += 1 + + self.n_supplier_cols = max(self.n_supplier_cols, idx) + + return supplier_part_data + + def get_manufacturer_data(self, bom_item: BomItem) -> dict: + """Return manufacturer data for a BomItem.""" + manufacturer_part_data = {} + + idx = 0 + + for manufacturer_part in bom_item.sub_part.manufacturer_parts.all(): + manufacturer_part_data.update({ + f'manufacturer_name_{idx}': manufacturer_part.manufacturer.name, + f'manufacturer_mpn_{idx}': manufacturer_part.MPN, + }) + + idx += 1 + + self.n_manufacturer_cols = max(self.n_manufacturer_cols, idx) + + return manufacturer_part_data + + def get_parameter_data(self, bom_item: BomItem) -> dict: + """Return parameter data for a BomItem.""" + parameter_data = {} + + for parameter in bom_item.sub_part.parameters.all(): + template = parameter.template + if template.pk not in self.parameters: + self.parameters[template.pk] = template.name + + parameter_data.update({f'parameter_{template.pk}': parameter.data}) + + return parameter_data diff --git a/src/backend/InvenTree/plugin/builtin/exporter/inventree_exporter.py b/src/backend/InvenTree/plugin/builtin/exporter/inventree_exporter.py new file mode 100644 index 0000000000..b94fb3dc83 --- /dev/null +++ b/src/backend/InvenTree/plugin/builtin/exporter/inventree_exporter.py @@ -0,0 +1,21 @@ +"""Generic data export plugin for InvenTree.""" + +from django.utils.translation import gettext_lazy as _ + +from plugin import InvenTreePlugin +from plugin.mixins import DataExportMixin + + +class InvenTreeExporter(DataExportMixin, InvenTreePlugin): + """Generic exporter plugin for InvenTree.""" + + NAME = 'InvenTree Exporter' + SLUG = 'inventree-exporter' + TITLE = _('InvenTree Generic Exporter') + DESCRIPTION = _('Provides support for exporting data from InvenTree') + VERSION = '1.0.0' + AUTHOR = _('InvenTree contributors') + + def supports_export(self, model_class: type, user, *args, **kwargs) -> bool: + """This exporter supports all model classes.""" + return True diff --git a/src/backend/InvenTree/plugin/builtin/labels/label_sheet.py b/src/backend/InvenTree/plugin/builtin/labels/label_sheet.py index ce904c40df..40f9d9639f 100644 --- a/src/backend/InvenTree/plugin/builtin/labels/label_sheet.py +++ b/src/backend/InvenTree/plugin/builtin/labels/label_sheet.py @@ -11,10 +11,11 @@ import weasyprint from rest_framework import serializers import report.helpers +from common.models import DataOutput from InvenTree.helpers import str2bool from plugin import InvenTreePlugin from plugin.mixins import LabelPrintingMixin, SettingsMixin -from report.models import LabelOutput, LabelTemplate +from report.models import LabelTemplate logger = structlog.get_logger('inventree') @@ -76,7 +77,7 @@ class InvenTreeLabelSheetPlugin(LabelPrintingMixin, SettingsMixin, InvenTreePlug PrintingOptionsSerializer = LabelPrintingOptionsSerializer def print_labels( - self, label: LabelTemplate, output: LabelOutput, items: list, request, **kwargs + self, label: LabelTemplate, output: DataOutput, items: list, request, **kwargs ): """Handle printing of the provided labels. diff --git a/src/backend/InvenTree/plugin/mixins/__init__.py b/src/backend/InvenTree/plugin/mixins/__init__.py index 40683be474..cb137237c3 100644 --- a/src/backend/InvenTree/plugin/mixins/__init__.py +++ b/src/backend/InvenTree/plugin/mixins/__init__.py @@ -8,6 +8,7 @@ from plugin.base.icons.mixins import IconPackMixin from plugin.base.integration.APICallMixin import APICallMixin from plugin.base.integration.AppMixin import AppMixin from plugin.base.integration.CurrencyExchangeMixin import CurrencyExchangeMixin +from plugin.base.integration.DataExport import DataExportMixin from plugin.base.integration.NavigationMixin import NavigationMixin from plugin.base.integration.ReportMixin import ReportMixin from plugin.base.integration.ScheduleMixin import ScheduleMixin @@ -25,6 +26,7 @@ __all__ = [ 'BarcodeMixin', 'BulkNotificationMethod', 'CurrencyExchangeMixin', + 'DataExportMixin', 'EventMixin', 'IconPackMixin', 'LabelPrintingMixin', diff --git a/src/backend/InvenTree/plugin/plugin.py b/src/backend/InvenTree/plugin/plugin.py index 81ccf03d18..2ba7de00b9 100644 --- a/src/backend/InvenTree/plugin/plugin.py +++ b/src/backend/InvenTree/plugin/plugin.py @@ -1,12 +1,13 @@ """Base Class for InvenTree plugins.""" +import enum import inspect import warnings from datetime import datetime from distutils.sysconfig import get_python_lib from importlib.metadata import PackageNotFoundError, metadata from pathlib import Path -from typing import Optional +from typing import Optional, Union from django.conf import settings from django.utils.text import slugify @@ -20,6 +21,36 @@ from plugin.helpers import get_git_log logger = structlog.get_logger('inventree') +class PluginMixinEnum(str, enum.Enum): + """Enumeration of the available plugin mixin types.""" + + def __str__(self): + """Return the string representation of the mixin.""" + return self.value + + BASE = 'base' + + ACTION = 'action' + API_CALL = 'api_call' + APP = 'app' + BARCODE = 'barcode' + CURRENCY_EXCHANGE = 'currencyexchange' + EVENTS = 'events' + EXPORTER = 'exporter' + ICON_PACK = 'icon_pack' + LABELS = 'labels' + LOCATE = 'locate' + NAVIGATION = 'navigation' + REPORT = 'report' + SCHEDULE = 'schedule' + SETTINGS = 'settings' + SETTINGS_CONTENT = 'settingscontent' + SUPPLIER_BARCODE = 'supplier-barcode' + URLS = 'urls' + USER_INTERFACE = 'ui' + VALIDATION = 'validation' + + class MetaBase: """Base class for a plugins metadata.""" @@ -126,12 +157,15 @@ class MixinBase: self._mixins = {} super().__init__(*args, **kwargs) - def mixin(self, key): + def mixin(self, key: str) -> bool: """Check if mixin is registered.""" + key = str(key).lower() return key in self._mixins - def mixin_enabled(self, key): + def mixin_enabled(self, key: str) -> bool: """Check if mixin is registered, enabled and ready.""" + key = str(key).lower() + if self.mixin(key): fnc_name = self._mixins.get(key) @@ -150,6 +184,8 @@ class MixinBase: def add_mixin(self, key: str, fnc_enabled=True, cls=None): """Add a mixin to the plugins registry.""" + key = str(key).lower() + self._mixins[key] = fnc_enabled self.setup_mixin(key, cls=cls) @@ -230,7 +266,7 @@ class InvenTreePlugin(VersionMixin, MixinBase, MetaBase): Set paths and load metadata. """ super().__init__() - self.add_mixin('base') + self.add_mixin(PluginMixinEnum.BASE) self.define_package() @@ -351,7 +387,7 @@ class InvenTreePlugin(VersionMixin, MixinBase, MetaBase): return self.check_package_path() @classmethod - def check_package_install_name(cls) -> [str, None]: + def check_package_install_name(cls) -> Union[str, None]: """Installable package name of the plugin. e.g. if this plugin was installed via 'pip install ', @@ -363,7 +399,7 @@ class InvenTreePlugin(VersionMixin, MixinBase, MetaBase): return getattr(cls, 'package_name', None) @property - def package_install_name(self) -> [str, None]: + def package_install_name(self) -> Union[str, None]: """Installable package name of the plugin. e.g. if this plugin was installed via 'pip install ', diff --git a/src/backend/InvenTree/plugin/registry.py b/src/backend/InvenTree/plugin/registry.py index a7257d29fe..81b5b3ed16 100644 --- a/src/backend/InvenTree/plugin/registry.py +++ b/src/backend/InvenTree/plugin/registry.py @@ -90,8 +90,14 @@ class PluginsRegistry: """Return True if the plugin registry is currently loading.""" return self.loading_lock.locked() - def get_plugin(self, slug, active=None): - """Lookup plugin by slug (unique key).""" + def get_plugin(self, slug, active=None, with_mixin=None): + """Lookup plugin by slug (unique key). + + Arguments: + slug {str}: The slug (unique key) of the plugin + active {bool, None}: Filter by 'active' status of plugin. Defaults to None. + with_mixin {str, None}: Filter by mixin. Defaults to None. + """ # Check if the registry needs to be reloaded self.check_reload() @@ -104,6 +110,9 @@ class PluginsRegistry: if active is not None and active != plg.is_active(): return None + if with_mixin is not None and not plg.mixin_enabled(with_mixin): + return None + return plg def get_plugin_config(self, slug: str, name: Union[str, None] = None): @@ -191,7 +200,9 @@ class PluginsRegistry: return plugin_func(*args, **kwargs) # region registry functions - def with_mixin(self, mixin: str, active=True, builtin=None): + def with_mixin( + self, mixin: str, active: bool = True, builtin: Optional[bool] = None + ) -> list: """Returns reference to all plugins that have a specified mixin enabled. Args: @@ -202,6 +213,8 @@ class PluginsRegistry: # Check if the registry needs to be loaded self.check_reload() + mixin = str(mixin).lower().strip() + result = [] for plugin in self.plugins.values(): @@ -841,7 +854,7 @@ class PluginsRegistry: Returns True if the registry has changed and was reloaded. """ - if settings.TESTING: + if settings.TESTING and not settings.PLUGIN_TESTING_RELOAD: # Skip if running during unit testing return False diff --git a/src/backend/InvenTree/plugin/templatetags/plugin_extras.py b/src/backend/InvenTree/plugin/templatetags/plugin_extras.py index 35e45925de..e8c945c11b 100644 --- a/src/backend/InvenTree/plugin/templatetags/plugin_extras.py +++ b/src/backend/InvenTree/plugin/templatetags/plugin_extras.py @@ -33,7 +33,7 @@ def mixin_enabled(plugin, key, *args, **kwargs): @register.simple_tag() -def mixin_available(mixin, *args, **kwargs): +def mixin_available(mixin: str, *args, **kwargs) -> bool: """Returns True if there is at least one active plugin which supports the provided mixin.""" return len(registry.with_mixin(mixin)) > 0 diff --git a/src/backend/InvenTree/report/admin.py b/src/backend/InvenTree/report/admin.py index 04cab87349..771cbc1b2c 100644 --- a/src/backend/InvenTree/report/admin.py +++ b/src/backend/InvenTree/report/admin.py @@ -3,14 +3,7 @@ from django.contrib import admin from .helpers import report_model_options -from .models import ( - LabelOutput, - LabelTemplate, - ReportAsset, - ReportOutput, - ReportSnippet, - ReportTemplate, -) +from .models import LabelTemplate, ReportAsset, ReportSnippet, ReportTemplate @admin.register(LabelTemplate) @@ -42,11 +35,3 @@ class ReportAssetAdmin(admin.ModelAdmin): """Admin class for the ReportAsset model.""" list_display = ('id', 'asset', 'description') - - -@admin.register(LabelOutput) -@admin.register(ReportOutput) -class TemplateOutputAdmin(admin.ModelAdmin): - """Admin class for the TemplateOutput model.""" - - list_display = ('id', 'output', 'progress', 'complete') diff --git a/src/backend/InvenTree/report/api.py b/src/backend/InvenTree/report/api.py index b90d68a6a8..c40a663f9f 100644 --- a/src/backend/InvenTree/report/api.py +++ b/src/backend/InvenTree/report/api.py @@ -18,9 +18,11 @@ import InvenTree.permissions import report.helpers import report.models import report.serializers -from InvenTree.api import BulkDeleteMixin, MetadataView -from InvenTree.filters import InvenTreeOrderingFilter, InvenTreeSearchFilter -from InvenTree.mixins import ListAPI, ListCreateAPI, RetrieveUpdateDestroyAPI +from common.models import DataOutput +from common.serializers import DataOutputSerializer +from InvenTree.api import MetadataView +from InvenTree.filters import InvenTreeSearchFilter +from InvenTree.mixins import ListCreateAPI, RetrieveUpdateDestroyAPI from plugin.builtin.labels.inventree_label import InvenTreeLabelPlugin @@ -161,8 +163,7 @@ class LabelPrint(GenericAPIView): if plugin_serializer: kwargs['plugin_serializer'] = plugin_serializer - serializer = super().get_serializer(*args, **kwargs) - return serializer + return super().get_serializer(*args, **kwargs) @method_decorator(never_cache) def post(self, request, *args, **kwargs): @@ -202,14 +203,17 @@ class LabelPrint(GenericAPIView): ): plugin_serializer.is_valid(raise_exception=True) - # Generate a new LabelOutput object to print against - output = report.models.LabelOutput.objects.create( - template=template, - plugin=plugin.slug, - user=request.user, + user = getattr(request, 'user', None) + + # Generate a new DataOutput object to print against + output = DataOutput.objects.create( + user=user if user and user.is_authenticated else None, + total=len(items_to_print), progress=0, - items=len(items_to_print), complete=False, + output_type=DataOutput.DataOutputTypes.LABEL, + plugin=plugin.slug, + template_name=template.name, output=None, ) @@ -224,9 +228,7 @@ class LabelPrint(GenericAPIView): output.refresh_from_db() - return Response( - report.serializers.LabelOutputSerializer(output).data, status=201 - ) + return Response(DataOutputSerializer(output).data, status=201) class LabelTemplateList(TemplatePermissionMixin, ListCreateAPI): @@ -274,18 +276,21 @@ class ReportPrint(GenericAPIView): """Print this report template against a number of provided items. This functionality is offloaded to the background worker process, - which will update the status of the ReportOutput object as it progresses. + which will update the status of the DataOutput object as it progresses. """ import report.tasks from InvenTree.tasks import offload_task - # Generate a new ReportOutput object - output = report.models.ReportOutput.objects.create( - template=template, - user=request.user, + user = getattr(request, 'user', None) + + # Generate a new DataOutput object + output = DataOutput.objects.create( + user=user if user and user.is_authenticated else None, + total=len(items_to_print), progress=0, - items=len(items_to_print), complete=False, + output_type=DataOutput.DataOutputTypes.REPORT, + template_name=template.name, output=None, ) @@ -296,9 +301,7 @@ class ReportPrint(GenericAPIView): output.refresh_from_db() - return Response( - report.serializers.ReportOutputSerializer(output).data, status=201 - ) + return Response(DataOutputSerializer(output).data, status=201) class ReportTemplateList(TemplatePermissionMixin, ListCreateAPI): @@ -347,44 +350,6 @@ class ReportAssetDetail(TemplatePermissionMixin, RetrieveUpdateDestroyAPI): serializer_class = report.serializers.ReportAssetSerializer -class TemplateOutputMixin: - """Mixin class for template output API endpoints.""" - - filter_backends = [InvenTreeOrderingFilter] - ordering_fields = ['created', 'model_type', 'user'] - ordering_field_aliases = {'model_type': 'template__model_type'} - - -class LabelOutputMixin(TemplatePermissionMixin, TemplateOutputMixin): - """Mixin class for a label output API endpoint.""" - - queryset = report.models.LabelOutput.objects.all() - serializer_class = report.serializers.LabelOutputSerializer - - -class LabelOutputList(LabelOutputMixin, BulkDeleteMixin, ListAPI): - """List endpoint for LabelOutput objects.""" - - -class LabelOutputDetail(LabelOutputMixin, RetrieveUpdateDestroyAPI): - """Detail endpoint for LabelOutput objects.""" - - -class ReportOutputMixin(TemplatePermissionMixin, TemplateOutputMixin): - """Mixin class for a report output API endpoint.""" - - queryset = report.models.ReportOutput.objects.all() - serializer_class = report.serializers.ReportOutputSerializer - - -class ReportOutputList(ReportOutputMixin, BulkDeleteMixin, ListAPI): - """List endpoint for ReportOutput objects.""" - - -class ReportOutputDetail(ReportOutputMixin, RetrieveUpdateDestroyAPI): - """Detail endpoint for ReportOutput objects.""" - - label_api_urls = [ # Printing endpoint path('print/', LabelPrint.as_view(), name='api-label-print'), @@ -411,16 +376,6 @@ label_api_urls = [ path('', LabelTemplateList.as_view(), name='api-label-template-list'), ]), ), - # Label outputs - path( - 'output/', - include([ - path( - '/', LabelOutputDetail.as_view(), name='api-label-output-detail' - ), - path('', LabelOutputList.as_view(), name='api-label-output-list'), - ]), - ), ] report_api_urls = [ @@ -449,18 +404,6 @@ report_api_urls = [ path('', ReportTemplateList.as_view(), name='api-report-template-list'), ]), ), - # Generated report outputs - path( - 'output/', - include([ - path( - '/', - ReportOutputDetail.as_view(), - name='api-report-output-detail', - ), - path('', ReportOutputList.as_view(), name='api-report-output-list'), - ]), - ), # Report assets path( 'asset/', diff --git a/src/backend/InvenTree/report/apps.py b/src/backend/InvenTree/report/apps.py index 33371ee0a5..debe7a7528 100644 --- a/src/backend/InvenTree/report/apps.py +++ b/src/backend/InvenTree/report/apps.py @@ -36,6 +36,8 @@ class ReportConfig(AppConfig): super().ready() + self.cleanup() + # skip loading if plugin registry is not loaded or we run in a background thread if ( not InvenTree.ready.isPluginRegistryLoaded() @@ -62,6 +64,15 @@ class ReportConfig(AppConfig): set_maintenance_mode(False) + def cleanup(self): + """Cleanup old label and report outputs.""" + try: + from report.tasks import cleanup_old_report_outputs + + cleanup_old_report_outputs() + except Exception: + pass + def file_from_template(self, dir_name: str, file_name: str) -> ContentFile: """Construct a new ContentFile from a template file.""" logger.info('Creating %s template file: %s', dir_name, file_name) diff --git a/src/backend/InvenTree/report/migrations/0029_remove_reportoutput_template_and_more.py b/src/backend/InvenTree/report/migrations/0029_remove_reportoutput_template_and_more.py new file mode 100644 index 0000000000..5d1858f5bd --- /dev/null +++ b/src/backend/InvenTree/report/migrations/0029_remove_reportoutput_template_and_more.py @@ -0,0 +1,27 @@ +# Generated by Django 4.2.19 on 2025-03-07 01:45 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ("report", "0028_labeltemplate_attach_to_model_and_more"), + ] + + operations = [ + migrations.RemoveField( + model_name="reportoutput", + name="template", + ), + migrations.RemoveField( + model_name="reportoutput", + name="user", + ), + migrations.DeleteModel( + name="LabelOutput", + ), + migrations.DeleteModel( + name="ReportOutput", + ), + ] diff --git a/src/backend/InvenTree/report/models.py b/src/backend/InvenTree/report/models.py index 7e97ac198a..a21f599bc3 100644 --- a/src/backend/InvenTree/report/models.py +++ b/src/backend/InvenTree/report/models.py @@ -5,7 +5,6 @@ import os import sys from django.conf import settings -from django.contrib.auth.models import User from django.core.exceptions import ValidationError from django.core.files.base import ContentFile from django.core.files.storage import default_storage @@ -25,10 +24,11 @@ import InvenTree.helpers import InvenTree.models import report.helpers import report.validators +from common.models import DataOutput from common.settings import get_global_setting from InvenTree.helpers_model import get_base_url from InvenTree.models import MetadataMixin -from plugin import InvenTreePlugin +from plugin import InvenTreePlugin, PluginMixinEnum from plugin.registry import registry try: @@ -338,7 +338,7 @@ class ReportTemplate(TemplateUploadMixin, ReportTemplateBase): } # Pass the context through to the plugin registry for any additional information - for plugin in registry.with_mixin('report'): + for plugin in registry.with_mixin(PluginMixinEnum.REPORT): try: plugin.add_report_context(self, instance, request, context) except Exception: @@ -348,16 +348,16 @@ class ReportTemplate(TemplateUploadMixin, ReportTemplateBase): return context - def print(self, items: list, request=None, output=None, **kwargs) -> 'ReportOutput': + def print(self, items: list, request=None, output=None, **kwargs) -> DataOutput: """Print reports for a list of items against this template. Arguments: items: A list of items to print reports for (model instance) - output: The ReportOutput object to use (if provided) + output: The DataOutput object to use (if provided) request: The request object (optional) Returns: - output: The ReportOutput object representing the generated report(s) + output: The DataOutput object representing the generated report(s) Raises: ValidationError: If there is an error during report printing @@ -369,8 +369,6 @@ class ReportTemplate(TemplateUploadMixin, ReportTemplateBase): Further work is required to allow the following extended features: - Render a single PDF file with the collated items (optional per template) - Render a raw file (do not convert to PDF) - allows for other file types - - Render using background worker, provide progress updates to UI - - Run report generation in the background worker process """ logger.info("Printing %s reports against template '%s'", len(items), self.name) @@ -381,18 +379,19 @@ class ReportTemplate(TemplateUploadMixin, ReportTemplateBase): # Start with a default report name report_name = None - report_plugins = registry.with_mixin('report') + report_plugins = registry.with_mixin(PluginMixinEnum.REPORT) - # If a ReportOutput object is not provided, create a new one + # If a DataOutput object is not provided, create a new one if not output: - output = ReportOutput.objects.create( - template=self, - items=len(items), + output = DataOutput.objects.create( + total=len(items), user=request.user if request and request.user and request.user.is_authenticated else None, progress=0, complete=False, + output_type=DataOutput.DataOutputTypes.REPORT, + template_name=self.name, output=None, ) @@ -547,7 +546,7 @@ class LabelTemplate(TemplateUploadMixin, ReportTemplateBase): context['page_style'] = self.generate_page_style() # Pass the context through to any registered plugins - plugins = registry.with_mixin('report') + plugins = registry.with_mixin(PluginMixinEnum.REPORT) for plugin in plugins: # Let each plugin add its own context data @@ -568,18 +567,18 @@ class LabelTemplate(TemplateUploadMixin, ReportTemplateBase): options=None, request=None, **kwargs, - ) -> 'LabelOutput': + ) -> DataOutput: """Print labels for a list of items against this template. Arguments: items: A list of items to print labels for (model instance) plugin: The plugin to use for label rendering - output: The LabelOutput object to use (if provided) + output: The DataOutput object to use (if provided) options: Additional options for the label printing plugin (optional) request: The request object (optional) Returns: - output: The LabelOutput object representing the generated label(s) + output: The DataOutput object representing the generated label(s) Raises: ValidationError: If there is an error during label printing @@ -592,15 +591,17 @@ class LabelTemplate(TemplateUploadMixin, ReportTemplateBase): ) if not output: - output = LabelOutput.objects.create( - template=self, - items=len(items), - plugin=plugin.slug, + output = DataOutput.objects.create( user=request.user if request and request.user.is_authenticated else None, + total=len(items), progress=0, complete=False, + output_type=DataOutput.DataOutputTypes.LABEL, + template_name=self.name, + plugin=plugin.slug, + output=None, ) if options is None: @@ -628,81 +629,6 @@ class LabelTemplate(TemplateUploadMixin, ReportTemplateBase): return output -class TemplateOutput(models.Model): - """Base class representing a generated file from a template. - - As reports (or labels) may take a long time to render, - this process is offloaded to the background worker process. - - The result is either a file made available for download, - or a message indicating that the output is handled externally. - """ - - class Meta: - """Metaclass options.""" - - abstract = True - - created = models.DateField(auto_now_add=True, editable=False) - - user = models.ForeignKey( - User, on_delete=models.SET_NULL, blank=True, null=True, related_name='+' - ) - - items = models.PositiveIntegerField( - default=0, verbose_name=_('Items'), help_text=_('Number of items to process') - ) - - complete = models.BooleanField( - default=False, - verbose_name=_('Complete'), - help_text=_('Report generation is complete'), - ) - - progress = models.PositiveIntegerField( - default=0, verbose_name=_('Progress'), help_text=_('Report generation progress') - ) - - -class ReportOutput(TemplateOutput): - """Class representing a generated report output file.""" - - template = models.ForeignKey( - ReportTemplate, on_delete=models.CASCADE, verbose_name=_('Report Template') - ) - - output = models.FileField( - upload_to='report/output', - blank=True, - null=True, - verbose_name=_('Output File'), - help_text=_('Generated output file'), - ) - - -class LabelOutput(TemplateOutput): - """Class representing a generated label output file.""" - - plugin = models.CharField( - max_length=100, - blank=True, - verbose_name=_('Plugin'), - help_text=_('Label output plugin'), - ) - - template = models.ForeignKey( - LabelTemplate, on_delete=models.CASCADE, verbose_name=_('Label Template') - ) - - output = models.FileField( - upload_to='label/output', - blank=True, - null=True, - verbose_name=_('Output File'), - help_text=_('Generated output file'), - ) - - class ReportSnippet(TemplateUploadMixin, models.Model): """Report template 'snippet' which can be used to make templates that can then be included in other reports. diff --git a/src/backend/InvenTree/report/serializers.py b/src/backend/InvenTree/report/serializers.py index 1c5e0dd4f1..fe542742ff 100644 --- a/src/backend/InvenTree/report/serializers.py +++ b/src/backend/InvenTree/report/serializers.py @@ -12,7 +12,6 @@ from InvenTree.serializers import ( InvenTreeAttachmentSerializerField, InvenTreeModelSerializer, ) -from users.serializers import UserSerializer class ReportSerializerBase(InvenTreeModelSerializer): @@ -161,59 +160,6 @@ class LabelTemplateSerializer(ReportSerializerBase): fields = [*ReportSerializerBase.base_fields(), 'width', 'height'] -class BaseOutputSerializer(InvenTreeModelSerializer): - """Base serializer class for template output.""" - - @staticmethod - def base_fields(): - """Basic field set.""" - return [ - 'pk', - 'created', - 'user', - 'user_detail', - 'model_type', - 'items', - 'complete', - 'progress', - 'output', - 'template', - ] - - output = InvenTreeAttachmentSerializerField() - model_type = serializers.CharField(source='template.model_type', read_only=True) - - user_detail = UserSerializer(source='user', read_only=True, many=False) - - -class LabelOutputSerializer(BaseOutputSerializer): - """Serializer class for the LabelOutput model.""" - - class Meta: - """Metaclass options.""" - - model = report.models.LabelOutput - fields = [*BaseOutputSerializer.base_fields(), 'plugin', 'template_detail'] - - template_detail = LabelTemplateSerializer( - source='template', many=False, read_only=True - ) - - -class ReportOutputSerializer(BaseOutputSerializer): - """Serializer class for the ReportOutput model.""" - - class Meta: - """Metaclass options.""" - - model = report.models.ReportOutput - fields = [*BaseOutputSerializer.base_fields(), 'template_detail'] - - template_detail = ReportTemplateSerializer( - source='template', many=False, read_only=True - ) - - class ReportSnippetSerializer(InvenTreeModelSerializer): """Serializer class for the ReportSnippet model.""" diff --git a/src/backend/InvenTree/report/tasks.py b/src/backend/InvenTree/report/tasks.py index ffddaa56fc..615f648aa6 100644 --- a/src/backend/InvenTree/report/tasks.py +++ b/src/backend/InvenTree/report/tasks.py @@ -1,43 +1,29 @@ """Background tasks for the report app.""" -from datetime import timedelta - import structlog from InvenTree.exceptions import log_error -from InvenTree.helpers import current_time -from InvenTree.tasks import ScheduledTask, scheduled_task -from report.models import LabelOutput, ReportOutput logger = structlog.get_logger('inventree') -@scheduled_task(ScheduledTask.DAILY) -def cleanup_old_report_outputs(): - """Remove old report/label outputs from the database.""" - # Remove any outputs which are older than 5 days - threshold = current_time() - timedelta(days=5) - - LabelOutput.objects.filter(created__lte=threshold).delete() - ReportOutput.objects.filter(created__lte=threshold).delete() - - def print_reports(template_id: int, item_ids: list[int], output_id: int, **kwargs): """Print multiple reports against the provided template. Arguments: template_id: The ID of the ReportTemplate to use item_ids: List of item IDs to generate the report against - output_id: The ID of the ReportOutput to use (if provided) + output_id: The ID of the DataOutput to use (if provided) This function is intended to be called by the background worker, - and will continuously update the status of the ReportOutput object. + and will continuously update the status of the DataOutput object. """ - from report.models import ReportOutput, ReportTemplate + from common.models import DataOutput + from report.models import ReportTemplate try: template = ReportTemplate.objects.get(pk=template_id) - output = ReportOutput.objects.get(pk=output_id) + output = DataOutput.objects.get(pk=output_id) except Exception: log_error('report.tasks.print_reports') return @@ -57,18 +43,19 @@ def print_labels( Arguments: template_id: The ID of the LabelTemplate to use item_ids: List of item IDs to generate the labels against - output_id: The ID of the LabelOutput to use (if provided) + output_id: The ID of the DataOutput to use (if provided) plugin_slug: The ID of the LabelPlugin to use (if provided) This function is intended to be called by the background worker, - and will continuously update the status of the LabelOutput object. + and will continuously update the status of the DataOutput object. """ + from common.models import DataOutput from plugin.registry import registry - from report.models import LabelOutput, LabelTemplate + from report.models import LabelTemplate try: template = LabelTemplate.objects.get(pk=template_id) - output = LabelOutput.objects.get(pk=output_id) + output = DataOutput.objects.get(pk=output_id) except Exception: log_error('report.tasks.print_labels') return diff --git a/src/backend/InvenTree/report/tests.py b/src/backend/InvenTree/report/tests.py index ee737fcc60..05333266ba 100644 --- a/src/backend/InvenTree/report/tests.py +++ b/src/backend/InvenTree/report/tests.py @@ -290,7 +290,7 @@ class ReportTest(InvenTreeAPITestCase): output = template.print(items) self.assertTrue(output.complete) - self.assertEqual(output.items, 5) + self.assertEqual(output.total, 5) self.assertIsNotNone(output.output) self.assertTrue(output.output.name.endswith('.pdf')) @@ -341,8 +341,9 @@ class LabelTest(InvenTreeAPITestCase): output = template.print(items=parts, plugin=plugin) self.assertTrue(output.complete) - self.assertEqual(output.items, 10) + self.assertEqual(output.total, 10) self.assertIsNotNone(output.output) + self.assertEqual(output.plugin, 'inventreelabel') self.assertTrue(output.output.name.endswith('.pdf')) @@ -461,7 +462,7 @@ class TestReportTest(PrintTestMixins, ReportTest): ) # There should be a link to the generated PDF - self.assertTrue(response.data['output'].startswith('/media/report/')) + self.assertTrue(response.data['output'].startswith('/media/data_output/')) self.assertTrue(response.data['output'].endswith('.pdf')) # By default, this should *not* have created an attachment against this stockitem diff --git a/src/backend/InvenTree/stock/api.py b/src/backend/InvenTree/stock/api.py index 386b6d1bc2..f298996779 100644 --- a/src/backend/InvenTree/stock/api.py +++ b/src/backend/InvenTree/stock/api.py @@ -25,8 +25,8 @@ from build.models import Build from build.serializers import BuildSerializer from company.models import Company, SupplierPart from company.serializers import CompanySerializer +from data_exporter.mixins import DataExportViewMixin from generic.states.api import StatusView -from importer.mixins import DataExportViewMixin from InvenTree.api import BulkUpdateMixin, ListCreateDestroyAPIView, MetadataView from InvenTree.filters import ( ORDER_FILTER_ALIAS, @@ -356,7 +356,7 @@ class StockLocationMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_queryset(self, *args, **kwargs): """Return annotated queryset for the StockLocationList endpoint.""" @@ -952,7 +952,7 @@ class StockApiMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class StockList(DataExportViewMixin, StockApiMixin, ListCreateDestroyAPIView): @@ -1253,7 +1253,7 @@ class StockItemTestResultMixin: kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) class StockItemTestResultDetail(StockItemTestResultMixin, RetrieveUpdateDestroyAPI): @@ -1394,7 +1394,7 @@ class StockTrackingList(DataExportViewMixin, ListAPI): kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + return super().get_serializer(*args, **kwargs) def get_delta_model_map(self) -> dict: """Return a mapping of delta models to their respective models and serializers. diff --git a/src/backend/InvenTree/stock/generators.py b/src/backend/InvenTree/stock/generators.py index f8487a0db6..7b11dc7db8 100644 --- a/src/backend/InvenTree/stock/generators.py +++ b/src/backend/InvenTree/stock/generators.py @@ -23,7 +23,7 @@ def generate_batch_code(**kwargs): Various kwargs can be passed to the function, which will be passed through to the plugin functions. """ # First, check if any plugins can generate batch codes - from plugin.registry import registry + from plugin import PluginMixinEnum, registry now = InvenTree.helpers.current_time() @@ -38,7 +38,7 @@ def generate_batch_code(**kwargs): **kwargs, } - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): generate = getattr(plugin, 'generate_batch_code', None) if not generate: diff --git a/src/backend/InvenTree/stock/models.py b/src/backend/InvenTree/stock/models.py index 0285cf9c51..ccc51008c2 100644 --- a/src/backend/InvenTree/stock/models.py +++ b/src/backend/InvenTree/stock/models.py @@ -530,12 +530,12 @@ class StockItem( This function hooks into the plugin system to allow for custom serial number conversion. """ - from plugin.registry import registry + from plugin import PluginMixinEnum, registry # First, let any plugins convert this serial number to an integer value # If a non-null value is returned (by any plugin) we will use that - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: serial_int = plugin.convert_serial_to_int(serial) except Exception: @@ -721,9 +721,9 @@ class StockItem( - Validation is performed by custom plugins. - By default, no validation checks are performed """ - from plugin.registry import registry + from plugin import PluginMixinEnum, registry - for plugin in registry.with_mixin('validation'): + for plugin in registry.with_mixin(PluginMixinEnum.VALIDATION): try: plugin.validate_batch_code(self.batch, self) except ValidationError as exc: diff --git a/src/backend/InvenTree/stock/serializers.py b/src/backend/InvenTree/stock/serializers.py index da7a941139..6f8669f853 100644 --- a/src/backend/InvenTree/stock/serializers.py +++ b/src/backend/InvenTree/stock/serializers.py @@ -28,8 +28,8 @@ import stock.filters import stock.status_codes from common.settings import get_global_setting from generic.states.fields import InvenTreeCustomStatusSerializerMixin -from importer.mixins import DataImportExportSerializerMixin from importer.registry import register_importer +from InvenTree.mixins import DataImportExportSerializerMixin from InvenTree.ready import isGeneratingSchema from InvenTree.serializers import InvenTreeCurrencySerializer, InvenTreeDecimalField from users.serializers import UserSerializer diff --git a/src/backend/InvenTree/stock/test_api.py b/src/backend/InvenTree/stock/test_api.py index 11891f3943..786d704cf2 100644 --- a/src/backend/InvenTree/stock/test_api.py +++ b/src/backend/InvenTree/stock/test_api.py @@ -1,17 +1,14 @@ """Unit testing for the Stock API.""" -import io import os import random from datetime import datetime, timedelta from enum import IntEnum -import django.http from django.contrib.contenttypes.models import ContentType from django.core.exceptions import ValidationError from django.urls import reverse -import tablib from djmoney.money import Money from rest_framework import status @@ -839,34 +836,9 @@ class StockItemListTest(StockAPITestCase): self.assertEqual(len(response['results']), n) - def export_data(self, filters=None): - """Helper to test exports.""" - if not filters: - filters = {} - - filters['export'] = 'csv' - - response = self.client.get(self.list_url, data=filters) - - self.assertEqual(response.status_code, 200) - - self.assertIsInstance(response, django.http.response.StreamingHttpResponse) - - file_object = io.StringIO(response.getvalue().decode('utf-8')) - - dataset = tablib.Dataset().load(file_object, 'csv', headers=True) - - return dataset - def test_export(self): """Test exporting of Stock data via the API.""" - dataset = self.export_data({}) - - # Check that *all* stock item objects have been exported - self.assertEqual(len(dataset), StockItem.objects.count()) - - # Expected headers - headers = [ + required_headers = [ 'Part', 'Customer', 'Stock Location', @@ -881,45 +853,40 @@ class StockItemListTest(StockAPITestCase): 'Supplier Part.MPN', ] - for h in headers: - self.assertIn(h, dataset.headers) - excluded_headers = ['metadata'] - for h in excluded_headers: - self.assertNotIn(h, dataset.headers) + with self.export_data(self.list_url) as data_file: + self.process_csv( + data_file, + required_cols=required_headers, + excluded_cols=excluded_headers, + required_rows=StockItem.objects.count(), + ) # Now, add a filter to the results - dataset = self.export_data({'location': 1, 'cascade': True}) + with self.export_data( + self.list_url, {'location': 1, 'cascade': True} + ) as data_file: + data = self.process_csv(data_file, required_rows=9) - self.assertEqual(len(dataset), 9) + for row in data: + item_id = int(row['ID']) + loc_id = int(row['Stock Location']) - # Read out the data - idx_id = dataset.headers.index('ID') - idx_loc = dataset.headers.index('Stock Location') - idx_loc_name = dataset.headers.index('Location.Name') - idx_part_name = dataset.headers.index('Part.Name') + item = StockItem.objects.get(pk=item_id) - for row in dataset: - item_id = int(row[idx_id]) - item = StockItem.objects.get(pk=item_id) + # Location should match ID + self.assertEqual(loc_id, item.location.pk) - loc_id = int(row[idx_loc]) + # Location name should match + self.assertEqual(row['Location.Name'], item.location.name) - # Location should match ID - self.assertEqual(int(loc_id), item.location.pk) - - # Location name should match - loc_name = row[idx_loc_name] - self.assertEqual(loc_name, item.location.name) - - # Part name should match - part_name = row[idx_part_name] - self.assertEqual(part_name, item.part.name) + # Part name should match + self.assertEqual(row['Part.Name'], item.part.name) # Export stock items with a specific part - dataset = self.export_data({'part': 25}) - self.assertEqual(len(dataset), 17) + with self.export_data(self.list_url, {'part': 25}) as data_file: + self.process_csv(data_file, required_rows=17) def test_filter_by_allocated(self): """Test that we can filter by "allocated" status. diff --git a/src/backend/InvenTree/users/api.py b/src/backend/InvenTree/users/api.py index 37e90be6ee..3fa33d4dda 100644 --- a/src/backend/InvenTree/users/api.py +++ b/src/backend/InvenTree/users/api.py @@ -190,7 +190,8 @@ class GroupMixin: params.get('permission_detail', None) ) kwargs['context'] = self.get_serializer_context() - return self.serializer_class(*args, **kwargs) + + return super().get_serializer(*args, **kwargs) class GroupDetail(GroupMixin, RetrieveUpdateDestroyAPI): diff --git a/src/backend/InvenTree/users/models.py b/src/backend/InvenTree/users/models.py index 7f49cc86cc..662090d385 100644 --- a/src/backend/InvenTree/users/models.py +++ b/src/backend/InvenTree/users/models.py @@ -222,10 +222,8 @@ class RuleSet(models.Model): 'auth_permission', 'users_apitoken', 'users_ruleset', - 'report_labeloutput', 'report_labeltemplate', 'report_reportasset', - 'report_reportoutput', 'report_reportsnippet', 'report_reporttemplate', 'account_emailaddress', @@ -336,6 +334,7 @@ class RuleSet(models.Model): # Models which currently do not require permissions 'common_attachment', 'common_customunit', + 'common_dataoutput', 'common_inventreesetting', 'common_inventreeusersetting', 'common_notificationentry', diff --git a/src/frontend/src/components/buttons/PrintingActions.tsx b/src/frontend/src/components/buttons/PrintingActions.tsx index 5bc136de63..afc63e38d1 100644 --- a/src/frontend/src/components/buttons/PrintingActions.tsx +++ b/src/frontend/src/components/buttons/PrintingActions.tsx @@ -1,19 +1,12 @@ import { t } from '@lingui/macro'; -import { notifications, showNotification } from '@mantine/notifications'; -import { - IconCircleCheck, - IconPrinter, - IconReport, - IconTags -} from '@tabler/icons-react'; +import { IconPrinter, IconReport, IconTags } from '@tabler/icons-react'; import { useQuery } from '@tanstack/react-query'; -import { useEffect, useMemo, useState } from 'react'; +import { useMemo, useState } from 'react'; import { api } from '../../App'; -import { useApi } from '../../contexts/ApiContext'; import { ApiEndpoints } from '../../enums/ApiEndpoints'; import type { ModelType } from '../../enums/ModelType'; import { extractAvailableFields } from '../../functions/forms'; -import { generateUrl } from '../../functions/urls'; +import useDataOutput from '../../hooks/UseDataOutput'; import { useCreateApiFormModal } from '../../hooks/UseForm'; import { apiUrl } from '../../states/ApiState'; import { @@ -22,94 +15,6 @@ import { } from '../../states/SettingsState'; import type { ApiFormFieldSet } from '../forms/fields/ApiFormField'; import { ActionDropdown } from '../items/ActionDropdown'; -import { ProgressBar } from '../items/ProgressBar'; - -/** - * Hook to track the progress of a printing operation - */ -function usePrintingProgress({ - title, - outputId, - endpoint -}: { - title: string; - outputId?: number; - endpoint: ApiEndpoints; -}) { - const api = useApi(); - - const [loading, setLoading] = useState(false); - - useEffect(() => { - if (!!outputId) { - setLoading(true); - showNotification({ - id: `printing-progress-${endpoint}-${outputId}`, - title: title, - loading: true, - autoClose: false, - withCloseButton: false, - message: - }); - } else { - setLoading(false); - } - }, [outputId, endpoint, title]); - - const progress = useQuery({ - enabled: !!outputId && loading, - refetchInterval: 750, - queryKey: ['printingProgress', endpoint, outputId], - queryFn: () => - api - .get(apiUrl(endpoint, outputId)) - .then((response) => { - const data = response?.data ?? {}; - - if (data.pk && data.pk == outputId) { - if (data.complete) { - setLoading(false); - notifications.hide(`printing-progress-${endpoint}-${outputId}`); - notifications.hide('print-success'); - - notifications.show({ - id: 'print-success', - title: t`Printing`, - message: t`Printing completed successfully`, - color: 'green', - icon: - }); - - if (data.output) { - const url = generateUrl(data.output); - window.open(url.toString(), '_blank'); - } - } else { - notifications.update({ - id: `printing-progress-${endpoint}-${outputId}`, - autoClose: false, - withCloseButton: false, - message: ( - - ) - }); - } - } - - return data; - }) - .catch(() => { - notifications.hide(`printing-progress-${endpoint}-${outputId}`); - setLoading(false); - return {}; - }) - }); -} export function PrintingActions({ items, @@ -142,16 +47,14 @@ export function PrintingActions({ const [labelId, setLabelId] = useState(undefined); const [reportId, setReportId] = useState(undefined); - const labelProgress = usePrintingProgress({ + const labelProgress = useDataOutput({ title: t`Printing Labels`, - outputId: labelId, - endpoint: ApiEndpoints.label_output + id: labelId }); - const reportProgress = usePrintingProgress({ + const reportProgress = useDataOutput({ title: t`Printing Reports`, - outputId: reportId, - endpoint: ApiEndpoints.report_output + id: reportId }); // Fetch available printing fields via OPTIONS request diff --git a/src/frontend/src/components/calendar/Calendar.tsx b/src/frontend/src/components/calendar/Calendar.tsx index 9d0e8b3909..9fd6c6d371 100644 --- a/src/frontend/src/components/calendar/Calendar.tsx +++ b/src/frontend/src/components/calendar/Calendar.tsx @@ -21,12 +21,12 @@ import { IconCalendarMonth, IconChevronLeft, IconChevronRight, + IconDownload, IconFilter } from '@tabler/icons-react'; import { useCallback, useState } from 'react'; import type { CalendarState } from '../../hooks/UseCalendar'; import { useLocalState } from '../../states/LocalState'; -import { DownloadAction } from '../../tables/DownloadAction'; import type { TableFilter } from '../../tables/Filter'; import { FilterSelectDrawer } from '../../tables/FilterSelectDrawer'; import { TableSearchInput } from '../../tables/Search'; @@ -35,7 +35,6 @@ import { ActionButton } from '../buttons/ActionButton'; import { StylishText } from '../items/StylishText'; export interface InvenTreeCalendarProps extends CalendarOptions { - downloadData?: (fileFormat: string) => void; enableDownload?: boolean; enableFilters?: boolean; enableSearch?: boolean; @@ -45,7 +44,6 @@ export interface InvenTreeCalendarProps extends CalendarOptions { } export default function Calendar({ - downloadData, enableDownload, enableFilters = false, enableSearch, @@ -88,6 +86,7 @@ export default function Calendar({ return ( <> + {state.exportModal.modal} {enableFilters && filters && (filters?.length ?? 0) > 0 && ( - + setFiltersVisible(!filtersVisible)} /> @@ -163,10 +162,14 @@ export default function Calendar({ )} {enableDownload && ( - + + + + + )} diff --git a/src/frontend/src/components/forms/ApiForm.tsx b/src/frontend/src/components/forms/ApiForm.tsx index 9ee6a4a60e..ca54e60ca6 100644 --- a/src/frontend/src/components/forms/ApiForm.tsx +++ b/src/frontend/src/components/forms/ApiForm.tsx @@ -80,6 +80,7 @@ export interface ApiFormProps { pk?: number | string; pk_field?: string; pathParams?: PathParams; + queryParams?: URLSearchParams; method?: 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE'; fields?: ApiFormFieldSet; focus?: string; @@ -123,8 +124,14 @@ export function OptionsApiForm({ const id = useId(pId); const url = useMemo( - () => constructFormUrl(props.url, props.pk, props.pathParams), - [props.url, props.pk, props.pathParams] + () => + constructFormUrl( + props.url, + props.pk, + props.pathParams, + props.queryParams + ), + [props.url, props.pk, props.pathParams, props.queryParams] ); const optionsQuery = useQuery({ @@ -252,7 +259,13 @@ export function ApiForm({ // Cache URL const url = useMemo( - () => constructFormUrl(props.url, props.pk, props.pathParams), + () => + constructFormUrl( + props.url, + props.pk, + props.pathParams, + props.queryParams + ), [props.url, props.pk, props.pathParams] ); @@ -445,6 +458,7 @@ export function ApiForm({ return api({ method: method, url: url, + params: method.toLowerCase() == 'get' ? jsonData : undefined, data: hasFiles ? formData : jsonData, timeout: timeout, headers: { diff --git a/src/frontend/src/components/forms/fields/ChoiceField.tsx b/src/frontend/src/components/forms/fields/ChoiceField.tsx index 432875fee0..d7dc0dcbd9 100644 --- a/src/frontend/src/components/forms/fields/ChoiceField.tsx +++ b/src/frontend/src/components/forms/fields/ChoiceField.tsx @@ -43,6 +43,11 @@ export function ChoiceField({ // Update form values when the selected value changes const onChange = useCallback( (value: any) => { + // Prevent blank values if the field is required + if (definition.required && !value) { + return; + } + field.onChange(value); // Run custom callback for this field (if provided) diff --git a/src/frontend/src/components/items/ProgressBar.tsx b/src/frontend/src/components/items/ProgressBar.tsx index 59224e8c90..dcbf5e51e7 100644 --- a/src/frontend/src/components/items/ProgressBar.tsx +++ b/src/frontend/src/components/items/ProgressBar.tsx @@ -6,6 +6,7 @@ export type ProgressBarProps = { maximum?: number; label?: string; progressLabel?: boolean; + animated?: boolean; size?: string; }; @@ -37,6 +38,7 @@ export function ProgressBar(props: Readonly) { color={progress < 100 ? 'orange' : progress > 100 ? 'blue' : 'green'} size={props.size ?? 'md'} radius='sm' + animated={props.animated} /> ); diff --git a/src/frontend/src/enums/ApiEndpoints.tsx b/src/frontend/src/enums/ApiEndpoints.tsx index 361c2cf29b..9c6689476a 100644 --- a/src/frontend/src/enums/ApiEndpoints.tsx +++ b/src/frontend/src/enums/ApiEndpoints.tsx @@ -67,6 +67,9 @@ export enum ApiEndpoints { barcode_unlink = 'barcode/unlink/', barcode_generate = 'barcode/generate/', + // Data output endpoints + data_output = 'data-output/', + // Data import endpoints import_session_list = 'importer/session/', import_session_accept_fields = 'importer/session/:id/accept_fields/', @@ -191,10 +194,8 @@ export enum ApiEndpoints { // Template API endpoints label_list = 'label/template/', label_print = 'label/print/', - label_output = 'label/output/', report_list = 'report/template/', report_print = 'report/print/', - report_output = 'report/output/', report_snippet = 'report/snippet/', report_asset = 'report/asset/', diff --git a/src/frontend/src/functions/forms.tsx b/src/frontend/src/functions/forms.tsx index c4e20ea6e1..02f7a40497 100644 --- a/src/frontend/src/functions/forms.tsx +++ b/src/frontend/src/functions/forms.tsx @@ -14,9 +14,16 @@ import { invalidResponse, permissionDenied } from './notifications'; export function constructFormUrl( url: ApiEndpoints | string, pk?: string | number, - pathParams?: PathParams + pathParams?: PathParams, + queryParams?: URLSearchParams ): string { - return apiUrl(url, pk, pathParams); + let formUrl = apiUrl(url, pk, pathParams); + + if (queryParams) { + formUrl += `?${queryParams.toString()}`; + } + + return formUrl; } /** diff --git a/src/frontend/src/hooks/UseCalendar.tsx b/src/frontend/src/hooks/UseCalendar.tsx index dfa163616e..9f9a0a78aa 100644 --- a/src/frontend/src/hooks/UseCalendar.tsx +++ b/src/frontend/src/hooks/UseCalendar.tsx @@ -7,7 +7,9 @@ import { api } from '../App'; import type { ApiEndpoints } from '../enums/ApiEndpoints'; import { showApiErrorMessage } from '../functions/notifications'; import { apiUrl } from '../states/ApiState'; +import useDataExport from './UseDataExport'; import { type FilterSetState, useFilterSet } from './UseFilterSet'; +import type { UseModalReturn } from './UseModal'; /* * Type definition for representing the state of a calendar: @@ -44,6 +46,7 @@ export type CalendarState = { currentMonth: () => void; selectMonth: (date: DateValue) => void; query: UseQueryResult; + exportModal: UseModalReturn; data: any; }; @@ -69,7 +72,7 @@ export default function useCalendar({ const [endDate, setEndDate] = useState(null); // Generate a set of API query filters - const queryFilters = useMemo(() => { + const queryFilters: Record = useMemo(() => { // Expand date range by one month, to ensure we capture all events let params = { @@ -144,6 +147,13 @@ export default function useCalendar({ [ref] ); + // Modal for exporting data from the calendar + const exportModal = useDataExport({ + url: apiUrl(endpoint), + enabled: true, + filters: queryFilters + }); + return { name, filterSet, @@ -160,6 +170,7 @@ export default function useCalendar({ setStartDate, endDate, setEndDate, + exportModal, query: query, data: query.data }; diff --git a/src/frontend/src/hooks/UseDataExport.tsx b/src/frontend/src/hooks/UseDataExport.tsx new file mode 100644 index 0000000000..d4ce63f7df --- /dev/null +++ b/src/frontend/src/hooks/UseDataExport.tsx @@ -0,0 +1,125 @@ +import { t } from '@lingui/macro'; +import { useQuery } from '@tanstack/react-query'; +import { useMemo, useState } from 'react'; +import type { ApiFormFieldSet } from '../components/forms/fields/ApiFormField'; +import { useApi } from '../contexts/ApiContext'; +import { extractAvailableFields } from '../functions/forms'; +import useDataOutput from './UseDataOutput'; +import { useCreateApiFormModal } from './UseForm'; + +/** + * Custom hook for managing data export functionality + * This is intended to be used from a table or calendar view, + * to export the data displayed in the table or calendar + */ +export default function useDataExport({ + url, + enabled, + filters, + searchTerm +}: { + url: string; + enabled: boolean; + filters: any; + searchTerm?: string; +}) { + const api = useApi(); + + // Selected plugin to use for data export + const [pluginKey, setPluginKey] = useState('inventree-exporter'); + + const [exportId, setExportId] = useState(undefined); + + const progress = useDataOutput({ + title: t`Exporting Data`, + id: exportId + }); + + // Construct a set of export parameters + const exportParams = useMemo(() => { + const queryParams: Record = { + export: true + }; + + if (!!pluginKey) { + queryParams.export_plugin = pluginKey; + } + + // Add in any additional parameters which have a defined value + for (const [key, value] of Object.entries(filters ?? {})) { + if (value != undefined) { + queryParams[key] = value; + } + } + + if (!!searchTerm) { + queryParams.search = searchTerm; + } + + return queryParams; + }, [pluginKey, filters, searchTerm]); + + // Fetch available export fields via OPTIONS request + const extraExportFields = useQuery({ + enabled: !!url && enabled, + queryKey: ['export-fields', pluginKey, url, exportParams], + gcTime: 500, + queryFn: () => + api + .options(url, { + params: exportParams + }) + .then((response: any) => { + return extractAvailableFields(response, 'GET') || {}; + }) + .catch(() => { + return {}; + }) + }); + + // Construct a field set for the export form + const exportFields: ApiFormFieldSet = useMemo(() => { + const extraFields: ApiFormFieldSet = extraExportFields.data || {}; + + const fields: ApiFormFieldSet = { + export_format: {}, + export_plugin: {}, + ...extraFields + }; + + fields.export_format = { + ...fields.export_format, + required: true + }; + + fields.export_plugin = { + ...fields.export_plugin, + required: true, + onValueChange: (value: string) => { + if (!!value) { + setPluginKey(value); + } + } + }; + + return fields; + }, [extraExportFields]); + + // Modal for exporting data + const exportModal = useCreateApiFormModal({ + url: url, + queryParams: new URLSearchParams(exportParams), + title: t`Export Data`, + method: 'GET', + fields: exportFields, + submitText: t`Export`, + successMessage: null, + timeout: 30 * 1000, + onFormSuccess: (response: any) => { + setExportId(response.pk); + setPluginKey('inventree-exporter'); + } + }); + + return exportModal; +} diff --git a/src/frontend/src/hooks/UseDataOutput.tsx b/src/frontend/src/hooks/UseDataOutput.tsx new file mode 100644 index 0000000000..45c0d4d94d --- /dev/null +++ b/src/frontend/src/hooks/UseDataOutput.tsx @@ -0,0 +1,109 @@ +import { t } from '@lingui/macro'; +import { notifications, showNotification } from '@mantine/notifications'; +import { IconCircleCheck } from '@tabler/icons-react'; +import { useQuery } from '@tanstack/react-query'; +import { useEffect, useState } from 'react'; +import { ProgressBar } from '../components/items/ProgressBar'; +import { useApi } from '../contexts/ApiContext'; +import { ApiEndpoints } from '../enums/ApiEndpoints'; +import { generateUrl } from '../functions/urls'; +import { apiUrl } from '../states/ApiState'; + +/** + * Hook for monitoring a data output process running on the server + */ +export default function useDataOutput({ + title, + id +}: { + title: string; + id?: number; +}) { + const api = useApi(); + + const [loading, setLoading] = useState(false); + + useEffect(() => { + if (!!id) { + setLoading(true); + showNotification({ + id: `data-output-${id}`, + title: title, + loading: true, + autoClose: false, + withCloseButton: false, + message: + }); + } else setLoading(false); + }, [id, title]); + + const progress = useQuery({ + enabled: !!id && loading, + refetchInterval: 500, + queryKey: ['data-output', id, title], + queryFn: () => + api + .get(apiUrl(ApiEndpoints.data_output, id)) + .then((response) => { + const data = response?.data ?? {}; + + if (data.complete) { + setLoading(false); + notifications.update({ + id: `data-output-${id}`, + loading: false, + autoClose: 2500, + title: title, + message: t`Process completed successfully`, + color: 'green', + icon: + }); + + if (data.output) { + const url = generateUrl(data.output); + window.open(url.toString(), '_blank'); + } + } else if (!!data.error) { + setLoading(false); + notifications.update({ + id: `data-output-${id}`, + loading: false, + autoClose: 2500, + title: title, + message: t`Process failed`, + color: 'red' + }); + } else { + notifications.update({ + id: `data-output-${id}`, + loading: true, + autoClose: false, + withCloseButton: false, + message: ( + 0} + animated + /> + ) + }); + } + + return data; + }) + .catch(() => { + setLoading(false); + notifications.update({ + id: `data-output-${id}`, + loading: false, + autoClose: 2500, + title: title, + message: t`Process failed`, + color: 'red' + }); + return {}; + }) + }); +} diff --git a/src/frontend/src/hooks/UseForm.tsx b/src/frontend/src/hooks/UseForm.tsx index 0a8e8c7cb4..c61fb9293b 100644 --- a/src/frontend/src/hooks/UseForm.tsx +++ b/src/frontend/src/hooks/UseForm.tsx @@ -108,7 +108,7 @@ export function useEditApiFormModal(props: ApiFormModalProps) { props.successMessage === null ? null : (props.successMessage ?? t`Item Updated`), - method: 'PATCH' + method: props.method ?? 'PATCH' }), [props] ); @@ -158,7 +158,7 @@ export function useDeleteApiFormModal(props: ApiFormModalProps) { const deleteProps = useMemo( () => ({ ...props, - method: 'DELETE', + method: props.method ?? 'DELETE', submitText: t`Delete`, submitColor: 'red', successMessage: diff --git a/src/frontend/src/hooks/UseModal.tsx b/src/frontend/src/hooks/UseModal.tsx index 16e9736ffd..bb1ab82852 100644 --- a/src/frontend/src/hooks/UseModal.tsx +++ b/src/frontend/src/hooks/UseModal.tsx @@ -15,7 +15,14 @@ export interface UseModalProps { closeOnClickOutside?: boolean; } -export function useModal(props: UseModalProps) { +export interface UseModalReturn { + open: () => void; + close: () => void; + toggle: () => void; + modal: React.ReactElement; +} + +export function useModal(props: UseModalProps): UseModalReturn { const onOpen = useCallback(() => { props.onOpen?.(); }, [props.onOpen]); diff --git a/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx b/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx index a610d8830b..071e61a4a8 100644 --- a/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx +++ b/src/frontend/src/pages/Index/Settings/AdminCenter/Index.tsx @@ -6,6 +6,7 @@ import { IconCpu, IconDevicesPc, IconExclamationCircle, + IconFileDownload, IconFileUpload, IconList, IconListDetails, @@ -69,7 +70,11 @@ const BarcodeScanHistoryTable = Loadable( lazy(() => import('../../../../tables/settings/BarcodeScanHistoryTable')) ); -const ImportSesssionTable = Loadable( +const ExportSessionTable = Loadable( + lazy(() => import('../../../../tables/settings/ExportSessionTable')) +); + +const ImportSessionTable = Loadable( lazy(() => import('../../../../tables/settings/ImportSessionTable')) ); @@ -114,7 +119,13 @@ export default function AdminCenter() { name: 'import', label: t`Data Import`, icon: , - content: + content: + }, + { + name: 'export', + label: t`Data Export`, + icon: , + content: }, { name: 'barcode-history', diff --git a/src/frontend/src/pages/Index/Settings/AdminCenter/LabelTemplatePanel.tsx b/src/frontend/src/pages/Index/Settings/AdminCenter/LabelTemplatePanel.tsx index 5bfc1c2339..36615b5f93 100644 --- a/src/frontend/src/pages/Index/Settings/AdminCenter/LabelTemplatePanel.tsx +++ b/src/frontend/src/pages/Index/Settings/AdminCenter/LabelTemplatePanel.tsx @@ -1,12 +1,6 @@ -import { t } from '@lingui/macro'; -import { Accordion } from '@mantine/core'; -import { StylishText } from '../../../../components/items/StylishText'; import { ApiEndpoints } from '../../../../enums/ApiEndpoints'; import { ModelType } from '../../../../enums/ModelType'; -import { - TemplateOutputTable, - TemplateTable -} from '../../../../tables/settings/TemplateTable'; +import { TemplateTable } from '../../../../tables/settings/TemplateTable'; function LabelTemplateTable() { return ( @@ -25,27 +19,5 @@ function LabelTemplateTable() { } export default function LabelTemplatePanel() { - return ( - - - - {t`Label Templates`} - - - - - - - - {t`Generated Labels`} - - - - - - - ); + return ; } diff --git a/src/frontend/src/pages/Index/Settings/AdminCenter/ReportTemplatePanel.tsx b/src/frontend/src/pages/Index/Settings/AdminCenter/ReportTemplatePanel.tsx index 7cd05e1c2f..dd00b7a520 100644 --- a/src/frontend/src/pages/Index/Settings/AdminCenter/ReportTemplatePanel.tsx +++ b/src/frontend/src/pages/Index/Settings/AdminCenter/ReportTemplatePanel.tsx @@ -1,14 +1,8 @@ import { t } from '@lingui/macro'; - -import { Accordion } from '@mantine/core'; import { YesNoButton } from '../../../../components/buttons/YesNoButton'; -import { StylishText } from '../../../../components/items/StylishText'; import { ApiEndpoints } from '../../../../enums/ApiEndpoints'; import { ModelType } from '../../../../enums/ModelType'; -import { - TemplateOutputTable, - TemplateTable -} from '../../../../tables/settings/TemplateTable'; +import { TemplateTable } from '../../../../tables/settings/TemplateTable'; function ReportTemplateTable() { return ( @@ -40,24 +34,5 @@ function ReportTemplateTable() { } export default function ReportTemplatePanel() { - return ( - - - - {t`Report Templates`} - - - - - - - - {t`Generated Reports`} - - - - - - - ); + return ; } diff --git a/src/frontend/src/tables/DownloadAction.tsx b/src/frontend/src/tables/DownloadAction.tsx deleted file mode 100644 index 239aa1c219..0000000000 --- a/src/frontend/src/tables/DownloadAction.tsx +++ /dev/null @@ -1,42 +0,0 @@ -import { t } from '@lingui/macro'; -import { - IconDownload, - IconFileSpreadsheet, - IconFileText, - IconFileTypeCsv -} from '@tabler/icons-react'; -import { useMemo } from 'react'; - -import { - ActionDropdown, - type ActionDropdownItem -} from '../components/items/ActionDropdown'; - -export function DownloadAction({ - downloadCallback -}: Readonly<{ - downloadCallback?: (fileFormat: string) => void; -}>) { - const formatOptions = [ - { value: 'csv', label: t`CSV`, icon: }, - { value: 'tsv', label: t`TSV`, icon: }, - { value: 'xlsx', label: t`Excel (.xlsx)`, icon: } - ]; - - const actions: ActionDropdownItem[] = useMemo(() => { - return formatOptions.map((format) => ({ - name: format.label, - icon: format.icon, - onClick: () => downloadCallback?.(format.value) - })); - }, [formatOptions, downloadCallback]); - - return ( - } - actions={actions} - /> - ); -} diff --git a/src/frontend/src/tables/InvenTreeTableHeader.tsx b/src/frontend/src/tables/InvenTreeTableHeader.tsx index f9bce56344..6afe3e5f9a 100644 --- a/src/frontend/src/tables/InvenTreeTableHeader.tsx +++ b/src/frontend/src/tables/InvenTreeTableHeader.tsx @@ -9,6 +9,7 @@ import { } from '@mantine/core'; import { IconBarcode, + IconDownload, IconExclamationCircle, IconFilter, IconRefresh, @@ -22,11 +23,10 @@ import { Boundary } from '../components/Boundary'; import { ActionButton } from '../components/buttons/ActionButton'; import { ButtonMenu } from '../components/buttons/ButtonMenu'; import { PrintingActions } from '../components/buttons/PrintingActions'; -import { useApi } from '../contexts/ApiContext'; +import useDataExport from '../hooks/UseDataExport'; import { useDeleteApiFormModal } from '../hooks/UseForm'; import type { TableState } from '../hooks/UseTable'; import { TableColumnSelect } from './ColumnSelect'; -import { DownloadAction } from './DownloadAction'; import type { TableFilter } from './Filter'; import { FilterSelectDrawer } from './FilterSelectDrawer'; import type { InvenTreeTableProps } from './InvenTreeTable'; @@ -52,48 +52,43 @@ export default function InvenTreeTableHeader({ filters: TableFilter[]; toggleColumn: (column: string) => void; }>) { - const api = useApi(); - // Filter list visibility const [filtersVisible, setFiltersVisible] = useState(false); - const downloadData = (fileFormat: string) => { - // Download entire dataset (no pagination) + // Construct export filters + const exportFilters = useMemo(() => { + const filters: Record = {}; - const queryParams = { - ...tableProps.params - }; + // Add in any additional parameters which have a defined value + for (const [key, value] of Object.entries(tableProps.params ?? {})) { + if (value != undefined) { + filters[key] = value; + } + } // Add in active filters if (tableState.filterSet.activeFilters) { tableState.filterSet.activeFilters.forEach((filter) => { - queryParams[filter.name] = filter.value; + filters[filter.name] = filter.value; }); } // Allow overriding of query parameters if (tableState.queryFilters) { for (const [key, value] of tableState.queryFilters) { - queryParams[key] = value; + if (value != undefined) { + filters[key] = value; + } } } + }, [tableProps.params, tableState.filterSet, tableState.queryFilters]); - // Add custom search term - if (tableState.searchTerm) { - queryParams.search = tableState.searchTerm; - } - - // Specify file format - queryParams.export = fileFormat; - - const downloadUrl = api.getUri({ - url: tableUrl, - params: queryParams - }); - - // Download file in a new window (to force download) - window.open(downloadUrl, '_blank'); - }; + const exportModal = useDataExport({ + url: tableUrl ?? '', + enabled: !!tableUrl && tableProps?.enableDownload != false, + filters: exportFilters, + searchTerm: tableState.searchTerm + }); const deleteRecords = useDeleteApiFormModal({ url: tableUrl ?? '', @@ -149,6 +144,7 @@ export default function InvenTreeTableHeader({ return ( <> + {exportModal.modal} {deleteRecords.modal} {tableProps.enableFilters && (filters.length ?? 0) > 0 && ( @@ -245,11 +241,12 @@ export default function InvenTreeTableHeader({ )} - {tableProps.enableDownload && ( - + {tableUrl && tableProps.enableDownload && ( + + + + + )} diff --git a/src/frontend/src/tables/settings/ExportSessionTable.tsx b/src/frontend/src/tables/settings/ExportSessionTable.tsx new file mode 100644 index 0000000000..ef242349ab --- /dev/null +++ b/src/frontend/src/tables/settings/ExportSessionTable.tsx @@ -0,0 +1,60 @@ +import { t } from '@lingui/macro'; +import { useMemo } from 'react'; +import { AttachmentLink } from '../../components/items/AttachmentLink'; +import { RenderUser } from '../../components/render/User'; +import { ApiEndpoints } from '../../enums/ApiEndpoints'; +import { useTable } from '../../hooks/UseTable'; +import { apiUrl } from '../../states/ApiState'; +import type { TableColumn } from '../Column'; +import { DateColumn } from '../ColumnRenderers'; +import { InvenTreeTable } from '../InvenTreeTable'; + +export default function ExportSessionTable() { + const table = useTable('exportsession'); + + const columns: TableColumn[] = useMemo(() => { + return [ + { + accessor: 'output', + sortable: false, + render: (record: any) => + }, + { + accessor: 'output_type', + title: t`Output Type`, + sortable: true + }, + { + accessor: 'plugin', + title: t`Plugin`, + sortable: true + }, + DateColumn({ + accessor: 'created', + title: t`Exported On`, + sortable: true + }), + { + accessor: 'user', + sortable: true, + title: t`User`, + render: (record: any) => RenderUser({ instance: record.user_detail }) + } + ]; + }, []); + + return ( + <> + + + ); +} diff --git a/src/frontend/src/tables/settings/ImportSessionTable.tsx b/src/frontend/src/tables/settings/ImportSessionTable.tsx index 12fe574683..4ffd0a6ed7 100644 --- a/src/frontend/src/tables/settings/ImportSessionTable.tsx +++ b/src/frontend/src/tables/settings/ImportSessionTable.tsx @@ -22,7 +22,7 @@ import { StatusFilterOptions, type TableFilter, UserFilter } from '../Filter'; import { InvenTreeTable } from '../InvenTreeTable'; import { type RowAction, RowDeleteAction } from '../RowActions'; -export default function ImportSesssionTable() { +export default function ImportSessionTable() { const table = useTable('importsession'); const [opened, setOpened] = useState(false); @@ -71,6 +71,7 @@ export default function ImportSesssionTable() { { accessor: 'user', sortable: false, + title: t`User`, render: (record: any) => RenderUser({ instance: record.user_detail }) }, { diff --git a/src/frontend/src/tables/settings/TemplateTable.tsx b/src/frontend/src/tables/settings/TemplateTable.tsx index 52c71fdded..162a091646 100644 --- a/src/frontend/src/tables/settings/TemplateTable.tsx +++ b/src/frontend/src/tables/settings/TemplateTable.tsx @@ -42,7 +42,7 @@ import { useTable } from '../../hooks/UseTable'; import { apiUrl } from '../../states/ApiState'; import { useUserState } from '../../states/UserState'; import type { TableColumn } from '../Column'; -import { BooleanColumn, DateColumn } from '../ColumnRenderers'; +import { BooleanColumn } from '../ColumnRenderers'; import type { TableFilter } from '../Filter'; import { InvenTreeTable } from '../InvenTreeTable'; import { @@ -401,78 +401,3 @@ export function TemplateTable({ ); } - -export function TemplateOutputTable({ - endpoint, - withPlugins = false -}: { - endpoint: ApiEndpoints; - withPlugins?: boolean; -}) { - const table = useTable(`${endpoint}-output`); - - const tableColumns: TableColumn[] = useMemo(() => { - return [ - { - accessor: 'output', - sortable: false, - switchable: false, - title: t`Report Output`, - noWrap: true, - noContext: true, - render: (record: any) => { - if (record.output) { - return ; - } else { - return '-'; - } - } - }, - { - accessor: 'template_detail.name', - sortable: false, - switchable: false, - title: t`Template` - }, - { - accessor: 'model_type', - sortable: true, - switchable: false, - title: t`Model Type` - }, - DateColumn({ - accessor: 'created', - title: t`Creation Date`, - switchable: false, - sortable: true - }), - { - accessor: 'plugin', - title: t`Plugin`, - hidden: !withPlugins - }, - { - accessor: 'user_detail.username', - sortable: true, - ordering: 'user', - title: t`Created By` - } - ]; - }, [withPlugins]); - - return ( - <> - - - ); -} diff --git a/src/frontend/tests/pages/pui_build.spec.ts b/src/frontend/tests/pages/pui_build.spec.ts index 0925f9e69a..d697c217cb 100644 --- a/src/frontend/tests/pages/pui_build.spec.ts +++ b/src/frontend/tests/pages/pui_build.spec.ts @@ -97,6 +97,11 @@ test('Build Order - Calendar', async ({ page }) => { await navigate(page, 'manufacturing/index/buildorders'); await activateCalendarView(page); + // Export calendar data + await page.getByLabel('calendar-export-data').click(); + await page.getByRole('button', { name: 'Export', exact: true }).click(); + await page.getByText('Process completed successfully').waitFor(); + // Check "part category" filter await page.getByLabel('calendar-select-filters').click(); await page.getByRole('button', { name: 'Add Filter' }).click(); @@ -104,6 +109,9 @@ test('Build Order - Calendar', async ({ page }) => { await page.getByRole('option', { name: 'Category', exact: true }).click(); await page.getByLabel('related-field-filter-category').click(); await page.getByText('Part category, level 1').waitFor(); + + // Required because we downloaded a file + await page.context().close(); }); test('Build Order - Edit', async ({ page }) => { diff --git a/src/frontend/tests/pui_exporting.spec.ts b/src/frontend/tests/pui_exporting.spec.ts new file mode 100644 index 0000000000..0ae18f96ad --- /dev/null +++ b/src/frontend/tests/pui_exporting.spec.ts @@ -0,0 +1,120 @@ +import test from '@playwright/test'; +import { globalSearch, loadTab, navigate } from './helpers'; +import { doQuickLogin } from './login'; + +// Helper function to open the export data dialog +const openExportDialog = async (page) => { + await page.waitForLoadState('networkidle'); + await page.getByLabel('table-export-data').click(); + await page.getByText('Export Format *', { exact: true }).waitFor(); + await page.getByText('Export Plugin *', { exact: true }).waitFor(); +}; + +// Test data export for various order types +test('Exporting - Orders', async ({ page }) => { + await doQuickLogin(page, 'steven', 'wizardstaff'); + + // Download list of purchase orders + await navigate(page, 'purchasing/index/purchase-orders'); + + await openExportDialog(page); + + // // Select export format + await page.getByLabel('choice-field-export_format').click(); + await page.getByRole('option', { name: 'Excel' }).click(); + + // // Select export plugin (should only be one option here) + await page.getByLabel('choice-field-export_plugin').click(); + await page.getByRole('option', { name: 'InvenTree Exporter' }).click(); + + // // Export the data + await page.getByRole('button', { name: 'Export', exact: true }).click(); + await page.getByText('Process completed successfully').waitFor(); + + // Download list of purchase order items + await page.getByRole('cell', { name: 'PO0011' }).click(); + await loadTab(page, 'Line Items'); + await openExportDialog(page); + await page.getByRole('button', { name: 'Export', exact: true }).click(); + await page.getByText('Process completed successfully').waitFor(); + + // Download a list of build orders + await navigate(page, 'manufacturing/index/buildorders/'); + await openExportDialog(page); + await page.getByRole('button', { name: 'Export', exact: true }).click(); + await page.getByText('Process completed successfully').waitFor(); + + // Finally, navigate to the admin center and ensure the export data is available + await navigate(page, 'settings/admin/export/'); + + // Check for expected outputs + await page + .getByRole('link', { name: /InvenTree_Build_.*\.csv/ }) + .first() + .waitFor(); + await page + .getByRole('link', { name: /InvenTree_PurchaseOrder_.*\.xlsx/ }) + .first() + .waitFor(); + await page + .getByRole('link', { name: /InvenTree_PurchaseOrderLineItem_.*\.csv/ }) + .first() + .waitFor(); + + // Delete all exported file outputs + await page.getByRole('cell', { name: 'Select all records' }).click(); + await page.getByLabel('action-button-delete-selected').click(); + await page.getByRole('button', { name: 'Delete', exact: true }).click(); + await page.getByText('Items Deleted').waitFor(); +}); + +// Test for custom BOM exporter +test('Exporting - BOM', async ({ page }) => { + await doQuickLogin(page, 'steven', 'wizardstaff'); + + await globalSearch(page, 'MAST'); + await page.getByLabel('search-group-results-part').locator('a').click(); + await page.waitForLoadState('networkidle'); + await loadTab(page, 'Bill of Materials'); + await openExportDialog(page); + + // Select export format + await page.getByLabel('choice-field-export_format').click(); + await page.getByRole('option', { name: 'TSV' }).click(); + await page.waitForLoadState('networkidle'); + + // Select BOM plugin + await page.getByLabel('choice-field-export_plugin').click(); + await page.getByRole('option', { name: 'BOM Exporter' }).click(); + await page.waitForLoadState('networkidle'); + + // Now, adjust the settings specific to the BOM exporter + await page.getByLabel('number-field-export_levels').fill('3'); + await page + .locator('label') + .filter({ hasText: 'Pricing DataInclude part' }) + .locator('span') + .nth(1) + .click(); + await page + .locator('label') + .filter({ hasText: 'Parameter DataInclude part' }) + .locator('span') + .nth(1) + .click(); + + await page.getByRole('button', { name: 'Export', exact: true }).click(); + await page.getByText('Process completed successfully').waitFor(); + + // Finally, navigate to the admin center and ensure the export data is available + await navigate(page, 'settings/admin/export/'); + + await page.getByRole('cell', { name: 'bom-exporter' }).first().waitFor(); + await page + .getByRole('link', { name: /InvenTree_BomItem_.*\.tsv/ }) + .first() + .waitFor(); + + // Required because we downloaded a file + await page.context().close(); +}); diff --git a/src/frontend/tests/pui_printing.spec.ts b/src/frontend/tests/pui_printing.spec.ts index ae50369b9a..573ae1ba4d 100644 --- a/src/frontend/tests/pui_printing.spec.ts +++ b/src/frontend/tests/pui_printing.spec.ts @@ -41,7 +41,7 @@ test('Label Printing', async ({ page }) => { await page.getByRole('button', { name: 'Print', exact: true }).isEnabled(); await page.getByRole('button', { name: 'Print', exact: true }).click(); - await page.getByText('Printing completed successfully').first().waitFor(); + await page.getByText('Process completed successfully').first().waitFor(); await page.context().close(); }); @@ -77,7 +77,7 @@ test('Report Printing', async ({ page }) => { await page.getByRole('button', { name: 'Print', exact: true }).isEnabled(); await page.getByRole('button', { name: 'Print', exact: true }).click(); - await page.getByText('Printing completed successfully').first().waitFor(); + await page.getByText('Process completed successfully').first().waitFor(); await page.context().close(); }); diff --git a/tasks.py b/tasks.py index 3d85d49940..228d6c8d94 100644 --- a/tasks.py +++ b/tasks.py @@ -169,12 +169,12 @@ def content_excludes( allow_plugins: bool = True, allow_sso: bool = True, ): - """Returns a list of content types to exclude from import/export. + """Returns a list of content types to exclude from import / export. Arguments: - allow_tokens (bool): Allow tokens to be exported/importe - allow_plugins (bool): Allow plugin information to be exported/imported - allow_sso (bool): Allow SSO tokens to be exported/imported + allow_tokens (bool): Allow tokens to be exported / imported + allow_plugins (bool): Allow plugin information to be exported / imported + allow_sso (bool): Allow SSO tokens to be exported / imported """ excludes = [ 'contenttypes', @@ -186,13 +186,12 @@ def content_excludes( 'django_q.ormq', 'exchange.rate', 'exchange.exchangebackend', + 'common.dataoutput', 'common.notificationentry', 'common.notificationmessage', 'importer.dataimportsession', 'importer.dataimportcolumnmap', 'importer.dataimportrow', - 'report.labeloutput', - 'report.reportoutput', ] # Optionally exclude user auth data