[WIP] Data importer (#6911)

* Adds new model for DataImportSession

* Add file extension validation

Expose to admin interface also

* Switch to new 'importer' app

* Refactoring to help prevent circular imports

* Add serializer registry

- Use @register_importer tag for any serializer class

* Cleanup migration file

- Do not use one-time hard-coded values here

* Refactor code into registry.py

* Add validation for the uploaded file

- Must be importable by tablib

* Refactoring

* Adds property to retrieve matching serializer class

* Update helper functions

* Add hook to auto-assign columns on initial creation

* Rename field

* Enforce initial status value

* Add model for individual rows in the data import

* Add DataImportRow model

* Extract data rows as dict

* Update fields

- Remove "progress" field (will be calculated)
- Added "timestamp" field
- Added "complete" field to DataImportRow

* Auto-map column names

- Provide "sensible" default values

* Add API endpoint for DataImportSession

* Offload data import operation

- For large data files this may take a significant amount of time
- Offload it to the background worker process

* Refactor data import code

* Update models

- Add "columns" field to DataImportSession
- Add "errors" field to DataImportRow

* Move field mapping to a new model type

- Simpler validation

* Save "valid" status for each data row

* Include session defaults when validating row data

* Update content_excludes

- Ignore importer models in import/export

* Remove port from ALLOWED_HOST entries

* Skip table events for importer models

* Bug fixes

* Serializer updates

* Add more endpoints

- DataImportColumnMappingList
- DataImportRowList

* further updates:

- Add 'get_api_url' method
- Handle case where

* Expose "available fields" to the DataImportSession serializer

Uses the (already available) inventree metadata middleware

* Add detail endpoints

* Clear existing column mappings

* Add endpoint for accepting column mappings

* Add API endpoint exposing available importer serializers

* Add simple playground area for testing data importer

* Adds simple form to start new import session

- Needs work, file field does not currently function correctly

* data_file is *not* read_only

* Add check for file type

* Remove debug statements

* Refactor column mapping

- Generate mapping for each column
- Remove "columns" field
- Column names are calculated dynamically

* Fix uniqueness requirements on mapping table

* Admin updates

- Prevent deletion of mappings
- Prevent addition of mappings

* API endpoint updates

- Prevent mappings from being deleted
- Prevent mappings from being created

* Update importer drawer

* Add widget for selecting data columns

* UI tweaks

* Delete import session when closing modal

* Allow empty string value

* Complete column mapping

* Adds ability to remove rows

* Adjust drawer specs

* Add column 'description' to serializer

* Add option to hide labels in API form field

* Update column heading

* Fix frontend linting errors

* Revert drawer position

* Return correct type

* Fix shadowing

* Fix f-string

* simplify frontend code

* Move importer app

* Update API version

* Reintroduce export formats

* Add new models to RuleSet

* typescript cleanup

* Typescript cleanup

* Improvement for Switch / boolean field

* Display original row data on popover

* Only display mapped columns

* Add DataExportMixin class

- Replaces existing APIDownloadMixin
- Uses DRF serializers for exporting
- *much* more efficient

* Create new file: importer.mixins.py

* Add new mixin to existing views which support data export

* Better error handling

* Cleanup:

- Remove references to APIDownloadMixin
- Remove download_queryset method
- All now handled by API-based export functionality

* Replace table with InvenTreeTable

- Paginate imported rows
- Data can be searched, ordered,

* Make 'pathstring' fields read-only

* Expose list of valid importer types to the API

* Exclude read-only fields

* Cleanup

* Updates for session model

- Column is now editable on mapping object
- Field is no  longer editable
- Improve admin integration

* Adds new custom hook for controlling data import session

* Refactor column mapping widget

* Refactor ImportDataSelector

* Working on ImportDataSelector component

* Adds method for editing fields in import table

- Cell edit mode
- Row edit mode
- Form submission still needs work!

* Adds background task for removing old import sessions

* Fix api_version.py

* Update src/frontend/src/components/importer/ImportDataSelector.tsx

Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>

* Update model verbose names

* Rename mixin class

* Add serializer mixin classes

- Will allow for fine-tuning of the import/export proces

* @register_importer requires specific mixin

* Check subclass for export

* Fix typos

* Refactor export serializer

- Keep operations local to the class

* Add shim class to process an individual row before exporting it

* Add mixin to existing serializers

* Add export functionality for company serializers

* Adds  placeholder for custom admin class

* Update mantine styling

* spacing -> gap

* Add functionality to pre-process form data before upload

* Remove old references to download_queryset

* Improvements for data import drawer:

- Pin title at top of drawer

* Further improvements

* Fix column selection input

* Formatting improvements

* Use a <Stepper> component for better progress display

* Cleanup text

* Add export-only fields to BuildItem queryset

* Expand "export" fields for BuildItem dataset

* Skip backup and static steps in CI

* Remove hard-coded paths

* Fix for "accept_mapping" method

* Present required fields first on import session

* Add "get_importable_fields" method

* Add method for commiting imported row to database

* Cleanup

* Save "complete" state after row import

* Allow prevention of column caching

* Remove debug statement

* Add basic admin table for import sessions

* Fix for table filter functions

- New mantine version requires string values

* Add filters for import session table

* Remove debug message

* fix for <FilterItem />

* Create new import session from admin page

* Cleanup playground

* Re-open an existing import session

* Memoize cell value

* Update <ImportDataSelector>

* Enable download of build line data

* Add extra detail fields

* Register data importers for the stock app

* Enable download of stock item tracking data

* Register importerrs for "company" app

* Register importers for the "order" app

* Add extra fields to purchase order line item serializer

* Update verbose names for order models

* Cleanup import data table rendering

* Pass session information through to cell renderer

* add separate 'field_overrides' field

* Expose 'field_overrides' to API

* Refactor import field selection

* Use override data if provided

* Fix data extraction

- Ignore columns which are not mapped

* Fix fields.pop

- Provide 'None' argument

* Update import data rendering

* Handle missing / empty column names when importing data

* Bug fixin'

* Update hook

* Adds button to upload data straight to table

* Cache "available_fields"

- Reduces API access time by 85%

* Fix calculation of completed_row_count

* Import individual rows from import session

* Allow import of multiple simultaneous records

* Improve extraction of metadata

- Especially for related fields
- Request object no longer required

* Implement suspended rendering of model instances

* Cleanup

* Implement more columns for StockTable

* Allow stock filtering by packaging field

* Fix "stock_value" column

* Improve metadata extraction

- Handle read_only_fields in Meta
- Handle write_only_fields in Meta

* Increase maximum number of importable rows

* Force data import to run on background worker

* Add export-only fields to StockItemSerializer class

* Data conversion when performing initial import

* Various tweaks

* Fix order of operations for data import

* Rename component

* Allow import/export of more model types

* Fix verbose name

* Import rows as a bulk db operation

* Enable download for PartCategoryTemplateTable

* Update stock item export

* Updates for unit tests

* Remove xls format for now

- Causes some bug in tablib
- Surely xlsx is OK?

* More unit test updates

* Future proof migration

* Updates

* unit tests

* Unit test fix

* Remove 'field_overrides'

- field_defaults will suffice

* Remove 'xls' as download option from frontend

* Add simple unit test for data import

* PUI tweaks

---------

Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>

src/backend/InvenTree/company/serializers.py

@@ -10,6 +10,8 @@ from sql_util.utils import SubqueryCount
 from taggit.serializers import TagListSerializerField
 
 import part.filters
+from importer.mixins import DataImportExportSerializerMixin
+from importer.registry import register_importer
 from InvenTree.serializers import (
     InvenTreeCurrencySerializer,
     InvenTreeDecimalField,
@@ -56,7 +58,8 @@ class CompanyBriefSerializer(InvenTreeModelSerializer):
     thumbnail = serializers.CharField(source='get_thumbnail_url', read_only=True)
 
 
-class AddressSerializer(InvenTreeModelSerializer):
+@register_importer()
+class AddressSerializer(DataImportExportSerializerMixin, InvenTreeModelSerializer):
     """Serializer for the Address Model."""
 
     class Meta:
@@ -100,9 +103,19 @@ class AddressBriefSerializer(InvenTreeModelSerializer):
         ]
 
 
-class CompanySerializer(NotesFieldMixin, RemoteImageMixin, InvenTreeModelSerializer):
+@register_importer()
+class CompanySerializer(
+    DataImportExportSerializerMixin,
+    NotesFieldMixin,
+    RemoteImageMixin,
+    InvenTreeModelSerializer,
+):
     """Serializer for Company object (full detail)."""
 
+    export_exclude_fields = ['url', 'primary_address']
+
+    import_exclude_fields = ['image']
+
     class Meta:
         """Metaclass options."""
 
@@ -183,17 +196,25 @@ class CompanySerializer(NotesFieldMixin, RemoteImageMixin, InvenTreeModelSeriali
         return self.instance
 
 
-class ContactSerializer(InvenTreeModelSerializer):
+@register_importer()
+class ContactSerializer(DataImportExportSerializerMixin, InvenTreeModelSerializer):
     """Serializer class for the Contact model."""
 
     class Meta:
         """Metaclass options."""
 
         model = Contact
-        fields = ['pk', 'company', 'name', 'phone', 'email', 'role']
+        fields = ['pk', 'company', 'company_name', 'name', 'phone', 'email', 'role']
+
+    company_name = serializers.CharField(
+        label=_('Company Name'), source='company.name', read_only=True
+    )
 
 
-class ManufacturerPartSerializer(InvenTreeTagModelSerializer):
+@register_importer()
+class ManufacturerPartSerializer(
+    DataImportExportSerializerMixin, InvenTreeTagModelSerializer
+):
     """Serializer for ManufacturerPart object."""
 
     class Meta:
@@ -225,13 +246,13 @@ class ManufacturerPartSerializer(InvenTreeTagModelSerializer):
         super().__init__(*args, **kwargs)
 
         if part_detail is not True:
-            self.fields.pop('part_detail')
+            self.fields.pop('part_detail', None)
 
         if manufacturer_detail is not True:
-            self.fields.pop('manufacturer_detail')
+            self.fields.pop('manufacturer_detail', None)
 
         if prettify is not True:
-            self.fields.pop('pretty_name')
+            self.fields.pop('pretty_name', None)
 
     part_detail = PartBriefSerializer(source='part', many=False, read_only=True)
 
@@ -246,7 +267,10 @@ class ManufacturerPartSerializer(InvenTreeTagModelSerializer):
     )
 
 
-class ManufacturerPartParameterSerializer(InvenTreeModelSerializer):
+@register_importer()
+class ManufacturerPartParameterSerializer(
+    DataImportExportSerializerMixin, InvenTreeModelSerializer
+):
     """Serializer for the ManufacturerPartParameter model."""
 
     class Meta:
@@ -270,14 +294,17 @@ class ManufacturerPartParameterSerializer(InvenTreeModelSerializer):
         super().__init__(*args, **kwargs)
 
         if not man_detail:
-            self.fields.pop('manufacturer_part_detail')
+            self.fields.pop('manufacturer_part_detail', None)
 
     manufacturer_part_detail = ManufacturerPartSerializer(
         source='manufacturer_part', many=False, read_only=True
     )
 
 
-class SupplierPartSerializer(InvenTreeTagModelSerializer):
+@register_importer()
+class SupplierPartSerializer(
+    DataImportExportSerializerMixin, InvenTreeTagModelSerializer
+):
     """Serializer for SupplierPart object."""
 
     class Meta:
@@ -341,17 +368,17 @@ class SupplierPartSerializer(InvenTreeTagModelSerializer):
         super().__init__(*args, **kwargs)
 
         if part_detail is not True:
-            self.fields.pop('part_detail')
+            self.fields.pop('part_detail', None)
 
         if supplier_detail is not True:
-            self.fields.pop('supplier_detail')
+            self.fields.pop('supplier_detail', None)
 
         if manufacturer_detail is not True:
-            self.fields.pop('manufacturer_detail')
-            self.fields.pop('manufacturer_part_detail')
+            self.fields.pop('manufacturer_detail', None)
+            self.fields.pop('manufacturer_part_detail', None)
 
         if prettify is not True:
-            self.fields.pop('pretty_name')
+            self.fields.pop('pretty_name', None)
 
     # Annotated field showing total in-stock quantity
     in_stock = serializers.FloatField(read_only=True, label=_('In Stock'))
@@ -435,7 +462,10 @@ class SupplierPartSerializer(InvenTreeTagModelSerializer):
         return supplier_part
 
 
-class SupplierPriceBreakSerializer(InvenTreeModelSerializer):
+@register_importer()
+class SupplierPriceBreakSerializer(
+    DataImportExportSerializerMixin, InvenTreeModelSerializer
+):
     """Serializer for SupplierPriceBreak object."""
 
     class Meta:
@@ -462,10 +492,10 @@ class SupplierPriceBreakSerializer(InvenTreeModelSerializer):
         super().__init__(*args, **kwargs)
 
         if not supplier_detail:
-            self.fields.pop('supplier_detail')
+            self.fields.pop('supplier_detail', None)
 
         if not part_detail:
-            self.fields.pop('part_detail')
+            self.fields.pop('part_detail', None)
 
     quantity = InvenTreeDecimalField()