[WIP] Data importer (#6911)

* Adds new model for DataImportSession

* Add file extension validation

Expose to admin interface also

* Switch to new 'importer' app

* Refactoring to help prevent circular imports

* Add serializer registry

- Use @register_importer tag for any serializer class

* Cleanup migration file

- Do not use one-time hard-coded values here

* Refactor code into registry.py

* Add validation for the uploaded file

- Must be importable by tablib

* Refactoring

* Adds property to retrieve matching serializer class

* Update helper functions

* Add hook to auto-assign columns on initial creation

* Rename field

* Enforce initial status value

* Add model for individual rows in the data import

* Add DataImportRow model

* Extract data rows as dict

* Update fields

- Remove "progress" field (will be calculated)
- Added "timestamp" field
- Added "complete" field to DataImportRow

* Auto-map column names

- Provide "sensible" default values

* Add API endpoint for DataImportSession

* Offload data import operation

- For large data files this may take a significant amount of time
- Offload it to the background worker process

* Refactor data import code

* Update models

- Add "columns" field to DataImportSession
- Add "errors" field to DataImportRow

* Move field mapping to a new model type

- Simpler validation

* Save "valid" status for each data row

* Include session defaults when validating row data

* Update content_excludes

- Ignore importer models in import/export

* Remove port from ALLOWED_HOST entries

* Skip table events for importer models

* Bug fixes

* Serializer updates

* Add more endpoints

- DataImportColumnMappingList
- DataImportRowList

* further updates:

- Add 'get_api_url' method
- Handle case where

* Expose "available fields" to the DataImportSession serializer

Uses the (already available) inventree metadata middleware

* Add detail endpoints

* Clear existing column mappings

* Add endpoint for accepting column mappings

* Add API endpoint exposing available importer serializers

* Add simple playground area for testing data importer

* Adds simple form to start new import session

- Needs work, file field does not currently function correctly

* data_file is *not* read_only

* Add check for file type

* Remove debug statements

* Refactor column mapping

- Generate mapping for each column
- Remove "columns" field
- Column names are calculated dynamically

* Fix uniqueness requirements on mapping table

* Admin updates

- Prevent deletion of mappings
- Prevent addition of mappings

* API endpoint updates

- Prevent mappings from being deleted
- Prevent mappings from being created

* Update importer drawer

* Add widget for selecting data columns

* UI tweaks

* Delete import session when closing modal

* Allow empty string value

* Complete column mapping

* Adds ability to remove rows

* Adjust drawer specs

* Add column 'description' to serializer

* Add option to hide labels in API form field

* Update column heading

* Fix frontend linting errors

* Revert drawer position

* Return correct type

* Fix shadowing

* Fix f-string

* simplify frontend code

* Move importer app

* Update API version

* Reintroduce export formats

* Add new models to RuleSet

* typescript cleanup

* Typescript cleanup

* Improvement for Switch / boolean field

* Display original row data on popover

* Only display mapped columns

* Add DataExportMixin class

- Replaces existing APIDownloadMixin
- Uses DRF serializers for exporting
- *much* more efficient

* Create new file: importer.mixins.py

* Add new mixin to existing views which support data export

* Better error handling

* Cleanup:

- Remove references to APIDownloadMixin
- Remove download_queryset method
- All now handled by API-based export functionality

* Replace table with InvenTreeTable

- Paginate imported rows
- Data can be searched, ordered,

* Make 'pathstring' fields read-only

* Expose list of valid importer types to the API

* Exclude read-only fields

* Cleanup

* Updates for session model

- Column is now editable on mapping object
- Field is no  longer editable
- Improve admin integration

* Adds new custom hook for controlling data import session

* Refactor column mapping widget

* Refactor ImportDataSelector

* Working on ImportDataSelector component

* Adds method for editing fields in import table

- Cell edit mode
- Row edit mode
- Form submission still needs work!

* Adds background task for removing old import sessions

* Fix api_version.py

* Update src/frontend/src/components/importer/ImportDataSelector.tsx

Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>

* Update model verbose names

* Rename mixin class

* Add serializer mixin classes

- Will allow for fine-tuning of the import/export proces

* @register_importer requires specific mixin

* Check subclass for export

* Fix typos

* Refactor export serializer

- Keep operations local to the class

* Add shim class to process an individual row before exporting it

* Add mixin to existing serializers

* Add export functionality for company serializers

* Adds  placeholder for custom admin class

* Update mantine styling

* spacing -> gap

* Add functionality to pre-process form data before upload

* Remove old references to download_queryset

* Improvements for data import drawer:

- Pin title at top of drawer

* Further improvements

* Fix column selection input

* Formatting improvements

* Use a <Stepper> component for better progress display

* Cleanup text

* Add export-only fields to BuildItem queryset

* Expand "export" fields for BuildItem dataset

* Skip backup and static steps in CI

* Remove hard-coded paths

* Fix for "accept_mapping" method

* Present required fields first on import session

* Add "get_importable_fields" method

* Add method for commiting imported row to database

* Cleanup

* Save "complete" state after row import

* Allow prevention of column caching

* Remove debug statement

* Add basic admin table for import sessions

* Fix for table filter functions

- New mantine version requires string values

* Add filters for import session table

* Remove debug message

* fix for <FilterItem />

* Create new import session from admin page

* Cleanup playground

* Re-open an existing import session

* Memoize cell value

* Update <ImportDataSelector>

* Enable download of build line data

* Add extra detail fields

* Register data importers for the stock app

* Enable download of stock item tracking data

* Register importerrs for "company" app

* Register importers for the "order" app

* Add extra fields to purchase order line item serializer

* Update verbose names for order models

* Cleanup import data table rendering

* Pass session information through to cell renderer

* add separate 'field_overrides' field

* Expose 'field_overrides' to API

* Refactor import field selection

* Use override data if provided

* Fix data extraction

- Ignore columns which are not mapped

* Fix fields.pop

- Provide 'None' argument

* Update import data rendering

* Handle missing / empty column names when importing data

* Bug fixin'

* Update hook

* Adds button to upload data straight to table

* Cache "available_fields"

- Reduces API access time by 85%

* Fix calculation of completed_row_count

* Import individual rows from import session

* Allow import of multiple simultaneous records

* Improve extraction of metadata

- Especially for related fields
- Request object no longer required

* Implement suspended rendering of model instances

* Cleanup

* Implement more columns for StockTable

* Allow stock filtering by packaging field

* Fix "stock_value" column

* Improve metadata extraction

- Handle read_only_fields in Meta
- Handle write_only_fields in Meta

* Increase maximum number of importable rows

* Force data import to run on background worker

* Add export-only fields to StockItemSerializer class

* Data conversion when performing initial import

* Various tweaks

* Fix order of operations for data import

* Rename component

* Allow import/export of more model types

* Fix verbose name

* Import rows as a bulk db operation

* Enable download for PartCategoryTemplateTable

* Update stock item export

* Updates for unit tests

* Remove xls format for now

- Causes some bug in tablib
- Surely xlsx is OK?

* More unit test updates

* Future proof migration

* Updates

* unit tests

* Unit test fix

* Remove 'field_overrides'

- field_defaults will suffice

* Remove 'xls' as download option from frontend

* Add simple unit test for data import

* PUI tweaks

---------

Co-authored-by: Lukas <76838159+wolflu05@users.noreply.github.com>

src/frontend/src/hooks/UseImportSession.tsx

@@ -0,0 +1,107 @@
+import { useCallback, useMemo } from 'react';
+
+import { api } from '../App';
+import { ApiEndpoints } from '../enums/ApiEndpoints';
+import { apiUrl } from '../states/ApiState';
+import { useInstance } from './UseInstance';
+
+/*
+ * Custom hook for managing the state of a data import session
+ */
+
+// TODO: Load these values from the server?
+export enum ImportSessionStatus {
+  INITIAL = 0,
+  MAPPING = 10,
+  IMPORTING = 20,
+  PROCESSING = 30,
+  COMPLETE = 40
+}
+
+export type ImportSessionState = {
+  sessionId: number;
+  sessionData: any;
+  refreshSession: () => void;
+  sessionQuery: any;
+  status: ImportSessionStatus;
+  availableFields: Record<string, any>;
+  availableColumns: string[];
+  mappedFields: any[];
+  columnMappings: any[];
+};
+
+export function useImportSession({
+  sessionId
+}: {
+  sessionId: number;
+}): ImportSessionState {
+  // Query manager for the import session
+  const {
+    instance: sessionData,
+    refreshInstance: refreshSession,
+    instanceQuery: sessionQuery
+  } = useInstance({
+    endpoint: ApiEndpoints.import_session_list,
+    pk: sessionId,
+    defaultValue: {}
+  });
+
+  // Current step of the import process
+  const status: ImportSessionStatus = useMemo(() => {
+    return sessionData?.status ?? ImportSessionStatus.INITIAL;
+  }, [sessionData]);
+
+  // List of available writeable database field definitions
+  const availableFields: any[] = useMemo(() => {
+    return sessionData?.available_fields ?? [];
+  }, [sessionData]);
+
+  // List of available data file columns
+  const availableColumns: string[] = useMemo(() => {
+    let cols = sessionData?.columns ?? [];
+
+    // Filter out any blank or duplicate columns
+    cols = cols.filter((col: string) => !!col);
+    cols = cols.filter(
+      (col: string, index: number) => cols.indexOf(col) === index
+    );
+
+    return cols;
+  }, [sessionData.columns]);
+
+  const columnMappings: any[] = useMemo(() => {
+    let mapping =
+      sessionData?.column_mappings?.map((mapping: any) => ({
+        ...mapping,
+        ...(availableFields[mapping.field] ?? {})
+      })) ?? [];
+
+    mapping = mapping.sort((a: any, b: any) => {
+      if (a?.required && !b?.required) return -1;
+      if (!a?.required && b?.required) return 1;
+      return 0;
+    });
+
+    return mapping;
+  }, [sessionData, availableColumns]);
+
+  // List of field which have been mapped to columns
+  const mappedFields: any[] = useMemo(() => {
+    return (
+      sessionData?.column_mappings?.filter((column: any) => !!column.column) ??
+      []
+    );
+  }, [sessionData]);
+
+  return {
+    sessionData,
+    sessionId,
+    refreshSession,
+    sessionQuery,
+    status,
+    availableFields,
+    availableColumns,
+    columnMappings,
+    mappedFields
+  };
+}