[Enhancement] Import by natural key fields (#11288)

* Data import flexibility

- Allow specification of alternative lookup fields for data import

* Observe field filters during data import

* Add alternative import fields for Part models

* More options for IMPORT_ID_FIELDS

* Update src/backend/InvenTree/importer/models.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Bump CHANGELOG

* Handle empty input values

* Add IMPORT_ID_FIELDS for more models

* PK field takes highest priority

* Update import docs

* Tweak return type

* Handle multiple date formats

* Add playwright testing

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+[#11222](https://github.com/inventree/InvenTree/pull/11222) adds support for data import using natural keys, allowing for easier association of related objects without needing to know their internal database IDs.
+
 ### Changed
 
 ### Removed

docs/docs/settings/import.md

@@ -19,6 +19,22 @@ External data can be imported via the admin interface, allowing for rapid integr
 
 To import data, the user must have the appropriate permissions. The user must be a *staff* user, and have the `change` permission for the model in question.
 
+### Mapping to Existing Data
+
+Many data models in InvenTree have relationships to other models. When importing data, the user must ensure that the related data is correctly mapped to existing records in the database. For example, when importing a list of parts, the user must ensure that the *part category* data has already been imported, and that the part category field in the imported file is correctly mapped to the existing part category records in the database.
+
+!!! warning "Multi Level Import"
+    Multi-level imports are explicitly not supported. Only one model can be imported at a time, and the user must ensure that any related data is imported beforehand.
+
+### Primary Key Fields
+
+The default field used to map to existing data (i.e. related models which have already been imported into the database) is using the `ID` (primary key) field. Thus, it is important to ensure that the imported data file contains the correct `ID` values for any related data, otherwise the import process will fail to correctly link the imported data to existing records in the database.
+
+Some models allow for mapping based on other "natural key" fields (e.g. the `reference` field for orders, or the `name` field for part categories). In such cases, the user must ensure that the correct field is mapped to the relevant column in the imported data file.
+
+!!! warning "Unique Identifiers"
+    If a unique identifier cannot be determined for any related field, the user must manually map the relevant field to the correct existing record in the database, during the import process.
+
 ## Import Session
 
 Importing data is a multi-step process, which is managed via an *import session*. An import session is created when the user initiates a data import, and is used to track the progress of the data import process.
@@ -33,6 +49,20 @@ Import sessions can be managed from the [Admin Center](./admin.md#admin-center)
 
 Depending on the type of data being imported, an import session can be created from an appropriate page context in the user interface. In such cases, the import session will be automatically linked to the relevant data type being imported.
 
+### Starting an Import Session
+
+An import session can be initiated from a number of different contexts within the user interface:
+
+### Admin Center
+
+Staff users can create an import session from within the [Admin Center](./admin.md#admin-center). This is a general-purpose import session, and the user will be required to select the type of data to import.
+
+Users can quickly navigate to the data import managemement page from the [spotlight search](../concepts/user_interface.md#spotlight), by searching for "import" and selecting the "Import data" option.
+
+### Data Tables
+
+Some data tables allow the user to create an import session directly from the table view. In such cases, the import session will be automatically linked to the relevant data type being imported, and additional [context information](#context-sensitive-importing) will be automatically provided.
+
 ## Import Process
 
 The following steps outline the process of importing data into InvenTree:

src/backend/InvenTree/build/models.py

@@ -111,6 +111,7 @@ class Build(
     """
 
     STATUS_CLASS = BuildStatus
+    IMPORT_ID_FIELDS = ['reference']
 
     class Meta:
         """Metaclass options for the BuildOrder model."""

src/backend/InvenTree/common/models.py

@@ -148,6 +148,8 @@ class UpdatedUserMixin(models.Model):
 class ProjectCode(InvenTree.models.InvenTreeMetadataModel):
     """A ProjectCode is a unique identifier for a project."""
 
+    IMPORT_ID_FIELDS = ['code']
+
     class Meta:
         """Class options for the ProjectCode model."""
 
@@ -2396,6 +2398,8 @@ class ParameterTemplate(
         enabled: Is this template enabled?
     """
 
+    IMPORT_ID_FIELDS = ['name']
+
     class Meta:
         """Metaclass options for the ParameterTemplate model."""
 

src/backend/InvenTree/company/models.py

@@ -111,6 +111,7 @@ class Company(
     """
 
     IMAGE_RENAME = rename_company_image
+    IMPORT_ID_FIELDS = ['name']
 
     class Meta:
         """Metaclass defines extra model options."""
@@ -297,6 +298,8 @@ class Contact(InvenTree.models.InvenTreeMetadataModel):
         role: position in company
     """
 
+    IMPORT_ID_FIELDS = ['name', 'email']
+
     class Meta:
         """Metaclass defines extra model options."""
 
@@ -494,6 +497,8 @@ class ManufacturerPart(
         description: Descriptive notes field
     """
 
+    IMPORT_ID_FIELDS = ['MPN']
+
     class Meta:
         """Metaclass defines extra model options."""
 
@@ -620,6 +625,8 @@ class SupplierPart(
         updated: Date that the SupplierPart was last updated
     """
 
+    IMPORT_ID_FIELDS = ['SKU']
+
     class Meta:
         """Metaclass defines extra model options."""
 

src/backend/InvenTree/importer/models.py

@@ -2,6 +2,7 @@
 
 import json
 from collections import OrderedDict
+from datetime import datetime
 from typing import Optional
 
 from django.contrib.auth.models import User
@@ -151,6 +152,27 @@ class DataImportSession(models.Model):
 
         return supported_models().get(self.model_type, None)
 
+    def get_related_model(self, field_name: str) -> models.Model:
+        """Return the related model for a given field name.
+
+        Arguments:
+            field_name: The name of the field to check
+
+        Returns:
+            The related model class, if one exists, or None otherwise
+        """
+        model_class = self.model_class
+
+        if not model_class:
+            return None
+
+        try:
+            related_field = model_class._meta.get_field(field_name)
+            model = related_field.remote_field.model
+            return model
+        except (AttributeError, models.FieldDoesNotExist):
+            return None
+
     def extract_columns(self) -> None:
         """Run initial column extraction and mapping.
 
@@ -597,6 +619,8 @@ class DataImportRow(models.Model):
 
         data = {}
 
+        self.related_field_map = {}
+
         # We have mapped column (file) to field (serializer) already
         for field, col in field_mapping.items():
             # Data override (force value and skip any further checks)
@@ -622,7 +646,9 @@ class DataImportRow(models.Model):
             if field_type == 'boolean':
                 value = InvenTree.helpers.str2bool(value)
             elif field_type == 'date':
-                value = value or None
+                value = self.convert_date_field(value)
+            elif field_type == 'related field':
+                value = self.lookup_related_field(field, value)
 
             # Use the default value, if provided
             if value is None and field in default_values:
@@ -667,6 +693,93 @@ class DataImportRow(models.Model):
         if commit:
             self.save()
 
+    def convert_date_field(self, value: str) -> str:
+        """Convert an incoming date field to the correct format for the database."""
+        if value in [None, '']:
+            return None
+
+        # Attempt conversion using accepted formats
+        date_formats = ['%Y-%m-%d', '%d/%m/%Y', '%m/%d/%Y', '%Y/%m/%d']
+
+        for fmt in date_formats:
+            try:
+                dt = datetime.strptime(value.strip(), fmt)
+
+                # If the date is valid, convert it to the standard format and return
+                return dt.strftime('%Y-%m-%d')
+            except ValueError:
+                continue
+
+        # If none of the formats matched, return the original value
+        return value
+
+    def lookup_related_field(self, field_name: str, value: str) -> Optional[int]:
+        """Try to perform lookup against a related field.
+
+        - This is used to convert a human-readable value (e.g. a supplier name) into a database reference (e.g. supplier ID).
+        - Reference the value against the related model's allowable import fields
+
+        Arguments:
+            field_name: The name of the field to perform the lookup against
+            value: The value to be looked up
+
+        Returns:
+            A primary key value
+        """
+        if value is None or value == '':
+            return value
+
+        if field_name is None or field_name == '':
+            return value
+
+        if field_name in self.related_field_map:
+            model = self.related_field_map[field_name]
+        else:
+            # Cache the related model for this field name
+            model = self.related_field_map[field_name] = self.session.get_related_model(
+                field_name
+            )
+
+        if not model:
+            raise DjangoValidationError({
+                'session': f'No related model found for field: {field_name}'
+            })
+
+        valid_items = set()
+
+        base_filters = (
+            self.session.field_filters.get(field_name, {})
+            if self.session.field_filters
+            else {}
+        )
+
+        # First priority is the PK (primary key) field
+        id_fields = ['pk']
+
+        if custom_id_fields := getattr(model, 'IMPORT_ID_FIELDS', None):
+            id_fields += custom_id_fields
+
+        # Iterate through the provided list - if any of the values match, we can perform the lookup
+        for id_field in id_fields:
+            try:
+                queryset = model.objects.filter(**{id_field: value}, **base_filters)
+            except ValueError:
+                continue
+
+            # Evaluate at most two results to determine if there is exactly one match
+            results = list(queryset[:2])
+            if len(results) == 1:
+                # We have a single match against this field
+                valid_items.add(results[0].pk)
+
+        if len(valid_items) == 1:
+            # We found a single valid match against the related model - return this value
+            return valid_items.pop()
+
+        # We found either zero or multiple values matching against the related model
+        # Return the original value and let the serializer validation handle any errors against this field
+        return value
+
     def serializer_data(self):
         """Construct data object to be sent to the serializer.
 

src/backend/InvenTree/order/models.py

@@ -299,6 +299,7 @@ class Order(
 
     REQUIRE_RESPONSIBLE_SETTING = None
     UNLOCK_SETTING = None
+    IMPORT_ID_FIELDS = ['reference']
 
     class Meta:
         """Metaclass options. Abstract ensures no database table is created."""

src/backend/InvenTree/part/models.py

@@ -84,8 +84,8 @@ class PartCategory(
     """
 
     ITEM_PARENT_KEY = 'category'
-
     EXTRA_PATH_FIELDS = ['icon']
+    IMPORT_ID_FIELDS = ['pathstring', 'name']
 
     class Meta:
         """Metaclass defines extra model properties."""
@@ -517,6 +517,7 @@ class Part(
 
     NODE_PARENT_KEY = 'variant_of'
     IMAGE_RENAME = rename_part_image
+    IMPORT_ID_FIELDS = ['IPN', 'name']
 
     objects = TreeManager()
 
@@ -3635,6 +3636,8 @@ class PartTestTemplate(InvenTree.models.InvenTreeMetadataModel):
     run on the model (refer to the validate_unique function).
     """
 
+    IMPORT_ID_FIELDS = ['key']
+
     class Meta:
         """Metaclass options for the PartTestTemplate model."""
 

src/backend/InvenTree/stock/models.py

@@ -65,6 +65,8 @@ class StockLocationType(InvenTree.models.MetadataMixin, models.Model):
         icon: icon class
     """
 
+    IMPORT_ID_FIELDS = ['name']
+
     class Meta:
         """Metaclass defines extra model properties."""
 
@@ -134,8 +136,8 @@ class StockLocation(
     """
 
     ITEM_PARENT_KEY = 'location'
-
     EXTRA_PATH_FIELDS = ['icon']
+    IMPORT_ID_FIELDS = ['pathstring', 'name']
 
     objects = TreeManager()
 
@@ -434,6 +436,7 @@ class StockItem(
         packaging: Description of how the StockItem is packaged (e.g. "reel", "loose", "tape" etc)
     """
 
+    IMPORT_ID_FIELDS = ['serial']
     STATUS_CLASS = StockStatus
 
     class Meta:

src/frontend/tests/fixtures/po_data_natural_keys.csv

@@ -0,0 +1,3 @@
+Project Code,Quantity,Reference,Target Date,Project Code Label,Build Order,Overdue,Received,Purchase price,Currency,Auto Pricing,Destination,Total price,SKU,MPN,Internal Part Number,Internal Part,Internal Part Name
+5,123,,30/01/2026,PRO-ZEN,,TRUE,0,0.48,USD,TRUE,,59.04,FUT-43861-DDU,,,55,C_100nF_0603
+,9999,my-custom-reference,,,,FALSE,0,0.1179,USD,TRUE,9,1178.8821,FUT-82092-CQB,,,60,C_10uF_0805

src/frontend/tests/pui_importing.spec.ts

@@ -172,3 +172,53 @@ test('Importing - Purchase Order', async ({ browser }) => {
   await page.getByRole('cell', { name: 'Database Field' }).waitFor();
   await page.getByRole('cell', { name: 'Field Description' }).waitFor();
 });
+
+test('Importing - Natural Keys', async ({ browser }) => {
+  const page = await doCachedLogin(browser, {
+    username: 'steven',
+    password: 'wizardstaff',
+    url: 'purchasing/purchase-order/15/line-items'
+  });
+
+  // Import line item data, but use natural keys as the import fields
+  await page
+    .getByRole('button', { name: 'action-button-import-line-' })
+    .click();
+
+  const fileInput = await page.locator('input[type="file"]');
+  await fileInput.setInputFiles('./tests/fixtures/po_data_natural_keys.csv');
+  await page.getByRole('button', { name: 'Submit' }).click();
+
+  // Attempt import with missing required fields
+  await page.getByRole('button', { name: 'Accept Column Mapping' }).click();
+  await page.getByText('Some required fields have not been mapped').waitFor();
+
+  // Select different columns for data import
+  // We will use the "SKU" field to map to the supplier part
+  await page.getByRole('textbox', { name: 'import-column-map-part' }).click();
+  await page.getByRole('option', { name: 'SKU' }).click();
+
+  // Other import fields will be left as default
+  await page.getByRole('button', { name: 'Accept Column Mapping' }).click();
+
+  // Check for expected values to be displayed
+  await page.getByText('PRO-ZEN').first().waitFor();
+  await page.getByText('Project Zenith').first().waitFor();
+  await page.getByText('my-custom-reference').first().waitFor();
+  await page.getByText('Factory/Mechanical Lab').first().waitFor();
+  await page.getByText('FUT-43861-DDU').first().waitFor();
+  await page.getByText('FUT-82092-CQB').first().waitFor();
+  await page.getByText('2026-01-30').first().waitFor();
+
+  // Let's import all the data
+  await page
+    .getByRole('row', { name: 'Select all records Row Not' })
+    .getByLabel('Select all records')
+    .click();
+  await page
+    .getByRole('button', { name: 'action-button-import-selected' })
+    .click();
+
+  await page.getByText('Data has been imported successfully').waitFor();
+  await page.getByRole('button', { name: 'Close' }).click();
+});