Add documentation on background worker configuration options (#11673)

docs/docs/start/config.md

@@ -185,7 +185,6 @@ Proxy configuration can be complex, and any configuration beyond the basic setup
 
 Refer to the [proxy server documentation](./processes.md#proxy-server) for more information.
 
-
 ## Admin Site
 
 Django provides a powerful [administrator interface]({% include "django.html" %}/ref/contrib/admin/) which can be used to manage the InvenTree database. This interface is enabled by default, and available at the `/admin/` URL.
@@ -468,6 +467,17 @@ The login-experience can be altered with the following settings:
 
 Custom authentication backends can be used by specifying them here. These can for example be used to add [LDAP / AD login](https://django-auth-ldap.readthedocs.io/en/latest/) to InvenTree
 
+## Background Worker Options
+
+The following options are available for configuring the InvenTree [background worker process](./processes.md#background-worker):
+
+| Environment Variable | Configuration File | Description | Default |
+| --- | --- | --- | --- |
+| INVENTREE_BACKGROUND_WORKERS | background.workers | Number of background worker processes | 1 |
+| INVENTREE_BACKGROUND_TIMEOUT | background.timeout | Timeout for background worker tasks (seconds) | 90 |
+| INVENTREE_BACKGROUND_RETRY | background.retry | Time to wait before retrying a background task (seconds) | 300 |
+| INVENTREE_BACKGROUND_MAX_ATTEMPTS | background.max_attempts | Maximum number of attempts for a background task | 5 |
+
 ## Sentry Integration
 
 The InvenTree server can be integrated with the [sentry.io](https://sentry.io) monitoring service, for error logging and performance tracking.

docs/docs/start/processes.md

@@ -143,3 +143,7 @@ InvenTree uses the [Redis](https://redis.io/) cache server to manage cache data.
 
 !!! tip "Enable Cache"
     While a redis container is provided in the default configuration, by default it is not enabled in the InvenTree server. You can enable redis cache support by following the [caching configuration guide](./config.md#caching)
+
+### Configuration
+
+Refer to the [background worker configuration options](./config.md#background-worker-options) for more information on configuring the background worker process.

src/backend/InvenTree/InvenTree/settings.py

@@ -834,10 +834,15 @@ GLOBAL_CACHE_ENABLED = is_global_cache_enabled()
 
 CACHES = {'default': get_cache_config(GLOBAL_CACHE_ENABLED)}
 
-_q_worker_timeout = int(
+BACKGROUND_WORKER_TIMEOUT = int(
     get_setting('INVENTREE_BACKGROUND_TIMEOUT', 'background.timeout', 90)
 )
 
+# Set the retry time for background workers to be slightly longer than the worker timeout, to ensure that workers have time to timeout before being retried
+BACKGROUND_WORKER_RETRY = max(
+    int(get_setting('INVENTREE_BACKGROUND_RETRY', 'background.retry', 300)),
+    BACKGROUND_WORKER_TIMEOUT + 120,
+)
 
 # Prevent running multiple background workers if global cache is disabled
 # This is to prevent scheduling conflicts due to the lack of a shared cache
@@ -851,16 +856,18 @@ BACKGROUND_WORKER_COUNT = (
 if 'sqlite' in DB_ENGINE:
     BACKGROUND_WORKER_COUNT = 1
 
+BACKGROUND_WORKER_ATTEMPTS = int(
+    get_setting('INVENTREE_BACKGROUND_MAX_ATTEMPTS', 'background.max_attempts', 5)
+)
+
 # django-q background worker configuration
 Q_CLUSTER = {
     'name': 'InvenTree',
     'label': 'Background Tasks',
     'workers': BACKGROUND_WORKER_COUNT,
-    'timeout': _q_worker_timeout,
+    'timeout': BACKGROUND_WORKER_TIMEOUT,
-    'retry': max(120, _q_worker_timeout + 30),
+    'retry': BACKGROUND_WORKER_RETRY,
-    'max_attempts': int(
+    'max_attempts': BACKGROUND_WORKER_ATTEMPTS,
-        get_setting('INVENTREE_BACKGROUND_MAX_ATTEMPTS', 'background.max_attempts', 5)
-    ),
     'save_limit': 1000,
     'queue_limit': 50,
     'catch_up': False,