[db] Backend setting improvements (#11500)

* Refactor database engine options

- Move to setting/db_backend.py
- Cleanup settings.py

* Fix documentation for postgres settings

* docs updates

* Add transaction_mode options for sqlite

* Update CHANGELOG with breaking changes

* Remove hard-coded database config

* Raise error on invalid backend

* Fix typos

* Fix broken redis link

* Limit to single worker thread for sqlite

* Update docs

* Add verbosity switch to dev.test task

* Add test timeout - kill hanging tests after 120s

* Set WAL mode for sqlite

* Use IMMEDIATE mode for background worker thread

* Use config to set WAL rather than custom hook

* Tweak pyproject settings

* Tweak code

* Increase timeouts

* Reset requirements to master

docs/docs/start/config.md

@@ -275,7 +275,7 @@ If running with a PostgreSQL database backend, the following additional options
 | INVENTREE_DB_TIMEOUT | database.timeout | Database connection timeout (s) | 2 |
 | INVENTREE_DB_TCP_KEEPALIVES | database.tcp_keepalives | TCP keepalive | 1 |
 | INVENTREE_DB_TCP_KEEPALIVES_IDLE | database.tcp_keepalives_idle | Idle TCP keepalive | 1 |
-| INVENTREE_DB_TCP_KEEPALIVES_INTERNAL | database.tcp_keepalives_internal | Internal TCP keepalive | 1|
+| INVENTREE_DB_TCP_KEEPALIVES_INTERVAL | database.tcp_keepalives_interval | TCP keepalive interval | 1|
 | INVENTREE_DB_TCP_KEEPALIVES_COUNT | database.tcp_keepalives_count | TCP keepalive count | 5 |
 | INVENTREE_DB_ISOLATION_SERIALIZABLE | database.serializable | Database isolation level configured to "serializable" | False |
 
@@ -287,6 +287,17 @@ If running with a MySQL database backend, the following additional options are a
 | --- | --- | --- | --- |
 | INVENTREE_DB_ISOLATION_SERIALIZABLE | database.serializable | Database isolation level configured to "serializable" | False |
 
+### SQLite Settings
+
+!!! warning "SQLite Performance"
+    SQLite is not recommended for production use, and should only be used for testing or development purposes. If you are using SQLite in production, you may want to adjust the following settings to improve performance.
+
+If running with a SQLite database backend, the following additional options are available:
+
+| Environment Variable | Configuration File | Description | Default |
+| --- | --- | --- | --- |
+| INVENTREE_DB_TIMEOUT | database.timeout | Database connection timeout (s) | 10 |
+
 ## Caching
 
 InvenTree can be configured to use [redis](https://redis.io) as a global cache backend.

docs/docs/start/installer.md

@@ -118,6 +118,9 @@ Extra python packages can be installed by setting the environment variable `SETU
 
 The used database backend can be configured with environment variables (before the first setup) or in the config file after the installation. Check the [configuration section](./config.md#database-options) for more information.
 
+!!! warning "SQLite Performance"
+    SQLite is not recommended for production use, as it is not designed for high concurrency.
+
 ## Moving Data
 
 To change the data storage location, link the new location to `/opt/inventree/data`. A rough outline of steps to achieve this could be:

docs/docs/start/processes.md

@@ -16,7 +16,22 @@ InvenTree supports a [number of database backends]({% include "django.html" %}/r
 
 Refer to the [database configuration guide](./config.md#database-options) for more information on selecting and configuring the database backend.
 
-In running InvenTree via [docker compose](./docker_install.md), the database process is managed by the `inventree-db` service which provides a [Postgres docker container](https://hub.docker.com/_/postgres).
+If running InvenTree via [docker compose](./docker_install.md), the database process is managed by the `inventree-db` service which provides a [Postgres docker container](https://hub.docker.com/_/postgres).
+
+!!! tip "Postgres Recommended"
+    We recommend using Postgres as the database backend for InvenTree, as it is a robust and scalable database which is well-suited to production use.
+
+#### SQLite Limitations
+
+!!! warning "SQLite Performance"
+    SQLite is not recommended for production use, as it is not designed for high concurrency.
+
+While SQLite is supported, it is strongly *not* recommended for a production installation, especially where there may be multiple users accessing the system concurrently. SQLite is designed for low-concurrency applications, and can experience performance issues when multiple users are accessing the database at the same time.
+
+In addition to concurrency issues, there are other structural limitations which exist in SQLite that can prevent operations on large querysets.
+
+If you are using SQLite, you should be aware of these limitations. It is important to ensure that the database file is stored on a fast storage medium (such as an SSD), and that the database options are configured correctly to minimize locking issues. Refer to the [database configuration guide](./config.md#database-options) for more information on configuring SQLite options.
+
 
 ### Web Server
 
@@ -112,6 +127,8 @@ If the background worker process is not running, InvenTree will not be able to p
 
 If the [cache server](#cache-server) is not running, the background worker will be limited to running a single threaded worker. This is because the background worker uses the cache server to manage task locking, and without a global cache server to communicate between processes, concurrency issues can occur.
 
+Additionally, if you are running SQLite as the database backend, the background worker will be limited to a single thread, due to database locking issues which can occur with SQLite when multiple threads are accessing the database concurrently.
+
 ### Cache Server
 
 The InvenTree cache server is used to store temporary data which is shared between the InvenTree web server and the background worker processes. The cache server is also used to store task information, and to manage task locking between the background worker processes.