mirror of
https://github.com/inventree/InvenTree.git
synced 2025-04-28 11:36:44 +00:00
Prevent multiple background workers without global cache (#8401)
* Prevent multiple background workers without global cache * Documentation updates - New page dedicated to InvenTree process stack - Update links - Consolidate information * Update mkdocs.yml
This commit is contained in:
Oliver
GitHub
parent
913a05cf45
commit
801b32e4e3
@ -40,7 +40,7 @@ To enable access to the InvenTree server from other computers on a local network
|
||||
|
||||
## Background Worker
|
||||
|
||||
The background task manager must also be started. The InvenTree server is already running in the foreground, so open a *new shell window* to start the server.
|
||||
The [background task manager](./processes.md#background-worker) must also be started. The InvenTree server is already running in the foreground, so open a *new shell window* to start the server.
|
||||
|
||||
### Activate Virtual Environment
|
||||
|
||||
@ -55,4 +55,4 @@ source ./env/bin/activate
|
||||
(env) invoke worker
|
||||
```
|
||||
|
||||
This will start the background process manager in the current shell.
|
||||
This will start an instance of the background worker in the current shell.
|
||||
|
||||
@ -19,7 +19,7 @@ The InvenTree web server is hosted using [Gunicorn](https://gunicorn.org/). Guni
|
||||
|
||||
### Supervisor
|
||||
|
||||
[Supervisor](http://supervisord.org/) is a process control system which monitors and controls multiple background processes. It is used in the InvenTree production setup to ensure that the server and background worker processes are always running.
|
||||
[Supervisor](http://supervisord.org/) is a process control system which monitors and controls multiple background processes. It is used in the InvenTree production setup to ensure that the [web server](./processes.md#web-server) and [background worker](./processes.md#background-worker) processes are always running.
|
||||
|
||||
## Gunicorn
|
||||
|
||||
@ -98,11 +98,12 @@ The InvenTree server (and background task manager) should now be running!
|
||||
In addition to the InvenTree server, you will need a method of delivering static and media files (this is *not* handled by the InvenTree server in a production environment).
|
||||
|
||||
!!! info "Read More"
|
||||
Refer to the [Serving Files](./serving_files.md) section for more details
|
||||
Refer to the [proxy server documentation](./processes.md#proxy-server) for more details
|
||||
|
||||
### Next Steps
|
||||
|
||||
You (or your system administrator) may wish to perform further steps such as placing the InvenTree server behind a reverse-proxy such as [caddy](https://caddyserver.com/), or [nginx](https://www.nginx.com/).
|
||||
You (or your system administrator) may wish to perform further steps such as placing the InvenTree server behind a [reverse proxy](./processes.md#proxy-server) such as [caddy](https://caddyserver.com/), or [nginx](https://www.nginx.com/).
|
||||
|
||||
As production environment options are many and varied, such tasks are outside the scope of this documentation.
|
||||
|
||||
There are many great online tutorials about running django applications in production!
|
||||
|
||||
@ -279,12 +279,12 @@ InvenTree requires some external directories for storing files:
|
||||
|
||||
| Environment Variable | Configuration File | Description | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| INVENTREE_STATIC_ROOT | static_root | [Static files](./serving_files.md#static-files) directory | *Not specified* |
|
||||
| INVENTREE_MEDIA_ROOT | media_root | [Media files](./serving_files.md#media-files) directory | *Not specified* |
|
||||
| INVENTREE_STATIC_ROOT | static_root | [Static files](./processes.md#static-files) directory | *Not specified* |
|
||||
| INVENTREE_MEDIA_ROOT | media_root | [Media files](./processes.md#media-files) directory | *Not specified* |
|
||||
| INVENTREE_BACKUP_DIR | backup_dir | Backup files directory | *Not specified* |
|
||||
|
||||
!!! tip "Serving Files"
|
||||
Read the [Serving Files](./serving_files.md) section for more information on hosting *static* and *media* files
|
||||
Read the [proxy server documentation](./processes.md#proxy-server) for more information on hosting *static* and *media* files
|
||||
|
||||
### Static File Storage
|
||||
|
||||
|
||||
@ -87,7 +87,7 @@ Plugins are supported natively when running under docker. There are two ways to
|
||||
The production docker compose configuration outlined on this page uses [Caddy](https://caddyserver.com/) to serve static files and media files. If you change this configuration, you will need to ensure that static and media files are served correctly.
|
||||
|
||||
!!! info "Read More"
|
||||
Refer to the [Serving Files](./serving_files.md) section for more details
|
||||
Refer to the [proxy server documentation](./processes.md#proxy-server) for more details
|
||||
|
||||
### SSL Certificates
|
||||
|
||||
@ -99,45 +99,11 @@ The example docker compose file launches the following containers:
|
||||
|
||||
| Container | Description |
|
||||
| --- | --- |
|
||||
| inventree-db | PostgreSQL database |
|
||||
| inventree-server | Gunicorn web server |
|
||||
| inventree-worker | django-q background worker |
|
||||
| inventree-proxy | Caddy file server and reverse proxy |
|
||||
| *inventree-cache* | *redis cache (optional)* |
|
||||
|
||||
#### PostgreSQL Database
|
||||
|
||||
A PostgreSQL database container which requires a username:password combination (which can be changed). This uses the official [PostgreSQL image](https://hub.docker.com/_/postgres).
|
||||
|
||||
#### Web Server
|
||||
|
||||
Runs an InvenTree web server instance, powered by a Gunicorn web server.
|
||||
|
||||
#### Background Worker
|
||||
|
||||
Runs the InvenTree background worker process. This spins up a second instance of the *inventree* container, with a different entrypoint command.
|
||||
|
||||
#### Proxy Server
|
||||
|
||||
Caddy working as a reverse proxy, separating requests for static and media files, and directing everything else to Gunicorn.
|
||||
|
||||
This container uses the official [caddy image](https://hub.docker.com/_/caddy).
|
||||
|
||||
!!! info "Nginx Proxy"
|
||||
An alternative is to run nginx as the reverse proxy. A sample configuration file is provided in the `./contrib/container/` source directory.
|
||||
|
||||
#### Redis Cache
|
||||
|
||||
Redis is used as cache storage for the InvenTree server. This provides a more performant caching system which can useful in larger installations.
|
||||
|
||||
This container uses the official [redis image](https://hub.docker.com/_/redis).
|
||||
|
||||
!!! info "Redis on Docker"
|
||||
Docker adds an additional network layer - that might lead to lower performance than bare metal.
|
||||
To optimize and configure your redis deployment follow the [official docker guide](https://redis.io/docs/getting-started/install-stack/docker/#configuration).
|
||||
|
||||
!!! tip "Enable Cache"
|
||||
While a redis container is provided in the default configuration, by default it is not enabled in the Inventree server. You can enable redis cache support by following the [caching configuration guide](./config.md#caching)
|
||||
| inventree-db | [PostgreSQL database](./processes.md#database) |
|
||||
| inventree-server | [InvenTree web server](./processes.md#web-server) |
|
||||
| inventree-worker | [django-q background worker](./processes.md#background-worker) |
|
||||
| inventree-proxy | [Caddy file server and reverse proxy](./processes.md#proxy-server) |
|
||||
| *inventree-cache* | [*redis cache (optional)*](./processes.md#cache-server) |
|
||||
|
||||
### Data Volume
|
||||
|
||||
|
||||
@ -104,10 +104,11 @@ docker compose up -d
|
||||
|
||||
This command launches the following containers:
|
||||
|
||||
- `inventree-db` - PostgreSQL database
|
||||
- `inventree-server` - InvenTree web server
|
||||
- `inventree-worker` - Background worker
|
||||
- `inventree-proxy` - Caddy reverse proxy
|
||||
- `inventree-db` - [PostgreSQL database](./processes.md#database)
|
||||
- `inventree-server` - [InvenTree web server](./processes.md#web-server)
|
||||
- `inventree-worker` - [Background worker](./processes.md#background-worker)
|
||||
- `inventree-proxy` - [Caddy reverse proxy](./processes.md#proxy-server)
|
||||
- `inventree-cache` - [Redis cache](./processes.md#cache-server)
|
||||
|
||||
!!! success "Up and Running!"
|
||||
You should now be able to view the InvenTree login screen at [http://inventree.localhost](http://inventree.localhost) (or whatever custom domain you have configured in the `.env` file).
|
||||
|
||||
@ -66,14 +66,14 @@ In addition to the location where the InvenTree source code is located, you will
|
||||
InvenTree requires a directory for storage of [static files](./config.md#static-file-storage).
|
||||
|
||||
!!! info "Read More"
|
||||
Refer to the [Serving Files](./serving_files.md) section for more details
|
||||
Refer to the [proxy server documentation](./processes.md#proxy-server) for more details
|
||||
|
||||
#### Media Files
|
||||
|
||||
InvenTree requires a directory for storage of [user uploaded files](./config.md#uploaded-file-storage)
|
||||
|
||||
!!! info "Read More"
|
||||
Refer to the [Serving Files](./serving_files.md) section for more details
|
||||
Refer to the [proxy server documentation](./processes.md#proxy-server) for more details
|
||||
|
||||
#### Backup Directory
|
||||
|
||||
|
||||
@ -132,11 +132,14 @@ To update InvenTree run `apt install --only-upgrade inventree` - this might need
|
||||
## Controlling InvenTree
|
||||
|
||||
### Services
|
||||
|
||||
InvenTree installs multiple services that can be controlled with your local system runner (`service` or `systemctl`).
|
||||
The service `inventree` controls everything, `inventree-web` the (internal) webserver and `inventree-worker` the background worker(s).
|
||||
The service `inventree` controls everything, `inventree-web` (the [InvenTree web server](./processes.md#web-server)) and `inventree-worker` the [background worker(s)](./processes.md#background-worker).
|
||||
|
||||
More instances of the worker can be instantiated from the command line. This is only meant for advanced users.
|
||||
|
||||
This sample script launches 3 services. By default, 1 is launched.
|
||||
|
||||
```bash
|
||||
inventree scale worker=3
|
||||
```
|
||||
|
||||
@ -29,30 +29,7 @@ Independent of the preferred installation method, InvenTree provides a number of
|
||||
|
||||
## System Components
|
||||
|
||||
The InvenTree server ecosystem consists of the following components:
|
||||
|
||||
### Database
|
||||
|
||||
A persistent database is required for data storage. By default, InvenTree is configured to use [PostgreSQL](https://www.postgresql.org/) - and this is the recommended database backend to use. However, InvenTree can also be configured to connect to any database backend [supported by Django]({% include "django.html" %}/ref/databases/)
|
||||
|
||||
|
||||
### Web Server
|
||||
|
||||
The bulk of the InvenTree code base supports the custom web server application. The web server application services user requests and facilitates database access. The webserver provides access to the [API](../api/api.md) for performing database query actions.
|
||||
|
||||
InvenTree uses [Gunicorn](https://gunicorn.org/) as the web server - a Python WSGI HTTP server.
|
||||
|
||||
### Background Tasks
|
||||
|
||||
A separate application handles management of [background tasks](../settings/tasks.md), separate to user-facing web requests. The background task manager is required to perform asynchronous tasks, such as sending emails, generating reports, and other long-running tasks.
|
||||
|
||||
InvenTree uses [django-q2](https://django-q2.readthedocs.io/en/master/) as the background task manager.
|
||||
|
||||
### File Storage
|
||||
|
||||
Uploaded *media* files (images, attachments, reports, etc) and *static* files (javascript, html) are stored to a persistent storage volume. A *file server* is required to serve these files to the user.
|
||||
|
||||
InvenTree uses [Caddy](https://caddyserver.com/) as a file server, which is configured to serve both *static* and *media* files. Additionally, Caddy provides SSL termination and reverse proxy services.
|
||||
The InvenTree software stack is composed of multiple components, each of which is required for a fully functional server environment. Your can read more about the [InvenTree processes here](./processes.md).
|
||||
|
||||
## OS Requirements
|
||||
|
||||
@ -132,4 +109,4 @@ So, for a production setup, you should set `INVENTREE_DEBUG=false` in the [confi
|
||||
Turning off DEBUG mode creates further work for the system administrator. In particular, when running in DEBUG mode, the InvenTree web server natively manages *static* and *media* files, which means that the InvenTree server can run "monolithically" without the need for a separate web server.
|
||||
|
||||
!!! info "Read More"
|
||||
Refer to the [Serving Files](./serving_files.md) section for more details
|
||||
Refer to the [proxy server documentation](./processes.md#proxy-server) for more details
|
||||
|
||||
103
docs/docs/start/processes.md
Normal file
103
docs/docs/start/processes.md
Normal file
@ -0,0 +1,103 @@
|
||||
---
|
||||
title: InvenTree Processes
|
||||
---
|
||||
|
||||
## InvenTree Processes
|
||||
|
||||
InvenTree is a complex application, and there are a number of processes which must be running in order for the system to operate correctly. Typically, these processes are started automatically when the InvenTree application stack is launched. However, in some cases, it may be necessary to start these processes manually.
|
||||
|
||||
System administrators should be aware of the following processes when configuring an InvenTree installation:
|
||||
|
||||
### Database
|
||||
|
||||
At the core of the InvenTree application is the SQL database. The database is responsible for storing all of the persistent data which is used by the InvenTree application.
|
||||
|
||||
InvenTree supports a [number of database backends]({% include "django.html" %}/ref/databases) - which typically require their own process to be running.
|
||||
|
||||
Refer to the [database configuration guide](./config.md#database-options) for more information on selecting and configuring the database backend.
|
||||
|
||||
In running InvenTree via [docker compose](./docker_install.md), the database process is managed by the `inventree-db` service which provides a [Postgres docker container](https://hub.docker.com/_/postgres).
|
||||
|
||||
### Web Server
|
||||
|
||||
The InvenTree web server is responsible for serving the InvenTree web interface to users. The web server is a [Django](https://www.djangoproject.com/) application, which is run using the [Gunicorn](https://gunicorn.org/) WSGI server.
|
||||
|
||||
The web server interfaces with the backend database and provides a [REST API](../api/api.md) (via the [Django REST framework](https://www.django-rest-framework.org/)) which is used by the frontend web interface.
|
||||
|
||||
In running InvenTree via [docker compose](./docker_install.md), the web server process is managed by the `inventree-server` service, which runs from a custom docker image.
|
||||
|
||||
### Proxy Server
|
||||
|
||||
In a production installation, the InvenTree web server application *does not* provide hosting of static files, or user-uploaded (media) files. Instead, these files should be served by a separate web server, such as [Caddy](https://caddyserver.com/), [Nginx](https://www.nginx.com/), or [Apache](https://httpd.apache.org/).
|
||||
|
||||
!!! info "Debug Mode"
|
||||
When running in [production mode](./bare_prod.md) (i.e. the `INVENTREE_DEBUG` flag is disabled), a separate web server is required for serving *static* and *media* files. In `DEBUG` mode, the django webserver facilitates delivery of *static* and *media* files, but this is explicitly not suitable for a production environment.
|
||||
|
||||
!!! tip "Read More"
|
||||
You can find further information in the [django documentation]({% include "django.html" %}/howto/static-files/deployment/).
|
||||
|
||||
A proxy server is required to store and serve static files (such as images, documents, etc) which are used by the InvenTree application. As django is not designed to serve static files in a production environment, a separate file server is required. For our docker compose setup, we use the `inventree-proxy` service, which runs a [Caddy](https://caddyserver.com/) proxy server to serve static files.
|
||||
|
||||
In addition to serving static files, the proxy server also provides a reverse proxy to the InvenTree web server, allowing the InvenTree web interface to be accessed via a standard HTTP/HTTPS port.
|
||||
|
||||
Further, it provides an authentication endpoint for accessing files in the `/static/` and `/media/` directories.
|
||||
|
||||
Finally, it provides a [Let's Encrypt](https://letsencrypt.org/) endpoint for automatic SSL certificate generation and renewal.
|
||||
|
||||
#### Static Files
|
||||
|
||||
Static files can be served without any need for authentication. In fact, they must be accessible *without* authentication, otherwise the unauthenticated views (such as the login screen) will not function correctly.
|
||||
|
||||
#### Media Files
|
||||
|
||||
It is highly recommended that the *media* files are served behind an authentication layer. This is because the media files are user-uploaded, and may contain sensitive information. Most modern web servers provide a way to serve files behind an authentication layer.
|
||||
|
||||
#### Example Configuration
|
||||
|
||||
The [docker production example](./docker.md) provides an example using [Caddy](https://caddyserver.com) to serve *static* and *media* files, and redirecting other requests to the InvenTree web server itself.
|
||||
|
||||
Caddy is a modern web server which is easy to configure and provides a number of useful features, including automatic SSL certificate generation.
|
||||
|
||||
#### Alternatives to Caddy
|
||||
|
||||
An alternative is to run nginx as the reverse proxy. A sample configuration file is provided in the `./contrib/container/` source directory.
|
||||
|
||||
#### Integrating with Existing Proxy
|
||||
|
||||
You may wish to integrate the InvenTree web server with an existing reverse proxy server. This is possible, but requires careful configuration to ensure that the static and media files are served correctly.
|
||||
|
||||
*Note: A custom configuration of the proxy server is outside the scope of this documentation!*
|
||||
|
||||
### Background Worker
|
||||
|
||||
The InvenTree background worker is responsible for handling [asynchronous tasks](../settings/tasks.md) which are not suitable for the main web server process. This includes tasks such as sending emails, generating reports, and other long-running tasks.
|
||||
|
||||
InvenTree uses the [django-q2](https://django-q2.readthedocs.io/en/master/) package to manage background tasks.
|
||||
|
||||
The background worker process is managed by the `inventree-worker` service in the [docker compose](./docker_install.md) setup. Note that this services runs a duplicate copy of the `inventree-server` container, but with a different entrypoint command which starts the background worker process.
|
||||
|
||||
#### Important Information
|
||||
|
||||
If the background worker process is not running, InvenTree will not be able to perform background tasks. This can lead to issues such as emails not being sent, or reports not being generated. Additionally, certain data may not be updated correctly if the background worker is not running.
|
||||
|
||||
!!! warning "Background Worker"
|
||||
The background worker process is a critical part of the InvenTree application stack. It is important that this process is running correctly in order for the InvenTree application to operate correctly.
|
||||
|
||||
#### Limitations
|
||||
|
||||
If the [cache server](#cache-server) is not running, the background worker will be limited to running a single threaded worker. This is because the background worker uses the cache server to manage task locking, and without a global cache server to communicate between processes, concurrency issues can occur.
|
||||
|
||||
### Cache Server
|
||||
|
||||
The InvenTree cache server is used to store temporary data which is shared between the InvenTree web server and the background worker processes. The cache server is also used to store task information, and to manage task locking between the background worker processes.
|
||||
|
||||
Using a cache server can significantly improve the performance of the InvenTree application, as it reduces the need to query the database for frequently accessed data.
|
||||
|
||||
InvenTree uses the [Redis](https://redis.io/) cache server to manage cache data. When running in docker, we use the official [redis image](https://hub.docker.com/_/redis).
|
||||
|
||||
!!! info "Redis on Docker"
|
||||
Docker adds an additional network layer - that might lead to lower performance than bare metal.
|
||||
To optimize and configure your redis deployment follow the [official docker guide](https://redis.io/docs/getting-started/install-stack/docker/#configuration).
|
||||
|
||||
!!! tip "Enable Cache"
|
||||
While a redis container is provided in the default configuration, by default it is not enabled in the Inventree server. You can enable redis cache support by following the [caching configuration guide](./config.md#caching)
|
||||
@ -1,29 +0,0 @@
|
||||
---
|
||||
title: Serving Static and Media Files
|
||||
---
|
||||
|
||||
## Serving Files
|
||||
|
||||
In a production installation, the InvenTree web server application *does not* provide hosting of static files, or user-uploaded (media) files. Instead, these files should be served by a separate web server, such as [Caddy](https://caddyserver.com/), [Nginx](https://www.nginx.com/), or [Apache](https://httpd.apache.org/).
|
||||
|
||||
!!! info "Debug Mode"
|
||||
When running in [production mode](./bare_prod.md) (i.e. the `INVENTREE_DEBUG` flag is disabled), a separate web server is required for serving *static* and *media* files. In `DEBUG` mode, the django webserver facilitates delivery of *static* and *media* files, but this is explicitly not suitable for a production environment.
|
||||
|
||||
!!! tip "Read More"
|
||||
You can find further information in the [django documentation]({% include "django.html" %}/howto/static-files/deployment/).
|
||||
|
||||
There are *many* different ways that a sysadmin might wish to handle this - and it depends on your particular installation requirements.
|
||||
|
||||
### Static Files
|
||||
|
||||
Static files can be served without any need for authentication. In fact, they must be accessible *without* authentication, otherwise the unauthenticated views (such as the login screen) will not function correctly.
|
||||
|
||||
### Media Files
|
||||
|
||||
It is highly recommended that the *media* files are served behind an authentication layer. This is because the media files are user-uploaded, and may contain sensitive information. Most modern web servers provide a way to serve files behind an authentication layer.
|
||||
|
||||
### Example Configuration
|
||||
|
||||
The [docker production example](./docker.md) provides an example using [Caddy](https://caddyserver.com) to serve *static* and *media* files, and redirecting other requests to the InvenTree web server itself.
|
||||
|
||||
Caddy is a modern web server which is easy to configure and provides a number of useful features, including automatic SSL certificate generation.
|
||||
@ -88,6 +88,7 @@ nav:
|
||||
- Security: security.md
|
||||
- Install:
|
||||
- Introduction: start/intro.md
|
||||
- Processes: start/processes.md
|
||||
- Configuration: start/config.md
|
||||
- Docker:
|
||||
- Introduction: start/docker.md
|
||||
@ -97,7 +98,6 @@ nav:
|
||||
- Installer: start/installer.md
|
||||
- Production: start/bare_prod.md
|
||||
- Development: start/bare_dev.md
|
||||
- Serving Files: start/serving_files.md
|
||||
- User Accounts: start/accounts.md
|
||||
- Data Backup: start/backup.md
|
||||
- Invoke: start/invoke.md
|
||||
|
||||
@ -221,6 +221,7 @@ class InfoView(AjaxView):
|
||||
'instance': InvenTree.version.inventreeInstanceName(),
|
||||
'apiVersion': InvenTree.version.inventreeApiVersion(),
|
||||
'worker_running': is_worker_running(),
|
||||
'worker_count': settings.BACKGROUND_WORKER_COUNT,
|
||||
'worker_pending_tasks': self.worker_pending_tasks(),
|
||||
'plugins_enabled': settings.PLUGINS_ENABLED,
|
||||
'plugins_install_disabled': settings.PLUGINS_INSTALL_DISABLED,
|
||||
|
||||
@ -851,13 +851,20 @@ _q_worker_timeout = int(
|
||||
get_setting('INVENTREE_BACKGROUND_TIMEOUT', 'background.timeout', 90)
|
||||
)
|
||||
|
||||
|
||||
# Prevent running multiple background workers if global cache is disabled
|
||||
# This is to prevent scheduling conflicts due to the lack of a shared cache
|
||||
BACKGROUND_WORKER_COUNT = (
|
||||
int(get_setting('INVENTREE_BACKGROUND_WORKERS', 'background.workers', 4))
|
||||
if GLOBAL_CACHE_ENABLED
|
||||
else 1
|
||||
)
|
||||
|
||||
# django-q background worker configuration
|
||||
Q_CLUSTER = {
|
||||
'name': 'InvenTree',
|
||||
'label': 'Background Tasks',
|
||||
'workers': int(
|
||||
get_setting('INVENTREE_BACKGROUND_WORKERS', 'background.workers', 4)
|
||||
),
|
||||
'workers': BACKGROUND_WORKER_COUNT,
|
||||
'timeout': _q_worker_timeout,
|
||||
'retry': max(120, _q_worker_timeout + 30),
|
||||
'max_attempts': int(
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user