2
0
mirror of https://github.com/inventree/InvenTree.git synced 2025-06-12 10:05:39 +00:00

Code structure refactor (#5582)

* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fit in changes from testing

* gather packager improvements (#149)

* Matmair/issue5578 (#143)

* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fix docker path

* use project dir

* move project dir command

* fixed docker paths

* another fix?

* seperate tasks out

* remove tasks

* some debugging

* ci: add .deepsource.toml

* Update .deepsource.toml

* also ignore migrations

* more debugging

* fix path issues

* remove debug script

* fix style

* change locale path

* Fixed paths for requirements

* Added dummy requirements to fool packager

* fixed exec path

* remove deepsource

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>

* Added docs for file structure

* Fixed style errors

* updated deepsource paths

* fix deepsource paths

* fixed reqs

* merge fixes

* move newly added dirs too

* fix reqs files

* another dep fix

* merge upstream/master

* revert removal of tags

* merge upstream

* enabled detection of old config files

* adapt coverage src

* also detect and support old location for plugins.txt

* style fix

* fix ~/init.sh location

* fix requirements path

* fix config to current master

* move new folders

* fix import order

* fix paths for qc_check

* fix docs build

* fix fix path

* set docker project dir

* just use a cd

* set image path?

* set file correct

* fix copy path

* fix tasks dir

* fix init path

* fix copy path

* set prject dir

* fix paths

* remove old prod files

* fix dev env path

* set docker file

* Fix devcontainer docker compose file

* fix login attempt values

* fix init.sh path

* Fix pathing for Docker

* Docker build fix

- Set INVENTREE_BACKEND_DIR separately

* Update init.sh

* Fix path

* Update requirements.txt

* merge

* fix rq merge

* fix docker compose usage

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>
Co-authored-by: Oliver <oliver.henry.walters@gmail.com>
This commit is contained in:
Matthias Mair
2024-04-03 02:16:59 +01:00
committed by GitHub
parent dd889e8eeb
commit 0bace3f3af
3317 changed files with 532 additions and 1194 deletions

49
contrib/container/.env Normal file
View File

@ -0,0 +1,49 @@
# InvenTree environment variables for docker compose deployment
# Specify the location of the external data volume
# By default, placed in local directory 'inventree-data'
INVENTREE_EXT_VOLUME=./inventree-data
# Ensure debug is false for a production setup
INVENTREE_DEBUG=False
INVENTREE_LOG_LEVEL=WARNING
# InvenTree admin account details
# Un-comment (and complete) these lines to auto-create an admin acount
#INVENTREE_ADMIN_USER=
#INVENTREE_ADMIN_PASSWORD=
#INVENTREE_ADMIN_EMAIL=
# Database configuration options
INVENTREE_DB_ENGINE=postgresql
INVENTREE_DB_NAME=inventree
INVENTREE_DB_HOST=inventree-db
INVENTREE_DB_PORT=5432
# Database credentials - These should be changed from the default values!
INVENTREE_DB_USER=pguser
INVENTREE_DB_PASSWORD=pgpassword
# Redis cache setup (disabled by default)
# Un-comment the following lines to enable Redis cache
# Note that you will also have to run docker-compose with the --profile redis command
# Refer to settings.py for other cache options
#INVENTREE_CACHE_HOST=inventree-cache
#INVENTREE_CACHE_PORT=6379
# Options for gunicorn server
INVENTREE_GUNICORN_TIMEOUT=90
# Enable custom plugins?
INVENTREE_PLUGINS_ENABLED=True
# Run migrations automatically?
INVENTREE_AUTO_UPDATE=True
# Image tag that should be used
INVENTREE_TAG=stable
# Site URL - update this to match your host
INVENTREE_SITE_URL="http://inventree.localhost"
COMPOSE_PROJECT_NAME=inventree

View File

@ -0,0 +1,56 @@
# Caddyfile for Inventree
# The following environment variables may be used:
# - INVENTREE_SITE_URL: The upstream URL of the Inventree site (default: inventree.localhost)
# - INVENTREE_SERVER: The internal URL of the Inventree container (default: http://inventree-server:8000)
(log_common) {
log {
output file /var/log/caddy/{args[0]}.access.log
}
}
(cors-headers) {
header Allow GET,HEAD,OPTIONS
header Access-Control-Allow-Origin *
header Access-Control-Allow-Methods GET,HEAD,OPTIONS
header Access-Control-Allow-Headers Authorization,Content-Type,User-Agent
@cors_preflight{args[0]} method OPTIONS
handle @cors_preflight{args[0]} {
respond "" 204
}
}
# Change the host to your domain (this will serve at inventree.localhost)
{$INVENTREE_SITE_URL:inventree.localhost} {
import log_common inventree
encode gzip
request_body {
max_size 100MB
}
handle_path /static/* {
import cors-headers static
root * /var/www/static
file_server
}
handle_path /media/* {
import cors-headers media
root * /var/www/media
file_server
header Content-Disposition attachment
forward_auth {$INVENTREE_SERVER:"http://inventree-server:8000"} {
uri /auth/
}
}
reverse_proxy {$INVENTREE_SERVER:"http://inventree-server:8000"}
}

View File

@ -0,0 +1,166 @@
# The InvenTree dockerfile provides two build targets:
#
# production:
# - Required files are copied into the image
# - Runs InvenTree web server under gunicorn
#
# dev:
# - Expects source directories to be loaded as a run-time volume
# - Runs InvenTree web server under django development server
# - Monitors source files for any changes, and live-reloads server
ARG base_image=python:3.11-alpine3.18
FROM ${base_image} AS inventree_base
# Build arguments for this image
ARG commit_tag=""
ARG commit_hash=""
ARG commit_date=""
ARG data_dir="data"
ENV PYTHONUNBUFFERED 1
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
ENV INVOKE_RUN_SHELL="/bin/ash"
ENV INVENTREE_DOCKER="true"
# InvenTree paths
ENV INVENTREE_HOME="/home/inventree"
ENV INVENTREE_DATA_DIR="${INVENTREE_HOME}/${data_dir}"
ENV INVENTREE_STATIC_ROOT="${INVENTREE_DATA_DIR}/static"
ENV INVENTREE_MEDIA_ROOT="${INVENTREE_DATA_DIR}/media"
ENV INVENTREE_BACKUP_DIR="${INVENTREE_DATA_DIR}/backup"
ENV INVENTREE_PLUGIN_DIR="${INVENTREE_DATA_DIR}/plugins"
# InvenTree configuration files
ENV INVENTREE_CONFIG_FILE="${INVENTREE_DATA_DIR}/config.yaml"
ENV INVENTREE_SECRET_KEY_FILE="${INVENTREE_DATA_DIR}/secret_key.txt"
ENV INVENTREE_PLUGIN_FILE="${INVENTREE_DATA_DIR}/plugins.txt"
# Worker configuration (can be altered by user)
ENV INVENTREE_GUNICORN_WORKERS="4"
ENV INVENTREE_BACKGROUND_WORKERS="4"
# Default web server address:port
ENV INVENTREE_WEB_ADDR=0.0.0.0
ENV INVENTREE_WEB_PORT=8000
LABEL org.label-schema.schema-version="1.0" \
org.label-schema.build-date=${DATE} \
org.label-schema.vendor="inventree" \
org.label-schema.name="inventree/inventree" \
org.label-schema.url="https://hub.docker.com/r/inventree/inventree" \
org.label-schema.vcs-url="https://github.com/inventree/InvenTree.git" \
org.label-schema.vcs-ref=${commit_tag}
# Install required system level packages
RUN apk add --no-cache \
git gettext py-cryptography \
# Image format support
libjpeg libwebp zlib \
# Weasyprint requirements : https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#alpine-3-12
py3-pip py3-pillow py3-cffi py3-brotli pango poppler-utils openldap \
# Postgres client
postgresql13-client \
# MySQL / MariaDB client
mariadb-client mariadb-connector-c \
&& \
# fonts
apk --update --upgrade --no-cache add fontconfig ttf-freefont font-noto terminus-font && fc-cache -f
EXPOSE 8000
RUN mkdir -p ${INVENTREE_HOME}
WORKDIR ${INVENTREE_HOME}
COPY contrib/container/requirements.txt base_requirements.txt
COPY src/backend/requirements.txt ./
COPY contrib/container/install_build_packages.sh .
RUN chmod +x install_build_packages.sh
# For ARMv7 architecture, add the piwheels repo (for cryptography library)
# Otherwise, we have to build from source, which is difficult
# Ref: https://github.com/inventree/InvenTree/pull/4598
RUN if [ `apk --print-arch` = "armv7" ]; then \
printf "[global]\nextra-index-url=https://www.piwheels.org/simple\n" > /etc/pip.conf ; \
fi
COPY tasks.py contrib/container/gunicorn.conf.py contrib/container/init.sh ./
RUN chmod +x init.sh
ENTRYPOINT ["/bin/ash", "./init.sh"]
FROM inventree_base AS prebuild
ENV PATH=/root/.local/bin:$PATH
RUN ./install_build_packages.sh --no-cache --virtual .build-deps && \
pip install --user -r base_requirements.txt -r requirements.txt --no-cache && \
apk --purge del .build-deps
# Frontend builder image:
FROM prebuild AS frontend
RUN apk add --no-cache --update nodejs npm && npm install -g yarn@v1.22.22
RUN yarn config set network-timeout 600000 -g
COPY src ${INVENTREE_HOME}/src
COPY tasks.py ${INVENTREE_HOME}/tasks.py
RUN cd ${INVENTREE_HOME} && inv frontend-compile
# InvenTree production image:
# - Copies required files from local directory
# - Starts a gunicorn webserver
FROM inventree_base AS production
ENV INVENTREE_DEBUG=False
# As .git directory is not available in production image, we pass the commit information via ENV
ENV INVENTREE_COMMIT_HASH="${commit_hash}"
ENV INVENTREE_COMMIT_DATE="${commit_date}"
# use dependencies and compiled wheels from the prebuild image
ENV PATH=/root/.local/bin:$PATH
COPY --from=prebuild /root/.local /root/.local
ENV INVENTREE_BACKEND_DIR="${INVENTREE_HOME}"
# Copy source code
COPY src/backend/InvenTree ./InvenTree
COPY --from=frontend ${INVENTREE_HOME}/src/backend/InvenTree/web/static/web ./src/backend/InvenTree/web/static/web
# Launch the production server
CMD gunicorn -c ./gunicorn.conf.py InvenTree.wsgi -b 0.0.0.0:8000 --chdir ./InvenTree
FROM inventree_base AS dev
ENV INVENTREE_BACKEND_DIR="${INVENTREE_HOME}/src/backend"
# Vite server (for local frontend development)
EXPOSE 5173
# Install packages required for building python packages
RUN ./install_build_packages.sh
RUN pip install uv==0.1.26 --no-cache-dir && pip install -r base_requirements.txt --no-cache
# Install nodejs / npm / yarn
RUN apk add --no-cache --update nodejs npm && npm install -g yarn@v1.22.22
RUN yarn config set network-timeout 600000 -g
# The development image requires the source code to be mounted to /home/inventree/
# So from here, we don't actually "do" anything, apart from some file management
ENV INVENTREE_DEBUG=True
# Location for python virtual environment
# If the INVENTREE_PY_ENV variable is set, the entrypoint script will use it!
ENV INVENTREE_PY_ENV="${INVENTREE_DATA_DIR}/env"
WORKDIR ${INVENTREE_HOME}
# Entrypoint ensures that we are running in the python virtual environment
ENTRYPOINT ["/bin/ash", "./contrib/container/init.sh"]
# Launch the development server
CMD ["invoke", "server", "-a", "${INVENTREE_WEB_ADDR}:${INVENTREE_WEB_PORT}"]

View File

@ -0,0 +1,69 @@
version: "3.8"
# Docker compose recipe for InvenTree development server
# - Runs PostgreSQL as the database backend
# - Uses built-in django webserver
# - Runs the InvenTree background worker process
# - Serves media and static content directly from Django webserver
# IMPORTANT NOTE:
# The InvenTree development image does not clone source code from git.
# Instead, it runs from source code on your local machine.
# The django server will auto-detect any code changes and reload the server.
# If you have cloned the InvenTree git repo, and not made any changes to this file,
# then the default setup in this file should work straight out of the box, without modification
services:
# Database service
# Use PostgreSQL as the database backend
# Note: This can be changed to a different backend if required
inventree-dev-db:
image: postgres:13
expose:
- 5432/tcp
environment:
- PGDATA=/var/lib/postgresql/data/pgdb
- POSTGRES_USER=pguser
- POSTGRES_PASSWORD=pgpassword
- POSTGRES_DB=inventree
volumes:
# Map 'data' volume such that postgres database is stored externally
- ./data:/var/lib/postgresql/data:z
restart: unless-stopped
# InvenTree web server service
# Runs the django built-in webserver application
inventree-dev-server:
depends_on:
- inventree-dev-db
build: &build_config
context: .
dockerfile: contrib/container/Dockerfile
target: dev
# Cache the built image to be used by the inventree-dev-worker process
image: inventree-dev-image
ports:
# Expose web server on port 8000
- 8000:8000
volumes:
# Mount local source directory to /home/inventree
- ./:/home/inventree:z
env_file:
- contrib/container/docker.dev.env
restart: unless-stopped
# Background worker process handles long-running or periodic tasks
inventree-dev-worker:
image: inventree-dev-image
build: *build_config
command: invoke worker
depends_on:
- inventree-dev-server
volumes:
# Mount local source directory to /home/inventree
- ./:/home/inventree:z
env_file:
- contrib/container/docker.dev.env
restart: unless-stopped

View File

@ -0,0 +1,122 @@
version: "3.8"
# Docker compose recipe for a production-ready InvenTree setup, with the following containers:
# - PostgreSQL as the database backend
# - gunicorn as the InvenTree web server
# - django-q as the InvenTree background worker process
# - Caddy as a reverse proxy
# - redis as the cache manager (optional, disabled by default)
# ---------------------
# READ BEFORE STARTING!
# ---------------------
# -----------------------------
# Setting environment variables
# -----------------------------
# Shared environment variables should be stored in the .env file
# Changes made to this file are reflected across all containers!
#
# IMPORTANT NOTE:
# You should not have to change *anything* within this docker-compose.yml file!
# Instead, make any changes in the .env file!
# ------------------------
# InvenTree Image Versions
# ------------------------
# By default, this docker-compose script targets the STABLE version of InvenTree,
# image: inventree/inventree:stable
#
# To run the LATEST (development) version of InvenTree,
# change the INVENTREE_TAG variable (in the .env file) to "latest"
#
# Alternatively, you could target a specific tagged release version with (for example):
# INVENTREE_TAG=0.7.5
#
services:
# Database service
# Use PostgreSQL as the database backend
inventree-db:
image: postgres:13
container_name: inventree-db
expose:
- ${INVENTREE_DB_PORT:-5432}/tcp
environment:
- PGDATA=/var/lib/postgresql/data/pgdb
- POSTGRES_USER=${INVENTREE_DB_USER:?You must provide the 'INVENTREE_DB_USER' variable in the .env file}
- POSTGRES_PASSWORD=${INVENTREE_DB_PASSWORD:?You must provide the 'INVENTREE_DB_PASSWORD' variable in the .env file}
- POSTGRES_DB=${INVENTREE_DB_NAME:?You must provide the 'INVENTREE_DB_NAME' variable in the .env file}
volumes:
# Map 'data' volume such that postgres database is stored externally
- ${INVENTREE_EXT_VOLUME:?You must specify the 'INVENTREE_EXT_VOLUME' variable in the .env file!}:/var/lib/postgresql/data/:z
restart: unless-stopped
# redis acts as database cache manager
# only runs under the "redis" profile : https://docs.docker.com/compose/profiles/
inventree-cache:
image: redis:7.0
container_name: inventree-cache
depends_on:
- inventree-db
profiles:
- redis
env_file:
- .env
expose:
- ${INVENTREE_CACHE_PORT:-6379}
restart: always
# InvenTree web server service
# Uses gunicorn as the web server
inventree-server:
# If you wish to specify a particular InvenTree version, do so here
image: inventree/inventree:${INVENTREE_TAG:-stable}
container_name: inventree-server
# Only change this port if you understand the stack.
expose:
- 8000
depends_on:
- inventree-db
env_file:
- .env
volumes:
# Data volume must map to /home/inventree/data
- ${INVENTREE_EXT_VOLUME}:/home/inventree/data:z
restart: unless-stopped
# Background worker process handles long-running or periodic tasks
inventree-worker:
# If you wish to specify a particular InvenTree version, do so here
image: inventree/inventree:${INVENTREE_TAG:-stable}
container_name: inventree-worker
command: invoke worker
depends_on:
- inventree-server
env_file:
- .env
volumes:
# Data volume must map to /home/inventree/data
- ${INVENTREE_EXT_VOLUME}:/home/inventree/data:z
restart: unless-stopped
# caddy acts as reverse proxy and static file server
# https://hub.docker.com/_/caddy
inventree-proxy:
container_name: inventree-proxy
image: caddy:alpine
restart: always
depends_on:
- inventree-server
ports:
- ${INVENTREE_WEB_PORT:-80}:80
- 443:443
env_file:
- .env
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- ${INVENTREE_EXT_VOLUME}/static:/var/www/static:z
- ${INVENTREE_EXT_VOLUME}/media:/var/www/media:z
- ${INVENTREE_EXT_VOLUME}:/var/log:z
- ${INVENTREE_EXT_VOLUME}:/data:z
- ${INVENTREE_EXT_VOLUME}:/config:z

View File

@ -0,0 +1,22 @@
# InvenTree environment variables for a development setup
# These variables will be used by the docker-compose.yml file
# Set DEBUG to True for a development setup
INVENTREE_DEBUG=True
INVENTREE_LOG_LEVEL=INFO
INVENTREE_DB_LOGGING=False
# Database configuration options
# Note: The example setup is for a PostgreSQL database (change as required)
INVENTREE_DB_ENGINE=postgresql
INVENTREE_DB_NAME=inventree
INVENTREE_DB_HOST=inventree-dev-db
INVENTREE_DB_PORT=5432
INVENTREE_DB_USER=pguser
INVENTREE_DB_PASSWORD=pgpassword
# Enable custom plugins?
INVENTREE_PLUGINS_ENABLED=True
# Auto run migrations?
INVENTREE_AUTO_UPDATE=True

View File

@ -0,0 +1,42 @@
"""Gunicorn configuration for InvenTree."""
import logging
import multiprocessing
import os
# Logger configuration
logger = logging.getLogger('inventree')
accesslog = '-'
errorlog = '-'
loglevel = os.environ.get('INVENTREE_LOG_LEVEL', 'warning').lower()
capture_output = True
# Worker configuration
# TODO: Implement support for gevent
# worker_class = 'gevent' # Allow multi-threading support
worker_tmp_dir = '/dev/shm' # Write temp file to RAM (faster)
threads = 4
# Worker timeout (default = 90 seconds)
timeout = os.environ.get('INVENTREE_GUNICORN_TIMEOUT', 90)
# Number of worker processes
workers = os.environ.get('INVENTREE_GUNICORN_WORKERS', None)
if workers is not None:
try:
workers = int(workers)
except ValueError:
workers = None
if workers is None:
workers = multiprocessing.cpu_count() * 2 + 1
logger.info('Starting gunicorn server with %s workers', workers)
max_requests = 1000
max_requests_jitter = 50
# preload app so that the ready functions are only executed once
preload_app = True

56
contrib/container/init.sh Normal file
View File

@ -0,0 +1,56 @@
#!/bin/ash
# exit when any command fails
set -e
# Required to suppress some git errors further down the line
if command -v git &> /dev/null; then
git config --global --add safe.directory /home/***
fi
# Create required directory structure (if it does not already exist)
if [[ ! -d "$INVENTREE_STATIC_ROOT" ]]; then
echo "Creating directory $INVENTREE_STATIC_ROOT"
mkdir -p $INVENTREE_STATIC_ROOT
fi
if [[ ! -d "$INVENTREE_MEDIA_ROOT" ]]; then
echo "Creating directory $INVENTREE_MEDIA_ROOT"
mkdir -p $INVENTREE_MEDIA_ROOT
fi
if [[ ! -d "$INVENTREE_BACKUP_DIR" ]]; then
echo "Creating directory $INVENTREE_BACKUP_DIR"
mkdir -p $INVENTREE_BACKUP_DIR
fi
# Check if "config.yaml" has been copied into the correct location
if test -f "$INVENTREE_CONFIG_FILE"; then
echo "Loading config file : $INVENTREE_CONFIG_FILE"
else
echo "Copying config file from $INVENTREE_BACKEND_DIR/InvenTree/config_template.yml to $INVENTREE_CONFIG_FILE"
cp $INVENTREE_BACKEND_DIR/InvenTree/config_template.yaml $INVENTREE_CONFIG_FILE
fi
# Setup a python virtual environment
# This should be done on the *mounted* filesystem,
# so that the installed modules persist!
if [[ -n "$INVENTREE_PY_ENV" ]]; then
if test -d "$INVENTREE_PY_ENV"; then
# venv already exists
echo "Using Python virtual environment: ${INVENTREE_PY_ENV}"
else
# Setup a virtual environment (within the "data/env" directory)
echo "Running first time setup for python environment"
python3 -m venv ${INVENTREE_PY_ENV} --system-site-packages --upgrade-deps
fi
# Now activate the venv
source ${INVENTREE_PY_ENV}/bin/activate
fi
cd ${INVENTREE_HOME}
# Launch the CMD *after* the ENTRYPOINT completes
exec "$@"

View File

@ -0,0 +1,12 @@
#!/bin/ash
# Install system packages required for building InvenTree python libraries
# Note that for postgreslql, we use the 13 version, which matches the version used in the InvenTree docker image
apk add gcc g++ musl-dev openssl-dev libffi-dev cargo python3-dev openldap-dev \
libstdc++ build-base linux-headers py3-grpcio \
jpeg-dev openjpeg-dev libwebp-dev zlib-dev \
sqlite sqlite-dev \
mariadb-connector-c-dev mariadb-client mariadb-dev \
postgresql13-dev postgresql-libs \
$@

View File

@ -0,0 +1,22 @@
# Base python requirements for docker containers
# Basic package requirements
invoke>=2.2.0 # Invoke build tool
pyyaml>=6.0.1
setuptools>=69.0.0
wheel>=0.41.0
# Database links
psycopg[binary, pool]
mysqlclient>=2.2.0
mariadb>=1.1.8
# gunicorn web server
gunicorn>=21.2.0
# LDAP required packages
django-auth-ldap # Django integration for ldap auth
python-ldap # LDAP auth support
# Upgraded python package installer
uv