feat(docs): add architecture/roadmap (#9624)

* feat(docs): Add architecture overview * add strucutre for rough roadmap * add stable reference * document stable links * test mermaid again * fix loading * fix format to reduce warnings * use local js * add architecture ovierview * add more sub-points * more structure * add general backend text * add sme bqsic docs for frontend * fix list syntax * fix typo
2025-06-14 11:05:41 +00:00 · 2025-06-13 03:28:02 +02:00
parent c70583097c
commit 4b1eeefb1f
12 changed files with 2796 additions and 27 deletions
--- a/docs/docs/develop/architecture.md
+++ b/docs/docs/develop/architecture.md
@ -0,0 +1,81 @@
+---
+title: Architecture
+---
+
+## Typical Deployment
+
+InvenTree is a classical Django Application and supports the WSGI interface standard. The following diagram shows a typical and recommended deployment architecture of InvenTree.
+
+``` mermaid
+flowchart LR
+    Request --- RP[1: Reverse Proxy]
+    RP --- Gunicorn[5: Gunicorn]
+    subgraph image:inventree/inventree
+    Gunicorn --- Django[6: Django Server processes]
+    end
+    Django --- Database@{ shape: cyl, label: "SQL Database" }
+    Django --- Redis
+    Worker[7: Q2 Worker] ---  Redis@{ shape: cyl, label: "9: Redis Cache" }
+      Database -- 8: DB holds tasks ---> Worker
+
+    subgraph caddy
+    RP --- file[2: file server]
+end
+
+    file --- s[3: Static Files]
+    file --- m[Media Files]
+    RP-. 4: API auth request .-> Gunicorn
+```
+
+1. A Request is received by a reverse proxy (e.g. Caddy, Nginx, Apache).
+2. Requests for static or media files are either served directly by the reverse proxy or forwarded to a dedicated file server
+3. Static or media files can be served by the different file server (placing static files in a CDN for example)
+4. Media files require an API authentication request to the Django server to ensure the user has access to the file
+5. API or frontend requests are forwarded from the reverse proxy to Gunicorn, which is a WSGI server that handles server Django processes.
+6. The Gunicorn processes are loosely coubled instances of Django application - which mainly serves a REST API as the frontend only needs a few full django calls (see [frontend architecture](#frontend-architecture) below).
+7. A properly configured InvenTree instance will also run background workers (Q2 Worker) which are responsible for processing long-running tasks, such as sending notifications, generating reports or calculating expensive updates. Q2 is used as the task processing library, which runs multiple loosely coupled worker processes.
+8. The database holds tasks and is queried by the workers. This enables relatively durable task processing, as the underlying server can be restarted with minimal task loss.
+9. Various components of InvenTree can benefit from a Redis or protocol compatible cache, which is used to store frequently accessed data, such as user sessions, API tokens, global or plugin settings, or other transient data. This can help to improve performance and reduce load on the database.
+
+## Code Architecture
+
+This sections describes various architectural aspects of the InvenTree codebase and mechanisms and lifecycles in the system.
+
+Knowledege of these aspects is not required to use InvenTree, but it is helpful for developers who want to contribute to the project or to understand where / how one might extend the system with plugins or by pathching in custom functionality.
+
+### Repository layout and separation of concerns
+
+All code that is intended to be executed on the server is located in the `src/` directory. Some code in contrib might be needed to deploy or maintain an instance.
+One exception is the `tasks.py` file, that contains definitions for various maintenance tasks that can be run from the command line. This file is located in the root directory of the repository to make instructions easier.
+
+Code styles are generally only applied to the code in the `src/` directory.
+
+### Backend Architecture
+
+InvenTree's backend is a Django application. It is generally structured in a way that follows Django's conventions, but also includes some customizations and extensions to support the specific needs of InvenTree.
+Most remarkable deviations from the Django standard are:
+- Manipulation of the django app mechanisms to enable the [plugin system](#plugin-system)
+- Usage of a custom role-mapping system to make permissions more approachable
+
+The backend aims to be:
+- API first, with a RESTful API that is used by the frontend and can also be used by other applications.
+- Modular, with a clear separation of concerns between different components and apps.
+- Tested reasonably throughout with transparent test coverage
+- Following the Django and generally accepted security conventions
+
+### Frontend Architecture
+
+InvenTree's frontend is primarily a single-page application (SPA) built with React, mantine and yarn/vite for bundling.
+
+#### Serving the frontend
+It is statically pre-build and is served by the Django application as a bundle of static files. No node executable or bundling technology is required to run the frontend in production. It is however possible to use the development server for the frontend stack together with a development or production instance of the backend.
+
+Frontend and backend do not need to be serverd by the same server or on the same domain, only the API versions need to match.
+
+#### Coupling to the backend
+
+The frontend is not coupled during the lifetime of the application but it can be primed via the `INVENTREE_SETTINGS` global variable (rendered by the `spa_settings` tag).
+
+These settings can configure the base url, available backends, environment details and more.
+
+To facilitate this the html structure for the frontend is run through a performance-optimized template. There is only a very limited amount of rendering done to keep response times and surface for  security threads low.
--- a/docs/docs/develop/contributing.md
+++ b/docs/docs/develop/contributing.md
@ -246,6 +246,12 @@ T002: Double quotes should be used in tags

 New features or updates to existing features should be accompanied by user documentation.

+### Stable link references
+
+The documentation framework enables addition of redirections. This is used to build stable references for linking in external resources.
+
+New references can be added in `docs/mkdocs.yml` in the `redirect_maps` section. Both external targets and documentation pages are possible targets. All references are linted in the docs CI pipeline.
+
 ## Translations

 Any user-facing strings *must* be passed through the translation engine.
--- a/docs/docs/develop/roadmap.md
+++ b/docs/docs/develop/roadmap.md
@ -0,0 +1,35 @@
+---
+title: High level roadmap
+---
+
+## General goals
+
+## High level Epics
+
+### 1.0
+
+Smaller items can be viewed [on the milestone](https://github.com/inventree/InvenTree/issues?q=is%3Aissue%20milestone%3A1.0.0).
+
+Aiming to stabelise several aspects of the software:
+
+- Only shipping the new frontend and removing reliance on templating
+- Stabelize API for client generation
+- Making data import- / export-mechanisms more stable
+- Updating and Re-Organising documentation to enable CII best practices compliance
+
+### 2.0
+
+Smaller items can be viewed [on the milestone](https://github.com/inventree/InvenTree/issues?q=is%3Aissue%20milestone%3A2.0.0).
+
+*Proposed* goals:
+
+- Reorganise permission system to support more entrerprise structures and reduce unneeded permissions [EPIC](https://github.com/inventree/InvenTree/issues/7466)
+- Add generalised file handling [EPIC](https://github.com/inventree/InvenTree/issues/5703)
+
+### Future
+
+There are several epics that target [the horizion](https://github.com/inventree/InvenTree/issues?q=is%3Aissue%20state%3Aopen%20type%3AEpic).
+
+## Non-Goals
+
+TBD
--- a/docs/docs/javascripts/mermaid.min.js
+++ b/docs/docs/javascripts/mermaid.min.js
--- a/docs/docs/plugins/mixins/export.md
+++ b/docs/docs/plugins/mixins/export.md
@ -24,7 +24,7 @@ Each plugin can dictate which datasets are supported using the `supports_export`
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 The default implementation returns `True` for all data types.

@ -40,7 +40,7 @@ The `generate_filename` method constructs a filename for the exported file.
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 ### Adjust Columns

@ -54,7 +54,7 @@ The `update_headers` method allows the plugin to adjust the columns selected to
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 ### Queryset Filtering

@ -68,7 +68,7 @@ The `filter_queryset` method allows the plugin to provide custom filtering to th
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 ### Export Data

@ -82,7 +82,7 @@ The `export_data` method performs the step of transforming a [Django QuerySet]({
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 Note that the default implementation simply uses the builtin tabulation functionality of the provided serializer class. In most cases, this will be sufficient.

--- a/docs/docs/plugins/mixins/ui.md
+++ b/docs/docs/plugins/mixins/ui.md
@ -24,7 +24,7 @@ The entrypoint for user interface plugins is the `UserInterfaceMixin` class, whi
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 Note here that the `get_ui_features` calls other methods to extract the available features from the plugin, based on the requested feature type. These methods can be overridden to provide custom functionality.

@ -43,7 +43,7 @@ The `get_ui_features` method should return a list of `UIFeature` objects, which
      summary: False
      members: []
      extra:
-        show_sources: True
+        show_source: True

 Note that the *options* field contains fields which may be specific to a particular feature type - read the documentation below on each feature type for more information.

@ -79,7 +79,7 @@ The InvenTree dashboard is a collection of "items" which are displayed on the ma
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -93,7 +93,7 @@ The *options* field in the returned `UIFeature` object can contain the following
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -115,7 +115,7 @@ Many of the pages in the InvenTree web interface are built using a series of "pa
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -129,7 +129,7 @@ The *options* field in the returned `UIFeature` object can contain the following
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -151,7 +151,7 @@ The `get_ui_template_editors` feature type can be used to provide custom templat
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -165,7 +165,7 @@ The `get_ui_template_previews` feature type can be used to provide custom templa
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

--- a/docs/docs/plugins/mixins/validation.md
+++ b/docs/docs/plugins/mixins/validation.md
@ -26,7 +26,7 @@ A custom plugin may implement the `validate_model_deletion` method to perform cu
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -42,7 +42,7 @@ Any plugin which inherits the `ValidationMixin` can implement the `validate_mode
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -116,7 +116,7 @@ If the custom method determines that the part name is *objectionable*, it should
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -130,7 +130,7 @@ Validation of the Part IPN (Internal Part Number) field is exposed to custom plu
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -144,7 +144,7 @@ Validation of the Part IPN (Internal Part Number) field is exposed to custom plu
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -162,7 +162,7 @@ The `validate_batch_code` method allows plugins to raise an error if a batch cod
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -176,7 +176,7 @@ The `generate_batch_code` method can be implemented to generate a new batch code
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -196,7 +196,7 @@ Custom serial number validation can be implemented using the `validate_serial_nu
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -236,7 +236,7 @@ A custom plugin can implement the `convert_serial_to_int` method to determine ho
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

@ -255,7 +255,7 @@ For custom serial number schemes, it is important to provide a method to generat
      show_root_heading: False
      show_root_toc_entry: False
      extra:
-        show_sources: True
+        show_source: True
      summary: False
      members: []

--- a/docs/main.py
+++ b/docs/main.py
@ -81,7 +81,7 @@ def get_repo_url(raw=False):
    mkdocs_yml = here.joinpath('mkdocs.yml')

    with open(mkdocs_yml, encoding='utf-8') as f:
-        mkdocs_config = yaml.safe_load(f)
+        mkdocs_config = yaml.load(f, yaml.BaseLoader)
        repo_name = mkdocs_config['repo_name']

    if raw:
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@ -91,6 +91,8 @@ nav:
        - Custom Barcodes: barcodes/custom.md
    - Development:
      - Contributing: develop/contributing.md
+      - Architecture: develop/architecture.md
+      - Roadmap: develop/roadmap.md
      - Devcontainer: develop/devcontainer.md
      - React Frontend: develop/react-frontend.md
    - Mobile App:
@ -261,6 +263,9 @@ plugins:
      opening_tag: "{!"
      closing_tag: "!}"
  - search
+  - mermaid2:
+      # version: 11.6.0
+      javascript: javascripts/mermaid.min.js
  - git-revision-date-localized
  - mkdocs-simple-hooks:
      hooks:
@ -286,6 +291,8 @@ plugins:
        'sref/contrib.md': 'develop/contributing.md'  #  https://github.com/inventree/InvenTree/blob/master/CONTRIBUTING.md.
        'sref/docs.md': 'index.md' #   https://docs.inventree.org/en/latest/
        'sref/api.md': 'api/index.md' #  https://demo.inventree.org/api-doc/
+        'sref/architecture.md': 'develop/architecture.md'
+        'sref/roadmap.md': 'develop/roadmap.md'
        'sref/code.md': 'https://github.com/inventree/InvenTree/tree/master' #  https://github.com/inventree/InvenTree/tree/master
        'sref/oci-image.md': 'https://hub.docker.com/r/inventree/inventree/tags' #  https://hub.docker.com/r/inventree/inventree/tags
        'sref/releases.md': 'https://github.com/inventree/InvenTree/releases' # https://github.com/inventree/InvenTree/releases
@ -312,7 +319,7 @@ markdown_extensions:
      custom_fences:
        - name: mermaid
          class: mermaid
-          # format: !!python/name:pymdownx.superfences.fence_code_format
+          format: !!python/name:mermaid2.fence_mermaid_custom
  # - pymdownx.emoji:
  #     emoji_index: !!python/name:materialx.emoji.twemoji
  #     emoji_generator: !!python/name:materialx.emoji.to_svg
--- a/docs/requirements.in
+++ b/docs/requirements.in
@ -7,3 +7,4 @@ mkdocs-simple-hooks>=0.1,<1.0
 mkdocs-include-markdown-plugin
 neoteroi-mkdocs
 mkdocstrings[python]>=0.25.0,<=0.29.1
+mkdocs-mermaid2-plugin
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@ -18,6 +18,10 @@ backrefs==5.8 \
    --hash=sha256:c67f6638a34a5b8730812f5101376f9d41dc38c43f1fdc35cb54700f6ed4465d \
    --hash=sha256:e3a63b073867dbefd0536425f43db618578528e3896fb77be7141328642a1585
    # via mkdocs-material
+beautifulsoup4==4.13.4 \
+    --hash=sha256:9bbbb14bfde9d79f38b8cd5f8c7c85f4b8f2523190ebed90e950a8dea4cb1c4b \
+    --hash=sha256:dbb3c4e1ceae6aefebdaf2423247260cd062430a410e38c66f2baa50a8437195
+    # via mkdocs-mermaid2-plugin
 bracex==2.5.post1 \
    --hash=sha256:12c50952415bfa773d2d9ccb8e79651b8cdb1f31a42f6091b804f6ba2b4a66b6 \
    --hash=sha256:13e5732fec27828d6af308628285ad358047cec36801598368cb28bc631dbaf6
@ -135,6 +139,10 @@ colorama==0.4.6 \
    # via
    #   griffe
    #   mkdocs-material
+editorconfig==0.17.0 \
+    --hash=sha256:8739052279699840065d3a9f5c125d7d5a98daeefe53b0e5274261d77cb49aa2 \
+    --hash=sha256:fe491719c5f65959ec00b167d07740e7ffec9a3f362038c72b289330b9991dfc
+    # via jsbeautifier
 essentials==1.1.6 \
    --hash=sha256:3fd26923f5f2ece51a219dbb17b1fb22c9190d70fa2104919be92a6419521877 \
    --hash=sha256:a470e693d83c13369ebf1f488d60236b4ea99400f38db6b7224e2808c1369256
@ -205,6 +213,10 @@ jinja2==3.1.6 \
    #   mkdocs-material
    #   mkdocstrings
    #   neoteroi-mkdocs
+jsbeautifier==1.15.4 \
+    --hash=sha256:5bb18d9efb9331d825735fbc5360ee8f1aac5e52780042803943aa7f854f7592 \
+    --hash=sha256:72f65de312a3f10900d7685557f84cb61a9733c50dcc27271a39f5b0051bf528
+    # via mkdocs-mermaid2-plugin
 markdown==3.8 \
    --hash=sha256:794a929b79c5af141ef5ab0f2f642d0f7b1872981250230e72682346f7cc90dc \
    --hash=sha256:7df81e63f0df5c4b24b7d156eb81e4690595239b7d70937d0409f1b0de319c6f
@ -306,6 +318,7 @@ mkdocs==1.6.1 \
    #   mkdocs-include-markdown-plugin
    #   mkdocs-macros-plugin
    #   mkdocs-material
+    #   mkdocs-mermaid2-plugin
    #   mkdocs-redirects
    #   mkdocs-simple-hooks
    #   mkdocstrings
@ -340,6 +353,10 @@ mkdocs-material-extensions==1.3.1 \
    --hash=sha256:10c9511cea88f568257f960358a467d12b970e1f7b2c0e5fb2bb48cab1928443 \
    --hash=sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31
    # via mkdocs-material
+mkdocs-mermaid2-plugin==1.2.1 \
+    --hash=sha256:22d2cf2c6867d4959a5e0903da2dde78d74581fc0b107b791bc4c7ceb9ce9741 \
+    --hash=sha256:9c7694c73a65905ac1578f966e5c193325c4d5a5bc1836727e74ac9f99d0e921
+    # via -r docs/requirements.in
 mkdocs-redirects==1.2.2 \
    --hash=sha256:3094981b42ffab29313c2c1b8ac3969861109f58b2dd58c45fc81cd44bfa0095 \
    --hash=sha256:7dbfa5647b79a3589da4401403d69494bd1f4ad03b9c15136720367e1f340ed5
@ -393,6 +410,7 @@ pymdown-extensions==10.15 \
    --hash=sha256:46e99bb272612b0de3b7e7caf6da8dd5f4ca5212c0b273feb9304e236c484e5f
    # via
    #   mkdocs-material
+    #   mkdocs-mermaid2-plugin
    #   mkdocstrings
 python-dateutil==2.9.0.post0 \
    --hash=sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3 \
@ -472,15 +490,23 @@ pyyaml-env-tag==1.1 \
 requests==2.32.3 \
    --hash=sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760 \
    --hash=sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6
-    # via mkdocs-material
+    # via
+    #   mkdocs-material
+    #   mkdocs-mermaid2-plugin
 rich==14.0.0 \
    --hash=sha256:1c9491e1951aac09caffd42f448ee3d04e58923ffe14993f6e83068dc395d7e0 \
    --hash=sha256:82f1bc23a6a21ebca4ae0c45af9bdbc492ed20231dcb63f297d6d1021a9d5725
    # via neoteroi-mkdocs
+setuptools==80.9.0 \
+    --hash=sha256:062d34222ad13e0cc312a4c02d73f059e86a4acbfbdea8f8f76b28c99f306922 \
+    --hash=sha256:f36b47402ecde768dbfafc46e8e4207b4360c654f1f3bb84475f0a28628fb19c
+    # via mkdocs-mermaid2-plugin
 six==1.17.0 \
    --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \
    --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81
-    # via python-dateutil
+    # via
+    #   jsbeautifier
+    #   python-dateutil
 smmap==5.0.2 \
    --hash=sha256:26ea65a03958fa0c8a1c7e8c7a58fdc77221b8910f6be2131affade476898ad5 \
    --hash=sha256:b30115f0def7d7531d22a0fb6502488d879e75b260a9db4d0819cfb25403af5e
@ -489,6 +515,10 @@ sniffio==1.3.1 \
    --hash=sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2 \
    --hash=sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc
    # via anyio
+soupsieve==2.7 \
+    --hash=sha256:6e60cc5c1ffaf1cebcc12e8188320b72071e922c2e897f737cadce79ad5d30c4 \
+    --hash=sha256:ad282f9b6926286d2ead4750552c8a6142bc4c783fd66b0293547c8fe6ae126a
+    # via beautifulsoup4
 super-collections==0.5.3 \
    --hash=sha256:907d35b25dc4070910e8254bf2f5c928348af1cf8a1f1e8259e06c666e902cff \
    --hash=sha256:94c1ec96c0a0d5e8e7d389ed8cde6882ac246940507c5e6b86e91945c2968d46
@ -502,6 +532,7 @@ typing-extensions==4.14.0 \
    --hash=sha256:a1514509136dd0b477638fc68d6a91497af5076466ad0fa6c338e44e359944af
    # via
    #   anyio
+    #   beautifulsoup4
    #   exceptiongroup
    #   mkdocstrings-python
    #   rich