Code structure refactor (#5582)

* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fit in changes from testing

* gather packager improvements (#149)

* Matmair/issue5578 (#143)

* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fix docker path

* use project dir

* move project dir command

* fixed docker paths

* another fix?

* seperate tasks out

* remove tasks

* some debugging

* ci: add .deepsource.toml

* Update .deepsource.toml

* also ignore migrations

* more debugging

* fix path issues

* remove debug script

* fix style

* change locale path

* Fixed paths for requirements

* Added dummy requirements to fool packager

* fixed exec path

* remove deepsource

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>

* Added docs for file structure

* Fixed style errors

* updated deepsource paths

* fix deepsource paths

* fixed reqs

* merge fixes

* move newly added dirs too

* fix reqs files

* another dep fix

* merge upstream/master

* revert removal of tags

* merge upstream

* enabled detection of old config files

* adapt coverage src

* also detect and support old location for plugins.txt

* style fix

* fix ~/init.sh location

* fix requirements path

* fix config to current master

* move new folders

* fix import order

* fix paths for qc_check

* fix docs build

* fix fix path

* set docker project dir

* just use a cd

* set image path?

* set file correct

* fix copy path

* fix tasks dir

* fix init path

* fix copy path

* set prject dir

* fix paths

* remove old prod files

* fix dev env path

* set docker file

* Fix devcontainer docker compose file

* fix login attempt values

* fix init.sh path

* Fix pathing for Docker

* Docker build fix

- Set INVENTREE_BACKEND_DIR separately

* Update init.sh

* Fix path

* Update requirements.txt

* merge

* fix rq merge

* fix docker compose usage

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>
Co-authored-by: Oliver <oliver.henry.walters@gmail.com>

.github/CODEOWNERS

@@ -1,2 +1,13 @@
 # General owner is the maintainers team
 *       @SchrodingersGat
+
+# plugins are co-owned
+/src/backend/InvenTree/plugin/      @SchrodingersGat @matmair
+/src/backend/InvenTree/plugins/     @SchrodingersGat @matmair
+
+# Installer functions
+.pkgr.yml               @matmair
+Procfile                @matmair
+runtime.txt             @matmair
+/contrib/installer      @matmair
+/contrib/packager.io    @matmair

.github/actions/migration/action.yaml

@@ -11,7 +11,7 @@ runs:
           invoke migrate
           invoke import-fixtures
           invoke export-records -f data.json
-          python3 ./InvenTree/manage.py flush --noinput
+          python3 ./src/backend/InvenTree/manage.py flush --noinput
           invoke migrate
           invoke import-records -f data.json
           invoke import-records -f data.json

.github/actions/setup/action.yaml

@@ -65,10 +65,11 @@ runs:
         with:
           node-version: ${{ env.node_version }}
           cache: 'npm'
+          cache-dependency-path: src/backend/package-lock.json
       - name: Install npm packages
         if: ${{ inputs.npm == 'true' }}
         shell: bash
-        run: npm install
+        run: cd src/backend && npm install
 
       # OS installs
       - name: Install OS Dependencies
@@ -77,12 +78,13 @@ runs:
         run: |
           sudo apt-get update
           sudo apt-get install ${{ inputs.apt-dependency }}
+          sudo apt-get install ${{ inputs.apt-dependency }}
 
       # Invoke commands
       - name: Install dev requirements
         if: ${{ inputs.dev-install == 'true' ||inputs.install == 'true' }}
         shell: bash
-        run: uv pip install -r requirements-dev.txt
+        run: uv pip install -r src/backend/requirements-dev.txt
       - name: Run invoke install
         if: ${{ inputs.install == 'true' }}
         shell: bash

.github/scripts/check_js_templates.py

@@ -0,0 +1,103 @@
+"""Test that the "translated" javascript files to not contain template tags which need to be determined at "run time".
+
+This is because the "translated" javascript files are compiled into the "static" directory.
+They should only contain template tags that render static information.
+"""
+
+import os
+import pathlib
+import re
+import sys
+
+here = os.path.abspath(os.path.dirname(__file__))
+template_dir = os.path.abspath(os.path.join(here, '..', 'InvenTree', 'templates'))
+
+# We only care about the 'translated' files
+js_i18n_dir = os.path.join(template_dir, 'js', 'translated')
+js_dynamic_dir = os.path.join(template_dir, 'js', 'dynamic')
+
+errors = 0
+
+print('=================================')
+print('Checking static javascript files:')
+print('=================================')
+
+
+def check_invalid_tag(data):
+    """Check for invalid tags."""
+    pattern = r'{%(\w+)'
+
+    err_count = 0
+
+    for idx, line in enumerate(data):
+        results = re.findall(pattern, line)
+
+        for result in results:
+            err_count += 1
+
+            print(f' - Error on line {idx + 1}: %{{{result[0]}')
+
+    return err_count
+
+
+def check_prohibited_tags(data):
+    """Check for prohibited tags."""
+    allowed_tags = [
+        'if',
+        'elif',
+        'else',
+        'endif',
+        'for',
+        'endfor',
+        'trans',
+        'load',
+        'include',
+        'url',
+    ]
+
+    pattern = r'{% (\w+)\s'
+
+    err_count = 0
+
+    for idx, line in enumerate(data):
+        for tag in re.findall(pattern, line):
+            if tag not in allowed_tags:
+                print(f" > Line {idx + 1} contains prohibited template tag '{tag}'")
+                err_count += 1
+
+    return err_count
+
+
+for filename in pathlib.Path(js_i18n_dir).rglob('*.js'):
+    print(f"Checking file 'translated/{os.path.basename(filename)}':")
+
+    with open(filename, 'r') as js_file:
+        data = js_file.readlines()
+
+    errors += check_invalid_tag(data)
+    errors += check_prohibited_tags(data)
+
+for filename in pathlib.Path(js_dynamic_dir).rglob('*.js'):
+    print(f"Checking file 'dynamic/{os.path.basename(filename)}':")
+
+    # Check that the 'dynamic' files do not contains any translated strings
+    with open(filename, 'r') as js_file:
+        data = js_file.readlines()
+
+    invalid_tags = ['blocktrans', 'blocktranslate', 'trans', 'translate']
+
+    err_count = 0
+
+    for idx, line in enumerate(data):
+        for tag in invalid_tags:
+            tag = '{% ' + tag
+            if tag in line:
+                err_count += 1
+
+                print(f" > Error on line {idx + 1}: Prohibited tag '{tag}' found")
+
+
+if errors > 0:
+    print(f'Found {errors} incorrect template tags')
+
+sys.exit(errors)

.github/scripts/check_migration_files.py

@@ -0,0 +1,28 @@
+"""Check that there are no database migration files which have not been committed."""
+
+import subprocess
+import sys
+
+print('Checking for unstaged migration files...')
+
+cmd = ['git', 'ls-files', '--exclude-standard', '--others']
+
+proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+out, err = proc.communicate()
+
+migrations = []
+
+for line in str(out.decode()).split('\n'):
+    if '/migrations/' in line:
+        migrations.append(line)
+
+if len(migrations) == 0:
+    sys.exit(0)
+
+print('There are {n} unstaged migration files:'.format(n=len(migrations)))
+
+for m in migrations:
+    print(' - {m}'.format(m=m))
+
+sys.exit(len(migrations))

.github/scripts/version_check.py

@@ -0,0 +1,196 @@
+"""Ensure that the release tag matches the InvenTree version number.
+
+Behaviour:
+master / main branch:
+    - version number must end with 'dev'
+
+tagged branch:
+    - version number must match tag being built
+    - version number cannot already exist as a release tag
+
+"""
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+
+import requests
+
+
+def get_existing_release_tags():
+    """Request information on existing releases via the GitHub API."""
+    # Check for github token
+    token = os.getenv('GITHUB_TOKEN', None)
+    headers = None
+
+    if token:
+        headers = {'Authorization': f'Bearer {token}'}
+
+    response = requests.get(
+        'https://api.github.com/repos/inventree/inventree/releases', headers=headers
+    )
+
+    if response.status_code != 200:
+        raise ValueError(
+            f'Unexpected status code from GitHub API: {response.status_code}'
+        )
+
+    data = json.loads(response.text)
+
+    # Return a list of all tags
+    tags = []
+
+    for release in data:
+        tag = release['tag_name'].strip()
+        match = re.match(r'^.*(\d+)\.(\d+)\.(\d+).*$', tag)
+
+        if len(match.groups()) != 3:
+            print(f"Version '{tag}' did not match expected pattern")
+            continue
+
+        tags.append([int(x) for x in match.groups()])
+
+    return tags
+
+
+def check_version_number(version_string, allow_duplicate=False):
+    """Check the provided version number.
+
+    Returns True if the provided version is the 'newest' InvenTree release
+    """
+    print(f"Checking version '{version_string}'")
+
+    # Check that the version string matches the required format
+    match = re.match(r'^(\d+)\.(\d+)\.(\d+)(?: dev)?$', version_string)
+
+    if not match or len(match.groups()) != 3:
+        raise ValueError(
+            f"Version string '{version_string}' did not match required pattern"
+        )
+
+    version_tuple = [int(x) for x in match.groups()]
+
+    # Look through the existing releases
+    existing = get_existing_release_tags()
+
+    # Assume that this is the highest release, unless told otherwise
+    highest_release = True
+
+    for release in existing:
+        if release == version_tuple and not allow_duplicate:
+            raise ValueError(f"Duplicate release '{version_string}' exists!")
+
+        if release > version_tuple:
+            highest_release = False
+            print(f'Found newer release: {str(release)}')
+
+    return highest_release
+
+
+if __name__ == '__main__':
+    if 'only_version' in sys.argv:
+        here = Path(__file__).parent.absolute()
+        version_file = here.joinpath('..', 'InvenTree', 'InvenTree', 'api_version.py')
+        text = version_file.read_text()
+        results = re.findall(r"""INVENTREE_API_VERSION = (.*)""", text)
+        print(results[0])
+        exit(0)
+    # GITHUB_REF_TYPE may be either 'branch' or 'tag'
+    GITHUB_REF_TYPE = os.environ['GITHUB_REF_TYPE']
+
+    # GITHUB_REF may be either 'refs/heads/<branch>' or 'refs/heads/<tag>'
+    GITHUB_REF = os.environ['GITHUB_REF']
+
+    GITHUB_REF_NAME = os.environ['GITHUB_REF_NAME']
+
+    GITHUB_BASE_REF = os.environ['GITHUB_BASE_REF']
+
+    # Print out version information, makes debugging actions *much* easier!
+    print(f'GITHUB_REF: {GITHUB_REF}')
+    print(f'GITHUB_REF_NAME: {GITHUB_REF_NAME}')
+    print(f'GITHUB_REF_TYPE: {GITHUB_REF_TYPE}')
+    print(f'GITHUB_BASE_REF: {GITHUB_BASE_REF}')
+
+    here = Path(__file__).parent.absolute()
+    version_file = here.joinpath(
+        '..', '..', 'src', 'backend', 'InvenTree', 'InvenTree', 'version.py'
+    )
+
+    version = None
+
+    with open(version_file, 'r') as f:
+        text = f.read()
+
+        # Extract the InvenTree software version
+        results = re.findall(r"""INVENTREE_SW_VERSION = '(.*)'""", text)
+
+        if len(results) != 1:
+            print(f'Could not find INVENTREE_SW_VERSION in {version_file}')
+            sys.exit(1)
+
+        version = results[0]
+
+    print(f"InvenTree Version: '{version}'")
+
+    # Check version number and look for existing versions
+    # If a release is found which matches the current tag, throw an error
+
+    allow_duplicate = False
+
+    # Note: on a 'tag' (release) we *must* allow duplicate versions, as this *is* the version that has just been released
+    if GITHUB_REF_TYPE == 'tag':
+        allow_duplicate = True
+
+    # Note: on a push to 'stable' branch we also allow duplicates
+    if GITHUB_BASE_REF == 'stable':
+        allow_duplicate = True
+
+    highest_release = check_version_number(version, allow_duplicate=allow_duplicate)
+
+    # Determine which docker tag we are going to use
+    docker_tags = None
+
+    if GITHUB_REF_TYPE == 'tag':
+        # GITHUB_REF should be of the form /refs/heads/<tag>
+        version_tag = GITHUB_REF.split('/')[-1]
+        print(f"Checking requirements for tagged release - '{version_tag}':")
+
+        if version_tag != version:
+            print(f"Version number '{version}' does not match tag '{version_tag}'")
+            sys.exit
+
+        if highest_release:
+            docker_tags = [version_tag, 'stable']
+        else:
+            docker_tags = [version_tag]
+
+    elif GITHUB_REF_TYPE == 'branch':
+        # Otherwise we know we are targeting the 'master' branch
+        docker_tags = ['latest']
+
+    else:
+        print('Unsupported branch / version combination:')
+        print(f'InvenTree Version: {version}')
+        print('GITHUB_REF_TYPE:', GITHUB_REF_TYPE)
+        print('GITHUB_BASE_REF:', GITHUB_BASE_REF)
+        print('GITHUB_REF:', GITHUB_REF)
+        sys.exit(1)
+
+    if docker_tags is None:
+        print('Docker tags could not be determined')
+        sys.exit(1)
+
+    print(f"Version check passed for '{version}'!")
+    print(f"Docker tags: '{docker_tags}'")
+
+    # Ref: https://getridbug.com/python/how-to-set-environment-variables-in-github-actions-using-python/
+    with open(os.getenv('GITHUB_ENV'), 'a') as env_file:
+        # Construct tag string
+        tags = ','.join([f'inventree/inventree:{tag}' for tag in docker_tags])
+
+        env_file.write(f'docker_tags={tags}\n')
+
+        if GITHUB_REF_TYPE == 'tag' and highest_release:
+            env_file.write('stable_release=true\n')

.github/workflows/check_translations.yaml

@@ -40,4 +40,4 @@ jobs:
       - name: Test Translations
         run: invoke translate
       - name: Check Migration Files
-        run: python3 ci/check_migration_files.py
+        run: python3 .github/scripts/check_migration_files.py

.github/workflows/docker.yaml

@@ -78,26 +78,26 @@ jobs:
         run: |
           pip install requests==2.31.0
           pip install pyyaml==6.0.1
-          python3 ci/version_check.py
+          python3 .github/scripts/version_check.py
           echo "git_commit_hash=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
           echo "git_commit_date=$(git show -s --format=%ci)" >> $GITHUB_ENV
       - name: Test Docker Image
         id: test-docker
         run: |
-          docker build . --target production --tag inventree-test
+          docker build . --target production --tag inventree-test -f contrib/container/Dockerfile
           docker run --rm inventree-test invoke --version
           docker run --rm inventree-test invoke --list
           docker run --rm inventree-test gunicorn --version
           docker run --rm inventree-test pg_dump --version
       - name: Build Docker Image
         # Build the development docker image (using docker-compose.yml)
-        run: docker-compose build --no-cache
+        run: docker compose --project-directory . -f contrib/container/dev-docker-compose.yml build --no-cache
       - name: Update Docker Image
         run: |
-          docker-compose run inventree-dev-server invoke update
-          docker-compose run inventree-dev-server invoke setup-dev
-          docker-compose up -d
-          docker-compose run inventree-dev-server invoke wait
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke update
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke setup-dev
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml up -d
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke wait
       - name: Check Data Directory
         # The following file structure should have been created by the docker image
         run: |
@@ -112,10 +112,10 @@ jobs:
           test -f data/secret_key.txt
       - name: Run Unit Tests
         run: |
-          echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" >> docker.dev.env
-          docker-compose run inventree-dev-server invoke test --disable-pty
-          docker-compose run inventree-dev-server invoke test --migrations --disable-pty
-          docker-compose down
+          echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" >> contrib/container/docker.dev.env
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke test --disable-pty
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke test --migrations --disable-pty
+          docker compose --project-directory . -f contrib/container/dev-docker-compose.yml down
       - name: Clean up test folder
         run: |
           rm -rf InvenTree/_testfolder

.github/workflows/qc_checks.yaml

@@ -42,9 +42,9 @@ jobs:
         with:
           filters: |
             server:
-              - 'InvenTree/**'
-              - 'requirements.txt'
-              - 'requirements-dev.txt'
+              - 'src/backend/InvenTree/**'
+              - 'src/backend/requirements.txt'
+              - 'src/backend/requirements-dev.txt'
             migrations:
               - '**/migrations/**'
               - '.github/workflows**'
@@ -68,12 +68,12 @@ jobs:
           install: true
       - name: Check Templated JS Files
         run: |
-          cd ci
+          cd .github/scripts
           python3 check_js_templates.py
       - name: Lint Javascript Files
         run: |
-          python InvenTree/manage.py prerender
-          npx eslint InvenTree/InvenTree/static_i18n/i18n/*.js
+          python src/backend/InvenTree/manage.py prerender
+          npx eslint src/backend/InvenTree/InvenTree/static_i18n/i18n/*.js
 
   pre-commit:
     name: Style [pre-commit]
@@ -93,7 +93,7 @@ jobs:
       - name: Check Version
         run: |
           pip install requests==2.31.0
-          python3 ci/version_check.py
+          python3 .github/scripts/version_check.py
 
   mkdocs:
     name: Style [Documentation]
@@ -147,12 +147,12 @@ jobs:
           dev-install: true
           update: true
       - name: Export API Documentation
-        run: invoke schema --ignore-warnings --filename InvenTree/schema.yml
+        run: invoke schema --ignore-warnings --filename src/backend/InvenTree/schema.yml
       - name: Upload schema
         uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # pin@v4.3.1
         with:
           name: schema.yml
-          path: InvenTree/schema.yml
+          path: src/backend/InvenTree/schema.yml
       - name: Download public schema
         if: needs.paths-filter.outputs.api == 'false'
         run: |
@@ -166,8 +166,8 @@ jobs:
       - name: Check for differences in API Schema
         if: needs.paths-filter.outputs.api == 'false'
         run: |
-          diff --color -u InvenTree/schema.yml api.yaml
-          diff -u InvenTree/schema.yml api.yaml && echo "no difference in API schema " || exit 2
+          diff --color -u src/backend/InvenTree/schema.yml api.yaml
+          diff -u src/backend/InvenTree/schema.yml api.yaml && echo "no difference in API schema " || exit 2
       - name: Check schema - including warnings
         run: invoke schema
         continue-on-error: true
@@ -273,7 +273,7 @@ jobs:
       - name: Test Translations
         run: invoke translate
       - name: Check Migration Files
-        run: python3 ci/check_migration_files.py
+        run: python3 .github/scripts/check_migration_files.py
       - name: Coverage Tests
         run: invoke test --coverage
       - name: Upload Coverage Report
@@ -512,9 +512,9 @@ jobs:
         run: cd src/frontend && npm run compile && npm run build
       - name: Zip frontend
         run: |
-          cd InvenTree/web/static
+          cd src/backend/InvenTree/web/static
           zip -r frontend-build.zip web/
       - uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # pin@v4.3.1
         with:
           name: frontend-build
-          path: InvenTree/web/static/web
+          path: src/backend/InvenTree/web/static/web

.github/workflows/release.yml

@@ -20,7 +20,7 @@ jobs:
       - name: Version Check
         run: |
           pip install requests==2.31.0
-          python3 ci/version_check.py
+          python3 .github/scripts/version_check.py
       - name: Push to Stable Branch
         uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df # pin@v0.8.0
         if: env.stable_release == 'true'
@@ -43,12 +43,12 @@ jobs:
         run: cd src/frontend && npm run compile && npm run build
       - name: Zip frontend
         run: |
-          cd InvenTree/web/static/web
+          cd src/backend/InvenTree/web/static/web
           zip -r ../frontend-build.zip *
       - uses: svenstaro/upload-release-action@04733e069f2d7f7f0b4aebc4fbdbce8613b03ccd # pin@2.9.0
         with:
           repo_token: ${{ secrets.GITHUB_TOKEN }}
-          file: InvenTree/web/static/frontend-build.zip
+          file: src/backend/InvenTree/web/static/frontend-build.zip
           asset_name: frontend-build.zip
           tag: ${{ github.ref }}
           overwrite: true