inventree/InvenTree
2
0
Fork 0
mirror of https://github.com/inventree/InvenTree.git synced 2025-08-05 19:41:41 +00:00
Code Activity

Code structure refactor (#5582)

Browse Source 
* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fit in changes from testing

* gather packager improvements (#149)

* Matmair/issue5578 (#143)

* moved docker files to /contrib/container

* changed code owners to make more precise

* updated CI to use new subdirs

* added manual trigger for testing

* moved ci files

* moved assets into subdir

* moved deploy template file to contrib

* moved django files to src/backend

* updated paths in scripts etc

* updated reqs path

* fixed version file path

* fixed flake8 path

* fixed path to node ressources

* fixed task paths

* added dep path for node

* removed unused yarn lockfile

* removed unused ci script

* updated internal backend paths for tasks

* updated translation stats path

* fixed source path for coverage

* fixed main commit repo path

* fix docker path

* use project dir

* move project dir command

* fixed docker paths

* another fix?

* seperate tasks out

* remove tasks

* some debugging

* ci: add .deepsource.toml

* Update .deepsource.toml

* also ignore migrations

* more debugging

* fix path issues

* remove debug script

* fix style

* change locale path

* Fixed paths for requirements

* Added dummy requirements to fool packager

* fixed exec path

* remove deepsource

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>

* Added docs for file structure

* Fixed style errors

* updated deepsource paths

* fix deepsource paths

* fixed reqs

* merge fixes

* move newly added dirs too

* fix reqs files

* another dep fix

* merge upstream/master

* revert removal of tags

* merge upstream

* enabled detection of old config files

* adapt coverage src

* also detect and support old location for plugins.txt

* style fix

* fix ~/init.sh location

* fix requirements path

* fix config to current master

* move new folders

* fix import order

* fix paths for qc_check

* fix docs build

* fix fix path

* set docker project dir

* just use a cd

* set image path?

* set file correct

* fix copy path

* fix tasks dir

* fix init path

* fix copy path

* set prject dir

* fix paths

* remove old prod files

* fix dev env path

* set docker file

* Fix devcontainer docker compose file

* fix login attempt values

* fix init.sh path

* Fix pathing for Docker

* Docker build fix

- Set INVENTREE_BACKEND_DIR separately

* Update init.sh

* Fix path

* Update requirements.txt

* merge

* fix rq merge

* fix docker compose usage

---------

Co-authored-by: deepsource-io[bot] <42547082+deepsource-io[bot]@users.noreply.github.com>
Co-authored-by: Oliver <oliver.henry.walters@gmail.com>
This commit is contained in:
Matthias Mair
2024-04-03 02:16:59 +01:00
committed by GitHub
parent dd889e8eeb
commit 0bace3f3af
3317 changed files with 532 additions and 1194 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
.github/CODEOWNERS vendored
View File

@@ -1,2 +1,13 @@
# General owner is the maintainers team
* @SchrodingersGat
# plugins are co-owned
/src/backend/InvenTree/plugin/ @SchrodingersGat @matmair
/src/backend/InvenTree/plugins/ @SchrodingersGat @matmair
# Installer functions
.pkgr.yml @matmair
Procfile @matmair
runtime.txt @matmair
/contrib/installer @matmair
/contrib/packager.io @matmair

2
.github/actions/migration/action.yaml vendored
View File

@@ -11,7 +11,7 @@ runs:
invoke migrate
invoke import-fixtures
invoke export-records -f data.json
python3 ./InvenTree/manage.py flush --noinput
python3 ./src/backend/InvenTree/manage.py flush --noinput
invoke migrate
invoke import-records -f data.json
invoke import-records -f data.json

6
.github/actions/setup/action.yaml vendored
View File

@@ -65,10 +65,11 @@ runs:
with:
node-version: ${{ env.node_version }}
cache: 'npm'
cache-dependency-path: src/backend/package-lock.json
- name: Install npm packages
if: ${{ inputs.npm == 'true' }}
shell: bash
run: npm install
run: cd src/backend && npm install
# OS installs
- name: Install OS Dependencies
@@ -77,12 +78,13 @@ runs:
run: |
sudo apt-get update
sudo apt-get install ${{ inputs.apt-dependency }}
sudo apt-get install ${{ inputs.apt-dependency }}
# Invoke commands
- name: Install dev requirements
if: ${{ inputs.dev-install == 'true' ||inputs.install == 'true' }}
shell: bash
run: uv pip install -r requirements-dev.txt
run: uv pip install -r src/backend/requirements-dev.txt
- name: Run invoke install
if: ${{ inputs.install == 'true' }}
shell: bash

103
.github/scripts/check_js_templates.py vendored Normal file
View File

@@ -0,0 +1,103 @@
"""Test that the "translated" javascript files to not contain template tags which need to be determined at "run time".
This is because the "translated" javascript files are compiled into the "static" directory.
They should only contain template tags that render static information.
"""
import os
import pathlib
import re
import sys
here = os.path.abspath(os.path.dirname(__file__))
template_dir = os.path.abspath(os.path.join(here, '..', 'InvenTree', 'templates'))
# We only care about the 'translated' files
js_i18n_dir = os.path.join(template_dir, 'js', 'translated')
js_dynamic_dir = os.path.join(template_dir, 'js', 'dynamic')
errors = 0
print('=================================')
print('Checking static javascript files:')
print('=================================')
def check_invalid_tag(data):
"""Check for invalid tags."""
pattern = r'{%(\w+)'
err_count = 0
for idx, line in enumerate(data):
results = re.findall(pattern, line)
for result in results:
err_count += 1
print(f' - Error on line {idx + 1}: %{{{result[0]}')
return err_count
def check_prohibited_tags(data):
"""Check for prohibited tags."""
allowed_tags = [
'if',
'elif',
'else',
'endif',
'for',
'endfor',
'trans',
'load',
'include',
'url',
]
pattern = r'{% (\w+)\s'
err_count = 0
for idx, line in enumerate(data):
for tag in re.findall(pattern, line):
if tag not in allowed_tags:
print(f" > Line {idx + 1} contains prohibited template tag '{tag}'")
err_count += 1
return err_count
for filename in pathlib.Path(js_i18n_dir).rglob('*.js'):
print(f"Checking file 'translated/{os.path.basename(filename)}':")
with open(filename, 'r') as js_file:
data = js_file.readlines()
errors += check_invalid_tag(data)
errors += check_prohibited_tags(data)
for filename in pathlib.Path(js_dynamic_dir).rglob('*.js'):
print(f"Checking file 'dynamic/{os.path.basename(filename)}':")
# Check that the 'dynamic' files do not contains any translated strings
with open(filename, 'r') as js_file:
data = js_file.readlines()
invalid_tags = ['blocktrans', 'blocktranslate', 'trans', 'translate']
err_count = 0
for idx, line in enumerate(data):
for tag in invalid_tags:
tag = '{% ' + tag
if tag in line:
err_count += 1
print(f" > Error on line {idx + 1}: Prohibited tag '{tag}' found")
if errors > 0:
print(f'Found {errors} incorrect template tags')
sys.exit(errors)

28
.github/scripts/check_migration_files.py vendored Normal file
View File

@@ -0,0 +1,28 @@
"""Check that there are no database migration files which have not been committed."""
import subprocess
import sys
print('Checking for unstaged migration files...')
cmd = ['git', 'ls-files', '--exclude-standard', '--others']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
migrations = []
for line in str(out.decode()).split('\n'):
if '/migrations/' in line:
migrations.append(line)
if len(migrations) == 0:
sys.exit(0)
print('There are {n} unstaged migration files:'.format(n=len(migrations)))
for m in migrations:
print(' - {m}'.format(m=m))
sys.exit(len(migrations))

196
.github/scripts/version_check.py vendored Normal file
View File

@@ -0,0 +1,196 @@
"""Ensure that the release tag matches the InvenTree version number.
Behaviour:
master / main branch:
- version number must end with 'dev'
tagged branch:
- version number must match tag being built
- version number cannot already exist as a release tag
"""
import json
import os
import re
import sys
from pathlib import Path
import requests
def get_existing_release_tags():
"""Request information on existing releases via the GitHub API."""
# Check for github token
token = os.getenv('GITHUB_TOKEN', None)
headers = None
if token:
headers = {'Authorization': f'Bearer {token}'}
response = requests.get(
'https://api.github.com/repos/inventree/inventree/releases', headers=headers
)
if response.status_code != 200:
raise ValueError(
f'Unexpected status code from GitHub API: {response.status_code}'
)
data = json.loads(response.text)
# Return a list of all tags
tags = []
for release in data:
tag = release['tag_name'].strip()
match = re.match(r'^.*(\d+)\.(\d+)\.(\d+).*$', tag)
if len(match.groups()) != 3:
print(f"Version '{tag}' did not match expected pattern")
continue
tags.append([int(x) for x in match.groups()])
return tags
def check_version_number(version_string, allow_duplicate=False):
"""Check the provided version number.
Returns True if the provided version is the 'newest' InvenTree release
"""
print(f"Checking version '{version_string}'")
# Check that the version string matches the required format
match = re.match(r'^(\d+)\.(\d+)\.(\d+)(?: dev)?$', version_string)
if not match or len(match.groups()) != 3:
raise ValueError(
f"Version string '{version_string}' did not match required pattern"
)
version_tuple = [int(x) for x in match.groups()]
# Look through the existing releases
existing = get_existing_release_tags()
# Assume that this is the highest release, unless told otherwise
highest_release = True
for release in existing:
if release == version_tuple and not allow_duplicate:
raise ValueError(f"Duplicate release '{version_string}' exists!")
if release > version_tuple:
highest_release = False
print(f'Found newer release: {str(release)}')
return highest_release
if __name__ == '__main__':
if 'only_version' in sys.argv:
here = Path(__file__).parent.absolute()
version_file = here.joinpath('..', 'InvenTree', 'InvenTree', 'api_version.py')
text = version_file.read_text()
results = re.findall(r"""INVENTREE_API_VERSION = (.*)""", text)
print(results[0])
exit(0)
# GITHUB_REF_TYPE may be either 'branch' or 'tag'
GITHUB_REF_TYPE = os.environ['GITHUB_REF_TYPE']
# GITHUB_REF may be either 'refs/heads/<branch>' or 'refs/heads/<tag>'
GITHUB_REF = os.environ['GITHUB_REF']
GITHUB_REF_NAME = os.environ['GITHUB_REF_NAME']
GITHUB_BASE_REF = os.environ['GITHUB_BASE_REF']
# Print out version information, makes debugging actions *much* easier!
print(f'GITHUB_REF: {GITHUB_REF}')
print(f'GITHUB_REF_NAME: {GITHUB_REF_NAME}')
print(f'GITHUB_REF_TYPE: {GITHUB_REF_TYPE}')
print(f'GITHUB_BASE_REF: {GITHUB_BASE_REF}')
here = Path(__file__).parent.absolute()
version_file = here.joinpath(
'..', '..', 'src', 'backend', 'InvenTree', 'InvenTree', 'version.py'
)
version = None
with open(version_file, 'r') as f:
text = f.read()
# Extract the InvenTree software version
results = re.findall(r"""INVENTREE_SW_VERSION = '(.*)'""", text)
if len(results) != 1:
print(f'Could not find INVENTREE_SW_VERSION in {version_file}')
sys.exit(1)
version = results[0]
print(f"InvenTree Version: '{version}'")
# Check version number and look for existing versions
# If a release is found which matches the current tag, throw an error
allow_duplicate = False
# Note: on a 'tag' (release) we *must* allow duplicate versions, as this *is* the version that has just been released
if GITHUB_REF_TYPE == 'tag':
allow_duplicate = True
# Note: on a push to 'stable' branch we also allow duplicates
if GITHUB_BASE_REF == 'stable':
allow_duplicate = True
highest_release = check_version_number(version, allow_duplicate=allow_duplicate)
# Determine which docker tag we are going to use
docker_tags = None
if GITHUB_REF_TYPE == 'tag':
# GITHUB_REF should be of the form /refs/heads/<tag>
version_tag = GITHUB_REF.split('/')[-1]
print(f"Checking requirements for tagged release - '{version_tag}':")
if version_tag != version:
print(f"Version number '{version}' does not match tag '{version_tag}'")
sys.exit
if highest_release:
docker_tags = [version_tag, 'stable']
else:
docker_tags = [version_tag]
elif GITHUB_REF_TYPE == 'branch':
# Otherwise we know we are targeting the 'master' branch
docker_tags = ['latest']
else:
print('Unsupported branch / version combination:')
print(f'InvenTree Version: {version}')
print('GITHUB_REF_TYPE:', GITHUB_REF_TYPE)
print('GITHUB_BASE_REF:', GITHUB_BASE_REF)
print('GITHUB_REF:', GITHUB_REF)
sys.exit(1)
if docker_tags is None:
print('Docker tags could not be determined')
sys.exit(1)
print(f"Version check passed for '{version}'!")
print(f"Docker tags: '{docker_tags}'")
# Ref: https://getridbug.com/python/how-to-set-environment-variables-in-github-actions-using-python/
with open(os.getenv('GITHUB_ENV'), 'a') as env_file:
# Construct tag string
tags = ','.join([f'inventree/inventree:{tag}' for tag in docker_tags])
env_file.write(f'docker_tags={tags}\n')
if GITHUB_REF_TYPE == 'tag' and highest_release:
env_file.write('stable_release=true\n')

2
.github/workflows/check_translations.yaml vendored
View File

@@ -40,4 +40,4 @@ jobs:
- name: Test Translations
run: invoke translate
- name: Check Migration Files
run: python3 ci/check_migration_files.py
run: python3 .github/scripts/check_migration_files.py

22
.github/workflows/docker.yaml vendored
View File

@@ -78,26 +78,26 @@ jobs:
run: |
pip install requests==2.31.0
pip install pyyaml==6.0.1
python3 ci/version_check.py
python3 .github/scripts/version_check.py
echo "git_commit_hash=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
echo "git_commit_date=$(git show -s --format=%ci)" >> $GITHUB_ENV
- name: Test Docker Image
id: test-docker
run: |
docker build . --target production --tag inventree-test
docker build . --target production --tag inventree-test -f contrib/container/Dockerfile
docker run --rm inventree-test invoke --version
docker run --rm inventree-test invoke --list
docker run --rm inventree-test gunicorn --version
docker run --rm inventree-test pg_dump --version
- name: Build Docker Image
# Build the development docker image (using docker-compose.yml)
run: docker-compose build --no-cache
run: docker compose --project-directory . -f contrib/container/dev-docker-compose.yml build --no-cache
- name: Update Docker Image
run: |
docker-compose run inventree-dev-server invoke update
docker-compose run inventree-dev-server invoke setup-dev
docker-compose up -d
docker-compose run inventree-dev-server invoke wait
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke update
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke setup-dev
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml up -d
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke wait
- name: Check Data Directory
# The following file structure should have been created by the docker image
run: |
@@ -112,10 +112,10 @@ jobs:
test -f data/secret_key.txt
- name: Run Unit Tests
run: |
echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" >> docker.dev.env
docker-compose run inventree-dev-server invoke test --disable-pty
docker-compose run inventree-dev-server invoke test --migrations --disable-pty
docker-compose down
echo "GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}" >> contrib/container/docker.dev.env
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke test --disable-pty
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml run inventree-dev-server invoke test --migrations --disable-pty
docker compose --project-directory . -f contrib/container/dev-docker-compose.yml down
- name: Clean up test folder
run: |
rm -rf InvenTree/_testfolder

28
.github/workflows/qc_checks.yaml vendored
View File

@@ -42,9 +42,9 @@ jobs:
with:
filters: |
server:
- 'InvenTree/**'
- 'requirements.txt'
- 'requirements-dev.txt'
- 'src/backend/InvenTree/**'
- 'src/backend/requirements.txt'
- 'src/backend/requirements-dev.txt'
migrations:
- '**/migrations/**'
- '.github/workflows**'
@@ -68,12 +68,12 @@ jobs:
install: true
- name: Check Templated JS Files
run: |
cd ci
cd .github/scripts
python3 check_js_templates.py
- name: Lint Javascript Files
run: |
python InvenTree/manage.py prerender
npx eslint InvenTree/InvenTree/static_i18n/i18n/*.js
python src/backend/InvenTree/manage.py prerender
npx eslint src/backend/InvenTree/InvenTree/static_i18n/i18n/*.js
pre-commit:
name: Style [pre-commit]
@@ -93,7 +93,7 @@ jobs:
- name: Check Version
run: |
pip install requests==2.31.0
python3 ci/version_check.py
python3 .github/scripts/version_check.py
mkdocs:
name: Style [Documentation]
@@ -147,12 +147,12 @@ jobs:
dev-install: true
update: true
- name: Export API Documentation
run: invoke schema --ignore-warnings --filename InvenTree/schema.yml
run: invoke schema --ignore-warnings --filename src/backend/InvenTree/schema.yml
- name: Upload schema
uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # pin@v4.3.1
with:
name: schema.yml
path: InvenTree/schema.yml
path: src/backend/InvenTree/schema.yml
- name: Download public schema
if: needs.paths-filter.outputs.api == 'false'
run: |
@@ -166,8 +166,8 @@ jobs:
- name: Check for differences in API Schema
if: needs.paths-filter.outputs.api == 'false'
run: |
diff --color -u InvenTree/schema.yml api.yaml
diff -u InvenTree/schema.yml api.yaml && echo "no difference in API schema " || exit 2
diff --color -u src/backend/InvenTree/schema.yml api.yaml
diff -u src/backend/InvenTree/schema.yml api.yaml && echo "no difference in API schema " || exit 2
- name: Check schema - including warnings
run: invoke schema
continue-on-error: true
@@ -273,7 +273,7 @@ jobs:
- name: Test Translations
run: invoke translate
- name: Check Migration Files
run: python3 ci/check_migration_files.py
run: python3 .github/scripts/check_migration_files.py
- name: Coverage Tests
run: invoke test --coverage
- name: Upload Coverage Report
@@ -512,9 +512,9 @@ jobs:
run: cd src/frontend && npm run compile && npm run build
- name: Zip frontend
run: |
cd InvenTree/web/static
cd src/backend/InvenTree/web/static
zip -r frontend-build.zip web/
- uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # pin@v4.3.1
with:
name: frontend-build
path: InvenTree/web/static/web
path: src/backend/InvenTree/web/static/web

6
.github/workflows/release.yml vendored
View File

@@ -20,7 +20,7 @@ jobs:
- name: Version Check
run: |
pip install requests==2.31.0
python3 ci/version_check.py
python3 .github/scripts/version_check.py
- name: Push to Stable Branch
uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df # pin@v0.8.0
if: env.stable_release == 'true'
@@ -43,12 +43,12 @@ jobs:
run: cd src/frontend && npm run compile && npm run build
- name: Zip frontend
run: |
cd InvenTree/web/static/web
cd src/backend/InvenTree/web/static/web
zip -r ../frontend-build.zip *
- uses: svenstaro/upload-release-action@04733e069f2d7f7f0b4aebc4fbdbce8613b03ccd # pin@2.9.0
with:
repo_token: ${{ secrets.GITHUB_TOKEN }}
file: InvenTree/web/static/frontend-build.zip
file: src/backend/InvenTree/web/static/frontend-build.zip
asset_name: frontend-build.zip
tag: ${{ github.ref }}
overwrite: true