Backup update (#10586)

* Update django-dbbackup version * Specify STORAGES option for dbbackup * Add more backup configuration * Support custom date formats * Add connector options * Extend functionality of invoke backup * Add extra options for restore task * Add invoke task for finding additional backups * Small tweaks * Add docs around backup / restore * Fix typo * Add example for GCS storage * More docs
2025-10-18 23:17:41 +00:00 · 2025-10-18 07:28:18 +11:00
parent de270a5fe7
commit d34f44221e
11 changed files with 336 additions and 48 deletions
--- a/docs/docs/develop/contributing.md
+++ b/docs/docs/develop/contributing.md
@@ -186,6 +186,10 @@ To see all the available options:
 invoke dev.test --help
 ```

+```
+{{ invoke_commands('dev.test --help') }}
+```
+
 #### Database Permission Issues

 For local testing django creates a test database and removes it after testing. If you encounter permission issues while running unit test, ensure that your database user has permission to create new databases.
--- a/docs/docs/start/backup.md
+++ b/docs/docs/start/backup.md
@@ -4,7 +4,7 @@ title: Data Backup

 ## Data Backup

-Backup functionality is provided natively using the [django-dbbackup library](https://django-dbbackup.readthedocs.io/en/master/). This library provides multiple options for creating backups of your InvenTree database and media files. In addition to local storage backup, multiple external storage solutions are supported (such as Amazon S3 or Dropbox).
+Backup functionality is provided natively using the [django-dbbackup library](https://archmonger.github.io/django-dbbackup/5.0.0/). This library provides multiple options for creating backups of your InvenTree database and media files. In addition to local storage backup, multiple external storage solutions are supported (such as Amazon S3 or Dropbox).

 Note that a *backup* operation is not the same as [migrating data](./migrate.md). While data *migration* exports data into a database-agnostic JSON file, *backup* exports a native database file and media file archive.

@@ -13,19 +13,38 @@ Note that a *backup* operation is not the same as [migrating data](./migrate.md)

 ## Configuration

+The django-dbbackup library provides [multiple configuration options](https://archmonger.github.io/django-dbbackup/5.0.0/configuration/), a subset of which are exposed via InvenTree.
+
 The following configuration options are available for backup:

 | Environment Variable | Configuration File | Description | Default |
 | --- | --- | --- | --- |
-| INVENTREE_BACKUP_STORAGE | backup_storage | Backup storage backend | django.core.files.storage.FileSystemStorage |
-| INVENTREE_BACKUP_DIR | backup_dir | Backup storage directory | *No default* |
-| INVENTREE_BACKUP_OPTIONS | backup_options | Specific backup options (dict) | *No default* |
+| INVENTREE_BACKUP_STORAGE | backup_storage | Backup storage backend. Refer to the [storage backend documentation](#storage-backend). | django.core.files.storage.FileSystemStorage |
+| INVENTREE_BACKUP_DIR | backup_dir | Backup storage directory. | *No default* |
+| INVENTREE_BACKUP_OPTIONS | backup_options | Specific options for the selected storage backend (dict) | *No default* |
+| INVENTREE_BACKUP_CONNECTOR_OPTIONS | backup_connector_options | Specific options for the database connector (dict). Refer to the [database connector options](#database-connector). | *No default* |
+| INVENTREE_BACKUP_SEND_EMAIL | backup_send_email | If True, an email is sent to the site admin when an error occurs during a backup or restore procedure. | False |
+| INVENTREE_BACKUP_EMAIL_PREFIX | backup_email_prefix | Prefix for the subject line of backup-related emails. | `[InvenTree Backup]` |
+| INVENTREE_BACKUP_GPG_RECIPIENT | backup_gpg_recipient | Specify GPG recipient if using encryption for backups. | *No default* |
+| INVENTREE_BACKUP_DATE_FORMAT | backup_date_format | Date format string used to format timestamps in backup filenames. | `%Y-%m-%d-%H%M%S` |
+| INVENTREE_BACKUP_DATABASE_FILENAME_TEMPLATE | backup_database_filename_template | Template string used to generate database backup filenames. | `InvenTree-db-{datetime}.{extension}` |
+| INVENTREE_BACKUP_MEDIA_FILENAME_TEMPLATE | backup_media_filename_template | Template string used to generate media backup filenames. | `InvenTree-media-{datetime}.{extension}` |

-### Storage Providers
+### Storage Backend

-If you want to use an external storage provider, extra configuration is required. As a starting point, refer to the [django-dbbackup documentation](https://django-dbbackup.readthedocs.io/en/master/storage.html).
+There are multiple backends available for storing and retrieving backup files. The default option is to use the local filesystem. Integration of other storage backends is provided by the django-storages library (which needs to be installed separately).

-Specific storage configuration options are specified using the `backup_options` dict (in the [configuration file](./config.md#backup-file-storage)).
+If you want to use an external storage provider, extra configuration is required. As a starting point, refer to the [django-dbbackup documentation](https://archmonger.github.io/django-dbbackup/5.0.0/storage/).
+
+Each storage backend may have its own specific configuration options and package requirements. Specific storage configuration options are specified using the `backup_options` dict (in the [configuration file](./config.md#backup-file-storage)), and passed through to the storage backend.
+
+### Database Connector
+
+Different database connection options are available, depending on the database backend in use.
+
+These options can be passed through via the `INVENTREE_BACKUP_CONNECTOR_OPTIONS` environment variable, or via the `backup_connector_options` value in the configuration file.
+
+Refer to the [database connector documentation](https://archmonger.github.io/django-dbbackup/5.0.0/databases/) for more information on the available options.

 ## Perform Backup

@@ -43,6 +62,10 @@ This will perform backup operation with the default parameters. To see all avail
 invoke backup --help
 ```

+```
+{{ invoke_commands('backup --help') }}
+```
+
 ### Backup During Update

 When performing an update of your InvenTree installation - via either [docker](./docker.md) or [bare metal](./install.md) - a backup operation is automatically performed.
@@ -71,9 +94,72 @@ To see all available options for restore, run:
 invoke restore --help
 ```

+```
+{{ invoke_commands('restore --help') }}
+```
+
+## View Backups
+
+To view a list of available backups, run the following command from the shell:
+
+```
+invoke listbackups
+```
+
+## Backup Filename Formatting
+
+There are multiple configuration options available to control the formatting of backup filenames. These options are described in the [configuration section](#configuration) above.
+
+For more information about the available formatting options, refer to the [django-dbbackup documentation](https://archmonger.github.io/django-dbbackup/latest/configuration/#dbbackup_filename_template).
+
 ## Advanced Usage

-Not all functionality of the db-backup library is exposed by default. For advanced usage (not covered by the documentation above), refer to the [django-dbbackup commands documentation](https://django-dbbackup.readthedocs.io/en/master/commands.html).
+Not all functionality of the db-backup library is exposed by default. For advanced usage (not covered by the documentation above), refer to the [django-dbbackup commands documentation](https://archmonger.github.io/django-dbbackup/5.0.0/commands/).

 !!! warning "Advanced Users Only"
    Any advanced usage assumes some underlying knowledge of django, and is not documented here.
+
+## Example: Google Cloud Storage
+
+By default, InvenTree backups are stored on the local filesystem. However, it is possible to configure remote storage backends, such as Google Cloud Storage (GCS). Below is a *brief* example of how you might configure GCS for backup storage. However, please note that this is for informational purposes only - the InvenTree project does not provide direct support for third-party storage backends.
+
+### External Documentation
+
+As a starting point, refer to the external documentation for django-dbbackup: https://archmonger.github.io/django-dbbackup/latest/storage/#google-cloud-storage
+
+### Install Dependencies
+
+You will need to install an additional package to enable GCS support:
+
+```bash
+pip install django-storages[google]
+```
+
+!!! tip "Python Environment"
+    Ensure you install the package into the same Python environment that InvenTree is installed in (e.g. virtual environment).
+
+### Select Storage Backend
+
+You will need to change the storage backend, which is set via the `INVENTREE_BACKUP_STORAGE` environment variable, or via `backup_storage` in the configuration file:
+
+```yaml
+backup_stoage: storages.backends.gcloud.GoogleCloudStorage
+```
+
+### Configure Backend Options
+
+You will need to also specify the required options for the GCS backend. This is done via the `INVENTREE_BACKUP_OPTIONS` environment variable, or via `backup_options` in the configuration file. An example configuration might look like:
+
+```yaml
+backup_options:
+  bucket_name: 'your_bucket_name'
+  project_id: 'your_project_id'
+```
+
+### Advanced Configuration
+
+There are other options available for the GCS storage backend - refer to the [GCS documentation](https://django-storages.readthedocs.io/en/latest/backends/gcloud.html) for more information.
+
+### Other Backends
+
+Other storage backends are also supported via the django-storages library, such as Amazon S3, Dropbox, and more. This is outside the scope of this documentation - refer to the external documentation links on this page for more information.
--- a/docs/docs/start/invoke.md
+++ b/docs/docs/start/invoke.md
@@ -95,6 +95,10 @@ For example, to find more information about the `update` task, run:
 invoke update --help
 ```

+```
+{{ invoke_commands('update --help') }}
+```
+
 ### Internal Tasks

 Tasks with the `int.` prefix are internal tasks, and are not intended for general use. These are called by other tasks, and should generally not be called directly.
--- a/docs/docs/start/migrate.md
+++ b/docs/docs/start/migrate.md
@@ -31,6 +31,10 @@ This will create JSON file at the specified location which contains all database
 !!! info "Specifying filename"
    The filename of the exported file can be specified using the `-f` option. To see all available options, run `invoke export-records --help`

+```
+{{ invoke_commands('export-records --help') }}
+```
+
 ### Initialize New Database

 Configure the new database using the normal processes (see [Configuration](./config.md))
--- a/docs/main.py
+++ b/docs/main.py
@@ -233,12 +233,12 @@ def define_env(env):
        return url

    @env.macro
-    def invoke_commands():
+    def invoke_commands(command: str = '--list'):
        """Provides an output of the available commands."""
        tasks = here.parent.joinpath('tasks.py')
        output = gen_base.joinpath('invoke-commands.txt')

-        command = f'invoke -f {tasks} --list > {output}'
+        command = f'invoke -f {tasks} {command} > {output}'

        assert subprocess.call(command, shell=True) == 0

--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -125,7 +125,7 @@ nav:
      - Project Security: security.md
      - Resources: project/resources.md
      - Privacy: privacy.md
-  - Install:
+  - Setup:
    - Introduction: start/index.md
    - Processes: start/processes.md
    - Configuration: start/config.md
--- a/src/backend/InvenTree/InvenTree/backup.py
+++ b/src/backend/InvenTree/InvenTree/backup.py
@@ -0,0 +1,133 @@
+"""Configuration options for InvenTree backup / restore functionality.
+
+We use the django-dbbackup library to handle backup and restore operations.
+
+Ref: https://archmonger.github.io/django-dbbackup/latest/configuration/
+"""
+
+import InvenTree.config
+
+
+def get_backup_connector_options() -> dict:
+    """Options which are specific to the selected backup connector.
+
+    These options apply to the database connector, not to the backup storage.
+
+    Ref: https://archmonger.github.io/django-dbbackup/latest/databases/
+    """
+    default_options = {'EXCLUDE': ['django_session']}
+
+    # Allow user to specify custom options here if necessary
+    connector_options = InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_CONNECTOR_OPTIONS',
+        'backup_connector_options',
+        default_value=default_options,
+        typecast=dict,
+    )
+
+    return connector_options
+
+
+def get_backup_storage_backend() -> str:
+    """Return the backup storage backend string."""
+    backend = InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_STORAGE',
+        'backup_storage',
+        'django.core.files.storage.FileSystemStorage',
+    )
+
+    # Validate that the selected backend is valid
+    # It must be able to be imported, and a class must be found
+    # It also must be a subclass of django.core.files.storage.Storage
+    try:
+        from django.core.files.storage import Storage
+        from django.utils.module_loading import import_string
+
+        backend_class = import_string(backend)
+
+        if not issubclass(backend_class, Storage):
+            raise TypeError(
+                f"Backup storage backend '{backend}' is not a valid Storage class"
+            )
+    except Exception as e:
+        raise ImportError(f"Could not load backup storage backend '{backend}': {e}")
+
+    return backend
+
+
+def get_backup_storage_options() -> dict:
+    """Return the backup storage options dictionary."""
+    # Default backend options which are used for FileSystemStorage
+    default_options = {'location': InvenTree.config.get_backup_dir()}
+
+    options = InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_OPTIONS',
+        'backup_options',
+        default_value=default_options,
+        typecast=dict,
+    )
+
+    if not isinstance(options, dict):
+        raise ValueError('Backup storage options must be a dictionary')
+
+    return options
+
+
+def backup_email_on_error() -> bool:
+    """Return whether to send emails to admins on backup failure."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_SEND_EMAIL',
+        'backup_send_email',
+        default_value=False,
+        typecast=bool,
+    )
+
+
+def backup_email_prefix() -> str:
+    """Return the email subject prefix for backup emails."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_EMAIL_PREFIX',
+        'backup_email_prefix',
+        default_value='[InvenTree Backup]',
+        typecast=str,
+    )
+
+
+def backup_gpg_recipient() -> str:
+    """Return the GPG recipient for encrypted backups."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_GPG_RECIPIENT',
+        'backup_gpg_recipient',
+        default_value='',
+        typecast=str,
+    )
+
+
+def backup_date_format() -> str:
+    """Return the date format string for database backups."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_DATE_FORMAT',
+        'backup_date_format',
+        default_value='%Y-%m-%d-%H%M%S',
+        typecast=str,
+    )
+
+
+def backup_filename_template() -> str:
+    """Return the filename template for database backups."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_DATABASE_FILENAME_TEMPLATE',
+        'backup_database_filename_template',
+        default_value='InvenTree-db-{datetime}.{extension}',
+        typecast=str,
+    )
+
+
+def backup_media_filename_template() -> str:
+    """Return the filename template for media backups."""
+    return InvenTree.config.get_setting(
+        'INVENTREE_BACKUP_MEDIA_FILENAME_TEMPLATE',
+        'backup_media_filename_template',
+        default_value='InvenTree-media-{datetime}.{extension}',
+        typecast=str,
+    )
--- a/src/backend/InvenTree/InvenTree/settings.py
+++ b/src/backend/InvenTree/InvenTree/settings.py
@@ -24,6 +24,7 @@ from django.http import Http404, HttpResponseGone
 import structlog
 from corsheaders.defaults import default_headers as default_cors_headers

+import InvenTree.backup
 from InvenTree.cache import get_cache_config, is_global_cache_enabled
 from InvenTree.config import (
    get_boolean_setting,
@@ -247,22 +248,32 @@ if DEBUG and 'collectstatic' not in sys.argv:
        STATICFILES_DIRS.append(BASE_DIR.joinpath('plugin', 'samples', 'static'))

 # Database backup options
-# Ref: https://django-dbbackup.readthedocs.io/en/master/configuration.html
-DBBACKUP_SEND_EMAIL = False
-DBBACKUP_STORAGE = get_setting(
-    'INVENTREE_BACKUP_STORAGE',
-    'backup_storage',
-    'django.core.files.storage.FileSystemStorage',
-)
+# Ref: https://archmonger.github.io/django-dbbackup/latest/configuration/

-# Default backup configuration
-DBBACKUP_STORAGE_OPTIONS = get_setting(
-    'INVENTREE_BACKUP_OPTIONS',
-    'backup_options',
-    default_value={'location': config.get_backup_dir()},
-    typecast=dict,
-)
+# For core backup functionality, refer to the STORAGES["dbbackup"] entry (below)

+DBBACKUP_DATE_FORMAT = InvenTree.backup.backup_date_format()
+DBBACKUP_FILENAME_TEMPLATE = InvenTree.backup.backup_filename_template()
+DBBACKUP_MEDIA_FILENAME_TEMPLATE = InvenTree.backup.backup_media_filename_template()
+
+DBBACKUP_GPG_RECIPIENT = InvenTree.backup.backup_gpg_recipient()
+
+DBBACKUP_SEND_EMAIL = InvenTree.backup.backup_email_on_error()
+DBBACKUP_EMAIL_SUBJECT_PREFIX = InvenTree.backup.backup_email_prefix()
+
+DBBACKUP_CONNECTORS = {'default': InvenTree.backup.get_backup_connector_options()}
+
+# Data storage options
+STORAGES = {
+    'default': {'BACKEND': 'django.core.files.storage.FileSystemStorage'},
+    'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.StaticFilesStorage'},
+    'dbbackup': {
+        'BACKEND': InvenTree.backup.get_backup_storage_backend(),
+        'OPTIONS': InvenTree.backup.get_backup_storage_options(),
+    },
+}
+
+# Enable django admin interface?
 INVENTREE_ADMIN_ENABLED = get_boolean_setting(
    'INVENTREE_ADMIN_ENABLED', config_key='admin_enabled', default_value=True
 )
--- a/src/backend/requirements.in
+++ b/src/backend/requirements.in
@@ -6,7 +6,7 @@ django-anymail[amazon_ses,postal]       # Email backend for various providers
 django-allauth[mfa,socialaccount,saml,openid]   # SSO for external providers via OpenID
 django-cleanup                          # Automated deletion of old / unused uploaded files
 django-cors-headers                     # CORS headers extension for DRF
-django-dbbackup                         # Backup / restore of database and media files
+django-dbbackup>=5.0.0                  # Backup / restore of database and media files
 django-error-report-2                   # Error report viewer for the admin interface
 django-filter                           # Extended filtering options
 django-flags                            # Feature flags
--- a/src/backend/requirements.txt
+++ b/src/backend/requirements.txt
@@ -438,9 +438,9 @@ django-cors-headers==4.9.0 \
    --hash=sha256:15c7f20727f90044dcee2216a9fd7303741a864865f0c3657e28b7056f61b449 \
    --hash=sha256:fe5d7cb59fdc2c8c646ce84b727ac2bca8912a247e6e68e1fb507372178e59e8
    # via -r src/backend/requirements.in
-django-dbbackup==4.3.0 \
-    --hash=sha256:3549c8ccfdb167f20ca2c26eb0dfa55c79aa3f31ea4e4dbfa0816bc18ceec6dc \
-    --hash=sha256:b4003b353d49d914ffbc033c793198426572d0a2b137ec795ddb2fb82225b960
+django-dbbackup==5.0.0 \
+    --hash=sha256:a0301b14a4bb3c7243a2fde76d09f8f572f16cd7639f75f4cd42d898fc1b82a2 \
+    --hash=sha256:aa9cc88e1413adfec0e547dd91e0afed6dbb91a02459697663a9b988dbc71f18
    # via -r src/backend/requirements.in
 django-error-report-2==0.4.2 \
    --hash=sha256:1dd99c497af09b7ea99f5fbaf910501838150a9d5390796ea00e187bc62f6c1b \
@@ -1295,10 +1295,6 @@ python3-saml==1.16.0 \
    --hash=sha256:97c9669aecabc283c6e5fb4eb264f446b6e006f5267d01c9734f9d8bffdac133 \
    --hash=sha256:c49097863c278ff669a337a96c46dc1f25d16307b4bb2679d2d1733cc4f5176a
    # via django-allauth
-pytz==2025.2 \
-    --hash=sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3 \
-    --hash=sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00
-    # via django-dbbackup
 pyyaml==6.0.3 \
    --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \
    --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \
--- a/tasks.py
+++ b/tasks.py
@@ -657,18 +657,37 @@ def translate(c, ignore_static=False, no_frontend=False):
@task(
    help={
        'clean': 'Clean up old backup files',
+        'compress': 'Compress the backup files',
+        'encrypt': 'Encrypt the backup files (requires GPG recipient to be set)',
        'path': 'Specify path for generated backup files (leave blank for default path)',
+        'quiet': 'Suppress informational output (only show errors)',
+        'skip_db': 'Skip database backup step (only backup media files)',
+        'skip_media': 'Skip media backup step (only backup database files)',
    }
 )
@state_logger('TASK04')
-def backup(c, clean=False, path=None):
+def backup(
+    c,
+    clean: bool = False,
+    compress: bool = True,
+    encrypt: bool = False,
+    path=None,
+    quiet: bool = False,
+    skip_db: bool = False,
+    skip_media: bool = False,
+):
    """Backup the database and media files."""
-    info('Backing up InvenTree database...')
+    cmd = '--noinput -v 2'

-    cmd = '--noinput --compress -v 2'
+    if compress:
+        cmd += ' --compress'

+    if encrypt:
+        cmd += ' --encrypt'
+
+    # A path to the backup dir can be specified here
+    # If not specified, the default backup dir is used
    if path:
-        # Resolve the provided path
        path = Path(path)
        if not os.path.isabs(path):
            path = local_dir().joinpath(path).resolve()
@@ -678,20 +697,34 @@ def backup(c, clean=False, path=None):
    if clean:
        cmd += ' --clean'

-    manage(c, f'dbbackup {cmd}')
-    info('Backing up InvenTree media files...')
-    manage(c, f'mediabackup {cmd}')
+    if quiet:
+        cmd += ' --quiet'

-    success('Backup completed successfully')
+    if skip_db:
+        info('Skipping database backup...')
+    else:
+        info('Backing up InvenTree database...')
+        manage(c, f'dbbackup {cmd}')
+
+    if skip_media:
+        info('Skipping media backup...')
+    else:
+        info('Backing up InvenTree media files...')
+        manage(c, f'mediabackup {cmd}')
+
+    if not skip_db or not skip_media:
+        success('Backup completed successfully')


@task(
    help={
        'path': 'Specify path to locate backup files (leave blank for default path)',
        'db_file': 'Specify filename of compressed database archive (leave blank to use most recent backup)',
+        'decrypt': 'Decrypt the backup files (requires GPG recipient to be set)',
        'media_file': 'Specify filename of compressed media archive (leave blank to use most recent backup)',
-        'ignore_media': 'Do not import media archive (database restore only)',
-        'ignore_database': 'Do not import database archive (media restore only)',
+        'skip_db': 'Do not import database archive (media restore only)',
+        'skip_media': 'Do not import media archive (database restore only)',
+        'uncompress': 'Uncompress the backup files before restoring (default behavior)',
    }
 )
 def restore(
@@ -699,11 +732,19 @@ def restore(
    path=None,
    db_file=None,
    media_file=None,
-    ignore_media=False,
-    ignore_database=False,
+    decrypt: bool = False,
+    skip_db: bool = False,
+    skip_media: bool = False,
+    uncompress: bool = True,
 ):
    """Restore the database and media files."""
-    base_cmd = '--noinput --uncompress -v 2'
+    base_cmd = '--noinput -v 2'
+
+    if uncompress:
+        base_cmd += ' --uncompress'
+
+    if decrypt:
+        base_cmd += ' --decrypt'

    if path:
        # Resolve the provided path
@@ -713,7 +754,7 @@ def restore(

        base_cmd += f' -I {path}'

-    if ignore_database:
+    if skip_db:
        info('Skipping database archive...')
    else:
        info('Restoring InvenTree database')
@@ -724,7 +765,7 @@ def restore(

        manage(c, cmd)

-    if ignore_media:
+    if skip_media:
        info('Skipping media restore...')
    else:
        info('Restoring InvenTree media files')
@@ -736,6 +777,14 @@ def restore(
        manage(c, cmd)


+@task()
+@state_logger()
+def listbackups(c):
+    """List available backup files."""
+    info('Finding available backup files...')
+    manage(c, 'listbackups')
+
+
@task(post=[rebuild_models, rebuild_thumbnails])
@state_logger('TASK05')
 def migrate(c):
@@ -1934,6 +1983,7 @@ ns = Collection(
    frontend_download,
    import_records,
    install,
+    listbackups,
    migrate,
    plugins,
    remove_mfa,