Bugfix/898 improve code related to bug in dependency update

.pre-commit-config.yaml

@@ -35,6 +35,7 @@ repos:
     rev: v5.0.0
     hooks:
       - id: check-yaml
+        exclude: ^exasol/toolbox/templates/github/workflows/
         stages: [ pre-commit ]
       - id: end-of-file-fixer
         stages: [ pre-commit ]

doc/changes/unreleased.md

@@ -2,7 +2,16 @@
 
 ## Summary
 
+This minor release adds automated vulnerability updates through Nox session
+`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
+few workflow-related bug fixes and documentation updates.
+
 ## Bug
 
 * #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
-* #910: Add `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
+* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
+
+## Feature
+
+* #898: Created Nox session `vulnerabilities:update` to automatically resolve
+  vulnerable dependencies and report the result for the dependency update workflow

doc/user_guide/features/github_workflows/index.rst

@@ -39,11 +39,11 @@ Workflows
 The PTB allows for two categories of workflows:
   #. those maintained by the PTB, which can be modified using the :ref:`workflow_patcher`.
   #. custom workflows, which are project-owned.
-  
-Custom workflows can optionally be 
+
+Custom workflows can optionally be
 * seeded by the PTB, i.e. PTB generates an initial version but ignores future changes.
 * extend PTB-provided workflows, i.e. ending in `-extension.yml`
- 
+
 Besides that, you can also create individual workflow files which are ignored by the PTB.
 
 Maintained by the PTB
@@ -311,14 +311,17 @@ checks for known vulnerabilities and tries to fix them by updating dependencies.
    :start-at: on:
    :end-at:  workflow_dispatch:
 
-The workflow first audits dependencies for known vulnerabilities:
+The workflow runs a dedicated ``vulnerabilities:update`` nox session that audits the
+current dependency set and attempts to resolve any detected vulnerabilities:
 
 * If no vulnerabilities are detected, then no update is needed.
-* If vulnerabilities are detected, it updates the dependencies using ``poetry update``.
+* If vulnerabilities are detected, the session updates the dependencies and
+  records the outcome for the pull request.
 
-   * If the ``poetry.lock`` is unchanged, then no further action is taken.
-   * If the ``poetry.lock`` is changed, then it creates a branch, stages the commit,
-     creates a pull request, and sends a Slack notification.
+   * If the update makes no effective change, then no further action is taken.
+   * If the update changes the dependency state, then the workflow pushes the
+     resulting commit, includes the post-update summary in the PR description,
+     opens a pull request, and sends a Slack notification.
 
 Afterwards, users need to perform some manual steps which are described in the PR description.
 

doc/user_guide/features/managing_dependencies/index.rst

@@ -19,12 +19,17 @@ Managing Dependencies and Vulnerabilities
      - ``report.yml``
      - Uses ``pip-licenses`` to return packages with their licenses.
    * - ``dependency:audit``
-     - ``dependency-update.yml``
+     - No
      - Uses ``pip-audit`` to report active vulnerabilities in our dependencies.
    * - ``vulnerabilities:resolved``
      - No
      - Uses ``pip-audit`` to report known vulnerabilities in dependencies that
        have been resolved in comparison to the last release.
+   * - ``vulnerabilities:update``
+     - ``dependency-update.yml``
+     - Uses ``pip-audit`` to update dependencies and commit ``poetry.lock`` when
+       vulnerabilities are found. It also produces a concise JSON summary for
+       the pull request description.
    * - ``workflow:audit``
      - ``checks.yml``
      - Uses ``zizmor`` to audit GitHub actions and workflows for security issues

exasol/toolbox/nox/_dependencies.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import argparse
 import json
 from pathlib import Path
 
@@ -18,9 +19,18 @@
 )
 from exasol.toolbox.util.dependencies.poetry_dependencies import get_dependencies
 from exasol.toolbox.util.dependencies.track_vulnerabilities import DependenciesAudit
+from exasol.toolbox.util.dependencies.update_dependencies import DependencyUpdater
 from noxconfig import PROJECT_CONFIG
 
 
+def _format_update_vulnerabilities_message(was_updated: bool, report_json: str) -> str:
+    if not was_updated:
+        return "No vulnerable dependencies were found."
+    if report_json == "[]":
+        return "No vulnerable dependencies remain after updating."
+    return report_json
+
+
 @nox.session(name="dependency:licenses", python=False)
 def dependency_licenses(session: Session) -> None:
     """Report licenses for all dependencies."""
@@ -35,16 +45,49 @@
 @nox.session(name="dependency:audit", python=False)
 def audit(session: Session) -> None:
     """Report known vulnerabilities."""
-
     try:
-        vulnerabilities = Vulnerabilities.load_from_pip_audit(working_directory=Path())
+        vulnerabilities = Vulnerabilities.load_from_pip_audit(
+            working_directory=PROJECT_CONFIG.root_path
+        )
     except PipAuditException as e:
         session.error(e.returncode, e.stdout, e.stderr)
 
     security_issue_dict = vulnerabilities.security_issue_dict
     print(json.dumps(security_issue_dict, indent=2))
 
 
+@nox.session(name="vulnerabilities:update", python=False)
+def update_vulnerabilities(session: Session) -> None:
+    """
+    Update vulnerabilities and optionally save the JSON of remaining vulnerabilities
+    to a file provided on the command line.
+    """
+    parser = argparse.ArgumentParser(
+        prog="nox -s vulnerabilities:update",
+        description="Update vulnerable dependencies and optionally write a report file.",
+    )
+    parser.add_argument(
+        "report_path",
+        nargs="?",
+        help="Optional path for the JSON report of remaining vulnerabilities.",
+    )
+    args = parser.parse_args(session.posargs)
+
+    try:
+        dependency_updater = DependencyUpdater(root_path=PROJECT_CONFIG.root_path)
+        was_updated, report_json = dependency_updater.update_vulnerable_dependencies()
+    except PipAuditException as e:
+        session.error(e.returncode, e.stdout, e.stderr)
+
+    if args.report_path is None:
+        print(_format_update_vulnerabilities_message(was_updated, report_json))
+        return
+
+    report_path = Path(args.report_path)
+    report_path.parent.mkdir(parents=True, exist_ok=True)
+    report_path.write_text(report_json + "\n", encoding="utf-8")
+
+
 @nox.session(name="vulnerabilities:resolved", python=False)
 def report_resolved_vulnerabilities(session: Session) -> None:
     """Report resolved vulnerabilities in dependencies."""

exasol/toolbox/nox/_release.py

@@ -62,9 +62,14 @@ def _get_changelogs(version: Version) -> Changelog:
     )
 
 
-def _add_files_to_index(session: Session, files: list[Path]) -> None:
-    for file in files:
-        session.run("git", "add", f"{file}")
+def _collect_release_files(session: Session, pm) -> tuple[Path, ...]:
+    return tuple(
+        path
+        for plugin_files in pm.hook.prepare_release_add_files(
+            session=session, config=PROJECT_CONFIG
+        )
+        for path in plugin_files
+    )
 
 
 class ReleaseError(Exception):
@@ -120,9 +125,7 @@ def prepare_release(session: Session) -> None:
     if not args.no_branch and not args.no_add:
         Git.create_and_switch_to_branch(f"release/prepare-{new_version}")
 
-    changed_files = (
-        _get_changelogs(version=new_version).prepare_release().get_changed_files()
-    )
+    changelogs = _get_changelogs(version=new_version).prepare_release()
 
     pm = NoxTasks.plugin_manager(PROJECT_CONFIG)
     pm.hook.prepare_release_update_version(
@@ -132,16 +135,11 @@ def prepare_release(session: Session) -> None:
     if args.no_add:
         return
 
-    changed_files += [
-        PROJECT_CONFIG.root_path / PoetryFiles.pyproject_toml,
-    ]
-    results = pm.hook.prepare_release_add_files(session=session, config=PROJECT_CONFIG)
-    changed_files += [f for plugin_response in results for f in plugin_response]
-    _add_files_to_index(
-        session,
-        changed_files,
-    )
-    session.run("git", "commit", "-m", f"Prepare release {new_version}")
+    version_files = (PROJECT_CONFIG.root_path / PoetryFiles.pyproject_toml,)
+    release_files = _collect_release_files(session=session, pm=pm)
+    changed_files = changelogs.get_changed_files() + release_files + version_files
+    Git.add(changed_files)
+    Git.commit(f"Prepare release {new_version}")
 
     if not args.no_pr:
         session.run(