Bugfix/898 improve code related to bug in dependency update

.pre-commit-config.yaml

@@ -35,6 +35,7 @@ repos:
     rev: v5.0.0
     hooks:
       - id: check-yaml
+        exclude: ^exasol/toolbox/templates/github/workflows/
         stages: [ pre-commit ]
       - id: end-of-file-fixer
         stages: [ pre-commit ]

doc/changes/unreleased.md

@@ -2,7 +2,16 @@
 
 ## Summary
 
+This minor release adds automated vulnerability updates through Nox session
+`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
+few  workflow-related bug fixes and documentation updates.
+
 ## Bug
 
 * #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
-* #910: Add `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
+* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
+
+## Feature
+
+* #898: Created Nox session `vulnerabilities:update` to automatically resolve
+  vulnerable dependencies and report the result for the dependency update workflow

doc/user_guide/features/github_workflows/index.rst

@@ -39,11 +39,11 @@ Workflows
 The PTB allows for two categories of workflows:
   #. those maintained by the PTB, which can be modified using the :ref:`workflow_patcher`.
   #. custom workflows, which are project-owned.
-  
-Custom workflows can optionally be 
+
+Custom workflows can optionally be
 * seeded by the PTB, i.e. PTB generates an initial version but ignores future changes.
 * extend PTB-provided workflows, i.e. ending in `-extension.yml`
- 
+
 Besides that, you can also create individual workflow files which are ignored by the PTB.
 
 Maintained by the PTB
@@ -311,14 +311,17 @@ checks for known vulnerabilities and tries to fix them by updating dependencies.
    :start-at: on:
    :end-at:  workflow_dispatch:
 
-The workflow first audits dependencies for known vulnerabilities:
+The workflow runs a dedicated ``vulnerabilities:update`` nox session that audits the
+current dependency set and attempts to resolve any detected vulnerabilities:
 
 * If no vulnerabilities are detected, then no update is needed.
-* If vulnerabilities are detected, it updates the dependencies using ``poetry update``.
+* If vulnerabilities are detected, the session updates the dependencies and
+  records the outcome for the pull request.
 
-   * If the ``poetry.lock`` is unchanged, then no further action is taken.
-   * If the ``poetry.lock`` is changed, then it creates a branch, stages the commit,
-     creates a pull request, and sends a Slack notification.
+   * If the update makes no effective change, then no further action is taken.
+   * If the update changes the dependency state, then the workflow pushes the
+     resulting commit, includes the post-update summary in the PR description,
+     opens a pull request, and sends a Slack notification.
 
 Afterwards, users need to perform some manual steps which are described in the PR description.
 

doc/user_guide/features/managing_dependencies/index.rst

@@ -19,12 +19,17 @@ Managing Dependencies and Vulnerabilities
      - ``report.yml``
      - Uses ``pip-licenses`` to return packages with their licenses.
    * - ``dependency:audit``
-     - ``dependency-update.yml``
+     - No
      - Uses ``pip-audit`` to report active vulnerabilities in our dependencies.
    * - ``vulnerabilities:resolved``
      - No
      - Uses ``pip-audit`` to report known vulnerabilities in dependencies that
        have been resolved in comparison to the last release.
+   * - ``vulnerabilities:update``
+     - ``dependency-update.yml``
+     - Uses ``pip-audit`` to update dependencies and commit ``poetry.lock`` when
+       vulnerabilities are found. It also produces a concise JSON summary for
+       the pull request description.
    * - ``workflow:audit``
      - ``checks.yml``
      - Uses ``zizmor`` to audit GitHub actions and workflows for security issues

exasol/toolbox/nox/_dependencies.py

@@ -18,6 +18,7 @@
 )
 from exasol.toolbox.util.dependencies.poetry_dependencies import get_dependencies
 from exasol.toolbox.util.dependencies.track_vulnerabilities import DependenciesAudit
+from exasol.toolbox.util.dependencies.update_dependencies import DependencyUpdater
 from noxconfig import PROJECT_CONFIG
 
 
@@ -35,16 +36,38 @@ def dependency_licenses(session: Session) -> None:
 @nox.session(name="dependency:audit", python=False)
 def audit(session: Session) -> None:
     """Report known vulnerabilities."""
-
     try:
-        vulnerabilities = Vulnerabilities.load_from_pip_audit(working_directory=Path())
+        vulnerabilities = Vulnerabilities.load_from_pip_audit(
+            working_directory=PROJECT_CONFIG.root_path
+        )
     except PipAuditException as e:
         session.error(e.returncode, e.stdout, e.stderr)
 
     security_issue_dict = vulnerabilities.security_issue_dict
     print(json.dumps(security_issue_dict, indent=2))
 
 
+@nox.session(name="vulnerabilities:update", python=False)
+def update_vulnerabilities(session: Session) -> None:
+    """
+    Update vulnerabilities, and save the JSON of remaining vulnerabilities to
+    `VULNERABILITIES_UPDATE_REPORT_PATH` if it is set in the environment.
+    """
+    try:
+        dependency_updater = DependencyUpdater(root_path=PROJECT_CONFIG.root_path)
+        report_json = dependency_updater.update_vulnerable_dependencies()
+    except PipAuditException as e:
+        session.error(e.returncode, e.stdout, e.stderr)
+
+    report_path_str = session.env.get("VULNERABILITIES_UPDATE_REPORT_PATH", None)
+    if report_json is None or report_path_str is None:
+        return
+
+    report_path = Path(report_path_str)
+    report_path.parent.mkdir(parents=True, exist_ok=True)
+    report_path.write_text(report_json + "\n", encoding="utf-8")
+
+
 @nox.session(name="vulnerabilities:resolved", python=False)
 def report_resolved_vulnerabilities(session: Session) -> None:
     """Report resolved vulnerabilities in dependencies."""

exasol/toolbox/nox/_release.py

@@ -62,9 +62,14 @@ def _get_changelogs(version: Version) -> Changelog:
     )
 
 
-def _add_files_to_index(session: Session, files: list[Path]) -> None:
-    for file in files:
-        session.run("git", "add", f"{file}")
+def _collect_release_files(session: Session, pm) -> tuple[Path, ...]:
+    return tuple(
+        path
+        for plugin_files in pm.hook.prepare_release_add_files(
+            session=session, config=PROJECT_CONFIG
+        )
+        for path in plugin_files
+    )
 
 
 class ReleaseError(Exception):
@@ -120,9 +125,7 @@ def prepare_release(session: Session) -> None:
     if not args.no_branch and not args.no_add:
         Git.create_and_switch_to_branch(f"release/prepare-{new_version}")
 
-    changed_files = (
-        _get_changelogs(version=new_version).prepare_release().get_changed_files()
-    )
+    changelogs = _get_changelogs(version=new_version).prepare_release()
 
     pm = NoxTasks.plugin_manager(PROJECT_CONFIG)
     pm.hook.prepare_release_update_version(
@@ -132,16 +135,11 @@ def prepare_release(session: Session) -> None:
     if args.no_add:
         return
 
-    changed_files += [
-        PROJECT_CONFIG.root_path / PoetryFiles.pyproject_toml,
-    ]
-    results = pm.hook.prepare_release_add_files(session=session, config=PROJECT_CONFIG)
-    changed_files += [f for plugin_response in results for f in plugin_response]
-    _add_files_to_index(
-        session,
-        changed_files,
-    )
-    session.run("git", "commit", "-m", f"Prepare release {new_version}")
+    version_files = (PROJECT_CONFIG.root_path / PoetryFiles.pyproject_toml,)
+    release_files = _collect_release_files(session=session, pm=pm)
+    changed_files = changelogs.get_changed_files() + release_files + version_files
+    Git.add(changed_files)
+    Git.commit(f"Prepare release {new_version}")
 
     if not args.no_pr:
         session.run(

exasol/toolbox/templates/github/workflows/dependency-update.yml

@@ -39,74 +39,69 @@ jobs:
           python-version: "(( minimum_python_version ))"
           poetry-version: "(( dependency_manager_version ))"
 
-      - name: Audit Dependencies
-        id: audit-dependencies
-        run: |
-          set -o pipefail
-          poetry self add poetry-plugin-export
-          # Pipeline purpose:
-          # - `2>&1`: merge stdout and stderr into one stream.
-          # - `tee /dev/stderr`: mirror the combined output back to stderr so it stays visible in the logs.
-          # - `sed -n '/^\[/,$p'`: keep only the JSON payload and write it to vulnerabilities.json.
-          # With `set -o pipefail`, any failure in the pipeline still fails this step and turns the workflow red.
-          poetry run -- nox -s dependency:audit 2>&1 | tee /dev/stderr | sed -n '/^\[/,$p' > vulnerabilities.json
-          LENGTH=$(jq 'length' vulnerabilities.json)
-          echo "count=$LENGTH" >> "$GITHUB_OUTPUT"
-
-      - name: Update Dependencies
-        id: update-dependencies
-        if: steps.audit-dependencies.outputs.count > 0
-        run: poetry update
-
-      - name: Check for poetry.lock Changes
-        id: check-for-poetry-lock-changes
-        if: steps.audit-dependencies.outputs.count > 0
-        run: |
-          if git diff --quiet -- poetry.lock; then
-            echo "changed=false" >> "$GITHUB_OUTPUT"
-          else
-            echo "changed=true" >> "$GITHUB_OUTPUT"
-          fi
-
       - name: Configure git
         id: configure-git
-        if: steps.check-for-poetry-lock-changes.outputs.changed == 'true'
         run: |
           git config --global user.email "opensource@exasol.com"
           git config --global user.name "Automatic Dependency Updater"
 
       - name: Create branch
         id: create-branch
-        if: steps.check-for-poetry-lock-changes.outputs.changed == 'true'
         run: |
           branch_name="dependency-update/$(date "+%Y-%m-%d")"
           echo "Creating branch $branch_name"
           git switch -C "$branch_name"
 
-      - name: Commit Changes & Push
+      - name: Record current commit
+        id: record-current-commit
+        run: |
+          echo "head_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+
+      - name: Update Dependencies
+        id: update-dependencies
+        env:
+          VULNERABILITIES_UPDATE_REPORT_PATH: vulnerabilities.json
+        run: |
+          poetry self add poetry-plugin-export
+          poetry run -- nox -s vulnerabilities:update
+
+      - name: Check for New Commit
+        id: check-for-new-commit
+        env:
+          PRE_UPDATE_HEAD_SHA: ${{ steps.record-current-commit.outputs.head_sha }}
+        run: |
+          if [ "$(git rev-parse HEAD)" = "$PRE_UPDATE_HEAD_SHA" ]; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Push Changes
         id: publish-branch
-        if: steps.check-for-poetry-lock-changes.outputs.changed == 'true'
+        if: steps.check-for-new-commit.outputs.changed == 'true'
         run: |
           branch_name=$(git rev-parse --abbrev-ref HEAD)
-          git add poetry.lock
-          git commit --message "Updated poetry.lock"
           git push --set-upstream origin "$branch_name"
 
       - name: Create Pull Request
         id: create-pr
-        if: steps.check-for-poetry-lock-changes.outputs.changed == 'true'
+        if: steps.check-for-new-commit.outputs.changed == 'true'
         env:
           GH_TOKEN: ${{ github.token }}
         run: |
           BASE_BRANCH=$(gh repo view --json defaultBranchRef -q .defaultBranchRef.name)
+          DEPENDENCY_UPDATE_REPORT="$(cat "$VULNERABILITIES_UPDATE_REPORT_PATH")"
 
           PR_BODY="Automated dependency update for \`poetry.lock\`.
           This PR was created by the workflow \`dependency-update.yml\`
 
+          Remaining vulnerable dependencies after the automated update:
+          \`\`\`json
+          ${DEPENDENCY_UPDATE_REPORT}
+          \`\`\`
+
           Please perform the following actions on a locally checked out branch:
           - [ ] Execute \`poetry run -- nox -s workflow:generate -- all\`
-          - [ ] Use \`poetry run -- nox -s dependency:audit\` to check for vulnerabilities requiring manual action
-          - [ ] Update file \`doc/changes/unreleased.md\`
           "
 
           PR_URL=$(gh pr create \