Skip to content

index.rst

Latest commit

 

History

History
360 lines (275 loc) · 12.3 KB

index.rst

File metadata and controls

360 lines (275 loc) · 12.3 KB
Raw

GitHub Workflow Templates

.. toctree::
    :maxdepth: 1
    :hidden:

    github_project_configuration
    create_and_update
    workflow_variables
    workflow_patcher

The PTB ships with configurable GitHub workflow templates covering the most common CI/CD setup variants for Python projects. The templates are defined in: exasol/toolbox/templates/github/workflows.

The PTB provides a command line interface (CLI) for managing workflows from the templates.

poetry run -- nox -s workflow:generate --help
poetry run -- nox -s workflow:check --help

Use workflow:generate to create or update workflows and workflow:check to compare the rendered workflow templates against the files in .github/workflows.

Attention!

In most cases, we recommend using all workflows without change to ensure consistent interdependencies. For more details, see :ref:`manual_activities` and :ref:`automated_activities`.

Workflows

The PTB allows for two categories of workflows:
  1. those maintained by the PTB, which can be modified using the :ref:`workflow_patcher`.
  2. custom workflows, which are project-owned.

Custom workflows can optionally be * seeded by the PTB, i.e. PTB generates an initial version but ignores future changes. * extend PTB-provided workflows, i.e. ending in -extension.yml

Besides that, you can also create individual workflow files which are ignored by the PTB.

Maintained by the PTB

Note

The matrix.yml workflow replaces the older matrix-all.yml, matrix-exasol.yml, and matrix-python.yml workflows. The associated nox sessions (matrix:all, matrix:exasol, and matrix:python) are deprecated and will be removed 2026-09-15.

Filename Run on Description
build-and-publish.yml Workflow call Packages the distribution and publishes it to PyPi and GitHub.
cd.yml Push with new tag Manages continuous delivery by creating and uploading build artifacts and documentation. See :ref:`cd_yml` for a graph of workflow calls.
check-release-tag.yml Workflow call Verifies that the release tag matches the project's internal versioning.
checks.yml Workflow call Executes many small & fast checks: builds documentation, validates cross-references and links in the documentation to be valid, runs various linters (security, type checks, etc.), and validates PTB generated workflows.
ci.yml Pull request Executes the continuous integration suite by calling merge-gate.yml and report.yml. See :ref:`ci_yml` for a graph of workflow calls.
fast-tests.yml Workflow call Executes unit tests.
dependency-update.yml Weekly and manual Audits project dependencies for known vulnerabilities, updates them with Poetry when needed, and creates a pull request if the poetry.lock was changed.
gh-pages.yml Workflow call Builds the documentation and deploys it to GitHub Pages.
matrix.yml Workflow call Calls Nox session matrix:generate to build a custom matrix from the PROJECT_CONFIG. See :ref:`workflow_matrix`.
merge-gate.yml Workflow call Acts as a final status check (gatekeeper) to ensure all required CI steps have passed before allowing to merge the branch of your pull request to the default branch of the repository. e.g. main.
periodic-validation.yml weekly Acts as a periodic validator that critical checks and tests are working as expected. See :ref:`periodic_validation_yml` for a graph of workflow calls.
pr-merge.yml Push to main Runs gh-pages.yml. See :ref:`pr_merge_yml` for a graph of the called workflow.
report.yml Workflow call Downloads results from code coverage analysis and linting, creates a summary displayed by GitHub as result of running the action, and uploads the results to Sonar.

Custom Workflows

Custom workflows are project-owned. The two custom workflow forms are:

  • workflows seeded once by the PTB and then owned by the project
  • workflow extensions that are enabled when the project adds the corresponding file

The PTB detects whether the relevant workflow file is present and activates a block where is it called in corresponding calling workflow. If the file is absent, the PTB does not add that block to the PTB-controlled workflows.

For these custom workflows to work correctly, the reusable workflow must follow the secret declaration format described in :ref:`custom_workflow_secrets` so the calling PTB workflow can pass the required secrets through.

Workflows Initially Seeded by the PTB

The PTB seeds these workflows for new projects, but after that the project owns them and PTB regeneration does not overwrite them.

Filename Run on Description
slow-checks.yml Workflow call Runs long-running checks, which typically involve an Exasol database instance.

Workflow Extensions

To use a workflow extension, a user must simply add the file to their project's .github/workflows directory. The PTB checks that this file exists, and if it does, then it automatically activates calling that workflow in the relevant parent workflow. These files are not otherwise checked or maintained by the PTB. The purpose behind the extensions is to allow project customization in a way that delineates what comes from and is maintained by the PTB and what is project-specific.

Filename Run on Description
fast-tests-extension.yml Workflow call This extends the fast-tests.yml and should include additional fast tests.
cd-extension.yml Workflow call This extends the cd.yml. Use it to add project-specific release steps. Runs after build-and-publish.yml workflow has finished successfully.
merge-gate-extension.yml Workflow call This extends the merge-gate.yml and the needs criteria of the job allow-merge. This extension is used to define additional requirements to the merge-gate, and it is most likely to include costly slows checks or tests. It's encouraged to add to this workflow extension additional approval requests, similar to approve-run-slow-tests.

Manual Activities

The following workflows are typically triggered directly by a developer during normal project maintenance.

Create a PR

Modifying any pull request triggers the workflow ci.yml.

.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/ci.yml
   :language: yaml
   :start-at: on:
   :end-at:    types: [opened, synchronize, reopened]

When configured as described on :ref:`github_project_configuration`, the run-slow-tests job requires a developer to manually approve executing the slower workflows, like slow-checks.yml. This allows developers to update their pull request more often and to only periodically run the more time-expensive tests.

The report.yml is called twice:

  1. after the steps in checks.yml successfully finish - this allows developers to get faster feedback for linting, security, and unit test coverage.
  2. after the steps in slow-checks.yml successfully finish - this gives developers an overview of the total coverage, as well as the information provided from running the checks.yml

In both scenarios, the results are posted in the PR and made available on Sonar's UI. Note that Sonar does not keep historical information, so it will only show the latest information provided to it.

If one of the jobs in the chain fails (or if run-slow-tests is not approved), then the subsequent jobs will not be started.

.. mermaid::

    graph TD
        %% Define Nodes
        checks[checks.yml]
        ci_job[ci.yml]
        fast-report[1st call to report.yml]
        fast-tests[fast-tests.yml]
        gate[merge-gate.yml]
        slow_run[run-slow-tests]
        slow_checks[slow-checks.yml]
        report[2nd call to report.yml]

        approver[approve-merge]

        %% Workflow Triggers
        ci_job --> gate
        gate --> checks
        gate --> fast-tests
        gate --> slow_run
        slow_run -.->|needs| slow_checks

        %% Dependencies
        checks -.->|needs| approver
        checks -.->|needs| fast-report
        fast-tests -.->|needs| fast-report
        fast-tests -.->|needs| approver
        slow_checks -.->|needs| approver
        approver -.->|needs| report

        %% Styling
        style approver fill:#fff,stroke:#333,stroke-dasharray: 5 5
        style slow_run fill:#fff,stroke:#333,stroke-dasharray: 5 5

Merge a PR

Merging a pull request to the default branch, activates the pr-merge.yml workflow.

.. mermaid::
   :name: merge-diagram

    graph TD
        %% Workflow Triggers (Solid Lines)
        pr-merge[pr-merge.yml] --> publish-docs[gh-pages.yml]

Create a Release

When the nox session release:trigger is used, a new tag is created & pushed to main. This starts the release process by activating the cd.yml workflow.

.. mermaid::
    :name: release-diagram

    graph TD
        %% Workflow Triggers (Solid Lines)
        cd[cd.yml] --> check-release-tag[check-release-tag.yml]
        cd --> cd-extension[cd-extension.yml]

        %% Dependencies / Waiting (Dotted Lines)
        check-release-tag -.->|needs| build-and-publish[build-and-publish.yml]
        build-and-publish -.->|needs| cd-extension[cd-extension.yml]
        build-and-publish -.->|needs| gh-pages[gh-pages.yml]

Automated Activities

The following workflows are triggered automatically by schedules.

Update Dependencies

The dependency-update.yml workflow runs on the default branch. It checks for known vulnerabilities and tries to fix them by updating dependencies.

.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/dependency-update.yml
   :language: yaml
   :start-at: on:
   :end-at:  workflow_dispatch:

The workflow runs a dedicated vulnerabilities:update nox session that audits the current dependency set and attempts to resolve any detected vulnerabilities:

  • If no vulnerabilities are detected, then no update is needed.

  • If vulnerabilities are detected, the session updates the dependencies and records the outcome for the pull request.

    • If the update makes no effective change, then no further action is taken.
    • If the update changes the dependency state, then the workflow pushes the resulting commit, includes the post-update summary in the PR description, opens a pull request, and sends a Slack notification.

Afterwards, users need to perform some manual steps which are described in the PR description.

Validate Periodically

The periodic-validation.yml runs on the default branch. It ensures that critical checks and tests continue to run and sends the results of the linting tools and test coverage to Sonar for an overall report.

.. literalinclude:: ../../../../exasol/toolbox/templates/github/workflows/periodic-validation.yml
   :language: yaml
   :start-at:   schedule:
   :end-at:  workflow_dispatch:


.. mermaid::

    graph TD
        %% Define Nodes
        checks[checks.yml]
        periodic_validation[periodic-validation.yml]
        fast-tests[fast-tests.yml]
        slow_checks[slow-checks.yml]
        report[report.yml]

        %% Workflow Triggers
        periodic_validation --> checks
        periodic_validation --> fast-tests
        periodic_validation --> slow_checks

        %% Dependencies
        checks -.->|needs| report
        fast-tests -.->|needs| report
        slow_checks -.->|needs| report