Addressing upstream and downstream packages in Git repos in Porch

[liamfallon]

The aim of this discussion is to

• describe how Porch references Package revisions in git repositories
• decide if the current referencing approach is adequate or does it require improvement

How Porch works now

Porch uses the concepts of Repository, Package, and PackageRevision to wrangle packages and their contents in a given namespace.

Repository

A Porch repository is NOT equivalent to a Git repository, see the Repository API definiton. A Porch Repository is a reference to a branch and directory in a git repository.

To fully dereference a Porch repository in a namespace, you must specify:

• the Git repo URL
• the branch on that repo to use
• the directory location on that branch to use

Package

The concept of a package is a weak concept in Porch, see the Package API definition. Porch does not support API calls on Package resources and internally Porch uses filtering to do package-oriented checks.

The PackageName (key) of a Package is of the form repository-name.[directory-path-in-repo-]package-name.workspace-name

• A package called "my-lovely-package" in the root of the Porch repository is called *my-lovely-package"
• A package called "my-lovely-package" in the directory "potential/eurovision/packages" of the Porch repository is called potential-eurovision-packages-my-lovely-package

There is no support for defeferencing packages in Porch

PackageRevision

The concept of a PackageRevision is the most important nad fundamental concept in Porch, see the PackageRevision API definition. Package Revisions are the entities that Porch manipulates to deliver its use cases.

To fully dereference a Porch PackageRevision in a namespace, you must specify:

• its Repository
• its Package name
• its WorkspaceName

The Name (key) of a PackageRevision is of the form repository-name.package-name.workspace-name

WorkspaceName on PackageRevision

A WorkspaceName is the unique identifier of a PackageRevision within a package. It is assigned at package init/clone time and can be any string that is compliant with DNS name conventions. This means that

• you can use whatever WorkspaceName you like eg "gonzo", "yadayada", "version1", "v1", "delete-me"
• the WorkspaceName must be unique on its package
• the same workspace name can be used on other packages, so this is legal
• my-repo.ran-package.cell1
• my-repo.ev-battery-package.cell1
• WorkspaceName only has scope within a package, so if the same WorkspaceName appears in two packages Porch treats those PackageRevisions completely independently

Examples:

NAMESPACE    NAME                                  PACKAGE              WORKSPACENAME   REVISION   LATEST   LIFECYCLE   REPOSITORY
porch-demo   porch-test.ev-battery-package.cell1   ev-battery-package   cell1           0          false    Draft       porch-test
porch-demo   porch-test.ran-package.cell1          ran-package          cell1           0          false    Draft       porch-test

Revision on PackageRevision

A Revision on a PackageRevision identifies the order of publishing on published package revisions. When a PackageRevision is published, it is assigned a Revision one higher than the highest existing revision.

The following rules apply:

• A PackageRevision that is published or deletion-proposed has a Revision > 0
• A PackageRevision that is in draft of proposed has a Revision = 0
• A placeholder PackageRevision has a Revision = -1

NAMESPACE    NAME                                                 PACKAGE              WORKSPACENAME          REVISION   LATEST   LIFECYCLE   REPOSITORY
porch-demo   porch-test.ev-battery-package.main                   ev-battery-package   main                   -1         false    Published   porch-test
porch-demo   porch-test.ev-battery-package.cell1                  ev-battery-package   cell1                  1          false    Published   porch-test
porch-demo   porch-test.ev-battery-package.cell1-change-coolant   ev-battery-package   cell1-change-coolant   2          false    Published   porch-test
porch-demo   porch-test.ev-battery-package.cell1-check-voltage    ev-battery-package   cell1-check-voltage    3          true     Published   porch-test
porch-demo   porch-test.ran-package.cell1                         ran-package          cell1                  0          false    Draft       porch-test

Note:

• Porch does not use the Revision of a PackageRevision cannot be used to uniquely identify a PackageRevision
• The Revision of a PackageRevision is not unique
• There is no connection between the Revision of a PackageRevision and its WorkspaceName.

The following listing shows some legal but strongly discouraged examples:

NAMESPACE  NAME                                PACKAGE              WORKSPACENAME REVISION LATEST LIFECYCLE REPOSITORY
porch-demo porch-test.coincidence-package.main coincidence-package  main          -1       false  Published porch-test
porch-demo porch-test.coincidence-package.1    coincidence-package  1              1        true  Published porch-test
porch-demo porch-test.cofusing-package.main    confusing-package    main          -1       false  Published porch-test
porch-demo porch-test.cofusing-package.2       confusing-package    2              1       false  Published porch-test
porch-demo porch-test.cofusing-package.1       confusing-package    1              2        true  Published porch-test

The placholder PackageRevision

The placeholder PackageRevision is a dummy PackageRevision reference that points at the latest PackageRevision of a package.

The placeholder PackageRevision is created by Porch when the first PackageRevision on a package is created and is updated (actually deleted and recreated) each time a new PackageRevision is published on a package.

The following rules apply

• there is always at most one placeholder PackageRevision for a package
• it always has a Revision of -1
• it is named **repository-name.package-name.branch-name, where the branch-name is the branch in Git on which the PackageRevision exists

Discussion Point Is there a better way of finding the latest package revision than using these placeholder package revisions?

Reading PackageRevision instances from git

When Porch starts, it reads the existing PackageRevision instances from the Porch repositories defined in Repository resources. Porch runs a periodic job that checks for updates on repositories and refreshes the PackageRevision instances if any changes are detected.

Porch uses the following approach:

1. Iterate over the specified branch of each repository and walks the repository tree from the root down each subdirectory
   a. the subdirectory contains a file called "KptFile"
   __i. Create a PackageRevision with the name repo-name.[directory-path-to-this-subdirectory-]subdirectory-name.branch-name
   __ii. The WorkspaceName of the PackageRevision is set to the branch name of the porch repository.
   __iii. Exit from this subdirectory to the next highest level
   b. The subdirectory does not contain a file called "KptFile"
   __i. Continue searching subdirectories of this subdirectory (go to a. above)

2. Iterate over all tags in the Porch repository that match a path in the repository on that tag, tag is of the form path-to-directory-containing-package-revision/version
   a. There is a "kptFile" at the "path-to-directory-containing-package-revision" location in the Porch repository
   __i. Create a PackageRevision with the name repo-name.path-to-directory-containing-package-revision.version
   __ii. The WorkspaceName of the PackageRevision is set to the version of the tag.
   __iii. If the version is of the form vx, set the Revision of the PackageRevision to the value of x
   __iiv. Continue to next tag
   b. Report a warning and continue to next tag

Consider the Porch Catalog git repository

It has the branches "main", "v4", and "v3".

The package "oai-ran-operator" has the following tags:

git tag --list | grep oai-ran-operator
workloads/oai/oai-ran-operator/v1
workloads/oai/oai-ran-operator/v2.0.0
workloads/oai/oai-ran-operator/v3.0.0

Consider the three Porch repositories defined below:

apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  name: catalog-main
  namespace: default
spec:
  content: Package
  deployment: false
  git:
    branch: main
    directory: 
    repo: https://github.com/nephio-project/catalog.git
  type: git
---
apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  name: catalog-v4-0-0
  namespace: default
spec:
  content: Package
  deployment: false
  git:
    branch: v4
    directory: 
    repo: https://github.com/nephio-project/catalog.git
  type: git
---
apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  name: catalog-v3-0-0
  namespace: default
spec:
  content: Package
  deployment: false
  git:
    branch: v3.0.0
    directory: 
    repo: https://github.com/nephio-project/catalog.git
  type: git

Porch discovers the following PackageRevision instances for the "oai-ran-operator" package"

NAMESPACE  NAME                                                  PACKAGE                         WORKSPACENAME   REVISION  LATEST  LIFECYCLE  REPOSITORY
default    catalog-main.workloads.oai.oai-ran-operator.main      workloads/oai/oai-ran-operator  main            -1        false   Published  catalog-main
default    catalog-main.workloads.oai.oai-ran-operator.v2.0.0    workloads/oai/oai-ran-operator  v2.0.0          -1        false   Published  catalog-main
default    catalog-main.workloads.oai.oai-ran-operator.v3.0.0    workloads/oai/oai-ran-operator  v3.0.0          -1        false   Published  catalog-main
default    catalog-main.workloads.oai.oai-ran-operator.v1        workloads/oai/oai-ran-operator  v1              1         true    Published  catalog-main
default    catalog-v3-0-0.workloads.oai.oai-ran-operator.v2.0.0  workloads/oai/oai-ran-operator  v2.0.0          -1        false   Published  catalog-v3-0-0
default    catalog-v3-0-0.workloads.oai.oai-ran-operator.v3.0.0  workloads/oai/oai-ran-operator  v3.0.0          -1        false   Published  catalog-v3-0-0
default    catalog-v3-0-0.workloads.oai.oai-ran-operator.v1      workloads/oai/oai-ran-operator  v1              1         false   Published  catalog-v3-0-0
default    catalog-v3-0-0.workloads.oai.oai-ran-operator.v3      workloads/oai/oai-ran-operator  v3              3         true    Published  catalog-v3-0-0
default    catalog-v4-0-0.workloads.oai.oai-ran-operator.v2.0.0  workloads/oai/oai-ran-operator  v2.0.0          -1        false   Published  catalog-v4-0-0
default    catalog-v4-0-0.workloads.oai.oai-ran-operator.v3.0.0  workloads/oai/oai-ran-operator  v3.0.0          -1        false   Published  catalog-v4-0-0
default    catalog-v4-0-0.workloads.oai.oai-ran-operator.v1      workloads/oai/oai-ran-operator  v1              1         false   Published  catalog-v4-0-0
default    catalog-v4-0-0.workloads.oai.oai-ran-operator.v4      workloads/oai/oai-ran-operator  v4              4         true    Published  catalog-v4-0-0

Writing PackageRevision instances to git

How Porch should work