feat: discover and parse project-maintained affiliations (CM-361) by skwowet · Pull Request #4280 · linuxfoundation/crowd.dev

skwowet · 2026-06-30T09:59:22Z

Summary

This PR adds processing for project-maintained contributor-to-employer files inside git_integration.

Many LF projects publish who works for which company in repo files (gitdm, .organizationmap, SIG YAML, etc.). Formats and filenames vary, so we reuse the existing clone pipeline: find the file on disk, parse it with AI into a normalized snapshot, track per-repo state, and resolve rows against CDP members and organizations.

The worker runs this on the first clone batch, alongside maintainer processing. Each repo gets one registry row (git.repoAffiliationRegistry) plus a serviceExecutions record with operation type Affiliation.

Run flow

Interval elapsed?
  no  → skip run
  yes ↓

Resolve file path
  saved path still on disk? → use it
  else discover:
    list text-like files at repo root → AI picks from candidates (path + 400 char preview per file)
  ↓

Parse
  file hash unchanged and snapshot valid? → reuse snapshot (no AI)
  else → AI extract flat rows, group by contributor into snapshot
  ↓

Fork? → drop stints already on parent repo's cached snapshot

Apply (dry run in this PR)
  for each stint: lookup member + org, run guards, build MO/MSA inserts
  actual INSERT calls are commented out
  ↓

Update registry (filePath, fileHash, status, snapshot) + serviceExecutions

Parsed snapshot shape

Each snapshot entry is one contributor with their organization stints over time:

Contributor: email or GitHub username required (email preferred for grouping and lookup). Name is optional.
Organizations: each stint has domain (required for CDP lookup), optional name, optional dateStart/dateEnd, and isUnaffiliated for independent/unaffiliated rows (normalized to individual-noaccount.com). AI returns flat contributor–org rows; code groups by contributor and dedupes stints. Rows missing email/github or domain are dropped during normalization.

Registry status: success (including zero-row parses), not_found, unusable (parsed but nothing applyable; stable until file content changes), error (unexpected parse failure; retry on next interval).

Apply rules (implemented, writes disabled for now)

Lookup only: no find-or-create for members or orgs. Unresolved rows are skipped and retried on the next run using the same snapshot.
Member identity: email first (git username identity), else GitHub username.
Org identity: verified primary-domain only (no display-name lookup).
Guards: skip MO/MSA insert when the same member + org + date range already exists, or an undated/open-ended row already covers an undated insert (MSA scoped to the repo segment).
MO source would be project-registry when inserts are enabled.

Repos without a segment skip apply. Unexpected parse failures set registry status to error, clear fileHash, and keep the previous snapshot so the next interval run retries parsing.

Changes

AffiliationService (services/affiliation/affiliation_service.py)

Root-level text file discovery; AI file picker on all candidates (no known-filename list or ripgrep). Batches candidates (20 per call) with short file previews; rejects scripts/governance/credits in the picker prompt.
AI extraction returns flat rows with optional dates and unaffiliated markers; group_parse_rows merges into contributor + organizations[] snapshot shape.
AI extraction with chunked parsing for files over 5k chars.
SHA-gated snapshot caching to avoid re-parsing unchanged files.
Separate update interval (24h when a file is found) and retry interval (30d when not found).
Fork repos: filter stints already present on parent repo snapshot before apply.
Full apply path: bulk identity resolution, dedupe, stint-level guard checks, insert list building. MO/MSA executemany calls commented out with a TODO.

Worker wiring (repository_worker.py, server.py)

AffiliationService registered and invoked on is_first_batch after maintainer processing.

Database layer (crud.py)

Registry get/upsert and snapshot JSON serialize/deserialize via Pydantic.
Bulk member/org identity lookups (single query per entity type, index-aligned results).
Fetch existing MO and segment-scoped MSA for guards.
Insert helpers for MO (ON CONFLICT DO NOTHING aligned to partial unique indexes) and MSA.

Shared infrastructure

Bedrock client moved from maintainer/bedrock.py to services/llm/bedrock.py; maintainer import updated.
New Pydantic models (affiliation_info.py), error types, AffiliationRegistryStatus, OperationType.AFFILIATION, and AFFILIATION_* settings in settings.py / conftest.py.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

This PR introduces a new AffiliationService to the git_integration worker pipeline to discover and AI-parse project-maintained contributor→employer/organization mapping files, cache parsed snapshots per repo, and (currently) dry-run the apply/guard logic while recording service executions and registry state.

Changes:

Added AffiliationService and wired it into RepositoryWorker/server lifespan for first-clone batches.
Added Bedrock LLM client module under services/llm/ and updated maintainer processing to import it.
Extended the git_integration DB CRUD layer with affiliation registry read/upsert plus bulk identity lookup helpers.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
services/apps/git_integration/src/test/conftest.py	Adds default env vars for affiliation retry/update intervals in tests.
services/apps/git_integration/src/crowdgit/worker/repository_worker.py	Injects and invokes `AffiliationService` in the repository processing pipeline.
services/apps/git_integration/src/crowdgit/settings.py	Adds `AFFILIATION_*` interval settings.
services/apps/git_integration/src/crowdgit/services/maintainer/maintainer_service.py	Switches Bedrock import to the new shared LLM module.
services/apps/git_integration/src/crowdgit/services/llm/bedrock.py	New shared Bedrock invocation helper (retry, JSON extraction, pydantic validation, cost calc).
services/apps/git_integration/src/crowdgit/services/llm/init.py	Initializes the new `llm` service package.
services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py	Implements discovery, AI parsing (including chunking), snapshot caching, and dry-run apply/guards.
services/apps/git_integration/src/crowdgit/services/init.py	Exports `AffiliationService` for DI/wiring.
services/apps/git_integration/src/crowdgit/server.py	Instantiates and passes `AffiliationService` into `RepositoryWorker`.
services/apps/git_integration/src/crowdgit/models/affiliation_info.py	Adds pydantic models for file-picker and parsed affiliation snapshot rows.
services/apps/git_integration/src/crowdgit/errors.py	Adds new affiliation-specific error types.
services/apps/git_integration/src/crowdgit/enums.py	Adds new error codes, registry status enum, and `OperationType.REPO_AFFILIATION`.
services/apps/git_integration/src/crowdgit/database/crud.py	Adds registry CRUD + snapshot (de)serialization + bulk identity lookup helpers + insert helpers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

…de text file extensions Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 4 comments.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 1 comment.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 1 comment.

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

…ntity Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

…pe and add date parsing util Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 13 changed files in this pull request and generated 2 comments.

+    def _parse_optional_date(value: str | None) -> date | None:
+        stripped = AffiliationService._strip(value)
+        if not stripped:
+            return None
+        return date.fromisoformat(stripped)


+    for idx, identity in enumerate(identities):
+        values_parts.append(
+            f"(${param_index}::int, ${param_index + 1}::text, ${param_index + 2}::boolean,"
+            f" ${param_index + 3}::text, ${param_index + 4}::text)"
+        )
+        params.extend(
+            [
+                idx,
+                identity["type"],
+                identity.get("verified", True),
+                identity.get("platform"),
+                identity["value"],
+            ]
+        )
+        param_index += 5


feat: discover and parse project-maintained affiliations

7f136af

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

skwowet requested a review from mbani01 June 30, 2026 09:59

skwowet self-assigned this Jun 30, 2026

Copilot AI review requested due to automatic review settings June 30, 2026 09:59

Copilot started reviewing on behalf of skwowet June 30, 2026 09:59 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

skwowet changed the title ~~feat: discover and parse project-maintained affiliations~~ feat: discover and parse project-maintained affiliations (CM-361) Jun 30, 2026

skwowet added 2 commits June 30, 2026 16:05

fix: resolve pr review comments

001b82c

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Merge branch 'main' into feat/CM-361-part-1

f0efc5a

Copilot AI review requested due to automatic review settings June 30, 2026 10:35

Copilot started reviewing on behalf of skwowet June 30, 2026 10:36 View session

Merge branch 'main' into feat/CM-361-part-1

9cbf4b9

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py

Copilot AI review requested due to automatic review settings June 30, 2026 10:39

Copilot started reviewing on behalf of skwowet June 30, 2026 10:39 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

skwowet added 2 commits June 30, 2026 16:50

fix: resolve pr review comments

31dc447

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

feat: enhance affiliation service with new status and refactor logging

e9cdd2a

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 30, 2026 12:25

Copilot started reviewing on behalf of skwowet June 30, 2026 12:25 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py Outdated

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py

mbani01 reviewed Jun 30, 2026

View reviewed changes

skwowet added 2 commits July 1, 2026 00:01

Merge branch 'main' into feat/CM-361-part-1

3e1c41c

fix: resolve pr review comments

17a99a3

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 30, 2026 18:31

Copilot started reviewing on behalf of skwowet June 30, 2026 18:31 View session

skwowet added 2 commits July 1, 2026 00:07

refactor: update glob pattern handling in AffiliationService to inclu…

48134bd

…de text file extensions Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: rm redundant check

dcb5657

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

skwowet requested a review from mbani01 June 30, 2026 18:47

Copilot AI reviewed Jun 30, 2026

Copilot AI reviewed Jul 1, 2026

View reviewed changes

skwowet added 2 commits July 1, 2026 21:51

fix: resolve pr review comments

202dc1d

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: affiliation registry writes and expected-run reporting

a101f37

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 1, 2026 17:26

Copilot started reviewing on behalf of skwowet July 1, 2026 17:27 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py Outdated

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py Outdated

skwowet added 2 commits July 1, 2026 23:16

refactor: batch affiliation filename search like maintainers

2da0d4f

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: resolve pr review comments

3d1cd30

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 1, 2026 17:48

Copilot started reviewing on behalf of skwowet July 1, 2026 17:48 View session

fix: make prettier and linter happy

ce8d853

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py Outdated

Copilot AI review requested due to automatic review settings July 1, 2026 18:00

Copilot started reviewing on behalf of skwowet July 1, 2026 18:01 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py Outdated

skwowet added 2 commits July 2, 2026 00:51

refactor: retry malformed affiliation parses once before unusable

8854ce3

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Merge branch 'main' into feat/CM-361-part-1

f1addda

Copilot AI review requested due to automatic review settings July 2, 2026 09:45

Copilot started reviewing on behalf of skwowet July 2, 2026 09:46 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/database/crud.py Outdated

skwowet added 2 commits July 2, 2026 22:30

feat: support affiliation stints and improve extraction coverage

17b6793

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

fix: prefer email over github and resolve git emails via username ide…

3bb2b9e

…ntity Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 2, 2026 17:35

Copilot started reviewing on behalf of skwowet July 2, 2026 17:35 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread services/apps/git_integration/src/crowdgit/models/affiliation_info.py Outdated

Comment thread services/apps/git_integration/src/crowdgit/services/affiliation/affiliation_service.py

skwowet added 2 commits July 2, 2026 23:23

fix: change date fields in AffiliationOrganizationFields to string ty…

c1cea25

…pe and add date parsing util Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>

Merge branch 'main' into feat/CM-361-part-1

1790067

Copilot AI review requested due to automatic review settings July 2, 2026 17:54

Copilot started reviewing on behalf of skwowet July 2, 2026 17:54 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

skwowet commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

skwowet commented Jun 30, 2026 •

edited

Loading