Air-Gapped Implementation 5 — Repository setup and CI trigger (air-gapped Gerrit + Zuul)¶

Goal¶

Create the required Gerrit projects (openstack/kolla, openstack/project-config) via UI.
Ensure All-Projects permissions and refs/meta/config settings allow the automation pushes.
Push Zuul tenant/project config and seed Kolla code into Gerrit.
Trigger a CI change on openstack/project-config so Zuul runs kolla-build-full (Kolla repo remains immutable).
Verify in Gerrit + Zuul UI.

0) Prerequisites (must be true before you start)¶

0.1 Export variables on the automation host¶

Run exactly:

# VM topology
export VM1_IP=192.0.2.10   # Nexus
export VM2_IP=192.0.2.11   # Gerrit + Zuul

# Gerrit
export GERRIT_SSH_PORT=29418

# Nexus (HTTP + Docker)
export NEXUS_HTTP_PORT=8081
export NEXUS_DOCKER_PROXY_PORT=8082     # pull-through proxy
export NEXUS_DOCKER_HOSTED_PORT=8083    # hosted (push target)

# Kolla / paths
export KOLLA_BRANCH=stable/2024.1
export INSTALL_DIR=/opt/sa-qi

# project-config branch used in Gerrit (choose one and keep it consistent)
export PROJECT_CONFIG_BRANCH=master

0.2 Confirm the SSH key used for pushes exists¶

This key must match what Gerrit knows for the admin user:

ls -la "${INSTALL_DIR}/zuul/ssh/id_rsa" "${INSTALL_DIR}/zuul/ssh/id_rsa.pub"

0.3 Define local workspace paths (create them)¶

mkdir -p "${INSTALL_DIR}/repos"

0.4 Network and service assumptions (air-gapped)¶

Gerrit Web UI reachable on VM2 (example: http://${VM2_IP}:8080 or your actual Gerrit web port).
Zuul dashboard reachable on VM2: http://${VM2_IP}:9000
Nexus reachable on VM1:
- http://${VM1_IP}:${NEXUS_HTTP_PORT}
- Docker proxy (pull): ${VM1_IP}:${NEXUS_DOCKER_PROXY_PORT}
- Docker hosted (push): ${VM1_IP}:${NEXUS_DOCKER_HOSTED_PORT}
- Gerrit is configured to accept push options (needed for -o skip-validation) in your deployment. If it is not enabled, the initial “skip-validation” pushes will fail.

0.5 Upstream references (source of truth for initial clone)¶

Kolla: https://opendev.org/openstack/kolla (checkout ${KOLLA_BRANCH})
Zuul configs reference: https://opengit.ir/devops/saqi/zuul-configs

1) Create Gerrit projects via Gerrit UI (required by your process) — VM2¶

Open Gerrit Web UI on VM2
Sign in as an admin user (must be in Administrators group)

1.2 Create project: `openstack/kolla`¶

Go to Browse → Repositories (or Projects → List, depending on Gerrit theme/version)
Click Create New Project
Fill in:
- Project Name: openstack/kolla
- Parent: All-Projects
- Create initial empty commit: Enabled
- Submit Type: keep default unless your standard requires a specific one
- Click Create Project

1.3 Create project: `openstack/project-config`¶

Repeat the same UI steps:

Project Name: openstack/project-config
Parent: All-Projects
Create initial empty commit: Enabled
Click Create Project

2) Configure access management in Gerrit UI (All-Projects) — VM2¶

You must ensure:

Administrators can forge and force-push where required.
Service users (Zuul) have the capabilities they need.
receive.maxBatchCommits = 0 is set once in refs/meta/config.

2.1 Update access rules (UI)¶

In Gerrit UI, open Projects → List
Open All-Projects
Go to the Access tab (or Permissions)

2.1.1 Ensure permissions for `refs/*`¶

Under reference refs/*, add or verify:

Forge permissions for Administrators
- Forge Author → group Administrators
- Forge Committer → group Administrators
- Forge Server As Committer → group Administrators
Push permissions for Administrators
- Push → +force → group Administrators
- Push Merge Commit (or Push Merge) → group Administrators

Save changes.

2.1.2 Ensure capabilities exist for Zuul/service identity¶

In All-Projects capability section, ensure:

[capability] accessDatabase = group Administrators
[capability] administrateServer = group Administrators
[capability] streamEvents = group Service Users
[capability] priority = batch group Service Users

If “Service Users” group does not exist:

Go to People → Groups
Create Service Users
Add your Zuul/service account(s) to this group

2.2 Set `receive.maxBatchCommits = 0` via UI config editor¶

In All-Projects
Open Edit Config (or equivalent UI action for editing refs/meta/config)
Locate [receive]
Ensure it contains exactly:

[receive]
        maxBatchCommits = 0

Save/Publish (Do not keep duplicate maxBatchCommits lines.)

3) Clone the source repositories locally (automation host)¶

3.1 Clone Kolla and checkout the branch¶

cd "${INSTALL_DIR}/repos"
git clone https://opendev.org/openstack/kolla kolla
cd kolla
git checkout "${KOLLA_BRANCH}"

3.2 Clone the Zuul config reference repo¶

cd "${INSTALL_DIR}/repos"
git clone https://opengit.ir/devops/saqi/zuul-configs project-config

4) Stage Zuul tenant configuration in `project-config` (local changes)¶

This step defines the tenant/project/jobs/playbooks and points dependencies to Nexus (air-gap compliance). Important: Nexus is VM1 and Docker uses two ports (proxy vs hosted).

4.1 Create required directories¶

cd "${INSTALL_DIR}/repos/project-config"
mkdir -p zuul.d playbooks ci-trigger

4.2 Create/Update these files¶

You must have (at minimum):

zuul.d/projects.yaml
zuul.d/kolla-jobs.yaml
playbooks/kolla-build.yaml

Populate them with your tenant/project/job definitions:

Define kolla-build-base as an abstract job.
Define kolla-build-full job.
Ensure the project-config change pipeline triggers kolla-build-full.
Ensure the job/playbook:
- fetches the immutable Kolla repo (openstack/kolla on ${KOLLA_BRANCH}) as an input repository,
- uses Nexus-only endpoints:
  - Nexus HTTP: http://${VM1_IP}:${NEXUS_HTTP_PORT}
  - Docker pull proxy: ${VM1_IP}:${NEXUS_DOCKER_PROXY_PORT}
  - Docker push hosted: ${VM1_IP}:${NEXUS_DOCKER_HOSTED_PORT}

4.3 Commit locally¶

cd "${INSTALL_DIR}/repos/project-config"
git add -A
git commit -m "Configure Nexus-Powered Build Pipeline"

5) Push `project-config` into Gerrit (seed the config repo) — VM2¶

5.1 Add Gerrit remote for `openstack/project-config`¶

cd "${INSTALL_DIR}/repos/project-config"
git remote remove gerrit 2>/dev/null || true
git remote add gerrit "ssh://admin@${VM2_IP}:${GERRIT_SSH_PORT}/openstack/project-config"

5.2 Push initial content to the target branch¶

Push to the branch you selected in ${PROJECT_CONFIG_BRANCH}:

git push gerrit HEAD:refs/heads/${PROJECT_CONFIG_BRANCH} -o skip-validation

Notes:

-o skip-validation is used because first seed can happen before all validation jobs are in place.
Keep ${PROJECT_CONFIG_BRANCH} consistent with your tenant configuration expectations.

6) Seed the Kolla repository into Gerrit (one-time) — VM2¶

Kolla is seeded once and then treated as immutable.

6.1 Add Gerrit remote for `openstack/kolla`¶

cd "${INSTALL_DIR}/repos/kolla"
git remote remove gerrit 2>/dev/null || true
git remote add gerrit "ssh://admin@${VM2_IP}:${GERRIT_SSH_PORT}/openstack/kolla"

6.2 Push the checked-out branch to Gerrit’s target branch (one-time seed)¶

This sets Gerrit branch stable/2024.1 to match your checked-out content:

git push gerrit HEAD:refs/heads/stable/2024.1 -o skip-validation --force
git push gerrit --tags

6.3 Make `openstack/kolla` immutable (recommended enforcement) — Gerrit UI¶

Gerrit UI → Projects → openstack/kolla → Access
For refs/heads/* (or refs/heads/stable/*):
- Remove Push and Push (force) from broad groups
- Keep only read access for Zuul/service identities as needed
- Save

7) Trigger CI (create a Gerrit change on `project-config`) — VM2¶

7.1 Create or update a trigger file inside `project-config`¶

Use a dedicated path to keep it clean:

cd "${INSTALL_DIR}/repos/project-config"
date -u +"%Y-%m-%dT%H:%M:%SZ" >> ci-trigger/build_trigger.txt
git add ci-trigger/build_trigger.txt
git commit -m "Trigger kolla build $(date -u +%Y-%m-%d)"

7.2 Push for review to start the pipeline (project-config repo)¶

Push to refs/for on the project-config branch, not stable/2024.1:

git push gerrit HEAD:refs/for/${PROJECT_CONFIG_BRANCH}

This creates a Gerrit change in openstack/project-config which Zuul should pick up and run kolla-build-full.

8) Verify success (Gerrit + Zuul UI) — VM2¶

8.1 Verify in Gerrit¶

Open Gerrit UI on VM2
Go to Changes → Open
Open the new change in openstack/project-config
Confirm:
- The change exists on ${PROJECT_CONFIG_BRANCH}
- Zuul reporting (labels/comments) updates as the job runs

8.2 Verify in Zuul dashboard¶

Open: http://${VM2_IP}:9000
Locate the pipeline that should run for Gerrit changes (commonly check)
Confirm you see a build for:
- Project: openstack/project-config
- Job: kolla-build-full
- Drill into the build and confirm:
- The playbook runs
- It pulls only from Nexus/internal endpoints:
  - http://${VM1_IP}:${NEXUS_HTTP_PORT}
  - ${VM1_IP}:${NEXUS_DOCKER_PROXY_PORT} for pulls
  - ${VM1_IP}:${NEXUS_DOCKER_HOSTED_PORT} for pushes
    - The Kolla repo is used as an immutable input (no changes pushed to openstack/kolla)