Description of problem:

When we activate the on-cluster-build functionality in a pool with yum based RHEL nodes, the pool is degraded reporting this error:

  - lastTransitionTime: "2023-09-20T15:14:44Z"
    message: 'Node ip-10-0-57-169.us-east-2.compute.internal is reporting: "error
      running rpm-ostree --version: exec: \"rpm-ostree\": executable file not found
      in $PATH"'
    reason: 1 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded

Version-Release number of selected component (if applicable):

4.14.0-0.nightly-2023-09-15-233408

How reproducible:

Always

Steps to Reproduce:

1. Create a cluster and add a yum based RHEL node to the worker pool

(we used RHEL8)

2. Create the necessary resources to enable the OCB functionality. Pull and push secrets and the on-cluster-build-config configmap.

For example we can use this if we want to use the internal registry:

cat << EOF | oc create -f -
apiVersion: v1
data:
  baseImagePullSecretName: $(oc get secret -n openshift-config pull-secret -o json | jq "del(.metadata.namespace, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.uid, .metadata.name)" | jq '.metadata.name="pull-copy"' | oc -n openshift-machine-config-operator create -f - &> /dev/null; echo -n "pull-copy")
  finalImagePushSecretName: $(oc get -n openshift-machine-config-operator sa builder -ojsonpath='{.secrets[0].name}')
  finalImagePullspec: "image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image"
  imageBuilderType: ""
kind: ConfigMap
metadata:
  name: on-cluster-build-config
  namespace: openshift-machine-config-operator
EOF

The configuration doesn't matter as long as the OCB functionality can work.

3. Label the worker pool so that the OCB functionality is enabled

$ oc label mcp/worker machineconfiguration.openshift.io/layering-enabled=

Actual results:

The RHEL node shows this log:


I0920 15:14:42.852742    1979 daemon.go:760] Preflight config drift check successful (took 17.527225ms)
I0920 15:14:42.852763    1979 daemon.go:2150] Performing layered OS update
I0920 15:14:42.868723    1979 update.go:1970] Starting transition to "image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/tc-67566@sha256:24ea4b12acf93095732ba457fc3e8c7f1287b669f2aceec65a33a41f7e8ceb01"
I0920 15:14:42.871625    1979 update.go:1970] drain is already completed on this node
I0920 15:14:42.874305    1979 rpm-ostree.go:307] Running captured: rpm-ostree --version
E0920 15:14:42.874388    1979 writer.go:226] Marking Degraded due to: error running rpm-ostree --version: exec: "rpm-ostree": executable file not found in $PATH
I0920 15:15:37.570503    1979 daemon.go:670] Transitioned from state: Working -> Degraded
I0920 15:15:37.570529    1979 daemon.go:673] Transitioned from degraded/unreconcilable reason  -> error running rpm-ostree --version: exec: "rpm-ostree": executable file not found in $PATH
I0920 15:15:37.574942    1979 daemon.go:2300] Not booted into a CoreOS variant, ignoring target OSImageURL quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3128a8e42fb70ab6fc276f7005e3c0839795e4455823c8ff3eca9b1050798b9
I0920 15:15:37.591529    1979 daemon.go:760] Preflight config drift check successful (took 16.588912ms)
I0920 15:15:37.591549    1979 daemon.go:2150] Performing layered OS update
I0920 15:15:37.591562    1979 update.go:1970] Starting transition to "image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/tc-67566@sha256:24ea4b12acf93095732ba457fc3e8c7f1287b669f2aceec65a33a41f7e8ceb01"
I0920 15:15:37.594534    1979 update.go:1970] drain is already completed on this node
I0920 15:15:37.597261    1979 rpm-ostree.go:307] Running captured: rpm-ostree --version
E0920 15:15:37.597315    1979 writer.go:226] Marking Degraded due to: error running rpm-ostree --version: exec: "rpm-ostree": executable file not found in $PATH
qI0920 15:16:37.613270    1979 daemon.go:2300] Not booted into a CoreOS variant, ignoring target OSImageURL quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3128a8e42fb70ab6fc276f7005e3c0839795e4455823c8ff3eca9b1050798b9



And the worker pool is degraded with this error:

  - lastTransitionTime: "2023-09-20T15:14:44Z"
    message: 'Node ip-10-0-57-169.us-east-2.compute.internal is reporting: "error
      running rpm-ostree --version: exec: \"rpm-ostree\": executable file not found
      in $PATH"'
    reason: 1 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded

Expected results:

The pool should not be degraded.

Additional info:

• https://github.com/openshift/machine-config-operator/pull/4442

Add a knob to CNO to control the installation of the IPAMClaim CRD.

Requires a new OpenShift feature gate only allowing the feature to be installed in Dev / Tech preview.

• https://github.com/openshift/cluster-network-operator/pull/2431
• https://github.com/openshift/cluster-network-operator/pull/2408

Description of problem:

InsightsRecommendationActive firing, description link results in "Invalid parameter: redirect_uri" on sso.redhat.com.

Insights recommendation "OpenShift cluster with more or less than 3 control plane node replicas is not supported by Red Hat" with total risk "Moderate" was detected on the cluster. More information is available at https://console.redhat.com/openshift/insights/advisor/clusters/<UID>?first=ccx_rules_ocp.external.rules.control_plane_replicas|CONTROL_PLANE_NODE_REPLICAS.

Version-Release number of selected component (if applicable):

4.15.14

How reproducible:

unknown

Steps to Reproduce:

1. Install 4.15.14 on a cluster that triggers this alert
2. Log out of Red Hat SSO
3. Clink link in alert description

Actual results:

"Invalid parameter: redirect_uri" on sso.redhat.com

Expected results:

Link successfully navigates through SSO

Additional info:

• https://github.com/openshift/insights-operator/pull/952

Description of problem:

We have a test test_cluster_base_domain_obfuscation that checks that when we set the insights-config configmap with the obfuscation parameter set to "networking", we expect the archive to not have instance of the api_url or the base_hostname of the cluster.
This is currently not happening in hypershift hosted clusters.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:

    1. Run test_cluster_base_domain_obfuscation
    Or
    1. Create insights_config configmap in the openshift-insights namespace, with 
dataReporting: 
   obfuscation: Networking
    2. wait until the obfuscation creation table exists
    3. Download the archive
    4. Check every path in the archive and search for instances of the api_url or the base_hostname (easier to do with automation than manually)
    

Actual results:

Instances are found. 

Expected results:

No instances are found since they've all been obfuscated.

Additional info:

• https://github.com/openshift/insights-operator/pull/947
• https://github.com/openshift/insights-operator/pull/936

 Description of problem:

Insights operator should replaces %s in https://console.redhat.com/api/gathering/v2/%s/gathering_rules error messages like the failed-to-bootstrap:

$ jq -r .content osd-ccs-gcp-ad-install.log | sed 's/\\n/\n/g' | grep 'Cluster operator insights'
time="2024-09-05T08:12:51Z" level=info msg="Cluster operator insights ClusterTransferAvailable is False with Unauthorized: failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%REDACTED%27+and+status+is+%27accepted%27 returned HTTP 401: REDACTED"
time="2024-09-05T08:12:51Z" level=info msg="Cluster operator insights Disabled is False with AsExpected: "
time="2024-09-05T08:12:51Z" level=info msg="Cluster operator insights RemoteConfigurationAvailable is False with HttpStatus401: received HTTP 401 Unauthorized from https://console.redhat.com/api/gathering/v2/%s/gathering_rules"
time="2024-09-05T08:12:51Z" level=info msg="Cluster operator insights RemoteConfigurationValid is Unknown with NoValidationYet: "
time="2024-09-05T08:12:51Z" level=info msg="Cluster operator insights SCAAvailable is False with Unauthorized: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 401: REDACTED
level=info msg=Cluster operator insights ClusterTransferAvailable is False with Unauthorized: failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27REDACTED%27+and+status+is+%27accepted%27 returned HTTP 401: REDACTED
level=info msg=Cluster operator insights Disabled is False with AsExpected: 
level=info msg=Cluster operator insights RemoteConfigurationAvailable is False with HttpStatus401: received HTTP 401 Unauthorized from https://console.redhat.com/api/gathering/v2/%s/gathering_rules
level=info msg=Cluster operator insights RemoteConfigurationValid is Unknown with NoValidationYet: 
level=info msg=Cluster operator insights SCAAvailable is False with Unauthorized: Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 401: REDACTED
level=info msg=Cluster operator insights UploadDegraded is True with NotAuthorized: Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: {\"errors\":[{\"meta\":{\"response_by\":\"gateway\"},\"detail\":\"UHC services authentication failed\",\"status\":401}]}

Version-Release number of selected component

Seen in 4.17 RCs. Also in this comment.

How reproducible

Unknown

Steps to Reproduce:

Unknown.

Actual results:

ClusterOperator conditions talking about https://console.redhat.com/api/gathering/v2/%s/gathering_rules

Expected results

URIs we expose in customer-oriented messaging to not have %s placeholders.

Additional detail

Seems like the template is coming in as conditionalGathererEndpoint here. Seems like insights-operator#964 introduced the %s, but I'm not finding the logic that's supposed to populate that placeholder.

• https://github.com/openshift/insights-operator/pull/998

Rapid recommendations enhancement defines this built-in configuration when the operator cannot reach the remote endpoint.

The issue is that the built-in configuration (though currently empty) is no taken into account - i.e the data requested in the built-configuration is not gathered.

• https://github.com/openshift/insights-operator/pull/1011

INSIGHTOCP-1557 is a rule to check for any custom Prometheus instances that may impact the management of corresponding resources.

Resource to gather:  Prometheus and Alertmanager in all namespaces

apiVersion: monitoring.coreos.com/v1
kind: Prometheus

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager

Backport:  OCP 4.12.z; 4.13.z; 4.14.z; 4.15.z

Additional info:
1) Get the Prometheus and Alertmanager in all namespaces

$ oc get prometheus -A 
NAMESPACE              NAME                             VERSION   DESIRED   READY   RECONCILED   AVAILABLE   AGE
openshift-monitoring   k8s                              2.39.1    2         1       True         Degraded    712d
test                   custom-prometheus                          1         0       True         False       25d

$ oc get alertmanager -A 
NAMESPACE              NAME                             VERSION   DESIRED   READY   RECONCILED   AVAILABLE   AGE
openshift-monitoring   main                             2.39.1    2         1       True         Degraded    712d
test                   custom-alertmanager                        1         0       True         False       25d
 

• https://github.com/openshift/insights-operator/pull/941

Business required:

We had a recommendation to check the certificate of the default ingress controller expiration after it has expired. From the referenced KCS, it seems that many customers(hundreds) hit this issue. So, Oscar Arribas Arribas suggests that if we can have a recommendation to alert customers before certificate expiration. 

Gathering method:

1. Gather all the ingresscontroller objects(we already gathered the default ingresscontroller) with commands: 
oc get ingresscontrollers -n openshift-ingress-operator
2. Gather operator auto-generated certificate's validate dates with commands:

$ oc get ingresscontrollers -n openshift-ingress-operator -o yaml | grep -A1 defaultCertificate
#### empty output here when certificate created by the operator

$ oc get secret router-ca -n openshift-ingress-operator -o yaml | grep crt | awk '{print $2}' | base64 -d | openssl x509 -noout -dates
notBefore=Dec 28 00:00:00 2022 GMT
notAfter=Jan 22 23:59:59 2024 GMT

$ oc get secret router-certs-default -n openshift-ingress -o yaml | grep crt | awk '{print $2}' | base64 -d | openssl x509 -noout -dates
notBefore=Dec 28 00:00:00 2022 GMT
notAfter=Jan 22 23:59:59 2024 GMT

3. Gather custom certificates' validate dates with commands:

$ oc get ingresscontrollers -n openshift-ingress-operator -o yaml | grep -A1 defaultCertificate
    defaultCertificate:
      name: [custom-cert-secret-1]

#### for each [custom-cert-secret] above
$ oc get secret [custom-cert-secret-1] -n openshift-ingress -o yaml | grep crt | awk '{print $2}' | base64 -d | openssl x509 -noout -dates
notBefore=Dec 28 00:00:00 2022 GMT
notAfter=Jan 22 23:59:59 2024 GMT
 

Other Information:

An RFE to create a cluster alert is under reveiwing: https://issues.redhat.com/browse/RFE-4269

• https://github.com/openshift/insights-operator/pull/945

• https://github.com/openshift/installer/pull/8610

• https://github.com/openshift/machine-config-operator/pull/4292

For tech preview we only allow files/units/etc. There are two potential use cases for directories:

1. namespaced image policy objects
2. hostname based networking policies

Which would allow the MCO to generally allow anything under a path to apply the policy. We should adapt the API and MCO logic to also allow paths.

• https://github.com/openshift/machine-config-operator/pull/4472

Add e2e tests that fit the guidelines mentioned in the openshift/api docs https://github.com/openshift/api/blob/master/README.md#defining-featuregate-e2e-tests 

• https://github.com/openshift/machine-config-operator/pull/4365

• https://github.com/openshift/installer/pull/8565

• https://github.com/openshift/installer/pull/8471

• https://github.com/openshift/installer/pull/8441

1. Design and implement an API at the NodePool (platform.kubevirt) that will allow exposing GPU passthrough or vGPU slicing from the infra cluster to the guest cluster.

2. Implement logic that sets up the GPU resources to be available to the guest cluster's workloads (by using nvidia-gpu-operator?)

• https://github.com/openshift/hypershift/pull/4101

• https://github.com/openshift/api/pull/1902
• https://github.com/openshift/cluster-storage-operator/pull/478
• https://github.com/openshift/cluster-storage-operator/pull/475
• https://github.com/openshift/origin/pull/28846

• https://github.com/openshift/origin/pull/28684

User Story

As a developer I want to add the field "CapacityReservationGroupID" to "AzureMachineProviderSpec" in openshift/api so that Azure capacity reservation can be supported.

Background

CFE-1036 adds the support of Capacity Reservation in upstream CAPZ (PR). The same support is needed to be added downstream also. Please refer the upstream PR for adding support downstream.

Slack discussion regarding the same: https://redhat-internal.slack.com/archives/CBZHF4DHC/p1713249202780119?thread_ts=1712582367.529309&cid=CBZHF4DHC_

Steps

• Add the field "CapacityReservationGroupID" to "AzureMachineProviderSpec"
• The new field should be immutable. Add validations for the same.
• Add tests to validate the immutability.

Stakeholders

• Cluster Infra
• CFE

Definition of Done

• The PR should be reviewed and approved.

• Docs

• Add appropriate godoc for the field explaining its purpose

• Testing

• Add tests to validate the immutability.

• https://github.com/openshift/api/pull/1866

Update the vendor to update in cluster-control-plane-machine-set-operator repository for capacity reservation Changes. 

• https://github.com/openshift/cluster-control-plane-machine-set-operator/pull/313

User Story

As a developer I want to add support of capacity reservation group in openshift/machine-api-provider-azure so that azure VMs can be associated to a capacity reservation group during the VM creation.

Background

CFE-1036 adds the support of Capacity Reservation in upstream CAPZ (PR). The same support is needed to be added downstream also. Please refer the upstream PR for adding support downstream.

Steps

• Import the latest API changes from openshift/api to get the new field "CapacityReservationGroupID" into openshift/machine-api-provider-azure.

• If a value is assigned to the field then use for associated a VM to the capacity reservation group during VM creation.

Stakeholders

• Cluster Infra
• CFE

Definition of Done

• The PR should be reviewed and approved.

• Testing

• Add unit tests to validate the implementation.

• https://github.com/openshift/machine-api-provider-azure/pull/107

As a developer I want to add the webhook validation for the "CapacityReservationGroupID" field of "AzureMachineProviderSpec" in openshift/machine-api-operator so that Azure capacity reservation can be supported.

Background

CFE-1036 adds the support of Capacity Reservation in upstream CAPZ (PR). The same support is needed to be added downstream also. Please refer the upstream PR for adding support downstream.

Slack discussion regarding the same: https://redhat-internal.slack.com/archives/CBZHF4DHC/p1713249202780119?thread_ts=1712582367.529309&cid=CBZHF4DHC_

Steps

• Add the validation for "CapacityReservationGroupID" to "AzureMachineProviderSpec"
• Add tests to validate.

Stakeholders

• Cluster Infra
• CFE

• https://github.com/openshift/machine-api-operator/pull/1250
• https://github.com/openshift/machine-api-operator/pull/1234

Develop hypershift e2e test that exercises attaching a secondary network to a HCP KubeVirt nodepool

• https://github.com/openshift/hypershift/pull/3902

This would involve updating the AMIs stored within the AWSProviderConfig(encapsulated within a MachineSet. The "updated" AMI values should be available in the golden configmap.

• https://github.com/openshift/machine-config-operator/pull/4492

This involves create a new feature gate for AWS boot image updates in openshift/api

• https://github.com/openshift/api/pull/1945

Done when:

PR to API to remove featuregate

Alert docs team

• https://github.com/openshift/api/pull/1975
• https://github.com/openshift/machine-config-operator/pull/4496

Currently errors are propagated via a prometheus alert. Before GA, we will need to make sure that we are placing a condition on the configuration object in addition to the current Prometheus mechanism. This will be done by the MSBIC, but it should be mindful as to not stomp on the operator, which updates the MachineConfiguration Status as well.  

• https://github.com/openshift/machine-config-operator/pull/4390

We currently have a kube version string used in the call to setup envtest. We should either git rid of this reference and grab it from elsewhere or update it with every kube bump we do.

In addition, the setup call now requires an additional argument to factor for openshift/api's kubebuilder divergence. So the value being used here may not be valid for every kube bump as the archive is not generated for every kube version. (Doing a bootstrap test run should be able to sus this out, if it doesn't error with the new version you should be ok)

• https://github.com/openshift/machine-config-operator/pull/4399

This was a temporary change and should be reverted. This was done because setup-envtest did a go bump that the MCO will probably not do in time for 4.16.

PR that pinned the tag: https://github.com/openshift/machine-config-operator/pull/4280

Slack thread: https://redhat-internal.slack.com/archives/GH7G2MANS/p1711372261617039?thread_ts=1711372068.123039&cid=GH7G2MANS

• https://github.com/openshift/machine-config-operator/pull/4458

Please review the following PR: https://github.com/openshift/machine-config-operator/pull/4380

The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.

Differences in upstream and downstream builds impact the fidelity of your CI signal.

If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.

Closing this issue without addressing the difference will cause the issue to
be reopened automatically.

• https://github.com/openshift/machine-config-operator/pull/4399

Template:

Networking Definition of Planned

Epic Template descriptions and documentation

Epic Goal

make sure we deliver a 1.30 kube-proxy standalone image

Why is this important?

Planning Done Checklist

The following items must be completed on the Epic prior to moving the Epic from Planning to the ToDo status

• Priority+ is set by engineering
• Epic must be Linked to a +Parent Feature
• Target version+ must be set
• Assignee+ must be set
• (Enhancement Proposal is Implementable
• (No outstanding questions about major work breakdown
• (Are all Stakeholders known? Have they all been notified about this item?
• Does this epic affect SD? {}Have they been notified{+}? (View plan definition for current suggested assignee)
  1. Please use the “Discussion Needed: Service Delivery Architecture Overview” checkbox to facilitate the conversation with SD Architects. The SD architecture team monitors this checkbox which should then spur the conversation between SD and epic stakeholders. Once the conversation has occurred, uncheck the “Discussion Needed: Service Delivery Architecture Overview” checkbox and record the outcome of the discussion in the epic description here.
  2. The guidance here is that unless it is very clear that your epic doesn’t have any managed services impact, default to use the Discussion Needed checkbox to facilitate that conversation.

Additional information on each of the above items can be found here: Networking Definition of Planned

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement
  details and documents.

...

Dependencies (internal and external)

1.

...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

• https://github.com/openshift/sdn/pull/630

Template:

Networking Definition of Planned

Epic Template descriptions and documentation

Epic Goal

Bump kube to 1.30 in CNCC

Why is this important?

Planning Done Checklist

The following items must be completed on the Epic prior to moving the Epic from Planning to the ToDo status

• Priority+ is set by engineering
• Epic must be Linked to a +Parent Feature
• Target version+ must be set
• Assignee+ must be set
• (Enhancement Proposal is Implementable
• (No outstanding questions about major work breakdown
• (Are all Stakeholders known? Have they all been notified about this item?
• Does this epic affect SD? {}Have they been notified{+}? (View plan definition for current suggested assignee)
  1. Please use the “Discussion Needed: Service Delivery Architecture Overview” checkbox to facilitate the conversation with SD Architects. The SD architecture team monitors this checkbox which should then spur the conversation between SD and epic stakeholders. Once the conversation has occurred, uncheck the “Discussion Needed: Service Delivery Architecture Overview” checkbox and record the outcome of the discussion in the epic description here.
  2. The guidance here is that unless it is very clear that your epic doesn’t have any managed services impact, default to use the Discussion Needed checkbox to facilitate that conversation.

Additional information on each of the above items can be found here: Networking Definition of Planned

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement
  details and documents.

...

Dependencies (internal and external)

1.

...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

• https://github.com/openshift/cloud-network-config-controller/pull/150

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/kubernetes-autoscaler/pull/306

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-aws/pull/62

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-provider-gcp/pull/84

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-ibm/pull/68

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-machine-approver/pull/235

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-provider-azure/pull/113

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-vsphere/pull/62

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-control-plane-machine-set-operator/pull/295

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-provider-ibmcloud/pull/38

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-operator/pull/1255

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-cloud-controller-manager-operator/pull/356

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-provider-aws/pull/104

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-autoscaler-operator/pull/327

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-azure/pull/113

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-nutanix/pull/31

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cloud-provider-gcp/pull/57

To align with the 4.17 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/machine-api-provider-nutanix/pull/75

 upgrade all OpenShift and Kubernetes components that cloud-credential-operator uses to v1.30 which keeps it on par with rest of the OpenShift components and the underlying cluster version.

• https://github.com/openshift/cloud-credential-operator/pull/725

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api-provider-gcp/pull/228

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api-provider-azure/pull/305

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api-provider-aws/pull/512

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-capi-operator/pull/176

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api/pull/210

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api-provider-ibmcloud/pull/82

To align with the 4.17 release, dependencies need to be updated to 1.29. This should be done by rebasing/updating as appropriate for the repository

• https://github.com/openshift/cluster-api-provider-vsphere/pull/44

1. [Vu Dinh] cluster-kube-controller-manager operator: https://github.com/openshift/cluster-kube-controller-manager-operator/pull/803
2. [Jan] cluster-policy-controller: https://github.com/openshift/cluster-policy-controller/pull/151
3. [Vu Dinh] cluster-kube-scheduler operator: https://github.com/openshift/cluster-kube-scheduler-operator/pull/540
4. [Jan] secondary-scheduler-operator: https://github.com/openshift/secondary-scheduler-operator/pull/142
5. [Jan] cluster-capacity: https://github.com/openshift/cluster-capacity/pull/93
6.  [Jan] run-once-duration-override-operator: https://github.com/openshift/run-once-duration-override-operator/pull/57
7.  [Jan] run-once-duration-override: https://github.com/openshift/run-once-duration-override/pull/35

Other teams:

1. [Jan] route-controller-manager: https://github.com/openshift/route-controller-manager/pull/44

If needed this card can be broken down into more cards with sublists, each card assigned to a different assignee.

• https://github.com/openshift/oc/pull/1789
• https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/352
• https://github.com/openshift/cluster-policy-controller/pull/151
• https://github.com/openshift/openshift-controller-manager/pull/313
• https://github.com/openshift/route-controller-manager/pull/44

User Story:

As a (user persona), I want to be able to:

• Use a pre-existing IAM profile in the install-config.yaml
• Use a user/role which doesn't have the permissions needed for instance profile creation.

so that I can achieve

• A cluster created with an existing IAM profile.

Acceptance Criteria:

Description of criteria:

• Upstream documentation
• Point 1
• Point 2
• Point 3

(optional) Out of Scope:

Detail about what is specifically not being delivered in the story

Engineering Details:

• (optional) https://github/com/link.to.enhancement/
• (optional) https://issues.redhat.com/link.to.spike
• Engineering detail 1
• Engineering detail 2

This requires/does not require a design proposal.
This requires/does not require a feature gate.

• https://github.com/openshift/installer/pull/8689

Once https://issues.redhat.com/browse/ETCD-473 is done this story will track the work required to move the "operator/v1 etcd spec.hardwareSpeed" field from behind the feature gate to GA.

• https://github.com/openshift/api/pull/1844

• https://github.com/openshift/cluster-etcd-operator/pull/1264

Feature Overview (aka. Goal Summary)  

Graduate the etcd tuning profiles feature delivered in https://issues.redhat.com/browse/ETCD-456 to GA

Goals (aka. expected user outcomes)

Remove the feature gate flag and ,ake the feature accessible to all customers 

Requirements (aka. Acceptance Criteria):

Requires fixes to apiserver to handle etcd client retries correctly

Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed.  Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

<your text here>

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

<your text here>

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

<your text here>

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

<your text here>

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

<your text here>

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.

<your text here>

Interoperability Considerations

Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

<your text here>

• https://github.com/openshift/origin/pull/28836
• https://github.com/openshift/origin/pull/28817

seeing various MCO issue across different platforms and repos on techpreview

• https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_clus[…]der-aws-master-e2e-aws-ovn-techpreview/1818614625273384960
• https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_clus[…]der-aws-master-e2e-aws-ovn-techpreview/1818592543001022464
• https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_clus[…]perator-main-e2e-azure-ovn-techpreview/1818562032329297920
• https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_clus[…]perator-main-e2e-azure-ovn-techpreview/1818592686244892672

 
@David Joshy found that it might be https://github.com/openshift/cluster-update-keys/pull/58
 
@jerzhang found diffs in sigstore-registries and policy.json. Which may be coming from this manifest. Is this available during bootstrap?

• https://github.com/openshift/installer/pull/8813
• https://github.com/openshift/cluster-update-keys/pull/59
• https://github.com/openshift/cluster-version-operator/pull/1078
• https://github.com/openshift/cluster-version-operator/pull/1076

As described in the OTA-1294 enhancement. The cluster-update-keys repository isn't actually managed by the OTA team, but I expect it will be me opening the pull, and there isn't a dedicated Jira project covering cluster-update-keys, so I'm creating this ticket under the OTA Epic just because I can't think of a better place to put it.

• https://github.com/openshift/cluster-update-keys/pull/58

Description of problem

As Miloslav Trmač reported upstream, when a ClusterImagePolicy is set on a scope to accept sigstore signatures, the underlying registry needs to be configured with use-sigstore-attachments: true. The current code:

 func generateSigstoreRegistriesdConfig(clusterScopePolicies map[string]signature.PolicyRequirements) ([]byte, error) { 

does do that for the configured scope; but the use-sigstore-attachments option applies not to the "logical name", but to each underlying mirror individually.

I.e. the option needs to be on every mirror of the scope. Without that, if the image is found on one of such mirrors, the c/image code will not be looking for signatures on the mirror, and policy enforcement is likely to fail.

Version-Release number of selected component

Seen in 4.17.0-0.nightly-2024-06-25-162526, but likely all releases which implement ClusterImagePolicy so far, because this is unlikely to be a regression.

How reproducible

Every time.

Steps to Reproduce

Apply the ClusterImagePolicy suggested in OTA-1294's enhancements#1633:

$ cat <<EOF >policy.yaml
apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy
metadata:
  name: openshift
  annotations:
    kubernetes.io/description: Require Red Hat signatures for quay.io/openshift-release-dev/ocp-release container images.
    exclude.release.openshift.io/internal-openshift-hosted: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    release.openshift.io/feature-set: TechPreviewNoUpgrade
spec:
  scopes:
  - quay.io/openshift-release-dev/ocp-release
  policy:
    rootOfTrust:
      policyType: PublicKey
      publicKey:
        keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUEzQzJlVGdJQUo3aGxveDdDSCtIcE1qdDEvbW5lYXcyejlHdE9NUmlSaEgya09ZalRadGVLSEtnWUJHcGViajRBcUpWYnVRaWJYZTZKYVFHQUFER0VOZXozTldsVXpCby9FUUEwaXJDRnN6dlhVbTE2cWFZMG8zOUZpbWpsVVovaG1VNVljSHhxMzR2OTh4bGtRbUVxekowR0VJMzNtWTFMbWFEM3ZhYmd3WWcwb3lzSTk1Z1V1Tk81TmdZUHA4WDREaFNoSmtyVEl5dDJLTEhYWW5BMExzOEJlbG9PWVJlTnJhZmxKRHNzaE5VRFh4MDJhQVZSd2RjMXhJUDArRTlZaTY1ZE4zKzlReVhEOUZ6K3MrTDNjZzh3bDdZd3ZZb1Z2NDhndklmTHlJbjJUaHY2Uzk2R0V6bXBoazRjWDBIeitnUkdocWpyajU4U2hSZzlteitrcnVhR0VuVGcyS3BWR0gzd3I4Z09UdUFZMmtqMnY1YWhnZWt4V1pFN05vazNiNTBKNEpnYXlpSnVSL2R0cmFQMWVMMjlFMG52akdsMXptUXlGNlZnNGdIVXYwaktrcnJ2QUQ4c1dNY2NBS00zbXNXU01uRVpOTnljTTRITlNobGNReG5xU1lFSXR6MGZjajdYamtKbnAxME51Z2lVWlNLeVNXOHc0R3hTaFNraGRGbzByRDlkVElRZkJoeS91ZHRQWUkrK2VoK243QTV2UVV4Wk5BTmZqOUhRbC81Z3lFbFV6TTJOekJ2RHpHellSNVdVZEVEaDlJQ1I4ZlFpMVIxNUtZU0h2Tlc3RW5ucDdZT2d5dmtoSkdwRU5PQkF3c1pLMUhhMkJZYXZMMk05NDJzSkhxOUQ1eEsrZyszQU81eXp6V2NqaUFDMWU4RURPcUVpY01Ud05LOENBd0VBQVE9PQotLS0tLUVORCBQVUJMSUMgS0VZLS0tLS0K
EOF
$ oc apply -f policy.yaml

Set up an ImageContentSourcePolicy such as the ones Cluster Bot jobs have by default:

cat <<EOF >mirror.yaml
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: pull-through-mirror
spec:
  repositoryDigestMirrors:
  - mirrors:
    - quayio-pull-through-cache-us-west-2-ci.apps.ci.l2s4.p1.openshiftapps.com
    source: quay.io
EOF
$ oc apply -f mirror.yaml

Set CRI-O debug logs, following these docs:

$ cat <<EOF >custom-loglevel.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: custom-loglevel
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ''
  containerRuntimeConfig:
    logLevel: debug
EOF
$ oc create -f custom-loglevel.yaml

Wait for that to roll out, as described in docs:

$ oc get machineconfigpool master

Launch a Sigstore-signed quay.io/openshift-release-dev/ocp-release image, by asking the cluster to update to 4.16.1:

$ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a

Check the debug CRI-O logs:

$ oc adm node-logs --role=master -u crio | grep -i1 sigstore | tail -n5

Actual results

Not looking for sigstore attachments: disabled by configuration entries like:

$ oc adm node-logs --role=master -u crio' | grep -i1 sigstore | tail -n5
--
Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317169116Z" level=debug msg=" Using transport \"docker\" specific policy section quay.io/openshift-release-dev/ocp-release" file="signature/policy_eval.go:150"
Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317207897Z" level=debug msg="Reading /var/lib/containers/sigstore/openshift-release-dev/ocp-release@sha256=c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a/signature-1" file="docker/docker_image_src.go:479"
Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317240227Z" level=debug msg="Not looking for sigstore attachments: disabled by configuration" file="docker/docker_image_src.go:556"
Jun 28 19:06:34.317335 ip-10-0-43-59 crio[2154]: time="2024-06-28 19:06:34.317277208Z" level=debug msg="Requirement 0: denied, done" file="signature/policy_eval.go:285"

Expected results

Something about "we're going to look for Sigstore signatures on quayio-pull-through-cache-us-west-2-ci.apps.ci.l2s4.p1.openshiftapps.com, since that's where we found the quay.io/openshift-release-dev/ocp-release@sha256:c17d4489c1b283ee71c76dda559e66a546e16b208a57eb156ef38fb30098903a image". At this point, it doesn't matter whether the retrieved signature is accepted or not, just that a signature lookup is attempted.

• https://github.com/openshift/machine-config-operator/pull/4449

Remove the openshift/api ClusterImagePolicy document about the restriction of scope field on OpenShift release image repository and install the updated manifest in MCO.

• https://github.com/openshift/api/pull/1927

enhancements#1633 is still in flight, but there seams to be some consensus around its API Extensions proposal to drop the following Godocs from ClusterImagePolicy and ImagePolicy:

// Please be aware that the scopes should not be nested under the repositories of OpenShift Container Platform images.
// If configured, the policies for OpenShift Container Platform repositories will not be in effect.

The backing implementation will also be removed. This guard was initially intended to protect cluster adminstrators from breaking their clusters by configuring policies that blocked critical images. And before Red Hat was publishing signatures for quay.io/openshift-release-dev/ocp-release releases, that made sense. But now that Red Hat is almost (OTA-1267) publishing Sigstore signatures for those release images, it makes sense to allow policies covering those images. And even if a cluster administrator creates a policy that blocks critical image pulls, PodDisriptionBudgets should keep the Kubernetes API server and related core workloads running for long enough for the cluster administrator to use the Kube API to remove or adjust the problematic policy.

There's a possibility that we replace the guard with some kind of pre-rollout validation, but that doesn't have to be part of the initial work.

We want this guard in place to unblock testing of enhancements#1633's proposed ClusterImagePolicy, so we can decide if it works as expected, or if it needs tweaks before being committed as a cluster-update-keys manifest. And we want that testing to establish confidence in the approach before we start in on the installer's internalTestingImagePolicy and installer-caller work.

• https://github.com/openshift/machine-config-operator/pull/4401

Update the manifest through MCO script:https://github.com/openshift/machine-config-operator/blob/master/hack/crds-sync.sh 

Make sure the document of ClusterImagePolicy and featuregate in the MCO manifest is up to date with API generated CRD.

• https://github.com/openshift/machine-config-operator/pull/4480

These seem like dup's, and we should remove ImagePolicy and consolidate around SigstoreImageVerification for clarity.

• https://github.com/openshift/api/pull/1964

User Story:

As a service provider, I want to be able to:

• Configure priority and fairness settings per HostedCluster size and force these settings to be applied on the resulting hosted cluster.

so that I can achieve

• Prevent user of hosted cluster from bringing down the HostedCluster kube apiserver with their workload.

Acceptance Criteria:

Description of criteria:

• HostedCluster priority and fairness settings should be configurable per cluster size in the ClusterSizingConfiguration CR
• Any changes in priority and fairness inside the HostedCluster should be prevented and overridden by whatever is configured on the provider side.
• With the proper settings, heavy use of the API from user workloads should not result in the KAS pod getting OOMKilled due to lack of resources.

This does not require a design proposal.
This does not require a feature gate.

• https://github.com/openshift/hypershift/pull/4488

• https://github.com/openshift/hypershift/pull/4075

Description of problem:

    In some instances the request serving node autoscaler helper fails to kick off a reconcile when there are pending pods.

Version-Release number of selected component (if applicable):

    4.17

How reproducible:

    sometimes

Steps to Reproduce:

    1. Setup a mgmt cluster with size tagging
    2. Create a hosted cluster and get it scheduled
    3. Scale down the machinesets of the request serving nodes for the hosted cluster.
    4. Wait for the hosted cluster to recover

Actual results:

    A placeholder pod is created for the missing node of the hosted cluster, but does not cause a scale up of the corresponding machineset.

Expected results:

    The hosted cluster recovers by scaling up the corresponding machinesets.

Additional info:

• https://github.com/openshift/hypershift/pull/4074

Description of problem:

    On Running PerfScale test on staging sectors, the script creates 1 HC per minute to load up a Management Cluster to its maximum capacity(64 HC). There were 2 clusters trying to use same serving node pair and got in to a deadlock

# oc get nodes -l osd-fleet-manager.openshift.io/paired-nodes=serving-12 
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-4-127.us-east-2.compute.internal    Ready    worker   34m   v1.27.11+d8e449a
ip-10-0-84-196.us-east-2.compute.internal   Ready    worker   34m   v1.27.11+d8e449a

Each node got assigned to 2 different cluster
# oc get nodes -l hypershift.openshift.io/cluster=ocm-staging-2bcimf68iudmq2pctkj11os571ahutr1-mukri-dysn-0017 
NAME                                       STATUS   ROLES    AGE   VERSION
ip-10-0-4-127.us-east-2.compute.internal   Ready    worker   33m   v1.27.11+d8e449a

# oc get nodes -l hypershift.openshift.io/cluster=ocm-staging-2bcind28698qgrugl87laqerhhb0u2c2-mukri-dysn-0019
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-84-196.us-east-2.compute.internal   Ready    worker   36m   v1.27.11+d8e449a

Taints were missing on those nodes, so metric-forwarder pod from other hostedclusters got scheduled on serving nodes.

# oc get pods -A -o wide | grep ip-10-0-84-196.us-east-2.compute.internal 
ocm-staging-2bcind28698qgrugl87laqerhhb0u2c2-mukri-dysn-0019   kube-apiserver-86d4866654-brfkb                                           5/5     Running                  0                40m     10.128.48.6      ip-10-0-84-196.us-east-2.compute.internal    <none>           <none>
ocm-staging-2bcins06s2acm59sp85g4qd43g9hq42g-mukri-dysn-0020   metrics-forwarder-6d787d5874-69bv7                                        1/1     Running                  0                40m     10.128.48.7      ip-10-0-84-196.us-east-2.compute.internal    <none>           <none>

and few more

Version-Release number of selected component (if applicable):

MC Version 4.14.17
HC version 4.15.10
HO Version quay.io/acm-d/rhtap-hypershift-operator:c698d1da049c86c2cfb4c0f61ca052a0654e2fb9

How reproducible:

Not Always

Steps to Reproduce:

    1. Create an MC with prod config (non-dynamic serving node)
    2. Create HCs on them at 1 HCP per minutes
    3. Cluster stuck at installing for more than 30 minutes
    

Actual results:

Only one replica of Kube-apiserver pods were up and the second stuck at pending state, upon checking the machine API has scaled both nodes in that machineset(serving-12) but only one got assigned(labelled). Further checking that node from one zone(serving-12a) was assigned to a specific hosted cluster(0017), and the other one(serving-12b) was assigned to a different hosted cluster(0019)

Expected results:

Kube-apiserver replica should be on the same machinesets and those node should be tainted.

Additional info: Slack

• https://github.com/openshift/hypershift/pull/4103

• https://github.com/openshift/hypershift/pull/4034

Description of problem:

    When creating a cluster with OCP < 4.16 and nodepools with a number of workers larger than smallest size, the placeholder pods for the hosted cluster get continually recycled and the hosted cluster is never scheduled.

Version-Release number of selected component (if applicable):

    4.17.0

How reproducible:

    Always

Steps to Reproduce:

0. Install hypershift with size tagging enabled    
1. Create a hosted cluster with nodepools large enough to not be able to use the default placeholder pods (or simply configure no placeholder pods)
    

Actual results:

    The cluster never schedules

Expected results:

    The cluster is scheduled and the control plane can come up

Additional info:

• https://github.com/openshift/hypershift/pull/4132

• https://github.com/openshift/machine-config-operator/pull/4535

Currently the operator catalog image is always being deleted in the delete feature. It can leads to catalogs broken in the clusters.

It is necessary to change the implementation to skip the deletion of the operator catalog image according with the following conditions:

• If in the DeleteImageSetConfiguration were specified packages (operators) under the operator catalog, so it means the customer wants to delete only the specified operators and the operator catalog image should not be deleted.

• If only the operator catalog was specified, it means all the operators under the specified operator catalog should be deleted AND also the operator catalog image.

• https://github.com/openshift/oc-mirror/pull/894

It is necessary to create a data structure that contains in which operator each related image is encountered, it is possible to get this information from the current loop already present in the collection phase.

Having this data structure will allow to tell customers which operators failed based on a image that failed during the mirroring.

For example: related image X failed during the mirroring, this related image is present in the operators a, b and c, so in the mirroring errors file already being generated is going to include the name of the operators instead of the name of the related image only.

• for a fail safe on a related image, stop mirroring the whole group of related images for that bundle
• AND especially, defer mirroring bundle images till all related images are mirrored. Otherwise, we'll be in the situation you described yesterday: the operator starts upgrading when its related images are missing
•  

• https://github.com/openshift/oc-mirror/pull/895

oc-mirror needs to identify when an OCP on EUS version is trying to upgrade and it is skiping one version (For example going from 4.12.14 to 4.14.18)

When this condition is identified, oc-mirror needs to show a warning in the log saying to customer to use the cincinnati web tool (upgrade tool) to identify versions required in the middle.

• https://github.com/openshift/oc-mirror/pull/874

This warning message is displayed by client-go library when running in cluster. Code: https://github.com/kubernetes/client-go/blob/e4e31fd32c91e1584fd774f58b2d39f135d23571/tools/clientcmd/client_config.go#L659

This message should be disabled by either passing in the kubconfig file correctly or building the config in a different way.

• https://github.com/openshift/installer/pull/8715

By default (and if available) the same pub ssh key used for the installation is reused to add a new node.
It could be useful to allow the user to (optionally) specify a different key in the nodes-config.yaml configuration file

• https://github.com/openshift/installer/pull/8779

Allow monitoring simultaneously more than one node.

It may be necessary to coalesce the different assisted-service pre-flight validations output accordingly

• https://github.com/openshift/installer/pull/8783
• https://github.com/openshift/installer/pull/8507

User Story:

As a cluster admin, I want to be able to:

• add a worker node to the existing cluster consuming the multi-arch payload having a different architecture from the controlplane one

Open questions:

• What are the effective use cases for an hybrid cluster (nodes with different archs)?

• https://github.com/openshift/installer/pull/8644

Currently Assisted Service, when adding a new node, configure the bootstrap ignition to fetch the node ignition from the insecure port (22624), even though it would be possible to use the secure one (22623). This could be an issue for the existing users that didn't want to use the insecure port for the add node operation.

Implementation notes
Extend the ClusterInfo asset to retrieve the initial ignition details (url and ca certificate) from the openshift-machine-api/worker-user-data-managed secret, if available in the target cluster. These information will then be used by the agent-installer-client when importing a new cluster, to configure the cluster ignition_endpoint

• https://github.com/openshift/installer/pull/8650

(see more context in comment https://github.com/openshift/installer/pull/8242#discussion_r1571664023)

To allow running the node-joiner in a pod into a cluster where FIPS was enabled:

• set CGO_ENABLED=1 in hack/build-node-joiner.sh
• set the fips=1 karg in the ISO (if not already set, to be verified)

• https://github.com/openshift/installer/pull/8760

Performs all the required preliminary validations for adding new nodes:

• Ensure that the (static) IPs/hostnames, if specified, do not conflict with the existing ones
  -If nmstate config is provided, it should be validated as usual- (already done)
• Validate that the target platform is a supported one

• https://github.com/openshift/installer/pull/8740
• https://github.com/openshift/oc/pull/1832

Allow running the node-image commands only for 4.17+ clusters.

Full support for 4.16 clusters will be implemented via https://issues.redhat.com/browse/OCPSTRAT-1528

• https://github.com/openshift/oc/pull/1837

This is the command responsible for monitoring the activity when adding new node(s) to the target cluster.
Similarly to the add-nodes-image command, also this one will be a simpler wrapper around the node-joiner's monitor command.

A list of the expected operations can be found in https://github.com/openshift/installer/blob/master/docs/user/agent/add-node/node-joiner-monitor.sh

• https://github.com/openshift/oc/pull/1836
• https://github.com/openshift/oc/pull/1835
• https://github.com/openshift/oc/pull/1823

On clusters where we use Cluster API instead of Machine API we need to create an empty file to report that the bootstrapping was successful. The file should be place in "/run/cluster-api/bootstrap-success.complete"

Normally there is a special controller for it, but in OpenShift we use MCO to bootstrap machines, so we have to create this file directly.

ToDo:

• Ensure that "/run/cluster-api" folder exists on the machine
• Create CAPI bootstrapping sentinel file "bootstrap-success.complete" in this folder

Links: 

https://cluster-api.sigs.k8s.io/developer/providers/bootstrap.html 
https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/azure/defaults.go#L81-L83 

• https://github.com/openshift/cluster-api-provider-azure/pull/301

User Story

As an OpenShift engineer I want the CAPI Providers repositories to use the new generator tool so that they can independently generate CAPI Provider transport ConfigMaps

Background

Once the new CAPI manifests generator tool is ready, we want to make use of that directly from the CAPI Providers repositories so we can avoid storing the generated configuration centrally and independently apply that based on the running platform.

Steps

• Install new CAPI manifest generator as a go `tool` to all the CAPI provider repositories
• Setup a make target under the `/openshift/Makefile` to invoke the generator. Make it output the manifests under `/openshift/manifests`
• Make sure `/openshift/manifests` is mapped to `/manifests` in the openshift/Dockerfile, so that the files are later picked up by CVO
• Make sure the manifest generation works by triggering a manual generation
• Check in the newly generated transport ConfigMap + Credential Requests (to let them be applied by CVO)

Stakeholders

• <Who is interested in this/where did they request this>

Definition of Done

• CAPI manifest generator tool is installed 

• Docs

• <Add docs requirements for this card>

• Testing

• <Explain testing that will be added>

• https://github.com/openshift/cluster-api-provider-azure/pull/300
• https://github.com/openshift/cluster-capi-operator/pull/170

Description of problem:

Port exposed in Dockerfile not observed in the Ports Dropdown in Git Import Form   

Version-Release number of selected component (if applicable):

    4.15

How reproducible:

Always

Steps to Reproduce:

 Use the https://github.com/Lucifergene/knative-do-demo/ to create application with the Dockerfile strateg     

Actual results:

    Ports not displayed

Expected results:

    Should display

Additional info:

• https://github.com/openshift/console/pull/13953

Description

As a user, I don't want to see jumping UI when the Import from Git flow (or other forms) waits that network call is handled correctly.

The PatternFly Button components have a progress indicator for that: https://www.patternfly.org/components/button/#progress-indicators

This should be preferred over the manually displayed indicator that is shown sometimes (esp on the Import from Git page) in the next line.

Acceptance Criteria

1. Use the button loading indicator instead of a loading indicator next to the button ("on all forms")
   1. Import from Git
   2. Import Container image
   3. Pipelines Builder (Pipelines create page)

Additional Details:

• https://github.com/openshift/console/pull/14007

Description

As a user, I want to quickly select a Git Type if Openshift cannot identify it. Currently, the UI involves opening a Dropdown and selecting the desired Git type.

But it would be great if I could select the Git Type directly without opening any dropdown. This will reduce the number of clicks required to complete the action.

Acceptance Criteria

1. Remove the Git Type dropdown
2. Use the Tile PF component to design the new Git Types
3. Make sure it's smaller than the Build Strategy Tiles
4. Update the E2E tests

Additional Details:

• https://github.com/openshift/console/pull/13936

Description

As a developer, I want to create a new Gitea service to be able to perform all kinds of import operations on repositories hosted in Gitea.

Acceptance Criteria

1. Analyse the APIs and filter the ones that are required for performing the import functions
2. Create the new gitea-service file.
3. Update the UI

Additional Details:

• https://github.com/openshift/console/pull/13928

Feature Overview (aka. Goal Summary)

OLM users can easily see in the console if an installed operator package is deprecated and learn how to stay within the support boundary by viewing the alerts/notifications that OLM emits, or by reviewing the operator status rendered by the console with visual representation.

Goals (aka. expected user outcomes)

• Pre-installation: OLM users can see the deprecation visual representation in the console UI and be warned/discouraged from installing a deprecated package, from deprecated channels, or in a deprecated version, and learn the recommended alternatives to stay within the supported path (with a short description).
• Post-installation: OLM users can see the deprecation visual representation in the console UI to tell if an installed operator is deprecated entirely, currently subscribed to a deprecated channel, or in a deprecated version, and know the alternatives as in package(s), update channel(s), or version(s) to stay within the support boundary.

Related Information

• Figma mocks: https://www.figma.com/proto/ZY4kXCx9AlcFWe7nOBI8JK/Untitled?page-id=140%3A1661&node-id=257-2465&viewport=1916%2C-2026%2C0.25&t=oUHBglFpyre6LgF4-1&scaling=min-zoom&starting-point-node-id=257%3A2465&show-proto-sidebar=1
• Design Brief (google doc): https://docs.google.com/document/d/1etcAcf2BCYYAVxjpYHnNsORsxzs3VKuTYBfTsLW6Dcg/edit?usp=sharing
• CatalogSource YAML for a catalog image with Operators carrying deprecation info: test-community-operator-deprecation.yaml (see in the attachment)
  • 3scale-community-operator:
    • deprecated channel: threescale-2.11
    • deprecated version: 3scale-community-operator.v0.8.2, 3scale-community-operator.v0.9.0

• • kiali-operator:
    • deprecated package: kiali-operator
    • deprecated channel: alpha
    • deprecated version: kiali-operator.v1.68.0

Acceptance Criteria

• Pre-installation
  • Operator Hub page - display the deprecation warning if the PackageManifest is deprecated
  • Install Operator details page
    • display the deprecation badge if the PackageManifest is deprecated
    • display the deprecation warning with deprecation message if PackageManifest, Channel or Version is deprecated
    • In both Channel and Version dropdown, show an warning icon next to the deprecated entry.
  • Operator install page
    • display the deprecation badge if the PackageManifest is deprecated
    • display the deprecation warning with deprecation message if PackageManifest, Channel or Version is deprecated
    • In both Channel and Version dropdown, show an warning icon next to the deprecated entry.
• Add integration and unit tests

• https://github.com/openshift/console/pull/13961

As a user, I would like to customize the modal displayed when a 'Create Project' button is clicked in the UI.

Acceptance criteria

• add an extension point so plugin creators can provide their own "Create Project" modal
• the console should implement this extension point using the "useModal" hook from the dynamic plugin SDK
• update docs with this extension point.
• add a a custom modal to the demo plugin for testing. The modal will create a project and quota for that project.

• https://github.com/openshift/console/pull/13825

Provide a simplified view of config files belonging to MachineConfig objects, to provide more convenient user experience. simpler management.

Current state:

• When a user/partner needs to retrieve contents from a MachineConfig they need to manually decode the file into a readable format from the URL-encoded one. For example: $ oc get mc $machineConfigName -o jsonpath=' {.spec.config.storage.files[1].contents.source}

  ' | sed "s@+@ @g;s@%@\\\\x@g" | xargs -0 printf "%b\n

Desired state:

• Instead of the command below the partner would like to get the content of the config files by default in a human-readable format via the OpenShift web dashboard. Please see the attached image. On this screen, a new tab can be introduced to be able to check the content of the configuration files.

AC:

• Add the Configuration Files section into the MachineConfig details page, which renders all the configuration that the resource contains.
• Add integration tests

RFE - https://issues.redhat.com/browse/RFE-5198

• https://github.com/openshift/console/pull/14015

Feature Overview (aka. Goal Summary)

OLM users can easily see in the console if an installed operator package is deprecated and learn how to stay within the support boundary by viewing the alerts/notifications that OLM emits, or by reviewing the operator status rendered by the console with visual representation.

Goals (aka. expected user outcomes)

• Pre-installation: OLM users can see the deprecation visual representation in the console UI and be warned/discouraged from installing a deprecated package, from deprecated channels, or in a deprecated version, and learn the recommended alternatives to stay within the supported path (with a short description).
• Post-installation: OLM users can see the deprecation visual representation in the console UI to tell if an installed operator is deprecated entirely, currently subscribed to a deprecated channel, or in a deprecated version, and know the alternatives as in package(s), update channel(s), or version(s) to stay within the support boundary.

Related Information

• Figma mocks: https://www.figma.com/proto/ZY4kXCx9AlcFWe7nOBI8JK/Untitled?page-id=140%3A1661&node-id=257-2465&viewport=1916%2C-2026%2C0.25&t=oUHBglFpyre6LgF4-1&scaling=min-zoom&starting-point-node-id=257%3A2465&show-proto-sidebar=1
• Design Brief (google doc): https://docs.google.com/document/d/1etcAcf2BCYYAVxjpYHnNsORsxzs3VKuTYBfTsLW6Dcg/edit?usp=sharing
• CatalogSource YAML for a catalog image with Operators carrying deprecation info: test-community-operator-deprecation.yaml (see in the attachment)
  • 3scale-community-operator:
    • deprecated channel: threescale-2.11
    • deprecated version: 3scale-community-operator.v0.8.2, 3scale-community-operator.v0.9.0

• • kiali-operator:
    • deprecated package: kiali-operator
    • deprecated channel: alpha
    • deprecated version: kiali-operator.v1.68.0

Acceptance Criteria

• • Installed operators page
    • Add the deprecation badge to the operator's Status field, if PackageManifest, Channel or Version is deprecated
  • Operator Details page 
    • Add deprecation badge next to the operator's name if PackageManifest, Channel or Version is deprecated
    • Add deprecation warning to the details page, if PackageManifest, Channel or Version is deprecated
      • Link the the operator's Subscription tab if Channel or Version is deprecated
      • In Subscription tab, the warning will contain "Update channel" link. In case of deprecated Channel, the "Change Subscription update channel" modal will need to get updated to contain warning icons next to deprecated channels.
• Add integration and unit tests

• https://github.com/openshift/console/pull/14048

Description

If there is any PDB with SufficientPods have allowed disruptions equal to 0, then when cluster admin tries to drain a node it will through error as Pod disruption budget is violated. So in order to avoid this, add a warning message in Topology page similar to Resource quota warning message to let the user know about this violation. 

Acceptance Criteria

1. Create a util where it will fetch PDB for namespace and return how many PDB with disruptionsAllowed is 0 and with SufficientPods, if the count is 1, return the name also so that we can redirect to it's details page
2. Use the util in Topology page and add Warning message similar to resource quota warning message.
3. On click of warning message, if there is one PDB which is violated then redirect to it's details page or else to PDB list page
4. Add YellowExclamationTriangleIcon to PDB list page to Allowed disruption column where rows having allowed disruption equal to 0. Add beside the count.
5. Create unit test for the util
6. Add e2e test (Automate this or manual????)

Additional Details:

Internal doc for reference - https://docs.google.com/document/d/1pa1jaYXPPMc-XhHt_syKecbrBaozhFg2_gKOz7E2JWw/edit

• https://github.com/openshift/console/pull/13921

Acceptance Criteria

• oc adm must-gather -help no longer shows the since and -since-time flags as experimental

• https://github.com/openshift/oc/pull/1811

Current state:

An update is in progress for 28m42s: Working towards 4.14.1: 700 of 859 done (81% complete), waiting on network

= Control Plane =
...
Completion:      91%

Improvement opportunities

1. Inconsistent info: CVO message says "700 of 859 done (81% complete)" but control plane section says "Completion: 91%"
2. Unclear measure of completion: CVO message counts manifest applied and control plane section says "Completion: 91%" which counts upgraded COs. Both messages do not state what they count. Manifest count is an internal implementation detail which users likely do not understand. COs are less so, but we should be more clear in what the completion means.
3. We could take advantage of this line and communicate progress with more details

Definition of Done

We'll only remove CVO message once the rest of the output functionally covers it, so the inconsistency stays until OTA-1154. Otherwise:

= Control Plane =
...
Completion:      91% (30 operators upgraded, 1 upgrading, 2 waiting)

Upgraded operators are COs that updated its version, no matter its conditions
Upgrading operators are COs that havent updated its version and are Progressing=True
Waiting operators are COs that havent updated its version and are Progressing=False

• https://github.com/openshift/oc/pull/1831

=Control Plane Upgrade=
...
Completion: 45% (Est Time Remaining: 35m)
                ^^^^^^^^^^^^^^^^^^^^^^^^^

Do not worry too much about the precision, we can make this more precise in the future. I am thinking of
1. Assigning a fixed amount of time per CO remaining for COs that do not have daemonsets
2. Assign an amount of time proportional to # of workers to each remaining CO that has daemonsets (network, dns)
3. Assign a special amount of time proportional to # of workers to MCO

We can probably take into account the "how long are we upgrading this operator right now" exposed by CVO in OTA-1160

• https://github.com/openshift/oc/pull/1742

Discovered by Evgeni Vakhonin during OTA-1245, the 4.16 code did not take nodes that are members of multiple pools into account. This surfaced in several ways:

Duplicate insights (=we iterate over nodes over pools, so we see problematic edges in each pool it is a member of):

= Update Health =
SINCE   LEVEL     IMPACT           MESSAGE
-	Error     Update Stalled   Node ip-10-0-26-198.us-east-2.compute.internal is degraded
-	Error     Update Stalled   Node ip-10-0-26-198.us-east-2.compute.internal is degraded

Such node is present in all pool listings, and in some cases such as paused pools the output is confusing (paused-ness is a property of a pool, so we list a node as paused in one pool but outdated pending in another):

= Worker Pool =
Worker Pool:     mcpfoo
Assessment:      Excluded
...

Worker Pool Node
NAME                                        ASSESSMENT   PHASE    VERSION   EST   MESSAGE
ip-10-0-26-198.us-east-2.compute.internal   Excluded     Paused   4.15.12   -

= Worker Pool =
Worker Pool:     worker
...
Worker Pool Nodes
NAME                                        ASSESSMENT   PHASE     VERSION                              EST   MESSAGE 
ip-10-0-26-198.us-east-2.compute.internal   Outdated     Pending   4.15.12                              ? 

It is not clear to me what would be the correct presentation of this case. Because this is an update status (and not node or cluster status) command, and only a single pool drives an update of a node, I'm thinking that maybe the best course of action would be to show only nodes whose version is driven by a given pool, or maybe come up with a "externally driven"-like assessment or whatever.

• https://github.com/openshift/oc/pull/1825
• https://github.com/openshift/oc/pull/1822

As an OTA engineer,
I would like to make sure the node in a single-node cluster is handled correctly in the upgrade-status command.

Context:
According to the discussion with the MCO team,
the node is in MCP/master but not worker.
This card is to make sure that the node are displayed that way too. My feeling is that the current code probably does the job already. In that case, we should add test coverage for the case to avoid regression in the future.

AC:

• The node is displayed in the master section in the output of the upgrade-status command.
• The node is NOT displayed in the worker section in the output of the upgrade-status command.
• A test case exists in https://github.com/openshift/oc/tree/master/pkg/cli/admin/upgrade/status/examples

• https://github.com/openshift/oc/pull/1829

• https://github.com/openshift/cloud-credential-operator/pull/776

Description of problem:

failed to create control-plane machines using GCP marketplace image

Version-Release number of selected component (if applicable):

4.16.0-0.nightly-multi-2024-06-11-205940 / 4.16.0-0.nightly-2024-06-10-211334

How reproducible:

Always

Steps to Reproduce:

1. "create install-config", and then edit it to insert osImage settings (see [1])
2. "create cluster" (see [2])

Actual results:

1. The bootstrap machine and the control-plane machines are not created.
2. Although it says "Waiting up to 15m0s (until 10:07AM CST)" for control-plane machines being provisioned, it did not time out until around 10:35AM CST.

Expected results:

The installation should succeed.

Additional info:

FYI a PROW CI test also has the issue: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/52816/rehearse-52816-periodic-ci-openshift-verification-tests-master-installer-rehearse-4.16-installer-rehearse-debug/1800431930391400448

• https://github.com/openshift/installer/pull/8665

• https://github.com/openshift/installer/pull/8931

Once support for private/internal clusters is added to CAPG in CORS-3252, we will need to integrate those changes into the installer:

• vendor updated capg
• update cluster manifest so internal cluster is default
• update load balancer creation (so external LB is created by installer and MCS configuration is added to CAPI-created LB)
• update DNS record creation (if needed) to ensure we are associating records with proper LB

• https://github.com/openshift/installer/pull/8421

• https://github.com/openshift/installer/pull/8948

Description of problem:

installing into Shared VPC stuck in waiting for network infrastructure ready

Version-Release number of selected component (if applicable):

4.17.0-0.nightly-2024-06-10-225505

How reproducible:

Always

Steps to Reproduce:

1. "create install-config" and then insert Shared VPC settings (see [1])
2. activate the service account which has the minimum permissions in the host project (see [2])
3. "create cluster"

FYI The GCP project "openshift-qe" is the service project, and the GCP project "openshift-qe-shared-vpc" is the host project. 

Actual results:

1. Getting stuck in waiting for network infrastructure to become ready, until Ctrl+C is pressed.
2. 2 firewall-rules are created in the service project unexpectedly (see [3]).

Expected results:

The installation should succeed, and there should be no any firewall-rule getting created either in the service project or in the host project.

Additional info:

• https://github.com/openshift/installer/pull/8780
• https://github.com/openshift/installer/pull/8706
• https://github.com/openshift/installer/pull/8583

When installconfig.platform.gcp.userTags is specified, all taggable resources should have the specified user tags.

This requires setting TechPreviewNoUpgrade featureSet to configure tags.

• https://github.com/openshift/installer/pull/8691

Use CAPG by default and remove it from TechPreview in 4.17.

• https://github.com/openshift/installer/pull/8723
• https://github.com/openshift/api/pull/1958

Once https://github.com/openshift/installer/pull/8359 merges, which adds a call to CAPI DestroyBootstrap, the GCP bootstrap firewall rule should be removed. This rule was added in https://github.com/openshift/installer/pull/8374.

• https://github.com/openshift/installer/pull/8523

When creating a Private Cluster with CAPG the cloud-controller-manager generates an error when the instance-group is created:

I0611 00:04:34.998546       1 event.go:376] "Event occurred" object="openshift-ingress/router-default" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: Resource 'projects/openshift-dev-installer/zones/us-east1-b/instances/bfournie-capg-test-6vn69-worker-b-rghf7' is expected to be in the subnetwork 'projects/openshift-dev-installer/regions/us-east1/subnetworks/bfournie-capg-test-6vn69-master-subnet' but is in the subnetwork 'projects/openshift-dev-installer/regions/us-east1/subnetworks/bfournie-capg-test-6vn69-worker-subnet'., wrongSubnetwork"

Three "k8s-ig" instance-groups were created for the Internal LoadBlancer. Of the 3, the first one is using the master subnet

subnetwork: https://www.googleapis.com/compute/v1/projects/openshift-dev-installer/regions/us-east1/subnetworks/bfournie-capg-test-6vn69-master-subnet

while the other two are using the worker-subnet. Since this IG uses the master-subnet and the instance is using the worker-subnet it results in this mismatch.

This looks similar to issue tracked (and closed) with cloud-provider-gcp
https://github.com/kubernetes/cloud-provider-gcp/pull/605

• https://github.com/openshift/cloud-provider-gcp/pull/66

Once a new version of CAPG is released we'll need to pick it up.

• https://github.com/openshift/installer/pull/8700

Related to https://issues.redhat.com/browse/CORS-3445, we need to ensure that pre-created ServiceAccounts can be passed in and used by a CAPG installation.

• https://github.com/openshift/installer/pull/8750

We are occasionally seeing this error when using GCP with TechPreview, i.e. using CAPG.

waiting for api to be available
level=warning msg=FeatureSet "TechPreviewNoUpgrade" is enabled. This FeatureSet does not allow upgrades and may affect the supportability of the cluster.
level=info msg=Creating infrastructure resources...
level=error msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed during pre-provisioning: failed to add worker roles: failed to set project IAM policy: googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff. The request's ETag '\007\006\033\255\347+\335\210' did not match the current policy's ETag '\007\006\033\255\347>%\332'., aborted
Installer exit with code 4
Install attempt 3 of 3

Here is an example:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-hive-master-periodic-e2e-gcp-weekly/1813424715671277568/artifacts/e2e-gcp-weekly/test/artifacts/cluster-test-hive-310440d4-8bfb-40f1-a489-ec0a44a7852e-0-m2gqq-provisio29dxg-installer.log

• https://github.com/openshift/installer/pull/8763

This is a clone of issue OCPBUGS-38152. The following is the description of the original issue:
—
Description of problem:

    Shared VPC installation using service account having all required permissions failed due to cluster operator ingress degraded, by telling error "error getting load balancer's firewall: googleapi: Error 403: Required 'compute.firewalls.get' permission for 'projects/openshift-qe-shared-vpc/global/firewalls/k8s-fw-a5b1f420669b3474d959cff80e8452dc'"

Version-Release number of selected component (if applicable):

    4.17.0-0.nightly-multi-2024-08-07-221959

How reproducible:

    Always