Jump to: Complete Features | Incomplete Features | Complete Epics | Incomplete Epics | Other Complete | Other Incomplete |
Note: this page shows the Feature-Based Change Log for a release
These features were completed when this image was assembled
The admin console's alert rule details page is provided by https://github.com/openshift/monitoring-plugin, but the dev console's equivalent page is still provided by code in the console codebase.
That dev console page is loaded from monitoring-plugin and the code for the page is removed from the console codebase.
Ensure removal of deprecated patternfly components from kebab-dropdown.tsx and alerting.tsx once this story and OU-257 are completed.
Technical debt led to a majority of the alerting components and all routing components being placed in a single file. This made the file very large, difficult to work with, and confusing for others to work with the routing items.
Alerting components have all been moved into their own separate files and the routing for the monitoring plugin is moved out of alerting.tsx into a routing specific file.
Placeholder feature for ccx-ocp-core maintenance tasks.
This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.
Description of problem:
The operatorLogLevel reverting back to `Normal` automatically for insightsoperator object
Version-Release number of selected component (if applicable):
4.14,4.15,4.16,4.17
How reproducible:
100%
Steps to Reproduce:
1. Edit the insightsoperator/cluster object: $ oc edit insightsoperator/cluster ... ... spec: logLevel: Normal managementState: Managed operatorLogLevel: Normal ====> Change it to Debug/Trace/TraceAll 2. After 1 Min check if the changes are constant: $ oc get insightsoperator/cluster -o yaml ... ... spec: logLevel: Normal managementState: Managed operatorLogLevel: Normal 3. The OC CLI DOC states that it supports below logLevels: $ oc explain insightsoperator.spec.operatorLogLevel GROUP: operator.openshift.io KIND: InsightsOperator VERSION: v1FIELD: operatorLogLevel <string> ENUM: "" Normal Debug Trace TraceAll DESCRIPTION: operatorLogLevel is an intent based logging for the operator itself. It does not give fine grained control, but it is a simple way to manage coarse grained logging choices that operators have to interpret for themselves. Valid values are: "Normal", "Debug", "Trace", "TraceAll". Defaults to "Normal".
Actual results:
The logLevel is set back to Normal automatically.
Expected results:
The logLevel should not be changed to Normal until manually modified.
Additional info:
The same issue is observed with insightsoperator.spec.logLevel where any logLevel other than Normal getting reverted.
Goal:
Track Insights Operator Data Enhancements epic in 2024
Description of problem:
Context OpenShift Logging is migrating from Elasticsearch to Loki. While the option to use Loki has existed forquite a while, the information about end of Elasticsearch support has not been available until recently. With the information available now, we can expect more and more customers to migrate and hit the issue described in INSIGHTOCP-1927. P.S. Note the bar chart in INSIGHTOCP-1927 which shows how frequently is the related KCS linked in customer cases. Data to gather LokiStack custom resources (any name, any namespace) Backports The option to use Loki is available since Logging 5.5 whose compatibility started at OCP 4.9. Considering the OCP life cycle, backports to up to OCP 4.14 would be nice. Unknowns Since Logging 5.7, Logging supports installation of multiple instances in customer namespaces. The Insights Operator would have to look for the CRs in all namespaces, which poses the following questions: What is the expected number of the LokiStack CRs in a cluster? Should the Insights operator look for the resource in all namespaces? Is there a way to narrow down the scope? The CR will contain the name of a customer namespaces which is a sensitive information. What is the API group of the CR? Is there a risk of LokiStack CRs in customer namespaces that would NOT be related to OpenShift Logging? SME Oscar Arribas Arribas
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
N/A
Actual results:
Expected results:
Additional info:
Volume Group Snapshots is a key new Kubernetes storage feature that allows multiple PVs to be grouped together and snapshotted at the same time. This enables customers to takes consistent snapshots of applications that span across multiple PVs.
This is also a key requirement for backup and DR solutions.
https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/
https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot
This feature tracks the Tech Preview implementation behind feature gate.
Productise the volume group snapshots feature as tech preview have docs, testing as well as a feature gate to enable it in order for customers and partners to test it in advance.
The feature should be graduated beta upstream to become TP in OCP. Tests and CI must pass and a feature gate should allow customers and partners to easily enable it. We should identify all OCP shipped CSI drivers that support this feature and configure them accordingly.
CSI drivers development/support of this feature.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
This allows backup vendors to implemented advanced feature by taking snapshots of multiple volumes at the same time a common use case in virtualisation.
Drivers must support this feature and enable it. Partners may need to change their operator and/or doc to support it.
Document how to enable the feature, what this feature does and how to use it. Update the OCP driver's table to include this capability.
Can be leveraged by ODF and OCP virt, especially around backup and DR scenarios.
Epic Goal*
What is our purpose in implementing this? What new capability will be available to customers?
Add Volume Group Snapshots as Tech Preview. This is a key new Kubernetes storage feature that allows multiple PVs to be grouped together and snapshotted at the same time. This enables customers to takes consistent snapshots of applications that span across multiple PVs.
We will rely on the newly beta promoted feature. This feature is driver dependent.
This will need a new external-snapshotter rebase + removal of the feature gate check in csi-snapshot-controller-operator. Freshly installed or upgraded from older release, will have group snapshot v1beta1 API enabled + enabled support for it in the snapshot-controller (+ ship corresponding external-snapshotter sidecar).
No opt-in, no opt-out.
OCP itself will not ship any CSI driver that supports it.
Why is this important? (mandatory)
What are the benefits to the customer or Red Hat? Does it improve security, performance, supportability, etc? Why is work a priority?
This is also a key requirement for backup and DR solutions specially for OCP virt.
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
External snapshotter rebase to the upstream version that include the beta API.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Since we don't ship any driver with OCP that support the feature we need to have testing with ODF
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
No risk, behind feature gate
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Crun is GA as non default since OCP 4.14 . We want to make it as default in 4.18 while still supporting runc as non-default
Benefits of Crun is covered here https://github.com/containers/crun
FAQ.: https://docs.google.com/document/d/1N7tik4HXTKsXS-tMhvnmagvw6TE44iNccQGfbL_-eXw/edit
***Note -> making Crun default does not means we will remove the support for runc nor we have any plans in foreseeable future to do that
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
iSCSI boot is supported in RHEL and since the implementation of OCPSTRAT-749 it's also available in RHCOS.
Customers require using this feature in different bare metal environments on-prem and cloud-based.
Assisted Installer implements support for it in Oracle Cloud Infrastructure (MGMT-16167) to support their bare metal standard "shapes".
This feature extends this support to make it generic and supported in the Agent-Based Installer, the Assisted Installer and in ACM/MCE.
Support iSCSI boot in bare metal nodes, including platform baremetal and platform "none".
Assisted installer can boot and install OpenShift on nodes with iSCSI disks.
Agent-Based Installer can boot and install OpenShift on nodes with iSCSI disks.
MCE/ACM can boot and install OpenShift on nodes with iSCSI disks.
The installation can be done on clusters with platform baremetal and clusters with platform "none".
Support booting from iSCSI using ABI starting OCP 4.16.
The following PRs are the gaps between release-4.17 branch and master that are needed to make the integration work on 4.17.
https://github.com/openshift/assisted-service/pull/6665
https://github.com/openshift/assisted-service/pull/6603
https://github.com/openshift/assisted-service/pull/6661
The feature has to be backported to 4.16 as well. TBD - list all the PRs that have to be backported.
Instructions to test the AI feature with local env - https://docs.google.com/document/d/1RnRhJN-fgofnVSBTA6mIKcK2_UW7ihbZDLGAVHSdpzc/edit#heading=h.bf4zg53460gu
When testing the agent-based installer using iSCSI with dev-scripts https://github.com/openshift-metal3/dev-scripts/pull/1727 it was found that the installer was not able to complete the installation when using multiple hosts. This same problem did not appear when using SNO.
The iscsi session from all the hosts work do their targets fine until coreos-installer is run, at which time (before reboot) the connection to the target is lost and the coreos-installer fails
Jan 09 16:12:23 master-1 kernel: session1: session recovery timed out after 120 secs Jan 09 16:12:23 master-1 kernel: sd 7:0:0:0: rejecting I/O to offline device Jan 09 16:12:23 master-1 kernel: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2 Jan 09 16:12:23 master-1 installer[2937]: time="2025-01-09T16:12:23Z" level=info msg="\nError: syncing data to disk\n\nCaused by:\n Input/output error (os error 5)\n\nResetting partition table\n" Jan 09 16:12:23 master-1 installer[2937]: time="2025-01-09T16:12:23Z" level=warning msg="Retrying after error: failed executing /usr/bin/nsenter [--target 1 --cgroup --mount --ipc --pid -- coreos-installer install --insecure -i /opt/install-dir/master-f3c24588-2129-483f-9dfb-8a8fe332a4bf.ign --append-karg rd.iscsi.firmware=1 --append-karg ip=enp6s0:dhcp --copy-network /dev/sdb], Error exit status 1, LastOutput \"Error: syncing data to disk\n\nCaused by:\n Input/output error (os error 5)\n\nResetting partition table\nError: syncing partition table to disk\n\nCaused by:\n Input/output error (os error 5)\""
On the host it can be seen that session shows as logged out
Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.2023-01.com.example:master-1 Iface IPaddress: [default] Iface HWaddress: default Iface Netdev: default SID: 1 iSCSI Connection State: Unknown iSCSI Session State: FREE Internal iscsid Session State: Unknown
The problem occurs because the iscsid service is not running. If it is started by iscsadm then coreos-installer can successfully write the image to disk.
Epic Goal*
Drive the technical part of the Kubernetes 1.31 upgrade, including rebasing openshift/kubernetes repositiry and coordination across OpenShift organization to get e2e tests green for the OCP release.
Why is this important? (mandatory)
OpenShift 4.18 cannot be released without Kubernetes 1.31
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
PRs:
Retro: Kube 1.31 Rebase Retrospective Timeline (OCP 4.18)
Retro recording: https://drive.google.com/file/d/1htU-AglTJjd-VgFfwE3z_dH5tKXT1Tes/view?usp=drive_web
Feature description
Oc-mirror v2 is focuses on major enhancements that include making oc-mirror faster and more robust and introduces caching as well as address more complex air-gapped scenarios. OC mirror v2 is a rewritten version with three goals:
There was a selected bundle feature on v2 that needs to be removed in 4.18 because of the its risk.
An alternative solution is required to unblock one of our customers.
Simplify debugging when a cluster fails to update to a new target release image, when that release image is unsigned or otherwise fails to pull.
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
– Kubelet/CRIO to verify RH images & release payload sigstore signatures
– ART will add sigstore signatures to core OCP images
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
These acceptance criteria are for all deployment flavors of OpenShift.
Deployment considerations | List applicable specific needs (N/A = not applicable) | |
Self-managed, managed, or both | both | |
Classic (standalone cluster) | yes | |
Hosted control planes | no | |
Multi node, Compact (three node), or Single node (SNO), or all | ||
Connected / Restricted Network | ||
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | ||
Operator compatibility | ||
Backport needed (list applicable versions) | Not Applicable | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | none, | |
Other (please specify) |
Add documentation for sigstore verification and gpg verification
For folks mirroring release images (e.g. disconnected/restricted-network):
Currently the CVO launches a Job and waits for it to complete to get manifests for an incoming release payload. But the Job controller doesn't bubble up details about why the pod has trouble (e.g. Init:SignatureValidationFailed), so to get those details, we need direct access to the Pod. The Job controller doesn't seem like it's adding much value here, so the goal of this Epic is to drop it and create and monitor the Pod ourselves, so we can deliver better reporting of version-Pod state.
When the version Pod fails to run, the cluster admin will likely need to take some action (clearing the update request, fixing a mirror registry, etc.). The more clearly we share the issues that the Pod is having with the cluster admin, the easier it will be for them to figure out their next steps.
oc adm upgrade and other ClusterVersion status UIs will be able to display Init:SignatureValidationFailed and other version-Pod failure modes directly. We don't expect to be able to give ClusterVersion consumers more detailed next-step advice, but hopefully the easier access to failure-mode context makes it easier for them to figure out next-steps on their own.
This change is purely and updates-team/OTA CVO pull request. No other dependencies.
Definition of done: failure modes like unretrievable image digests (e.g. quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000) or images with missing or unacceptable Sigstore signatures with OTA-1304's ClusterImagePolicy) have failure-mode details in ClusterVersion's RetrievePayload message, instead of the current Job was active longer than specified deadline.
Limited audience, and failures like Init:SignatureValidationFailed are generic, while CVO version-Pod handling is pretty narrow. This may be redundant work if we end up getting nice generic init-Pod-issue handling like RFE-5627. But even if the work ends up being redundant, thinning the CVO stack by removing the Job controller is kind of nice.
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Currently the CVO launches a Job and waits for it to complete to get manifests for an incoming release payload. But the Job controller doesn't bubble up details about why the pod has trouble (e.g. Init:SignatureValidationFailed), so to get those details, we need direct access to the Pod. The Job controller doesn't seem like it's adding much value here, so we probably want to drop it and create and monitor the Pod ourselves.
Definition of done: failure modes like unretrievable image digests (e.g. quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000) or images with missing or unacceptable Sigstore signatures with OTA-1304's ClusterImagePolicy) have failure-mode details in ClusterVersion's RetrievePayload message, instead of the current Job was active longer than specified deadline.
Not clear to me what we want to do with reason, which is currently DeadlineExceeded. Keep that? Split out some subsets like SignatureValidationFailed and whatever we get for image-pull-failures? Other?
See the UDN Sync Meeting notes: https://docs.google.com/document/d/1wjT8lSDfG6Mj-b45_p9SlbpCVG0af-pjDlmAUnM5mOI/edit#bookmark=id.zemsagq71ou2
In our current UDN API, subnets field is mandatory always for primary role and optional for secondary role. This is because users are allowed to have a pure L2 without subnets for secondary networks. However, in the future if we want to add egress support on secondary networks, we might need subnets...
CNV has many different use cases:
This card tracks the design changes to the API and the code changes needed to implement this. See https://github.com/openshift/enhancements/pull/1708 for details.
We want to do Network Policies not MultiNetwork POlicies
Enable sharing ConfigMap and Secret across namespaces
Requirement | Notes | isMvp? |
---|---|---|
Secrets and ConfigMaps can get shared across namespaces | YES |
NA
NA
Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them.
Questions to be addressed:
* What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
* Does this feature have doc impact?
* New Content, Updates to existing content, Release Note, or No Doc Impact
* If unsure and no Technical Writer is available, please contact Content Strategy.
* What concepts do customers need to understand to be successful in [action]?
* How do we expect customers will use the feature? For what purpose(s)?
* What reference material might a customer want/need to complete [action]?
* Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
* What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic Goal*
Remove the Shared Resource CSI Driver as a tech preview feature.
Why is this important? (mandatory)
Shared Resources was originally introduced as a tech preview feature in OpenShift Container Platform. After extensive review, we have decided to GA this component through the Builds for OpenShift layered product.
Expected GA will be alongside OpenShift 4.16. Therefore it is safe to remove in OpenShift 4.17
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Drawbacks or Risk (optional)
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
We need to do a lot of R&D and fix some known issues (e.g., see linked BZs).
R&D targetted at 4.16 and productisation of this feature in 4.17
Goal
To make the current implementation of the HAProxy config manager the default configuration.
Objectives
The goal of this user story is to combine the code from the smoke test user story and results from the spike into an implementation PR.
Since multiple gaps were discovered a feature gate will be needed to ensure stability of OCP before the feature can be enabled by default.
Phase 2 Goal:
for Phase-1, incorporating the assets from different repositories to simplify asset management.
Overarching Goal
Move to using the upstream Cluster API (CAPI) in place of the current implementation of the Machine API for standalone Openshift.
Phase 1 & 2 covers implementing base functionality for CAPI.
Phase 2 also covers migrating MAPI resources to CAPI.
There must be no negative effect to customers/users of the MAPI, this API must continue to be accessible to them though how it is implemented "under the covers" and if that implementation leverages CAPI is open
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
The CAPI operator should ensure that, for clusters that are upgraded into a version of openshift supporting CAPI, that a Cluster object exists in the openshift-cluster-api namespace with the name as the infratructure ID of the Cluster.
The cluster spec should be populated with the reference to the infrastructure object and the status should be updated to reflect that the control plane is initialized.
Goal Summary
This feature aims to make sure that the HyperShift operator and the control-plane it deploys uses Managed Service Identities (MSI) and have access to scoped credentials (also via access to AKS's image gallery potentially). Additionally, for operators deployed in customers account (system components), they would be scoped with Azure workload identities.
Requirements (aka. Acceptance criteria)
Today Azure installation requires manually created service principal which involves relations, permission granting, credential setting, credential storage, credentials rotation, credentials clean up, and service principal deletion. This is not only mundane and time-consuming but also less secure and risks access to resources by adversaries due to lack of credential rotation.
Employ Azure managed credentials which drastically reduce the steps required to just managed identity creation, permission granting, and resource deletion.
Ideally, this should be a HyperShift-native functionality. I.e., HyperShift should use managed identities for the control plane, the kubelet, and any add-on that needs access to Azure resources.
Currently, we were using contributor role for all control plane identities on ARO HCP. We should go ahead and restrict the identities we know there are existing ARO roles for already.
Determine the azure permissions to operate hosted clusters (control plane side). Specifically:
The Cluster Ingress Operator can authenticate with Service Principal backed by a certificate stored in an Azure Key Vault. The Secrets CSI driver will be used to mount the certificate as a volume on the image registry deployment in a hosted control plane.
Azure SDK
Which degree of coverage should run on AKS e2e vs on existing e2es
CI - Existing CI is running, tests are automated and merged.
CI - AKS CI is running, tests are automated and merged.
DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
DEV - Downstream build attached to advisory: <link to errata>
QE - Test plans in Polarion: <link or reference to Polarion>
QE - Automated tests merged: <link or reference to automated tests>
DOC - Downstream documentation merged: <link to meaningful PR>
The Cluster Network Operator can authenticate with Service Principal backed by a certificate stored in an Azure Key Vault. The Secrets CSI driver will be used to mount the certificate as a volume on the image registry deployment in a hosted control plane.
Azure SDK
Which degree of coverage should run on AKS e2e vs on existing e2es
CI - Existing CI is running, tests are automated and merged.
CI - AKS CI is running, tests are automated and merged.
DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
DEV - Downstream build attached to advisory: <link to errata>
QE - Test plans in Polarion: <link or reference to Polarion>
QE - Automated tests merged: <link or reference to automated tests>
DOC - Downstream documentation merged: <link to meaningful PR>
Requirement | Notes | isMvp? |
---|---|---|
CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
This Section:
This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.
Questions to be addressed:
Provide a simple way to get a VM-friendly networking setup, without having to configure the underlying physical network.
Provide a network solution working out of the box, meeting expectations of a typical VM workload.
Ensure the feature can be used on non dev|tech preview clusters.
Involves a PR to the OpenShift API - need to involve the API team.
This task requires periodic e2e tests in openshift/origin and openshift/ovn-kubernetes asserting the correct behavior of the gated feature. Currently focused on that (must add a virt-aware TP lane).
This is the script that decied if FG can be GA or not
https://github.com/openshift/api/blob/master/tools/codegen/cmd/featuregate-test-analyzer.go
Interesing slack thread https://redhat-internal.slack.com/archives/CE4L0F143/p1729612519105239?thread_ts=1729607662.848259&cid=CE4L0F143
Primary used-defined networks can be managed from the UI and the user flow is seamless.
When this image was assembled, these features were not yet completed. Therefore, only the Jira Cards included here are part of this release
The EPIC purpose is to hold all the required works to integrate CAPI & CAPI providers (CAPA & CAPZ) into MCE
It is important for end users to be able to have life cycle management for ROSA-HCP and ARO-HCP clusters
...
...
We need to test using the redhat cert-manager operator instead of using the upstream helm chart for certificate manager as our doc mentioned here
Placeholder feature for ccx-ocp-core maintenance tasks.
This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.
The following Insights APIs use duration attributes:
The kubebuilder validation patterns are defined as
^0|([1-9][0-9]*(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$
and
^([1-9][0-9]*(\\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$
Unfortunately this is not enough and it fails in case of updating the resource e.g with value "2m0s".
The validation pattern must allow these values.
It looks like the insights-operator doesn't work with IPv6, there are log errors like this:
E1209 12:20:27.648684 37952 run.go:72] "command failed" err="failed to run groups: failed to listen on secure address: listen tcp: address fd01:0:0:5::6:8000: too many colons in address"
It's showing up in metal techpreview jobs.
The URL isn't being constructed correctly, use NetJoinHostPort over Sprintf. Some more details here https://github.com/stbenjam/no-sprintf-host-port. There's a non-default linter in golangci-lint for this.
Component Readiness has found a potential regression in the following test:
[sig-architecture] platform pods in ns/openshift-insights should not exit an excessive amount of times
Test has a 56.36% pass rate, but 95.00% is required.
Sample (being evaluated) Release: 4.18
Start Time: 2024-12-02T00:00:00Z
End Time: 2024-12-09T16:00:00Z
Success Rate: 56.36%
Successes: 31
Failures: 24
Flakes: 0
View the test details report for additional context.
In order for customers or internal teams to troubleshoot better, they need to be able to see the dashboards created using Perses inside the OpenShift console. We will use the monitoring plugin which already supports console dashboards comming from Grafana, to provide the Perses dashboard funcionallity
Create a component in the monitoring plugin that can render a Perses dashboard based on the dashboard schema returned by the Perses API.
There are 2 dropdowns, one for selecting namespaces and another for selecting the dashboards in the selected namespace.
Previous work
Create dashboard flow chart
In order to allow customers and internal teams to see dashboards created using Perses, we must add them as new elements on the current dashboard list
In order for customers or internal teams to troubleshoot better, they need to be able to see the dashboards created using Perses inside the OpenShift console. We will use the monitoring plugin which already supports console dashboards coming from Grafana, to provide the Perses dashboard functionality
Create a component in the monitoring plugin that can render a Perses dashboard based on the dashboard schema returned by the Perses API.
There are 2 dropdowns, one for selecting namespaces and another for selecting the dashboards in the selected namespace.
Previous work
Create dashboard flow chart
1. Proposed title of this feature request:
Insights Operator Entitlements for Multi-Arch Clusters
2. What is the nature and description of the request?
When the Insights Operator syncs the cluster's Simple Content Access (SCA) certificate, it should survey the architectures of the worker nodes on the cluster and enable repositories for the appropriate architectures.
3. Why does the customer need this? (List the business requirements here)
Konflux and other customers needs this to produce multi-arch container images that install RHEL content via yum/dnf.
4. List any affected packages or components.
Insights Operator periodically pulls down the entitlements certificates into the cluster. Technically it means there's a HTTP POST request to the
https://api.openshift.com/api/accounts_mgmt/v1/certificates. The payload of the POST requests looks like:
`{"type": "sca","arch": "x86_64"}`
And this is currently the limitation. It prevents builds using the entitlement certificates on different architectures as s390x, ppc64le, arm64.
https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetes-io-arch label can be used
1. Proposed title of this feature request:
Insights Operator Entitlements for Multi-Arch Clusters
2. What is the nature and description of the request?
When the Insights Operator syncs the cluster's Simple Content Access (SCA) certificate, it should survey the architectures of the worker nodes on the cluster and enable repositories for the appropriate architectures.
3. Why does the customer need this? (List the business requirements here)
Konflux and other customers needs this to produce multi-arch container images that install RHEL content via yum/dnf.
4. List any affected packages or components.
tldr: three basic claims, the rest is explanation and one example
While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.
One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.
I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.
We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.
Relevant links:
Add a NID alias to OWNERS_ALIASES and update the OWNERS file in test/extended/router and add OWNERS file to test/extended/dns
As a result of Hashicorp's license change to BSL, Red Hat OpenShift needs to remove the use of Hashicorp's Terraform from the installer – specifically for IPI deployments which currently use Terraform for setting up the infrastructure.
To avoid an increased support overhead once the license changes at the end of the year, we want to provision IBM Cloud VPC infrastructure without the use of Terraform.
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
High-level list of items that are out of scope. Initial completion during Refinement status.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
As a cluster-admin, I want to run update in discrete steps. Update control plane and worker nodes independently.
I also want to back-up and restore incase of a problematic upgrade.
Background:
This Feature is a continuation of https://issues.redhat.com/browse/OCPSTRAT-180.
Customers are asking for improvements to the upgrade experience (both over-the-air and disconnected). This is a feature tracking epics required to get that work done.Below is the list of done tasks.
These are alarming conditions which may frighten customers, and we don't want to see them in our own, controlled, repeatable update CI. This example job had logs like:
Feb 18 21:11:25.799 E clusteroperator/openshift-apiserver changed Degraded to True: APIServerDeployment_UnavailablePod: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
And the job failed, but none of the failures were "something made openshift-apiserver mad enough to go Degraded".
Definition of done:
This feature aims to enable customers of OCP to integrate 3rd party KMS solutions for encrypting etcd values at rest in accordance with:
https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/
Scenario:
For an OCP cluster with external KMS enabled:
How doe the above scenario(s) impact the cluster? The API may be unavailable
Goal:
Investigation Steps:
Detection:
Actuation:
User stories that might result in KCS:
Plugins research:
POCs:
Acceptance Criteria:
We did something similar for the aesgcm encryption type in https://github.com/openshift/api/pull/1413/.
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
Provide a long term solution to SELinux context labeling in OCP. Continue the implementation with RWO/RWX PVs which are the most expected from the field. Start with a TechPreview support grade.
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
As of today when selinux is enabled, the PV's files are relabeled when attaching the PV to the pod, this can cause timeout when the PVs contains lot of files as well as overloading the storage backend.
https://access.redhat.com/solutions/6221251 provides few workarounds until the proper fix is implemented. Unfortunately these workaround are not perfect and we need a long term seamless optimised solution.
This feature tracks the long term solution where the PV FS will be mounted with the right selinux context thus avoiding to relabel every file. This covers RWO/RWX PVs, RWOP is already being implemented and should GA in 4.17.
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
Should pass all regular regression CI. All the drivers we ship should have it enabled and partners may enable it if they want it to be consumed.
Performances should drascillaly improved and security should remain the same as the legacy chcon approach.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | both |
Classic (standalone cluster) | Y |
Hosted control planes | Y |
Multi node, Compact (three node), or Single node (SNO), or all | all |
Connected / Restricted Network | Both |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | all |
Operator compatibility | AWS EBS, Azure Disk, GCP PD, IBM VPC block, OSP cinder, vSphere |
Backport needed (list applicable versions) | no |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | No need |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
As we are relying on mount context there should not be any relabeling (chcon) because all files / folders will inherit the context from the mount context
More on design & scenarios in the KEP
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
RWOP PVs
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Lots of support cases due to pod taking too long to start because of selinux relabeling with chcon. This epics covers the most "unpopular" RWX case specially for PVs with lots of files and backends that are "slow" at updating metadata.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Most cases / concerns are on RWX, RWOP was the first step and has limited customer's impact though it is easier to implement first and gather feedback / metrics. https://access.redhat.com/solutions/6221251
This feature track TP for RWO/RWX PVs
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
Relnotes + table of drivers supporting it.
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
Partners may want to enable the feature.
Epic Goal*
What is our purpose in implementing this? What new capability will be available to customers?
Tracking upstream beta promotion of the SELinux context mounts for RWX/RWO PVs
Why is this important? (mandatory)
What are the benefits to the customer or Red Hat? Does it improve security, performance, supportability, etc? Why is work a priority?
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Description of problem:
Seeing this error occasionally when using the agent-based installer in 4.19.0-ec.2. This causes the installation to fail.
Feb 24 19:56:00 master-0 node-image-pull.sh[5150]: Currently on CoreOS version 9.6.20250121-0 Feb 24 19:56:00 master-0 node-image-pull.sh[5150]: Target node image is quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b38df9ee395756ff57e9fdedd5c366ac9b129330aea9d3b07f1431fe9bcbab1 Feb 24 19:56:00 master-0 ostree-containe[5317]: Fetching ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b38df9ee395756ff57e9fdedd5c366ac9b129330aea9d3b07f1431fe9bcbab1 Feb 24 19:56:01 master-0 node-image-pull.sh[5317]: layers already present: 11; layers needed: 40 (691.0 MB) Feb 24 19:56:01 master-0 ostree-containe[5317]: layers already present: 11; layers needed: 40 (691.0 MB) Feb 24 19:56:01 master-0 node-image-pull.sh[5317]: error: Importing: Unencapsulating base: Layer sha256:2cda1fda6bb7b2171d8d65eb2f66101e513f33808a5e5ce0bf31fa0314e442ed: mkdirat: Read-only file system Feb 24 19:56:01 master-0 node-image-pull.sh[5150]: Failed to fetch release image; retrying...
Version-Release number of selected component (if applicable):
openshift 4.19.0-ec.2
How reproducible: Happens occasionally and inconsistently.
Steps to Reproduce:
1. Create the agent-installer iso via 'openshift-install agent create image'
2. Boot the ISO on 3 hosts - compact cluster
3.
Actual results:
Two of the hosts are installed correctly, in the 3rd host running bootstrap we see the error as above.
Expected results:
Installation succeeds.
As part of https://issues.redhat.com/browse/COS-2572, the OpenShift installer needs to be adapted so that it can work with bootimages which have no OCP content.
We want to ship OpenShift 4.19 with CAPO synced to upstream 0.12.x.
Two tasks:
1. In openshift/release, update merge-bot to sync CAPO main (current 4.19) to sync with upstream release-0.12 (instead of release-0.11)
2. In openshift/cluster-api-provider-openstack, wait for the bot to submit a sync and work out the PR to pass CI and merge.
This depends on OSASINFRA-3717. Per the discussion here, once that is complete and ORC is built by ART, we will need to update the PR for 2 to override the image.
Condition of satisfaction: ART has produced an image of CAPO synced to upstream 0.12.x.
In essence, customers can:
1) Pre-installation:
2) Installation:
3) Update:
4) Uninstallation/Deletion:
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Our customers will experience a streamlined approach to managing layered capabilities and workloads delivered through operators, operators packaged in Helm charts, or even plain Helm charts. The next generation OLM will power this central distribution mechanism within the OpenShift in the future.
Customers will be able to explore and discover the layered capabilities or workloads, and then install those offerings and make them available on their OpenShift clusters. Similar to the experience with the current OperatorHub, customers will be able to sort and filter the available offerings based on the delivery mechanism (i.e., operator-backed or plain helm charts), source type (i.e., from Red Hat or ISVs), valid subscriptions, infrastructure features, etc. Once click on a specific offering, they see the details which include the description, usage, and requirements of the offering, the provided services in APIs, and the rest of the relevant metadata for making the decisions.
The next-gen OLM aims to unify workload management. This includes operators packaged for current OLM, operators packaged in Helm charts, and even plain Helm charts for workloads. We want to leverage the current support for managing plain Helm charts within OpenShift and the console for leveraging our investment over the years.
Refer to the “Documentation Considerations” section of the OLM v1 GA feature.
The RFC written for https://github.com/operator-framework/catalogd/issues/426 identified a desire to formalize the catalogd web API, and divided work into a set of v1.0-blocking changes to enable versioned web interfaces ([phase 1](https://github.com/operator-framework/catalogd/issues/427)) and non-blocking changes to express and extend a formalized API specification (phase 2).
This epic is to track the design and implementation work associated with phase 2. During phase 1 RFC review we identified that we needed more work to capture the extensibility design but didn't want to slow progress on the v1.0 blocking changes so the first step should be an RFC to capture the design goals for phase 2, and then any implementation trackers we feel are necessary.
Work here will be behind a feature gate.
Downstreaming this feature
We need to follow this guide to downstream this feature: https://docs.google.com/document/d/1krN-4vwaE47aLRW9QjwD374-0Sh80taa_7goqVNi_-s/edit?tab=t.0#heading=h.9l63b4k04g9e
Modify cluster-olm-operator to watch for the new catalogd web api feature gate and implement reconciliation logic (openshift/ cluster-olm-operator)
See dev guide for more info: https://docs.google.com/document/d/1krN-4vwaE47aLRW9QjwD374-0Sh80taa_7goqVNi_-s/edit?tab=t.0#heading=h.th7069vduyim
This goals of this features are:
Given Microsoft's constraints on IPv4 usage, there is a pressing need to optimize IP allocation and management within Azure-hosted environments.
Interoperability Considerations
There's currently multiple ingress strategies we support for hosted cluster service endpoints (kas, nodePort, router...).
In a context of uncertainty about what use cases would be more critical to support, we initially exposed this in a flexible API that enables to potentially choose any combination of ingress strategies and endpoints.
ARO has internal restrictions on IPv4 usage. Because of this, to simplify the above and to be more cost effective in terms of infra we'd want to have a common shared ingress solution for all hosted clusters fleet.
Goal:
Graduate to GA (full support) Gateway API with Istio to unify the management of cluster ingress with a common, open, expressive, and extensible API.
Description:
Gateway API is the evolution of upstream Kubernetes Ingress APIs. The upstream project is part of Kubernetes, working under SIG-NETWORK. OpenShift is contributing to the development, building a leadership position, and preparing OpenShift to support Gateway API, with Istio as our supported implementation.
The plug-able nature of the implementation of Gateway API enables support for additional and optional 3rd-party Ingress technologies.
The team agrees, we should be running the upstream GWAPI conformance tests, as they are readily available and we are an integration product with GWAPI.
We need to answer these questions asked at the March 23, 2023 GWAPI team meeting:
Would it make sense to do it as an optional job in the cluster-ingress-operator?
Is OSSM running the Gateway Conformance test in their CI?
Review what other implementers do with conformance tests to understand what we should do (Do we fork the repo? Clone it? Make a new repo?)
Gateway API is the next generation of the Ingress API in upstream Kubernetes.
OpenShift Service Mesh (OSSM) and several other offering of ours like Kuadrant, Microshift and OpenShift AI all have critical dependencies on Gateway API's API resources. However, even though Gateway API is an official Kubernetes project its API resources are not available in the core API (like Ingress) and instead require the installation of Custom Resource Definitions (CRDs).
OCP will be fully in charge of managing the life-cycle of the Gateway API CRDs going forward. This will make the Gateway API a "core-like" API on OCP. If the CRDs are already present on a cluster when it upgrades to the new version where they are managed, the cluster admin is responsible for the safety of existing Gateway API implementations. The Cluster Ingress Operator (CIO) enacts a process called "CRD Management Succession" to ensure the transfer of control occurs safely, which includes multiple pre-upgrade checks and CIO startup checks.
The organization as a whole needs to be made aware of this as new projects will continue to pop up with Gateway API support over the years. This includes (but is not limited to)
Importantly our cluster infrastructure work with Cluster API (CAPI) is working through similar dilemmas for CAPI CRDs, and so we need to make sure to work directly with them as they've already broken a lot of ground here. Here are the relevant docs with the work they've done so far:
The purpose of this task is to provide API validation on OCP that blocks upgrades to Gateway API CRDs from all entities except the platform itself.
See the description of NE-1898.
We will use a Validating Admission Policy (VAP) to block ALL actions on the Gateway API CRDs from ALL entities besides the Cluster Ingress Operator (CIO).
Blocking in the VAP should occur at the group level, meaning only the CIO is capable of creating or changing any CRDs across the entire group at any version. As such this VAP will block access to ALL Gateway API CRDs, not just the ones we use (GatewayClass, Gateway, HTTPRoute, GRPCRoute, ReferenceGrant). Note that this means experimental APIs (e.g. TCPRoute, UDPRoute, TLSRoute) and older versions of APIs (e.g. v1beta1.HTTPRoute) are restricted as well from creation/modification. The effect should be that only the standard versions of GatewayClass, Gateway, HTTPRoute, GRPCRoute and ReferenceGrant (at the time of writing, these fully represent the standard APIs) are present and nobody can modify those, or deploy any others.
This VAP should be deployed alongside the CIO manifests, such that it is deployed along with the CIO itself.
Example of a VAP restricting actions to a single entity: https://github.com/openshift/cluster-cloud-controller-manager-operator/blob/master/pkg/cloud/azure/assets/validating-admission-policy.yaml
Here's where the current operator manifests can be found: https://github.com/openshift/cluster-ingress-operator/tree/edf5e71e8b08ef23a4d8f0b3fee5630c66625967/manifests
As a developer, I need a featuregate to develop behind so that the Gateway API work does not impact other development teams until tests pass.
The featuregate is currently in the DevPreviewNoUpgrade featureset. We need to graduate it to the TechPreviewNoUpgrade featureset to give us more CI signal and testing. Ultimately the featuregate needs to graduate to GA (default on) once tests pass so that the feature can GA.
See also:
Update the existing feature gate to enable Gateway API in clusters with either the DevPreviewNoUpgrade or TechPreviewNoUpgrade feature set.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
As an openshift developer, I want to extend the fields of ClusterImagePolicy CRD for BYOPKI verification, so the containerruntimeconfig controller can roll out the configurationto policy.json for verification.
OVN Kubernetes BGP support as a routing protocol for User Defined Network (Segmentation) pod and VM addressability.
OVN-Kubernetes BGP support enables the capability of dynamically exposing cluster scoped network entities into a provider’s network, as well as program BGP learned routes from the provider’s network into OVN.
OVN-Kubernetes currently has no native routing protocol integration, and relies on a Geneve overlay for east/west traffic, as well as third party operators to handle external network integration into the cluster. This enhancement adds support for BGP as a supported routing protocol with OVN-Kubernetes. The extent of this support will allow OVN-Kubernetes to integrate into different BGP user environments, enabling it to dynamically expose cluster scoped network entities into a provider’s network, as well as program BGP learned routes from the provider’s network into OVN. In a follow-on release, this enhancement will provide support for EVPN, which is a common data center networking fabric that relies on BGP.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Importing Routes from the Provider Network
Today in OpenShift there is no API for a user to be able to configure routes into OVN. In order for a user to change how cluster traffic is routed egress into the cluster, the user leverages local gateway mode, which forces egress traffic to hop through the Linux host's networking stack, where a user can configure routes inside of the host via NM State. This manual configuration would need to be performed and maintained across nodes and VRFs within each node.
Additionally, if a user chooses to not manage routes within the host and use local gateway mode, then by default traffic is always sent to the default gateway. The only other way to affect egress routing is by using the Multiple External Gateways (MEG) feature. With this feature the user may choose to have multiple different egress gateways per namespace to send traffic to.
As an alternative, configuring BGP peers and which route-targets to import would eliminate the need to manually configure routes in the host, and would allow dynamic routing updates based on changes in the provider’s network.
Exporting Routes into the Provider Network
There exists a need for provider networks to learn routes directly to services and pods today in Kubernetes. Metal LB is already one solution whereby load balancer IPs are advertised by BGP to provider networks, and this feature development does not intend to duplicate or replace the function of Metal LB. Metal LB should be able to interoperate with OVN-Kubernetes, and be responsible for advertising services to a provider’s network.
However, there is an alternative need to advertise pod IPs on the provider network. One use case is integration with 3rd party load balancers, where they terminate a load balancer and then send packets directly to OCP nodes with the destination IP address being the pod IP itself. Today these load balancers rely on custom operators to detect which node a pod is scheduled to and then add routes into its load balancer to send the packet to the right node.
By integrating BGP and advertising the pod subnets/addresses directly on the provider network, load balancers and other entities on the network would be able to reach the pod IPs directly.
Extending OVN-Kubernetes VRFs into the Provider Network
This is the most powerful motivation for bringing support of EVPN into OVN-Kubernetes. A previous development effort enabled the ability to create a network per namespace (VRF) in OVN-Kubernetes, allowing users to create multiple isolated networks for namespaces of pods. However, the VRFs terminate at node egress, and routes are leaked from the default VRF so that traffic is able to route out of the OCP node. With EVPN, we can now extend the VRFs into the provider network using a VPN. This unlocks the ability to have L3VPNs that extend across the provider networks.
Utilizing the EVPN Fabric as the Overlay for OVN-Kubernetes
In addition to extending VRFs to the outside world for ingress and egress, we can also leverage EVPN to handle extending VRFs into the fabric for east/west traffic. This is useful in EVPN DC deployments where EVPN is already being used in the TOR network, and there is no need to use a Geneve overlay. In this use case, both layer 2 (MAC-VRFs) and layer 3 (IP-VRFs) can be advertised directly to the EVPN fabric. One advantage of doing this is that with Layer 2 networks, broadcast, unknown-unicast and multicast (BUM) traffic is suppressed across the EVPN fabric. Therefore the flooding domain in L2 networks for this type of traffic is limited to the node.
Multi-homing, Link Redundancy, Fast Convergence
Extending the EVPN fabric to OCP nodes brings other added benefits that are not present in OCP natively today. In this design there are at least 2 physical NICs and links leaving the OCP node to the EVPN leaves. This provides link redundancy, and when coupled with BFD and mass withdrawal, it can also provide fast failover. Additionally, the links can be used by the EVPN fabric to utilize ECMP routing.
OVN Kubernetes support for BGP as a routing protocol.
Additional information on each of the above items can be found here: Networking Definition of Planned
...
1.
...
1. …
1. …
CNO should deploy the new RouteAdvertisements OVN-K CRD.
When the OCP API flag to enable BGP support in the cluster is set, CNO should enable support on OVN-K through a CLI arg.
Customers in highly regulated environments are required to adopt strong ciphers. For control-plane components, this means all components must support the modern TLSProfile with TLS 1.3.
Note: Post-Quantum Cryptography support in TLS will ONLY be available with TLS 1.3, thus support is required. For more information about PQC support, see https://issues.redhat.com/browse/OCPSTRAT-1858)
During internal discussions [1] for RFE-4992 and follow-up conversations, it became clear we need a dedicated CI job to track and validate that OpenShift control-plane components are aligned and functional with the TLSProfiles configured on the system.
[1] https://redhat-internal.slack.com/archives/CB48XQ4KZ/p1713288937307359
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | both |
Classic (standalone cluster) | yes |
Hosted control planes | yes |
Multi node, Compact (three node), or Single node (SNO), or all | all |
Connected / Restricted Network | all |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | all |
Operator compatibility | ?? |
Backport needed (list applicable versions) | n/a |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | n/a |
Other (please specify) | unknown |
If we try to enable a Modern TLS profile:
EnvVarControllerDegraded: no supported cipherSuites not found in observedConfig
also, if we do manage to pass along the Modern TLS profile cipher suit, we see:
http2: TLSConfig.CipherSuites is missing an HTTP/2-required AES_128_GCM_SHA256 cipher (need at least one of TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 or TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256)
This is Image mode on OpenShift. It uses the rpm-ostree native containers interface and not bootc but that is an implementation detail.
In the initial delivery of CoreOS Layering, it is required that administrators provide their own build environment to customize RHCOS images. That could be a traditional RHEL environment or potentially an enterprising administrator with some knowledge of OCP Builds could set theirs up on-cluster.
The primary virtue of an on-cluster build path is to continue using the cluster to manage the cluster. No external dependency, batteries-included.
On-cluster, automated RHCOS Layering builds are important for multiple reasons:
This work describes the tech preview state of On Cluster Builds. Major interfaces should be agreed upon at the end of this state.
As a cluster admin of user provided infrastructure,
when I apply the machine config that opts a pool into On Cluster Layering,
I want to also be able to remove that config and have the pool revert back to its non-layered state with the previously applied config.
As a cluster admin using on cluster layering,
when an image build has failed,
I want it to retry 3 times automatically without my intervention and show me where to find the log of the failure.
As a cluster admin,
when I enable On Cluster Layering,
I want to know that the builder image I am building with is stable and will not change unless I change it
so that I keep the same API promises as we do elsewhere in the platform.
To test:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster and the Cluster Version Operator is not available,
I want the upgrade operation to be blocked.
As a cluster admin,
when I use a disconnected environment,
I want to still be able to use On Cluster Layering.
As a cluster admin using On Cluster layering,
When there has been config drift of any sort that degrades a node and I have resolved the issue,
I want to it to resync without forcing a reboot.
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and references an internal registry
I want that registry available on the host network so that the pool can successfully scale up
(MCO-770, MCO-578, MCO-574 )
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and I want to scale up nodes,
the nodes should have the same config as the other nodes in the pool.
Maybe:
Entitlements: MCO-1097, MCO-1099
Not Likely:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster,
I want the upgrade operation to succeed at the same rate as non-OCL upgrades do.
As a follow up to https://issues.redhat.com/browse/MCO-1284, the one field we identified that is best updated pre-GA is to make the baseImagePullSecret optional. The builder pod should have the logic to fetch from baseImagePullSecret if the user does not specify this via a MachineOSConfig object.
To make OCL ready for GA, the first step would be graduating the MCO's APIs from v1alpha1 to v1. This requires changes in the openshift/api repo.
This work describes the tech preview state of On Cluster Builds. Major interfaces should be agreed upon at the end of this state.
As a cluster admin of user provided infrastructure,
when I apply the machine config that opts a pool into On Cluster Layering,
I want to also be able to remove that config and have the pool revert back to its non-layered state with the previously applied config.
As a cluster admin using on cluster layering,
when an image build has failed,
I want it to retry 3 times automatically without my intervention and show me where to find the log of the failure.
As a cluster admin,
when I enable On Cluster Layering,
I want to know that the builder image I am building with is stable and will not change unless I change it
so that I keep the same API promises as we do elsewhere in the platform.
To test:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster and the Cluster Version Operator is not available,
I want the upgrade operation to be blocked.
As a cluster admin,
when I use a disconnected environment,
I want to still be able to use On Cluster Layering.
As a cluster admin using On Cluster layering,
When there has been config drift of any sort that degrades a node and I have resolved the issue,
I want to it to resync without forcing a reboot.
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and references an internal registry
I want that registry available on the host network so that the pool can successfully scale up
(MCO-770, MCO-578, MCO-574 )
As a cluster admin using on cluster layering,
when a pool is using on cluster layering and I want to scale up nodes,
the nodes should have the same config as the other nodes in the pool.
Maybe:
Entitlements: MCO-1097, MCO-1099
Not Likely:
As a cluster admin using on cluster layering,
when I try to upgrade my cluster,
I want the upgrade operation to succeed at the same rate as non-OCL upgrades do.
The original scope of this task is represented across this story & MCO-1494.
With OCL GA'ing soon, we'll need a blocking path within our e2e test suite that must pass before a PR can be merged. This story represents the first stage in creating the blocking path:
Description of problem:
Intermittently, when OCL is disabled and the nodes are updated with the new non-ocl image the process is not honoring the maxUnavailable value in the MCP.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Intermittent
Steps to Reproduce:
In the second must-gather file the issue happened when we did: 1. Enable OCL in the worker pool 2. Remove the MOSC resource to disable OCL The result was that the worker pool had 2 nodes being drained at the same time.
Actual results:
The worker pool has 2 nodes being drained at the same time.
Expected results:
Since MCP has no maxUnavailable value defined, the default value is 1. Hence, there should be only 1 node being drained at a time. Moreover, they should be updated in the right order.
Additional info:
[REGRESSION] We need to reinvent the wheel for triggering rebuild functionality and the rebuild mechanism as pool labeling and annotation is no longer a favorable way to interact with layered pools
There are a few situations in which a cluster admin might want to trigger a rebuild of their OS image in addition to situations where cluster state may dictate that we should perform a rebuild. For example, if the custom Dockerfile changes or the machine-config-osimageurl changes, it would be desirable to perform a rebuild in that case. To that end, this particular story covers adding the foundation for a rebuild mechanism in the form of an annotation that can be applied to the target MachineConfigPool. What is out of scope for this story is applying this annotation in response to a change in cluster state (e.g., custom Dockerfile change).
Done When:
Today VMs for a single nodepool can “clump” together on a single node after the infra cluster is updated. This is due to live migration shuffling around the VMs in ways that can result in VMs from the same nodepool being placed next to each other.
Through a combination of TopologySpreadConstraints and the De-Scheduler, it should be possible to continually redistributed VMs in a nodepool (via live migration) when clumping occurs. This will provide stronger HA guarantees for nodepools
VMs within a nodepool should re-distribute via live migration in order to best satisfy topology spread constraints.
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Today VMs for a single nodepool can “clump” together on a single node after the infra cluster is updated. This is due to live migration shuffling around the VMs in ways that can result in VMs from the same nodepool being placed next to each other.
Through a combination of TopologySpreadConstraints and the De-Scheduler, it should be possible to continually redistributed VMs in a nodepool (via live migration) when clumping occurs. This will provide stronger HA guarantees for nodepools
VMs within a nodepool should re-distribute via live migration in order to best satisfy topology spread constraints.
In the hypershift upstream documentation, outline how the de-scheduler can be used to continually redistribute VMs in a nodepool when clumping of VMs occur after live migration.
As a cluster admin for standalone OpenShift, I want to customize the prefix of the machine names created by CPMS due to company policies related to nomenclature. Implement the Control Plane Machine Set (CPMS) feature in OpenShift to support machine names where user can set custom names prefixes. Note the prefix will always be suffixed by "<5-chars>-<index>" as this is part of the CPMS internal design.
A new field called machineNamePrefix has been added to CPMS CR.
This field would allow the customer to specify a custom prefix for the machine names.
The machine names would then be generated using the format: <machineNamePrefix><5-chars><index>
Where:
<machineNamePrefix> is the custom prefix provided by the customer
<5-chars> is a random 5 character string (this is required and cannot be changed)
<index> represents the index of the machine (0, 1, 2, etc.)
Ensure that if the machineNamePrefix is changed, the operator reconciles and succeeds in rolling out the changes.
Bump openshift/api to vendor machineNamePrefix field and CPMSMachineNamePrefix feature-gate into cpms-operator
Define a new feature gate in openshift/api for this feature so that all the implementation can be safe guarded behind this gate.
To be able to gather test data for this feature, we will need to introduce tech preview periodics, so we need to duplicate each of https://github.com/openshift/release/blob/8f8c7c981c3d81c943363a9435b6c48005eec6e3[…]control-plane-machine-set-operator-release-4.19__periodics.yaml and add the techpreview configuration.
It's configured as an env var, so copy each job, add the env var, and change the name to include -techpreview as a suffic
env: FEATURE_SET: TechPreviewNoUpgrade
Description of problem:
Consecutive points are not allowed in machineNamePrefix, but from the prompt "spec.machineNamePrefix: Invalid value: "string": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character.", consecutive points should be allowed. And I can create machine on providers console with consecutive points https://drive.google.com/file/d/1p5QLhkL4VI3tt3viO98zYG8uqb1ePTnB/view?usp=sharing
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-01-08-165032
How reproducible:
Always
Steps to Reproduce:
1.Update machineNamePrefix containing consecutive points in controlplanemachineset, like machineNamePrefix: huliu..azure 2.Cannot save, get prompt # * spec.machineNamePrefix: Invalid value: "string": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character. 3. If change to machineNamePrefix: huliu.azure then it can save successfully, and rollout succeed.
Actual results:
Cannot save, get prompt
Expected results:
Can save successfully
Additional info:
New feature testing for https://issues.redhat.com/browse/OAPE-22
Provide a new field to the CPMS that allows to define a Machine name prefix
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Include `CPMSMachineNamePrefix` feature gate name in e2e tests.
This is required for sippy tool to filter the e2e tests specific to this featuregate.
xRef: https://github.com/openshift/api?tab=readme-ov-file#defining-featuregate-e2e-tests
Enable OpenShift to be deployed on Confidential VMs on GCP using Intel TDX technology
Users deploying OpenShift on GCP can choose to deploy Confidential VMs using Intel TDX technology to rely on confidential computing to secure the data in use
As a user, I can choose OpenShift Nodes to be deployed with the Confidential VM capability on GCP using Intel TDX technology at install time
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
This is a piece of a higher-level effort to secure data in use with OpenShift on every platform
Documentation on how to use this new option must be added as usual
Edge customers requiring computing on-site to serve business applications (e.g., point of sale, security & control applications, AI inference) are asking for a 2-node HA solution for their environments. Only two nodes at the edge, because the 3d node induces too much cost, but still they need HA for critical workload. To address this need, a 2+1 topology is introduced. It supports a small cheap arbiter node that can optionally be remote/virtual to reduce onsite HW cost.
Support OpenShift on 2+1 topology, meaning two primary nodes with large capacity to run workload and control plan, and a third small “arbiter” node which ensure quorum. See requirements for more details.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | self-managed |
Classic (standalone cluster) | yes |
Hosted control planes | no |
Multi node, Compact (three node), or Single node (SNO), or all | Multi node and Compact (three node) |
Connected / Restricted Network | both |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86_86 and ARM |
Operator compatibility | full |
Backport needed (list applicable versions) | no |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | no |
Other (please specify) | n/a |
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
See requirements - there are two main groups of customers: co-located arbiter node, and remote arbiter node.
Authentication Operator will need to be updated to take in the newly created `HighlyAvailableArbiter` topology flag, and allow the minimum of kube-apiservers running to equal 2.
We need to add the `HighlyAvailableArbiter` value to the controlPlaneTopology in ocp/api package as a TechPreview
https://github.com/openshift/api/blob/master/config/v1/types_infrastructure.go#L95-L103
We need to update the installer to support the machines for the arbiter node during install.
This will be flushed out with explicit direction, currently discussing next steps in EP comments.
Bump the openshift/api dependency to latest for the arbiter node enum.
We will need to make sure the MCO contains the bootstrap files for the arbiter node similar to the files it contains for master nodes.
The console operator performs a check for HighlyAvailable, we will need to add the HighlyAvailableArbiter check there so that it supports it on the same level as HA
The console operand checks for valid infra control plane topology fields.
We'll need to add the HighlyAvailableArbiter flag to that check
Once the HighlyAvailableArbiter has been added to the ocp/api, we need to update the cluster-config-operator dependencies to reference the new change, so that it propagates to cluster installs in our payloads.
If arbiter is set in the install config, then we want to set it's default values.
We need to update CEO (cluster etcd operator) to understand what an arbiter/witness node is so it can properly assign an etcd member on our less powerful node.
Update the dependencies for CEO for library-go and ocp/api to support the Arbiter additions, doing this in a separate PR to keep things clean and easier to test.
We need to make sure kubelet respects the new node role type for arbiter. This will be a simple update on the well_known_openshift_labels.go list to include the new node role.
Make sure pacemaker configured in TNF controls etcd.
Verify the following scenarios:
For TNF we need to replace our currently running etcd after the installation with the one that is managed by pacemaker.
This allows us to keep the following benefits:
AC:
As a developer of TNF, I need:
Acceptance Criteria
As a developer of 2NO, I need:
Acceptance Criteria
This epic is used to track tasks/stories delivered from OCP core engineering control plane gorup.
As a developer of TNF, I need:
Acceptance Criteria
As a developer of TNF, I need:
Acceptance Criteria
As a developer of TNF, I need:
Acceptance Criteria
We need to bump the go.mod for the cluster-config-operator to make sure the DualReplica CRD is applied at bootstrap after we merge in the topology change to the ocp/api repo.
As a developer of 2NO, I need:
Acceptance Criteria
As an OpenShift Container Platform (OCP) user, I will be able to install the GA version of OpenShift Lightspeed (OLS) from the Operator Hub and access information about OLS in the OCP documentation.
1) disconnected
2) FIPS
3) Multi arch -> Arm for GA
4) HCP
5) Konflux
As a OLS developer, I want to verify if OLS can be configured on custom namespace on 2 different HCP clusters provisioned from same Management Cluster, so that SD team can run on their HCP clusters.
Support mapping OpenShift zones to vSphere host groups, in addition to vSphere clusters.
When defining zones for vSphere administrators can map regions to vSphere datacenters and zones to vSphere clusters.
There are use cases where vSphere clusters have only one cluster construct with all their ESXi hosts but the administrators want to divide the ESXi hosts in host groups. A common example is vSphere stretched clusters, where there is only one logical vSphere cluster but the ESXi nodes are distributed across to physical sites, and grouped by site in vSphere host groups.
In order for OpenShift to be able to distribute its nodes on vSphere matching the physical grouping of hosts, OpenShift zones have to be able to map to vSphere host groups too.
Support mapping OpenShift zones to vSphere host groups, in addition to vSphere clusters.
When defining zones for vSphere administrators can map regions to vSphere datacenters and zones to vSphere clusters.
There are use cases where vSphere clusters have only one cluster construct with all their ESXi hosts but the administrators want to divide the ESXi hosts in host groups. A common example is vSphere stretched clusters, where there is only one logical vSphere cluster but the ESXi nodes are distributed across to physical sites, and grouped by site in vSphere host groups.
In order for OpenShift to be able to distribute its nodes on vSphere matching the physical grouping of hosts, OpenShift zones have to be able to map to vSphere host groups too.
{}USER STORY:{}
As someone that installs openshift on vsphere, I want to install zonal via host and vm groups so that I can use a stretched physical cluster or use a cluster as a region and hosts as zones .
{}DESCRIPTION:{}
{}Required:{}
{}Nice to have:{}
{}ACCEPTANCE CRITERIA:{}
{}ENGINEERING DETAILS:{}
Configuration steps:
https://github.com/openshift/installer/compare/master...jcpowermac:installer:test-vm-host-group
As an openshift engineer enable host vm group zonal in mao so that compute nodes properly are deployed
Acceptance Criteria:
As an openshift engineer enable host vm group zonal in CPMS so that control plane nodes properly are redeployed
Acceptance Criteria:
As an openshift engineer update the infrastructure and the machine api objects so it can support host vm group zonal
Acceptance criteria
Overarching Goal
Move to using the upstream Cluster API (CAPI) in place of the current implementation of the Machine API for standalone Openshift.
Phase 1 & 2 covers implementing base functionality for CAPI.
Phase 2 also covers migrating MAPI resources to CAPI.
Phase 2 Goal:
for Phase-1, incorporating the assets from different repositories to simplify asset management.
There must be no negative effect to customers/users of the MAPI, this API must continue to be accessible to them though how it is implemented "under the covers" and if that implementation leverages CAPI is open
Update: We did this for the corecluster controller, we will create separate cards to do it for the other cluster-capi-operator controllers too.
–
The current ClusterOperator status conditions—Available, Progressing, Upgradeable, and Degraded—are set by the corecluster controller independently of the status of other controllers.
This approach does not align with the intended purpose of these conditions, which are meant to reflect the overall status of the operator, considering all the controllers it manages.
To address this, we should introduce controller-level conditions similar to the top-level ones. These conditions would influence an aggregated top-level status, which a new controller (the ClusterOperator controller) would then consolidate into the Available, Progressing, Upgradeable, and Degraded conditions.
Moreover, when running `oc get co` against the cluster-capi-operator, only the name and version are returned. The status is not rolled up into the additional columns as expected.
Before GA, this state information should be visible from `oc get co`
OpenShift is traditionally a single-disk system, meaning the OpenShift installer is designed to install the OpenShift filesystem on one disk. However, new customer use cases have highlighted the need for the ability to configure additional disks during installation. Here are the specific use cases:
User Story:
As an OpenShift administrator, I need to be able to configure my OpenShift cluster to have additional disks on each vSphere VM so that I can use the new data disks for various OS needs.
Description:
This goal of this epic is to be able to allow the cluster administrator to install and configure after install new machines with additional disks attached to each virtual machine for various OS needs.
Required:
Nice to Have:
Acceptance Criteria:
Notes:
User Story:
As an OpenShift Engineer I need to create a new feature gate and CRD changes for vSphere multi disk so that we can gate the new function until all bugs are ironed out.
Description:
This task is to create the new feature gate to be used by all logical changes around multi disk support for vSphere. We also need to update the types for vsphere machine spec to include new array field that contains data disk definitions.
Acceptance Criteria:
- New feature gate exists for components to use.
- Initial changes to the CRD for data disks are present for components to start using.
User Story:
As an OpenShift Engineer I need to ensure the the MAPI Operator.
Description:
This task is to verify if any changes are needed in the MAPI Operator to handle the change data disk definitions in the CPMS.
Acceptance Criteria:
- Adding a disk to MachineSet does not result in new machines being rolled out.
- Removing a disk from MachineSet does not result in new machines being rolled out.
- After making changes to a MachineSet related to data disks, when MachineSet is scaled down and then up, new machines contain the new data disk configurations.
- All attempts to modify existing data disk definitions in an existing Machine definition are blocked by the webhook.
Notes:
The behaviors for the data disk field should be the same as all other provider spec level fields. We want to make sure that the new fields are no different than the others. This field is not hot swap capable for running machines. A new VM must be created for this feature to work.
User Story:
As an OpenShift Engineer I need to enhance the OpenShift installer to support creating a cluster with additional disks added to control plane and compute nodes so that I can use the new data disks for various OS needs.
Description:
This task is to enhance the installer to allow configuring data disks in the install-config.yaml. This will also require setting the necessary fields in machineset and machine definitions. The important one being for CAPV to do the initial creation of disks for the configured masters.
Acceptance Criteria:
- install-config.yaml supports configuring data disks in all machinepools.
- CAPV has been updated with new multi disk support.
- CAPV machines are created that result in control plane nodes with data disks.
- MachineSet definitions for compute nodes are created correctly with data disk values from compute pool.
- CPMS definition for masters has the data disks configured correctly.
Notes:
We need to be sure that after installing a cluster, the cluster remains stable and has all correct configurations.
User Story:
As an OpenShift Engineer I need to ensure the the CPMS Operator now works with detecting any changes needed when data disks are added to the CPMS definition.
Description:
This task is to verify if any changes are needed in the CPMS Operator to handle the change data disk definitions in the CPMS.
Acceptance Criteria:
- CPMS does not roll out changes when initial install is performed.
- Adding a disk to CPMS results in control plane roll out.
- Removing a disk from CPMS results in control plane roll out.
- No errors logged as a result of data disks being present in the CPMS definition.
Notes:
Ideally we just need to make sure the operator is updated to pull in the new CRD object definitions that contain the new data disk field.
GA User Name Space in OpenShift 4.19
continue work from https://issues.redhat.com/browse/OCPSTRAT-207
https://github.com/openshift/kubernetes/pull/2104 and https://github.com/openshift/cluster-kube-apiserver-operator/pull/1754 have some updated comments
As part of being a first party Azure offering, ARO HCP needs to adhere to Microsoft secure supply chain software requirements. In order to do this, we require setting a label on all pods that run in the hosted cluster namespace.
Implement Mechanism for Labeling Hosted Cluster Control Plane Pods
kubernetes.azure.com/managedby: sub_1d3378d3-5a3f-4712-85a1-2485495dfc4b
Review, refine and harden the CAPI-based Installer implementation introduced in 4.16
From the implementation of the CAPI-based Installer started with OpenShift 4.16 there is some technical debt that needs to be reviewed and addressed to refine and harden this new installation architecture.
Review existing implementation, refine as required and harden as possible to remove all the existing technical debt
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
There should not be any user-facing documentation required for this work
We need a place to add tasks that are not feature oriented.
Most of the platform subdirectories don't have OWNERS files
we should add the aliases for everything that's missing
backport to 4.16
This feature aims to comprehensively refactor and standardize various components across HCP, ensuring consistency, maintainability, and reliability. The overarching goal to increase customer satisfaction by increasing speed to market and saving engineering budget by reducing incidents/bugs. This will be achieved by reducing technical debt, improving code quality, and simplifying the developer experience across multiple areas, including CLI consistency, NodePool upgrade mechanisms, networking flows, and more. By addressing these areas holistically, the project aims to create a more sustainable and scalable codebase that is easier to maintain and extend.
Over time, the HyperShift project has grown organically, leading to areas of redundancy, inconsistency, and technical debt. This comprehensive refactor and standardization effort is a response to these challenges, aiming to improve the project's overall health and sustainability. By addressing multiple components in a coordinated way, the goal is to set a solid foundation for future growth and development.
Ensure all relevant project documentation is updated to reflect the refactored components, new abstractions, and standardized workflows.
This overarching feature is designed to unify and streamline the HCP project, delivering a more consistent, maintainable, and reliable platform for developers, operators, and users.
Goal
Refactor and modularize controllers and other components to improve maintainability, scalability, and ease of use.
As a (user persona), I want to be able to:
https://issues.redhat.com//browse/HOSTEDCP-1801 introduced a new abstraction to be used by ControlPlane components. However, when a component or a sub-resources predicate changes to false, the resources are not removed from the cluster. All resources should be deleted from the cluster.
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
As a (user persona), I want to be able to:
https://issues.redhat.com//browse/HOSTEDCP-1801 introduced a new abstraction to be used by ControlPlane components. We need to refactor every component to use this abstraction.
Description of criteria:
All ControlPlane Components are refactored:
Example PR to refactor cloud-credential-operator : https://github.com/openshift/hypershift/pull/5203
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Provide a PR with a CSI standard refactor
Example PR to refactor HCCO: https://github.com/openshift/hypershift/pull/4860
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Provide a PR with a CNO standard refactor
Example PR to refactor HCCO: https://github.com/openshift/hypershift/pull/4860
docs: https://github.com/openshift/hypershift/blob/main/support/controlplane-component/README.md
Improve the consistency and reliability of APIs by enforcing immutability and clarifying service publish strategy support.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Allow customers to migrate CNS volumes (i.e vsphere CSI volumes) from one datastore to another.
This operator relies on a new VMware CNS API and requires 8.0.2 or 7.0 Update 3o minimum versions
https://docs.vmware.com/en/VMware-vSphere/8.0/rn/vsphere-vcenter-server-802-release-notes/index.html
In 4.17 we shipped a devpreview CLI tool (OCPSTRAT-1619) to cover existing urgent requests. This CLI tool will be removed as soon as this feature is available in OCP.
Often our customers are looking to migrate volumes between datastores because they are running out of space in current datastore or want to move to more performant datastore. Previously this was almost impossible or required modifying PV specs by hand to accomplish this. It was also very error prone.
As a first version, we developed a CLI tool that is shipped as part of the vsphere CSI operator. We keep this tooling internal for now, support can guide customers on a per request basis. This is to manage current urgent customer's requests, a CLI tool is easier and faster to develop it can also easily be used in previous OCP releases.
After multiple discussion with VMware we now have confidence that we can rely on their built-in vSphere UI tool to migrate CNS volume from one datastore to another. This includes attached and detached volumes. Vmware confirmed they have confidence in this scenario and they fully support this operation for attached volumes.
SInce the feature is external to OCP, it is mostly a matter of testing it works as expected with OCP but customers will be redirected to Vmware documentation as all the steps are done through the vSphere UI.
Perform testing for attached and detached volumes + special cases such as RWX, zonal, encrypted.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | both |
Classic (standalone cluster) | Yes |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | Yes |
Connected / Restricted Network | both |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86 |
Operator compatibility | vsphere CSI operator but feature is external to OCP |
Backport needed (list applicable versions) | no |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | no done through vsphere UI |
Other (please specify) | OCP on vsphere only |
As a admin - want to migrate all my PVs or optional PVCs belonging to certain namespace to a different datastore within cluster without potentially requiring extended downtime.
Get full support confirmation from vmware that their CNS volume migration feature
Limited to what VMware supports. At the moment only one volume can be migrated at a time.
We had a lot of requests to migrate volumes between datastore for multiple reason. Up until now it was not natively supported by VMware. In 8.0.2 they added a CNS API and a vsphere UI feature to perform volume migration.
In 4.17 we shipped a devpreview CLI tool (OCPSTRAT-1619) to cover existing urgent requests. This CLI tool will be removed as soon as this feature is available in OCP.
This feature also includes the work needed to remove the CLI tool
Need to be explicit on requirements and limitations.
Documented as part of the vsphere CSI OCP documentation.
Specify min vsphere version. Document any limitation found during testing
Redirect to vmware documentation.
Announce removal of the CLI tool + update KB.
OCP on vSphere only
Epic Goal*
We need to document and support CNS volume migration using native vCenter UI, so as customers can migrate volumes between datastores.
Why is this important? (mandatory)
Often our customers are looking to migrate volumes between datastores because they are running out of space in current datastore or want to move to more performant datastore. Previously this was almost impossible or required modifying PV specs by hand to accomplish this. It was also very error prone.
Scenarios (mandatory)
As an vCenter/Openshift admin, I want to migrate CNS volumes between datastores for existing vSphere CSI persistent volumes (PVs).
This should cover attached and detached volumes. Special cases such as RWX, zonal or encrypted should also be tested to confirm is there is any limitation we should document.
Dependencies (internal and external) (mandatory)
This feature depends on VMware vCenter Server 7.0 Update 3o or vCenter Server 8.0 Update 2.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
This is mostly a testing / documentation epic, which will change current wording about unsupported CNS volume migration using vCenter UI.
As part of this epic, we also want to remove the CLI tool we developed for https://github.com/openshift/vmware-vsphere-csi-driver-operator/blob/master/docs/cns-migration.md from the payload.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
We have decided to remove cns-migration CLI tool for now.
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
Upgrade the OCP console to Pattern Fly 6.
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
The core OCP Console should be upgraded to PF 6 and the Dynamic Plugin Framework should add support for PF6 and deprecate PF4.
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
Console, Dynamic Plugin Framework, Dynamic Plugin Template, and Examples all should be upgraded to PF6 and all PF4 code should be removed.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
As a company we have all agreed to getting our products to look and feel the same. The current level is PF6.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Console is adopting PF6 and removing the PF4 support. It creates lots of UI issues in the Developer Console which we need to support to fix.
Fix all the UI issues in the ODC related to PF6 upgrade
Description of problem:
Label/tag on software catalog card is left align but it should be align right.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Navigate to software catalog 2. 3.
Actual results:
Label/tag on software catalog card is left align
Expected results:
Label/tag on software catalog card should be right align
Additional info:
Per slack discussion, the following files contain invalid tokens that need to be fixed/updated/removed:
knative-plugin/src/components/functions/GettingStartedSection.scss
pipelines-plugin/src/components/pipelineruns/PipelineRunDetailsPage.scss
pipelines-plugin/src/components/pipelineruns/list-page/PipelineRunList.scss
pipelines-plugin/src/components/pipelines/detail-page-tabs/pipeline-details/PipelineVisualizationStepList.scss
pipelines-plugin/src/components/taskruns/TaskRunDetailsPage.scss
pipelines-plugin/src/components/taskruns/list-page/TaskRunsRow.scss
topology/src/components/graph-view/components/GraphComponent.scss
topology/src/components/graph-view/components/edges/ServiceBinding.scss
topology/src/components/graph-view/components/groups/GroupNode.scss
topology/src/components/page/TopologyView.scss
The background color of the tab should be solid and not transparent
The layout and functionality of the Edit upstream configuration modal could be improved.
https://github.com/patternfly/patternfly/issues/7256 is a bug in PatternFly that impacts console. Once it is fixed upstream, we need to update PatternFly to a version that includes the fix.
Some of the PatternFly releases in https://github.com/openshift/console/pull/14621 are prereleases. Once final releases are available (v.6.2.0 is scheduled for the end of March), we should update to them.
Also update https://github.com/openshift/console/blob/900c19673f6f3cebc1b57b6a0a9cadd1573950d9/dynamic-demo-plugin/package.json#L21-L24, and https://github.com/openshift/console-crontab-plugin to the same versions.
Before we can adopt PatternFly 6, we need to drop PatternFly 4. We should drop 4 first so we can understand what impact if any that will have on plugins.
AC:
https://github.com/openshift/console/blob/master/frontend/packages/console-shared/src/components/actions/menu/ActionMenuItem.tsx#L3 contains a reference to `@patternfly/react-core/deprecated`. In order to drop PF4 and adopt PF6, this reference needs to be removed.
This component was never finished and should be removed as it includes a reference to `@patternfly/react-core/deprecated`, which blocks the removal of PF4 and the adoption of PF6.
AC:
Check https://www.patternfly.org/get-started/upgrade/#potential-test-failures
The API Explorer > Resource details > Access review page utilizes a custom Checkbox filter. This custom component is unnecessary as PatternFly offers comparable functionality. We should replace this customer component with a PatternFly one.
AC: Replace the CheckBox component for the Switch in the API Explorer
Instances of LogViewer are hard coded to use the dark theme. We should make that responsive to the user's choice the same we we are with the CodeEditor.
The checkboxes at the top of the ResourceLog component should be changed to Switches as that same change is being made for the YAMLEditor.
AC: Replace CheckBox component in the ResourceLog component with Switch from PF6
Throughout the code base, there are many instances of `<h1>` - `<h6>`. As a result, we have to manually style these elements to align with PatternFly. By replacing the html elements with a PatternFly component, we get the correct styling for free.
PopupKebabMenu is orphaned and contains a reference to `@patternfly/react-core/deprecated`. It and related code should be removed so we can drop PF4 and adopt PF6.
The dynamic-demo-plugin in the openshift/console repository currently relies on PatternFly 5. To align with the latest design standards and to ensure consistency with other parts of the platform, the plugin must be upgraded to use PatternFly 6.
This involves updating dependencies, refactoring code where necessary to comply with PatternFly 6 components and APIs, and testing to ensure that functionality and UI are not disrupted by the changes.
AC:
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
<your text here>
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
As a user, I want to use Topology components in the dynamic plugin
getModifyApplicationAction
baseDataModelGetter
getWorkloadResources
contextMenuActions
CreateConnector
createConnectorCallback (e
Description:
Update ODC e2e automation tests
Acceptance Criteria
As a user, I want to favorite pages in the Console admin perspective
Design https://www.figma.com/design/CMGLcRam4M523WVDqdugWz/Favoriting?node-id=0-1&p=f&t=y07SBX01YxDa6pTv-0
Description of problem:
Namespace does not persist when switching to developer view from the topology page of the admin page. This is based on the changes from pr https://github.com/openshift/console/pull/14588
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Go to topology page on ain view after login 2. Select/Create a namespace 3. Switch to developer perspective
Actual results:
Namespace do not persist
Expected results:
Namespace should persist
Additional info:
https://drive.google.com/file/d/1VYg-pWt4ZCYKtmPx4bIK6L2Lkt3WpOPv/view?usp=sharing
As a user, I want to see actions regarding the merge perspective in getting started resources section
As a user, I want to know how I can enable the developer perspective in the Web console
Steps to enable dev perspective through UI
Description of problem:
Disable Guided tour in admin perspective for now as e2e is failing because of it and will re-enable it once we fix the e2e.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
As a user, I want to know about the nav option added to the Admin perspective from the Developer perspective.
https://www.figma.com/proto/xSf03qhQdjL1JP0VXegckQ/Navigation?node-id=262-2537
As a user, I want to use Topology components in the dynamic plugin
Utils and component needs to be exposed
https://docs.google.com/spreadsheets/d/1B0TLMtRY9uUR-ma0po3ma0rwgEE7T5w-v-k6f0Ak_tk/edit?gid=0#gid=0
As a developer, I do not want to maintain the code for a project already dead.
K8s 1.31 introduces VolumeAttributesClass as beta (code in external provisioner). We should make it available to customers as tech preview.
VolumeAttributesClass allows PVC to be modified after their creation and while attached. There is as vast number of parameters that can be updated but the most popular is to change the QoS values. Parameters that can be changed depend on the driver.
Productise VolumeAttributesClass as TP in anticipation for GA. Customer can start testing VolumeAttributesClass.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | both |
Classic (standalone cluster) | yes |
Hosted control planes | yes |
Multi node, Compact (three node), or Single node (SNO), or all | all |
Connected / Restricted Network | both |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | all |
Operator compatibility | N/A core storage |
Backport needed (list applicable versions) | None |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | TBD for TP |
Other (please specify) | n/A |
As an OCP user, I want to change parameters of my existing PVC such as the QoS attributes.
Get list of drivers that supports it (from the ones we ship)
No UI for TP
There's been some limitations and complains on the fact that PVC attributes are sealed after their creation avoiding customers to update them. This is particularly impacting for a specific QoS is set and the volume requirements are changing.
Customer should not use it in production atm. The driver used by customers must support this feature.
Document VolumeAttributesClass creation and how to update PVC. Mention any limitation. Mention it's tech preview no upgrade. Add drivers support if needed.
Check which drivers support it for which parameters.
Support upstream feature "VolumeAttributesClass" in OCP as Beta, i.e. test it and have docs for it.
Red Hat Product Security recommends that pods be deployed with readOnlyRootFilesystem set to true in the SecurityContext, but does not require it because a successful attack can only be carried out with a combination of weaknesses and OpenShift runs with a variety of mitigating controls.
However, customers are increasingly asking questions about why pods from Red Hat, and deployed as part of OpenShift, do not follow common hardening recommendations.
Note that setting readOnlyRootFilesystem to true ensures that the container's root filesystem is mounted as read-only. This setting has nothing to do with host access.
For more information, see
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
Setting the readOnlyRootFilesystem flag to true reduces the attack surface of your containers, preventing an attacker from manipulating the contents of your container and its root file system.
If your container needs to write temporary files, you can specify the ability to mount an emptyDir in the Security Context for your pod as described here. https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
The following containers have been identified by customer scans as needing remediation. If your pod will not function with readOnlyRootFilesystem set to true, please document why so that we can document the reason for the exception.
1. Proposed title of this feature request
[openshift-cloud-credential-operator] - readOnlyRootFilesystem should be explicitly to true and if required to false for security reason
2. What is the nature and description of the request?
According to security best practice, it's recommended to set readOnlyRootFilesystem: true for all containers running on kubernetes. Given that openshift-cloud-credential-operator does not set that explicitly, it's requested that this is being evaluated and if possible set to readOnlyRootFilesystem: true or otherwise to readOnlyRootFilesystem: false with a potential explanation why the file-system needs to be write-able.
3. Why does the customer need this? (List the business requirements here)
Extensive security audits are run on OpenShift Container Platform 4 and are highlighting that many vendor specific container is missing to set readOnlyRootFilesystem: true or else justify why readOnlyRootFilesystem: false is set.
4. List any affected packages or components.
openshift-cloud-credential-operator
Enable readOnlyRootFilesystem on all of the cloud-credential-operator pods. This will require reverting prior changes that caused the tls-ca-bundler.pem to be mounted in a temporary location and then moved to the default location as part of the cloud-credential-operator pod's command.
A common concern with dealing with escalations/incidents in Managed OpenShift Hosted Control Planes is the resolution time incurred when the fix needs to be delivered in a component of the solution that ships within the OpenShift release payload. This is because OpenShift's release payloads:
This feature seeks to provide mechanisms that put the upper time boundary in delivering such fixes to match the current HyperShift Operator <24h expectation
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | managed (ROSA and ARO) |
Classic (standalone cluster) | No |
Hosted control planes | Yes |
Multi node, Compact (three node), or Single node (SNO), or all | All supported ROSA/HCP topologies |
Connected / Restricted Network | All supported ROSA/HCP topologies |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All supported ROSA/HCP topologies |
Operator compatibility | CPO and Operators depending on it |
Backport needed (list applicable versions) | TBD |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | No |
Other (please specify) | No |
Discussed previously during incident calls. Design discussion document
SOP needs to be defined for:
The default PR posting and pushing tekton file that Konflux generates builds always. We can be more efficient with resources.
It is necessary to get the builds off of main for CPO overrides.
Rolling out new versions of HyperShift Operator or Hosted Control Plane components such as HyperShift's Control Plane Operator will no longer carry the possibility of triggering a Node rollout that can affect customer workloads running on those nodes
Customer Nodepool rollouts exhaustive cause list will be:
Customers will have visibility on rollouts that are pending so that they can effect a rollout of their affected nodepools at their earliest convenience
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | Managed (ROSA and ARO) |
Classic (standalone cluster) | No |
Hosted control planes | Yes |
Multi node, Compact (three node), or Single node (SNO), or all | All supported Managed Hosted Control Plane topologies and configurations |
Connected / Restricted Network | All supported Managed Hosted Control Plane topologies and configurations |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All supported Managed Hosted Control Plane topologies and configurations |
Operator compatibility | N/A |
Backport needed (list applicable versions) | All supported Managed Hosted Control Plane releases |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | Yes. Console represtation of Nodepools/Machinepools should indicate pending rollouts and allow them to be triggered |
Other (please specify) |
Past incidents with fixes to ignition generation resulting in rollout unexpected by the customer with workload impact
There should be an easy way to see, understand the content and trigger queued updates
SOPs for the observability above
ROSA documentation for queued updates
ROSA/HCP and ARO/HCP
test to capture HO updates causing nodePool reboots
The OpenShift IPsec implementation will be enhanced for a growing set of enterprise use cases, and for larger scale deployments.
The OpenShift IPsec implementation was originally built for purpose-driven use cases from telco NEPs, but was also be useful for a specific set of other customer use cases outside of that context. As customer adoption grew and it was adopted by some of the largest (by number of cluster nodes) deployments in the field, it became obvious that some redesign is necessary in order to continue to deliver enterprise-grade IPsec, for both East-West and North-South traffic, and for some of our most-demanding customer deployments.
Key enhancements include observability and blocked traffic across paths if IPsec encryption is not functioning properly.
Requirement | Notes | isMvp? |
---|---|---|
CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
The OpenShift IPsec feature is fundamental to customer deployments for ensuring that all traffic between cluster nodes (East-West) and between cluster nodes and external-to-the-cluster entities that also are configured for IPsec (North-South) is encrypted by default. This encryption must scale to the largest of deployments.
Questions to be addressed:
While running IPsec e2e tests in the CI, the data plane traffic is not flowing with desired traffic type esp or udp. For example, ipsec mode external, the traffic type seems to seen as esp for EW traffic, but it's supposed to be geneve (udp) taffic.
This issue was reproducible on a local cluster after many attempts and noticed ipsec states are not cleanup on the node which is a residue from previous test run with ipsec full mode.
[peri@sdn-09 origin]$ kubectl get networks.operator.openshift.io cluster -o yaml
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
creationTimestamp: "2024-05-13T18:55:57Z"
generation: 1362
name: cluster
resourceVersion: "593827"
uid: 10f804c9-da46-41ee-91d5-37aff920bee4
spec:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
defaultNetwork:
ovnKubernetesConfig:
egressIPConfig: {}
gatewayConfig:
ipv4: {}
ipv6: {}
routingViaHost: false
genevePort: 6081
ipsecConfig:
mode: External
mtu: 1400
policyAuditConfig:
destination: "null"
maxFileSize: 50
maxLogFiles: 5
rateLimit: 20
syslogFacility: local0
type: OVNKubernetes
deployKubeProxy: false
disableMultiNetwork: false
disableNetworkDiagnostics: false
logLevel: Normal
managementState: Managed
observedConfig: null
operatorLogLevel: Normal
serviceNetwork:
- 172.30.0.0/16
unsupportedConfigOverrides: null
useMultiNetworkPolicy: false
status:
conditions:
- lastTransitionTime: "2024-05-13T18:55:57Z"
status: "False"
type: ManagementStateDegraded
- lastTransitionTime: "2024-05-14T10:13:12Z"
status: "False"
type: Degraded
- lastTransitionTime: "2024-05-13T18:55:57Z"
status: "True"
type: Upgradeable
- lastTransitionTime: "2024-05-14T11:50:26Z"
status: "False"
type: Progressing
- lastTransitionTime: "2024-05-13T18:57:13Z"
status: "True"
type: Available
readyReplicas: 0
version: 4.16.0-0.nightly-2024-05-08-222442
[peri@sdn-09 origin]$ oc debug node/worker-0
Starting pod/worker-0-debug-k6nlm ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.111.23
If you don't see a command prompt, try pressing enter.
sh-5.1# chroot /host
sh-5.1# toolbox
Checking if there is a newer version of registry.redhat.io/rhel9/support-tools available...
Container 'toolbox-root' already exists. Trying to start...
(To remove the container and start with a fresh toolbox, run: sudo podman rm 'toolbox-root')
toolbox-root
Container started successfully. To exit, type 'exit'.
[root@worker-0 /]# tcpdump -i enp2s0 -c 1 -v --direction=out esp and src 192.168.111.23 and dst 192.168.111.24
dropped privs to tcpdump
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:07:01.854214 IP (tos 0x0, ttl 64, id 20451, offset 0, flags [DF], proto ESP (50), length 152)
worker-0 > worker-1: ESP(spi=0x52cc9c8d,seq=0xe1c5c), length 132
1 packet captured
6 packets received by filter
0 packets dropped by kernel
[root@worker-0 /]# exit
exit
sh-5.1# ipsec whack --trafficstatus
006 #20: "ovn-1184d9-0-in-1", type=ESP, add_time=1715687134, inBytes=206148172, outBytes=0, maxBytes=2^63B, id='@1184d960-3211-45c4-a482-d7b6fe995446'
006 #19: "ovn-1184d9-0-out-1", type=ESP, add_time=1715687112, inBytes=0, outBytes=40269835, maxBytes=2^63B, id='@1184d960-3211-45c4-a482-d7b6fe995446'
006 #27: "ovn-185198-0-in-1", type=ESP, add_time=1715687419, inBytes=71406656, outBytes=0, maxBytes=2^63B, id='@185198f6-7dde-4e9b-b2aa-52439d2beef5'
006 #26: "ovn-185198-0-out-1", type=ESP, add_time=1715687401, inBytes=0, outBytes=17201159, maxBytes=2^63B, id='@185198f6-7dde-4e9b-b2aa-52439d2beef5'
006 #14: "ovn-922aca-0-in-1", type=ESP, add_time=1715687004, inBytes=116384250, outBytes=0, maxBytes=2^63B, id='@922aca42-b893-496e-bb9b-0310884f4cc1'
006 #13: "ovn-922aca-0-out-1", type=ESP, add_time=1715686986, inBytes=0, outBytes=986900228, maxBytes=2^63B, id='@922aca42-b893-496e-bb9b-0310884f4cc1'
006 #6: "ovn-f72f26-0-in-1", type=ESP, add_time=1715686855, inBytes=115781441, outBytes=98, maxBytes=2^63B, id='@f72f2622-e7dc-414e-8369-6013752ea15b'
006 #5: "ovn-f72f26-0-out-1", type=ESP, add_time=1715686833, inBytes=9320, outBytes=29002449, maxBytes=2^63B, id='@f72f2622-e7dc-414e-8369-6013752ea15b'
sh-5.1# ip xfrm state; echo ' '; ip xfrm policy
src 192.168.111.21 dst 192.168.111.23
proto esp spi 0x7f7ddcf5 reqid 16413 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x6158d9a0f4a28598500e15f81a40ef715502b37ecf979feb11bbc488479c8804598011ee 128
lastused 2024-05-14 16:07:11
anti-replay esn context:
seq-hi 0x0, seq 0x18564, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.21/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.21
proto esp spi 0xda57e42e reqid 16413 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x810bebecef77951ae8bb9a46cf53a348a24266df8b57bf2c88d4f23244eb3875e88cc796 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.21/32 proto udp sport 6081
src 192.168.111.21 dst 192.168.111.23
proto esp spi 0xf84f2fcf reqid 16417 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x0f242efb072699a0f061d4c941d1bb9d4eb7357b136db85a0165c3b3979e27b00ff20ac7 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.21/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.21
proto esp spi 0x9523c6ca reqid 16417 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xe075d39b6e53c033f5225f8be48efe537c3ba605cee2f5f5f3bb1cf16b6c53182ecf35f7 128
lastused 2024-05-14 16:07:11
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x10fb2
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.21/32 proto udp dport 6081
src 192.168.111.20 dst 192.168.111.23
proto esp spi 0x459d8516 reqid 16397 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xee778e6db2ce83fa24da3b18e028451bbfcf4259513bca21db832c3023e238a6b55fdacc 128
lastused 2024-05-14 16:07:13
anti-replay esn context:
seq-hi 0x0, seq 0x3ec45, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.20/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.20
proto esp spi 0x3142f53a reqid 16397 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x6238fea6dffdd36cbb909f6aab48425ba6e38f9d32edfa0c1e0fc6af8d4e3a5c11b5dfd1 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.20/32 proto udp sport 6081
src 192.168.111.20 dst 192.168.111.23
proto esp spi 0xeda1ccb9 reqid 16401 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xef84a90993bd71df9c97db940803ad31c6f7d2e72a367a1ec55b4798879818a6341c38b6 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.20/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.20
proto esp spi 0x02c3c0dd reqid 16401 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x858ab7326e54b6d888825118724de5f0c0ad772be2b39133c272920c2cceb2f716d02754 128
lastused 2024-05-14 16:07:13
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x26f8e
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.20/32 proto udp dport 6081
src 192.168.111.24 dst 192.168.111.23
proto esp spi 0xc9535b47 reqid 16405 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xd7a83ff4bd6e7704562c597810d509c3cdd4e208daabf2ec074d109748fd1647ab2eff9d 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x53d4c, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.24/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.24
proto esp spi 0xb66203c8 reqid 16405 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xc207001a7f1ed7f114b3e327308ddbddc36de5272a11fe0661d03eaecc84b6761c7ec9c4 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.24/32 proto udp sport 6081
src 192.168.111.24 dst 192.168.111.23
proto esp spi 0x2e4d4deb reqid 16409 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x91e399d83aa1c2626424b502d4b8dae07d4a170f7ef39f8d1baca8e92b8a1dee210e2502 128
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.24/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.24
proto esp spi 0x52cc9c8d reqid 16409 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0xb605451f32f5dd7a113cae16e6f1509270c286d67265da2ad14634abccf6c90f907e5c00 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0xe2735
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.24/32 proto udp dport 6081
src 192.168.111.22 dst 192.168.111.23
proto esp spi 0x973119c3 reqid 16389 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x87d13e67b948454671fb8463ec0cd4d9c38e5e2dd7f97cbb8f88b50d4965fb1f21b36199 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x2af9a, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
ffffffff ffffffff ffffffff ffffffff
sel src 192.168.111.22/32 dst 192.168.111.23/32 proto udp dport 6081
src 192.168.111.23 dst 192.168.111.22
proto esp spi 0x4c3580ff reqid 16389 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x2c09750f51e86d60647a60e15606f8b312036639f8de2d7e49e733cda105b920baade029 128
lastused 2024-05-14 14:36:43
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x1
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.22/32 proto udp sport 6081
src 192.168.111.22 dst 192.168.111.23
proto esp spi 0xa3e469dc reqid 16393 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x1d5c5c232e6fd4b72f3dad68e8a4d523cbd297f463c53602fad429d12c0211d97ae26f47 128
lastused 2024-05-14 14:18:42
anti-replay esn context:
seq-hi 0x0, seq 0xb, oseq-hi 0x0, oseq 0x0
replay_window 128, bitmap-length 4
00000000 00000000 00000000 000007ff
sel src 192.168.111.22/32 dst 192.168.111.23/32 proto udp sport 6081
src 192.168.111.23 dst 192.168.111.22
proto esp spi 0xdee8476f reqid 16393 mode transport
replay-window 0 flag esn
aead rfc4106(gcm(aes)) 0x5895025ce5b192a7854091841c73c8e29e7e302f61becfa3feb44d071ac5c64ce54f5083 128
lastused 2024-05-14 16:07:14
anti-replay esn context:
seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0x1f1a3
replay_window 128, bitmap-length 4
00000000 00000000 00000000 00000000
sel src 192.168.111.23/32 dst 192.168.111.22/32 proto udp dport 6081
src 192.168.111.23/32 dst 192.168.111.21/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16413 mode transport
src 192.168.111.21/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16413 mode transport
src 192.168.111.23/32 dst 192.168.111.21/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16417 mode transport
src 192.168.111.21/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16417 mode transport
src 192.168.111.23/32 dst 192.168.111.20/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16397 mode transport
src 192.168.111.20/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16397 mode transport
src 192.168.111.23/32 dst 192.168.111.20/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16401 mode transport
src 192.168.111.20/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16401 mode transport
src 192.168.111.23/32 dst 192.168.111.24/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16405 mode transport
src 192.168.111.24/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16405 mode transport
src 192.168.111.23/32 dst 192.168.111.24/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16409 mode transport
src 192.168.111.24/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16409 mode transport
src 192.168.111.23/32 dst 192.168.111.22/32 proto udp sport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16389 mode transport
src 192.168.111.22/32 dst 192.168.111.23/32 proto udp dport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16389 mode transport
src 192.168.111.23/32 dst 192.168.111.22/32 proto udp dport 6081
dir out priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16393 mode transport
src 192.168.111.22/32 dst 192.168.111.23/32 proto udp sport 6081
dir in priority 1360065 ptype main
tmpl src 0.0.0.0 dst 0.0.0.0
proto esp reqid 16393 mode transport
src ::/0 dst ::/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src ::/0 dst ::/0
socket out priority 0 ptype main
src ::/0 dst ::/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0 ptype main
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir out priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir fwd priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 135
dir in priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir out priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir fwd priority 1 ptype main
src ::/0 dst ::/0 proto ipv6-icmp type 136
dir in priority 1 ptype main
sh-5.1# cat /etc/ipsec.conf
# /etc/ipsec.conf - Libreswan 4.0 configuration file
#
# see 'man ipsec.conf' and 'man pluto' for more information
#
# For example configurations and documentation, see https://libreswan.org/wiki/
config setup
# If logfile= is unset, syslog is used to send log messages too.
# Note that on busy VPN servers, the amount of logging can trigger
# syslogd (or journald) to rate limit messages.
#logfile=/var/log/pluto.log
#
# Debugging should only be used to find bugs, not configuration issues!
# "base" regular debug, "tmi" is excessive and "private" will log
# sensitive key material (not available in FIPS mode). The "cpu-usage"
# value logs timing information and should not be used with other
# debug options as it will defeat getting accurate timing information.
# Default is "none"
# plutodebug="base"
# plutodebug="tmi"
#plutodebug="none"
#
# Some machines use a DNS resolver on localhost with broken DNSSEC
# support. This can be tested using the command:
# dig +dnssec DNSnameOfRemoteServer
# If that fails but omitting '+dnssec' works, the system's resolver is
# broken and you might need to disable DNSSEC.
# dnssec-enable=no
#
# To enable IKE and IPsec over TCP for VPN server. Requires at least
# Linux 5.7 kernel or a kernel with TCP backport (like RHEL8 4.18.0-291)
# listen-tcp=yes
# To enable IKE and IPsec over TCP for VPN client, also specify
# tcp-remote-port=4500 in the client's conn section.
# if it exists, include system wide crypto-policy defaults
include /etc/crypto-policies/back-ends/libreswan.config
# It is best to add your IPsec connections as separate files
# in /etc/ipsec.d/
include /etc/ipsec.d/*.conf
sh-5.1# cat /etc/ipsec.d/openshift.conf
# Generated by ovs-monitor-ipsec...do not modify by hand!
config setup
uniqueids=yes
conn %default
keyingtries=%forever
type=transport
auto=route
ike=aes_gcm256-sha2_256
esp=aes_gcm256
ikev2=insist
conn ovn-f72f26-0-in-1
left=192.168.111.23
right=192.168.111.22
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@f72f2622-e7dc-414e-8369-6013752ea15b
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-f72f26-0-out-1
left=192.168.111.23
right=192.168.111.22
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@f72f2622-e7dc-414e-8369-6013752ea15b
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-1184d9-0-in-1
left=192.168.111.23
right=192.168.111.20
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@1184d960-3211-45c4-a482-d7b6fe995446
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-1184d9-0-out-1
left=192.168.111.23
right=192.168.111.20
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@1184d960-3211-45c4-a482-d7b6fe995446
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-922aca-0-in-1
left=192.168.111.23
right=192.168.111.24
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@922aca42-b893-496e-bb9b-0310884f4cc1
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-922aca-0-out-1
left=192.168.111.23
right=192.168.111.24
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@922aca42-b893-496e-bb9b-0310884f4cc1
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
conn ovn-185198-0-in-1
left=192.168.111.23
right=192.168.111.21
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@185198f6-7dde-4e9b-b2aa-52439d2beef5
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp/6081
rightprotoport=udp
conn ovn-185198-0-out-1
left=192.168.111.23
right=192.168.111.21
leftid=@cf36db5c-5c54-4329-9141-b83679b18ecc
rightid=@185198f6-7dde-4e9b-b2aa-52439d2beef5
leftcert="ovs_certkey_cf36db5c-5c54-4329-9141-b83679b18ecc"
leftrsasigkey=%cert
rightca=%same
leftprotoport=udp
rightprotoport=udp/6081
sh-5.1#
The e2e-aws-ovn-ipsec-upgrade job is currently an optional job and always_run: false because the job not reliable and success rate is so low. This must be made as mandatory CI lane after fixing its relevant issues.
The CNO rolls out ipsec mc plugin for rolling out IPsec for the cluster, but it doesn't really check master and work role machine config pools status to confirm if that's successfully installed in the cluster nodes.
Hence CNO should be made to listen for MachineConfigPool status object updates and set network operator condition accordingly based on ipsec mc plugin rollout status.
Placeholder to track GA work for CFE-811
Update API godoc to document that manual intervention is required for using .spec.tls.externalCertificate. Something simple like: "The Router service account needs to be granted with read-only access to this secret, please refer to openshift docs for additional details."
As a developer, I want to bump o/api into o/kubernetes repo to update API godoc added in OAPE-94
Networking Definition of Planned
Epic Template descriptions and documentation
Additional information on each of the above items can be found here: Networking Definition of Planned
...
1.
...
1. ...
1. ...
Epic Goal*
Drive the technical part of the Kubernetes 1.32 upgrade, including rebasing openshift/kubernetes repository and coordination across OpenShift organization to get e2e tests green for the OCP release.
Why is this important? (mandatory)
OpenShift 4.19 cannot be released without Kubernetes 1.32
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Slack Discussion Channel - https://redhat.enterprise.slack.com/archives/C07V32J0YKF
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
To align with the 4.19 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.30. This should be done by rebasing/updating as appropriate for the repository
As a developer, I want to upgrade the Kubernetes dependencies to 1.32
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.32. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.32. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
To align with the 4.19 release, dependencies need to be updated to 1.31. This should be done by rebasing/updating as appropriate for the repository
As a MCO developer, I want to pick up the openshift/kubernetes updates for the 1.32 k8s rebase to track the k8s version as rest of the OpenShift 1.32 cluster.
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
Following up this issue, it's added the support for CapacityReservations but missing the support for Capacity Blocks for ML which is essential to launch the capacity-blocks.
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Add support to Dxv6 machine series to be used on OpenShift deployment in Azure
As a user, I can deploy OpenShift in Azure using Dxv6 machine series so both the Control Plane and Compute Nodes can run on these machine series
The new machines series that will be available in Azure soon, Dxv6 can be selected at install time to be used to deploy OpenShift on Azure
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
These machines series will be available for OpenShift to be used once they are declared GA by Microsoft
ARO will need to support Dxv6 instance types supported. These are currently in preview. The specific instance types are:
Usual documentation to list these machine series as tested
This feature will be consumed by ARO later
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a customer of self managed OpenShift or an SRE managing a fleet of OpenShift clusters I should be able to determine the progress and state of an OCP upgrade and only be alerted if the cluster is unable to progress. Support a cli-status command and status-API which can be used by cluster-admin to monitor the progress. status command/API should also contain data to alert users about potential issues which can make the updates problematic.
Here are common update improvements from customer interactions on Update experience
oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0 True True 16s Working towards 4.12.4: 9 of 829 done (1% complete)
Update docs for UX and CLI changes
Reference : https://docs.google.com/presentation/d/1cwxN30uno_9O8RayvcIAe8Owlds-5Wjr970sDxn7hPg/edit#slide=id.g2a2b8de8edb_0_22
Epic Goal*
Add a new command `oc adm upgrade status` command which is backed by an API. Please find the mock output of the command output attached in this card.
Why is this important? (mandatory)
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Upgrade of the ota-stage cluster from 4.14.1 to 4.15.0-ec.2 got stuck because of the operator-lifecycle-manager-packageserver ClusterOperator which never reaches the desired version (likely because its Pods are CrashLooping, which is a separate issue discussed now on Slack and OCPBUGS-23538 was filed for it)
However, I would expect CVO to enter its waiting for operator-lifecycle-manager-packageserver up to 40 minutes state, eventually hit that deadline and signal the upgrade as stuck via a Failing=True condition, but that did not happen and CVO does not signal anything problematic in this stuck state.
upgrade from 4.14.1 to 4.15.0-ec.2
Unsure
1. upgrade from 4.14.1 to 4.15.0-ec.2 and hope you get stuck the way ota-stage did
$ OC_ENABLE_CMD_UPGRADE_STATUS=true ./oc adm upgrade status An update is in progress for 2h8m20s: Working towards 4.15.0-ec.2: 695 of 863 done (80% complete), waiting on operator-lifecycle-manager-packageserver
$ oc adm upgrade status Failing=True Reason: operator-lifecycle-manager-packageserver is stuck (or whatever is the message) An update is in progress for 2h8m20s: Working towards 4.15.0-ec.2: 695 of 863 done (80% complete), waiting on operator-lifecycle-manager-packageserver
Attached CVO log and the waited-on CO yaml dump
Health insights have a lifecycle that is not suitable for the async producer/consumer architecture USC has right now, where update informers send individual insights to the controller that maintains the API instance. Health insights are expected to disappear and appear following the trigger condition, and this needs to be respected through controller restart, API unavailability etc. Basically this means that the informer should ideally work as a standard, full-reconciliation controller over a set of its insights.
We should also have an easy method to test health insight lifecycle: easy on/off switch for an artificial health insight to be reported or not, to avoid relying on true problematic conditions for testing the controller operation. Something like an existence of a label or an annotation on a resource to trigger a health insight directly.
After OTA-960 is fixed, ClusterVersion/version and oc adm upgrade can be used to monitor the process of migrating a cluster to multi-arch.
$ oc adm upgrade info: An upgrade is in progress. Working towards 4.18.0-ec.3: 761 of 890 done (85% complete), waiting on machine-config Upgradeable=False Reason: PoolUpdating Message: Cluster operator machine-config should not be upgraded between minor versions: One or more machine config pools are updating, please see `oc get mcp` for further details Upstream: https://api.integration.openshift.com/api/upgrades_info/graph Channel: candidate-4.18 (available channels: candidate-4.18) No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
But oc adm upgrade status reports COMPLETION 100% while the migration/upgrade is still ongoing.
$ OC_ENABLE_CMD_UPGRADE_STATUS=true oc adm upgrade status Unable to fetch alerts, ignoring alerts in 'Update Health': failed to get alerts from Thanos: no token is currently in use for this session = Control Plane = Assessment: Completed Target Version: 4.18.0-ec.3 (from 4.18.0-ec.3) Completion: 100% (33 operators updated, 0 updating, 0 waiting) Duration: 15m Operator Status: 33 Healthy Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-95-224.us-east-2.compute.internal Unavailable Updated 4.18.0-ec.3 - Node is unavailable ip-10-0-33-81.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - ip-10-0-45-170.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Completed 100% 3 Total, 2 Available, 0 Progressing, 0 Outdated, 0 Draining, 0 Excluded, 0 Degraded Worker Pool Nodes: worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-72-40.us-east-2.compute.internal Unavailable Updated 4.18.0-ec.3 - Node is unavailable ip-10-0-17-117.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - ip-10-0-22-179.us-east-2.compute.internal Completed Updated 4.18.0-ec.3 - = Update Health = SINCE LEVEL IMPACT MESSAGE - Warning Update Speed Node ip-10-0-95-224.us-east-2.compute.internal is unavailable - Warning Update Speed Node ip-10-0-72-40.us-east-2.compute.internal is unavailable Run with --details=health for additional description and links to related online documentation $ oc get clusterversion version NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-ec.3 True True 14m Working towards 4.18.0-ec.3: 761 of 890 done (85% complete), waiting on machine-config $ oc get co machine-config NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE machine-config 4.18.0-ec.3 True True False 63m Working towards 4.18.0-ec.3
The reason is that PROGRESSING=True is not detected for co/machine-config as the status command checks only operator.Status.Versions[name=="operator"] and it needs to check ClusterOperator.Status.Versions[name=="operator-image"] as well.
For grooming:
It will be challenging for the status command to check the operator image's pull spec because it does not know the expected value. CVO knows it because CVO holds the manifests (containing the expected value) from the multi-arch payload.
One "hacky" workaround is that the status command gets the pull spec from the MCO deployment:
oc get deployment -n openshift-machine-config-operator machine-config-operator -o json | jq -r '.spec.template.spec.containers[]|select(.name=="machine-config-operator")|.image' quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787a505ca594b0a727549353c503dec9233a9d3c2dcd6b64e3de5f998892a1d5
Note this co/machine-config -> deployment/machine-config-operator trick may not be feasible if we want to extend it to all cluster operators. But it should work as a hacky workaround to check only MCO.
We may claim that the status command is not designed for monitoring the multi-arch migration and suggest to use oc adm upgrade instead. In that case, we can close this card as Obsolete/Won'tDo.
manifests.ziphas the mockData/manifests for the status cmd that are taken during the migration.
oc#1920 started the work for the status command to recognize the migration and we need to extend the work to cover (the comments from Petr's review):
Implement a new Informer controller in the Update Status Controller to watch Node resources in the cluster and maintain an update status insight for each. The informer will need to interact with additional resources such as MachineConfigPools and MachineConfigs, e.g. to discover the OCP version tied to config that is being reconciled on the Node, but should not attempt to maintain the MachineConfigPool status insights. Generally the node status insight should carry enough data for any client to be able to render a line that the oc adm upgrade status currently shows:
NAME ASSESSMENT PHASE VERSION EST MESSAGE
build0-gstfj-ci-prowjobs-worker-b-9lztv Degraded Draining 4.16.0-ec.2 ? failed to drain node: <node> after 1 hour. Please see machine-config-controller logs for more informatio
build0-gstfj-ci-prowjobs-worker-d-ddnxd Unavailable Pending ? ? Machine Config Daemon is processing the node
build0-gstfj-ci-tests-worker-b-d9vz2 Unavailable Pending ? ? Not ready
build0-gstfj-ci-tests-worker-c-jq5rk Unavailable Updated 4.16.0-ec.3 - Node is marked unschedulable
The basic expectations for Node status insights are described in the design docs but the current source of truth for the data structure is the NodeStatusInsight structure from https://github.com/openshift/api/pull/2012 .
Extend the Control Plane Informer in the Update Status Controller so it watches ClusterOperator resources in the cluster and maintains an update status insight for each.
The actual API structure for an update status insights needs to be taken from whatever state https://github.com/openshift/api/pull/2012 is at the moment. The story does not include the actual API form nor how it is exposed by the cluster (depending on the state of API approval, the USC may still expose the API as a ConfigMap or an actual custom resource), it includes just the logic to watch ClusterOperator resources and producing a matching set of cluster operator status insights.
The basic expectations for cluster operator status insights are described in the design docs
On the call to discuss oc adm upgrade status roadmap to server side-implementation (notes) we agreed on basic architectural direction and we can starting moving in that direction:
Let's start building this controller; we can implement the controller perform the functionality currently present in the client, and just expose it through an API. I am not sure how to deal with the fact that we won't have the API merged until it merges into o/api, which is not soon. Maybe we can implement the controller over a temporary fork of o/api and rely on manually inserting the CRD into the cluster when we test the functionality? Not sure.
We need to avoid committing to implementation details and investing effort into things that may change though.
This card only brings a skeleton of the desired functionality to the DevPreviewNoUpgrade feature set. Its purpose is mainly to enable further development by putting the necessary bits in place so that we can start developing more functionality. There's not much point in automating testing of any of the functionality in this card, but it should be useful to start getting familiar with how the new controller is deployed and what are its concepts.
For seeing the new controller in action:
1. Launch a cluster that includes both the code and manifests. As of Nov 11, #1107 is not yet merged so you need to use launch 4.18,openshift/cluster-version-operator#1107 aws,no-spot
2. Enable the DevPreviewNoUpgrade feature set. CVO will restart and will deploy all functionality gated by this feature set, including the USC. It can take a bit of time, ~10-15m should be enough though.
3. Eventually, you should be able to see the new openshift-update-status-controller Namespace created in the cluster
4. You should be able to see a update-status-controller Deployment in that namespace
5. That Deployment should have one replica running and being ready. It should not crashloop or anything like that. You can inspect its logs for obvious failures and such. At this point, its log should, near its end, say something like "the ConfigMap does not exist so doing nothing"
6. Create the ConfigMap that mimics the future API (make sure to create it in the openshift-update-status-controller namespace): oc create configmap -n openshift-update-status-controller status-api-cm-prototype
7. The controller should immediately-ish insert a usc-cv-version key into the ConfigMap. Its content is a YAML-serialized ClusterVersion status insight (see design doc). As of OTA-1269 the content is not that important, but the (1) reference to the CV (2) versions field should be correct.
8. The status insight should have a condition of Updating type. It should be False at this time (the cluster is not updating).
9. Start upgrading the cluster (it's a cluster bot cluster with ephemeral 4.18 version so you'll need to use --to-image=pullspec and probably force it
10. While updating, you should be able to observe the controller activity in the log (it logs some diffs), but also the content of the status insight in the ConfigMap changing. The versions field should change appropriately (and startedAt too), and the Updating condition should become True.
11. Eventually the update should finish and the Updating condition should flip to False again.
Some of these will turn into automated testcases, but it does not make sense to implement that automation while we're using the ConfigMap instead of the API.
Spun out of https://issues.redhat.com/browse/MCO-668
This aims to capture the work required to rotate the MCS-ignition CA + cert.
Original description copied from MCO-668:
Today in OCP there is a TLS certificate generated by the installer , which is called "root-ca" but is really "the MCS CA".
A key derived from this is injected into the pointer Ignition configuration under the "security.tls.certificateAuthorities" section, and this is how the client verifies it's talking to the expected server.
If this key expires (and by default the CA has a 10 year lifetime), newly scaled up nodes will fail in Ignition (and fail to join the cluster).
The MCO should take over management of this cert, and the corresponding user-data secret field, to implement rotation.
Reading:
- There is a section in the customer facing documentation that touches on this: https://docs.openshift.com/container-platform/4.13/security/certificate_types_descriptions/machine-config-operator-certificates.html
- There's a section in the customer facing documentation for this: https://docs.openshift.com/container-platform/4.13/security/certificate_types_descriptions/machine-config-operator-certificates.html that needs updating for clarification.
- There's a pending PR to openshift/api: https://github.com/openshift/api/pull/1484/files
- Also see old (related) bug: https://issues.redhat.com/browse/OCPBUGS-9890
- This is also separate to https://issues.redhat.com/browse/MCO-499 which describes the management of kubelet certs
The CA/cert generated by the installer is not currently managed and also does not preserve the signing key; so the cert controller we are adding in the MCO(leveraged from library-go), throws away everything and starts fresh. Normally this happens fairly quickly so both the MCS and the -user-data secrets are updated together. However, in certain cases(such as agent based installations) where a bootstrap node joins the cluster late, it will have the old CA from installer, and unfortunately the MCS will have a TLS cert signed by the new CA - resulting in invalid TLS cert errors.
To account for such cases, we have to ensure the first CA embedded in any machine is matching the format expected by the cert controller. To do this, we'll have to do the following in the installer:
The machinesets in the machine-api namespace reference a user-data secret (per pool and can be customized) which stores the initial ignition stub configuration pointing to the MCS, and the TLS cert. This today doesn't get updated after creation.
The MCO now has the ability to manage some fields of the machineset object as part of the managed bootimage work. We should extend that to also sync in the updated user-data secrets for the ignition tls cert.
The MCC should be able to parse both install-time-generated machinesets as well as user-created ones, so as to not break compatibility. One way users are using this today is to use a custom secret + machineset to do non-MCO compatible ignition fields, for example, to partition disks for different device types for nodes in the same pool. Extra care should be taken not to break this use case
As a cluster-admin I can use a single command to see all upgrade checklist before I trigger an update.
Create a Update precheck command that helps customers identify potential issues before triggering an OpenShift cluster upgrade, without blocking the upgrade process. This tool aims to reduce upgrade failures and support tickets by surfacing common issues beforehand.
==============
{}New additions in 2025{}
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | Self-managed |
Classic (standalone cluster) | standalone |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | All |
Connected / Restricted Network | All |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Use Cases (Optional):
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Add docs for precheck command
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
OCPSTRAT-1834 is requesting an oc precheck command that helps customers identify potential issues before triggering an OpenShift cluster upgrade. For 4.18, we'd built a tech-preview oc adm upgrade recommend (OTA-1270, product docs) that is in this "Anything I should think about before updating my cluster [to 4.y.z]?" space, and this Epic is about extending that subcommand with alerts to deliver the coverage requested by OCPSTRAT-1834.
We currently document some manual checks for customer admins to run before launching an update. For example, RFE-5104 is up asking to automate whatever we're hoping customer are supposed to look for vs. critical alerts. But updating the production OpenShift Update Service is complicated, and it's easier to play around in a tech-preview oc subcommand, while we get a better idea of what information is helpful, and which presentation approaches are most accessible. 4.18's OTA-902 / cvo#1907 folded the Upgradeable condition in as a client-side Conditional Update risk, and this Epic proposes to continue in that direction by retrieving update-relevant alerts and folding those in as additional client-side Conditional Update risks.
As a cluster administrator interested in launching an OCP update, I want to run an oc command that talks to me about my next-hop options, including any information related to known regressions with those target releases, and also including any information about things I should consider addressing in my current cluster state.
The initial implementation can be delivered unilaterally by the OTA updates team. The implementation may surface ambiguous or hard-to-actuate alert messages, and those messages will need to be improved by the component team responsible for maintaining that alert.
OCPSTRAT-1834 customer is happy
Client-side Conditional Update risks are helpful for cluster administrators who use that particular client. But admins who use older oc or who are using the in-cluster web-console and similar will not see risks known only to newer oc. If we can clearly tie a particular cluster state to update risk, declaring that risk via the OpenShift Update Service would put the information in front of all cluster administrators, regardless of which update interface they use.
However, trialing update risks client-side in tech-preview oc and then possibly promoting them to risks served by the OpenShift Update Service in the future might help us identify cluster state that's only weakly coupled to update success but still interesting enough to display. Or help us find more accessible ways of displaying that context before putting the message in front of large chunks of the fleet.
oc adm upgrade recommend should retrieve alerts from the cluster (similar to how oc adm upgrade status already does), and inject them as conditional update risks (similar to how OTA-902 / cvo#1907 injected Upgradeable issues). The set of alerts to include is:
Definition of done / test-plan:
The precheck functionality introduced by OTA-1426 flubbed the Route lookup implementation, and crashes with a seg-fault. Thanks to Billy Pearson for reporting.
4.19 code behind both the OC_ENABLE_CMD_UPGRADE_RECOMMEND=true and OC_ENABLE_CMD_UPGRADE_RECOMMEND_PRECHECK=true feature gates.
Every time.
$ export OC_ENABLE_CMD_UPGRADE_RECOMMEND=true # overall 'recommend' command still tech-preview $ export OC_ENABLE_CMD_UPGRADE_RECOMMEND_PRECHECK=true # pre-check functionality even more tech-preview $ ./oc adm upgrade recommend
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3bd3cdd] goroutine 1 [running]: github.com/openshift/client-go/route/clientset/versioned/typed/route/v1.NewForConfig(0x24ec35c?) /go/src/github.com/openshift/oc/vendor/github.com/openshift/client-go/route/clientset/versioned/typed/route/v1/route_client.go:31 +0x1d github.com/openshift/oc/pkg/cli/admin/upgrade/recommend.(*options).alerts(0xc000663130, {0x5ed4828, 0x8350a80}) /go/src/github.com/openshift/oc/pkg/cli/admin/upgrade/recommend/alerts.go:34 +0x170 github.com/openshift/oc/pkg/cli/admin/upgrade/recommend.(*options).precheck(0x0?, {0x5ed4828, 0x8350a80}) ...
Successful precheck execution and reporting.
Console enhancements based on customer RFEs that improve customer user experience.
Requirement | Notes | isMvp? |
---|
CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
This Section:
This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.
Questions to be addressed:
Introduce a "Created Time" column to the Job listing in the OpenShift Container Platform (OCP) console to enhance the ability to sort jobs by their creation date. This feature will help users efficiently manage and navigate through numerous jobs, particularly in environments with frequent CronJob executions and a high volume of job runs.
We need to maintain our dependencies across all the libraries we use in order to stay in compliance.
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
<your text here>
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
<enter general Feature acceptance here>
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
<your text here>
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
<your text here>
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.
Inconsistency in the loader/spinner/dots component used throughout the unified console. The dots animation is used widely through the spoke clusters, but it is not a Patternfly component(This component was originally inhereted from CoreOS console). Spoke clusters also uses skeleton states on certain pages, which is a Patternfly component. Hub uses a mix of dots animation first for half a second, and then spinners and skeletons.
Currently there is a discussion with PF to update with clearer guidelines. According to the current PF guidelines, we should be using the large spinner if we cannot anticipate the data being loaded, and the skeleton state if we do know. Link to doc
Currently console is using TypeScript 4, which is preventing us from upgrading to NodeJS-22. Due to that we need to update TypeScript 5 (not necessarily latest version).
AC:
Note: In case of higher complexity we should be splitting the story into multiple stories, per console package.
Part of lifting feature gate for the CSP API we need to enable e2e CSP tests for console-operator.
In Console 4.18 we introduced an initial Content Security Policy (CSP) implementation (CONSOLE-4263).
This affects both Console web application as well as any dynamic plugins loaded by Console. In production, CSP violations are sent to telemetry service for analysis (CONSOLE-4272).
We need a reliable way to detect new CSP violations as part of our automated CI checks. We can start with testing the main dashboard page of Console and expand to more pages as necessary.
Acceptance criteria:
As a developer I want to make sure we are running the latest version of webpack in order to take advantage of the latest benefits and also keep current so that future updating is a painless as possible.
We are currently on v4.47.0.
Changelog: https://webpack.js.org/blog/2020-10-10-webpack-5-release/
By updating to version 5 we will need to update following pkgs as well:
AC: Update webpack to version 5 and determine what should be the ideal minor version.
The GenericSecretForm component needs to be refactored to address several tech debt issues: * Rename to OpaqueSecretForm
The KeyValueEntryForm component needs to be refactored to address several tech debt issues: * Rename to OpaqueSecretFormEntry
The BasicAuthSubform component needs to be refactored to address several tech debt issues: * Rename to BasicAuthSecretForm
The SourceSecretForm component needs to be refactored to address several tech debt issues: * Rename to AuthSecretForm
Networking Definition of Planned
Epic Template descriptions and documentation
Provide quality user experience for customers connecting their Pods and VMs to the underlying physical network through OVN Kubernetes localnet.
This is a continuation to https://issues.redhat.com/browse/SDN-5313.
It covers the UDN API for localnet and other improvements
Additional information on each of the above items can be found here: Networking Definition of Planned
...
IPAM is not in the scope of this epic. See RFE-6947.
1.
...
1. …
1. …
In order for Managed OpenShift Hosted Control Planes to run as part of the Azure Redhat OpenShift, it is necessary to support the new AKS design for secrets/identities.
Hosted Cluster components use the secrets/identities provided/referenced in the Hosted Cluster resources creation.
All OpenShift Hosted Cluster components running with the appropriate managed or workload identity.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | Managed |
Classic (standalone cluster) | No |
Hosted control planes | Yes |
Multi node, Compact (three node), or Single node (SNO), or all | All supported ARO/HCP topologies |
Connected / Restricted Network | All supported ARO/HCP topologies |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | All supported ARO/HCP topologies |
Operator compatibility | All core operators |
Backport needed (list applicable versions) | OCP 4.18.z |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | No |
Other (please specify) |
This is a follow-up to OCPSTRAT-979 required by an AKS sweeping change to how identities need to be handled.
Should only affect ARO/HCP documentation rather than Hosted Control Planes documentation.
Does not affect ROSA or any of the supported self-managed Hosted Control Planes platforms
As an ARO HCP user, I want to be able to:
so that I can
Description of criteria:
N/A
This does not require a design proposal.
This does not require a feature gate.
As an ARO HCP user, I want to be able to:
so that I can
Description of criteria:
N/A
This does not require a design proposal.
This does not require a feature gate.
As an ARO HCP user, I want to be able to:
so that
Description of criteria:
Updating any external OpenShift components that run in the HCP
This does not require a design proposal.
This does not require a feature gate.
As an ARO HCP user, I want to be able to:
so that
Description of criteria:
Updating any HyperShift-only components that run in the HCP
This does not require a design proposal.
This does not require a feature gate.
As an ARO HCP user, I want to be able to:
so that I can
Description of criteria:
Removing the deprecated client ID and certificate name fields. This should be done at a later date when CS only supports the new API
This does not require a design proposal.
This requires a feature gate.
The installation process for the OpenShift Virtualization Engine (OVE) has been identified as a critical area for improvement to address customer concerns regarding its complexity compared to competitors like VMware, Nutanix, and Proxmox. Customers often struggle with disconnected environments, operator configuration, and managing external dependencies, making the initial deployment challenging and time-consuming.
To resolve these issues, the goal is to deliver a streamlined, opinionated installation workflow that leverages existing tools like the Agent-Based Installer, the Assisted Installer, and the OpenShift Appliance (all sharing the same underlying technology) while pre-configuring essential operators and minimizing dependencies, especially the need for an image registry before installation.
By focusing on enterprise customers, particularly VMware administrators working in isolated networks, this effort aims to provide a user-friendly, UI-based installation experience that simplifies cluster setup and ensures quick time-to-value.
VMware administrators transitioning to OpenShift Virtualization in isolated/disconnected environments.
The first area of focus is a disconnected environment. We target these environments with the Agent-Based Installer.
The current docs for installing on disconnected environment are very long and hard to follow.
The image registry is required in disconnected installations before the installation process can start. We must simplify this point so that users can start the installation with one image, without having to explicitly install one.
This isn't a new requirement and in the past we've analyzed options for this and even did a POC, we could revisit this point, see Deploy OpenShift without external registry in disconnected environments.
The OpenShift Appliance can in fact be installed without a registry.
Additionally, we started work in this direction AGENT-262 (Strategy to complete installations where there isn't a pre-existing registry).
We also had the field (Brandon Jozsa) doing a POC which was promising:
https://gist.github.com/v1k0d3n/cbadfb78d45498b79428f5632853112a
The type of users coming from VMware vSphere expect a UI. They aren't used to writing YAML files and this has been identified as a blocker for some of them. We must provide a simple UI to stand up a cluster.
Simplified Disconnected Installation:
This document written by Zane Bitter as the Staff Engineer of this area contains an interactive workflow proposal.
This is the workflow proposed in the above document.
This sub-command will be used to generate the ignition file based on the interactive disconnected workflow.
This command will be invoked by the builder script (currently within the appliance tool) for supporting generating the ISO.
It will also consume, in future, the eventual (portion of) install configuration that the user will provide via the connected UI (above the sea level)
Description (2/20/25):
Create a script in agent-installer-utils/tools/ove-builder to build a live ISO using the appliance. The script will:
1. Checkout the appliance code.
2. Generate appliance-config.yaml.
3. Build the live ISO with the command:
sudo podman run --rm -it --pull newer --privileged --net=host -v $APPLIANCE_ASSETS:/assets:Z $APPLIANCE_IMAGE build live-iso
4. Take the pull secret as input
5. Unpack the ISO, embed TUI and Web UI components
6. Repack the ISO.
Refer to Appliance PR #335 for how release images are added in
Acceptance Criteria:
Original description:
Deploy a script (in dev-scripts) that will take as input a release image version.
The script should perform the following tasks (possibly within a container):
This will be used as a starting point for generating the image that the user will download to install the OVE cluster, and subsequently could be expanded with the additional images required (for example, the UI)
Notes:
The ove builder script should include all the necessary agent TUI artifacts within the generated ISO
(If the former option is selected, the IP address should be displayed so that it can be entered in the other hosts.)
As an admin, I want to be able to:
so that I can achieve
Description of criteria:
This does not require a design proposal.
This does not require a feature gate.
Remove all persistent volumes and claims. Also check if there are any CNS volumes that could be removed but the pv/pvc deletion should check for that.
When an OpenShift cluster on vSphere with CSI volumes is destroyed the volumes are not deleted, leaving behind multiple objects within vSphere. This leads to storage usage by orphan volumes that must be manually deleted.
Multiple customers have requested this feature and we need this feature for CI. PV(s) are not cleaned up and leave behind CNS orphaned volumes that cannot be removed.
The goal of this epic is upon destroy to remove all persistent volumes and claims. Also check if there are any CNS volumes that could be removed but the pv/pvc deletion should check for that.
As a openshift engineer I want the installer to make sure there are no CNS volumes so we are not leaking volumes that could be taking disk space or alerting in vCenter.
acceptance criteria
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
Some Kubernetes clusters do not have direct Internet access and rely solely on proxies for communication, so OLM v1 needs to support proxies to enable this communication.
The observable functionality that the user now has as a result of receiving this feature. Include the anticipated primary user type/persona and which existing features, if any, will be expanded. Complete during New status.
Some Kubernetes clusters do not have direct Internet access and rely solely on proxies for communication. This may be done for isolation, testing or to enhance security and minimise vulnerabilities. This is a fully supported configuration in OpenShift, with origin tests designed to validate functionality in proxy-based environments. Supporting proxies is essential to ensure your solution operates reliably within these secure and compliant setups.
To address this need, we have two key challenges to solve:
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | Restricted Network |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | 4.18 |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
OpenShift’s centralized proxy control via the proxies.config.openshift.io (a.k.a. proxy.config.openshift.io) resource makes managing proxies across a cluster easier. At the same time, vanilla Kubernetes requires a manual and decentralized proxy configuration, making it more complex to manage, especially in large clusters. There is no native Kubernetes solution that can adequately address the need for centralized proxy management.
Kubernetes lacks a built-in unified API like OpenShift’s proxies.config.openshift.io, which can streamline proxy configuration and management across any Kubernetes vendor. Consequently, Kubernetes requires more manual work to ensure the proxy configuration is consistent across the cluster, and this complexity increases with the scale of the environment. As such, vanilla Kubernetes does not provide a solution that can natively address proxy configuration across all clusters and vendors without relying on external tools or complex manual processes (such as that devised by OpenShift).
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
<your text here>
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
<your text here>
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
See RFC for more details
It looks like OLMv1 doesn't handle proxies correctly, aws-ovn-proxy job is permafailing https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-proxy/1861444783696777216
I suspect it's on the OLM operator side, are you looking at the cluster-wide proxy object and wiring it into your containers if set?
Once CCM was moved out-of-tree for Azure the 'azurerm_user_assigned_identity' resource the Installer creates is not required anymore. To make sure the Installer only creates the minimum permissions required to deploy OpenShift on Azure this resource created at install time needs to be removed
The installer doesn't create the 'azurerm_user_assigned_identity' resource anymore that is no longer required for the Nodes
**
The Installer only creates the minimum permissions required to deploy OpenShift on Azure
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
Once CCM was moved out-of-tree this permission is not required anymore. We are implementing this change into 4.19 and backported to 4.18.z
At the same time, for customers running previous OpenShift releases, we will test upgrades between EUS releases (4.14.z - 4.16.z - 4.18.z) when `azurerm_user_assigned_identity` resource is removed previously to ensure the upgrade process is working with no issues and OpenShift is not reporting any issues because of this change
A KCS will be created for customers running previous OpenShift releases who want to remove this resource
The new permissions requirements will be documented
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Volume Group Snapshots is a key new Kubernetes storage feature that allows multiple PVs to be grouped together and snapshotted at the same time. This enables customers to takes consistent snapshots of applications that span across multiple PVs.
This is also a key requirement for backup and DR solutions.
https://kubernetes.io/blog/2023/05/08/kubernetes-1-27-volume-group-snapshot-alpha/
https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot
Productise the volume group snapshots feature as GA, have docs updated, testing as well as removing feature gate to enable it by default.
Tests and CI must pass. We should identify all OCP shipped CSI drivers that support this feature and configure them accordingly.
CSI drivers development/support for this feature.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
This allows backup vendors to implemented advanced feature by taking snapshots of multiple volumes at the same time a common use case in virtualisation.
Drivers must support this feature and enable it. Partners may need to change their operator and/or doc to support it.
We already have TP doc content. Update the OCP driver's table to include this capability. Check if any new driver supports it beyond ODF
Can be leveraged by ODF and OCP virt, especially around backup and DR scenarios.
Support upstream feature "VolumeGroupSnapshot"" in OCP as Beta GA, i.e. test it and have docs for it.
We need to make sure that we have e2e tests in Openshift that exercise this feature.
Enable OpenShift to be deployed on Confidential VMs on GCP using AMD SEV-SNP technology
Users deploying OpenShift on GCP can choose to deploy Confidential VMs using AMD SEV-SNP technology to rely on confidential computing to secure the data in use
As a user, I can choose OpenShift Nodes to be deployed with the Confidential VM capability on GCP using AMD SEV-SNP technology at install time
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | |
Classic (standalone cluster) | |
Hosted control planes | |
Multi node, Compact (three node), or Single node (SNO), or all | |
Connected / Restricted Network | |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | |
Operator compatibility | |
Backport needed (list applicable versions) | |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | |
Other (please specify) |
This is a piece of a higher-level effort to secure data in use with OpenShift on every platform
Documentation on how to use this new option must be added as usual
OCP 4 clusters still maintain pinned boot images. We have numerous clusters installed that have boot media pinned to first boot images as early as 4.1. In the future these boot images may not be certified by the OEM and may fail to boot on updated datacenter or cloud hardware platforms. These "pinned" boot images should be updateable so that customers can avoid this problem and better still scale out nodes with boot media that matches the running cluster version.
In phase 1 provided tech preview for GCP.
In phase 2, GCP support goes to GA and AWS goes to TP.
In phase 3, AWS support goes to GA .
In phase 4, vsphere opt-in goes to TP and GCP goes to opt-out.
This epic will encompass work required to switch boot image updates on GCP to be opt-out.
The origin tests should be:
This should land before MCO-1584 lands.
Based on discussion on the enhancement, we have decided that we'd like to add an explicit opt-out option and a status field for the ManagedBootImages knob in the MachineConfiguration object.
More context here:
https://github.com/openshift/enhancements/pull/1761#discussion_r1987873170
The boot image controller should should ensure `-user-data` stub secrets are at least spec 3. This requires the cert management work to land first.
To ensure maximum coverage and minimum risk, we will only attempt to upgrade stub secrets that are currently spec 2. While we could potentially upgrade all stubs to the newest(which at the moment is 3.4.0) supported by the MCO, this may cause issues like https://issues.redhat.com/browse/MCO-1589 for some boot images that only support early spec 3 ignition(certain older boot images can't do can only do 3.0.0 and 3.1.0 ignition). Newer boot images can support all spec 3 stubs, so to preserve scaling ability as much as we can, we'll leave spec 3 stubs as is for the moment.
As a cluster administrator, I want to use Karpenter on an OpenShift cluster running in AWS to scale nodes instead of Cluster Autoscalar(CAS). I want to automatically manage heterogeneous compute resources in my OpenShift cluster without the additional manual task of managing node pools. Additional features I want are:
This feature covers the work done to integrate upstream Karpenter 1.x with ROSA HCP. This eliminates the need for manual node pool management while ensuring cost-effective compute selection for workloads. Red Hat manages the node lifecycle and upgrades.
The goal is roll this out with ROSA-HCP (AWS) since it has more mature Karpenter ecosystem, followed by ARO-HCP (Azure) implementation (refer to OCPSTRAT-1498).
This feature will be delivered in 3 Phases:
The Dev Preview release will expose AutoNode capabilities on Hosted Control Planes for AWS (note this is not meant to be productized on self-managed OpenShift). It includes the following capabilities:
OpenShift AutoNode (a.k.a. Karpenter) Proposal
As a cluster-admin or SRE I should be able to configure Karpenter with OCP on AWS. Both cli and UI should enable users to configure Karpenter and disable CAS.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | Managed, i.e. ROSA-HCP |
Classic (standalone cluster) | N/A |
Hosted control planes | yes |
Multi node, Compact (three node), or Single node (SNO), or all | MNO |
Connected / Restricted Network | Connected |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86_x64, ARM (aarch64) |
Operator compatibility | |
Backport needed (list applicable versions) | No |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | yes - console |
Other (please specify) | OCM, rosa-cli, ACM, cost management |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
Move from karpenter nodepool to programatic generate userdata
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
This is currently hardcoded to this value and clobbered by the controller in the default ec2nodeclass. https://github.com/openshift/hypershift/blob/f21f250592ea74b38e8d79555ab720982869ef5e/karpenter-operator/controllers/karpenter/karpenter_controller.go#L362
We need to default to a more neutral name and possibly stop clobbering it so it can be changed.
I see creating the role as an admin task. For hypershift "the project" this can be automated by the cli, so it will be created when cli create cluster with known name e.g karpenter-role-infra-id
While VAP is ok for implementing shared ownership it has some drawbacks. e.g. it forces us to change the builtin CEL rules of the API for required fields which is a maintenance burden and error prone. Besides it doesn't gives control to future api pivots we might need to execute to satisfy business needs. E.g. expose a field for dual stream support which requires picking a different ami, e.g expose a field for kubeletconfig that let us include that in the payload generation...
we should be ready to pivot to have our own class which exposes only a subset of the upstream and have a controller which just renders the upstream one
The ability in OpenShift to create trust and directly consume access tokens issued by external OIDC Authentication Providers using an authentication approach similar to upstream Kubernetes.
BYO Identity will help facilitate CLI only workflows and capabilities of the Authentication Provider (such as Keycloak, Dex, Azure AD) similar to upstream Kubernetes.
Ability in OpenShift to provide a direct, pluggable Authentication workflow such that the OpenShift/K8s API server can consume access tokens issued by external OIDC identity providers. Kubernetes provides this integration as described here. Customer/Users can then configure their IDPs to support the OIDC protocols and workflows they desire such as Client credential flow.
OpenShift OAuth server is still available as default option, with the ability to tune in the external OIDC provider as a Day-2 configuration.
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
_Items listed for GA https://issues.redhat.com/browse/OCPSTRAT-1804_
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Provide information that needs to be considered and planned so that documentation will meet customer needs. Initial completion during Refinement status.
Which other projects and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
Epic Goal
The ability to provide a direct authentication workflow such that OpenShift can consume bearer tokens issued by external OIDC identity providers, replacing the built-in OAuth stack by deactivating/removing its components as necessary.
Why is this important? (mandatory)
OpenShift has its own built-in OAuth server which can be used to obtain OAuth access tokens for authentication to the API. The server can be configured with an external identity provider (including support for OIDC), however it is still the built-in server that issues tokens, and thus authentication is limited to the capabilities of the oauth-server.
Scenarios (mandatory)
Dependencies (internal and external) (mandatory)
Contributing Teams(and contacts) (mandatory)
Acceptance Criteria (optional)
Drawbacks or Risk (optional)
Done - Checklist (mandatory)
In order to remove the RoleBindingRestriction CRD from the cluster, as outlined in the updates to the original OIDC enhancement proposal in CNTRLPLANE-69, we will have to make it such that CVO no longer manages it. This means updating the cluster-authentication-operator such that it is responsible for ensuring the CRD is present on the cluster.
The CAO and KAS-o both need to work and enable structured authentication configuration for the KAS static pods.
CAO:
KAS-o:
Description of problem:
This is a bug found during pre-merge test of 4.18 epic AUTH-528 PRs and filed for better tracking per existing "OpenShift - Testing Before PR Merges - Left-Shift Testing" google doc workflow.
co/console degraded with AuthStatusHandlerDegraded after OCP BYO external oidc is configured and then removed (i.e. reverted back to OAuth IDP).
Version-Release number of selected component (if applicable):
Cluster-bot build which is built at 2024-11-25 09:39 CST (UTC+800) build 4.18,openshift/cluster-authentication-operator#713,openshift/cluster-authentication-operator#740,openshift/cluster-kube-apiserver-operator#1760,openshift/console-operator#940
How reproducible:
Always (tried twice, both hit it)
Steps to Reproduce:
1. Launch a TechPreviewNoUpgrade standalone OCP cluster with above build. Configure htpasswd IDP. Test users can login successfully. 2. Configure BYO external OIDC in this OCP cluster using Microsoft Entra ID. KAS and console pods can roll out successfully. oc login and console login to Microsoft Entra ID can succeed. 3. Remove BYO external OIDC configuration, i.e. go back to original htpasswd OAuth IDP: [xxia@2024-11-25 21:10:17 CST my]$ oc patch authentication.config/cluster --type=merge -p=' spec: type: "" oidcProviders: null ' authentication.config.openshift.io/cluster patched [xxia@2024-11-25 21:15:24 CST my]$ oc get authentication.config cluster -o yaml apiVersion: config.openshift.io/v1 kind: Authentication metadata: annotations: include.release.openshift.io/ibm-cloud-managed: "true" include.release.openshift.io/self-managed-high-availability: "true" release.openshift.io/create-only: "true" creationTimestamp: "2024-11-25T04:11:59Z" generation: 5 name: cluster ownerReferences: - apiVersion: config.openshift.io/v1 kind: ClusterVersion name: version uid: e814f1dc-0b51-4b87-8f04-6bd99594bf47 resourceVersion: "284724" uid: 2de77b67-7de4-4883-8ceb-f1020b277210 spec: oauthMetadata: name: "" serviceAccountIssuer: "" type: "" webhookTokenAuthenticator: kubeConfig: name: webhook-authentication-integrated-oauth status: integratedOAuthMetadata: name: oauth-openshift oidcClients: - componentName: cli componentNamespace: openshift-console - componentName: console componentNamespace: openshift-console conditions: - lastTransitionTime: "2024-11-25T13:10:23Z" message: "" reason: OIDCConfigAvailable status: "False" type: Degraded - lastTransitionTime: "2024-11-25T13:10:23Z" message: "" reason: OIDCConfigAvailable status: "False" type: Progressing - lastTransitionTime: "2024-11-25T13:10:23Z" message: "" reason: OIDCConfigAvailable status: "True" type: Available currentOIDCClients: - clientID: 95fbae1d-69a7-4206-86bd-00ea9e0bb778 issuerURL: https://login.microsoftonline.com/6047c7e9-b2ad-488d-a54e-dc3f6be6a7ee/v2.0 oidcProviderName: microsoft-entra-id KAS and console pods indeed can roll out successfully; and now oc login and console login indeed can succeed using the htpasswd user and password: [xxia@2024-11-25 21:49:32 CST my]$ oc login -u testuser-1 -p xxxxxx Login successful. ... But co/console degraded, which is weird: [xxia@2024-11-25 21:56:07 CST my]$ oc get co | grep -v 'True *False *False' NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.18.0-0.test-2024-11-25-020414-ci-ln-71cvsj2-latest True False True 9h AuthStatusHandlerDegraded: Authentication.config.openshift.io "cluster" is invalid: [status.oidcClients[1].currentOIDCClients[0].issuerURL: Invalid value: "": oidcClients[1].currentOIDCClients[0].issuerURL in body should match '^https:\/\/[^\s]', status.oidcClients[1].currentOIDCClients[0].oidcProviderName: Invalid value: "": oidcClients[1].currentOIDCClients[0].oidcProviderName in body should be at least 1 chars long]
Actual results:
co/console degraded, as above.
Expected results:
co/console is normal.
Additional info:
In order to fully transition the management of the RoleBindingRestriction CRD to the cluster-authentication-operator, we also need to update the openshift/installer to use the cluster-authentication-operator render subcommand to add the RoleBindingRestriction CRD to the set of manifests applied during cluster bootstrapping.
Without the RoleBindingRestriction CRD included in the set of bootstrap manifests, the authorization.openshift.io/RestrictSubjectBindings admission plugin will prevent the creation of system:* RoleBindings during the installation process.
Feature Overview
This is a TechDebt and doesn't impact OpenShift Users.
As the autoscaler has become a key feature of OpenShift, there is the requirement to continue to expand it's use bringing all the features to all the cloud platforms and contributing to the community upstream. This feature is to track the initiatives associated with the Autoscaler in OpenShift.
Goals
Requirements
Requirement | Notes | isMvp? |
---|---|---|
vSphere autoscaling from zero | No | |
Upstream E2E testing | No | |
Upstream adapt scale from zero replicas | No | |
Out of Scope
n/a
Background, and strategic fit
Autoscaling is a key benefit of the Machine API and should be made available on all providers
Assumptions
Customer Considerations
Documentation Considerations
please note, the changes described by this epic will happen in OpenShift controllers and as such there is no "upstream" relationship in the same sense as the Kubernetes-based controllers.
As a developer, in order to deprecate the old annotations, we will need to carry both for at least one release cycle. Updating the CAO to apply the upstream annotations, and the CAS to accept both (preferring upstream), will allow me to properly deprecate the old annotations.
to help the transition to the upstream scale from zero annotations, we need to have the CAS recognize both sets of annotations, preferring the upstream, for at least one release cycle. this will allow us to have a window of deprecation on the old annotations.
Upstream K8s deprecated PodSecurityPolicy and replaced it with a new built-in admission controller that enforces the Pod Security Standards (See here for the motivations for deprecation).] There is an OpenShift-specific dedicated pod admission system called Security Context Constraints. Our aim is to keep the Security Context Constraints pod admission system while also allowing users to have access to the Kubernetes Pod Security Admission.
With OpenShift 4.11, we are turned on the Pod Security Admission with global "privileged" enforcement. Additionally we set the "restricted" profile for warnings and audit. This configuration made it possible for users to opt-in their namespaces to Pod Security Admission with the per-namespace labels. We also introduced a new mechanism that automatically synchronizes the Pod Security Admission "warn" and "audit" labels.
With OpenShift 4.19, we intend to move the global configuration to enforce the "restricted" pod security profile globally. With this change, the label synchronization mechanism will also switch into a mode where it synchronizes the "enforce" Pod Security Admission label rather than the "audit" and "warn".
Epic Goal
Get Pod Security admission to be run in "restricted" mode globally by default alongside with SCC admission.
When creating a custom SCC, it is possible to assign a priority that is higher than existing SCCs. This means that any SA with access to all SCCs might use the higher priority custom SCC, and this might mutate a workload in an unexpected/unintended way.
To protect platform workloads from such an effect (which, combined with PSa, might result in rejecting the workload once we start enforcing the "restricted" profile) we must pin the required SCC to all workloads in platform namespaces (openshift-, kube-, default).
Each workload should pin the SCC with the least-privilege, except workloads in runlevel 0 namespaces that should pin the "privileged" SCC (SCC admission is not enabled on these namespaces, but we should pin an SCC for tracking purposes).
The following tables track progress.
# namespaces | 4.19 | 4.18 | 4.17 | 4.16 | 4.15 | 4.14 |
---|---|---|---|---|---|---|
monitored | 82 | 82 | 82 | 82 | 82 | 82 |
fix needed | 68 | 68 | 68 | 68 | 68 | 68 |
fixed | 39 | 39 | 35 | 32 | 39 | 1 |
remaining | 29 | 29 | 33 | 36 | 29 | 67 |
~ remaining non-runlevel | 8 | 8 | 12 | 15 | 8 | 46 |
~ remaining runlevel (low-prio) | 21 | 21 | 21 | 21 | 21 | 21 |
~ untested | 3 | 2 | 2 | 2 | 82 | 82 |
# | namespace | 4.19 | 4.18 | 4.17 | 4.16 | 4.15 | 4.14 |
---|---|---|---|---|---|---|---|
1 | oc debug node pods | ![]() |
![]() |
![]() |
![]() |
![]() |
|
2 | openshift-apiserver-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
3 | openshift-authentication | ![]() |
![]() |
![]() |
![]() |
![]() |
|
4 | openshift-authentication-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
5 | openshift-catalogd | ![]() |
![]() |
![]() |
![]() |
![]() |
|
6 | openshift-cloud-credential-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
7 | openshift-cloud-network-config-controller | ![]() |
![]() |
![]() |
#2496 | ||
8 | openshift-cluster-csi-drivers | #118 #5310 #135 | #524 #131 #306 #265 #75 | #170 #459 | ![]() |
||
9 | openshift-cluster-node-tuning-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
10 | openshift-cluster-olm-operator | ![]() |
![]() |
![]() |
![]() |
n/a | n/a |
11 | openshift-cluster-samples-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
12 | openshift-cluster-storage-operator | ![]() |
![]() |
#459 #196 | ![]() |
||
13 | openshift-cluster-version | #1038 | ![]() |
||||
14 | openshift-config-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
15 | openshift-console | ![]() |
![]() |
![]() |
![]() |
![]() |
|
16 | openshift-console-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
17 | openshift-controller-manager | ![]() |
![]() |
![]() |
![]() |
![]() |
|
18 | openshift-controller-manager-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
19 | openshift-e2e-loki | ![]() |
![]() |
![]() |
![]() |
![]() |
|
20 | openshift-image-registry | #1008 | ![]() |
||||
21 | openshift-ingress | #1032 | |||||
22 | openshift-ingress-canary | #1031 | |||||
23 | openshift-ingress-operator | #1031 | |||||
24 | openshift-insights | ![]() |
![]() |
![]() |
![]() |
![]() |
|
25 | openshift-kni-infra | ![]() |
![]() |
![]() |
![]() |
![]() |
|
26 | openshift-kube-storage-version-migrator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
27 | openshift-kube-storage-version-migrator-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
28 | openshift-machine-api | #1308 #1317 | #1311 | #407 | #315 #282 #1220 #73 #50 #433 | ![]() |
|
29 | openshift-machine-config-operator | ![]() |
![]() |
#4219 | #4384 | ![]() |
|
30 | openshift-manila-csi-driver | ![]() |
![]() |
![]() |
![]() |
![]() |
|
31 | openshift-marketplace | ![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
32 | openshift-metallb-system | ![]() |
![]() |
![]() |
#241 | ||
33 | openshift-monitoring | #2298 #366 | #2498 | #2335 | ![]() |
||
34 | openshift-network-console | ![]() |
![]() |
||||
35 | openshift-network-diagnostics | ![]() |
![]() |
![]() |
#2496 | ||
36 | openshift-network-node-identity | ![]() |
![]() |
![]() |
#2496 | ||
37 | openshift-nutanix-infra | ![]() |
![]() |
![]() |
![]() |
![]() |
|
38 | openshift-oauth-apiserver | ![]() |
![]() |
![]() |
![]() |
![]() |
|
39 | openshift-openstack-infra | ![]() |
![]() |
![]() |
![]() |
||
40 | openshift-operator-controller | ![]() |
![]() |
![]() |
![]() |
![]() |
|
41 | openshift-operator-lifecycle-manager | ![]() |
![]() |
![]() |
![]() |
![]() |
|
42 | openshift-route-controller-manager | ![]() |
![]() |
![]() |
![]() |
![]() |
|
43 | openshift-service-ca | ![]() |
![]() |
![]() |
![]() |
![]() |
|
44 | openshift-service-ca-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
45 | openshift-sriov-network-operator | ![]() |
![]() |
![]() |
![]() |
![]() |
|
46 | openshift-user-workload-monitoring | ![]() |
![]() |
![]() |
![]() |
![]() |
|
47 | openshift-vsphere-infra | ![]() |
![]() |
![]() |
![]() |
![]() |
|
48 | (runlevel) kube-system | ||||||
49 | (runlevel) openshift-cloud-controller-manager | ||||||
50 | (runlevel) openshift-cloud-controller-manager-operator | ||||||
51 | (runlevel) openshift-cluster-api | ||||||
52 | (runlevel) openshift-cluster-machine-approver | ||||||
53 | (runlevel) openshift-dns | ||||||
54 | (runlevel) openshift-dns-operator | ||||||
55 | (runlevel) openshift-etcd | ||||||
56 | (runlevel) openshift-etcd-operator | ||||||
57 | (runlevel) openshift-kube-apiserver | ||||||
58 | (runlevel) openshift-kube-apiserver-operator | ||||||
59 | (runlevel) openshift-kube-controller-manager | ||||||
60 | (runlevel) openshift-kube-controller-manager-operator | ||||||
61 | (runlevel) openshift-kube-proxy | ||||||
62 | (runlevel) openshift-kube-scheduler | ||||||
63 | (runlevel) openshift-kube-scheduler-operator | ||||||
64 | (runlevel) openshift-multus | ||||||
65 | (runlevel) openshift-network-operator | ||||||
66 | (runlevel) openshift-ovn-kubernetes | ||||||
67 | (runlevel) openshift-sdn | ||||||
68 | (runlevel) openshift-storage |
The PodSecurityAdmissionLabelSynchronizationController in cluster-policy-controller should set an annotation, that tells us what it decided wrt given namespace.
Requirement | Notes | isMvp? |
---|---|---|
CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
This Section:
For users with strict regulatory policies, Private Service Connect allows private consumption of services across VPC networks that belong to different groups, teams, projects, or organizations. Supporting OpenShift to consume these private endpoints is key for these customers to be able to deploy the platform on GCP and be complaint with their regulatory policies.
Questions to be addressed:
Requirement | Notes | isMvp? |
---|---|---|
CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
This Section:
For users with strict regulatory policies, Private Service Connect allows private consumption of services across VPC networks that belong to different groups, teams, projects, or organizations. Supporting OpenShift to consume these private endpoints is key for these customers to be able to deploy the platform on GCP and be complaint with their regulatory policies.
Questions to be addressed:
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Add the ability to choose subnets for IngressControllers with LoadBalancer-type Services for AWS in the Installer. This install config should be applies to the default IngressController and all future IngressControllers (the design is similar to installconfig.platform.aws.lbtype).
Cluster Admins may have dedicated subnets for their load balancers due to security reasons or infrastructure constraints. With the implementation in NE-705, Cluster Admins will be able to specify subnets for IngressControllers for Day 2 operations. Service Delivery needs a way to configure IngressController subnets for the default IngressController for ROSA.
If the cluster is spread across multiple subnets then we can have a way to select the subnet while creating ingresscontroller of type LoadBalancerService.
Description of criteria:
.
Static validations (no API connection required)
Description of criteria:
Some validations are extracted from API validation (i.e. the intstaller does not handle CEL at this time) and Xvalidation markers (i.e. defined in the enhancement proposal).
Validations that require access to the AWS API will go in pkg/asset (different card)
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As an openshift-instlal user I want to be able to specify AWS subnets with roles.
Description of criteria:
Validations will be handled in a different card.
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Implement Migration core for MAPI to CAPI for AWS
When customers use CAPI, There must be no negative effect to switching over to using CAPI . Seamless migration of Machine resources. the fields in MAPI/CAPI should reconcile from both CRDs.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
VolumeSize on the block device mapping spec in MAPA is currently optional (and if is not set we send an empty value to AWS and let it choose for us), where it is required and a minimum of 8gb in CAPA.
We need to determine an appropriate behaviour for when the value is unset.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
To enable CAPI MachineSets to still mirror MAPI MachineSets accurately, and to enable MAPI MachineSets to be implemented by CAPI MachineSets in the future, we need to implement a way to convert CAPI Machines back into MAPI Machines.
These steps assume that the CAPI Machine is authoritative, or, that there is no MAPI Machines.
For the MachineSet controller, we need to implement a forward conversion, converting the MachineAPI MachineSet to ClusterAPI.
This will involve creating the CAPI MachineSet if it does not exist, and managing the Infrastructure templates.
This card covers the case where MAPI is currently authoritative.
As QE have tried to test upstream CAPI pausing, we've hit a few issues with running the migration controller. & cluster capi operator on a real cluster vs envtest.
This card captures the work required to iron out these kinks, and get things running (i.e not crashing).
I also think we want an e2e or some sort of automated testing to ensure we don't break things again.
Goal: Stop the CAPI operator crashing on startup in a real cluster.
Non goals: get the entire conversion flow running from CAPI -> MAPI and MAPI -> CAPI. We still need significant feature work before we're here.
For the Machine controller, we need to implement a forward conversion, converting the MachineAPI Machine to ClusterAPI.
This will involve creating the CAPI Machine if it does not exist, and managing the Infrastructure Machine.
This card covers the case where MAPI is currently authoritative.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
As a user I want to be able to add labels/taints/annotations to my machines, and have them propagate to the nodes. This will allow me to use the labels for other tasks e.g selectors.
Currently, MAPI supports propagating labels from machines to nodes, but CAPI does not. When we move to CAPI we will lose this feature.
_See https://issues.redhat.com/browse/OCPBUGS-37236_
Relevant upstream issues:
Epic Goal
This is the epic tracking the work to collect a list of TLS artifacts (certificates, keys and CA bundles).
This list will contain a set of required an optional metadata. Required metadata examples are ownership (name of Jira component) and ability to auto-regenerate certificate after it has expired while offline. In most cases metadata can be set via annotations to secret/configmap containing the TLS artifact.
Components not meeting the required metadata will fail CI - i.e. when a pull request makes a component create a new secret, the secret is expected to have all necessary metadata present to pass CI.
This PR will enforce it WIP API-1789: make TLS registry tests required
Description of problem:
In order to make TLS registry tests required we need to make sure all OpenShift variants are using the same metadata for kube-apiserver certs. Hypershift uses several certs stored in the secret without accompanying metadata (namely component ownership).
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
In order to keep track of existing certs/CA bundles and ensure that they adhere to requirements we need to have a TLS artifact registry setup.
The registry would:
Ref: API-1622
To improve automation, governance and security, AWS customers extensively use AWS Tags to track resources. Customers wants the ability to change user tags on day 2 without having to recreate a new cluster to have 1 or more tags added/modified.
The observable functionality that the user now has as a result of receiving this feature. Complete during New status.
A list of specific needs or objectives that a feature must deliver in order to be considered complete. Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc. Initial completion during Refinement status.
This feature will only apply to ROSA with Hosted Control Planes, and ROSA Classic / standalone is excluded.
As a cluster administrator, I want to use Karpenter on an OpenShift cluster running in AWS to scale nodes instead of Cluster Autoscalar(CAS). I want to automatically manage heterogeneous compute resources in my OpenShift cluster without the additional manual task of managing node pools. Additional features I want are:
This feature covers the work done to integrate upstream Karpenter 1.x with ROSA HCP. This eliminates the need for manual node pool management while ensuring cost-effective compute selection for workloads. Red Hat manages the node lifecycle and upgrades.
The goal is roll this out with ROSA-HCP (AWS) since it has more mature Karpenter ecosystem, followed by ARO-HCP (Azure) implementation (refer to OCPSTRAT-1498).
This feature will be delivered in 3 Phases:
The Dev Preview release will expose AutoNode capabilities on Hosted Control Planes for AWS (note this is not meant to be productized on self-managed OpenShift) as APIs for Managed Services (ROSA) to consume. It includes the following capabilities:
OpenShift AutoNode (a.k.a. Karpenter) Proposal
As a cluster-admin or SRE I should be able to configure Karpenter with OCP on AWS. Both cli and UI should enable users to configure Karpenter and disable CAS.
Anyone reviewing this Feature needs to know which deployment configurations that the Feature will apply to (or not) once it's been completed. Describe specific needs (or indicate N/A) for each of the following deployment scenarios. For specific configurations that are out-of-scope for a given release, ensure you provide the OCPSTRAT (for the future to be supported configuration) as well.
Deployment considerations | List applicable specific needs (N/A = not applicable) |
Self-managed, managed, or both | Managed, i.e. ROSA-HCP |
Classic (standalone cluster) | N/A |
Hosted control planes | yes |
Multi node, Compact (three node), or Single node (SNO), or all | MNO |
Connected / Restricted Network | Connected |
Architectures, e.g. x86_x64, ARM (aarch64), IBM Power (ppc64le), and IBM Z (s390x) | x86_x64, ARM (aarch64) |
Operator compatibility | |
Backport needed (list applicable versions) | No |
UI need (e.g. OpenShift Console, dynamic plugin, OCM) | yes - console |
Other (please specify) | OCM, rosa-cli, ACM, cost management |
Include use case diagrams, main success scenarios, alternative flow scenarios. Initial completion during Refinement status.
<your text here>
Include a list of refinement / architectural questions that may need to be answered before coding can begin. Initial completion during Refinement status.
<your text here>
High-level list of items that are out of scope. Initial completion during Refinement status.
Provide any additional context is needed to frame the feature. Initial completion during Refinement status.
Provide any additional customer-specific considerations that must be made when designing and delivering the Feature. Initial completion during Refinement status.
Provide information that needs to be considered and planned so that documentation will meet customer needs. If the feature extends existing functionality, provide a link to its current documentation. Initial completion during Refinement status.
Which other projects, including ROSA/OSD/ARO, and versions in our portfolio does this feature impact? What interoperability test scenarios should be factored by the layered products? Initial completion during Refinement status.
<your text here>
For CAPI/MAPI driven machine management the cluster-machine-approver uses the machine.status.ips to match the CSRs. In karpenter there's no Machine resources
We'll need to implement something similar. Some ideas:
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
We'll need to implement something similar. Some ideas:
– Explore using the nodeClaim resource info like status.providerID to match the CSRs
– Store the requesting IP when the ec2 instances query ignition and follow similar comparison criteria than machine approver to match CSRs
– Query AWS to get info and compare info to match CSRs
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
https://github.com/openshift/hypershift/pull/5349 introduced a new controller to implement auto approval for kubelet client CSRs. We need to extend to also approve serving CSRs since they are not auto approved by the cluster-machine-approver
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
We decided to start by using validating admission policies to implement ownership of ec2NodeClass fields. So we can restrict fields crud to a particular service account.
This has some caveats:
If using validating policies for this proves to be satisfactory we'll need to consider alternatives, e.g:
Bring https://issues.redhat.com/browse/CNF-11805 to a GA solution:
- end to end CI with sunny and rainy days scenarios: we want to make sure that the control plane cannot be affected by any workload, ideally, and worse case, clearly document workloads profiles that will put the control plane at risk.
Business justification: request from NEPs and end customers (not listed in the title, please check then tracker links) who do want to have one single type of servers in their clusters, those servers hosting more than 200 CPUs (512 is now common for AMD based servers, and e can expect more in the future). Having 3 servers dedicated just to run control plane is not OPEX neither CAPEX efficient, as 90%+ of the server capacity will never be used. This is even worse for 30 nodes clusters (very common size).
Finally, our HUB clusters (hosting ACM), infrastructure clusters (hosting HCP for instance) are tiny clusters, and the above applies: we cannot require customers to use servers with less than 32 CPUs (for instance).
Phase 1 goals:
Come up with a proposal of what level of isolation and workload limitations (only easy workload? DPDK?) are acceptable on the control plane nodes. In other words, lets take all the easy hanging fruits and see what the platform can do while maintaining control plane stability.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
To date our work within the telecommunications radio access network space has focused primarily on x86-based solutions. Industry trends around sustainability, and more specific discussions with partners and customers, indicate a desire to progress towards ARM-based solutions with a view to production deployments in roughly a 2025 timeframe. This would mean being able to support one or more RAN partners DU applications on ARM-based servers.
Depending on source 75-85% of service provider network power consumption is attributable to the RAN sites, with data centers making up the remainder. This means that in the face of increased downward pressure on both TCO and carbon footprint (the former for company performance reasons, the later for regulatory reasons) it is an attractive place to make substantial improvements using economies of scale.
There are currently three main obvious thrusts to how to go about this:
This BU priority focuses on the third of these approaches.
Reference Documents:
The PerformanceProfile currently allows the user to select either the standard kernel (by default) or the realtime kernel, using the realTimeKernel field. However, for some use cases (e.g. Nvidia based ARM server) a kernel with 64k page size is required. This is supported through the MachineConfig kernelType, which currently supports the following options:
At some point it is likely that 64k page support will be added to the realtime kernel, which would likely mean another "realtime-64k-pages" option (or similar) would be added.
The purpose of this epic is to allow the 64k-pages (standard kernel with 64k pages) option to be selected in the PerformanceProfile and make it easy to support new kernelTypes added to the MachineConfig. There is a workaround for this today, by applying an additional MachineConfig CR, which overrides the kernelType, but this is awkward for the user.
One option to support this in the PerformanceProfile would be to deprecate the existing realTimeKernel option and replace it with a new kernelType option. The kernelType option would support the same values as the MachineConfig kernelType (i.e. default, realtime, 64k-pages). The old option could be supported for backwards compatibility - attempting to use both options at the same time would be treated as an error. Another option would be to add a new kernelPageSize option (with values like default or 64k) and then internally map that to the MachineConfig kernelType (after validation that the combination of kernel type and page size was allowed).
This will require updates to the customer documentation and to the performance-profile-creator to support the new option.
This will also might require updates to the workloadhints kernel related sections.
https://docs.kernel.org/mm/vmemmap_dedup.html
Reading this doc, we can see a mapping of kernel page size and hugepages support.
We want to enforce that the combination of hugepages and kernelPageSize entered by the users in the performance profile is valid (done by validation webhook).
Acceptance criteria:
PERSONAS:
The following personas are borrowed from Hypershift docs used in the user stories below.
USER STORY:
ACCEPTANCE CRITERIA:
What is "done", and how do we measure it? You might need to duplicate this a few times.
Given a
When b
Then c
CUSTOMER EXPERIENCE:
Only fill this out for Product Management / customer-driven work. Otherwise, delete it.
BREADCRUMBS:
Where can SREs look for additional information? Mark with "N/A" if these items do not exist yet so Functional Teams know they need to create them.
NOTES:
If there's anything else to add.
To reconcile capabilities at runtime correctly, we need to constantly reconcile the CVO capabilities in the HostedClusterConfigOperator.
This was introduced in https://github.com/openshift/hypershift/pull/1687 and causes the capabilities to reset after being set initially after the installation.
This logic needs to take into account the enabled/disabled state of the image registry and the desired state of the hypershift control plane config CRD to render the capabilities correctly.
Ref from ARO-13685 where this was simply removed: https://github.com/openshift/hypershift/pull/5315/files#diff-c888020c5dc46c458d818b931db97131d6e35b90661fe1030a39ebeac8859b19L1224
Mark with an X when done; strikethrough for non-applicable items. All items
must be considered before closing this issue.
[ ] Ensure all pull request (PR) checks, including ci & e2e, are passing
[ ] Document manual test steps and results
[ ] Manual test steps executed by someone other than the primary implementer or a test artifact such as a recording are attached
[ ] All PRs are merged
[ ] Ensure necessary actions to take during this change's release are communicated and documented
[ ] Troubleshooting Guides (TSGs), ADRs, or other documents are updated as necessary
When the cluster is installed and configured, the CVO container has init containers that apply the initial CRDs and remove some manifests from CVO.
Depending on the configuration state of the image registry, it needs to remove or keep the capability as enabled. The bootstrap init container needs to set the initial capabilities as AdditionalEnabledCapabilities and the BaselineCapabilities as None.
Note there is a gotcha on the ordering of the capabilities, they always need to be sorted ascending for the API to accept them - otherwise they are silently ignored.
Note, we need to implement the same change in V1 and V2 of the operator:
1.
2.
XREF implementation from ARO-13685: https://github.com/openshift/hypershift/pull/5315/files#diff-520b6ecfad21e6c9bc0cbf244ff694cf5296ffc8c0318cb2248eb7185a36cd8aR363-R366
Mark with an X when done; strikethrough for non-applicable items. All items
must be considered before closing this issue.
[ ] Ensure all pull request (PR) checks, including ci & e2e, are passing
[ ] Document manual test steps and results
[ ] Manual test steps executed by someone other than the primary implementer or a test artifact such as a recording are attached
[ ] All PRs are merged
[ ] Ensure necessary actions to take during this change's release are communicated and documented
[ ] Troubleshooting Guides (TSGs), ADRs, or other documents are updated as necessary
Currently the ImagePolicyConfig has the internal image registry url hardcoded
When the image registry is disabled, we need to ensure that this is properly accounted for, as the registry won't be reachable under this address anymore.
Mark with an X when done; strikethrough for non-applicable items. All items
must be considered before closing this issue.
[ ] Ensure all pull request (PR) checks, including ci & e2e, are passing
[ ] Document manual test steps and results
[ ] Manual test steps executed by someone other than the primary implementer or a test artifact such as a recording are attached
[ ] All PRs are merged
[ ] Ensure necessary actions to take during this change's release are communicated and documented
[ ] Troubleshooting Guides (TSGs), ADRs, or other documents are updated as necessary
Enhancement https://github.com/openshift/enhancements/pull/1729
We want to add the disabled capabilities to the hosted cluster CRD as described in the above enhancement.
AC:
This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were completed when this image was assembled
This epic tracks any part of our codebase / solutions we implemented taking shortcuts.
Whenever a shortcut is taken, we should add a story here not to forget to improve it in a safer and more maintainabile way.
maintanability and debuggability, and in general fighting the technical debt, is critical to keep velocity and ensure overall high quality
https://issues.redhat.com/browse/CNF-796
https://issues.redhat.com/browse/CNF-1479
https://issues.redhat.com/browse/CNF-2134
https://issues.redhat.com/browse/CNF-6745
https://issues.redhat.com/browse/CNF-8036
https://issues.redhat.com/browse/CNF-9566
https://issues.redhat.com/browse/OCPNODE-2217
Since OCP 4.18, crun will be used as the default runtime instead of runc.
We need to adjust NTO to conform with this change.
Collection of Auth stories that are of organizational kind.
Reference: https://issues.redhat.com/browse/KONFLUX-1611?filter=-1
ACM/MCE flow: https://miro.com/app/board/uXjVLqVek9E=/ (last diagram)
List of component:
A lot of time our pipelines as well as other teams' pipelines are stuck because they are unable to provision hosts with different architectures to build the images.
Because we currently don't use the multi-arch images we build with konflux, we will stop building multi-arch for now and readd those architectures when we need them.
Currently each component is in it's own application which means we need a Release object for each one.
We want to have a single application that has all of our components in order to be able to release with a single Release object.
The way to do this is:
All components need to build images for the following architectures:
List of components:
In order to release our products with konflux we need to pass the registry-standard EnterpriseContractPolicy.
There are a few things small things we need to configure for all of our components:
Integration uses latest tags, so update the push-event pipelines to also push latest tag for the following components:
The ignores need to be listed in the tekton pipelines.
The relevant repos are:
Currently our base images use ubi9 and MintMaker wants to upgrade it to 9.5, because MintMaker looks at the repo and tries to upgrade to the newest tag.
We should change the base image to use a repo that is just for rhel9.4.
Adept the current dockerfiles and add them to the upstream repos.
List of components:
Add the dockerfiles for the following components:
List of repo
- assisted-events-stream
- assisted-image-service
- assisted-service
- auto-report
- bug-master-bot
- jira-unfurl-bot (auto-merge)
- prow-jobs-scraper
- assisted-installer (all config)
- assisted-installer-agent (all config)
Epic Goal
Kubesan requires these host level dependencies:
To ensure that operator-provisioned nodes support kubesan, these deps must be added to the machine-config-operator's config.
Revert the PRs that added the extension and packages needed for kSAN Storage.
https://github.com/openshift/machine-config-operator/pull/4792 and https://github.com/openshift/os/pull/1701
Note: This epic only tracks the phase one of the work listed in the RFC
This effort is tracked at: https://github.com/operator-framework/operator-controller/issues/1646
UPSTREAM:
DOWNSTREAM:
Consolidate the catalogd and operator-controller kustomize configuration dirs. e.g.
```
$ tree -d config
config
├── catalogd
└── operator-controller
```
Goal: We should be able to render separate overlays for each project. However, de-duplication of kustomize configuration files could be done.
As an SRE, I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were not completed when this image was assembled
What
Merge upstream kube-rbac-proxy v0.18.2 into downstream.
Why
We don't want to log the tokens even if the verbosity is set to high
Make sure that we have dependencies updated and that we are building on the proper base images and with the proper toolchain for OpenShift 4.19.
if possible we might be able to reuse upstream tests, based on decisions from PODAUTO-323.
As a developer, I want to update all go dependencies:
Note: As this is the first time we are doing this, we should provide best-effort here by upgrading the dependencies that have no conflict, and creating cards for the ones that need more effort. Those cards can be linked to the 4.20 Regular Maintenance epic if/when created so we can better plan for the additional effort.
The OpenShift API currently defines two CustomResourceDefinitions (APIs) related to Insights. These are:
Those CRDs/APIs are currently in v1alpha1 version, which means that they are available only in the techpreview clusters (featureSet=TechPreviewNoUpgrade). Note that the two CRDs share definitions/attributes. The goal of this epic is to promote these CRDs to some higher version (v1beta or v1)
Previous work (on the CRDs/APIs) is tracked in the:
We would like to add an option to specify the PVC to be used by on-demand gathering jobs for storing the final archive. We plan to extend the existing API of the DataGather and InsightsDataGather CRD to allow specifying the PVC name and mount path
Capture the necessary accidental work to get CI / Konflux unstuck during the 4.19 cycle
Due to capacity problems on the s390x environment, the Konflux team recommended disabling the s390x platform from the PR pipeline.
Dev and QE CI improvements to cover upgrades of both the management and the hosted clusters
As an Admin I want to have an easy view of the alerts that are firing in my cluster.
If I have the same alert that fires many time it is very hard to identify the issues.
We can simplify the existing Alerts page to make it much clearer by the following quick fix:
1. Aggregate the alerts by alert name and severity
2. For each aggregated line add the "Total alerts number"
3. When pressing on the aggregated line it can be expanded with the list of alerts
4.Optional - Add the namespace label to the expanded list of alerts, where each alert can have a different namespace. Note: Not all alerts have this label.
Initial mockup by Foday Kargbo:
https://www.figma.com/proto/JlYlMG19eF8IJasy7DF4ND/Alert-Grouping-(Quick-fix)?page-id=32%3A1503&node-id=197-850&viewport=972%2C-400%2C0.59&t=sCVxsT84aZy1kLu2-1&scaling=min-zoom&content-scaling=fixed
Given we have several alerts of the same name and severity
When we go to the Alerts page and view the alerts
Then We would see a single line for each "alert name" and "severity", the number of times its shown and I can click on the line to expend the line and get the full list of alerts of that name and severity.
As an Admin I want to have an easy view of the alerts that are firing in my cluster.
If I have the same alert that fires many time it is very hard to identify the issues.
We can simplify the existing Alerts page to make it much clearer by the following quick fix:
1. Aggregate the alerts by alert name and severity
2. For each aggregated line add the "Total alerts number"
3. When pressing on the aggregated line it can be expanded with the list of alerts
4.Optional - Add the namespace label to the expanded list of alerts, where each alert can have a different namespace. Note: Not all alerts have this label.
Initial mockup by Foday Kargbo:
https://www.figma.com/proto/JlYlMG19eF8IJasy7DF4ND/Alert-Grouping-(Quick-fix)?page-id=32%3A1503&node-id=197-850&viewport=972%2C-400%2C0.59&t=sCVxsT84aZy1kLu2-1&scaling=min-zoom&content-scaling=fixed
Given we have several alerts of the same name and severity
When we go to the Alerts page and view the alerts
Then We would see a single line for each "alert name" and "severity", the number of times its shown and I can click on the line to expend the line and get the full list of alerts of that name and severity.
Description of problem:
@types/react-* was not updated with react itself, hiding many type errors
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Location:
PF component:
AC:
As a user already accustomed to PatternFly-based web applications, when I use the CodeEditor within OpenShift Console, I expect the same experience as the PF CodeEditor
An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.
When running ./build-frontend.sh, I am getting the following warnings in the build log:
warning " > cypress-axe@0.12.0" has unmet peer dependency "axe-core@^3 || ^4". warning " > cypress-axe@0.12.0" has incorrect peer dependency "cypress@^3 || ^4 || ^5 || ^6".
To fix:
As part of improving the security, traceability, and reliability of our build processes, we are enabling hermetic builds for the Cluster Observability Operator (COO). Hermetic builds ensure that the build process is self-contained, with no external dependencies fetched during runtime, thus protecting against potential supply chain threats. This EPIC will involve modifying the current build pipelines, adding the necessary configuration to enforce hermetic builds, and verifying that all dependencies are properly declared in advance to meet security best practices.
Description: Modify the Containerfile for the ui-monitoring to ensure compatibility with hermetic builds. The build should not rely on external dependencies and should ensure all required resources are available locally before the build starts.
Acceptance Criteria:
The installer makes heavy use of it's data/data directory, which contains hundreds of files in various subdirectories that are mostly used for inserting into ignition files. From these files, autogenerated code is created that includes the contents in the installer binary.
Unfortunately, subdirectories that do not contain .go files are not regarded by as golang packages and are therefore not included when building the installer as a library: https://go.dev/wiki/Modules#some-needed-files-may-not-be-present-in-populated-vendor-directory
This is currently handled in the installer fork repo by deleting the compile-time autogeneration and instead doing a one-time autogeneration that is checked in to the repo: https://github.com/openshift/installer-aro/pull/27/commits/26a5ed5afe4df93b6dde8f0b34a1f6b8d8d3e583
Since this does not exist in the upstream installer, we will need some way to copy the data/data associated with the current installer version into the wrapper repo - we should probably encapsulate this in a make vendor target. The wiki page above links to https://github.com/goware/modvendor which unfortunately doesn't work, because it assumes you know the file extensions of all of the files (e.g. .c, .h), and it can't handle directory names matching the glob. We could probably easily fix this by forking the tool and teaching it to ignore directories in the source. Alternatively, John Hixson has a script that can do something similar.
Rebase openshift/etcd release-4.19 to upstream etcd 3.5.18
Rebase openshift/etcd release-4.19 to upstream etcd 3.5.19
We are constantly bumping up against quotas when trying to create new ServicePrincipals per test. Example:
=== NAME TestCreateClusterV2
hypershift_framework.go:291: failed to create cluster, tearing down: failed to create infra: ERROR: The directory object quota limit for the Tenant has been exceeded. Please ask your administrator to increase the quota limit or delete objects to reduce the used quota.
We need to create a set of ServicePrincipals to use during testing, and we need to reuse them while executing the e2e-aks.
When adding the assignContributorRole to assign contributor roles for appropriate scopes to existing SPs we missed the assignment of the role over the DNS RG scope
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
With the new changes in powervs workspace delete process, need to make sure all the child resources are cleaned up before attempting to delete powervs instance.
Child resources:
The error propagation is generally speaking not 1-to-1. The operator status will generally capture the pool status, but the full error from Controller/Daemon does not fully bubble up to pool/operator, and the journal logs with error generally don’t get bubbled up at all. This is very confusing for customers/admins working with the MCO without full understanding of the MCO’s internal mechanics:
Using “unexpected on-disk state” as an example, this can be caused by any amount of the following:
Etc. etc.
Since error use cases are wide and varied, there are many improvements we can perform for each individual error state. This epic aims to propose targeted improvements to error messaging and propagation specifically. The goals being:
With a side objective of observability, including reporting all the way to the operator status items such as:
Approaches can include:
MCD will send an alert when a node failes to pivot to another MachineConfig. This could prevent an OS upgrade from succeeding. The alert contains the information on what logs to look for
The alerts describes the following
"Error detected in pivot logs on {{ $labels.node }} , upgrade may be blocked. For more details: oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon "
It is possible that admin may not be able to interpret exact action to be taken after looking at MCD pod logs. Adding runbook (https://github.com/openshift/runbooks) can help admin in better troubleshooting and taking appropriate action.
Acceptance Criteria:
MCO will send an alert when a node for 15 minutes, a specific node is using more memory than is reserved.
The alerts describes the following
"summary: "Alerts the user when, for 15 minutes, a specific node is using more memory than is reserved"
description: "System memory usage of {{ $value | humanize }} on {{ $labels.node }} exceeds 95% of the reservation. Reserved memory ensures system processes can function even when the node is fully allocated and protects against workload out of memory events impacting the proper functioning of the node. The default reservation is expected to be sufficient for most configurations and should be increased (https://docs.openshift.com/container-platform/latest/nodes/nodes/nodes-nodes-managing.html) when running nodes with high numbers of pods (either due to rate of change or at steady state).""
It is possible that admin may not be able to interpret exact action to be taken after looking at the alert and the cluster state. Adding runbook (https://github.com/openshift/runbooks) can help admin in better troubleshooting and taking appropriate action.
Acceptance Criteria:
MCC sends drain alert when a node fails to reboot in a span of 5 minutes This is to make sure that admin takes appropriate action if required by looking at the pod logs. Alert contains the information on where to look for the logs.
Example alert looks like:
Reboot failed on {{ $labels.node }} , update may be blocked. For more details: oc logs -f -n {{ $labels.namespace }} {{ $labels.pod }} -c machine-config-daemon
It is possible that admin may not be able to interpret exact action to be taken after looking at MCC pod logs. Adding runbook (https://github.com/openshift/runbooks) can help admin in better troubleshooting and taking appropriate action.
Acceptance Criteria:
Currently, there are mainly three update paths built in parallel within the MCO. They separately take care of non-image updates, image updates, and updates for pools that have opted in to On Cluster Layering. As a new bootc update path will be added in with the introduction of this enhancement, MCO is looking for a better way to better manage these four update paths who manage different types of update but also shares a lot of things in common (e.g. check reconcilability). Interest and Proposals in refactoring the MCD functions and creating a unified update interface has been raised several times in previous discussions:
Description of problem:
When the reboot process is broken a MCDRebootError alert should be raised. Nevertheless, the alert is not raise, and the mcp is degraded with a wrong message E1028 17:22:38.515751 45330 writer.go:226] Marking Degraded due to: failed to update OS to quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3: error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3: error: Old and new refs are equal: ostree-unverified-registry:quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3 If the reboot process is fixed the node cannot be recovered and remains stuck reporting the " Old and new refs are equal" error.
Version-Release number of selected component (if applicable):
IPI on AWS: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.nightly-2024-10-28-052434 True False 8h Error while reconciling 4.18.0-0.nightly-2024-10-28-052434: an unknown error has occurred: MultipleErrors
How reproducible:
Always
Steps to Reproduce:
1. Enable OCL 2. Break the reboot $ oc debug node/$(oc get nodes -l node-role.kubernetes.io/worker -ojsonpath="{.items[0].metadata.name}") -- chroot /host sh -c "mount -o remount,rw /usr; mv /usr/bin/systemd-run /usr/bin/systemd-run2" Starting pod/sregidor-ver1-w48rv-worker-a-rln2vcopenshift-qeinternal-debug ... To use host binaries, run `chroot /host` 3. Wait for a MCDRebootError to be raised and check that the MCP is degraded with message: "reboot command failed, something is seriously wrong"'
Actual results:
The MCDRebootError alert is not raised and the MCP is degraded with the wrong message - lastTransitionTime: "2024-10-28T16:40:43Z" message: 'Node ip-10-0-51-0.us-east-2.compute.internal is reporting: "failed to update OS to quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3: error running rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3: error: Old and new refs are equal: ostree-unverified-registry:quay.io/mcoqe/layering@sha256:c56f19230be27cbc595d9467bcbc227858e097964ac5e5e7e74c5242aaca61e3\n: exit status 1"' reason: 1 nodes are reporting degraded status on sync status: "True" type: NodeDegraded
Expected results:
The alert should be raised and the mcp should be degraded with the right message
Additional info:
If OCL is disabled this functionality works as expected.
These are items that the team has prioritized to address in 4.18.
In OCP 4.7 and before, you were able to see the MCD logs of the previous container post upgrade. Now it seems that we no longer do in newer versions. I am not sure if this is a change in kube pod logging behaviour, how the pod gets shutdown and brought up, or something in the MCO.
This however makes it relatively hard to debug in newer versions of the MCO, and in numerous bugs we could not pinpoint the source of the issue since we no longer have necessary logs. We should find a way to properly save the previous boot MCD logs if possible.
Today the MCO bootstraps with a bootstrap MCC/MCS to generate and serve master configs. When the in-cluster MCC comes up, it then tries to regen the same MachineConfig via the in-cluster MCs at the time.
This often causes a drift and for the install to fail. See https://github.com/openshift/machine-config-operator/issues/2114 and https://docs.google.com/document/d/1fgP6Kv1D-75e1Ot0Kg-W2qPyxWDp2_CALltlBLuseec/edit#heading=h.ny6l9ud82fxx for more context. For the most recent occurrence of this, see: https://github.com/openshift/machine-config-operator/pull/3513
Early on this helped us see differences between bootstrap and in-cluster behaviour more easily, but we do have the bootstrap machineconfig on-disk on the masters. In theory, we should just be able to use that directly and attempt to consolidate the changes.
In the case of a drift, instead of failing, we can consider doing an immediate update instead to the latest version.
In newer versions of OCP, we have changed our draining mechanism to only fail after 1 hour. This also means that the event which captures the failing drain was also moved to the failure at the 1hr mark.
Today, upgrade tests oft fail with timeouts related to drain errors (PDB or other). There exists no good way to distinguish what pods are failing and for what reason, so we cannot easily aggregate this data in CI to tackle issues related to PDBs to improve upgrade and CI pass rate.
If the MCD, upon a drain run failure, emits the failing pod and reason (PDB, timeout) as an event, it would be easier to write a test to aggregate this data.
Context in this thread: https://coreos.slack.com/archives/C01CQA76KMX/p1633635861184300
This is intended to be a place to capture general "tech debt" items so they don't get lost. I very much doubt that this will ever get completed as a feature, but that's okay, the desire is more that stores get pulled out of here and put with feature work "opportunistically" when it makes sense.
If you find a "tech debt" item, and it doesn't have an obvious home with something else (e.g. with MCO-1 if it's metrics and alerting) then put it here, and we can start splitting these out/marrying them up with other epics when it makes sense.
In https://issues.redhat.com/browse/MCO-1469, we are migrating my helper binaries into the MCO repository. I had to make changes to several of my helpers in the original repository to address bugs and other issues in order to unblock https://github.com/openshift/release/pull/58241. Because of the changes I requested during the PR review to make the integration easier, it may be a little tricky to incorporate all of my changes into the MCO repository, but it is still doable.
Done When:
This epic has been repurposed for handling bugs and issues related to DataImage api ( see comments by Zane and slack discussion below ). Some issues have already been added, will add more issues to improve the stability and reliability of this feature.
Reference links :
Issue opened for IBIO : https://issues.redhat.com/browse/OCPBUGS-43330
Slack discussion threads :
https://redhat-internal.slack.com/archives/CFP6ST0A3/p1729081044547689?thread_ts=1728928990.795199&cid=CFP6ST0A3
https://redhat-internal.slack.com/archives/C0523LQCQG1/p1732110124833909?thread_ts=1731660639.803949&cid=C0523LQCQG1
Description of problem:
After deleting a BaremetalHost which has a related DataImage, the DataImage is still present. I'd expect that together with the bmh deletion the dataimage gets deleted as well.
Version-Release number of selected component (if applicable):
4.17.0-rc.0
How reproducible:
100%
Steps to Reproduce:
1. Create BaremetalHost object as part of the installation process using Image Based Install operator 2. Image Based Install operator will create a dataimage as part of the install process 3. Delete the BaremetalHost object 4. Check the DataImage assigned to the BareMetalHost
Actual results:
While the BaremetalHost was deleted the DataImage is still present: oc -n kni-qe-1 get bmh No resources found in kni-qe-1 namespace. oc -n kni-qe-1 get dataimage -o yaml apiVersion: v1 items: - apiVersion: metal3.io/v1alpha1 kind: DataImage metadata: creationTimestamp: "2024-09-24T11:58:10Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2024-09-24T14:06:15Z" finalizers: - dataimage.metal3.io generation: 2 name: sno.kni-qe-1.lab.eng.rdu2.redhat.com namespace: kni-qe-1 ownerReferences: - apiVersion: metal3.io/v1alpha1 blockOwnerDeletion: true controller: true kind: BareMetalHost name: sno.kni-qe-1.lab.eng.rdu2.redhat.com uid: 0a8bb033-5483-4fe8-8e44-06bf43ae395f resourceVersion: "156761793" uid: 2358cae9-b660-40e6-9095-7daabb4d9e48 spec: url: https://image-based-install-config.multicluster-engine.svc:8000/images/kni-qe-1/ec274bfe-a295-4cd4-8847-4fe4d232b255.iso status: attachedImage: url: https://image-based-install-config.multicluster-engine.svc:8000/images/kni-qe-1/ec274bfe-a295-4cd4-8847-4fe4d232b255.iso error: count: 0 message: "" lastReconciled: "2024-09-24T12:03:28Z" kind: List metadata: resourceVersion: ""
Expected results:
The DataImage gets deleted when the BaremetalHost owner gets deleted.
Additional info:
This is impacting automated test pipelines which use ImageBasedInstall operator as the cleanup stage gets stuck waiting for the namespace deletion which still holds the DataImage. Also deleting the DataImage gets stuck and it can only be deleted by removing the finalizer. oc get namespace kni-qe-1 -o yaml apiVersion: v1 kind: Namespace metadata: annotations: openshift.io/sa.scc.mcs: s0:c33,c2 openshift.io/sa.scc.supplemental-groups: 1001060000/10000 openshift.io/sa.scc.uid-range: 1001060000/10000 creationTimestamp: "2024-09-24T11:40:03Z" deletionTimestamp: "2024-09-24T14:06:14Z" labels: app.kubernetes.io/instance: clusters cluster.open-cluster-management.io/managedCluster: kni-qe-1 kubernetes.io/metadata.name: kni-qe-1 name: kni-qe-1-namespace open-cluster-management.io/cluster-name: kni-qe-1 pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/audit-version: v1.24 pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: v1.24 pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/warn-version: v1.24 name: kni-qe-1 resourceVersion: "156764765" uid: ee984850-665a-4f5e-8f17-0c44b57eb925 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2024-09-24T14:06:23Z" message: All resources successfully discovered reason: ResourcesDiscovered status: "False" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: 'Some resources are remaining: dataimages.metal3.io has 1 resource instances' reason: SomeResourcesRemain status: "True" type: NamespaceContentRemaining - lastTransitionTime: "2024-09-24T14:06:23Z" message: 'Some content in the namespace has finalizers remaining: dataimage.metal3.io in 1 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining phase: Terminating
Description of problem:
After deleting a BaremetalHost which has a related DataImage, the DataImage is still present. I'd expect that together with the bmh deletion the dataimage gets deleted as well.
Version-Release number of selected component (if applicable):
4.17.0-rc.0
How reproducible:
100%
Steps to Reproduce:
1. Create BaremetalHost object as part of the installation process using Image Based Install operator 2. Image Based Install operator will create a dataimage as part of the install process 3. Delete the BaremetalHost object 4. Check the DataImage assigned to the BareMetalHost
Actual results:
While the BaremetalHost was deleted the DataImage is still present: oc -n kni-qe-1 get bmh No resources found in kni-qe-1 namespace. oc -n kni-qe-1 get dataimage -o yaml apiVersion: v1 items: - apiVersion: metal3.io/v1alpha1 kind: DataImage metadata: creationTimestamp: "2024-09-24T11:58:10Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2024-09-24T14:06:15Z" finalizers: - dataimage.metal3.io generation: 2 name: sno.kni-qe-1.lab.eng.rdu2.redhat.com namespace: kni-qe-1 ownerReferences: - apiVersion: metal3.io/v1alpha1 blockOwnerDeletion: true controller: true kind: BareMetalHost name: sno.kni-qe-1.lab.eng.rdu2.redhat.com uid: 0a8bb033-5483-4fe8-8e44-06bf43ae395f resourceVersion: "156761793" uid: 2358cae9-b660-40e6-9095-7daabb4d9e48 spec: url: https://image-based-install-config.multicluster-engine.svc:8000/images/kni-qe-1/ec274bfe-a295-4cd4-8847-4fe4d232b255.iso status: attachedImage: url: https://image-based-install-config.multicluster-engine.svc:8000/images/kni-qe-1/ec274bfe-a295-4cd4-8847-4fe4d232b255.iso error: count: 0 message: "" lastReconciled: "2024-09-24T12:03:28Z" kind: List metadata: resourceVersion: ""
Expected results:
The DataImage gets deleted when the BaremetalHost owner gets deleted.
Additional info:
This is impacting automated test pipelines which use ImageBasedInstall operator as the cleanup stage gets stuck waiting for the namespace deletion which still holds the DataImage. Also deleting the DataImage gets stuck and it can only be deleted by removing the finalizer. oc get namespace kni-qe-1 -o yaml apiVersion: v1 kind: Namespace metadata: annotations: openshift.io/sa.scc.mcs: s0:c33,c2 openshift.io/sa.scc.supplemental-groups: 1001060000/10000 openshift.io/sa.scc.uid-range: 1001060000/10000 creationTimestamp: "2024-09-24T11:40:03Z" deletionTimestamp: "2024-09-24T14:06:14Z" labels: app.kubernetes.io/instance: clusters cluster.open-cluster-management.io/managedCluster: kni-qe-1 kubernetes.io/metadata.name: kni-qe-1 name: kni-qe-1-namespace open-cluster-management.io/cluster-name: kni-qe-1 pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/audit-version: v1.24 pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: v1.24 pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/warn-version: v1.24 name: kni-qe-1 resourceVersion: "156764765" uid: ee984850-665a-4f5e-8f17-0c44b57eb925 spec: finalizers: - kubernetes status: conditions: - lastTransitionTime: "2024-09-24T14:06:23Z" message: All resources successfully discovered reason: ResourcesDiscovered status: "False" type: NamespaceDeletionDiscoveryFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: All legacy kube types successfully parsed reason: ParsedGroupVersions status: "False" type: NamespaceDeletionGroupVersionParsingFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: All content successfully deleted, may be waiting on finalization reason: ContentDeleted status: "False" type: NamespaceDeletionContentFailure - lastTransitionTime: "2024-09-24T14:06:23Z" message: 'Some resources are remaining: dataimages.metal3.io has 1 resource instances' reason: SomeResourcesRemain status: "True" type: NamespaceContentRemaining - lastTransitionTime: "2024-09-24T14:06:23Z" message: 'Some content in the namespace has finalizers remaining: dataimage.metal3.io in 1 resource instances' reason: SomeFinalizersRemain status: "True" type: NamespaceFinalizersRemaining phase: Terminating
Tracking all things Konflux related for the Metal Platform Team
Full enablement should happen during OCP 4.19 development cycle
Description of problem:
The host that gets used in production builds to download the iso will change soon. It would be good to allow this host to be set through configuration from the release team / ocp-build-data
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
tracking here all the work that needs to be done to configure the ironic containers (ironic-image and ironic-agent-image) to be ready for OCP 4.20
this includes also CI configuration, tools and documentation updates
all the configuration bits need to happen at least one sprint BEFORE 4.20 branching (current target April 18 2025)
docs tasks can be completed after the configuration tasks
the CI tasks need to be completed RIGHT AFTER 4.20 branching happens
tag creation is now automated during OCP tags creation
builder creation has been automated
this needs to be done in ironic-image and ironic-agent-image
we may need to bump dependencies
we've been installing the sushy package because of dependencies in sushy-oem-idrac and proliantutils packages, and then installing sushy from source de facto updating the code in the image during build time
this is not a great practice, and we also want to remove dependency from the pkg entirely
both sushy-oem-idrac and proliantutils don't get too many updates so it should be fine to install them from source, and we can always rollback if things go sideways
CAPI Agent Control Plane Provider and CAPI Bootstrap Provider will provide an easy way to install clusters through CAPI.
Those providers will not be generic OpenShift providers, as they are geared towards Bare Metal. Those providers will leverage Assisted Installer ZTP flow, and will benefit BM users by avoiding to provision a bootstrap node (as opposed to regular OpenShift install where the bootstrap node is required, but it will comply better to CAPI interface)
milestones:
Yes
With the standard image (non liveISO) flow, we decoupled from BMHs.
The challenge now is that we cannot set the label from the controller directly, as the data will be available in-host only.
We should now:
Flows goes like this: * M3Machine will set providerID on itselffrom bmh and from the label on the node - it expects them to be the same else it wont' succeed. This is how it shares data with the host (eventually)
Please describe what this feature is going to do.
To allow for installs on platforms where ISOs are not easily used/supported the assisted installer should have an install flow that requires only a disk image and a user-data service.
Please describe what conditions must be met in order to mark this feature as "done".
Installation using a disk image must work as expected.
If the answer is "yes", please make sure to check the corresponding option.
This allows assisted installer to more closely map to the assumptions for installing using CAPI. It also allows us to more easily support installs on platforms where booting a live-iso isn't an option or is very difficult (OCI and AWS, for example)
I don't know. I assume some data about installs using this method vs ISO methods might exist, but I don't have it.
N/A this isn't so much a feature of the installer as a way of installing.
Today users can customize installer args when we run coreos-installer
The new install process should parse those args and preserve the functionality in a way that is seamless for the user.
The most relevant ones from https://coreos.github.io/coreos-installer/cmd/install/ seem to be:
We shouldn't have to worry about the partitioning flags since we're not writing a new disk and the options for fetching ignitions shouldn't apply since we're always getting ignition from the service and writing it locally.
When discovery is running on the real device rather in a live ISO assisted-installer needs to know how to install using ostree to the local disk rather than using coreos-installer.
Create that new install flow.
The agent or installer may be able to detect if we're running on the install target disk or a live ISO.
If they detect we're not running on a live ISO they should switch automatically to the new install flow.
If it's not possible to detect this we'll need some kind of API for the user to choose a flow.
Allow users to do a basic OpenShift AI intallation with one click in the "operators" page of the cluster creation wizard, similar to how the ODF or MCE operators can be installed.
This feature will be done when users can click on the "OpenShift AI" check box on the operators page of the cluster creation wizard, and end having an installation that can be used for basic tasks.
Yes.
Feature origin (who asked for this feature?)
Currently installing the OpenShift AI operator requires at least one supported GPU. For NVIDIA GPUs it is also necessary to disable secure boot, because otherwise it isn't possible to load the NVIDIA drivers. This ticket is about adding that validation, so that the problem will be detected and reported to the user before installing the cluster.
Dan Clark mentions that:
In both OCP 3 and 4, MTU issues present in very bizarre behavior.
In OCP 4, the 2 SSL warning pages for the console would load and then just a white screen of doom forever
In the latest case, he was using ABI with the interface MTUs set to 9000 using the NMState interface config, but the network switches were not configured for jumbo frames. The result was that none of the control plane nodes ever rebooted. According to Dan 'generally it's SSL frames like OAUTH that have "no fragment" marked' and thus get lost.
External configuration issues that "present in very bizarre behaviour" are ideal candidates for assisted installer validations. We already verify connectivity between nodes using ping. We could verify the expected MTU by additionally doing (the equivalent of):
mtu="$(ip -j link show "${ifname}" | jq '.[0].mtu')" ping "${ip_addr}" -c 3 -M do -s $((mtu - 28)) -I "${ifname}" ping6 "${ip6_addr}%${ifname}" -c 3 -M do -s $((mtu - 48)) -I "${ifname}"
i.e. send a maximum-sized packet with No Fragment set and see if we get a response. This will be sufficient to validate connectivity even in cases where ICMP "Fragmentation Needed" packets are dropped (and therefore Path-Based MTU Discovery will not work).
Probably we should limit this to interfaces where the MTU is set explicitly? Rather than interrogate the NMState data, perhaps verifying all interfaces where the MTU is not 1500 would be the simplest approach.
Additionally, adjust the headers to account only for IP headers.
Deprecate high_availability_mode as it was replaced by control_plane_count
high_availability_mode is no longer used in our code
Yes
Centralize view of issues resulting from the ACM 2.13 feature freeze. See more details here - https://docs.google.com/document/d/1WXRr_-HZkVrwbXBFo4gGhHUDhSO4-VgOPHKod1RMKng
We need to close https://issues.redhat.com/browse/ACM-17845 as well once done
Now downstream images are built with Kofnflux, therefore we need to change the references in the master branches in all components to the new ACM version.
Currently, the monitoring stack is configured using a configmap. In OpenShift though the best practice is to configure operators using custom resources.
To start the effort we should create a feature gate behind which we can start implementing a CRD config approach. This allows us to iterate in smaller increments without having to support full feature parity with the config map from the start. We can start small and add features as they evolve.
One proposal for a minimal DoD was:
Feature parity should be planned in one or more separate epics.
This story covers the implementation of our initial CRD in CMO. When the feature gate is enabled, CMO watches a singleton CR (name tbd) and acts on changes. The inital feature could be a boolean flag (defaults to true) that tells CMO to merge the configmap settings. If a user sets this flag to false, the config map is ignored and default settings are applied.
This epic is to track stories that are not completed in MON-3865
For the issue https://issues.redhat.com//browse/OCPBUGS-32510 we had identified that we need to have separate metrics client cert for metrics server but for that we need to add approver for metrics-server .
alertmanager v1 no longer supported
The `PrometheusPossibleNarrowSelectors` alert was added to help identify label selectors misuses after the Prometheus v3 update (More details below)
Setting Prometheus/Thanos log level to "debug" (see
https://docs.openshift.com/container-platform/latest/observability/monitoring/configuring-the-monitoring-stack.html#setting-log-levels-for-monitoring-components_configuring-the-monitoring-stack)
should provide insights into the affected queries and relabeling configs.
See attached PR for how to fix.
If assistance is needed, please leave a comment.
—
With Prometheus v3, the classic histogram's "le" and summary's "quantile" labels values will be floats.
All queries (in Alerts, Recording rules, dashboards, or interactive ones) with selectors that assume "le"/"quantile" values to be integers only should be adjusted.
Same applies to Relabel Configs.
Queries:
foo_bucket{le="1"} may need to be turned into foo_bucket{le=~"1(.0)?"} foo_bucket{le=~"1|3"} may need to be turned into foo_bucket{le=~"1|3(.0)?"}
(same applies to the "quantile" label)
Relabel configs:
- action: foo regex: foo_bucket;(1|3|5|15.5) sourceLabels: - __name__ - le may need to be adjusted - action: foo regex: foo_bucket;(1|3|5|15.5)(\.0)? sourceLabels: - __name__ - le
(same applies to the "quantile" label)
Also, from upstream Prometheus:
Aggregation by the `le` and `quantile` labels for vectors that contain the old and new formatting will lead to unexpected results, and range vectors that span the transition between the different formatting will contain additional series. The most common use case for both is the quantile calculation via `histogram_quantile`, e.g. `histogram_quantile(0.95, sum by (le) (rate(histogram_bucket[10m])))`. The `histogram_quantile` function already tries to mitigate the effects to some extent, but there will be inaccuracies, in particular for shorter ranges that cover only a few samples.
A warning about this should suffice, as adjusting the queries would be difficult, if not impossible. Additionally, it might complicate things further.
See attached PRs for examples.
A downstream check to help surface such misconfigurations was added. An alert will fire for configs that aren't enabled by default and that may need to be adjusted.
For more details, see https://docs.google.com/document/d/11c0Pr2-Zn3u3cjn4qio8gxFnu9dp0p9bO7gM45YKcNo/edit?tab=t.0#bookmark=id.f5p0o1s8vyjf
thanos needs to be upgraded to support prometheus3
all origin tests were failing
This hsould give us an idea what, if any, changes we need in our stack to support and ship Prometheus 3.
Additionally it would be great to get an idea what improvements we could ship based on Prometheus 3.
See if we want to add the checks in https://github.com/openshift/prometheus/pull/227 (last commits) to thanos ruler and querier.
Queries with broken selectors may be sent to thanos querier without necessarily reaching the checks added into Prometheus (no alert on prometheus would be triggered)
Thanos ruler may be evaluating rules with broken selectors.
The check will consists of adding the same code into the Prometheus parser that Thanos imports.
Add fallbackScrapeProtocol to ScrapeClass Prometheus-Operator
And have it set downstream.
(no need to wait for a prom-operator release, we can cherry-pick it one merged, that would be considered an extra test for the change)
or whatever is the name of that alert will be.
Warn about possible misuses of selectors on the "le" or/an "quantile" labels values.
See https://issues.redhat.com/browse/MON-4129 for what the runbook could include
The history of this epic starts with this PR which triggered a lengthy conversation around the workings of the image API with respect to importing imagestreams images as single vs manifestlisted. The imagestreams today by default have the `importMode` flag set to `Legacy` to avoid breaking behavior of existing clusters in the field. This makes sense for single arch clusters deployed with a single arch payload, but when users migrate to use the multi payload, more often than not, their intent is to add nodes of other architecture types. When this happens - it gives rise to problems when using imagestreams with the default behavior of importing a single manifest image. The oc commands do have a new flag to toggle the importMode, but this breaks functionality of existing users who just want to create an imagestream and use it with existing commands.
There was a discussion with David Eads and other staff engineers and it was decided that the approach to be taken is to default imagestreams' importMode to `preserveOriginal` if the cluster is installed with/ upgraded to a multi payload. So a few things need to happen to achieve this:
Some open questions:
Add featuregated automation tests to exercise this functionality :
Once these tests run for a release, we will have enough confidence that this signal proves we can make the featuregate default.
Bump vendored Kubernetes packages (k8s.io/api, k8s.io/apimachinery, k8s.io/client-go, etc.) to v0.32.0 or newer version.
Keep vendored packages up to date.
1. Other vendored dependencies (such as openshift/api and controller-runtime) may also need to be updated to Kubernetes 1.32.
1. NE-1875.
None.
The openshift/cluster-ingress-operator repository vendors k8s.io/* v0.31.1. OpenShift 4.19 is based on Kubernetes 1.32.
4.19.
Always.
Check https://github.com/openshift/cluster-ingress-/blob/release-4.19/go.mod.
The k8s.io/* packages are at v0.31.1.
The k8s.io/* packages are at v0.32.0 or newer.
This epic is a placeholder for code refactoring, code cleanups, and other maintenance work in cluster-ingress-operator, keepalived, and other components that the Network Ingress & DNS team maintains that doesn't already have an associated Jira issue.
This work has no impact on docs, QE, or PX. This epic strictly tracks internal code changes that is not customer facing and that otherwise has no Jira issues to track it.
See the child issues for specific maintenance work under this epic.
Our code needs refactoring to remove duplicated logic, as well as cleanups to improve readability. These code changes will make it easier to find and fix defects and extend the code with new features.
All child issues under this epic are closed.
None. If any major dependencies are discovered in the course of the work under this epic, we will split that work into a separate epic and plan it appropriately.
NE-1182 is an example of previous refactoring and code cleanups.
None.
Move the FakeCache, FakeController, and FakeClientRecorder test helpers, which are currently duplicated in cluster-ingress-operator's gateway-service-dns and gatewayapi controller test code, into a dedicated package, so that the controllers' respective tests can import it instead of each having copies of the same helpers.
As a developer I want rebasebot to periodically update cluster api repositories.
Created this card to track merging my PR that was hit with few blockers. We ran the required rebases as rehearsal jobs bud did not merge the PR.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
This story is to track the CI jobs that need to be added to test `ClusterImagePolicy` and `ImagePolicy` API.
coverd configuration:
testing the verification coordinates configs from image.config.openshift.io/cluster: support scopes from allowedRegistries (https://access.redhat.com/solutions/6958257)
covered root of trust type: Public key,(with Rekor)
The tests should cover all supported configuration. track future tests OCPNODE-2951
Different from case failure, this task will focus on case enhancement. Trace the automation work not coved by QE e2e tasks, such as:
This epic covers the scope of automation-related stories in ODC
Automation enhancements for ODC
Description of problem:
Update CI config to reflect video in CI results
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Networking Definition of Planned
Epic Template descriptions and documentation
Track work that needs to happen in 4.18 but was not part of the original planning.
Additional information on each of the above items can be found here: Networking Definition of Planned
...
1.
...
1. …
1. …
This is a followup to https://issues.redhat.com//browse/OCPBUGS-44694 to make the monitor resilient in all configurations, including things like 5 node control planes. Instead of relying on a longer fall time, we can just let HAProxy report its own ability to reach any backend, which means under ordinary circumstances this check will never fail. There will no longer be any issue with pathologically bad call chains where we happen to hit backends that are down but haven't been detected yet.
We will use this to address tech debt in OLM in the 4.10 timeframe.
Items to prioritize are:
CI e2e flakes
The operator framework portfolio consists of a number of projects that
have an upstream project. The verify-commits makefile target had been
introduced in each of the downstream projects to assist with the
downstreaming efforts. The commit checking logic was eventually moved
into a container and was rolled out as the verify-commits-shared test,
which ran alongside the verify-commits test to ensure that it worked
as expected.
We are now confident that the verify-commits-shared test is running as
expected, and can replace the logic used in the verify-commits test with
that of the verify-commits-shared test, which will allow us to remove
the verify-commits Makefile targets from each of the repos, simplifying
the code base and reducing duplication of code.
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
Modify cluster-olm-operator to allow it to respect feature gates
AC:
See RFC for additional background and details
As documented in OCPCLOUD-1578, OpenShift would like to migrate from Machine API to Cluster API and eventually remove Machine API. This effort is going to require work from all affected platforms including OpenStack. This epic tracks the implementation of the OpenStack part of the mapi2capi and capi2mapi translation layer being added to cluster-capi-operator, based on the scoping done in OSASINFRA-3440.
Note that it is important that we implement both MAPI to CAPI and CAPI to MAPI for all platforms including OpenStack. This ensures we will always have a mirror copy on both sides, which is particularly useful for third-party components that don't have a way to interact with CAPI yet. In these situations, users can create CAPI resources while these components can (until updated) continue to fetch the MAPI mirror and work out the state of the cluster that way.
Repeating from OCPCLOUD-1578:
From an OpenStack perspective, we simply need to ensure we follow suit with other platforms.
TBD.
TBD.
We need to add support for fake MAPI and CAPI cluster/machineset/machine builders to the openshift/cluster-api-actuator-pkg package. These already exist for other providers, so this is likely to be an exercise in copy-pasting-tweaking.
We don't use a feature gate for determining whether to deploy CAPO or not. Rather, we check for a whole feature set: specifically, the TechPreviewNoUpgrade. However, the MachineAPIMigration controller from cluster-capi-operator does use a feature gate, MachineAPIMigration, which is not included in the TechPreviewNoUpgrade feature set and must be enabled manually using the CustomNoUpgrade feature set. This leaves us in a bit of a catch-22. We can resolve this by ensuring CAPO is also deployed when the CustomNoUpgrade feature set is used.
See https://github.com/openshift/cluster-capi-operator/pull/238 for the changes to all the other controllers.
This EPIC regroups the tasks that need to be finished so we can deliver Hosted Control Plane on OpenStack as TechPreview.
Some tasks were initially in this EPIC and were de-prioritized to be done later once we have customer feedback. What remain in the current list are things we think can be achieved within the 4.19 cycle.
This EPIC will be used by QE to test the quality of the product and the outcome will have a direct impact on whether this can be TechPreview or now. The HCP team will only accept it as TechPreview if we have QE coverage.
The initial scenario that QE agreed to start with is the following:
CAPO doesn't currently support Glance image creation.
This will have to be via upload for now rather than web-download because:
the RHCOS image is gzipped
glance can automatically decompress an image if the image_decompression plugin is enabled, but
we have no way to determine if the image_decompression plugin is enabled, and
glance will report success but create a corrupt image if it is not
This task focuses on ensuring that all OpenStack resources automatically created by Hypershift for Hosted Control Planes are tagged with a unique identifier, such as the HostedCluster ID. These resources include, but are not limited to, servers, ports, and security groups. Proper tagging will enable administrators to clearly identify and manage resources associated with specific OpenShift clusters.
Acceptance Criteria:
Once we rebase downstream CAPO to 0.12, ORC will need to be taken care so Hypershift will deploy ORC since CAPO won't.
Valuable slack thread giving details on what and how to do: https://redhat-external.slack.com/archives/C01C8502FMM/p1738877046453869
We broke the 4.18 CI jobs because now ORC is being deployed but it's not in the release payload. We should only install ORC on 4.19 and beyond.
When deploying a HostedCluster, etcd will be using the default CSI StorageClass but we can override it with --etcd-storage-class.
Customers should use a local storage CSI Storage class in production to avoid performance issues in their clusters.
Spike / Document: https://docs.google.com/document/d/1qdYHb7YAQJKSDLOziqLG8jaBQ80qEMchsozgVnI7qyc
This task also includes a doc change in Hypershift repo but depends on OSASINFRA-3681.
Epic Goal*
What is our purpose in implementing this? What new capability will be available to customers?
Why is this important? (mandatory)
What are the benefits to the customer or Red Hat? Does it improve security, performance, supportability, etc? Why is work a priority?
Scenarios (mandatory)
Provide details for user scenarios including actions to be performed, platform specifications, and user personas.
Dependencies (internal and external) (mandatory)
What items must be delivered by other teams/groups to enable delivery of this epic.
Contributing Teams(and contacts) (mandatory)
Our expectation is that teams would modify the list below to fit the epic. Some epics may not need all the default groups but what is included here should accurately reflect who will be involved in delivering the epic.
Acceptance Criteria (optional)
Provide some (testable) examples of how we will know if we have achieved the epic goal.
Drawbacks or Risk (optional)
Reasons we should consider NOT doing this such as: limited audience for the feature, feature will be superseded by other work that is planned, resulting feature will introduce substantial administrative complexity or user confusion, etc.
Done - Checklist (mandatory)
The following points apply to all epics and are what the OpenShift team believes are the minimum set of criteria that epics should meet for us to consider them potentially shippable. We request that epic owners modify this list to reflect the work to be completed in order to produce something that is potentially shippable.
Feature gates must demonstrate completeness and reliability.
As per https://github.com/openshift/api?tab=readme-ov-file#defining-featuregate-e2e-tests:
If your FeatureGate lacks automated testing, there is an exception process that allows QE to sign off on the promotion by commenting on the PR.
The introduced functionality is not that complex. The only newly introduced ability is to modify the CVO log level using the API. However, we should still introduce an e2e test or tests to demonstrate that the CVO correctly reconciles the new configuration API.
The tests may be:
Definition of Done:
We need to send an enhancement proposal that would contain the design changes we suggest in openshift/api/config/v1/types_cluster_version.go to allow changing the log level of the CVO using an API configuration before implementing such changes in the API.
Definition of Done:
The ClusterVersionOperator API has been introduced in the DevPreviewNoUpgrade feature set. Enable the CVO in standalone OpenShift to change its log level based on the new API.
Definition of Done:
The CVO is configurable using a manifest file. We now can proceed to HyperShift. Introduce the HyperShift API changes described in the enhancement.
Definition of Done:
As the monitoring plugin migrated away from nginx into a lightweight Go backend, changes in CMO were required to remove the ConfigMap that stored the nginx configuration.
In order to gracefully handle CMO upgrades the ConfigMap needs to be deleted if exists in 4.18. For 4.19 this log can be removed as the ConfigMap is not there anymore.
Previous PR: https://github.com/openshift/cluster-monitoring-operator/pull/2412
This epic will serve as a parent issue for all the bugs RFEs for the incidents section in the monitoring plugin
OCP conformance has a test that monitors how many times a container exits, and it appears managed-upgrade-operator restarts up to 10 times in a run
{ 1 containers with multiple restarts namespace/openshift-managed-upgrade-operator node/ip-10-0-24-14.us-west-2.compute.internal pod/managed-upgrade-operator-775587df-sbgds uid/e7ac3064-a9f6-4864-8b5f-58918b30c54f container/managed-upgrade-operator restarted 10 times at: non-zero exit at 2024-10-25 15:42:48.147733774 +0000 UTC m=+151.061055807: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 15:48:00.093547308 +0000 UTC m=+463.006869341: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 15:53:08.363301669 +0000 UTC m=+771.276623702: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 15:58:15.514874824 +0000 UTC m=+1078.428196857: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:03:18.539399543 +0000 UTC m=+1381.452721576: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:08:25.585326381 +0000 UTC m=+1688.498648414: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:13:35.622986532 +0000 UTC m=+1998.536308555: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:18:39.307340212 +0000 UTC m=+2302.220662235: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:23:49.56614045 +0000 UTC m=+2612.479462473: cause/Error code/1 reason/ContainerExit non-zero exit at 2024-10-25 16:29:00.143738226 +0000 UTC m=+2923.057060299: cause/Error code/1 reason/ContainerExit}
Goal:
Key result:
Acceptance criteria:
Description of problem:
OpenShift 4.19 clusters cannot be created on PowerVS CI. The only cluster operator active is cloud-controller-manager 4.19.0-0.nightly-multi-2025-01-24-235841 True False False 14m
Version-Release number of selected component (if applicable):
rhcos-9-6-20250121-0-ppc64le-powervs.ova.gz
How reproducible:
Always
Steps to Reproduce:
1. Use the IPI installer to create an OpenShift cluster.
Actual results:
Expected results:
Additional info:
[core@localhost ~]$ sudo systemctl status afterburn-hostname.service --no-pager -l × afterburn-hostname.service - Afterburn Hostname Loaded: loaded (/etc/systemd/system/afterburn-hostname.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Tue 2025-02-11 13:33:59 UTC; 9min ago Process: 1920 ExecStart=/usr/bin/afterburn --provider powervs --hostname=/etc/hostname (code=exited, status=1/FAILURE) Main PID: 1920 (code=exited, status=1/FAILURE) CPU: 5ms Feb 11 13:33:59 localhost.localdomain systemd[1]: Starting Afterburn Hostname... Feb 11 13:33:59 localhost.localdomain afterburn[1920]: Error: failed to run Feb 11 13:33:59 localhost.localdomain afterburn[1920]: Caused by: Feb 11 13:33:59 localhost.localdomain afterburn[1920]: 0: fetching metadata from provider Feb 11 13:33:59 localhost.localdomain afterburn[1920]: 1: failed to create temporary directory Feb 11 13:33:59 localhost.localdomain afterburn[1920]: 2: Permission denied (os error 13) at path "/tmp/afterburn-n4dSuU" Feb 11 13:33:59 localhost.localdomain systemd[1]: afterburn-hostname.service: Main process exited, code=exited, status=1/FAILURE Feb 11 13:33:59 localhost.localdomain systemd[1]: afterburn-hostname.service: Failed with result 'exit-code'. Feb 11 13:33:59 localhost.localdomain systemd[1]: Failed to start Afterburn Hostname. [core@localhost ~]$ sudo grep afterburn /var/log/audit/audit.log type=AVC msg=audit(1739280490.147:59): avc: denied { create } for pid=3394 comm="afterburn" name="afterburn-tZsJKP" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir permissive=0 type=SYSCALL msg=audit(1739280490.147:59): arch=c0000015 syscall=39 success=no exit=-13 a0=7fffee6b8120 a1=1ff a2=16 a3=16 items=0 ppid=1 pid=3394 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)ARCH=ppc64le SYSCALL=mkdir AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" type=SERVICE_START msg=audit(1739280490.147:60): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=afterburn-hostname comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset" type=AVC msg=audit(1739280839.555:63): avc: denied { create } for pid=1920 comm="afterburn" name="afterburn-n4dSuU" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir permissive=0 type=SYSCALL msg=audit(1739280839.555:63): arch=c0000015 syscall=39 success=no exit=-13 a0=7ffff52ed5a0 a1=1ff a2=16 a3=16 items=0 ppid=1 pid=1920 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)ARCH=ppc64le SYSCALL=mkdir AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" type=SERVICE_START msg=audit(1739280839.565:64): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=afterburn-hostname comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset"
OCP/Telco Definition of Done
Epic Template descriptions and documentation.
<--- Cut-n-Paste the entire contents of this description into your new Epic --->
An elevator pitch (value statement) that describes the Feature in a clear, concise way. Complete during New status.
We have decided to remove IPI and UPI support for Alibaba Cloud, which until recently has been in Tech Preview due to the following reasons:
(1) Low customer interest of using Openshift on Alibaba Cloud
(2) Removal of Terraform usage
(3) MAPI to CAPI migration
(4) CAPI adoption for installation (Day 1) in OpenShift 4.16 and Day 2 (post-OpenShift 4.16)
Impacted areas based on CI:
alibaba-cloud-csi-driver/openshift-alibaba-cloud-csi-driver-release-4.16.yaml
alibaba-disk-csi-driver-operator/openshift-alibaba-disk-csi-driver-operator-release-4.16.yaml
cloud-provider-alibaba-cloud/openshift-cloud-provider-alibaba-cloud-release-4.16.yaml
cluster-api-provider-alibaba/openshift-cluster-api-provider-alibaba-release-4.16.yaml
cluster-cloud-controller-manager-operator/openshift-cluster-cloud-controller-manager-operator-release-4.16.yaml
machine-config-operator/openshift-machine-config-operator-release-4.16.yaml
Acceptance Criteria
Traditionally we did these updates as bugfixes, because we did them after the feature freeze (FF).
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update OCP release number in OLM metadata manifests of:
OLM metadata of the operators are typically in /config/manifest directory of each operator. Example of such a bump: https://github.com/openshift/aws-efs-csi-driver-operator/pull/56
We should do it early in the release, so QE can identify new operator builds easily and they are not mixed with the old release.
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update all OCP and kubernetes libraries in storage operators to the appropriate version for OCP release.
Please wait for openshift/api, openshift/library-go, and openshift/client-go are updated to the newest Kubernetes release! There may be non-trivial changes in these libraries.
This includes (but is not limited to):
Operators:
(please cross-check with *-operator + vsphere-problem-detector in our tracking sheet)
Transferred ownership to AOPE team:
EOL, do not upgrade:
The following operators were migrated to csi-operator, do not update these obsolete repos:
tools/library-bump.py and tools/bump-all may be useful. For 4.16, this was enough:
mkdir 4.16-bump cd 4.16-bump ../library-bump.py --debug --web <file with repo list> STOR-1574 --run "$PWD/../bump-all github.com/google/cel-go@v0.17.7" --commit-message "Bump all deps for 4.16"
4.17 perhaps needs an older prometheus:
../library-bump.py --debug --web <file with repo list> STOR-XXX --run "$PWD/../bump-all github.com/google/cel-go@v0.17.8 github.com/prometheus/common@v0.44.0 github.com/prometheus/client_golang@v1.16.0 github.com/prometheus/client_model@v0.4.0 github.com/prometheus/procfs@v0.10.1" --commit-message "Bump all deps for 4.17"
4.18 special:
Add "spec.unhealthyEvictionPolicy: AlwaysAllow" to all PodDisruptionBudget objects of all our operators + operands. See WRKLDS-1490 for details
There has been change in library-go function called `WithReplicasHook`. See https://github.com/openshift/library-go/pull/1796.
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
This epic is part of the 4.18 initiatives we discussed, it includes:
The user is running openshift-tests from git against a much older cluster, the tests image won't have k8s-tests-ext, but it doesn't error until we try to run it.
Looks like there's no check the image extraction was successful.
INFO[0002] Run image extract for release image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143e5aa28ec8a2345e934373b7bfe234f50a825691c67286aecc031424c38c43" and src "/usr/bin/k8s-tests-ext.gz" INFO[0013] Completed image extract for release image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:143e5aa28ec8a2345e934373b7bfe234f50a825691c67286aecc031424c38c43" in 11.020672636s Suite run returned error: encountered errors while extracting binaries: failed to decompress external binary "/usr/bin/k8s-tests-ext.gz": failed to open gzip file: open /home/shmoran/.cache/openshift-tests/quay_io_openshift-release-dev_ocp-release_sha256_bada2d7626c8652e0fb68d3237195cb37f425e960347fbdd747beb17f671cf13_0cfab9de1754/k8s-tests-ext.gz: no such file or directory error running options: encountered errors while extracting binaries: failed to decompress external binary "/usr/bin/k8s-tests-ext.gz": failed to open gzip file: open /home/shmoran/.cache/openshift-tests/quay_io_openshift-release-dev_ocp-release_sha256_bada2d7626c8652e0fb68d3237195cb37f425e960347fbdd747beb17f671cf13_0cfab9de1754/k8s-tests-ext.gz: no such file or directoryerror: encountered errors while extracting binaries: failed to decompress external binary "/usr/bin/k8s-tests-ext.gz": failed to open gzip file: open /home/shmoran/.cache/openshift-tests/quay_io_openshift-release-dev_ocp-release_sha256_bada2d7626c8652e0fb68d3237195cb37f425e960347fbdd747beb17f671cf13_0cfab9de1754/k8s-tests-ext.gz: no such file or directory
origin should support calling an external test binary implemented using openshift-tests-extension. There's an external test binary already in the hyperkube repo: https://github.com/openshift/kubernetes/tree/master/openshift-hack/cmd/k8s-tests-ext
Here's the existing external binary using the legacy interface:
https://github.com/openshift/origin/blob/master/pkg/test/ginkgo/cmd_runsuite.go#L174-L179
That can just be removed and replaced with k8s-test-ext.
MVP requires for k8s-tests:
Additional things for later:
The design should be flexible enough to allow a scheduling algorithm that takes into account available resources/isolation, but the first pass doesn't need to implement it yet.
`openshift-tests images` identifies the locations of all test images referenced by the test suite and outputs a mirror list for use with 'oc image mirror' to copy those images to a private registry.
We need to make this work with external binaries, including making the `LocationFor` helper work with OTE-enabled binaries.
If an extension wants to produce artifacts we need to tell them where to write them, i.e. EXTENSION_ARTIFACT_DIR environment variable.
The annotation code in origin and k8s-tests should be removed and replaced, or refactored to at least not inject the annotations into the test names themselves. After TRT-1840 and TRT-1852 you can skip based on labels and other criteria. Skip information should be decided at run-time, and should not require revendoring.
Add a caching layer so external binaries only need to be extracted once when running locally
This section includes Jira cards that are not linked to either an Epic or a Feature. These tickets were completed when this image was assembled
Description of problem:
Created a service for DNS server for secondary networks in Openshift-Virtualizaion, using MetalLB, but the IP is still pending, when accessing the service from the UI, it crash.
Version-Release number of selected component (if applicable):
4.17
How reproducible:
Steps to Reproduce:
1. Create an IP pool (for example 1 IP) for Metal LB and fully utilize the IP range (which other service) 2. Allocate a new IP using the oc expose command like below 3. Check the service status on the UI
Actual results:
UI crash
Expected results:
Should show the service status
Additional info:
oc expose -n openshift-cnv deployment/secondary-dns --name=dns-lb --type=LoadBalancer --port=53 --target-port=5353 --protocol='UDP'
Description of problem:
The period is placed inside the quotes of the missingKeyHandler i18n error
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
Always when there is a missingKeyHandler error
Steps to Reproduce:
1. Check browser console 2. Observe period is placed inside the quites 3.
Actual results:
It is placed inside the quotes
Expected results:
It should be placed outside the quotes
Additional info:
Description of problem:
Due the recent changes, using oc 4.17 adm node-image commands on a 4.18 ocp cluster doesn't work
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. oc adm node-image create / monitor 2. 3.
Actual results:
The commands fail
Expected results:
The commands should work as expected
Additional info:
Description of problem:
dev console, select one project that has alerts, example: openshift-monitoring, silence one Alert, example Watchdog, go to the "Silence details" page, click the Watchdog link under "Firing alerts" section, "No Alert found" shows, which should go to the alert details page, see screen recording: https://drive.google.com/file/d/1lUKLoHpmBKuzd8MmEUaUJRPgIkI1LjCj/view?usp=drive_link
the issue is happen with 4.18.0-0.nightly-2025-01-04-101226/4.19.0-0.nightly-2025-01-07-234605, no issue withe 4.17
checked the links for Watchdog link under "Firing alerts" section, there is undefined in the link, which should be namespace(openshift-monitoring) like 4.17
4.19
https://${console_url}/dev-monitoring/ns/undefined/alerts/1067612101?alertname=Watchdog&namespace=openshift-monitoring&severity=none
4.18
https://${console_url}/dev-monitoring/ns/undefined/alerts/1086044860?alertname=Watchdog&namespace=openshift-monitoring&severity=none
4.17
https://${console_url}/dev-monitoring/ns/openshift-monitoring/alerts/3861382580?namespace=openshift-monitoring&prometheus=openshift-monitoring%2Fk8s&severity=none&alertname=Watchdog
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always for 4.18+
Steps to Reproduce:
1. see the description
Actual results:
"No Alert found" shows
Expected results:
no error
Description of problem: [UDN pre-merge testing] not able to create layer3 UDN from CRD on dualstack cluster
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. On a dualstack cluster, created a UDN namespace with label
2. Attempted to create a layer3 UDN from CRD
$ cat /tmp/e2e-test-udn-networking-udn-bhmwv-n3wn8mngresource.json
{
"kind": "List",
"apiVersion": "v1",
"metadata": {},
"items": [
{
"apiVersion": "k8s.ovn.org/v1",
"kind": "UserDefinedNetwork",
"metadata":
,
"spec": {
"layer3": {
"mtu": 1400,
"role": "Primary",
"subnets": [
,
]
},
"topology": "Layer3"
}
}
]
}
3. got the following error message:
The UserDefinedNetwork "udn-network-77827-ns1" is invalid: spec.layer3.subnets[1]: Invalid value: "object": HostSubnet must < 32 for ipv4 CIDR
The UserDefinedNetwork "udn-network-77827-ns1" is invalid: spec.layer3.subnets[1]: Invalid value: "object": HostSubnet must < 32 for ipv4 CIDR
subnets[1] is ipv6 hostsubnet, but it was compared with IPv4 CIDR
Actual results: Not able to create UDN in UDN namespace on dualstack cluster
Expected results: should be able to create UDN in UDN namespace on dualstack cluster
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
oc-mirror throws error when trying to mirror release payloads using digest with graph=true for nightly, rc and ec builds and does not generate signatures for ec and rc builds.
Version-Release number of selected component (if applicable):
[fedora@knarra-fedora knarra]$ ./oc-mirror version W0210 13:19:40.701416 143622 mirror.go:102] ⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2 WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202502041931.p0.gc7144d5.assembly.stream.el9-c7144d5", GitCommit:"c7144d5d2c2b0345f163299ed04a400f2f93d340", GitTreeState:"clean", BuildDate:"2025-02-04T20:04:49Z", GoVersion:"go1.22.9 (Red Hat 1.22.9-1.module+el8.10.0+22500+aee717ef) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. use the nightly image set config as shown below and see that it throws an error 2. use the rc candidate as shown below and see that it throws an error and does not generate signature relatted files 3. use the ec candidate as shown below and see that it throws an error and does not generate signature related files.
Actual results:
2025/02/10 12:15:49 [INFO] : === Results === 2025/02/10 12:15:49 [INFO] : ✓ 191 / 191 release images mirrored successfully 2025/02/10 12:15:49 [INFO] : 📄 Generating IDMS file... 2025/02/10 12:15:49 [INFO] : internlrelease/working-dir/cluster-resources/idms-oc-mirror.yaml file created 2025/02/10 12:15:49 [INFO] : 📄 No images by tag were mirrored. Skipping ITMS generation. 2025/02/10 12:15:49 [INFO] : 📄 No catalogs mirrored. Skipping CatalogSource file generation. 2025/02/10 12:15:49 [INFO] : 📄 No catalogs mirrored. Skipping ClusterCatalog file generation. 2025/02/10 12:15:49 [INFO] : 📄 Generating UpdateService file... 2025/02/10 12:15:49 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2025/02/10 12:15:49 [ERROR] : unknown image : reference name is empty
Expected results:
No errors should be seen and for ec and rc candidate signature should be generated as well.
Additional info:
with nightly payload and graph==true: =============================== kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: graph: true release: quay.io/openshift-release-dev/ocp-release@sha256:e0907823bc8989b02bb1bd55d5f08262dd0e4846173e792c14e7684fbd476c0d with rc payload and graph==true: =========================== [fedora@knarra-fedora knarra]$ cat /tmp/internal.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: release: quay.io/openshift-release-dev/ocp-release@sha256:f0de3be10be2f5fc1a5b1c208bcfe5d3a71a70989cacbca57ebf7c5fe6e14b09 graph: true with ec payload and graph==true: ============================ [fedora@knarra-fedora knarra]$ cat /tmp/internal.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: release: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 graph: true
Description of problem:
when TechPreviewNoUpgrade feature gate is enabled, console will show a customized 'Create Project' modal to all users. In the customized modal, 'Display name' and 'Description' values user typed are not taking effect
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-16-065305
How reproducible:
Always when TechPreviewNoUpgrade feature gate is enabled
Steps to Reproduce:
1. Enable TechPreviewNoUpgrade feature gate $ oc patch featuregate cluster -p '{"spec": {"featureSet": "TechPreviewNoUpgrade"}}' --type merge 2. any user login to console and create a project from web, set 'Display name' and 'Description' then click on 'Create' 3. Check created project YAML $ oc get project ku-5 -o json | jq .metadata.annotations { "openshift.io/description": "", "openshift.io/display-name": "", "openshift.io/requester": "kube:admin", "openshift.io/sa.scc.mcs": "s0:c28,c17", "openshift.io/sa.scc.supplemental-groups": "1000790000/10000", "openshift.io/sa.scc.uid-range": "1000790000/10000" }
Actual results:
display-name and description are all empty
Expected results:
display-name and description should be set to the values user had configured
Additional info:
once TP is enabled, customized create project modal is looking like https://drive.google.com/file/d/1HmIlm0u_Ia_TPsa0ZAGyTloRmpfD0WYk/view?usp=drive_link
Description of problem:
When deploying with endpoint overrides, the block CSI driver will try to use the default endpoints rather than the ones specified.
Description of problem:
[IPSEC] 'encapsulation=no' is not supported in OVN, 'Never' option in API might need to be removed
Version-Release number of selected component (if applicable):
Pre-merge testing build openshift/cluster-network-operator#2573
How reproducible:
Always
Steps to Reproduce:
1. % oc patch networks.operator.openshift.io cluster --type=merge -p '{ "spec":{ "defaultNetwork":{ "ovnKubernetesConfig":{ "ipsecConfig":{ "mode":"Full", "full":{"encapsulation": "Never"}}}}}}'
From https://redhat-internal.slack.com/archives/C01G7T6SYSD/p1732621519461779,
encapsulation=no is not supported currently.
2.
3.
Actual results:
No 'encapsulation=no' as parameter added in /etc/ipsec.d/openshift.conf, still use default setting which is auto.
Expected results:
Suggest remove 'Never' from API.
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
Users could not install 4.19 ocp clusters with 4.19 oc-mirror it fails with error below [core@ci-op-r0wcschh-0f79b-mxw75-bootstrap ~]$ sudo crictl ps FATA[0000] validate service connection: validate CRI v1 runtime API for endpoint "unix:///var/run/crio/crio.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/crio/crio.sock: connect: no such file or directory" [core@ci-op-r0wcschh-0f79b-mxw75-bootstrap ~]$ sudo crictl img FATA[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/crio/crio.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/crio/crio.sock: connect: no such file or directory" [core@ci-op-r0wcschh-0f79b-mxw75-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service Dec 27 04:04:04 ci-op-r0wcschh-0f79b-mxw75-bootstrap release-image-download.sh[2568]: Error: initializing source docker://registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: (Mirrors also failed: [ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: reading manifest sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1 in ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release: manifest unknown]): registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: pinging container registry registry.build02.ci.openshift.org: Get "https://registry.build02.ci.openshift.org/v2/": dial tcp 34.74.144.21:443: i/o timeout Dec 27 04:04:04 ci-op-r0wcschh-0f79b-mxw75-bootstrap podman[2568]: 2024-12-27 04:04:04.637824679 +0000 UTC m=+243.178748520 image pull-error registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1 initializing source docker://registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: (Mirrors also failed: [ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: reading manifest sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1 in ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release: manifest unknown]): registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: pinging container registry registry.build02.ci.openshift.org: Get "https://registry.build02.ci.openshift.org/v2/": dial tcp 34.74.144.21:443: i/o timeout Dec 27 04:04:04 ci-op-r0wcschh-0f79b-mxw75-bootstrap release-image-download.sh[2107]: Pull failed. Retrying registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1... Dec 27 04:05:04 ci-op-r0wcschh-0f79b-mxw75-bootstrap release-image-download.sh[2656]: time="2024-12-27T04:05:04Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: (Mirrors also failed: [ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: reading manifest sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1 in ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release: manifest unknown]): registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: pinging container registry registry.build02.ci.openshift.org: Get \"https://registry.build02.ci.openshift.org/v2/\": dial tcp 34.74.144.21:443: i/o timeout" Dec 27 04:06:05 ci-op-r0wcschh-0f79b-mxw75-bootstrap release-image-download.sh[2656]: time="2024-12-27T04:06:05Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: (Mirrors also failed: [ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1: reading manifest sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e8a24cc83f6eb6c1eda1 in ci-op-r0wcschh-0f79b.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ci-op-r0wcschh/release/openshift/release: manifest unknown]): registry.build02.ci.openshift.org/ci-op-r0wcschh/release@sha256:bb836eae0322852a45fc1787716d8c4ddc935f0cfc44e
Version-Release number of selected component (if applicable):
Running command: 'oc-mirror' version --output=yaml W1227 14:24:55.919668 102 mirror.go:102] ⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2 clientVersion: buildDate: "2024-12-17T11:21:05Z" compiler: gc gitCommit: 27a04ae182eda7a668d0ad99c66a5f1e0435010b gitTreeState: clean gitVersion: 4.19.0-202412170739.p0.g27a04ae.assembly.stream.el9-27a04ae goVersion: go1.23.2 (Red Hat 1.23.2-1.el9) X:strictfipsruntime major: "" minor: "" platform: linux/amd64
How reproducible:
Always
Steps to Reproduce:
1. Install OCP4.19 cluster via oc-mirror 4.19 2. 3.
Actual results:
Users see the error as described in the Description
Expected results:
Installation should be successful.
Additional info:
More details in jira https://issues.redhat.com/browse/OCPQE-27853 More details in thread https://redhat-internal.slack.com/archives/C050P27C71S/p1735550241970219
Currently the location of the cache directory can be set via the environment variable `OC_MIRROR_CACHE`. The only problem is that the env var is not easily discoverable by users. It would be better if we had a command line option (e.g `-cache-dir <dir>`) which is discoverable via `-help`.
Description of problem:
For same image , v1 has more than 1 tags , but for v2 only has 1 tag, when delete executed, may remain some tags created by v1.
Version-Release number of selected component (if applicable):
./oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.2.0-alpha.1-309-g63a5556", GitCommit:"63a5556a", GitTreeState:"clean", BuildDate:"2024-10-23T02:42:55Z", GoVersion:"go1.23.0", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. mirror catalog package for v1: kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: sandboxed-containers-operator `oc-mirror -c config-catalog-v1.yaml docker://localhost:5000/catalog --dest-use-http` 2. mirror same package for v2 : kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: sandboxed-containers-operator `oc-mirror -c config-catalog-v2.yaml --workspace file://ws docker://localhost:5000/catalog --v2 --dest-tls-verify=false` 3. generate the delete image list with config : kind: DeleteImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 delete: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: sandboxed-containers-operator `oc-mirror delete -c config-d-catalog-v2.yaml --workspace file://ws docker://localhost:5000/catalog --v2 --dest-tls-verify=false --generate` 4. Execute the delete action: `oc-mirror delete --delete-yaml-file ws/working-dir/delete/delete-images.yaml docker://localhost:5000/catalog --v2 --dest-tls-verify=false `
Actual results:
1. v1 has more than 1 tags: [fedora@preserve-fedora-yinzhou ~]$ curl localhost:5000/v2/catalog/openshift4/ose-kube-rbac-proxy/tags/list |json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 81 100 81 0 0 24382 0 --:--:-- --:--:-- --:--:-- 27000 { "name": "catalog/openshift4/ose-kube-rbac-proxy", "tags": [ "cb9a8d8a", "d07492b2" ] } 2. v2 only has 1 tag: [fedora@preserve-fedora-yinzhou ~]$ curl localhost:5000/v2/catalog/openshift4/ose-kube-rbac-proxy/tags/list |json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 148 100 148 0 0 60408 0 --:--:-- --:--:-- --:--:-- 74000 { "name": "catalog/openshift4/ose-kube-rbac-proxy", "tags": [ "cb9a8d8a", "f6c37678f1eb3279e603f63d2a821b72394c52d25c2ed5058dc29d4caa15d504", "d07492b2" ] } 4. after delete , we could see still has tags remaining: [fedora@preserve-fedora-yinzhou ~]$ curl localhost:5000/v2/catalog/openshift4/ose-kube-rbac-proxy/tags/list |json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 70 100 70 0 0 28056 0 --:--:-- --:--:-- --:--:-- 35000 { "name": "catalog/openshift4/ose-kube-rbac-proxy", "tags": [ "d07492b2" ] }
Expected results:
4. delete all the tags for the same image.
Additional info:
Component Readiness has found a potential regression in the following test:
[sig-network][Feature:Router][apigroup:route.openshift.io] The HAProxy router converges when multiple routers are writing conflicting status [Suite:openshift/conformance/parallel]
Significant regression detected.
Fishers Exact probability of a regression: 99.98%.
Test pass rate dropped from 99.51% to 93.75%.
Sample (being evaluated) Release: 4.18
Start Time: 2024-10-29T00:00:00Z
End Time: 2024-11-05T23:59:59Z
Success Rate: 93.75%
Successes: 40
Failures: 3
Flakes: 5
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 99.51%
Successes: 197
Failures: 1
Flakes: 6
Description of problem:
After the upgrade to OpenShift Container Platform 4.17, it's being observed that aws-efs-csi-driver-controller-metrics/aws-efs-csi-driver-controller-metrics is reporting target down state. When checking the newly created Container one can find the below logs, that may explain the effect seen/reported. $ oc logs aws-efs-csi-driver-controller-5b8d5cfdf4-zwh67 -c kube-rbac-proxy-8211 W1119 07:53:10.249934 1 deprecated.go:66] ==== Removed Flag Warning ====================== logtostderr is removed in the k8s upstream and has no effect any more. =============================================== I1119 07:53:10.250382 1 kube-rbac-proxy.go:233] Valid token audiences: I1119 07:53:10.250431 1 kube-rbac-proxy.go:347] Reading certificate files I1119 07:53:10.250645 1 kube-rbac-proxy.go:395] Starting TCP socket on 0.0.0.0:9211 I1119 07:53:10.250944 1 kube-rbac-proxy.go:402] Listening securely on 0.0.0.0:9211 I1119 07:54:01.440714 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:54:19.860038 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:54:31.432943 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:54:49.852801 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:55:01.433635 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:55:19.853259 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:55:31.432722 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:55:49.852606 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:56:01.432707 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:56:19.853137 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:56:31.440223 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:56:49.856349 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:57:01.432528 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:57:19.853132 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:57:31.433104 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:57:49.852859 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:58:01.433321 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused I1119 07:58:19.853612 1 log.go:245] http: proxy error: dial tcp 127.0.0.1:8211: connect: connection refused
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.17
How reproducible:
Always
Steps to Reproduce:
1. Install OpenShift Container Platform 4.17 2. Install aws-efs-csi-driver-operator 3. Create efs.csi.aws.com CSIDriver object and wait for the aws-efs-csi-driver-controller to roll out.
Actual results:
The below Target Down Alert is being raised 50% of the aws-efs-csi-driver-controller-metrics/aws-efs-csi-driver-controller-metrics targets in Namespace openshift-cluster-csi-drivers namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.
Expected results:
The ServiceMonitor endpoint should be reachable and properly responding with the desired information to monitor the health of the component.
Additional info:
Description of problem:
We are getting the below error on OCP 4.16 while creating the egressfirewall with uppercase:
~~~
[a-z0-9])?\.?$'
#
~~~
When I check the code
https://github.com/openshift/ovn-kubernetes/blob/release-4.15/go-controller/pkg/crd/egressfirewall/v1/types.go#L80-L82
types.go
~~~
// dnsName is the domain name to allow/deny traffic to. If this is set, cidrSelector and nodeSelector must be unset.
// kubebuilder:validation:Pattern=^([A-Za-z0-9-]\.)*[A-Za-z0-9-]+\.?$
DNSName string `json:"dnsName,omitempty"`
~~~
https://github.com/openshift/ovn-kubernetes/blob/release-4.16/go-controller/pkg/crd/egressfirewall/v1/types.go#L80-L85
types.go
~~~
// dnsName is the domain name to allow/deny traffic to. If this is set, cidrSelector and nodeSelector must be unset.
// For a wildcard DNS name, the '' will match only one label. Additionally, only a single '' can be
// used at the beginning of the wildcard DNS name. For example, '*.example.com' will match 'sub1.example.com' // but won't match 'sub2.sub1.example.com'.
// kubebuilder:validation:Pattern=`^(*\.)?([A-Za-z0-9-]\.)*[A-Za-z0-9-]+\.?$`
DNSName string `json:"dnsName,omitempty"`
~~~
Code looks its supported for the upper case.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Deploy the cluster with OCP 4.16.x
2. Create EgressFirewall with the upper case for dnsName
~~~
apiVersion: k8s.ovn.org/v1
kind: EgressFirewall
metadata:
name: default
spec:
egress:
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
When the cluster is created with a secure proxy enabled, and certificate is set in configuration.proxy.trustCA, the cluster fails to complete provisioning. 4.19.0-0.nightly-2024-12-10-040415
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. Create a cluster with a secure proxy, certificate is set in .spec.configuration.proxy.trustCA. 3.
Actual results:
cluster does not complete provisioning
Expected results:
cluster completes
Additional info:
root cause is that certificate in additionalTrustBunlde isn't propagated into ingress proxy. slack: https://redhat-internal.slack.com/archives/G01QS0P2F6W/p1734047816669079?thread_ts=1734023627.636019&cid=G01QS0P2F6W
Description of problem:
SecretProviderClass Doesn't Allow Object Encoding
Version-Release number of selected component (if applicable):
How reproducible:
Every time
Steps to Reproduce:
1. 2. 3.
Actual results:
SecretProviderClass Doesn't Allow Object Encoding Per CP MI
Expected results:
SecretProviderClass Allows Object Encoding Per CP MI
Additional info:
Description of problem:
I get this build warning when building: warning Pattern ["asn1js@latest"] is trying to unpack in the same destination "~/.cache/yarn/v6/npm-asn1js-2.0.26-0a6d435000f556a96c6012969d9704d981b71251-integrity/node_modules/asn1js" as pattern ["asn1js@^2.0.26"]. This could result in non-deterministic behavior, skipping.
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
Always
Steps to Reproduce:
1. Run ./clean-frontend && ./build-frontend.sh 2. Observe build output 3.
Actual results:
I get the warning
Expected results:
No warning
Additional info:
Description of problem:
for vSphere multi-nic setting, it should have validation that allow vSphere network count from 1 to 10. but installer didn't print any error info when I set 11 nics in install-config.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-01-25-163410
How reproducible:
setting 11 nics under install-config.platform.vsphere.failureDomains.topology.networks and continue installation
Steps to Reproduce:
1. 2. 3.
Actual results:
installation continue and failed
Expected results:
should print validation info.
Additional info:
Moving the bug https://bugzilla.redhat.com/show_bug.cgi?id=1802553 to this Jira card
Currently we had a customer that triggered the upgrade from 4.1.27 to 4.3, having intermediate versions on 4.2 in partial state. We have asked for details of the CVO from the customer to understand better the procedure taken, but we might need to implement a way to either stop the upgrade in case customer makes a mistake or block the upgrade if the customer changes the channel on the console to a version which the upgrade does not support, like in this case.
OTA - Inhibit minor version upgrades when an upgrade is in progress
We should inhibit minor version upgrades via Upgradeable=False whenever an existing upgrade is in progress. This prevents retargetting of upgrades before we've reached a safe point.
Imagine:
Be running 4.6.z.
Request an update to 4.7.z'.
CVO begins updating to 4.7.z'.
CVO requests recommended updates from 4.7.z', and hears about 4.8.z".
User accepts recommended update to 4.8.z" before the 4.7.z' OLM operator had come out to check its children's max versions against 4.8 and set Upgradeable=False.
Cluster core hits 4.8.z" and some OLM operators fail on compat violations.
This should not inhibit further z-stream upgrades, but we should be sure that we catch the case of 4.6.z to 4.7.z to 4.7.z+n to 4.8.z whenever 4.7.z was not marked as Complete.
Update:
Eventually, the output of this card:
This issue tracks the updation of k8s and related openshift APIs to a recent version, to keep in-line with other MAPI providers.
The python3-inotify package is installed as dependency but it should be explicitly mentioned in the packages list to avoid issues like okd-project/okd#2113
Description of problem:
Filter 'Name' on resource list page doesn't align well when language is set to Chinese/Japanese/Korean
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-02-26-172353
How reproducible:
Always
Steps to Reproduce:
1.Set language as Chinese, Japanese or Korean. Check column 'Name' on resource list page, eg, on pods list page. 2. 3.
Actual results:
1. Filter 'Name' on resource list page doesn't align well. screenshots: https://drive.google.com/drive/folders/1w1-sXSj3RdIWIVbAGl8q0hZaHuFHHCJM?usp=drive_link
Expected results:
1. Should align well and keep consistent with other filter input.
Additional info:
Description of problem:
Tracking per-operator fixes for the following related issues static pod node, installer, and revision controllers: https://issues.redhat.com/browse/OCPBUGS-45924 https://issues.redhat.com/browse/OCPBUGS-46372 https://issues.redhat.com/browse/OCPBUGS-48276
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
In an effort to ensure all HA components are not degraded by design during normal e2e test or upgrades, we are collecting all operators that are blipping Degraded=True during any payload job run. This card captures machine-config operator that blips Degraded=True during upgrade runs. Example Job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.18-upgrade-from-stable-4.17-e2e-azure-ovn-upgrade/1843023092004163584 Reasons associated with the blip: RenderConfigFailed For now, we put an exception in the test. But it is expected that teams take action to fix those and remove the exceptions after the fix go in. Exceptions are defined here: See linked issue for more explanation on the effort.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following tests:
install should succeed: infrastructure
install should succeed: overall
Significant regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 99.24% to 89.63%.
Sample (being evaluated) Release: 4.18
Start Time: 2024-12-13T00:00:00Z
End Time: 2024-12-20T12:00:00Z
Success Rate: 89.63%
Successes: 120
Failures: 17
Flakes: 27
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 99.24%
Successes: 939
Failures: 8
Flakes: 99
View the test details report for additional context.
Description of problem:
Tracking per-operator fixes for the following related issues static pod node, installer, and revision controllers: https://issues.redhat.com/browse/OCPBUGS-45924 https://issues.redhat.com/browse/OCPBUGS-46372 https://issues.redhat.com/browse/OCPBUGS-48276
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Visit `/project-details/ns/default/` and note the breadcrumbs are not present. There is container for them, but no actual content.
https://drive.google.com/file/d/1FE0GHmWDKNqV8EXbTXx-lBSjQyK6--g_/view?usp=sharing
Update openshift/api to k8s 1.32
Description of problem:
When a new configuration is picked up by the authentication operator, it rolls out a new revision of oauth-server pods. However, since the pods define `terminationGracePeriodSeconds`, the old-revision pods would still be running even after the oauth-server deployment reports that all pods have been updated to the newest revision, which triggers the authentication operator to report Progressing=false. The above behavior might cause that tests (and possible any observers) that expect the newer revision would be still routed to the old pods, causing confusion.
Version-Release number of selected component (if applicable):
4.13
How reproducible:
Always
Steps to Reproduce:
1. Trigger new oauth-server rollout 2. Observe the authentication operator reporting Progressing while also watching the number of pods in the openshift-authentication namespace
Actual results:
CAO reports Progressing=false even though there are more than the expected number of pods running.
Expected results:
CAO waits to report Progressing=false when only the new revision of pods is running in the openshift-authentication NS.
Additional info:
-
Description of problem:
OLMv1 is being GA'd with OCP 4.18, together with OLMv0. The long(ish)-term plan right now is for OLM v0 and v1 to be able to coexist on a cluster. However, their access to installable extensions is through different resources. v0 uses CatalogSource and v1 uses ClusterCatalog. We expect to see catalog content begin to diverge at some point, but don't have a specific timeline for it yet. oc-mirror v2 should generate ClusterCatalog YAML along with CatalogSource YAML. We also work with docs team to document ClusterCatalog YAML is only needed to be applied when managing Operator catalogs with OLM v1.
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
ClusterCatalog resource is generated for operators.
Additional info:
OLMv1 currently only works for a small subset of operators in the catalog.
Description of problem:
Below error is occuring while creating Persistent Volume using LUN IDs,
Error "Invalid value: 300: must be between 0 and 255, inclusive" for field "spec.fc.lun".
Actual results:
Persistent volume is not creating.
Expected results:
Persistent Volume should create while using fiber channel
The "oc adm pod-network" command for working with openshift-sdn multitenant mode is now totally useless in OCP 4.17 and newer clusters (since it's only useful with openshift-sdn, and openshift-sdn no longer exists as of OCP 4.17). Of course, people might use a new oc binary to talk to an older cluster, but probably the built-in documentation should make it clearer that this is not a command that will be useful to 99% of users.
If it's possible to make "pod-network" not show up as a subcommand in "oc adm -h" that would probably be good. If not, it should probably have a description like "Manage OpenShift-SDN Multitenant mode networking [DEPRECATED]", and likewise, the longer descriptions of the pod-network subcommands should talk about "OpenShift-SDN Multitenant mode" rather than "the redhat/openshift-ovs-multitenant network plugin" (which is OCP 3 terminology), and maybe should explicitly say something like "this has no effect when using the default OpenShift Networking plugin (OVN-Kubernetes)".
Description of problem:
The e2e-ibmcloud-operator presubmit job for the cluster-ingress-operator repo introduced in https://github.com/openshift/release/pull/56785 always fails due to DNS. Note that this job has `always_run: false` and `optional: true` so it requires calling /test e2e-ibmcloud-operator on a PR to make it appear. These failures are not blocking any PRs from merging. Example failure.
The issue is that IBM Cloud has DNS propagation issues, similar to the AWS DNS issues (OCPBUGS-14966), except:
The PR https://github.com/openshift/cluster-ingress-operator/pull/1164 was an attempt at fixing the issue by both resolving the DNS name inside of the cluster and allowing for a couple minute "warmup" interval to avoid negative caching. I found (via https://github.com/openshift/cluster-ingress-operator/pull/1132) that the SOA TTL is ~30 minutes, which if you trigger negative caching, you will have to wait 30 minutes for the IBM DNS Resolver to refresh the DNS name.
However, I found that if you wait ~7 minutes for the DNS record to propagate and don't query the DNS name, it will work after that 7 minute wait (I call it the "warmup" period).
The tests affected are any tests that use a DNS name (wildcard or load balancer record):
The two paths I can think of are:
Version-Release number of selected component (if applicable):
4.19
How reproducible:
90-100%
Steps to Reproduce:
1. Run /test e2e-ibmcloud-operator
Actual results:
Tests are flakey
Expected results:
Tests should work reliably
Additional info:
Description of problem:
Following error returns in IPI Baremetal install with recent 4.18 builds. In bootstrap vm, https is not configured on 6180 port used in boot iso url. openshift-master-1: inspection error: Failed to inspect hardware. Reason: unable to start inspection: HTTP POST https://[2620:52:0:834::f1]:8000/redfish/v1/Managers/7fffdce9-ff4a-4e6a-b598-381c58564ca5/VirtualMedia/Cd/Actions/VirtualMedia.InsertMedia returned code 500. Base.1.0.GeneralError: Failed fetching image from URL https://[2620:52:0:834:f112:3cff:fe47:3a0a]:6180/redfish/boot-93d79ad0-0d56-4c8f-a299-6dc1b3f40e74.iso: HTTPSConnectionPool(host='2620:52:0:834:f112:3cff:fe47:3a0a', port=6180): Max retries exceeded with url: /redfish/boot-93d79ad0-0d56-4c8f-a299-6dc1b3f40e74.iso (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1131)'))) Extended information: [{'@odata.type': '/redfish/v1/$metadata#Message.1.0.0.Message', 'MessageId': 'Base.1.0.GeneralError'}]"
Version-Release number of selected component (if applicable):
4.18 ec.4, 4.18.0-0.nightly-2024-11-27-162407
How reproducible:
100%
Steps to Reproduce:
1. trigger ipi baremetal install with dual stack config using virtual media 2. 3.
Actual results:
inspection fails at fetching boot iso
Expected results:
Additional info:
# port 6180 used in ironic ipv6 url is not configured for https. Instead, ssl service is running # at https://[2620:52:0:834:f112:3cff:fe47:3a0a]:6183. # May be introduced by OCPBUGS-39404. [root@api core]# cat /etc/metal3.env AUTH_DIR=/opt/metal3/auth IRONIC_ENDPOINT="http://bootstrap-user:pJ0R9XXsxUfoYVK2@localhost:6385/v1" IRONIC_EXTERNAL_URL_V6="https://[2620:52:0:834:f112:3cff:fe47:3a0a]:6180/" METAL3_BAREMETAL_OPERATOR_IMAGE="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e142d5989415da3c1035d04f84fa765c127bf2cf3406c4612e36607bb03384d9" [root@api core]# echo "" | openssl s_client -connect localhost:6180 CONNECTED(00000003) 405CE187787F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:354: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 5 bytes and written 295 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
Description of problem:
When creating a new Shipwright Build via the form (ODC-7595), the form shows all the params available on the build strategy.
Expected results:
The parameters should be hidden behind an "Advanced" botton.
Additional info:
Except the following parameters, the rest for each build strategy should be hidden and moved to the advanced section.
Description of problem:
checked on 4.18.0-0.nightly-2024-12-07-130635/4.19.0-0.nightly-2024-12-07-115816, admin console, go to alert details page, "No datapoints found." on alert details graph. see picture for CannotRetrieveUpdates alert: https://drive.google.com/file/d/1RJCxUZg7Z8uQaekt39ux1jQH_kW9KYXd/view?usp=drive_link
issue exists in 4.18+, no such issue with 4.17
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always on 4.18+
Steps to Reproduce:
1. see the description
Actual results:
"No datapoints found." on alert details graph
Expected results:
show correct graph
Description of problem:
As a User when we attempt to "ReRun" a resolver based pipelinerun from OpenshiftConsole, UI errors with message "Invalid PipelineRun configuration, unable to start Pipeline." Slack thread: https://redhat-internal.slack.com/archives/CG5GV6CJD/p1730876734675309
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Create a resolver based pipelinerun 2. Attempt to "ReRun" the same from Console
Actual results:
ReRun Errors
Expected results:
ReRun should be triggered successfully
Additional info:
Description of problem:
The alertmanager-user-workload Service Account has "automountServiceAccountToken: true"
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
1. Enable Alertmanager for user-defined monitoring. 2. oc get sa -n openshift-user-workload-monitoring alertmanager-user-workload -o yaml 3.
Actual results:
Has "automountServiceAccountToken: true"
Expected results:
Has "automountServiceAccountToken: false" or no mention of automountServiceAccountToken.
Additional info:
It is recommended to not enable token automount for service accounts in general.
Description of problem:
The notification drawer close button throws a runtime error when clicked.
Version-Release number of selected component (if applicable):
4.19
How reproducible:
Always
Steps to Reproduce:
1. Click the notification drawer icon in the masthead 2. Click the close button in the notifcation drawer 3.
Actual results:
A runtime error is thrown
Expected results:
The notification drawer closes
Additional info:
This was a regression introduced by https://github.com/openshift/console/pull/14680
When deploying with an endpoint override for the resourceController, the Power VS machine API provider will ignore the override.
Description of problem:
Check on route create form/yaml, the Path field is set "/" by default, when create passthrough route, it prompts error about not supporting "spec.path", user needs to clear this field.
Version-Release number of selected component (if applicable):
4.17.0-0.nightly-2024-09-13-193731
How reproducible:
Steps to Reproduce:
1.Go to Networking->Routes page, click "Create Route", check the path field. 2.Choose Passthrough type for insecure set and click "Create" button. 3.
Actual results:
1. It has "/" set by default. 2. Error info shows up: Error "Invalid value: "/": passthrough termination does not support paths" for field "spec.path".
Expected results:
1. The field should be null by default.
Additional info:
Managed services marks a couple of nodes as "infra" so user workloads don't get scheduled on them. However, platform daemonsets like iptables-alerter should run there – and the typical toleration for that purpose should be:
tolerations:
- operator: Exists
instead the toleration is
tolerations: - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule"
Examples from other platform DS:
$ for ns in openshift-cluster-csi-drivers openshift-cluster-node-tuning-operator openshift-dns openshift-image-registry openshift-machine-config-operator openshift-monitoring openshift-multus openshift-multus openshift-multus openshift-network-diagnostics openshift-network-operator openshift-ovn-kubernetes openshift-security; do echo "NS: $ns"; oc get ds -o json -n $ns|jq '.items.[0].spec.template.spec.tolerations'; done NS: openshift-cluster-csi-drivers [ { "operator": "Exists" } ] NS: openshift-cluster-node-tuning-operator [ { "operator": "Exists" } ] NS: openshift-dns [ { "key": "node-role.kubernetes.io/master", "operator": "Exists" } ] NS: openshift-image-registry [ { "operator": "Exists" } ] NS: openshift-machine-config-operator [ { "operator": "Exists" } ] NS: openshift-monitoring [ { "operator": "Exists" } ] NS: openshift-multus [ { "operator": "Exists" } ] NS: openshift-multus [ { "operator": "Exists" } ] NS: openshift-multus [ { "operator": "Exists" } ] NS: openshift-network-diagnostics [ { "operator": "Exists" } ] NS: openshift-network-operator [ { "effect": "NoSchedule", "key": "node-role.kubernetes.io/master", "operator": "Exists" } ] NS: openshift-ovn-kubernetes [ { "operator": "Exists" } ] NS: openshift-security [ { "operator": "Exists" } ]
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Requests allow up to 30s for etcd to respond. Readiness probes only allow 9s for etcd to respond. When etcd latency is between 10-30s, standard requests will succeed, but due to the readiness probe configuration we lose every apiserver endpoint at the same time. This requires correction in the pod definitions and the load balancers. Making the ongoing readiness check `readyz?exclude=etcd` should correct the issue.
Off the top of my head this will include
This is a low cost, low risk, high benefit change.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The UDN Details page shows a "group" of attributes called "Layer configuration". That does not really add any benefit and the name is just confusing. Let's just remove the grouping and keep the attributes flat.
Version-Release number of selected component (if applicable):
rc.4
How reproducible:
Always
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Below tests fail on ipv6primary dualstack cluster because the router deployed is not prepared for dualstack:
[sig-network][Feature:Router][apigroup:image.openshift.io] The HAProxy router should serve a route that points to two services and respect weights [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:operator.openshift.io] The HAProxy router should respond with 503 to unrecognized hosts [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:operator.openshift.io] The HAProxy router should serve routes that were created from an ingress [apigroup:route.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io][apigroup:operator.openshift.io] The HAProxy router should support reencrypt to services backed by a serving certificate automatically [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io] The HAProxy router should override the route host for overridden domains with a custom value [apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io] The HAProxy router should override the route host with a custom value [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io] The HAProxy router should run even if it has no access to update status [apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io] The HAProxy router should serve the correct routes when scoped to a single namespace and label set [Skipped:Disconnected] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router][apigroup:route.openshift.io] when FIPS is disabled the HAProxy router should serve routes when configured with a 1024-bit RSA key [Feature:Networking-IPv4] [Suite:openshift/conformance/parallel] [sig-network][Feature:Router] The HAProxy router should expose prometheus metrics for a route [apigroup:route.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
That is confirmed by accessing to the router pod and checking the connectivity locally:
sh-4.4$ curl -k -s -m 5 -o /dev/null -w '%{http_code}\n' --header 'Host: FIRST.example.com' "http://127.0.0.1/Letter" 200 sh-4.4$ echo $? 0
sh-4.4$ curl -k -s -m 5 -o /dev/null -w '%{http_code}\n' --header 'Host: FIRST.example.com' "http://fd01:0:0:5::551/Letter" 000 sh-4.4$ echo $? 3
sh-4.4$ curl -k -s -m 5 -o /dev/null -w '%{http_code}\n' --header 'Host: FIRST.example.com' "http://[fd01:0:0:5::551]/Letter" 000 sh-4.4$ echo $? 7
The default router deployed in the cluster supports dualstack. Hence it's possible and required to update the router image configuration usedin the tests to be able to answer both ipv4 and ipv6.
Version-Release number of selected component (if applicable): https://github.com/openshift/origin/tree/release-4.15/test/extended/router/
How reproducible: Always.
Steps to Reproduce: Run the tests in ipv6primary dualstack cluster.
Actual results: Tests failing as below:
<*errors.errorString | 0xc001eec080>:
last response from server was not 200:
{
s: "last response from server was not 200:\n",
}
occurred
Ginkgo exit error 1: exit with code 1
Expected results: Test passing
Description of problem:
After OCPBUGS-13726, Hypershift honors ImageConfig provided by the user in the HostedCluster. Providing both allowedRegistries and blockedRegistries is forbidden https://github.com/openshift/api/blob/1e963d8dc4663f4f004f44fd58459381a771bdb5/config/v1/types_image.go#L126 If we do that in HyperShift, it will block any NodePool creation but no error is visible in the HostedCluster, so it is not easy to identify the error. The error should instead be visible in the existing HC condition ValidHostedControlPlaneConfiguration
Version-Release number of selected component (if applicable):
4.15.6
How reproducible:
Always
Steps to Reproduce:
1. Create HC 2. oc patch hc -n $HC_NS $HC_NAME -p '{"spec":{"configuration":{"image":{"registrySources":{"allowedRegistries":["docker.io"], "blockedRegistries":["test.io"]}}}}}' --type=merge 3. New node pools not coming up and condition visible in the NP
- lastTransitionTime: "2024-04-11T12:49:32Z"
message: 'Failed to generate payload: error getting ignition payload: failed to
execute machine-config-controller: machine-config-controller process failed:
exit status 255'
observedGeneration: 1
reason: InvalidConfig
status: ""
type: ValidGeneratedPayload
Actual results:
HC condition successful
- lastTransitionTime: "2024-04-11T08:59:01Z" message: Configuration passes validation observedGeneration: 4 reason: AsExpected status: "True" type: ValidHostedControlPlaneConfiguration
Expected results:
Above HC condition to be failed
Additional info:
Description of problem:
IBM Cloud has a IBM Cloud CAPI argument for metrics-bind-addr, which it appears is being dropped in a future cluster-api release. Drop the argument now to account for when this argument is no longer available.
Version-Release number of selected component (if applicable):
4.19
How reproducible:
100%
Steps to Reproduce:
1. Update IBM Cloud CAPI and Cluster-API components (Cluster-API to 1.9.3) 2. Attempt to create cluster for IBM Cloud, using CAPI support
Actual results:
time="2025-01-24T20:50:00Z" level=info msg="Running process: ibmcloud infrastructure provider with args [--provider-id-fmt=v2 -v=2 --metrics-bind-addr=0 --health-addr=127.0.0.1:35849 --leader-elect=false --webhook-port=44443 --webhook-cert-dir=/tmp/envtest-serving-certs-3963032230 --namespace=openshift-cluster-api-guests --kubeconfig=/root/cluster-deploys/capi-drop-v1-1/.clusterapi_output/envtest.kubeconfig]" time="2025-01-24T20:50:00Z" level=debug msg="unknown flag: --metrics-bind-addr" time="2025-01-24T20:50:00Z" level=debug msg="Usage of cluster-deploys/capi-drop-v1-1/cluster-api/cluster-api-provider-ibmcloud:" time="2025-01-24T20:50:00Z" level=debug msg=" --add_dir_header If true, adds the file directory to the header of the log messages" time="2025-01-24T20:50:00Z" level=debug msg=" --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)" time="2025-01-24T20:50:00Z" level=debug msg=" --diagnostics-address string The address the diagnostics endpoint binds to. Per default metrics are served via https and withauthentication/authorization. To serve via http and without authentication/authorization set --insecure-diagnostics. If --insecure-diagnostics is not set the diagnostics endpoint also serves pprof endpoints and an endpoint to change the log level. (default \":8443\")" time="2025-01-24T20:50:00Z" level=debug msg=" --health-addr string The address the health endpoint binds to. (default \":9440\")" time="2025-01-24T20:50:00Z" level=debug msg=" --insecure-diagnostics Enable insecure diagnostics serving. For more details see the description of --diagnostics-address." time="2025-01-24T20:50:00Z" level=debug msg=" --kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster." time="2025-01-24T20:50:00Z" level=debug msg=" --leader-elect Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager." time="2025-01-24T20:50:00Z" level=debug msg=" --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)" time="2025-01-24T20:50:00Z" level=debug msg=" --log-json-info-buffer-size quantity [Alpha] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this." time="2025-01-24T20:50:00Z" level=debug msg=" --log-json-split-stream [Alpha] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this." time="2025-01-24T20:50:00Z" level=debug msg=" --log-text-info-buffer-size quantity [Alpha] In text format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). Enable the LoggingAlphaOptions feature gate to use this." time="2025-01-24T20:50:00Z" level=debug msg=" --log-text-split-stream [Alpha] In text format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. Enable the LoggingAlphaOptions feature gate to use this." time="2025-01-24T20:50:00Z" level=debug msg=" --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)" time="2025-01-24T20:50:00Z" level=debug msg=" --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)" time="2025-01-24T20:50:00Z" level=debug msg=" --log_file string If non-empty, use this log file (no effect when -logtostderr=true)" time="2025-01-24T20:50:00Z" level=debug msg=" --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)" time="2025-01-24T20:50:00Z" level=debug msg=" --logging-format string Sets the log format. Permitted formats: \"json\" (gated by LoggingBetaOptions), \"text\". (default \"text\")" time="2025-01-24T20:50:00Z" level=debug msg=" --logtostderr log to standard error instead of files (default true)" time="2025-01-24T20:50:00Z" level=debug msg=" --namespace string Namespace that the controller watches to reconcile cluster-api objects. If unspecified, the controller watches for cluster-api objects across all namespaces." time="2025-01-24T20:50:00Z" level=debug msg=" --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)" time="2025-01-24T20:50:00Z" level=debug msg=" --provider-id-fmt string ProviderID format is used set the Provider ID format for Machine (default \"v2\")" time="2025-01-24T20:50:00Z" level=debug msg=" --service-endpoint string Set custom service endpoint in semi-colon separated format: ${ServiceRegion1}:${ServiceID1}=${URL1},${ServiceID2}=${URL2};${ServiceRegion2}:${ServiceID1}=${URL1}" time="2025-01-24T20:50:00Z" level=debug msg=" --skip_headers If true, avoid header prefixes in the log messages" time="2025-01-24T20:50:00Z" level=debug msg=" --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)" time="2025-01-24T20:50:00Z" level=debug msg=" --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=true) (default 2)" time="2025-01-24T20:50:00Z" level=debug msg=" --sync-period duration The minimum interval at which watched resources are reconciled. (default 10m0s)" time="2025-01-24T20:50:00Z" level=debug msg=" --tls-cipher-suites strings Comma-separated list of cipher suites for the webhook server and metrics server (the latter only if --insecure-diagnostics is not set to true). If omitted, the default Go cipher suites will be used. " time="2025-01-24T20:50:00Z" level=debug msg=" Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256. " time="2025-01-24T20:50:00Z" level=debug msg=" Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_RC4_128_SHA." time="2025-01-24T20:50:00Z" level=debug msg=" --tls-min-version string The minimum TLS version in use by the webhook server and metrics server (the latter only if --insecure-diagnostics is not set to true)." time="2025-01-24T20:50:00Z" level=debug msg=" Possible values are VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13. (default \"VersionTLS12\")" time="2025-01-24T20:50:00Z" level=debug msg=" -v, --v Level number for the log level verbosity (default 0)" time="2025-01-24T20:50:00Z" level=debug msg=" --vmodule pattern=N,... comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)" time="2025-01-24T20:50:00Z" level=debug msg=" --webhook-cert-dir string The webhook certificate directory, where the server should find the TLS certificate and key. (default \"/tmp/k8s-webhook-server/serving-certs/\")" time="2025-01-24T20:50:00Z" level=debug msg=" --webhook-port int The webhook server port the manager will listen on. (default 9443)" time="2025-01-24T20:50:00Z" level=debug msg="unknown flag: --metrics-bind-addr"
Expected results:
Successful cluster deployment using updated IBM Cloud CAPI
Additional info:
IBM Cloud CAPI needs to be updated to resolve known bugs. in the process of doing this, this requires updating Cluster-API to 1.9.3 https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/commit/ffa141c7edd53a58396403b7c0e3995c91580161 However, this appears to cause issues with the current IBM Cloud CAPI arguments within IPI, metrics-bind-addr. IBM Cloud is working on a patch to drop that argument, to allow for updating IBM Cloud CAPI to a newer level for bug fixes.
Description of problem:
When overriding the VPC endpoint in a PowerVS deployment, the VPC endpoint override is ignored by MAPI
Version-Release number of selected component (if applicable):
How reproducible:
easily
Steps to Reproduce:
1. Deploy a disconnected cluster 2. network operator will fail to come up 3.
Actual results:
Deploy fails and endpoint is ignored
Expected results:
Deploy should succeed with endpoint honored
Additional info:
Description of problem:
A user without CSR read permission can not view the Node page in Openshift console.
Version-Release number of selected component (if applicable):
4.16.30
How reproducible:
Always
Steps to Reproduce:
1. Create a 4.16.30 ROSA/OSD cluster 2. Assign a user with dedicated-admins group. dedicated-admins has get node permission but without CSR related permissions. 3. Open console access Compute - Node page
Actual results:
The page only shows an error, without other content: certificatesigningrequests.certificates.k8s.io is forbidden: User "xxxx" cannot list resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
Expected results:
The user should still be able to view nodes.
Additional info:
OSD-28173 - card for SRE tracking.
it is meant to select projects that don't even exist yet - it should work with label selectors,
Description of problem:
If the bootstrap fails, the installer will try to get the VM console logs via the AWS SDK which requires the ec2:GetConsoleOutput permission.
Version-Release number of selected component (if applicable):
all versions where we enabled VM console log gathering
How reproducible:
always
Steps to Reproduce:
1. Use minimal permissions and force a bootstrap failure 2. 3.
Actual results:
level=info msg=Pulling VM console logs level=error msg=UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::460538899914:user/ci-op-xgq2j8ch-f93c7-minimal-perm is not authorized to perform: ec2:GetConsoleOutput on resource: arn:aws:ec2:us-west-1:460538899914:instance/i-0fa40c9966e9f1ab9 because no identity-based policy allows the ec2:GetConsoleOutput action. Encoded authorization failure message: XYfLhyZ0pKnDzJrs9ZbOH8z8YkG03aPhT6U57EoqiLH8iS5PZvFgbgONlBuZfDswNpaNBVOfZcdPc1dWYoIsoPTXtQ_n32tzrdxloK7qpVbvkuesHtb8ytV8iLkpmOGyArMqp7Muphn2yXG9DQ5aijx-zQh_ShwirruMTZWhkZdx7_f1WtfjnCBVJGRwAc-rMZ_Xh82-jjxQlQbtBfgJ8COc3kQm7E_iJ1Ngyrcmu6bmVKCS6cEcGIVwRi03PZRtiemfejZfUT7yhKppB-zeeRm5bBWWVmRiuJhswquIW4dH0E9obNvq76-C0b2PR_V9ep-t0udUcypKGilqzqT1DY51gaP66GlSEfN5b4CTLTQxEhE73feZn4xEK0Qq4MkatPFJeGsUcxY5TXEBsGMooj4_D7wPFwkY46QEle41oqs-KNCWEifZSlV5f4IUyiSear85LlUIxBS9-_jfitV90Qw7MZM4z8ggIinQ_htfvRKgnW9tjREDj6hzpydQbViaeAyBod3Q-qi2vgeK6uh7Q6kqK3f8upu1hS8I7XD_TH-oP-npbVfkiPMIQGfy3vE3J5g1AyhQ24LUjR15y-jXuBOYvGIir21zo9oGKc0GEWRPdZr4suSbbx68rZ9TnTHXfwa0jrhIns24uwnANdR9U2NStE6XPJk9KWhbbz6VD6gRU72qbr2V7QKPiguNpeO_P5uksRDwEBWxDfQzMyDWx1zOhhPPAjOQRup1-vsPpJhkgkrsdhPebN0duz6Hd4yqy0RiEyb1sSMaQn_8ac_2vW9CLuWWbbt5qo2WlRllo3U7-FpvlP6BRGTPjv5z3O4ejrGsnfDxm7KF0ANvLU0KT2dZvKugB6j-Kkz56HXHebIzpzFPRpmo0B6H3FzpQ5IpzmYiWaQ6sNMoaatmoE2z420AJAOjSRBodqhgi2cVxyHDqHt0E0PQKM-Yt4exBGm1ZddC5TUPnCrDnZpdu2WLRNHMxEBgKyOzEON_POuDaOP0paEXFCflt7kNSlBRMRqAbOpGI_F96wlNmDO58KZDbPKgdOfomwkaR5icdeS-tQyQk2PnhieOTNL1M5hQZpLrzWVeJzZEtmZ_0vsePUdvXYusvL828ldyg8VCwq-B2oGD_ym_iPCINBC7sIy8Q0HVb5v5dzbs4l2UKcC7OzTG-TMlxphV20DqNmC5yCnHEdmnleNA48J69HdTMw_G7N9mo5IrXw049MjvYnia4NwarMGUvoBYnxROfQ2jprN7_BW-Cdyp2Ca2P9uU9AeSubeeQdzieazkXNeR9_4Su_EGsbQm Instance=i-0fa40c9966e9f1ab9
Expected results:
No failures.
Additional info:
See https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/57437/rehearse-57437-pull-ci-openshift-installer-master-e2e-aws-ovn-user-provisioned-dns/1860020284598259712 for an example of a failed job
Description of problem:
The OpenShift installer fails to create a cluster on an OpenStack Single-stack IPv6 environment - failed to run cluster api system
Version-Release number of selected component (if applicable):
Installer version: openshift-install version openshift-install 4.18.0-rc.3 built from commit 0f87b38910a84cfe3243fb878436bc052afc3187 release image registry.ci.openshift.org/ocp/release@sha256:668c92b06279cb5c7a2a692860b297eeb9013af10d49d2095f2c3fe9ad02baaa WARNING Release Image Architecture not detected. Release Image Architecture is unknown release architecture unknown default architecture amd64
RHOSO version:
[zuul@controller-0 ~]$ oc get openstackversions.core.openstack.org NAME TARGET VERSION AVAILABLE VERSION DEPLOYED VERSION controlplane 18.0.4-trunk-20241112.1 18.0.4-trunk-20241112.1 18.0.4-trunk-20241112.1
How reproducible:
Always
Steps to Reproduce:
1. Prepare openstack infra for openshift installation with Single-stack IPv6 (see the install-config.yaml below) 2. openshift-install create cluster
install-config.yaml:
apiVersion: v1 baseDomain: "shiftstack.local" controlPlane: name: master platform: openstack: type: "master" replicas: 3 compute: - name: worker platform: openstack: type: "worker" replicas: 2 metadata: name: "ostest" networking: clusterNetworks: - cidr: fd01::/48 hostPrefix: 64 machineNetwork: - cidr: "fd2e:6f44:5dd8:c956::/64" serviceNetwork: - fd02::/112 networkType: "OVNKubernetes" platform: openstack: cloud: "shiftstack" region: "regionOne" apiVIPs: ["fd2e:6f44:5dd8:c956::5"] ingressVIPs: ["fd2e:6f44:5dd8:c956::7"] controlPlanePort: fixedIPs: - subnet: name: "subnet-ssipv6" pullSecret: <omitted> sshKey: <omitted>
Actual results:
The openshift-install fails to start the controlplane - kube-apiserver:
INFO Started local control plane with envtest E0109 13:17:36.425059 30979 server.go:328] "unable to start the controlplane" err="timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)" logger="controller-runtime.test-env" tries=0 E0109 13:17:38.365005 30979 server.go:328] "unable to start the controlplane" err="timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)" logger="controller-runtime.test-env" tries=1 E0109 13:17:40.142385 30979 server.go:328] "unable to start the controlplane" err="timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)" logger="controller-runtime.test-env" tries=2 E0109 13:17:41.947245 30979 server.go:328] "unable to start the controlplane" err="timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)" logger="controller-runtime.test-env" tries=3 E0109 13:17:43.761197 30979 server.go:328] "unable to start the controlplane" err="timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)" logger="controller-runtime.test-env" tries=4 DEBUG Collecting applied cluster api manifests... ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to run cluster api system: failed to run local control plane: unable to start control plane itself: failed to start the controlplane. retried 5 times: timeout waiting for process kube-apiserver to start successfully (it may have failed to start, or stopped unexpectedly before becoming ready)
Additional info:
After the openshift-install failure, we observe that the kube-apiserver attempts to find service IPv4, even though our environment exclusively supports IPv6:
$ cat ostest/.clusterapi_output/kube-apiserver.log I0109 13:17:36.402549 31041 options.go:228] external host was not specified, using fd01:0:0:3::97 E0109 13:17:36.403397 31041 run.go:72] "command failed" err="service IP family \"10.0.0.0/24\" must match public address family \"fd01:0:0:3::97\"" I0109 13:17:38.351573 31096 options.go:228] external host was not specified, using fd01:0:0:3::97 E0109 13:17:38.352116 31096 run.go:72] "command failed" err="service IP family \"10.0.0.0/24\" must match public address family \"fd01:0:0:3::97\"" I0109 13:17:40.129451 31147 options.go:228] external host was not specified, using fd01:0:0:3::97 E0109 13:17:40.130026 31147 run.go:72] "command failed" err="service IP family \"10.0.0.0/24\" must match public address family \"fd01:0:0:3::97\"" I0109 13:17:41.517490 31203 options.go:228] external host was not specified, using fd01:0:0:3::97 E0109 13:17:41.518118 31203 run.go:72] "command failed" err="service IP family \"10.0.0.0/24\" must match public address family \"fd01:0:0:3::97\"" I0109 13:17:43.750048 31258 options.go:228] external host was not specified, using fd01:0:0:3::97 E0109 13:17:43.750649 31258 run.go:72] "command failed" err="service IP family \"10.0.0.0/24\" must match public address family \"fd01:0:0:3::97\""
$ ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0@if174: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 0a:58:19:b4:10:b3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fd01:0:0:3::97/64 scope global valid_lft forever preferred_lft forever inet6 fe80::858:19ff:feb4:10b3/64 scope link valid_lft forever preferred_lft forever
Component Readiness has found a potential regression in the following test:
[sig-instrumentation] Prometheus [apigroup:image.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
Significant regression detected.
Fishers Exact probability of a regression: 99.95%.
Test pass rate dropped from 99.06% to 93.75%.
Sample (being evaluated) Release: 4.18
Start Time: 2025-01-06T00:00:00Z
End Time: 2025-01-13T16:00:00Z
Success Rate: 93.75%
Successes: 45
Failures: 3
Flakes: 0
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 99.06%
Successes: 210
Failures: 2
Flakes: 0
View the test details report for additional context.
From the test details link, two of the three referenced failures are as follows:
[ { "metric": { "__name__": "ALERTS", "alertname": "OperatorHubSourceError", "alertstate": "firing", "container": "catalog-operator", "endpoint": "https-metrics", "exported_namespace": "openshift-marketplace", "instance": "[fd01:0:0:1::1a]:8443", "job": "catalog-operator-metrics", "name": "community-operators", "namespace": "openshift-operator-lifecycle-manager", "pod": "catalog-operator-6c446dcbbb-sxvjz", "prometheus": "openshift-monitoring/k8s", "service": "catalog-operator-metrics", "severity": "warning" }, "value": [ 1736751753.045, "1" ] } ]
This looks to always happen sparodically in CI lately: https://search.dptools.openshift.org/?search=OperatorHubSourceError&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Though overall it looks quite rare.
What is happening to cause these alerts to fire?
At this moment, it's a regression for 4.18 and thus a release blocker. I suspect it will clear naturally, but it might be a good opportunity to look for a reason why. Could use some input from OLM on what exactly is happening in the runs such as these two:
Description of problem:
2024/10/08 12:50:50 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/10/08 12:50:50 [INFO] : 👋 Hello, welcome to oc-mirror 2024/10/08 12:50:50 [INFO] : ⚙️ setting up the environment for you... 2024/10/08 12:50:50 [INFO] : 🔀 workflow mode: diskToMirror 2024/10/08 12:52:19 [INFO] : 🕵️ going to discover the necessary images... 2024/10/08 12:52:19 [INFO] : 🔍 collecting release images... 2024/10/08 12:52:19 [ERROR] : [ReleaseImageCollector] open /home/fedora/test-oc-mirror/hold-release/working-dir/release-images/ocp-release/4.14.20-x86_64/release-manifests/image-references: no such file or directory 2024/10/08 12:52:19 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/10/08 12:52:19 [ERROR] : [ReleaseImageCollector] open /home/fedora/test-oc-mirror/hold-release/working-dir/release-images/ocp-release/4.14.20-x86_64/release-manifests/image-references: no such file or directory
Version-Release number of selected component (if applicable):
[fedora@preserve-fedora-yinzhou test]$ ./oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.17.0-202409120935.p0.gc912303.assembly.stream.el9-c912303", GitCommit:"c9123030d5df99847cf3779856d90ff83cf64dcb", GitTreeState:"clean", BuildDate:"2024-09-12T09:57:57Z", GoVersion:"go1.22.5 (Red Hat 1.22.5-1.module+el8.10.0+22070+9237f38b) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. Install 4.17 cluster and 4.17 oc-mirror 2. Now use the ImageSetConfig.yaml below and perform mirror2disk using the command below [root@bastion-dsal oc-mirror]# cat imageset.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: channels: - name: stable-4.14 minVersion: 4.14.20 maxVersion: 4.14.20 shortestPath: true graph: true oc-mirror -c /tmp/imagesetconfig.yaml file:///home/fedora/test-oc-mirror/release-images --v2 3. Now perform disk2mirror using the command below oc-mirror -c /tm/imagesetconfig.yaml --from file:///home/fedora/test-oc-mirror/release-images --v2 --dry-run
Actual results:
When performing disk2mirror errors are seen as below [fedora@preserve-fedora-yinzhou test]$ ./oc-mirror -c /tmp/imageset.yaml --from file:///home/fedora/test-oc-mirror/release-images docker://localhost:5000 --v2 --dest-tls-verify=false --dry-run 2024/10/08 12:50:50 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/10/08 12:50:50 [INFO] : 👋 Hello, welcome to oc-mirror 2024/10/08 12:50:50 [INFO] : ⚙️ setting up the environment for you... 2024/10/08 12:50:50 [INFO] : 🔀 workflow mode: diskToMirror 2024/10/08 12:52:19 [INFO] : 🕵️ going to discover the necessary images... 2024/10/08 12:52:19 [INFO] : 🔍 collecting release images... 2024/10/08 12:52:19 [ERROR] : [ReleaseImageCollector] open /home/fedora/test-oc-mirror/hold-release/working-dir/release-images/ocp-release/4.14.20-x86_64/release-manifests/image-references: no such file or directory 2024/10/08 12:52:19 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/10/08 12:52:19 [ERROR] : [ReleaseImageCollector] open /home/fedora/test-oc-mirror/hold-release/working-dir/release-images/ocp-release/4.14.20-x86_64/release-manifests/image-references: no such file or directory
Expected results:
No errors should be seen when performing disk2mirror
Additional info:
If not using nestedpaths for file i.e like file:///home/fedora/test-oc-mirror/release-images and just using file://test-oc-mirror things work fine and no error as above is seen.
Description of problem:
The data in the table column overlaps on the helm rollback page when the screen width is shrunk
Version-Release number of selected component (if applicable):
How reproducible:
Every time
Steps to Reproduce:
1. Open a quick start window 2. Create a helm chart and upgrade it 3. Now select the rollback option from the action menu or kebab menu of the helm chart
Actual results:
The Rollback page has a messed-up UI
https://drive.google.com/file/d/1YXz80YsR5pkRG4dQmqFxpTkzgWQnQWLe/view?usp=sharing
Expected results:
The UI should be similar to the build config page with quick start open
https://drive.google.com/file/d/1UYxdRdV2kGC1m-MjBifTNdsh8gtpYnaU/view?usp=sharing
Additional info:
Description of problem:
There is a new menu "UserDefinedNetworks" under "Networking", it shows 404 error on the page after nav to this menu.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-20-085127
How reproducible:
Always
Steps to Reproduce:
1.Go to Networking->UserDefinedNetworks page. 2. 3.
Actual results:
1. 404 error is shown on the page : 404: Page Not Found The server doesn't have a resource type "UserDefinedNetwork" in "k8s.ovn.org/v1". Try refreshing the page if it was recently added.
Expected results:
1. Should not show 404 error.
Additional info:
Description of problem:
A cluster with a default (empty) `configs.spec.samplesRegistry` field but with whitelist entries in `image.spec.registrySources.allowedRegistries` causes openshift-samples CO in degraded state.
Version-Release number of selected component (if applicable):
4.13.30, 4.13.32
How reproducible:
100%
Steps to Reproduce:
1. Add the whitelist entries in image.spec.registrySources.allowedRegistries: ~~~ oc get image.config/cluster -o yaml spec: registrySources: allowedRegistries: - registry.apps.example.com - quay.io - registry.redhat.io - image-registry.openshift-image-registry.svc:5000 - ghcr.io - quay.apps.example.com ~~~ 2. Delete the pod, so it recreates: ~~~ oc delete pod -l name=cluster-samples-operator -n openshift-cluster-samples-operator ~~~ 3. The openshift-samples go to degraded state: ~~~ # oc get co openshift-samples NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-samples 4.13.30 True True True 79m Samples installation in error at 4.13.30: &errors.errorString{s:"global openshift image configuration prevents the creation of imagestreams using the registry "} ~~~ 4. The configs.samples spec is empty: ~~~ # oc get configs.samples.operator.openshift.io cluster -o jsonpath='{.spec}{"\n"}' {"architectures":["x86_64"],"managementState":"Managed"} ~~~
Actual results:
The openshift-sample go to degraded state.
Expected results:
The openshift-sample should remain in healthy state.
Additional info:
We had a Bug (https://bugzilla.redhat.com/show_bug.cgi?id=2027745) earlier which was fixed in OCP 4.10.3 as per erratta (https://access.redhat.com/errata/RHSA-2022:0056). One of my customer faced this issue when they upgraded the cluster from 4.12 to 4.13.32. As a workaround updating the below lines under `image.config.spec` helped. ~~~ allowedRegistriesForImport - domainName: registry.redhat.io insecure: false ~~~~
Description of problem:
vmware-vsphere-csi-driver-operator goes in panic mode when the vcenter address is incorrect
Version-Release number of selected component (if applicable):
Vsphere IPI 4.17
How reproducible:
100%
$ oc logs vmware-vsphere-csi-driver-operator-74f65d4444-ljt78 -n openshift-cluster-csi-drivers E1007 09:22:39.958324 1 config_yaml.go:208] Unmarshal failed: yaml: unmarshal errors: line 1: cannot unmarshal !!seq into config.CommonConfigYAML I1007 09:22:39.958515 1 config.go:272] ReadConfig INI succeeded. INI-based cloud-config is deprecated and will be removed in 2.0. Please use YAML based cloud-config. I1007 09:22:39.959600 1 config.go:283] Config initialized W1007 09:22:39.959738 1 vspherecontroller.go:910] vCenter vcenter1.vmware.gsslab.pnq2.redhat.com is missing from vCenter map E1007 09:22:39.959815 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 731 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2cf0f80, 0x54f9210}) k8s.io/apimachinery@v0.30.3/pkg/util/runtime/runtime.go:75 +0x85 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0xc000b0f4e8, 0x1, 0xc001adc000?}) k8s.io/apimachinery@v0.30.3/pkg/util/runtime/runtime.go:49 +0x6b panic({0x2cf0f80?, 0x54f9210?}) runtime/panic.go:770 +0x132
Actual results:
The storage CO goes in panic: $ oc get co storage NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE storage 4.17.0 True False True 6d21h VSphereCSIDriverOperatorCRDegraded: VMwareVSphereControllerDegraded: panic caught:...
Expected results:
The vmware-vsphere-csi-driver-operator should not go into panic even if the configuration is missing or incorrect.
Additional info:
Missing RBAC causes an error when OVNK tries to annotate the network ID on the NADs. Regression noticed when testing coverage for secondary networks was added.
Description of problem:
v1alpha1 schema is still present in the v1 ConsolePlugin CRD and should be removed manually since the generator is re-adding it automatically.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
with the default install-config (not zones field),create manifests panic
Version-Release number of selected component (if applicable):
4.19
How reproducible:
Always
Steps to Reproduce:
1.openshift-install create install-config 2.enable CAPI in install-config.yaml featureSet: CustomNoUpgrade featureGates: ["ClusterAPIInstall=true"] 3. openshift-install create manifests
Actual results:
INFO Consuming Install Config from target directory panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3b93434]goroutine 1 [running]: github.com/openshift/installer/pkg/asset/manifests/ibmcloud.GenerateClusterAssets(0xc0017a3080, 0xc001999700, {0xc0006847e9, 0x31}) /go/src/github.com/openshift/installer/pkg/asset/manifests/ibmcloud/cluster.go:73 +0x754 github.com/openshift/installer/pkg/asset/manifests/clusterapi.(*Cluster).Generate(0x2718f500, {0x5?, 0x89d1e67?}, 0xc00135bd10) /go/src/github.com/openshift/installer/pkg/asset/manifests/clusterapi/cluster.go:142 +0x7ef github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000c12180, {0x224f8690, 0xc000ffbe00}, {0x224d4a90, 0x2718f500}, {0x0, 0x0}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6e2 github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000c12180, {0x224f8690?, 0xc000ffbe00?}, {0x224d4a90, 0x2718f500}, {0x27151c80, 0x6, 0x6}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc0006c6be0, {0x224f8690, 0xc000ffbe00}, {0x27151c80, 0x6, 0x6}) /go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b main.newCreateCmd.runTargetCmd.func3({0x7ffde172857f?, 0x1?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:307 +0x6a main.newCreateCmd.runTargetCmd.func4(0x2715d2c0, {0xc000a32a60?, 0x4?, 0x8986d1f?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:321 +0x102 github.com/spf13/cobra.(*Command).execute(0x2715d2c0, {0xc000a32a20, 0x2, 0x2}) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xa91 github.com/spf13/cobra.(*Command).ExecuteC(0xc0009cef08) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x390 main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
Expected results:
create manifest file succeed.
Additional info:
the default install-config.yaml without zones field: controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: {} replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 metadata: name: maxu-capi1 platform: ibmcloud: region: eu-gb
After adding the zones field, create manifests succeed.
controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: ibmcloud: zones: - eu-gb-1 - eu-gb-2 - eu-gb-3 replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: ibmcloud: zones: - eu-gb-1 - eu-gb-2 - eu-gb-3 replicas: 3 metadata: name: maxu-capi2 platform: ibmcloud: region: eu-gb
The cluster-baremetal-operator sets up a number of watches for resources using Owns() that have no effect because the Provisioning CR does not (and should not) own any resources of the given type or using EnqueueRequestForObject{}, which similarly has no effect because the resource name and namespace are different from that of the Provisioning CR.
The commit https://github.com/openshift/cluster-baremetal-operator/pull/351/commits/d4e709bbfbae6d316f2e76bec18b0e10a45ac93e should be reverted as it adds considerable complexity to no effect whatsoever.
The correct way to trigger a reconcile of the provisioning CR is using EnqueueRequestsFromMapFunc(watchOCPConfigPullSecret) (note that the map function watchOCPConfigPullSecret() is poorly named - it always returns the name/namespace of the Provisioning CR singleton, regardless of the input, which is what we want). We should replace the ClusterOperator, Proxy, and Machine watches with ones of this form.
See https://github.com/openshift/cluster-baremetal-operator/pull/423/files#r1628777876 and https://github.com/openshift/cluster-baremetal-operator/pull/351/commits/d4e709bbfbae6d316f2e76bec18b0e10a45ac93e#r1628776168 for commentary.
Description of problem:
Trying to setup a disconnected HCP cluster with self-managed image registry. After the cluster installed, all the imagestream failed to import images. With error: ``` Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client ``` The imagestream will talk to openshift-apiserver and get the image target there. After login to the hcp namespace, figured out that I cannot access any external network with https protocol.
Version-Release number of selected component (if applicable):
4.14.35
How reproducible:
always
Steps to Reproduce:
1. Install the hypershift hosted cluster with above setup 2. The cluster can be created successfully and all the pods on the cluster can be running with the expected images pulled 3. Check the internal image-registry 4. Check the openshift-apiserver pod from management cluster
Actual results:
All the imagestreams failed to sync from the remote registry. $ oc describe is cli -n openshift Name: cli Namespace: openshift Created: 6 days ago Labels: <none> Annotations: include.release.openshift.io/ibm-cloud-managed=true include.release.openshift.io/self-managed-high-availability=true openshift.io/image.dockerRepositoryCheck=2024-11-06T22:12:32Z Image Repository: image-registry.openshift-image-registry.svc:5000/openshift/cli Image Lookup: local=false Unique Images: 0 Tags: 1latest updates automatically from registry quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d ! error: Import failed (InternalError): Internal error occurred: [122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-1@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-2@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-3@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-4@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, 122610517469.dkr.ecr.us-west-2.amazonaws.com/ocp-mirror-5@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/": http: server gave HTTP response to HTTPS client, quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:49baeac68e90026799d0b62609e04adf285be5b83bdb5dbd372de2b14442be5d: Get "https://quay.io/v2/": http: server gave HTTP response to HTTPS client] Access the external network from the openshift-apiserver pod: sh-5.1$ curl --connect-timeout 5 https://quay.io/v2 curl: (28) Operation timed out after 5001 milliseconds with 0 out of 0 bytes received sh-5.1$ curl --connect-timeout 5 https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/ curl: (28) Operation timed out after 5001 milliseconds with 0 out of 0 bytes received sh-5.1$ env | grep -i http.*proxy HTTPS_PROXY=http://127.0.0.1:8090 HTTP_PROXY=http://127.0.0.1:8090
Expected results:
The openshift-apiserver should be able to talk to the remote https services.
Additional info:
It is working after set the registry to no_proxy sh-5.1$ NO_PROXY=122610517469.dkr.ecr.us-west-2.amazonaws.com curl --connect-timeout 5 https://122610517469.dkr.ecr.us-west-2.amazonaws.com/v2/ Not Authorized
Description of problem:
OWNERS file updated to include prabhakar and Moe as owners and reviewers
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
This is to fecilitate easy backport via automation
Description of problem:
The secret created with Basic authentication has an incorrect type, the type will show as Opaque, compare with the same behavior with OCP 4.18, the type should show as kubernetes.io/basic-auth
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-01-16-064700
How reproducible:
Always
Steps to Reproduce:
1. Navigate to Secrect page, and Create a Source secrect eg: /k8s/ns/default/secrets/~new/source 2. Make sure the Authentication type is selected as 'Basic authentication' 3. Create the secret, and check the secret type on Secret list and Secret details page
Actual results:
The secret created with Basic authentication, its type shown as Opaque which is incorrect
Expected results:
Compare with the same behavior on OCP 4.18, it should shown as kubernetes.io/basic-auth
Additional info:
Description of problem:
Our carry patch intended to retry retriable requests that fail due to leader change will retry any etcd error with code "Unavailable": https://github.com/openshift/kubernetes/blob/4b2db1ec33faa3ffc305e5ffa7376908cc955370/staging/src/k8s.io/apiserver/pkg/storage/etcd3/etcd3retry/retry_etcdclient.go#L135-L145, but this includes reasons like "timeout" and does not distinguish between writes and reads. So a "timeout" error on a writing request might be retried even though a "timeout" observed by a client does not indicate that the effect of the write has not been persisted.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Current output message for oc adm top pvc -n xxxx "error: no persistentvolumeclaims found in xxxx namespace" even though there exists persistent volumes which could mislead the end user.
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-02-11-161912
How reproducible:
Always
Steps to Reproduce:
1. Create sc with Immediate VOLUMEBINDINGMODE, pvc which consume sc 2. Check the pvc. oc get pvc -n testropatil there exists persistent volume claims/volume 3. Check the output message oc adm top pvc oc adm top pvc --insecure-skip-tls-verify=true -n testropatil error: no persistentvolumeclaims found in testropatil namespace. sc_pvc.yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: mysc parameters: encrypted: "true" type: gp3 provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: Immediate --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mypvc-fs namespace: testropatil spec: accessModes: - ReadWriteOnce volumeMode: Filesystem storageClassName: mysc resources: requests: storage: 1Gi oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE mypvc-fs Bound pvc-8257a331-c808-47c0-b127-53c817f090a7 1Gi RWO mysc <unset> 4s oc get pv | grep "testropatil" pvc-8257a331-c808-47c0-b127-53c817f090a7 1Gi RWO Delete Bound testropatil/mypvc-fs mysc <unset> 82s
Actual results:
error: no persistentvolumeclaims found in testropatil namespace
Expected results:
error: no persistentvolumeclaims found or mounted in xxxx namespace
Additional info:
Description of problem:
oc-mirror can't support mirror image with bundle
Version-Release number of selected component (if applicable):
oc-mirror version W1121 06:10:37.581138 159435 mirror.go:102] ⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202411191706.p0.g956fc31.assembly.stream.el9-956fc31", GitCommit:"956fc318cc67769aedb2db8c0c4672bf7ed9f909", GitTreeState:"clean", BuildDate:"2024-11-19T18:08:35Z", GoVersion:"go1.22.7 (Red Hat 1.22.7-1.el9_5) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. mirror the image with bundles : kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16 packages: - name: cluster-kube-descheduler-operator bundles: - name: clusterkubedescheduleroperator.v5.0.1 - name: clusterkubedescheduleroperator.4.13.0-202309181427 - catalog: registry.redhat.io/redhat/community-operator-index:v4.14 packages: - name: 3scale-community-operator bundles: - name: 3scale-community-operator.v0.11.0 oc-mirror -c /tmp/config-73420.yaml file://out73420 --v2
Actual results:
1. hit error : oc-mirror -c /tmp/ssss.yaml file:///home/cloud-user/outss --v22024/11/21 05:57:40 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/21 05:57:40 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/21 05:57:40 [INFO] : ⚙️ setting up the environment for you... 2024/11/21 05:57:40 [INFO] : 🔀 workflow mode: mirrorToDisk 2024/11/21 05:57:40 [INFO] : 🕵️ going to discover the necessary images... 2024/11/21 05:57:40 [INFO] : 🔍 collecting release images... 2024/11/21 05:57:40 [INFO] : 🔍 collecting operator images... ✓ (43s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.16 ⠼ (20s) Collecting catalog registry.redhat.io/redhat/community-operator-index:v4.14 ✗ (20s) Collecting catalog registry.redhat.io/redhat/community-operator-index:v4.14 2024/11/21 05:58:44 [ERROR] : filtering on the selected bundles leads to invalidating channel "threescale-2.11" for package "3scale-community-operator": cha ✗ (20s) Collecting catalog registry.redhat.io/redhat/community-operator-index:v4.14
Expected results:
1. no error
Additional info:
no such issue with older version : 4.18.0-ec3
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
every time
Steps to Reproduce:
1. Create the dashboard with a bar chart and sort query result asc. 2. 3.
Actual results:
bar goes outside of the border
Expected results:
bar should not goes outside of the border
Additional info:
screenshot: https://drive.google.com/file/d/1xPRgenpyCxvUuWcGiWzmw5kz51qKLHyI/view?usp=drive_link
Description of problem:
`sts:AssumeRole` is required by creating Shared-VPC [1], otherwise which will cause the error: level=fatal msg=failed to fetch Cluster Infrastructure Variables: failed to fetch dependency of "Cluster Infrastructure Variables": failed to generate asset "Platform Provisioning Check": aws.hostedZone: Invalid value: "Z01991651G3UXC4ZFDNDU": unable to retrieve hosted zone: could not get hosted zone: Z01991651G3UXC4ZFDNDU: AccessDenied: User: arn:aws:iam::301721915996:user/ci-op-1c2w7jv2-ef4fe-minimal-perm-installer is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::641733028092:role/ci-op-1c2w7jv2-ef4fe-shared-role level=fatal msg= status code: 403, request id: ab7160fa-ade9-4afe-aacd-782495dc9978 Installer exit with code 1 [1]https://docs.openshift.com/container-platform/4.17/installing/installing_aws/installing-aws-account.html
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-03-174639
How reproducible:
Always
Steps to Reproduce:
1. Create install-config for Shared-VPC cluster 2. Run openshift-install create permissions-policy 3. Create cluster by using the above installer-required policy.
Actual results:
See description
Expected results:
sts:AssumeRole is included in the policy file when Shared VPC is configured.
Additional info:
The configuration of Shared-VPC is like: platform: aws: hostedZone: hostedZoneRole:
Slack context: https://redhat-internal.slack.com/archives/C02CZNQHGN8/p1739880614072559?thread_ts=1739804170.104739&cid=C02CZNQHGN8
The MCD appears to be inconsistently emitting OSUpdateStarted events when the NodeDisruptionPolicy functionality is in-use, causing failures in the openshift origin test suite.
TL;DR: I suppose that a combination of resource constraints together with thousands of systemd services could trigger an underlying racy behavior in the MCO.
In the combination of rpm-ostree, the machineconfig operator and when we actually reboot nodes.
Here's the graph of how an rpm-ostree update works according to my understanding (read bottom to top):
Note: Red Hat viewers, please see https://drive.google.com/file/d/1zYirNlsbFsBRNRZ8MVLugdsTsMV6yCQR/view?usp=sharing for an ASCII version that wasn't defigured by Jira's horrible UI ...
ostree-finalize-staged-hold.service stopped ^ | ostree-finalize-staged-hold.service stopping ^ | Stopped ostree-finalize-staged.service <------------------------ | | Stopping ostree-finalize-staged.service --(runs)--> /usr/bin/ostree admin finalize-staged -----------------------------(reads)----------------------------| ^ | | | (on service stopping, e.g. reboot) | | | Finished ostree-finalize-staged.service(is a noop)[active (exited)] <-----------------(dependency met)------------------- | | | | | ostree-finalize-staged.service --(depends on, starts and runs)--> ostree-finalize-staged-hold.service starting --> ostree-finalize-staged-hold.service started | ^ | | | (triggers) | | | | ostree-finalize-staged.path | | | (monitors) | | | v | rpm-ostree start-daemon --(writes)--> /run/ostree/staged-deployment <----------------------------------------------------------------------------------------------------------------------| ^ | (starts and instructs via dbus) | rpm-ostree kargs
In the journal, on a healthy run from a lab of mine, this plays out as:
Feb 19 15:04:28 ocp-on-osp.workload.bos2.lab systemd[1]: Starting Hold /boot Open for OSTree Finalize Staged Deployment... # <--- comes from ostree-finalize-staged-hold.service, pulled in via dep relationship with ostree-finalize-staged.service Feb 19 15:04:28 ocp-on-osp.workload.bos2.lab systemd[1]: Started Hold /boot Open for OSTree Finalize Staged Deployment. # <--- remains running now Feb 19 15:04:28 ocp-on-osp.workload.bos2.lab systemd[1]: Finished OSTree Finalize Staged Deployment. # <--- is a noop, remains active (exited) - comes from ostree-finalize-staged.service (...) Feb 19 15:06:50 ocp-on-osp.workload.bos2.lab systemd[1]: Stopping OSTree Finalize Staged Deployment... # <--- stop logic starting now meaining ostree admin finalize-staged Feb 19 15:06:54 ocp-on-osp.workload.bos2.lab systemd[1]: ostree-finalize-staged.service: Deactivated successfully. Feb 19 15:06:54 ocp-on-osp.workload.bos2.lab systemd[1]: Stopped OSTree Finalize Staged Deployment. Feb 19 15:06:54 ocp-on-osp.workload.bos2.lab systemd[1]: Stopping Hold /boot Open for OSTree Finalize Staged Deployment... # <--- pulled in via dep, starts before, stops after Feb 19 15:06:54 ocp-on-osp.workload.bos2.lab systemd[1]: Stopped Hold /boot Open for OSTree Finalize Staged Deployment. # <--- same
In the attached support case, this logic fails with the following log messages:
Jan 08 00:28:44 worker0 systemd[1]: Started machine-config-daemon: Node will reboot into config rendered-worker-39e251cb69d9f430a38dd14b0d3bae3c. Jan 08 00:28:44 worker0 systemd[1]: Starting Hold /boot Open for OSTree Finalize Staged Deployment... Jan 08 00:28:44 worker0 root[15374]: machine-config-daemon[10214]: reboot successful Jan 08 00:28:44 worker0 systemd-logind[1796]: The system will reboot now! Jan 08 00:28:44 worker0 ovs-vswitchd[1947]: ovs|00075|connmgr|INFO|br-int<->unix#2: 2317 flow_mods in the 7 s starting 10 s ago (2111 adds, 206 deletes) Jan 08 00:28:44 worker0 kernel: ice 0000:6c:00.0 ens14f0: Setting MAC 36:4a:79:c2:61:e7 on VF 34. VF driver will be reinitialized Jan 08 00:28:44 worker0 kernel: iavf 0000:6c:05.2: Reset indication received from the PF Jan 08 00:28:44 worker0 kernel: iavf 0000:6c:05.2: Scheduling reset task Jan 08 00:28:44 worker0 kernel: iavf 0000:6c:05.2: Removing device Jan 08 00:28:44 worker0 systemd[1]: Started Hold /boot Open for OSTree Finalize Staged Deployment. Jan 08 00:28:44 worker0 systemd-logind[1796]: System is rebooting. Jan 08 00:28:44 worker0 systemd[1]: Requested transaction contradicts existing jobs: Transaction for ostree-finalize-staged.service/start is destructive (ostree-finalize-staged-hold.service has 'stop' job queued, but 'start' is included in transaction). Jan 08 00:28:44 worker0 systemd[1]: ostree-finalize-staged.path: Failed to queue unit startup job: Transaction for ostree-finalize-staged.service/start is destructive (ostree-finalize-staged-hold.service has 'stop' job queued, but 'start' is included in transaction). Jan 08 00:28:44 worker0 systemd[1]: ostree-finalize-staged.path: Failed with result 'resources'. # <------------- here (...) Jan 08 00:29:01 worker0 systemd[1]: Stopping Hold /boot Open for OSTree Finalize Staged Deployment... (...) Jan 08 00:29:01 worker0 systemd[1]: Stopped Hold /boot Open for OSTree Finalize Staged Deployment. (...)
Message ` Requested transaction contradicts existing jobs: Transaction for ostree-finalize-staged.service/start is destructive (ostree-finalize-staged-hold.service has 'stop' job queued, but 'start'
is included in transaction).` means that ostree-finalize-staged requires ostree-finalize-staged-hold to be running, but a reboot was triggered right after ostree-finalize-staged-hold completed, and before ostree-finalize-staged resumed its own start sequence again. At that point, the dependency relationship is never fulfilled, and ostree-finalize-staged can never start. However, rpm-ostree applies all changes in the ExecStop of ostree-finalize-staged, and that's why the changes are never applied.
Here's the sequence of what's happening when it's going wrong:
0) ostree-finalize-staged.path sees that /run/ostree/staged-deployment is created and it wants to start ostree-finalize-staged.service
1) ostree-finalize-staged.service wants to run, but requires to start ostree-finalize-staged-hold.service
2) ostree-finalize-staged-hold.service starting as requested by ostree-finalize-staged.service
3) reboot triggered
4) ostree-finalize-staged-hold.service started
5) ----> ASSUMPTION: reboot queues stop job for ostree-finalize-staged-hold.service
6) ostree-finalize-staged.service wants to run, but requires ostree-finalize-staged-hold.service to be started, but it is currently stopping
ostree-finalize-staged.service can never start
7) ostree-finalize-staged.path: Failed with result 'resources' (because what it wanted to start, ostree-finalize-staged.service, can't start)
Regardless of why the systemd hold service is delayed so much that the reboot affects it ...
A race can occur here because rpm-ostree kargs (or any rpm-ostree update operation) blocks only until the rpm-ostree daemon writes /run/ostree/staged-deployment (the changes are staged in this file).
rpm-ostree then returns.
Everything else happens asynchronously: ostree-finalize-staged.path detects that the file was created, it starts ostree-finalize-staged.service which in ExecStop runs /usr/bin/ostree admin finalize-staged.
This last command actually applies the staged changes.
The MCO daemon is actually blocking only on rpm-ostree kargs. It's not waiting for ostree-finalize-staged.service to be active.
Meaning that the MCO will reboot the node immediately after the rpm-ostree kargs changes were staged, but it is not waiting for the actual daemon that applies the changes to be active.
Here lies the potential for a race condition which - I believe - the customer is hitting due to a very specific node configuration.
Whereas I cannot reproduce the exact error message in the customer's journal, it's super easy to reproduce a failure by exploiting this async mechanism.
Spawn a cluster with cluster bot, e.g. `launch 4.16.30 aws,no-spot`, then run the following test:
#!/bin/bash TIMEOUT=900 echo "#####################################" echo "Creating MCP for a single worker node" echo "#####################################" first_node=$(oc get nodes -l node-role.kubernetes.io/worker= -o name | head -1) oc label "${first_node}" node-role.kubernetes.io/worker-test= cat <<EOF | oc apply -f - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: name: worker-test spec: machineConfigSelector: matchExpressions: - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-test]} nodeSelector: matchLabels: node-role.kubernetes.io/worker-test: "" EOF echo "#####################################" echo " Creating reproducer conditions " echo "#####################################" f=$(mktemp) cat <<EOF>$f [Service] ExecStartPre=/bin/bash -c "echo 'exec start pre'; /bin/sleep 15; echo 'exec start pre end'" EOF cat <<EOF | oc apply -f - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker-test name: 99-worker-test-smoketest-prep spec: baseOSExtensionsContainerImage: "" config: ignition: config: replace: verification: {} proxy: {} security: tls: {} timeouts: {} version: 3.2.0 passwd: {} storage: files: - contents: source: data:text/plain;charset=utf-8;base64,$(cat $f | base64 -w0) verification: {} group: {} mode: 600 path: /etc/systemd/system/ostree-finalize-staged-hold.service.d/override.conf EOF echo "Sleeping for a bit ..." sleep 60 echo "Waiting for MCP to be updated" oc wait --for=condition=Updated=true mcp/worker-test --timeout=${TIMEOUT}s echo "#####################################" echo " Updating kernel arguments " echo "#####################################" cat <<EOF | oc apply -f - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker-test name: 99-worker-test-kernelarg-smoketest spec: kernelArguments: - abc=def EOF echo "Sleeping for a bit ..." sleep 60 echo "Waiting for MCP to be updated" oc wait --for=condition=Updated=true mcp/worker-test --timeout=${TIMEOUT}s & oc wait --for=condition=Degraded=true mcp/worker-test --timeout=${TIMEOUT}s & wait -n echo "#####################################" echo " End of test " echo "#####################################" oc get mcp
Wnereas the above test delays the ostree-finalize-staged-hold.service by 15 seconds, and it does not reproduce the exact customer error message ...
It still manages to reproduce an issue where the rpm-ostree staged changes are not applied. The 15 seconds sleep is of course extreme.
But it demonstrates that any delay in starting the -hold service, for example, can lead to a race condition which makes the process fail.
With the above test, the MCP will go into degraded state with:
[akaris@workstation 04029338 (master)]$ oc get mcp worker-test -o yaml | grep abc -C4 type: Updating - lastTransitionTime: "2025-02-21T12:13:03Z" message: 'Node ip-10-0-107-203.us-west-2.compute.internal is reporting: "unexpected on-disk state validating against rendered-worker-test-bf73a6e5a7892ac4151522bbf11a8f72: missing expected kernel arguments: [abc=def]"' reason: 1 nodes are reporting degraded status on sync status: "True" type: NodeDegraded - lastTransitionTime: "2025-02-21T12:13:03Z"
In the journal:
Feb 21 12:09:11 ip-10-0-107-203 root[8530]: machine-config-daemon[2561]: Running rpm-ostree [kargs --delete=systemd.unified_cgroup_hierarchy=1 --delete=cgroup_no_v1="all" --delete=psi=0 --append=systemd.unified_cgroup_hierarchy=1 --append=cgroup_no_v1="all" --append=psi=0 --append=abc=def] Feb 21 12:09:11 ip-10-0-107-203 rpm-ostree[6705]: client(id:machine-config-operator dbus:1.254 unit:crio-958faf4c29a580eace124aeb7cf8ecffe66b326216226daee367e207e3cb67d3.scope uid:0) added; new total=1 Feb 21 12:09:11 ip-10-0-107-203 rpm-ostree[6705]: Loaded sysroot Feb 21 12:09:11 ip-10-0-107-203 rpm-ostree[6705]: Locked sysroot Feb 21 12:09:11 ip-10-0-107-203 rpm-ostree[6705]: Initiated txn KernelArgs for client(id:machine-config-operator dbus:1.254 unit:crio-958faf4c29a580eace124aeb7cf8ecffe66b326216226daee367e207e3cb67d3.scope uid:0): /org/projectatomic/rpmostree1/rhcos Feb 21 12:09:11 ip-10-0-107-203 rpm-ostree[6705]: Process [pid: 8531 uid: 0 unit: crio-958faf4c29a580eace124aeb7cf8ecffe66b326216226daee367e207e3cb67d3.scope] connected to transaction progress Feb 21 12:09:11 ip-10-0-107-203 kubenswrapper[2259]: I0221 12:09:11.564171 2259 patch_prober.go:28] interesting pod/loki-promtail-z7ll9 container/promtail namespace/openshift-e2e-loki: Readiness probe status=failure output="Get \"http://10.129.2.3:3101/ready\": context deadline exceeded (Client.Timeout exceeded wh ile awaiting headers)" start-of-body= Feb 21 12:09:11 ip-10-0-107-203 kubenswrapper[2259]: I0221 12:09:11.564237 2259 prober.go:107] "Probe failed" probeType="Readiness" pod="openshift-e2e-loki/loki-promtail-z7ll9" podUID="c31172d3-6a1e-457a-8e66-c49db072c977" containerName="promtail" probeResult="failure" output="Get \"http://10.129.2.3:3101/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Feb 21 12:09:16 ip-10-0-107-203 systemd[1]: Starting Hold /boot Open for OSTree Finalize Staged Deployment... Feb 21 12:09:16 ip-10-0-107-203 bash[8598]: exec start pre Feb 21 12:09:16 ip-10-0-107-203 rpm-ostree[6705]: Created new deployment /ostree/deploy/rhcos/deploy/7c3910540178cc5e35b9c819a09cb58831b264384e9ff7a071a73a379dc32718.4 Feb 21 12:09:16 ip-10-0-107-203 rpm-ostree[6705]: sanitycheck(/usr/bin/true) successful Feb 21 12:09:16 ip-10-0-107-203 rpm-ostree[6705]: Txn KernelArgs on /org/projectatomic/rpmostree1/rhcos successful Feb 21 12:09:16 ip-10-0-107-203 rpm-ostree[6705]: Unlocked sysroot Feb 21 12:09:16 ip-10-0-107-203 rpm-ostree[6705]: Process [pid: 8531 uid: 0 unit: crio-958faf4c29a580eace124aeb7cf8ecffe66b326216226daee367e207e3cb67d3.scope] disconnected from transaction progress Feb 21 12:09:17 ip-10-0-107-203 rpm-ostree[6705]: client(id:machine-config-operator dbus:1.254 unit:crio-958faf4c29a580eace124aeb7cf8ecffe66b326216226daee367e207e3cb67d3.scope uid:0) vanished; remaining=0 Feb 21 12:09:17 ip-10-0-107-203 rpm-ostree[6705]: In idle state; will auto-exit in 63 seconds Feb 21 12:09:17 ip-10-0-107-203 root[8608]: machine-config-daemon[2561]: Rebooting node Feb 21 12:09:17 ip-10-0-107-203 root[8609]: machine-config-daemon[2561]: initiating reboot: Node will reboot into config rendered-worker-test-bf73a6e5a7892ac4151522bbf11a8f72 Feb 21 12:09:17 ip-10-0-107-203 systemd[1]: Started machine-config-daemon: Node will reboot into config rendered-worker-test-bf73a6e5a7892ac4151522bbf11a8f72. Feb 21 12:09:17 ip-10-0-107-203 root[8612]: machine-config-daemon[2561]: reboot successful Feb 21 12:09:17 ip-10-0-107-203 systemd-logind[913]: The system will reboot now! Feb 21 12:09:17 ip-10-0-107-203 systemd-logind[913]: System is rebooting.
And here are the MCO daemon logs:
2025-02-21T12:09:11.464509464+00:00 stderr F I0221 12:09:11.464485 2561 update.go:2641] Running rpm-ostree [kargs --delete=systemd.unified_cgroup_hierarchy=1 --delete=cgroup_no_v1="all" --delete=psi=0 --append=systemd.unified_cgroup_hierarchy=1 --append=cgroup_no_v1="all" --append=psi=0 --append=abc=def] 2025-02-21T12:09:11.466273927+00:00 stderr F I0221 12:09:11.466231 2561 update.go:2626] Running: rpm-ostree kargs --delete=systemd.unified_cgroup_hierarchy=1 --delete=cgroup_no_v1="all" --delete=psi=0 --append=systemd.unified_cgroup_hierarchy=1 --append=cgroup_no_v1="all" --append=psi=0 --append=abc=def 2025-02-21T12:09:16.367223780+00:00 stdout F Staging deployment...done 2025-02-21T12:09:17.184102954+00:00 stdout F Changes queued for next boot. Run "systemctl reboot" to start a reboot
So we can see here that the MCO daemon reboots the node even though rpm-ostree isn't ready, yet, because the corresponding daemon is not yet active.
Description of problem:
We have two EAP application server clusters and for each of them there is a service created. We have a route configured to the one of the services. When we update the route programmatically to lead to the second service/cluster the response shows it is still being attached to the same service.
Steps to Reproduce:
1. Create two separate clusters of the EAP servers
2. Create one service for the first cluster (hsc1) and one for the second one (hsc2)
3. Create a route for the first service (hsc1)
4. Start both of the clusters and assure the replication works
5. Send a request to the first cluster using the route URL - response should contain identification of the first cluster (hsc-1-xxx)
[2024-08-29 11:30:44,544] INFO - [ForkJoinPool-1-worker-1] 1st request responseString hsc-1-2-0c872b89-ef3e-48c6-8163-372e447e013d 1 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com [2024-08-29 11:30:44,654] INFO - [ForkJoinPool-1-worker-1] 2nd request responseString hsc-1-2-0c872b89-ef3e-48c6-8163-372e447e013d 2 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
6. update the route programatically to redirect to the second service (hsc2)
...
builder.editSpec().editTo().withName("hsc2").endTo().endSpec();
...
7. Send the request again using the same route - in the response there is the same identification of the first cluster
[2024-08-29 11:31:45,098] INFO - [ForkJoinPool-1-worker-1] responseString after route update hsc-1-2-0c872b89-ef3e-48c6-8163-372e447e013d 3 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
although the service was updated in the route yaml:
... kind: Service name: hsc2
When creating a new route hsc2 for a service hsc2 and using it for the third request we can see the second cluster was targetted correctly with his own separate replication working
[2024-08-29 13:43:13,679] INFO - [ForkJoinPool-1-worker-1] 1st request responseString hsc-1-2-00594ca9-f70c-45de-94b8-354a6e1fc293 1 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
[2024-08-29 13:43:13,790] INFO - [ForkJoinPool-1-worker-1] 2nd request responseString hsc-1-2-00594ca9-f70c-45de-94b8-354a6e1fc293 2 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
[2024-08-29 13:44:14,056] INFO - [ForkJoinPool-1-worker-1] responseString after second route for service hsc2 was used hsc-2-2-614582a9-3c71-4690-81d3-32a616ed8e54 1 with route hsc2-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
I also did a different attempt.
I Stopped the test in debug mode after the two requests were executed
[2024-08-30 14:23:43,101] INFO - [ForkJoinPool-1-worker-1] 1st request responseString hsc-1-1-489d4239-be4f-4d5e-9343-3211ae479d51 1 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com [2024-08-30 14:23:43,210] INFO - [ForkJoinPool-1-worker-1] 2nd request responseString hsc-1-1-489d4239-be4f-4d5e-9343-3211ae479d51 2 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
Then manually changed the route yaml to use the hsc2 service and send the request manually:
curl http://hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com/Counter
hsc-2-2-84fa1d7e-4045-4708-b89e-7d7f3cd48541 1
responded correctly with the second service/cluster.
Then resumed the test run in debug mode and sent the request programmatically
[2024-08-30 14:24:59,509] INFO - [ForkJoinPool-1-worker-1] responseString after route update hsc-1-1-489d4239-be4f-4d5e-9343-3211ae479d51 3 with route hsc-cihak2.apps.cpqe037-cnid.eapqe.psi.redhat.com
responded with the wrong first service/cluster.
Actual results: Route directs to the same service and EAP cluster
Expected results: After the update the route should direct to the second service and EAP cluster
Additional info:
This issue started to occur from OCP 4.16. Going through the 4.16 release notes and suggested route configuration didn't lead to any possible configuration chnages which should have been applied.
The code of the MultipleClustersTest.twoClustersTest where was this issue discovered is available here.
All the logs as well as services and route yamls are attached to the EAPQE jira.
The cloud provider health check runs every time through the HCP reconcile loop and result in ~800k calls to DescribeVpcEndpoints per day. This is 25% of our total AWS API call volume in our CI account and is contibuting to API throttling.
Description of problem:
The UserDefinedNetwork page lists UDN and CUDN objects. UDN is namespaces scope, CUDN is clsuter scope. The list view "namespace" column for CUDN objects presents "All Namespaces" which is confusing, making me think the CUDN selects all namespaces in the clsuter.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
100%
Steps to Reproduce:
1. Create CUDN, check "Namespace" column in the UDN page list view 2. 3.
Actual results:
UDN page list view "Namespace" column present "All Namespaces" for CUDN objects
Expected results:
I expect "Namespace" column to not present "All Namespaces" for CUDN objects because its confusing. I think its better for "Namespace" to remain empty for CUDNs objects.
Additional info:
The CUDN spec has namespace selector, controlling which namespaces the CUDN is affecting, I think this is the source for confusion. Maybe having the "Namespace" column preset "All Namespaces" for cluster-scope objects make sense, but in this particular case I find it confusing.
Description of problem:
In ASH arm template 06_workers.json[1], there is an unused variable "identityName" defined, this is harmless, but little weird to be present in official upi installation doc[2], which might confuse user when installing UPI cluster on ASH. [1] https://github.com/openshift/installer/blob/master/upi/azurestack/06_workers.json#L52 [2] https://docs.openshift.com/container-platform/4.17/installing/installing_azure_stack_hub/upi/installing-azure-stack-hub-user-infra.html#installation-arm-worker_installing-azure-stack-hub-user-infra suggest to remove it from arm template.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Today, when source images are by digest only, oc-mirror applies a default tag:
This should be unified.
Description of problem:
Our carry patch intended to retry retriable requests that fail due to leader change will retry any etcd error with code "Unavailable": https://github.com/openshift/kubernetes/blob/4b2db1ec33faa3ffc305e5ffa7376908cc955370/staging/src/k8s.io/apiserver/pkg/storage/etcd3/etcd3retry/retry_etcdclient.go#L135-L145, but this includes reasons like "timeout" and does not distinguish between writes and reads. So a "timeout" error on a writing request might be retried even though a "timeout" observed by a client does not indicate that the effect of the write has not been persisted.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following test:
operator conditions kube-apiserver
Significant regression detected.
Fishers Exact probability of a regression: 99.96%.
Test pass rate dropped from 97.09% to 91.78%.
Sample (being evaluated) Release: 4.19
Start Time: 2025-02-18T00:00:00Z
End Time: 2025-02-25T12:00:00Z
Success Rate: 91.78%
Successes: 67
Failures: 6
Flakes: 0
Base (historical) Release: 4.18
Start Time: 2025-01-26T00:00:00Z
End Time: 2025-02-25T12:00:00Z
Success Rate: 97.09%
Successes: 334
Failures: 10
Flakes: 0
View the test details report for additional context.
The problem involved may exist in 4.18 and only be appearing in 4.19 because machine set operator jobs are lumped into a larger set, which has shrunk in 4.19. However, there appears to be a common test failure causing this which looks important to the functionality of the job and should be fixed, in addition to the need to get this cell red.
The test this always seems to fail on is:
E2E Suite: [It] ControlPlaneMachineSet Operator With an active ControlPlaneMachineSet and the instance type is changed should perform a rolling update [Periodic]
Our unit test runtime is slow. It seems to run anywhere from ~16-20 minutes locally. On CI it can take at least 30 minutes to run. Investigate whether or not any changes can be made to improve the unit test runtime.
Update HyperShift to use k8s 1.32
During review of ARO MiWi permissions, some permissions in the CCM CredentialsRequest for Azure having other permissions identified through a linked action that are missing.
A linked access check is an action performed by Azure Resource Manager during a incoming request. For example, when you issue a create operation to a network interface ( Microsoft.Network/networkInterfaces/write ) you specify a subnet in the payload. ARM parses the payload, sees you're setting a subnet property, and as a result requires the linked access check Microsoft.Network/virtualNetworks/subnets/join/action to the subnet resource specified in the network interface. If you update a resource but don't include the property in the payload, it will not perform the permission check.
The following permissions were identified as possibly needed in CCM CredsRequest as they are specified as linked action of one of CCM's existing permissions
Microsoft.Network/applicationGateways/backendAddressPools/join/action Microsoft.Network/applicationSecurityGroups/joinIpConfiguration/action Microsoft.Network/applicationSecurityGroups/joinNetworkSecurityRule/action Microsoft.Network/ddosProtectionPlans/join/action Microsoft.Network/gatewayLoadBalancerAliases/join/action Microsoft.Network/loadBalancers/backendAddressPools/join/action Microsoft.Network/loadBalancers/frontendIPConfigurations/join/action Microsoft.Network/loadBalancers/inboundNatRules/join/action Microsoft.Network/networkInterfaces/join/action Microsoft.Network/networkSecurityGroups/join/action Microsoft.Network/publicIPAddresses/join/action Microsoft.Network/publicIPPrefixes/join/action Microsoft.Network/virtualNetworks/subnets/join/action
Each permission needs to be validated as to whether it is needed by CCM through any of its code paths.
Description of problem:
In CAPI, we use a random machineNetwork instead of using the one passed in by the user.
Description of problem:
Bare Metal UPI cluster Nodes lose communication with other nodes and this affects the pod communication on these nodes as well. This issue can be fixed with an OVN rebuild on the nodes db that are hitting the issue but eventually the nodes will degrade again and lose communication again. Note despite an OVN Rebuild fixing the issue temporarily Host Networking is set to True so it's using the kernel routing table. **update: observed on Vsphere with routingViaHost: false, ipForwarding: global configuration as well.
Version-Release number of selected component (if applicable):
4.14.7, 4.14.30
How reproducible:
Can't reproduce locally but reproducible and repeatedly occurring in customer environment
Steps to Reproduce:
identify a host node who's pods can't be reached from other hosts in default namespaces ( tested via openshift-dns). observe curls to that peer pod consistently timeout. TCPdumps to target pod observe that packets are arriving and are acknowledged, but never route back to the client pod successfully. (SYN/ACK seen at pod network layer, not at geneve; so dropped before hitting geneve tunnel).
Actual results:
Nodes will repeatedly degrade and lose communication despite fixing the issue with a ovn db rebuild (db rebuild only provides hours/days of respite, no permanent resolve).
Expected results:
Nodes should not be losing communication and even if they did it should not happen repeatedly
Additional info:
What's been tried so far ======================== - Multiple OVN rebuilds on different nodes (works but node will eventually hit issue again) - Flushing the conntrack (Doesn't work) - Restarting nodes (doesn't work) Data gathered ============= - Tcpdump from all interfaces for dns-pods going to port 7777 (to segregate traffic) - ovnkube-trace - SOSreports of two nodes having communication issues before an OVN rebuild - SOSreports of two nodes having communication issues after an OVN rebuild - OVS trace dumps of br-int and br-ex ==== More data in nested comments below.
linking KCS: https://access.redhat.com/solutions/7091399
Description of problem:
ImageStream cannot import image tags when ImageTagMirrorSet is set to NeverContactSource. The same issue does not apply for pods
Version-Release number of selected component (if applicable):
4.15.35
Steps to Reproduce:
1. Create a disconnected cluster with no internet access 2. Create a "pull-through" image registry [1] 3. Create the following ImageTagMirrorSet and ImageDigestMirrorSet ~~~ apiVersion: config.openshift.io/v1 kind: ImageDigestMirrorSet metadata: name: image-mirrors spec: imageDigestMirrors: - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/docker-remote source: docker.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/registry.access.redhat.com source: registry.access.redhat.com - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/quay.io source: quay.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/registry.redhat.io source: registry.redhat.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/gcr.io source: gcr.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/ghcr.io source: ghcr.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/com.redhat.connect.registry source: registry.connect.redhat.com - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/nvcr.io source: nvcr.io --- apiVersion: config.openshift.io/v1 kind: ImageTagMirrorSet metadata: name: image-mirrors spec: imageTagMirrors: - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/docker-remote source: docker.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/registry.access.redhat.com source: registry.access.redhat.com - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/quay.io source: quay.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/registry.redhat.io source: registry.redhat.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/gcr.io source: gcr.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/ghcr.io source: ghcr.io - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/com.redhat.connect.registry source: registry.connect.redhat.com - mirrorSourcePolicy: NeverContactSource mirrors: - <local-registry-url>/nvcr.io source: nvcr.io ~~~ 4. Import an image [2] [1] https://docs.redhat.com/en/documentation/red_hat_quay/3.13/html/use_red_hat_quay/quay-as-cache-proxy [2] https://docs.openshift.com/container-platform/4.15/openshift_images/image-streams-manage.html#images-imagestream-import-images-image-streams
Actual results:
Unable to import images
Expected results:
Being able to import images
A similar issue is reported in OCPBUGS-17975
Description of problem:
The kube scheduler pod should use the /livez endpoint rather than /healthz for its liveness probe.
Version-Release number of selected component (if applicable):
How reproducible:
N/A
Steps to Reproduce:
N/A
Actual results:
livenessProbe: failureThreshold: 5 httpGet: path: /healthz
Expected results:
livenessProbe: failureThreshold: 5 httpGet: path: /livez
Additional info:
Description of problem:
[must-gather] should collect the 3rd driver operator clustercsidriver resources
Version-Release number of selected component (if applicable):
Client Version: 4.18.0-ec.3 Kustomize Version: v5.4.2 Server Version: 4.18.0-0.nightly-2024-11-05-163516
How reproducible:
Always
Steps to Reproduce:
1. Install an Openshift cluster on azure. 2. Deploy the smb csi driver operator and create the cluster csidriver. 3. Use oc adm must-gather --dest-dir=./gather-test command gather the cluster info.
Actual results:
In step 3 the gathered data does not contain the clustercsidriver smb.csi.k8s.io object $ wangpenghao@pewang-mac ~ omc get clustercsidriver NAME AGE disk.csi.azure.com 3h file.csi.azure.com 3h wangpenghao@pewang-mac ~ oc get clustercsidriver NAME AGE disk.csi.azure.com 4h45m efs.csi.aws.com 40m file.csi.azure.com 4h45m smb.csi.k8s.io 4h13m wangpenghao@pewang-mac ~ ls -l ~/gather-test/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-2a3aa11d261a312215bcba80827ab6c75527f44d1ebde54958e7b7798673787c/cluster-scoped-resources/operator.openshift.io/clustercsidrivers total 32 -rwxr-xr-x@ 1 wangpenghao staff 7191 Nov 6 13:55 disk.csi.azure.com.yaml -rwxr-xr-x@ 1 wangpenghao staff 7191 Nov 6 13:55 file.csi.azure.com.yaml
Expected results:
In step 3 the gathered data should contain the clustercsidriver smb.csi.k8s.io object
Additional info:
aws efs, gcp filestore also have the same issue
Description of problem:
The default of the default cert in the router, default_pub_keys.pem, uses SHA1 and fails to load if any of the DEFAULT_CERTIFICATE, DEFAULT_CERTIFICATE_PATH, or DEFAULT_CERTIFICATE_DIR are NOT specified on the router deployment. This isn't an active problem for our supported router scenarios because default_pub_keys.pem is never used since DEFAULT_CERTIFICATE_DIR is always specified. But it does impact E2E testing such as when we create router deployments with no default cert, which attempts to load default_pub_keys.pem, which HAProxy fails on now because it's SHA1. So, both a completeness fix, and a fix to help make E2E tests simpler in origin.
Version-Release number of selected component (if applicable):
4.16+
How reproducible:
100%
Steps to Reproduce:
1. openssl x509 -in ./images/router/haproxy/conf/default_pub_keys.pem -noout -text
Actual results:
... Signature Algorithm: sha1WithRSAEncryption ...
Expected results:
... Signature Algorithm: sha256WithRSAEncryption ...
Additional info:
Description of problem:
Some coderefs in console are broken and runtime errors will occur if these coderefs are loaded
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Always
Steps to Reproduce:
1. Import every codeRef used by console 2. Observe runtime errors if any 3.
Actual results:
There are errors
Expected results:
No errors
Additional info:
Since 4.8, cvo#431 has the CVO checking to see whether the requested image's version matches the requested version, and erroring out if they don't match. For example, asking a 4.17.12 cluster to move to a 4.17.13 pullspec but claiming it will have a 4.17.99 version:
oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/desiredUpdate", "value": {"image": "quay.io/openshift-release-dev/ocp-release@sha256:82aa2a914d4cd964deda28b99049abbd1415f96c0929667b0499dd968864a8dd", "version": "4.17.99"}}]'
fails with ReleaseAccepted=False:
$ oc adm upgrade Cluster version is 4.17.12 ReleaseAccepted=False Reason: VerifyPayloadVersion Message: Verifying payload failed version="4.17.99" image="quay.io/openshift-release-dev/ocp-release@sha256:82aa2a914d4cd964deda28b99049abbd1415f96c0929667b0499dd968864a8dd" failure=release image version 4.17.13 does not match the expected upstream version 4.17.99 Upstream: https://api.integration.openshift.com/api/upgrades_info/graph Channel: candidate-4.17 (available channels: candidate-4.17, candidate-4.18, fast-4.17) Recommended updates: VERSION IMAGE 4.17.13 quay.io/openshift-release-dev/ocp-release@sha256:82aa2a914d4cd964deda28b99049abbd1415f96c0929667b0499dd968864a8dd
API godocs should be updated to accurately explain this behavior.
The current CVO behavior dates back to 4.8, cvo#431. The current (and incorrect) API godocs date back to 4.13, api#1339.
Every time.
1. View ClusterVersion API docs, e.g. 4.17's.
See strings like "version is ignored if image is specified".
Have the actual cluster-version operator behavior accurately described.
cvo#431 landing in 4.8:
cvo$ git diff origin/release-4.7..origin/release-4.8 -- pkg/cvo/sync_worker.go | grep -2 FIXME err = fmt.Errorf("release image version %s does not match the expected upstream version %s", payloadUpdate.Release.Version, work.Desired.Version) w.eventRecorder.Eventf(cvoObjectRef, corev1.EventTypeWarning, "VerifyPayloadVersionFailed", "verifying payload failed version=%q image=%q failure=%v", work.Desired.Version, work.Desired.Image, err) - /* FIXME: Ignore for now. I will make this fatal in a follow-up pivot reporter.Report(SyncWorkerStatus{ Generation: work.Generation,
api#1339 landing in 4.13:
api$ git diff origin/release-4.12..origin/release-4.13 -- config/v1/types_cluster_version.go | grep 'version is ignored if image is specified' + // version is ignored if image is specified and required if
Description of problem:
CUDN cannot be created thorugh the UI if there is no existing UDN
Version-Release number of selected component (if applicable):
4.18 rc.6
How reproducible:
Always
Steps to Reproduce:
1. Go to a fresh cluster without any existing UDNs 2. Go to UserDefinedNetworks menu
Actual results:
It only allows the user to create UDN
Expected results:
It should allow both UDN and CUDN
Additional info:
Description of problem:
console-demo-plugin adds a `Demo Plugin` tab for project resource, every time we click on `Demo Plugin` tab, it will append /demo-plugin suffix as part of URL path, in this situation visiting any tab will just return empty page since that's invalid URL
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-02-11-161912
How reproducible:
Always
Steps to Reproduce:
1. deploy console-demo-plugin manifests and enable it $ oc apply -f https://raw.githubusercontent.com/openshift/console/refs/heads/master/dynamic-demo-plugin/oc-manifest.yaml 2. Navigate to any project details page, such as openshift-console 3. switch between tabs including 'Demo Plugin' tab
Actual results:
3. After we click on 'Demo Plugin', nav through any project tab will just return empty page
Expected results:
3. return correct page
Additional info:
Description of problem:
On azure(or vsphere) TP cluster upgrade failed from 4.15.0-rc.5-> 4.15.0-rc.7 or 4.15.0-rc.4-> 4.15.0-rc.5, stuck in cluster-api. Seems this only happened on platforms don't support capi, this couldn't be reproduced on aws and gcp, .
Version-Release number of selected component (if applicable):
4.15.0-rc.5-> 4.15.0-rc.7 or 4.15.0-rc.4-> 4.15.0-rc.5
How reproducible:
always
Steps to Reproduce:
1.Build a TP cluster 4.15.0-rc.5 on azure(or vsphere) 2.Upgrade to 4.15.0-rc.7 3.
Actual results:
Upgrade stuck in cluster-api. must-gather: https://drive.google.com/file/d/12ykhEVZvqY_0eNdLwJOWFSxTSdQQrm_y/view?usp=sharing $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.15.0-rc.5 True True 82m Working towards 4.15.0-rc.7: 257 of 929 done (27% complete), waiting on cluster-api I0222 04:53:18.733907 1 sync_worker.go:1134] Update error 198 of 929: ClusterOperatorUpdating Cluster operator cluster-api is updating versions (*errors.errorString: cluster operator cluster-api is available and not degraded but has not finished updating to target version) E0222 04:53:18.733944 1 sync_worker.go:638] unable to synchronize image (waiting 2m44.892272217s): Cluster operator cluster-api is updating versions $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.15.0-rc.5 True False False 99m baremetal 4.15.0-rc.5 True False False 123m cloud-controller-manager 4.15.0-rc.7 True False False 128m cloud-credential 4.15.0-rc.5 True False False 135m cluster-api 4.15.0-rc.5 True False False 124m cluster-autoscaler 4.15.0-rc.5 True False False 123m config-operator 4.15.0-rc.7 True False False 124m console 4.15.0-rc.5 True False False 101m control-plane-machine-set 4.15.0-rc.7 True False False 113m csi-snapshot-controller 4.15.0-rc.5 True False False 112m dns 4.15.0-rc.5 True False False 115m etcd 4.15.0-rc.7 True False False 122m image-registry 4.15.0-rc.5 True False False 107m ingress 4.15.0-rc.5 True False False 106m insights 4.15.0-rc.5 True False False 118m kube-apiserver 4.15.0-rc.7 True False False 108m kube-controller-manager 4.15.0-rc.7 True False False 121m kube-scheduler 4.15.0-rc.7 True False False 120m kube-storage-version-migrator 4.15.0-rc.5 True False False 115m machine-api 4.15.0-rc.7 True False False 111m machine-approver 4.15.0-rc.5 True False False 124m machine-config 4.15.0-rc.5 True False False 121m marketplace 4.15.0-rc.5 True False False 123m monitoring 4.15.0-rc.5 True False False 106m network 4.15.0-rc.5 True False False 126m node-tuning 4.15.0-rc.5 True False False 112m olm 4.15.0-rc.5 True False False 106m openshift-apiserver 4.15.0-rc.5 True False False 115m openshift-controller-manager 4.15.0-rc.5 True False False 115m openshift-samples 4.15.0-rc.5 True False False 111m operator-lifecycle-manager 4.15.0-rc.5 True False False 123m operator-lifecycle-manager-catalog 4.15.0-rc.5 True False False 123m operator-lifecycle-manager-packageserver 4.15.0-rc.5 True False False 112m platform-operators-aggregated 4.15.0-rc.5 True False False 73m service-ca 4.15.0-rc.5 True False False 124m storage 4.15.0-rc.5 True False False 107m
Expected results:
Upgrade is successful
Additional info:
upgrade succeed from 4.15.0-rc.3-> 4.15.0-rc.4
Description of problem:
Following a 4.12.53 > 4.13.48 > 4.14.35 cluster upgrade path, a customer scaled up one of their cluster's MachineSets, at while points their spotted the following error: "ocp-lmwfc-infra-westeurope3b-8c48v: reconciler failed to Create machine: error launching instance: googleapi: Error 400: Invalid value for field 'resource.shieldedInstanceConfig': '{ "enableVtpm": true, "enableIntegrityMonitoring": true}'. Shielded VM Config can only be set when using a UEFI-compatible disk., invalid" At that point their noticed the following new parameters in their machineSet: `.spec.template.spec.providerSpec.value.shieldedInstanceConfig` The above seems to be related to commit 8bc61bd, introduced in RHOCP 4.13: - https://github.com/openshift/machine-api-provider-gcp/commit/8bc61bddf5cf01fce2462808afad3ab4e773c13e - https://issues.redhat.com/browse/OCPSTRAT-632
Version-Release number of selected component (if applicable):
4.14.35
Actual results:
As of now, shieldedInstanceConfig seems to be reconciled automátically into the MachineSet, even when the cluster may be using non UEFI-compatible disks
Expected results:
shieldedInstanceConfig to only be enabled when the cluster is using UEFI-compatible disks
Additional info:
- The customer workaround this, by disabling VTPM & IntegrityMonitoring in their MachineSet ShieldedInstanceConfig - The `compute-api.json` seems to suggest shieldedInstanceConfig is enabled by default (which breaks compatibility with non UEFI-compatible disks: $ curl -s https://raw.githubusercontent.com/openshift/machine-api-provider-gcp/refs/heads/release-4.13/vendor/google.golang.org/api/compute/v1/compute-api.json | sed -n '61048,61066p' "ShieldedInstanceConfig": { "description": "A set of Shielded Instance options.", "id": "ShieldedInstanceConfig", "properties": { "enableIntegrityMonitoring": { "description": "Defines whether the instance has integrity monitoring enabled. Enabled by default.", <<<---------- "type": "boolean" }, "enableSecureBoot": { "description": "Defines whether the instance has Secure Boot enabled. Disabled by default.", "type": "boolean" }, "enableVtpm": { "description": "Defines whether the instance has the vTPM enabled. Enabled by default.", <<<---------- "type": "boolean" } }, "type": "object" },
Description of problem:
While creating the provisioning resource on UPI cluster of 4.19, the metal3 pods fail to come up, because the init-container machine-os-images fails.
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-arm64-2025-02-14-044543
How reproducible:
always in arm64 cluster
Steps to Reproduce:
1. On a UPI cluster create provisioning resource 2. Check the pods being created on openshift-machine-api namespace
Actual results:
$ cat provisioning-upi.yaml apiVersion: metal3.io/v1alpha1 kind: Provisioning metadata: name: provisioning-configuration spec: provisioningNetwork: "Disabled" watchAllNamespaces: false $ oc get pods -n openshift-machine-api NAME READY STATUS RESTARTS AGE cluster-autoscaler-operator-fd96c97bf-psbt7 2/2 Running 1 (82m ago) 91m cluster-baremetal-operator-fb7cbbb65-dp7zb 2/2 Running 1 (82m ago) 92m control-plane-machine-set-operator-68596d754-ckfwq 1/1 Running 1 (82m ago) 91m ironic-proxy-j7whx 1/1 Running 0 28s ironic-proxy-nchz6 1/1 Running 0 28s ironic-proxy-wjgj6 1/1 Running 0 28s machine-api-operator-7dcd67759b-nqnv8 2/2 Running 0 92m metal3-5d77cd875-xpcsf 0/3 Init:CrashLoopBackOff 1 (6s ago) 34s metal3-baremetal-operator-9b465b848-j2ztt 1/1 Running 0 33s metal3-image-customization-767bd55db5-49pnn 0/1 Init:CrashLoopBackOff 1 (5s ago) 29s $ oc logs -f metal3-image-customization-767bd55db5-49pnn machine-os-images -n openshift-machine-api extracting PXE files... /shared/html/images/coreos-aarch64-initrd.img /shared/html/images/coreos-aarch64-rootfs.img /shared/html/images/coreos-aarch64-vmlinuz gzip: /shared/html/images/coreos-aarch64-vmlinuz.gz: not in gzip format
Expected results:
All 3 metal pods should be in Running state
QE has testing for this which detected OCPBUGS-43357 but we should make our own test and verify this in our e2e as well
The following test is failing more than expected:
Undiagnosed panic detected in pod
See the sippy test details for additional context.
Observed in 4.18-e2e-telco5g-cnftests/1863644602574049280 and 4.18-e2e-telco5g/1863677472923455488
Undiagnosed panic detected in pod { pods/openshift-kube-apiserver_kube-apiserver-cnfdu3-master-1_kube-apiserver.log.gz:E1202 22:12:02.806740 12 audit.go:84] "Observed a panic" panic="context deadline exceeded" panicGoValue="context.deadlineExceededError{}" stacktrace=<}
Undiagnosed panic detected in pod { pods/openshift-kube-apiserver_kube-apiserver-cnfdu11-master-2_kube-apiserver.log.gz:E1202 22:11:42.359004 14 timeout.go:121] "Observed a panic" panic=<}
Description of problem:
1 Client can not connect to the kube-apiserver via kubernetes svc, as the kubernetes svc is not in the cert SANs 2 The kube-apiserver-operator generate apiserver certs, and insert the kubernetes svc ip from the network cr status.ServiceNetwork 3 When the temporary control plane is down, and the network cr is not ready yet, Client will not connect to apiserver again
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. I have just met this for very rare conditions, especially when the machine performance is poor 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
After adding any directive to the ConsolePlugin CR a hard refresh is required for the changes to actually reflect, but we are not getting a refresh popover for this.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Enable feature gate (CSP feature is behind the FG in 4.18). 2. Add "DefaultSrc" directive to any ConsolePlugin CR. 3.
Actual results:
No refresh popover getting displayed, we need to manually refresh for the changes to get reflected.
Expected results:
No manual refresh. An automatic popover should be rendered.
Additional info:
ex: https://github.com/openshift/enhancements/blob/master/enhancements/console/dynamic-plugins.md#example
If the vCenter cluster has no esxi hosts importing the ova fails. Add a more sane error message
Description of problem:
when there is only one console plugin within operator, we don't display the console plugin name on CSV details page, only show a `Enabled` or `Disabled` button
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-29-211757
How reproducible:
Always
Steps to Reproduce:
1. goes to Operators -> OperatorHub page, find 'Node Health Check Operator' and install it, Enable or Disable the associated console plugin during operator installation process, either option is OK 2. Check CSV details page
Actual results:
2. We only show a 'Enabled' or 'Disabled' button in Console plugin section
Expected results:
2. We should also display plugin name no matter the plugin count, otherwise user has no idea which plugin we are enabling or disabling
Additional info:
The kind folks at Pure Storage tell us that if customers upgrade to 4.18 without the following patch issues will occur in CSI migration.
Kube 1.31 backport https://github.com/kubernetes/kubernetes/pull/129675
Master branch PR will full issue description and testing procedure
https://github.com/kubernetes/kubernetes/pull/129630
Description of problem:
While creating a project and move to network, select "Refer an existing ClusterUserDefinedNetwork", the title "Project name" is not correct, which should be "UserDefinedNetwork name".
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Open Questions
Go to Nodes > Node details > Logs and click one of the Selects above the log and then click outside the menu. Note the menu does not close but should.
Description of problem:
Audit-logs container was adjusted previously to handle SIGTERM using a trap and cleanup function. However due to behavior described here https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#container-v1-core the `$$` used to grab PID is not properly interpreted. This issue is to fix the script to handle this behavior correctly and make the cleanup be consistent with the changes to apply-bootstrap container for a similar ignored SIGTERM issue.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create a hypershift cluster with auditing enabled 2. Delete apiserver pods and observe the script does not correctly handle sigterm. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
"destroy cluster" doesn't delete the PVC disks which have the label "kubernetes-io-cluster-<infra-id>: owned"
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-multi-2024-11-27-162629
How reproducible:
Always
Steps to Reproduce:
1. include the step which sets the cluster default storageclass to the hyperdisk one before ipi-install (see my debug PR https://github.com/openshift/release/pull/59306) 2. "create cluster", and make sure it succeeds 3. "destroy cluster" Note: although we confirmed with issue with disk type "hyperdisk-balanced", we believe other disk types have the same issue.
Actual results:
The 2 PVC disks of hyperdisk-balanced type are not deleted during "destroy cluster", although the disks have the label "kubernetes-io-cluster-<infra-id>: owned".
Expected results:
The 2 PVC disks should be deleted during "destroy cluster", because they have the correct/expected labels according to which the uninstaller should be able to detect them.
Additional info:
FYI the PROW CI debug job: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/59306/rehearse-59306-periodic-ci-openshift-verification-tests-master-installer-rehearse-4.18-installer-rehearse-debug/1861958752689721344
Description of problem:
When setting up the "webhookTokenAuthenticator" the oauth configure "type" is set to "None". Then controller sets the console configmap with "authType=disabled". Which will cause that the console pod goes in the crash loop back due to the not allowed type: Error: validate.go:76] invalid flag: user-auth, error: value must be one of [oidc openshift], not disabled. This worked before on 4.14, stopped working on 4.15.
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.15
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
The console can't start, seems like it is not allowed to change the console.
Expected results:
Additional info:
Description of problem:
Backport https://github.com/prometheus/prometheus/pull/15723
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
We merged this ART PR which bumps base images. And then bumper [reverted the changes here|https://github.com/openshift/operator-framework-operator-controller/pull/88/files].
I still see the ART bump commit in main, but there is "Add OpenShift specific files" commit on top of it with older images. Actually now we have two "Add OpenShift specific files" commits in main:
And every UPSTREAM: <carry>-prefixed commit seems to be duplicated on top of synced changes.
Expected result:
Description of problem:
Currently check-patternfly-modules.sh checks them serially, which could be improved by checking them in parallel. Since yarn why does not write to anything, this should be easily parallelizable as there is no race condition with writing back to the yarn.lock
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
Always
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The StaticPodOperatorStatus API validations permit: - nodeStatuses[].currentRevision can be cleared and can decrease - more than one entry in nodeStatuses can have a targetRevision > 0 But both of these signal a bug in one or more of the static pod controllers that write to them.
Version-Release number of selected component (if applicable):
This has been the case ~forever but we are aware of bugs in 4.18+ that are resulting in controllers trying to make these invalid writes. We also have more expressive validation mechanisms today that make it possible to plug the holes.
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
The bug in 4.18+ that are resulting in some static pod node/installer controllers trying to make these invalid write requests.
Expected results:
To add some validation rules to help to see them
Additional info:
Description of problem:
The security job is failing on a new test added in October. I'm not sure we actually need to worry about it since we don't deal with user input so it may not be exploitable, but I think just bumping our logrus module would fix it so we should probably just do that.
Version-Release number of selected component (if applicable):
4.19, but possibly other branches were the security job is enabled too.
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Our e2e setup `go install` a few packages with the `@latest` tag. `go install` does not take `go.mod` into consideration, so in older branches we can pull package versions not compatible with the system Go version.
Version-Release number of selected component (if applicable):
All branches using Go < 1.23
How reproducible:
always on branch <= 4.18
Steps to Reproduce:
1. 2. 3.
Actual results:
./test/e2e/e2e-simple.sh ././bin/oc-mirror /go/src/github.com/openshift/oc-mirror/test/e2e/operator-test.17343 /go/src/github.com/openshift/oc-mirror go: downloading github.com/google/go-containerregistry v0.20.3 go: github.com/google/go-containerregistry/cmd/crane@latest: github.com/google/go-containerregistry@v0.20.3 requires go >= 1.23.0 (running go 1.22.9; GOTOOLCHAIN=local) /go/src/github.com/openshift/oc-mirror/test/e2e/lib/util.sh: line 17: PID_DISCONN: unbound variable https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_oc-mirror/1006/pull-ci-openshift-oc-mirror-release-4.18-e2e/1879913390239911936
Expected results:
The package version selected is compatible with the system Go version.
Additional info:
Description of problem:
While upgrading the Fusion operator, IBM team is facing the following error in the operator's subscription: error validating existing CRs against new CRD's schema for "fusionserviceinstances.service.isf.ibm.com": error validating service.isf.ibm.com/v1, Kind=FusionServiceInstance "ibm-spectrum-fusion-ns/odfmanager": updated validation is too restrictive: [].status.triggerCatSrcCreateStartTime: Invalid value: "number": status.triggerCatSrcCreateStartTime in body must be of type integer: "number" question here, "triggerCatSrcCreateStartTime" has been present in the operator for the past few releases and it's datatype (integer) hasn't changed in the latest release as well. There was one "FusionServiceInstance" CR present in the cluster when this issue was hit and the value of "triggerCatSrcCreateStartTime" field being "1726856593000774400".
Version-Release number of selected component (if applicable):
Its impacting between OCP 4.16.7 and OCP 4.16.14 versions
How reproducible:
Always
Steps to Reproduce:
1.Upgrade the fusion operator ocp version 4.16.7 to ocp 4.16.14 2. 3.
Actual results:
Upgrade fails with error in description
Expected results:
Upgrade should not be failed
Additional info:
Description of problem:
Customer is trying to install Self managed OCP cluster in aws. This customer use AWS VPC DHCPOptionSet. where it has a trailing dot (.) at the end of domain name in dhcpoptionset. due to this setting Master nodes hostname also has trailing dot & this cause failure in OpenShift installation.
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1.Please create a aws vpc with DHCPOptionSet, where DHCPoptionSet has trailing dot at the domain name. 2.Try installation of cluster with IPI.
Actual results:
Openshift Installer should allowed to create AWS Master nodes, where domain has trailing dot(.).
Expected results:
Additional info:
Description of problem:
Some bundles in the Catalog have been given the property in the FBC (and not in the bundle's CSV) which does not get propagated through to the helm chart annotations.
Version-Release number of selected component (if applicable):
How reproducible:
Install elasticsearch 5.8.13
Steps to Reproduce:
1. 2. 3.
Actual results:
cluster is upgradeable
Expected results:
cluster is not upgradeable
Additional info:
Description of problem:
With balance-slb and nmstate a node got stuck on reboot.
[root@master-1 core]# systemctl list-jobs JOB UNIT TYPE STATE 307 wait-for-br-ex-up.service start running 341 afterburn-checkin.service start waiting 187 multi-user.target start waiting 186 graphical.target start waiting 319 crio.service start waiting 292 kubelet.service start waiting 332 afterburn-firstboot-checkin.service start waiting 306 node-valid-hostname.service start waiting 293 kubelet-dependencies.target start waiting 321 systemd-update-utmp-runlevel.service start waiting systemctl status wait-for-br-ex-up.service Dec 10 20:11:39 master-1.ostest.test.metalkube.org systemd[1]: Starting Wait for br-ex up event from NetworkManager...
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-04-113014
How reproducible:
Sometimes
Steps to Reproduce:
1. create nmstate config
interfaces: - name: bond0 type: bond state: up copy-mac-from: eno2 ipv4: enabled: false link-aggregation: mode: balance-xor options: xmit_hash_policy: vlan+srcmac balance-slb: 1 port: - eno2 - eno3 - name: br-ex type: ovs-bridge state: up ipv4: enabled: false dhcp: false ipv6: enabled: false dhcp: false bridge: port: - name: bond0 - name: br-ex - name: br-ex type: ovs-interface state: up copy-mac-from: eno2 ipv4: enabled: true address: - ip: "192.168.111.111" prefix-length: 24 ipv6: enabled: false dhcp: false - name: eno1 type: interface state: up ipv4: enabled: false ipv6: enabled: false dns-resolver: config: server: - 192.168.111.1 routes: config: - destination: 0.0.0.0/0 next-hop-address: 192.168.111.1 next-hop-interface: br-ex
2. reboot
3.
Actual results:
systemctl status wait-for-br-ex-up.service Dec 10 20:11:39 master-1.ostest.test.metalkube.org systemd[1]: Starting Wait for br-ex up event from NetworkManager...
bond0 fails, network is in odd state
[root@master-1 core]# ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 90:e2:ba:ca:9f:28 brd ff:ff:ff:ff:ff:ff
altname enp181s0f0
inet6 fe80::92e2:baff:feca:9f28/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 30:d0:42:56:66:bb brd ff:ff:ff:ff:ff:ff
altname enp23s0f0
4: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 90:e2:ba:ca:9f:29 brd ff:ff:ff:ff:ff:ff
altname enp181s0f1
inet6 fe80::92e2:baff:feca:9f29/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 30:d0:42:56:66:bc brd ff:ff:ff:ff:ff:ff
altname enp23s0f1
6: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 30:d0:42:56:66:bd brd ff:ff:ff:ff:ff:ff
altname enp23s0f2
inet 192.168.111.34/24 brd 192.168.111.255 scope global dynamic noprefixroute eno3
valid_lft 3576sec preferred_lft 3576sec
inet6 fe80::32d0:42ff:fe56:66bd/64 scope link noprefixroute
valid_lft forever preferred_lft forever
7: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 30:d0:42:56:66:be brd ff:ff:ff:ff:ff:ff
altname enp23s0f3
8: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 56:92:14:97:ed:10 brd ff:ff:ff:ff:ff:ff
9: ovn-k8s-mp0: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000
link/ether ae:b9:9e:dc:17:d1 brd ff:ff:ff:ff:ff:ff
10: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether e6:68:4d:df:e0:bd brd ff:ff:ff:ff:ff:ff
inet6 fe80::e468:4dff:fedf:e0bd/64 scope link
valid_lft forever preferred_lft forever
11: br-int: <BROADCAST,MULTICAST> mtu 1400 qdisc noop state DOWN group default qlen 1000
link/ether 32:5b:1f:35:ce:f5 brd ff:ff:ff:ff:ff:ff
12: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue master ovs-system state DOWN group default qlen 1000
link/ether aa:c8:8c:e3:71:aa brd ff:ff:ff:ff:ff:ff
13: bond0.104@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master ovs-system state LOWERLAYERDOWN group default qlen 1000
link/ether aa:c8:8c:e3:71:aa brd ff:ff:ff:ff:ff:ff
14: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 30:d0:42:56:66:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.111.111/24 brd 192.168.111.255 scope global noprefixroute br-ex
valid_lft forever preferred_lft forever
Expected results:
System reboots correctly.
Additional info:
br-ex up/down re-generates the event
[root@master-1 core]# nmcli device down br-ex ; nmcli device up br-ex
Description of problem:
Customers need to be able to configure the DNS nameservers for the OpenStack subnet created by Hypershift (through Cluster API Provider for OpenStack). Without that, the default subnet wouldn't have DNS nameservers and resolution can fail in some environments.
Version-Release number of selected component (if applicable):
4.19, 4.18
How reproducible:
In default RHOSO 18 we don't have DNS forwarded to the DHCP agent so we need to set the DNS nameservers in every subnet that is created.
If you use a hyperlink in the value of operator.openshift.io/uninstall-message it is not parsed by the UI.
Description of problem:
For the PowerVS IPI CAPI installer, creating an OpenShift cluster in the Madrid zone will fail because it cannot import the RHCOS image. This is due to not using the correct bucket name.
Description of problem:
Using the same registry, oc and oc-mirror 4.16 versions but different installer versions in different directory structures, 4.16 returns the following error while generating the ISO file while 4.14 does not: DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:0e71cb61694473b40e8d95f530eaf250a62616debb98199f31b4034808687dae ERROR Release Image arch could not be found: command '[oc adm release info quay.io/openshift-release-dev/ocp-release@sha256:0e71cb61694473b40e8d95f530eaf250a62616debb98199f31b4034808687dae -o=go-template={{if and .metadata.metadata (index . "metadata" "metadata" "release.openshift.io/architecture")}}{{index . "metadata" "metadata" "release.openshift.io/architecture"}}{{else}}{{.config.architecture}}{{end}} --insecure=true --registry-config=/tmp/registry-config2248417039]' exited with non-zero exit code 1: ERROR error: unable to read image quay.io/openshift-release-dev/ocp-release@sha256:0e71cb61694473b40e8d95f530eaf250a62616debb98199f31b4034808687dae: Get "http://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ERROR WARNING Unable to validate the release image architecture, skipping validation This is valid for disconnected environments. This doesn't affect creating the image, it's more a cosmetic error. The ISO file is still being created and works as expected.
Version-Release number of selected component (if applicable):
openshift-install-linux-4.16
How reproducible:
./openshift-install agent create image --dir=./abi/ --log-level=debug
Steps to Reproduce:
1. Create 2 directory structures ocp4.14 and ocp4.16 2. Get from mirror.openshift.com oc client, oc-mirror for OCP 4.16 and openshift-install for both 4.14 and 4.16 3. Clone catalogs for both versions 4.14 and 4.16 to private registry 4. Use same install-config.yaml and agent-config.yaml for both "openshift-install agent create image" command with different versions
Actual results:
Error shows up for version 4.16
Expected results:
No error should show as in version 4.14
Additional info:
There are logs and config files in linked case. I'm attaching here the files and logs from my own reproduction.
Description of problem:
In OCL, while a new image is being built the MCP is reporting Updating=false.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-14-090045
How reproducible:
Always
Steps to Reproduce:
1. Enable techpreview 2. Create a MOSC
Actual results:
A builder pod is created, a machineosbuild resource is reporting "building" status but the MCP is reporting Updating=false Once the MCO starts applying the image to the nodes, then the MCP starts reporting Updating=true. For example, using an infra custom pool we get this scenario # The MOSB reports a building status $ oc get machineosbuild NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED mosc-infra-1adf9d0871a38cfcb8a0a3242fb78269 False True False False False # The builder pod is running $ oc get pods NAME READY STATUS RESTARTS AGE build-mosc-infra-1adf9d0871a38cfcb8a0a3242fb78269 2/2 Running 0 53s kube-rbac-proxy-crio-ip-10-0-17-74.us-east-2.compute.internal 1/1 Running 8 10h # But the infra pool is reporting Updating=false $ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE infra rendered-infra-8336591613a94060cb7d8a1a8319dc8e True False False 0 0 0 0 62m master rendered-master-ae24125af2c010fe672af84ce06153d9 True False False 3 3 3 0 10h worker rendered-worker-1c0c28ca4046a899927f4417754955c6 True False False 2 2 2 0 10h
Expected results:
The pool should report Updating=true when the a new MC is rendered and MCO starts building a new image.
Additional info:
Currently our automation for OCL relies on the MCP reporting Updating=true while the new image is being built so automation for OCL cannot be used until this issue is fixed.
the master branch is going to be renamed to 'main'
we need to update the automation for the sync to reduce breakage
Description of problem:
see https://issues.redhat.com/browse/OCPBUGS-44111
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Reported by customer IHAC, see https://redhat-internal.slack.com/archives/C02A3BM5DGS/p1736514939074049 "The timeout needs to be increased for Nutanix IPI installations using OpenShift versions >= 4.16, as image creation that takes more than 5 minutes will fail. OpenShift versions 4.11-4.15 work as expected with the OpenShift installer because the image creation timeout is set to a value greater than 5 minutes."
Version-Release number of selected component (if applicable):
How reproducible:
In some slow Prism-Central env. (slow network, etc.)
Steps to Reproduce:
In some slow Prism-Central env. (such as slow network), run the installer (4.16 and later) to create a Nutanix OCP cluster. The installation will fail with timeout when trying to upload the RHCOS image to PC.
Actual results:
The installation failed with timeout when uploading the RHCOS image to PC.
Expected results:
The installation successfully create the OCP cluster.
Additional info:
In some slow Prism-Central env. (such as slow network), run the installer (4.16 and later) to create a Nutanix OCP cluster. The installation will failed with timeout when trying to upload the RHCOS image to PC.
Description of problem:
https://github.com/openshift/console/commit/4b29cd8d77d4dcbf5cec1cd947f4877bd95bb684#diff-b7cc128ed1e2d7ad6eba033d9e76a4d8794bd1820ec5d132dd05c12bc993fa73L95-L109 removed `logoutOpenShift`, but it is still in use (see https://github.com/openshift/console/blob/7ba2bdcadf64b9e51157cb77b4b284cd6654504d/frontend/public/components/masthead-toolbar.jsx#L559). Cursory investigation shows `logout` alone will not successfully log out kubeadmin in OpenShift.
In order to fix security issue https://github.com/openshift/assisted-service/security/dependabot/94
time="2025-02-21T22:51:33Z" level=warning msg="unable to log into vCenter vcenter-1.ci.ibmc.devcluster.openshift.com, Post \"https://vcenter-1.ci.ibmc.devcluster.openshift.com/sdk\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
Description of problem:
Request to add a CLI option in hypershift create cluster to pass user configurable ETCD disk size. It allows user to pass ETCD Storage Class name but disk size, this request to accept storage size to override current default size of 8Gi. This disk size won't be sufficient for larger clusters, ROSA-HCP default is already set to 32Gi. It would be a good addition to make it configurable in CLI for self-hosted installs.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
hypershift create cluster azure --name hosted-cp --etcd-storage-class premium --etcd-storage-size 32Gi
Additional info:
Analyze the data from the new tests and determine what, if anything, we should do.
Description of problem:
The Power VS Machine API provider ignores the authentication endpoint override.
Testing OCP Console 4.17.9, the NetworkAttachmentDefinition Creation Button is misspelled as NetworkArrachmentDefinition.
I have attached a picture.
https://issues.redhat.com/secure/attachment/13328391/console_bug.png
When the CCO updates a CredentialsRequest's status, the current logs are not clear on what's changing:
time="2024-12-05T21:44:49Z" level=info msg="status has changed, updating" controller=credreq cr=openshift-cloud-credential-operator/aws-ebs-csi-driver-operator secret=openshift-cluster-csi-drivers/ebs-cloud-credentials
We should make it possible to get the CCO to log the diff it's trying to push, even if that requires bumping the operator's log level to debug. That would make it easier to understand hotloops like OCPBUGS-47505.
Description of problem:
oVirt support was dropped in 4.13, so the Machine API operator no longer needs to reference the oVirt image, nor does it need to know how to ship it
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
AlertmanagerConfig with missing options causes Alertmanager to crash
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
A cluster administrator has enabled monitoring for user-defined projects. CMO ~~~ config.yaml: | enableUserWorkload: true prometheusK8s: retention: 7d ~~~ A cluster administrator has enabled alert routing for user-defined projects. UWM cm / CMO cm ~~~ apiVersion: v1 kind: ConfigMap metadata: name: user-workload-monitoring-config namespace: openshift-user-workload-monitoring data: config.yaml: | alertmanager: enabled: true enableAlertmanagerConfig: true ~~~ verify existing config: ~~~ $ oc exec -n openshift-user-workload-monitoring alertmanager-user-workload-0 -- amtool config show --alertmanager.url http://localhost:9093 global: resolve_timeout: 5m http_config: follow_redirects: true smtp_hello: localhost smtp_require_tls: true pagerduty_url: https://events.pagerduty.com/v2/enqueue opsgenie_api_url: https://api.opsgenie.com/ wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/ telegram_api_url: https://api.telegram.org route: receiver: Default group_by: - namespace continue: false receivers: - name: Default templates: [] ~~~ create alertmanager config without options "smtp_from:" and "smtp_smarthost" ~~~ apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: example namespace: example-namespace spec: receivers: - emailConfigs: - to: some.username@example.com name: custom-rules1 route: matchers: - name: alertname receiver: custom-rules1 repeatInterval: 1m ~~~ check logs for alertmanager: the following error is seen. ~~~ ts=2023-09-05T12:07:33.449Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="no global SMTP smarthost set" ~~~
Actual results:
Alertmamnager fails to restart.
Expected results:
CRD should be pre validated.
Additional info:
Reproducible with and without user workload Alertmanager.
Description of problem:
Custom CPMS name fails on OpenStack. This was caught by the CI jobs:
Notes:
The problem that I recently noticed with the existing expression is that when we compute the overall burnrate from write and read requests, we take the ratio of successful read requests and we sum it to the one of write requests. But both of these ratios are calculated against their relevant request type, not the total number of requests. This is only correct when the proportion of write and read requests is equal.
For example, let's imagine a scenario where 40% of requests are write requests and their success during a disruption is only 50%. Whilst for read requests we have 90% of success.
apiserver_request:burnrate1h
{verb="write"}would be equal to 2/4 and apiserver_request:burnrate1h
{verb="read"} would be 1/6.
The sum of these as these by the alert today would be equal to 2/4+1/6=2/3 when in reality, the ratio of successful requests should be 2/10*1/10=3/10. So there is quite a huge difference today when we don't account for the total number of requests.
Description of problem:
When user tries to run oc-mirror delete command with `--force-cache-delete=true` after a (M2D + D2M) for catalog operators, it only delete the manifests on local cache, don't delete the blobs,which is not expected , from the help information , we should also delete the blobs for catalog operators : --force-cache-delete Used to force delete the local cache manifests and blobs
Version-Release number of selected component (if applicable):
oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202410011141.p0.g227a9c4.assembly.stream.el9-227a9c4", GitCommit:"227a9c499b6fd94e189a71776c83057149ee06c2", GitTreeState:"clean", BuildDate:"2024-10-01T20:07:43Z", GoVersion:"go1.22.5 (Red Hat 1.22.5-1.module+el8.10.0+22070+9237f38b) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. Using follow imagesetconfig to do mirror2disk+disk2mirror: kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: quay.io/openshifttest/hello-openshift@sha256:61b8f5e1a3b5dbd9e2c35fd448dc5106337d7a299873dd3a6f0cd8d4891ecc27 operators: - catalog: oci:///test/redhat-operator-index packages: - name: aws-load-balancer-operator - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: devworkspace-operator 2. Generate delete file : cat delete.yaml kind: DeleteImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 delete: additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: quay.io/openshifttest/hello-openshift@sha256:61b8f5e1a3b5dbd9e2c35fd448dc5106337d7a299873dd3a6f0cd8d4891ecc27 operators: - catalog: oci:///test/redhat-operator-index packages: - name: aws-load-balancer-operator - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: devworkspace-operator 3. execute the delete with --force-cache-delete=true `oc-mirror delete --v2 --delete-yaml-file out/working-dir/delete/delete-images.yaml --force-cache-delete=true docker://localhost:5000 --dest-tls-verify=false`
Actual results:
3. Check the local cache, didn't see any blobs deleted.
Expected results:
3. Not only delete the manifest for catalog operator , should also delete the blobs.
Additional info:
This error is resolved upon using --src-tls-verify=false with the oc-mirror delete --generate command More details in the slack thread here https://redhat-internal.slack.com/archives/C050P27C71S/p1722601331671649?thread_ts=1722597021.825099&cid=C050P27C71S
Also the logs show some logs from the registry, when --force-cache-delete is true
Description of problem:
Root issue reported in: https://issues.redhat.com/browse/RHEL-70334
Version-Release number of selected component (if applicable):
(4.18, 4.17, 4.16 as reported)
How reproducible:
Difficult. Relies on testing as mentioned in RHEL-70344
Additional info:
We're pending a downstream sync of the upstream repositories, and we'll want to account for this refactor eventually as it stands, so, 4.19 is likely a good place to bring it in, and see if it improves testing results in RHEL-70344
Description of problem:
The plugins name is shown as {{plugin}} on Operator details page
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-26-075648
How reproducible:
Always
Steps to Reproduce:
1. Prepare an operator has `console.openshift.io/plugins` annotation, or create a catalogsource with image quay.io/openshifttest/dynamic-plugin-oprs:latest annotations: alm-examples: 'xxx' console.openshift.io/plugins: '["prometheus-plugin1", "prometheus-plugin2"]' 2. Install operator, on operator installation page, choose Enable or Disable associated plugins 3. check Operator details page
Actual results:
2. on Operator installation page, associated plugin names are correctly shown 3. There is Console plugins section on Operator details page, in this section all plugins name is shown as {{plugin}}
Expected results:
3. plugin name associated with operator should be correctly displayed
Additional info:
Description of problem:
MCO failed to roll out imagepolicy configuration with imagepoliy objects for different namespaces
Version-Release number of selected component (if applicable):
How reproducible:
Create ImagePolicy for testnamespace and mynamespace
apiVersion: config.openshift.io/v1alpha1 kind: ImagePolicy metadata: name: p1 namespace: testnamespace spec: scopes: - example.com/global/image - example.com policy: rootOfTrust: policyType: PublicKey publicKey: keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFVW9GVW9ZQVJlS1hHeTU5eGU1U1FPazJhSjhvKwoyL1l6NVk4R2NOM3pGRTZWaUl2a0duSGhNbEFoWGFYL2JvME05UjYyczAvNnErK1Q3dXdORnVPZzhBPT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t signedIdentity: matchPolicy: ExactRepository exactRepository: repository: example.com/foo/bar apiVersion: config.openshift.io/v1alpha1 kind: ImagePolicy metadata: name: p2 namespace: mynamespace spec: scopes: - registry.namespacepolicy.com policy: rootOfTrust: policyType: PublicKey publicKey: keyData: Zm9vIGJhcg== signedIdentity: matchPolicy: ExactRepository exactRepository: repository: example.com/foo/bar
Steps to Reproduce:
1.create namespace test-namespace, the first imagepolicy 2.create the second namespace and imagepolicy
Actual results:
only the first imagepolicy got rolled out machineconfig controller log error: $ oc logs -f machine-config-controller-c997df58b-9dk8t I0108 23:05:09.141699 1 container_runtime_config_controller.go:499] Error syncing image config openshift-config: could not Create/Update MachineConfig: could not update namespace policy JSON from imagepolicy: error decoding policy json for namespaced policies: EOF
Expected results:
both /etc/crio/policies/mynamespace.json and /etc/crio/policies/testnamespace.json created
Additional info:
Currently the parallel-layers flag is not being shown in the --help of the mirror and delete command.
Description of problem:
OWNERS file updated to include prabhakar and Moe as owners and reviewers
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
This is to fecilitate easy backport via automation
Description of problem:
Our carry patch intended to retry retriable requests that fail due to leader change will retry any etcd error with code "Unavailable": https://github.com/openshift/kubernetes/blob/4b2db1ec33faa3ffc305e5ffa7376908cc955370/staging/src/k8s.io/apiserver/pkg/storage/etcd3/etcd3retry/retry_etcdclient.go#L135-L145, but this includes reasons like "timeout" and does not distinguish between writes and reads. So a "timeout" error on a writing request might be retried even though a "timeout" observed by a client does not indicate that the effect of the write has not been persisted.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When deploying a disconnected cluster, creating the iso by "openshifit-install agent create image" is failing (authentication required), when the release image resides in a secured local-registry. Actually the issue is this: openshift-install generates registry-config out of the install-config.yaml, and it's only the local regustry credentials (disconnected deploy), but it's not creating an icsp-file to get the image from local registry.
Version-Release number of selected component (if applicable):
How reproducible:
Run an agent-based iso image creation of a disconnected clutser. choose a version (nightly), where the image is in secured registry (such as registry.ci). it will fail on authentication required.
Steps to Reproduce:
1.openshift-install agant create image 2. 3.
Actual results:
failing on authentication required
Expected results:
iso to be created
Additional info:
Payload 4.19.0-0.nightly-2025-02-13-083804 showed aws regression failures likely from cloud-provider-aws#102 which came in but needs cluster-cloud-controller-manager-operator in an accepted payload to upgrade out of.
CBO-installed Ironic unconditionally has TLS, even though we don't do proper host validation just yet (see bug OCPBUGS-20412). Ironic in the installer does not use TLS (mostly for historical reasons). Now that OCPBUGS-36283 added a TLS certificate for virtual media, we can use the same for Ironic API. At least initially, it will involve disabling host validation for IPA.
ISSUE:
The cluster storage operator is in a degraded state because it is unable to find the UUID for the Windows node.
DESCRIPTIONS:
The customer has one Windows node in the OCP environment, the OCP environment is installed on vSphere. Storage CO is in a degraded state with the following error:
~~~
'VSphereCSIDriverOperatorCRDegraded: VMwareVSphereOperatorCheckDegraded:
unable to find VM win-xx-xx by UUID
vSphere CSI driver operator is trying to search UUID of that windows machine which should not be intended.
~~~
2024-09-27T15:44:27.836266729Z E0927 15:44:27.836234 1 check_error.go:147] vsphere driver install failed with unable to find VM win-ooiv8vljg7 by UUID , found existing driver
2024-09-27T15:44:27.860300261Z W0927 15:44:27.836249 1 vspherecontroller.go:499] Marking cluster as degraded: vcenter_api_error unable to find VM win--xx-xx by UUID
~~~
So, the operator pod should exclude the Windows node and should not go in a 'Degraded' state.
As a ARO HCP user, I would like MachineIdentityID to be removed from the Azure HyperShift API since this field is not needed for ARO HCP.
Description of problem:
clicking on 'create a Project' button on Getting Started page doesn't work
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-10-25-210451
How reproducible:
Always
Steps to Reproduce:
1. A new normal user login to OCP web console, user will be redirected to Getting Started page 2. try to create a project via 'create a Project' button in message "Select a Project to start adding to it or create a Project." 3.
Actual results:
click on 'create a Project' button doesn't open project creation modal
Expected results:
as it indicated 'create a Project' should open project creation modal
Additional info:
Description of problem:
Sometimes the ovs-configuration cannot be started with errors as below: Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + add_nm_conn br-ex type ovs-bridge conn.interface br-ex 802-3-ethernet.mtu 1500 connection.autoconnect-slaves 1 Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + nmcli c add save no con-name br-ex type ovs-bridge conn.interface br-ex 802-3-ethernet.mtu 1500 connection.autoconnect-slaves 1 connection.autoconnect no Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9781]: Connection 'br-ex' (eb9fdfa0-912f-4ee2-b6ac-a5040b290183) successfully added. Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + nmcli connection show ovs-port-phys0 Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + ovs-vsctl --timeout=30 --if-exists del-port br-ex ens1f0np0 Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + add_nm_conn ovs-port-phys0 type ovs-port conn.interface ens1f0np0 master br-ex connection.autoconnect-slaves 1 Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: + nmcli c add save no con-name ovs-port-phys0 type ovs-port conn.interface ens1f0np0 master br-ex connection.autoconnect-slaves 1 connection.autoconnect no Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9790]: Error: failed to modify connection.port-type: 'ovs-interface' not among [bond, bridge, ovs-bridge, ovs-port, team, vrf]. Sep 08 12:45:34 openshift-qe-024.lab.eng.rdu2.redhat.com configure-ovs.sh[9472]: ++ handle_exit However there is workaround is remove the existing `ovs-if-br-ex` by `nmcli connection delete ovs-if-br-ex` can fix this issue.
Version-Release number of selected component (if applicable):
4.17.0-0.nightly-2024-09-07-151850
How reproducible:
not always
Steps to Reproduce:
1. Create many bond interface by nmstate nncp 2. reboot worker 3.
Actual results:
ovs-configuration service cannot be started up
Expected results:
ovs-configuration service should be started without any issue
Additional info:
Not sure if these bond interface affected this issue. However there is workaround is remove the existing `ovs-if-br-ex` by `nmcli connection delete ovs-if-br-ex` can fix this issue. [root@openshift-qe-024 ~]# nmcli c NAME UUID TYPE DEVICE ens1f0np0 701f8b4e-819d-56aa-9dfb-16c00ea947a8 ethernet ens1f0np0 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet eno1 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens1f1np1 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens2f2 bond12 ba986131-f4d2-460c-b883-a1d6a9ddfdcb bond bond12 bond12.101 46bc3df0-e093-4096-a747-0e6717573f82 vlan bond12.101 bond12.102 93d68598-7453-4666-aff6-87edfcf2f372 vlan bond12.102 bond12.103 be6013e1-6b85-436f-8ce8-24655db0be17 vlan bond12.103 bond12.104 fabf9a76-3635-48d9-aace-db14ae2fd9c3 vlan bond12.104 bond12.105 0fab3700-ce50-4815-b329-35af8f830cb1 vlan bond12.105 bond12.106 68c20304-f3e9-4238-96d7-5bcce05b3827 vlan bond12.106 bond12.107 f1029614-2e6e-4e20-b9b6-79902dd12ac9 vlan bond12.107 bond12.108 27669b6f-e24d-4ac2-a8ba-35ca0b6c5b05 vlan bond12.108 bond12.109 d421e0bb-a441-4305-be23-d1964cb2bb46 vlan bond12.109 bond12.110 c453e70c-e460-4e80-971c-88fac4bd1d9e vlan bond12.110 bond12.111 2952a2c6-deb4-4982-8a4b-2a962c3dda96 vlan bond12.111 bond12.112 5efe4b2d-2834-4b0b-adb2-8caf153cef2d vlan bond12.112 bond12.113 2ec39bea-2704-4b8a-83fa-d48e1ef1c472 vlan bond12.113 bond12.114 8fc8ae53-cc8f-4412-be7d-a05fc3abdffe vlan bond12.114 bond12.115 58f9e047-fe4f-475d-928f-7dec74cf379f vlan bond12.115 bond12.116 d4d133cb-13cc-43f3-a636-0fbcb1d2b65d vlan bond12.116 bond12.117 3a2d10a1-3fd8-4839-9836-56eb6cab76a7 vlan bond12.117 bond12.118 8d1a22da-efa0-4a06-ab6d-6840aa5617ea vlan bond12.118 bond12.119 b8556371-eba8-43ba-9660-e181ec16f4d2 vlan bond12.119 bond12.120 989f770f-1528-438b-b696-eabcb5500826 vlan bond12.120 bond12.121 b4c651f6-18d7-47ce-b800-b8bbeb28ed60 vlan bond12.121 bond12.122 9a4c9ec2-e5e4-451a-908c-12d5031363c6 vlan bond12.122 bond12.123 aa346590-521a-40c0-8132-a4ef833de60c vlan bond12.123 bond12.124 c26297d6-d965-40e1-8133-a0d284240e46 vlan bond12.124 bond12.125 24040762-b6a0-46f7-a802-a86b74c25a1d vlan bond12.125 bond12.126 24df2984-9835-47c2-b971-b80d911ede8d vlan bond12.126 bond12.127 0cc62ca7-b79d-4d09-8ec3-b48501053e41 vlan bond12.127 bond12.128 bcf53331-84bd-400c-a95c-e7f1b846e689 vlan bond12.128 bond12.129 88631a53-452c-4dfe-bebe-0b736633d15a vlan bond12.129 bond12.130 d157ffb0-2f63-4844-9a16-66a035315a77 vlan bond12.130 bond12.131 a36f8fb2-97d6-4059-8802-ce60faffb04a vlan bond12.131 bond12.132 94aa7a8e-b483-430f-8cd1-a92561719954 vlan bond12.132 bond12.133 7b3a2b6e-72ad-4e0a-8f37-6ecb64d1488c vlan bond12.133 bond12.134 68b80892-414f-4372-8247-9276cea57e88 vlan bond12.134 bond12.135 08f4bdb2-469f-4ff7-9058-4ed84226a1dd vlan bond12.135 bond12.136 a2d13afa-ccac-4efe-b295-1f615f0d001b vlan bond12.136 bond12.137 487e29dc-6741-4406-acec-47e81bed30d4 vlan bond12.137 bond12.138 d6e2438f-2591-4a7a-8a56-6c435550c3ae vlan bond12.138 bond12.139 8a2e21c3-531b-417e-b747-07ca555909b7 vlan bond12.139 bond12.140 8e3c5d65-5098-48a5-80c4-778d41b24634 vlan bond12.140 bond12.141 7aaca678-27e1-4219-9410-956649313c52 vlan bond12.141 bond12.142 6765c730-3240-48c8-ba29-88113c703a88 vlan bond12.142 bond12.143 3e9cef84-4cb1-4f17-98eb-de9a13501453 vlan bond12.143 bond12.144 ebaa63ee-10be-483d-9096-43252757b7fa vlan bond12.144 bond12.145 1ba28e89-0578-4967-85d3-95c03677f036 vlan bond12.145 bond12.146 75ac1594-a761-4066-9ac9-a2f4cc853429 vlan bond12.146 bond12.147 b8c7e473-8179-49f7-9ea8-3494ce4a0244 vlan bond12.147 bond12.148 4c643923-8412-4550-b43c-cdb831dd28e9 vlan bond12.148 bond12.149 418fa841-24ba-4d6f-bc5a-37c8ffb25d45 vlan bond12.149 bond12.150 1eb8d1ce-256e-42f3-bacd-e7e5ac30bd9a vlan bond12.150 bond12.151 aaab839b-0fbc-4ba9-9371-c460172566a2 vlan bond12.151 bond12.152 de2559c4-255b-45ac-8602-968796e647a6 vlan bond12.152 bond12.153 52b5d827-c212-45f1-975d-c0e5456c19e9 vlan bond12.153 bond12.154 26fc0abd-bfe5-4f66-a3a5-fadefdadb9df vlan bond12.154 bond12.155 0677f4a8-9260-475c-93ca-e811a47d5780 vlan bond12.155 bond12.156 4b4039f4-1e7e-4427-bc3a-92fe37bec27e vlan bond12.156 bond12.157 38b7003e-a20c-4ef6-8767-e4fdfb7cd61b vlan bond12.157 bond12.158 7d073e1b-1cf7-4e49-9218-f96daf97150a vlan bond12.158 bond12.159 3d8c5222-e59c-45c9-acb6-1a6169e4eb6d vlan bond12.159 bond12.160 764bce7a-ec99-4f8b-9e39-d47056733c0c vlan bond12.160 bond12.161 63ee9626-2c17-4335-aa17-07a38fa820d8 vlan bond12.161 bond12.162 6f8298ff-4341-42a6-93a8-66876042ca16 vlan bond12.162 bond12.163 7bb90042-f592-49c6-a0c9-f4d2cf829674 vlan bond12.163 bond12.164 3fd8b04f-8bd0-4e8d-b597-4fd37877d466 vlan bond12.164 bond12.165 06268a05-4533-4bd2-abb8-14c80a6d0411 vlan bond12.165 bond12.166 4fa1f0c1-e55d-4298-bfb5-3602ad446e61 vlan bond12.166 bond12.167 494e1a43-deb2-4a69-90da-2602c03400fb vlan bond12.167 bond12.168 d2c034cd-d956-4d02-8b6e-075acfcd9288 vlan bond12.168 bond12.169 8e2467b7-80dd-45b6-becc-77cbc632f1f0 vlan bond12.169 bond12.170 3df788a3-1715-4a1c-9f5d-b51ffd3a5369 vlan bond12.170 dummy1 b4d7daa3-b112-4606-8b9c-cb99b936b2b9 dummy dummy1 dummy2 c99d8aa1-0627-47f3-ae57-f3f397adf0e8 dummy dummy2 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet enp138s0np0 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens2f3 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens4f2 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens4f3 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens8f0 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens8f1 Wired Connection b7361c63-fb2a-4a95-80f4-c669fd368bbf ethernet ens8f3 lo ae4bbedd-1a2e-4c97-adf7-4339cf8fb226 loopback lo ovs-if-br-ex 90af89d6-a3b0-4497-b6d0-7d2cc2d5098a ovs-interface --
Description of problem:
When an invalid packages are included in the ImageSetConfiguration and the operator catalog becomes without any valid package, d2m fails. The operator catalog image should be skipped in case no operators were found for that operator catalog.
With the following ImageSetConfiguration:
kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 packages: - name: netscaler-operator
./bin/oc-mirror -c ./alex-tests/alex-isc/pr-1093.yaml file://alex-tests/pr-1093 --v2
2. I removed the working-dir under the folder pr-1093 and the oc-mirror cache to simulate a d2m from scratch on a disconnected environment where I only have the tarball
rm -rf ~/.oc-mirror/ && rm -rf ./alex-tests/pr-1093/working-dir
3. d2m
./bin/oc-mirror -c ./alex-tests/alex-isc/pr-1093.yaml --from file://alex-tests/pr-1093 docker://localhost:6000 --dest-tls-verify=false --v2
Actual results:
./bin/oc-mirror -c ./alex-tests/alex-isc/pr-1093.yaml --from file://alex-tests/pr-1093 docker://localhost:6000 --dest-tls-verify=false --v2 2025/03/07 12:29:46 [INFO] : :wave: Hello, welcome to oc-mirror 2025/03/07 12:29:46 [INFO] : :gear: setting up the environment for you... 2025/03/07 12:29:46 [INFO] : :twisted_rightwards_arrows: workflow mode: diskToMirror 2025/03/07 12:30:11 [INFO] : 🕵 going to discover the necessary images... 2025/03/07 12:30:11 [INFO] : :mag: collecting release images... 2025/03/07 12:30:11 [INFO] : :mag: collecting operator images... 2025/03/07 12:30:11 [ERROR] : [OperatorImageCollector] stat .: no such file or directory ✗ () Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.17 2025/03/07 12:30:11 [INFO] : :wave: Goodbye, thank you for using oc-mirror 2025/03/07 12:30:11 [ERROR] : stat .: no such file or directory
Expected results:
m2d should should fail when there is no related images found for the specified catalog (invalid operator netscaler-operator for the ImageSetConfiguration used above)
Additional info:
Description of problem:
documentationBaseURL still points to 4.18
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-01-16-064700
How reproducible:
Always
Steps to Reproduce:
1. check documentationBaseURL on a 4.19 cluster $ oc get cm console-config -n openshift-console -o yaml | grep documentationon documentationBaseURL: https://access.redhat.com/documentation/en-us/openshift_container_platform/4.18/ 2. 3.
Actual results:
documentationBaseURL still links to 4.18
Expected results:
documentationBaseURL should link to 4.19
Additional info:
Description of problem:
When I use the UI to provision a primary UDN into a namespace, I get the following error (after indicating which project I want the UDN in, and which subnet it'll output the following error: """ Admission Webhook Warning UserDefinedNetwork primary-udn violates policy 299 - "unknown field \"spec.layer2.ipamLifecycle\" """
Version-Release number of selected component (if applicable):
4.99
How reproducible:
Always
Steps to Reproduce:
1. create a project 2. create a UDN in said project; define a subnet 3. watch error
Actual results:
The UDN is created, but the ipam.lifecycle attribute is *not* set to persistent, which for virtualization means you'll have a useless network.
Expected results:
The UDN must be created with ipam.lifecycle set to Persistent for the VMs to have stable IPs across live-migration and restart / stop / start.
Additional info:
Description of problem:
https://github.com/openshift/hypershift/blob/5d03cfa7e250e06cf408fd0b6c46d207ab915c1e/control-plane-operator/controllers/hostedcontrolplane/olm/catalogs.go#L126 We should check registry overrides here and retrive the images from the mirrored registry We currently dont check this which means it could prevent the cluster operators coming up if the registry.redhat.io/redhat registry is being overriden.
In addition to the above the releaseImageProvider we use in the HCCO and HCP rencociler doesnt use the registry overrides set at the HO level which could cause confusions for users that expect those overrides to be propagated to the HCP.
Version-Release number of selected component (if applicable):
4.19
How reproducible:
100%
Steps to Reproduce:
1. Set Registry override for registry.redhat.io/redhat 2. Notice catalog operatores still using registry.redhat.io/redhat images 3.
Actual results:
catalog operators using registry.redhat.io/redhat images
Expected results:
catalog operators use override set at HO level or set through OLM override flag on HC
Additional info:
Description of problem:
The apply-bootstrap container does not terminate within the `TerminationGracePeriodSeconds` timer. This is because the container does not respect a SIGTERM signal. This is an issue to address this bug and ensure the container respects a SIGTERM when issued and thus terminates within the window.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create hypershift enabled cluster 2. Delete API server pods and observe the apply-bootstrap container be force killed instead of terminating gracefully. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
In integration, creating a rosa HostedCluster with a shared vpc will result in a VPC endpoint that is not available.
Version-Release number of selected component (if applicable):
4.17.3
How reproducible:
Sometimes (currently every time in integration, but could be due to timing)
Steps to Reproduce:
1. Create a HostedCluster with shared VPC 2. Wait for HostedCluster to come up
Actual results:
VPC endpoint never gets created due to errors like: {"level":"error","ts":"2024-11-18T20:37:51Z","msg":"Reconciler error","controller":"awsendpointservice","controllerGroup":"hypershift.openshift.io","controllerKind":"AWSEndpointService","AWSEndpointService":{"name":"private-router","namespace":"ocm-int-2f4labdgi2grpumbq5ufdsfv7nv9ro4g-cse2etests-gdb"},"namespace":"ocm-int-2f4labdgi2grpumbq5ufdsfv7nv9ro4g-cse2etests-gdb","name":"private-router","reconcileID":"bc5d8a6c-c9ad-4fc8-8ead-6b6c161db097","error":"failed to create vpc endpoint: UnauthorizedOperation","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
Expected results:
VPC endpoint gets created
Additional info:
Deleting the control plane operator pod will get things working. The theory is that if the control plane operator pod is delayed in obtaining a web identity token, then the client will not assume the role that was passed to it. Currently the client is only created once at the start, we should create it on every reconcile.
Description of problem:
The story is to track i18n upload/download routine tasks which are perform every sprint.
A.C.
- Upload strings to Memosource at the start of the sprint and reach out to localization team
- Download translated strings from Memsource when it is ready
- Review the translated strings and open a pull request
- Open a followup story for next sprint
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The user-provided edge subnets (BYO VPC), created on zone type local-zone, is not tagged with kubernetes cluster tag `kubernetes.io/cluster/<infraID>` and value `shared` in install time. Subnets in regular/default zones are correctly tagged. The edge subnets created by installer in IPI is also tagged with value `owner`, so we need to check if there is an issue BYO VPC scenario or the implementation was just not replicated to edge subnets.
Version-Release number of selected component (if applicable):
4.19 (or since edge subnets, 4.14+?)
How reproducible:
always
Steps to Reproduce:
1. create vpc 2. create subnet in local zone 3. create install-config with regular zones, and edge zones 4. create the cluster 5. check the tags of subnets in local-zones
Actual results:
$ aws ec2 describe-subnets --subnet-ids $SUBNET_ID_PUB_WL | jq -r '.Subnets[] | [.AvailabilityZone, .Tags]' [ "us-east-1-nyc-1a", [ { "Key": "openshift_creationDate", "Value": "2025-01-24T00:14:44.445494+00:00" }, { "Key": "aws:cloudformation:stack-id", "Value": "arn:aws:cloudformation:us-east-1:[redacted]:stack/lzdemo-subnets-nyc-1a/10effe00-d9e0-11ef-b2ba-0ecca22ca195" }, { "Key": "aws:cloudformation:logical-id", "Value": "PublicSubnet" }, { "Key": "Name", "Value": "lzdemo-public-us-east-1-nyc-1a" }, { "Key": "aws:cloudformation:stack-name", "Value": "lzdemo-subnets-nyc-1a" } ] ]
Expected results:
$ aws ec2 describe-subnets --subnet-ids $SUBNET_ID_PUB_WL | jq -r '.Subnets[] | [.AvailabilityZone, .Tags]' [ "us-east-1-nyc-1a", [ { "Key": "openshift_creationDate", "Value": "2025-01-24T00:14:44.445494+00:00" }, { "Key": "aws:cloudformation:stack-id", "Value": "arn:aws:cloudformation:us-east-1:[redacted]:stack/lzdemo-subnets-nyc-1a/10effe00-d9e0-11ef-b2ba-0ecca22ca195" }, { "Key": "aws:cloudformation:logical-id", "Value": "PublicSubnet" }, { "Key": "Name", "Value": "lzdemo-public-us-east-1-nyc-1a" }, { "Key": "aws:cloudformation:stack-name", "Value": "lzdemo-subnets-nyc-1a" }, + { + "Key": "kubernetes.io/cluster/lzdemo-4znjd", + "Value": "shared" + }, ] ]
Additional info:
- Example of result in IPI deployment with edge zone (fully created by installer) ``` $ aws ec2 describe-subnets --subnet-ids subnet-08d8d32c7ee4b629c | jq -r '.Subnets[] | [.AvailabilityZone, .Tags]' [ "us-east-1-nyc-1a", [ { "Key": "kubernetes.io/role/elb", "Value": "1" }, { "Key": "Name", "Value": "lzipi-ljgzl-subnet-public-us-east-1-nyc-1a" }, { "Key": "sigs.k8s.io/cluster-api-provider-aws/role", "Value": "public" }, { "Key": "openshift_creationDate", "Value": "2025-01-24T00:14:44.445494+00:00" }, { "Key": "sigs.k8s.io/cluster-api-provider-aws/cluster/lzipi-ljgzl", "Value": "owned" }, { "Key": "kubernetes.io/cluster/lzipi-ljgzl", "Value": "owned" } ] ] ```
Description of problem:
Applying a performance profile with unsupported hugepages size, (In this example 512) fails to create the performanceprofile and it becomes degraded.
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Label one of the worker nodes with worker-cnf 2. Create an mcp for worker-cnf 3. Apply this performanceprofile apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: cpu: isolated: 1-3,4-6 reserved: 0,7 hugepages: defaultHugepagesSize: 512M pages: - count: 1 node: 0 size: 512M - count: 128 node: 1 size: 2M machineConfigPoolSelector: machineconfiguration.openshift.io/role: worker-cnf net: userLevelNetworking: true nodeSelector: node-role.kubernetes.io/worker-cnf: '' kernelPageSize: 64k numa: topologyPolicy: single-numa-node realTimeKernel: enabled: false workloadHints: highPowerConsumption: true perPodPowerManagement: false realTime: true Status: Conditions: Last Heartbeat Time: 2025-02-04T10:14:52Z Last Transition Time: 2025-02-04T10:14:52Z Status: False Type: Available Last Heartbeat Time: 2025-02-04T10:14:52Z Last Transition Time: 2025-02-04T10:14:52Z Status: False Type: Upgradeable Last Heartbeat Time: 2025-02-04T10:14:52Z Last Transition Time: 2025-02-04T10:14:52Z Status: False Type: Progressing Last Heartbeat Time: 2025-02-04T10:14:52Z Last Transition Time: 2025-02-04T10:14:52Z Message: can not convert size "512M" to kilobytes Reason: ComponentCreationFailed Status: True Type: Degraded Runtime Class: performance-performance Tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-performance Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Creation failed 11m (x19 over 33m) performance-profile-controller Failed to create all components: can not convert size "512M" to kilobytes
Actual results:
Expected results:
Additional info:
This was accidentally broken when we merged https://github.com/openshift/machine-config-operator/pull/4756, and should be fixed to prevent unnecessary API noise.
We think that low disk space is likely the cause of https://issues.redhat.com/browse/OCPBUGS-37785
It's not immediately obvious that this happened during the run without digging into the events.
Could we create a new test to enforce that the kubelet never reports disk pressure during a run?
Description of problem:Description of problem:
When we enable techpreview and we try to scale up a new node using a 4.5 base image, the node cannot join the cluster
Version-Release number of selected component (if applicable):
IPI on AWS $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.nightly-2024-08-19-165854 True False 5h25m Cluster version is 4.17.0-0.nightly-2024-08-19-165854
How reproducible:
Always
Steps to Reproduce:
1. Create a new machineset using a 4.5 base image and a 2.2.0 ignition version Detailed commands to create this machineset can be found here: [OCP-52822-Create new config resources with 2.2.0 ignition boot image nodes|https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-52822] 2. Scale up this machineset to create a new worker node
Actual results:
The node cannot join the cluster. We can find this message in the machine-config-daemon-pull.service in the failed node Wed 2024-08-21 13:02:19 UTC ip-10-0-29-231 machine-config-daemon-pull.service[1971]: time="2024-08-21T13:02:19Z" level=warning msg="skip_mount_home option is no longer supported, ignoring option" Wed 2024-08-21 13:02:20 UTC ip-10-0-29-231 machine-config-daemon-pull.service[1971]: Error: error pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a0afcde0e240601cb4a761e95f8311984b02ee76f827527d425670be3a39797": unable to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a0afcde0e240601cb4a761e95f8311984b02ee76f827527d425670be3a39797: invalid policy in "/etc/containers/policy.json": Unknown policy requirement type "sigstoreSigned"
Expected results:
Nodes should join the cluster
Additional info:
If techpreview is not enabled, the node can join the cluster without problems The podman version in a 4.5 base image is: $ podman version WARN[0000] skip_mount_home option is no longer supported, ignoring option Version: 1.9.3 RemoteAPI Version: 1 Go Version: go1.13.4 OS/Arch: linux/amd64
Description of problem:
The observe section is not bound only to monitoring anymore. Users that do not have access to all the namespaces or prometheus flags should be still able to see the menus under the observe section.
Version-Release number of selected component (if applicable):
4.19,4.18,4.17,4.16,4.15,4.14,4.12
How reproducible:
Always
Steps to Reproduce:
1. Enter the admin console without admin access to all namespaces
Actual results:
The observe section is not visible
Expected results:
The observe should be visible if plugins contribute menu items
Additional info:
If there are no items the observe section should be hidden
Description of problem:
There's a missmatch between the .configuration API vendored for the HC controller and the one in those older versions of the CPO controller. The HO computes the hash including "" for that field. The cpo doesn't see the field at all to compute the mcs hash. That'd cause the missmatch. Slack thread https://redhat-internal.slack.com/archives/C04EUL1DRHC/p1739540592972569?thread_ts=1739527507.065809&cid=C04EUL1DRHC
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create an HostedCluster with ImageConfig set in an 4.17.15 2. Check NodePool rollout 3. Ignition is failing
Actual results:
Expected results:
Additional info:
Cloned to OpenStack as we have the same issue.
Description of problem:
Customer is trying to install Self managed OCP cluster in aws. This customer use AWS VPC DHCPOptionSet. where it has a trailing dot (.) at the end of domain name in dhcpoptionset. due to this setting Master nodes hostname also has trailing dot & this cause failure in OpenShift installation.
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1.Please create a aws vpc with DHCPOptionSet, where DHCPoptionSet has trailing dot at the domain name. 2.Try installation of cluster with IPI.
Actual results:
Openshift Installer should allowed to create AWS Master nodes, where domain has trailing dot(.).
Expected results:
Additional info:
The following test is failing more than expected:
Undiagnosed panic detected in pod
See the sippy test details for additional context.
Observed in 4.18-e2e-azure-ovn/1864410356567248896 as well as pull-ci-openshift-installer-master-e2e-azure-ovn/1864312373058211840
: Undiagnosed panic detected in pod { pods/openshift-cloud-controller-manager_azure-cloud-controller-manager-5788c6f7f9-n2mnh_cloud-controller-manager_previous.log.gz:E1204 22:27:54.558549 1 iface.go:262] "Observed a panic" panic="interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.EndpointSlice" panicGoValue="&runtime.TypeAssertionError{_interface:(*abi.Type)(0x291daa0), concrete:(*abi.Type)(0x2b73880), asserted:(*abi.Type)(0x2f5cc20), missingMethod:\"\"}" stacktrace=<}
Description of problem:
Based on what discussed in bug OCPBUGS-46514 the openshift installer should not allow the creation of a cluster with different hostPrefix for the ClusterNetwork CIDRs of the same IP family.
Version-Release number of selected component (if applicable):
all the supported releases.
Description of problem:
The create button on MultiNetworkPolicies and NetworkPolicies list page is in wrong position, it should on the top right.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Unit tests for openshift/builder permanently failing for v4.18
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
1. Run PR against openshift/builder
Actual results:
Test fails: --- FAIL: TestUnqualifiedClone (0.20s) source_test.go:171: unable to add submodule: "Cloning into '/tmp/test-unqualified335202210/sub'...\nfatal: transport 'file' not allowed\nfatal: clone of 'file:///tmp/test-submodule643317239' into submodule path '/tmp/test-unqualified335202210/sub' failed\n" source_test.go:195: unable to find submodule dir panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference
Expected results:
Tests pass
Additional info:
Example: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_builder/401/pull-ci-openshift-builder-master-unit/1853816128913018880
Description of problem:
It is the negative testing scenario of QE test case OCP-36887, i.e. specifying non-existing encryption keyring/name for compute & control-plane, and the expectation is creating instance error due to the problem keyring. But the testing results is, the installer does not tell the error in stdout, and keeps waiting until timeout.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-15-202509
How reproducible:
Always
Steps to Reproduce:
1. "create install-config", and then insert the interested settings (see [1]) 2. "create cluster" (see [2])
Actual results:
The installer doesn't tell the real error and quit soon, although finally it tells below error, ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to provision control-plane machines: machines are not ready: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
Expected results:
The installer should tell the keyring error and quit soon.
Additional info:
Description of problem:
openshift-install has no zsh completion
Version-Release number of selected component (if applicable):
How reproducible:
everytime
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
PR opened at https://github.com/openshift/installer/pull/9116
Description of problem:
The machine-os-builder deployment manifest does not set the openshift.io/required-scc annotation, which appears to be required for the upgrade conformance suite to pass. The rest of the MCO components currently set this annotation and we can probably use the same setting for the Machine Config Controller (which is restricted-v2). What I'm unsure of is whether this also needs to be set on the builder pods as well and what the appropriate setting would be for that case.
Version-Release number of selected component (if applicable):
How reproducible:
This always occurs in the new CI jobs, e2e-aws-ovn-upgrade-ocb-techpreview and e2e-aws-ovn-upgrade-ocb-conformance-suite-techpreview. Here's two examples from rehearsal failures:
Steps to Reproduce:
Run either of the aforementioned CI jobs.
Actual results:
Test [sig-auth] all workloads in ns/openshift-machine-config-operator must set the 'openshift.io/required-scc' annotation fails.
Expected results:
Test{{ [sig-auth] all workloads in ns/openshift-machine-config-operator must set the 'openshift.io/required-scc' annotation}} should pass.
Additional info:
Description of problem:
Currently related images failures are being treated as warning in the catalog_handler.go (handleRelatedImages func). If one related image of a bundle fails during the collection phase, the bundle should be removed from the list of collected images, otherwise the batch will copy bundles with related images missing.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Bundles are being added to the list of collected images even if one of its related images failed.
Expected results:
Bundles with related images that failed during the collection phase should be removed from the list of collected images in the collection phase.
Additional info:
Description of problem:
Sort function on Access review table is not working as expected
Version-Release number of selected component (if applicable):
4.19.0-0.test-2025-02-28-070949
How reproducible:
Always
Steps to Reproduce:
1. Navigate to Hhome -> API Explorer -> Resource details -> Accessreview tab eg: /api-resource/all-namespaces/core~v1~Binding/access 2. Click on column headers Subject/Type to sort 3.
Actual results:
the sort function is not work as expected
Expected results:
clicking on the column headers can correctly sort the data in ascending or descending order as expected
Additional info:
click on column headers to sort, the table isn't using the sorted data, but the original filtered data instead https://github.com/openshift/console/blob/e9d6ead6852600993cffef3d50cb9b122d64c068/frontend/public/components/api-explorer.tsx#L575
Because we disable SNAT when deploying a disconnected cluster in PowerVS, the private PowerVS endpoints are unreachable. This causes workers to not launch and breaks disconnected deploys.
Description of problem:
when run the oc-mirror command twice , the rebuild catalog failed with error: [ERROR] : unable to rebuild catalog oci:///test/yinzhou/out20/working-dir/operator-catalogs/redhat-operator-index/33dd53f330f4518bd0427772debd3331aa4e21ef4ff4faeec0d9064f7e4f24a9/catalog-image: filtered declarative config not found
Version-Release number of selected component (if applicable):
oc-mirror version W1120 10:40:11.056507 6751 mirror.go:102] ⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.2.0-alpha.1-324-gbae91d5", GitCommit:"bae91d55", GitTreeState:"clean", BuildDate:"2024-11-20T02:06:04Z", GoVersion:"go1.23.0", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. run the mirror2mirror twice with same imageseconfig and same workspace, the twice command failed with error: cat config-20.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: quay.io/openshifttest/hello-openshift@sha256:61b8f5e1a3b5dbd9e2c35fd448dc5106337d7a299873dd3a6f0cd8d4891ecc27 operators: - catalog: oci:///test/redhat-operator-index packages: - name: aws-load-balancer-operator - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: devworkspace-operator `oc-mirror -c config-20.yaml docker://my-route-e2e-test-ocmirrorv2-pxbg4.apps.yinzhou11202.qe.devcluster.openshift.com --workspace file://out20 --v2 --dest-tls-verify=false`
Actual results:
oc-mirror -c config-20.yaml docker://my-route-e2e-test-ocmirrorv2-pxbg4.apps.yinzhou11202.qe.devcluster.openshift.com --workspace file://out20 --v2 --dest-tls-verify=false2024/11/20 10:34:00 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/20 10:34:00 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/20 10:34:00 [INFO] : ⚙️ setting up the environment for you... 2024/11/20 10:34:00 [INFO] : 🔀 workflow mode: mirrorToMirror 2024/11/20 10:34:00 [INFO] : 🕵️ going to discover the necessary images... 2024/11/20 10:34:00 [INFO] : 🔍 collecting release images... 2024/11/20 10:34:00 [INFO] : 🔍 collecting operator images... ✓ () Collecting catalog oci:///test/redhat-operator-index ✓ (2s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.15 2024/11/20 10:34:02 [INFO] : 🔍 collecting additional images... 2024/11/20 10:34:02 [INFO] : 🔍 collecting helm images... 2024/11/20 10:34:02 [INFO] : 🔂 rebuilding catalogs 2024/11/20 10:34:02 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/11/20 10:34:02 [ERROR] : unable to rebuild catalog oci:///test/yinzhou/out20/working-dir/operator-catalogs/redhat-operator-index/33dd53f330f4518bd0427772debd3331aa4e21ef4ff4faeec0d9064f7e4f24a9/catalog-image: filtered declarative config not found
Expected results:
no error
Additional info:
delete the workspace file , run again, no issue.
Description of problem:
Consoleplugin could be enabled repeatedly when it's already enabled.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-14-090045
How reproducible:
Always
Steps to Reproduce:
1.Go to console operator's 'Console Plugins' tab("/k8s/cluster/operator.openshift.io~v1~Console/cluster/console-plugins"), choose on console plugin, enable consoleplugin from modal by cicking 'Enabled/Disabled' edit button, try several times even though the plugin has been enabled. 2.Check console operator yaml. 3.
Actual results:
1. Could enable consoleplugin repeatedly. 2. The same consoleplugin are added in console operator several times. $ oc get consoles.operator.openshift.io cluster -ojson | jq '.spec.plugins' [ "monitoring-plugin", "monitoring-plugin", "networking-console-plugin", "networking-console-plugin" ]
Expected results:
1.Should not enable repeatedly. 2. Should not add the same consoleplugin multiple times in console operator
Additional info:
We can even add the same plugin name in console operator yaml directly, that's not corect.
Description of problem:
CAPI install got ImageReconciliationFailed when creating vpc custom image
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-06-101930
How reproducible:
always
Steps to Reproduce:
1.add the following in install-config.yaml featureSet: CustomNoUpgrade featureGates: [ClusterAPIInstall=true] 2. create IBMCloud cluster with IPI
Actual results:
level=info msg=Done creating infra manifests level=info msg=Creating kubeconfig entry for capi cluster ci-op-h3ykp5jn-32a54-xprzg level=info msg=Waiting up to 30m0s (until 11:25AM UTC) for network infrastructure to become ready... level=error msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: infrastructure was not ready within 30m0s: client rate limiter Wait returned an error: context deadline exceeded
in IBMVPCCluster-openshift-cluster-api-guests log
reason: ImageReconciliationFailed message: 'error failure trying to create vpc custom image: error unknown failure creating vpc custom image: The IAM token that was specified in the request has expired or is invalid. The request is not authorized to access the Cloud Object Storage resource.'
Expected results:
create cluster succeed
Additional info:
the resources created when install failed: ci-op-h3ykp5jn-32a54-xprzg-cos dff97f5c-bc5e-4455-b470-411c3edbe49c crn:v1:bluemix:public:cloud-object-storage:global:a/fdc2e14cf8bc4d53a67f972dc2e2c861:f648897a-2178-4f02-b948-b3cd53f07d85:: ci-op-h3ykp5jn-32a54-xprzg-vpc is.vpc crn:v1:bluemix:public:is:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861::vpc:r022-46c7932d-8f4d-4d53-a398-555405dfbf18 copier-resurrect-panzer-resistant is.security-group crn:v1:bluemix:public:is:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861::security-group:r022-2367a32b-41d1-4f07-b148-63485ca8437b deceiving-unashamed-unwind-outward is.network-acl crn:v1:bluemix:public:is:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861::network-acl:r022-b50286f6-1052-479f-89bc-fc66cd9bf613
Description of problem:
the contextId property to allow users of HorizontalNav to receive dynamic tabs contributions is not available
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
1.Import HorizontalNav from the dynamic plugins SDK 2.'contextId' property does not exist 3.
Actual results:
Expected results:
Additional info:
The helper doesn't have all the namespaces in it, and we're getting some flakes in CI like this:
{{batch/v1/Job/openshift-backplane-managed-scripts/<batch_job>/container/osd-delete-backplane-script-resources
does not have a cpu request (rule: "batch/v1/Job/openshift-backplane-managed-scripts/<batch_job>/container/osd-delete-backplane-script-resources/request[cpu]")}}
Description of problem:
when installing cluster with multiple NICs in failureDomains, it always report "Duplicate value" error # ./openshift-install create cluster --dir cluster --log-level debug ... INFO Creating infra manifests... INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests DEBUG {"level":"info","ts":"2025-01-01T11:28:56Z","msg":"Starting workers","controller":"nutanixcluster","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"NutanixCluster","worker count":10} DEBUG {"level":"info","ts":"2025-01-01T11:28:57Z","msg":"Starting workers","controller":"nutanixmachine","controllerGroup":"infrastructure.cluster.x-k8s.io","controllerKind":"NutanixMachine","worker count":10} INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=sgao-nutanix-zonal-l96qg DEBUG I0101 11:28:58.918576 2309035 recorder.go:104] "Cluster sgao-nutanix-zonal-l96qg is Provisioning" logger="events" type="Normal" object={"kind":"Cluster","namespace":"openshift-cluster-api-guests","name":"sgao-nutanix-zonal-l96qg","uid":"d86c6f80-0f60-431d-80fc-bddd7b1f2d7c","apiVersion":"cluster.x-k8s.io/v1beta1","resourceVersion":"257"} reason="Provisioning" DEBUG Collecting applied cluster api manifests... DEBUG I0101 11:28:58.924319 2309035 warning_handler.go:65] "metadata.finalizers: \"cluster.cluster.x-k8s.io\": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers" logger="KubeAPIWarningLogger" ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to create infrastructure manifest: NutanixCluster.infrastructure.cluster.x-k8s.io "sgao-nutanix-zonal-l96qg" is invalid: spec.failureDomains[0].subnets[1]: Duplicate value: map[string]interface {}{"type":"uuid"} INFO Shutting down local Cluster API controllers... INFO Stopped controller: Cluster API WARNING process cluster-api-provider-nutanix exited with error: signal: killed INFO Stopped controller: nutanix infrastructure provider INFO Shutting down local Cluster API control plane... INFO Local Cluster API system has completed operations
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-01-04-101226
How reproducible:
always
Steps to Reproduce:
1. set multiple NICs in failureDomains and install cluster e.g. failureDomains: ... subnetUUIDs: - 512c1d6f-c6e7-4746-8ae2-9c3e1db2aba6 - a94cb75c-24ff-4ee2-85cf-c2f906ee9fe5 - name: failure-domain-2 ... subnetUUIDs: - d1b1b617-23de-4a9d-b53f-4b386fc27600 - name: failure-domain-3 ... subnetUUIDs: - 3624b067-61e2-4703-b8bf-3810de5cbac1 2. 3.
Actual results:
Install failed
Expected results:
Install should succeed with multiple NICs configured
Additional info:
slack discussion pls refer to https://redhat-external.slack.com/archives/C0211848DBN/p1735790959497809
Since we merged https://github.com/openshift/hypershift/pull/5679
It seems like we need to have a rollout but in some cases, the HostedCluster is already rolled out to the latest image.
We need to support that case.
Example of error: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-openstack-aws/1897225681364848640
In the hypershift CLI we currently support creating a cluster with a proxy with the `--enable-proxy` flag.
This is to add a new flag `--enable-secure-proxy` that will create a cluster configured to use a secure proxy.
Description of problem:
Project dropdown is partially hidden due to web terminal
Version-Release number of selected component (if applicable):
How reproducible:
Every time
Steps to Reproduce:
1. Install and initialize web-terminal 2. Open the Project bar 3.
Actual results:
Attaching screenshot:
https://drive.google.com/file/d/1AaYXCZsEcBiXVCIBqXkavKvXbjFb1YlP/view?usp=sharing
Expected results:
Project namespace bar should be at the front
Additional info:
Tracker issue for bootimage bump in 4.19. This issue should block issues which need a bootimage bump to fix.
The previous bump was OCPBUGS-44977.
Implementing RFE-3017.
As a bare Story, without a Feature or Epic, because I'm trying to limit the amount of MCO-side paperwork required to get my own RFE itch scratched. As a Story and not a NO-ISSUE pull, because OCP QE had a bit of trouble handling mco#4637 when I went NO-ISSUE on that one, and I think this might be worth a 4.19 release note.
https://github.com/sclorg/nodejs-ex
image-registry.openshift-image-registry.svc:5000/demo/nodejs-app
image-registry.openshift-image-registry.svc:5000/openshift/nodejs
BuildRun fails with the following error:
[User error] invalid input params for task nodejs-app-6cf5j-gk6f9: param types don't match the user-specified type: [registries-block registries-insecure]
BuildRun runs successfully
https://gist.github.com/vikram-raj/fa67186f1860612b5ad378655085745e
Description of problem:
The new `managed-trust-bundle` VolumeMount / `trusted-ca-bundle-managed` ConfigMap has recently been required given this latest change here: https://github.com/openshift/hypershift/pull/5667. However, this should be optional since folks that bring their own PKI shouldn't need this.
Version-Release number of selected component (if applicable):
4.18.2
How reproducible:
Every time.
Steps to Reproduce:
1. Deploy ROKS (HyperShift) version 4.18.2 cluster.
Actual results:
Cluster fails to deploy as the OpenShift API server fails to come up since it expects the `trusted-ca-bundle-managed` ConfigMap to exist.
Expected results:
Cluster should deploy successfully.
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
Delete the openshift-monitoring/monitoring-plugin-cert secret, SCO will re-create a new one with different content
Actual results:
- monitoring-plugin is still using the old cert content. - If the cluster doesn’t show much activity, the hash may take time to be updated.
Expected results:
CMO should detect that exact change and run a sync to recompute and set the new hash.
Additional info:
- We shouldn't rely on another changeto trigger the sync loop. - CMO should maybe watch that secret? (its name isn't known in advance).
Description of problem:
Reviewing some 4.17 cluster-ingress-operator logs, I found many (215) of these even when the GatewayAPI feature was disabled: 024-09-03T08:20:03.726Z INFO operator.gatewayapi_controller controller/controller.go:114 reconciling {"request": {"name":"cluster"}} This makes it look like the feature was enabled when it was not. Also check for same in the other gatewayapi controllers in the gateway-service-dns and gatewayclass folders. A search for r.config.GatewayAPIEnabled should show where we are checking whether the feature is enabled.
Version-Release number of selected component (if applicable):
4.17, 4.18 should have the fix
How reproducible:
This was showing up in the CI logs for the e2e-vsphere-ovn-upi test: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-upi/1830872105227390976/artifacts/e2e-vsphere-ovn-upi/gather-extra/artifacts/pods/openshift-ingress-operator_ingress-operator-64df7b9cd4-hqkmh_ingress-operator.log It is probably showing up in all logs to varying degrees.
Steps to Reproduce:
1. Deploy 4.17 2. Review cluster-ingress-operator logs
Actual results:
Seeing a log that makes it look like the GatewayAPI feature is enabled even when it is not.
Expected results:
Only see the log when the GatewayAPI feature is enabled.
Additional info:
The GatewayAPI feature is enabled in the e2e-aws-gatewayapi PR test and any techpreview PR test, and can be manually enabled on a test cluster by running: oc patch featuregates/cluster --type=merge --patch='{"spec":{"featureSet":"CustomNoUpgrade","customNoUpgrade":{"enabled":["GatewayAPI"]}}}'
Description of problem:
There are duplicate external link icon on Operator details modal for 'Purchase' button
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-18-013707
How reproducible:
Always
Steps to Reproduce:
1. find one Marketplace operator and click on operator tile
Actual results:
1. on Operator details modal, there is `Purchase` button and there are two duplicate external link icon displayed beside Purchase button
Expected results:
1. only one external link icon is required
Additional info:
screenshot https://drive.google.com/file/d/1uGCXxXdR8ayXRafhabHepW5mVqwePzcq/view?usp=drive_link
Description of problem:
When installing ROSA/OSD operators OLM "locks up" the Subscription object with "ConstraintsNotSatisfiable" 3-15% of the times, depending on the environment.
Version-Release number of selected component (if applicable):
Recently tested on: - OSD 4.17.5 - 4.18 nightly (from cluster bot) Though prevalence across the ROSA fleet suggests this is not a new issue.
How reproducible:
Very. This is very prevalent across the OSD/ROSA Classic cluster fleet. Any new OSD/ROSA Classic cluster has a good chance of at least one of its ~12 OSD-specific operators being affected on install time.
Steps to Reproduce:
0. Set up a cluster using cluster bot. 1. Label at least one worker node with node-role.kubernetes.io=infra 2. Install must gather operator with "oc apply -f mgo.yaml" (file attached) 3. Wait for the pods to come up. 4. Start this loop: for i in `seq -w 999`; do echo -ne ">>>>>>> $i\t\t"; date; oc get -n openshift-must-gather-operator subscription/must-gather-operator -o yaml >mgo-sub-$i.yaml; oc delete -f mgo.yaml; oc apply -f mgo.yaml; sleep 100; done 5. Let it run for a few hours.
Actual results:
Run "grep ConstraintsNotSatisfiable *.yaml" You should find a few of the Subscriptions ended up in a "locked" state from which there is no upgrade without manual intervention: - message: 'constraints not satisfiable: @existing/openshift-must-gather-operator//must-gather-operator.v4.17.281-gd5416c9 and must-gather-operator-registry/openshift-must-gather-operator/stable/must-gather-operator.v4.17.281-gd5416c9 originate from package must-gather-operator, subscription must-gather-operator requires must-gather-operator-registry/openshift-must-gather-operator/stable/must-gather-operator.v4.17.281-gd5416c9, subscription must-gather-operator exists, clusterserviceversion must-gather-operator.v4.17.281-gd5416c9 exists and is not referenced by a subscription' reason: ConstraintsNotSatisfiable status: "True" type: ResolutionFailed
Expected results:
Each installation attempt should've worked fine.
Additional info:
mgo.yaml:
apiVersion: v1 kind: Namespace metadata: name: openshift-must-gather-operator annotations: package-operator.run/collision-protection: IfNoController package-operator.run/phase: namespaces openshift.io/node-selector: "" labels: openshift.io/cluster-logging: "true" openshift.io/cluster-monitoring: 'true' --- apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: must-gather-operator-registry namespace: openshift-must-gather-operator annotations: package-operator.run/collision-protection: IfNoController package-operator.run/phase: must-gather-operator labels: opsrc-datastore: "true" opsrc-provider: redhat spec: image: quay.io/app-sre/must-gather-operator-registry@sha256:0a0610e37a016fb4eed1b000308d840795838c2306f305a151c64cf3b4fd6bb4 displayName: must-gather-operator icon: base64data: '' mediatype: '' publisher: Red Hat sourceType: grpc grpcPodConfig: securityContextConfig: restricted nodeSelector: node-role.kubernetes.io: infra tolerations: - effect: NoSchedule key: node-role.kubernetes.io/infra operator: Exists --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: must-gather-operator namespace: openshift-must-gather-operator annotations: package-operator.run/collision-protection: IfNoController package-operator.run/phase: must-gather-operator spec: channel: stable name: must-gather-operator source: must-gather-operator-registry sourceNamespace: openshift-must-gather-operator --- apiVersion: operators.coreos.com/v1alpha2 kind: OperatorGroup metadata: name: must-gather-operator namespace: openshift-must-gather-operator annotations: package-operator.run/collision-protection: IfNoController package-operator.run/phase: must-gather-operator olm.operatorframework.io/exclude-global-namespace-resolution: 'true' spec: targetNamespaces: - openshift-must-gather-operator
Description of problem:
cluster-version-operator pod is crashing repeatedly. It is a disconnected cluster. Customer used to ABI (agent based installation) and it's bare metal. # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.7 True False 134d Cluster version is 4.16.7 $ omc get pods NAME READY STATUS RESTARTS AGE cluster-version-operator-74688c8bc5-fbzlf 0/1 Error 1425 7d ==== 2025-02-07T10:24:37.750428195+05:30 I0207 04:54:37.750394 1 start.go:565] FeatureGate found in cluster, using its feature set "" at startup 2025-02-07T10:24:37.750437148+05:30 I0207 04:54:37.750427 1 payload.go:307] Loading updatepayload from "/" 2025-02-07T10:24:37.750603865+05:30 I0207 04:54:37.750583 1 payload.go:403] Architecture from release-metadata (4.16.7) retrieved from runtime: "amd64" 2025-02-07T10:24:37.751094450+05:30 I0207 04:54:37.751078 1 metrics.go:154] Metrics port listening for HTTPS on 0.0.0.0:9099 2025-02-07T10:24:37.751105629+05:30 E0207 04:54:37.751091 1 metrics.go:166] Collected metrics HTTPS server goroutine: listen tcp 0.0.0.0:9099: bind: address already in use 2025-02-07T10:24:37.754776401+05:30 E0207 04:54:37.754748 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) 2025-02-07T10:24:37.754776401+05:30 goroutine 236 [running]: 2025-02-07T10:24:37.754776401+05:30 k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1bdfc40?, 0x311dc90}) 2025-02-07T10:24:37.754776401+05:30 /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85 2025-02-07T10:24:37.754776401+05:30 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0004fc370?}) 2025-02-07T10:24:37.754776401+05:30 /go/src/github.com/openshift/cluster-version-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b 2025-02-07T10:24:37.754776401+05:30 panic({0x1bdfc40?, 0x311dc90?}) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/runtime/panic.go:914 +0x21f 2025-02-07T10:24:37.754776401+05:30 crypto/tls.(*listener).Close(0x1e67240?) 2025-02-07T10:24:37.754776401+05:30 <autogenerated>:1 +0x1e 2025-02-07T10:24:37.754776401+05:30 net/http.(*onceCloseListener).close(...) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/net/http/server.go:3502 2025-02-07T10:24:37.754776401+05:30 sync.(*Once).doSlow(0xc000701b68?, 0x414e32?) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/sync/once.go:74 +0xbf 2025-02-07T10:24:37.754776401+05:30 sync.(*Once).Do(...) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/sync/once.go:65 2025-02-07T10:24:37.754776401+05:30 net/http.(*onceCloseListener).Close(0xc000346ab0) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/net/http/server.go:3498 +0x45 2025-02-07T10:24:37.754776401+05:30 net/http.(*Server).closeListenersLocked(0xa6?) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/net/http/server.go:2864 +0x96 2025-02-07T10:24:37.754776401+05:30 net/http.(*Server).Shutdown(0xc0006c00f0, {0x21f3160, 0xc0002c85f0}) 2025-02-07T10:24:37.754776401+05:30 /usr/lib/golang/src/net/http/server.go:2790 +0x96 2025-02-07T10:24:37.754776401+05:30 github.com/openshift/cluster-version-operator/pkg/cvo.RunMetrics({0x21f3160, 0xc0002c8640}, {0x21f3160, 0xc0002c85f0}, {0x7ffff38f18a7, 0xc}, {0x7ffff38f18c8, 0x1d}, {0x7ffff38f18f9, 0x1d}) 2025-02-07T10:24:37.754776401+05:30 /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/metrics.go:292 +0x130c 2025-02-07T10:24:37.754776401+05:30 github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func3.1.1() 2025-02-07T10:24:37.754776401+05:30 /go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:248 +0x85 2025-02-07T10:24:37.754776401+05:30 created by github.com/openshift/cluster-version-operator/pkg/start.(*Options).run.func3.1 in goroutine 234 2025-02-07T10:24:37.754776401+05:30 /go/src/github.com/openshift/cluster-version-operator/pkg/start/start.go:246 +0x176 2025-02-07T10:24:37.757971014+05:30 panic: runtime error: invalid memory address or nil pointer dereference [recovered] 2025-02-07T10:24:37.757971014+05:30 panic: runtime error: invalid memory address or nil pointer dereference 2025-02-07T10:24:37.757971014+05:30 [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7c5afe] ====
Description of problem:
Bootstrap process failed due to API_URL and API_INT_URL are not resolvable: Feb 06 06:41:49 yunjiang-dn16d-657jf-bootstrap systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE Feb 06 06:41:49 yunjiang-dn16d-657jf-bootstrap systemd[1]: bootkube.service: Failed with result 'exit-code'. Feb 06 06:41:49 yunjiang-dn16d-657jf-bootstrap systemd[1]: bootkube.service: Consumed 1min 457ms CPU time. Feb 06 06:41:54 yunjiang-dn16d-657jf-bootstrap systemd[1]: bootkube.service: Scheduled restart job, restart counter is at 1. Feb 06 06:41:54 yunjiang-dn16d-657jf-bootstrap systemd[1]: Stopped Bootstrap a Kubernetes cluster. Feb 06 06:41:54 yunjiang-dn16d-657jf-bootstrap systemd[1]: bootkube.service: Consumed 1min 457ms CPU time. Feb 06 06:41:54 yunjiang-dn16d-657jf-bootstrap systemd[1]: Started Bootstrap a Kubernetes cluster. Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Check if API and API-Int URLs are resolvable during bootstrap Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Checking if api.yunjiang-dn16d.qe.gcp.devcluster.openshift.com of type API_URL is resolvable Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Starting stage resolve-api-url Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Unable to resolve API_URL api.yunjiang-dn16d.qe.gcp.devcluster.openshift.com Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Checking if api-int.yunjiang-dn16d.qe.gcp.devcluster.openshift.com of type API_INT_URL is resolvable Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Starting stage resolve-api-int-url Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Unable to resolve API_INT_URL api-int.yunjiang-dn16d.qe.gcp.devcluster.openshift.com Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[8905]: https://localhost:2379 is healthy: successfully committed proposal: took = 7.880477ms Feb 06 06:41:58 yunjiang-dn16d-657jf-bootstrap bootkube.sh[7781]: Starting cluster-bootstrap... Feb 06 06:41:59 yunjiang-dn16d-657jf-bootstrap bootkube.sh[8989]: Starting temporary bootstrap control plane... Feb 06 06:41:59 yunjiang-dn16d-657jf-bootstrap bootkube.sh[8989]: Waiting up to 20m0s for the Kubernetes API Feb 06 06:42:00 yunjiang-dn16d-657jf-bootstrap bootkube.sh[8989]: API is up install logs: ... time="2024-02-06T06:54:28Z" level=debug msg="Unable to connect to the server: dial tcp: lookup api-int.yunjiang-dn16d.qe.gcp.devcluster.openshift.com on 169.254.169.254:53: no such host" time="2024-02-06T06:54:28Z" level=debug msg="Log bundle written to /var/home/core/log-bundle-20240206065419.tar.gz" time="2024-02-06T06:54:29Z" level=error msg="Bootstrap failed to complete: timed out waiting for the condition" time="2024-02-06T06:54:29Z" level=error msg="Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane." ...
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-02-05-184957,openshift/machine-config-operator#4165
How reproducible:
Always.
Steps to Reproduce:
1. Enable custom DNS on gcp: platform.gcp.userProvisionedDNS:Enabled and featureSet:TechPreviewNoUpgrade 2. Create cluster 3.
Actual results:
Failed to complete bootstrap process.
Expected results:
See description.
Additional info:
I believe 4.15 is affected as well once https://github.com/openshift/machine-config-operator/pull/4165 backport to 4.15, currently, it failed at an early phase, see https://issues.redhat.com/browse/OCPBUGS-28969
Description of problem:
A minor component regression - [sig-auth] all workloads in ns/openshift-machine-config-operator must set the 'openshift.io/required-scc' annotation is failing for arbiter templates. See https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&LayeredProduct=none&Network=ovn&Network=ovn&NetworkAccess=default&Platform=metal&Platform=metal&Procedure=none&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2025-03-13%2023%3A59%3A59&baseRelease=4.18&baseStartTime=2025-02-11%2000%3A00%3A00&capability=Other&columnGroupBy=Architecture%2CNetwork%2CPlatform%2CTopology&component=Machine%20Config%20Operator&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20metal%20unknown%20ha%20none&flakeAsFailure=false&ignoreDisruption=true&ignoreMissing=false&includeMultiReleaseAnalysis=true&includeVariant=Architecture%3Aamd64&includeVariant=CGroupMode%3Av2&includeVariant=ContainerRuntime%3Acrun&includeVariant=ContainerRuntime%3Arunc&includeVariant=FeatureSet%3Adefault&includeVariant=FeatureSet%3Atechpreview&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=LayeredProduct%3Anone&includeVariant=Network%3Aovn&includeVariant=Owner%3Aeng&includeVariant=Owner%3Aservice-delivery&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Arosa&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&includeVariant=Topology%3Amicroshift&minFail=3&passRateAllTests=0&passRateNewTests=95&pity=5&sampleEndTime=2025-03-13%2023%3A59%3A59&sampleRelease=4.19&sampleStartTime=2025-03-06%2000%3A00%3A00&testBasisRelease=4.18&testId=openshift-tests%3Ac1e4aa5075ed2c622d44645fe531a7a1&testName=%5Bsig-auth%5D%20all%20workloads%20in%20ns%2Fopenshift-machine-config-operator%20must%20set%20the%20%27openshift.io%2Frequired-scc%27%20annotation
Version-Release number of selected component (if applicable):
4.19 only
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
After following the procedure described at https://github.com/openshift/installer/blob/main/docs/user/openstack/observability.md the PromQL query:
sum by (vm_instance) ( group by (vm_instance, resource) (ceilometer_cpu) / on (resource) group_right(vm_instance) ( group by (node, resource) ( label_replace(kube_node_info, "resource", "$1", "system_uuid", "(.+)") ) / on (node) group_left group by (node) ( cluster:master_nodes ) ) )
returns an empty result because the cluster:master_nodes metric is not being scraped from the shift-on-stack cluster.
Version-Release number of selected component (if applicable):
4.18.0-rc.6
How reproducible:
Always
Steps to Reproduce: https://github.com/openshift/installer/blob/main/docs/user/openstack/observability.md
Actual results:
Empty result
Expected results:
The number of OpenShift master nodes per OpenStack host
Additional info:
Just querying the 'cluster:master_nodes' metric returns empty
The procedure described in the docs needs to be updated and include the cluster:master_nodes metric retrieval in the scrapeconfig definition.
spec: params: 'match[]': - '{__name__=~"kube_node_info|kube_persistentvolume_info|cluster:master_nodes"}'
Description of problem:
When the ImageStreamTag is updated, the pod does not reflect the new image automatically for Deployments created via the OpenShift Console.
Version-Release number of selected component (if applicable):
Red Had Openshift Container Platform
Steps to Reproduce:
1. Create a Deployment via OpenShift console and check that the trigger annotation has a `paused` field as string (e.g., `"paused":"false"`) instead of a boolean value (e.g., `"paused":false`) 2. Make changes on the ImageStreamTag
Actual results:
No changes on the deployment or new Pods created.
Expected results:
The deployment should have the new changes of the ImageStreamTag.
Description of problem:
We have recently enabled a few endpoint overrides, but ResourceManager was accidentally excluded.
Beginning with payload 4.19.0-0.nightly-2025-02-21-125506 panics in aggregated-hypershift-ovn-conformance are causing failures due to
[sig-arch] events should not repeat pathologically for ns/openshift-console-operator event happened 772 times, something is wrong: namespace/openshift-console-operator deployment/console-operator hmsg/ac4940667c - reason/Console-OperatorPanic Panic observed: runtime error: invalid memory address or nil pointer dereference (14:06:18Z) result=reject
That payload contains console-operator#950 will begin investigation there.
{Failed === RUN TestAutoscaling/EnsureHostedCluster/EnsurePodsWithEmptyDirPVsHaveSafeToEvictAnnotations
util.go:1232:
the pod openstack-manila-csi-controllerplugin-676cc65ffc-tnnkb is not in the audited list for safe-eviction and should not contain the safe-to-evict-local-volume annotation
Expected
<string>: socket-dir
to be empty
--- FAIL: TestAutoscaling/EnsureHostedCluster/EnsurePodsWithEmptyDirPVsHaveSafeToEvictAnnotations (0.02s)
}
Description of problem:
CGO_ENABLED=0 GO111MODULE=on GOWORK=off GOFLAGS=-mod=vendor go build -gcflags=all='-N -l' -ldflags '-extldflags "-static"' -o bin/hypershift . github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v5: /usr/lib/golang/pkg/tool/linux_amd64/compile: signal: killed make: *** [Makefile:118: hypershift] Error 1
Version-Release number of selected component (if applicable):
How reproducible:
25%
Steps to Reproduce:
1.Send a PR 2.Job builds the container
Actual results:
The build aborts due to memory issues when processing AzureSDK imports
Expected results:
The build completes
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
The openshift/origin/text/extended/router/http2.go tests don't run on AWS. We disabled this sometime ago. Let's enable this to see if it is still an issue. I have been running the http2 tests on AWS this week and I have not run into the original issue highlighted by the original bugzilla bug.
// platformHasHTTP2LoadBalancerService returns true where the default // router is exposed by a load balancer service and can support http/2 // clients. func platformHasHTTP2LoadBalancerService(platformType configv1.PlatformType) bool { switch platformType { case configv1.AzurePlatformType, configv1.GCPPlatformType: return true case configv1.AWSPlatformType: e2e.Logf("AWS support waiting on https://bugzilla.redhat.com/show_bug.cgi?id=1912413") fallthrough default: return false } }
Description of problem:
If the serverless function is not running and on click of Test Serverless button, nothing is happening.
Version-Release number of selected component (if applicable):
4.15
How reproducible:
Always
Steps to Reproduce:
1.Install serverless operator 2.Create serverless function and make sure the status is false 3.Click on Test Serverless function
Actual results:
No response
Expected results:
May be an alert or may be we can hide that option if function is not ready?
Additional info:
Description of problem:
After creating a pair of self-signed tls cert and private key, then add it into trustde-ca-bundle by using the following cmd: oc patch proxy/cluster \ --type=merge \ --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}' The insights-runtime-extractor pod will return response with 500 status code, this is the https flow details: * Trying 10.129.2.15:8000... * Connected to exporter.openshift-insights.svc.cluster.local (10.129.2.15) port 8000 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /var/run/configmaps/service-ca-bundle/service-ca.crt * TLSv1.0 (OUT), TLS header, Certificate Status (22): * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS header, Certificate Status (22): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS header, Finished (20): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Request CERT (13): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.2 (OUT), TLS header, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.3 (OUT), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=*.exporter.openshift-insights.svc * start date: Jan 2 02:19:07 2025 GMT * expire date: Jan 2 02:19:08 2027 GMT * subjectAltName: host "exporter.openshift-insights.svc.cluster.local" matched cert's "exporter.openshift-insights.svc.cluster.local" * issuer: CN=openshift-service-serving-signer@1735784302 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.2 (OUT), TLS header, Unknown (23): * Using Stream ID: 1 (easy handle 0x5577a19094a0) * TLSv1.2 (OUT), TLS header, Unknown (23): > GET /gather_runtime_info HTTP/2 > Host: exporter.openshift-insights.svc.cluster.local:8000 > accept: */* > user-agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/_f19625f5-ee5f-40c0-bc49-23a8ba1abe61_ > authorization: Bearer sha256~x9jj_SnjJf6LVlhhWFdUG8UqnPDHzZW0xMYa0WU05Gw > * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.2 (OUT), TLS header, Unknown (23): * TLSv1.2 (IN), TLS header, Unknown (23): * TLSv1.2 (IN), TLS header, Unknown (23): < HTTP/2 500 < content-type: text/plain; charset=utf-8 < date: Thu, 02 Jan 2025 08:18:59 GMT < x-content-type-options: nosniff < content-length: 33 < * TLSv1.2 (IN), TLS header, Unknown (23): stat : no such file or directory
Version-Release number of selected component (if applicable):
4.19
How reproducible:
True
Steps to Reproduce:
1. Create a pair of self-signed tls cert and key 2. Update trusted-ca-bundle by using following cmd: oc patch proxy/cluster \ --type=merge \ --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}' 3. Pull a request to insights-runtime-extractor pod via the following cmd: curl -v --cacert /var/run/configmaps/trusted-ca-bundle/ca-bundle.crt -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/_<cluster_id>_" -H "Authorization: <token>" -H 'Cache-Control: no-cache' https://api.openshift.com/api/accounts_mgmt/v1/certificates
Actual results:
3. The status code of response to this request is 500
Expected results:
3. The status code of response to this request should be 200 and return the runtime info as expected.
Additional info:
Description of problem:
The documentation in the MCO repo on supported extensions does not match the experienced functionality for 4.17.
Version-Release number of selected component (if applicable):
4.17/4.18
How reproducible:
Always
Steps to Reproduce:
1. Start a 4.17.x cluster. 2. Apply a MC to install the sysstat extension. apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 80-worker-extensions spec: config: ignition: version: 3.2.0 extensions: - sysstat 3. Wait for the MCP targeted by the MC in step 2 to degrade & check the reason in the controller logs. Degraded Machine: ip-10-0-21-15.us-west-2.compute.internal and Degraded Reason: invalid extensions found: [sysstat]
Actual results:
Applying an MC to install sysstat results in a degraded MCP.
Expected results:
Documentation in the MCO repo suggests that `sysstat` is supported for OCP version 4.17. Either the extension should be supported in 4.17 or the documentation should be updated to indicate the correct version where support was added.
Additional info:
Referenced MCO Repo Documentation can be found here.
Description of problem:
Initially, the clusters at version 4.16.9 were having issues with reconciling the IDP. The error which was found in Dynatrace was
"error": "failed to update control plane: failed to reconcile openshift oauth apiserver: failed to reconcile oauth server config: failed to generate oauth config: failed to apply IDP AAD config: Service Unavailable",
Initially it was assumed that the IDP service was unavialble but the CU confirmed that they also had the GroupSync operator running inside all clusters, which can successfully connect to the customer IDP and sync User + Group information from the IDP into the cluster.
The CU was advised to upgrade to 4.16.18 keeping in mind few of the other OCPBUGS which were related to proxy and would be resolved by upgrading to 4.16.15+
However, after upgrade the IDP is still failing to apply it seems. It looks like IDP reconciler isn't considering the Additional Trust Bundle for the customer proxy
Checking DT Logs, it seems to fail to verify the certificate
"error": "failed to update control plane: failed to reconcile openshift oauth apiserver: failed to reconcile oauth server config: failed to generate oauth config: failed to apply IDP AAD config: tls: failed to verify certificate: x509: certificate signed by unknown authority", "error": "failed to update control plane: [failed to reconcile openshift oauth apiserver: failed to reconcile oauth server config: failed to generate oauth config: failed to apply IDP AAD config: tls: failed to verify certificate: x509: certificate signed by unknown authority, failed to update status: Operation cannot be fulfilled on hostedcontrolplanes.hypershift.openshift.io \"rosa-staging\": the object has been modified; please apply your changes to the latest version and try again]",
Version-Release number of selected component (if applicable):
4.16.18
How reproducible:
Customer has a few clusters deployed and each of them has the same issue.
Steps to Reproduce:
1. Create a HostedCluster with a proxy configuration that specifies an additionalTrustBundle, and an OpenID idp that can be publicly verified (ie. EntraID or Keycloak with LetsEncrypt certs) 2. Wait for the cluster to come up and try to use the IDP 3.
Actual results:
IDP is failing to work for HCP
Expected results:
IDP should be working for the clusters
Additional info:
The issue will happen only if the IDP does not require a custom trust bundle to be verified.
Description of problem:
As more systems have been added to Power VS, the assumption that every zone in a region has the same set of systypes has been broken. To properly represent what system types are available, the powervs_regions struct needed to be altered and parts of the installer referencing it needed to be updated.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Try to deploy with s1022 in dal10 2. SysType not available, even though it is a valid option in Power VS. 3.
Actual results:
Expected results:
Additional info:
The HyperShift Control Plane Operator doesn't honor the set *_PROXY variables in some places in the code. Despite the proxy vars being set. HTTP requests for validating KAS health fail when all egress traffic is blocked except through a proxy.
I would like to check the complexity of replacing this library by archive/tar (from standard library).
Outcome:
Description of problem:
When listing the UserDefinedNetwork via the UI (or CLI), the MTU is not reported.
This makes the user flow cumbersome, since they won't know which MTU they're using unless they log into the VM, and actually check what's there.
Version-Release number of selected component (if applicable):
4.18.0-rc4
How reproducible:
Always
Steps to Reproduce:
1. Provision a primary UDN (namespace scoped)
2. Read the created UDN data.
Actual results:
The MTU of the network is not available to the user.
Expected results:
The MTU of the network should be available in the UDN contents.
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
All
bootstrap API server should be terminated only after API is HA, we should wait for API to be available on at least 2 master nodes, these are the steps:
We should note the difference between a) the bootstrap node itself existing, and b) API being available on the bootstrap node. Today inside the cluster bootstrap, we remove the bootstrap API (b) as soon as two master nodes appear. This is what happens today on the bootstrap node:
a) create the static assets
b) wait for 2 master nodes to appear
c) remove the kube-apiserver from the bootstrap node
d) mark the bootstrap process as completed
But we already might have a time window where API is not available [starting from c, and until api is available on a master node].
cluster bootstrap executable is invoked here:
https://github.com/openshift/installer/blob/c534bb90b780ae488bc6ef7901e0f3f6273e2764/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L541
start --tear-down-early=false --asset-dir=/assets --required-pods="${REQUIRED_PODS}"
Then, cluster bootstrap removes the bootstrap API here: https://github.com/openshift/cluster-bootstrap/blob/bcd73a12a957ce3821bdfc0920751b8e3528dc98/pkg/start/start.go#L203-L209
but the wait for API to be HA is done here: https://github.com/openshift/installer/blob/c534bb90b780ae488bc6ef7901e0f3f6273e2764/data/data/bootstrap/files/usr/local/bin/report-progress.sh#L24
The wait should happen from within cluster-bootstrap, this PR moves the wait before cluster bootstrap tears down the bootstrap API/control plane
util.go:153: InfrastructureReady=False: WaitingOnInfrastructureReady(private-router load balancer is not provisioned: Error syncing load balancer: failed to ensure load balancer: error creating load balancer: "Throttling: Rate exceeded\n\tstatus code: 400, request id: 3cdc703c-49f3-489b-b1bf-60278686d946")
util.go:153: InfrastructureReady=False: WaitingOnInfrastructureReady(private-router load balancer is not provisioned: Error syncing load balancer: failed to ensure load balancer: error describing load balancer: "Throttling: Rate exceeded\n\tstatus code: 400, request id: 216792b1-767b-4388-a14b-be5b749a1434")
`aws-cloud-controller-manager` is very aggressive with AWS calls and across all mgmt clusters, we are getting throttled.
Step 7:
Unique names for AZURE_DISK_MI_NAME, AZURE_FILE_MI_NAME, IMAGE_REGISTRY_MI_NAME so they don't overlap with other names.
Step 10:
AKS_CLUSTER_NAME variable never made throughout steps (add to variables at the start).
Step 19:
"$CLUSTER_NAME" should be replaced with "$HC_NAME" from previous steps
https://hypershift-docs.netlify.app/how-to/azure/create-azure-cluster_on_aks/
Description of problem:
ingress-to-route controller does not provide any information about failed conversions from ingress to route. This is a big issue in environments heavily dependent on the ingress objects. The only way to find why a route is not created is guess and try as the only information one can get is that the route is not created.
Version-Release number of selected component (if applicable):
OCP 4.14
How reproducible:
100%
Steps to Reproduce:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: route.openshift.io/termination: passthrough name: hello-openshift-class namespace: test spec: ingressClassName: openshift-default rules: - host: ingress01-rhodain-test01.apps.rhodain03.sbr-virt.gsslab.brq2.redhat.com http: paths: - backend: service: name: myapp02 port: number: 8080 path: / pathType: Prefix tls: - {}
Actual results:
Route is not created and no error is logged
Expected results:
En error is provided in the events or at least in the controllers logs. The events are prefered as the ingress objects are mainly created by uses without cluster admin privileges.
Additional info:
Description of problem:
As admin, I can configure a defaultCertificate for the cluster domain (e.g. name.example.com) with SAN's for a custom domain (e.g. name.example.org). Cluster users can create application routes exposed on the custom domain (myapp.apps.example.org) without including a certificate in the rout e definition. As an admin, I cannot expose the console over the custom domain and rely on the defaultCertificate without specifying a `ingress.spec.componentRoutes.servingCertKeyPairSecret`.
Version-Release number of selected component (if applicable):
How reproducible:
100%
Steps to Reproduce:
1. configure defaultCertificate with SAN's for both .net and .org domains $ openssl x509 -in *.apps.name.example.net.crt -ext subjectAltName X509v3 Subject Alternative Name: DNS:*.apps.name.example.net, DNS:*.apps.name.example.org $ oc create configmap custom-ca --from-file=ca-bundle.crt=rootCA.crt -n openshift-config $ oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}' $ oc create secret tls custom-ingress-cert --cert=*.apps.name.example.net.crt --key=*.apps.name.example.net.key -n openshift-ingress $ oc patch ingresscontroller.operator default --type=merge -p '{"spec":{"defaultCertificate": {"name": "custom-ingress-cert"}}}' -n openshift-ingress-operator 2. create and expose user routes on default and custom domain, without specifying server certificate $ oc new-project san-multidom-wildcard $ kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname $ oc expose deployment/hello-node --port 9376 $ oc create route edge --service=hello-node hello-node-default $ oc create route edge --service=hello-node hello-node-custom --hostname=hello-node-custom-san-multidom-wildcard.apps.name.example.org $ curl --cacert rootCA.crt https://$(oc get route hello-node-default -ojsonpath='{.spec.host}') hello-node-8dd54cb99-27j5h $ curl --cacert rootCA.crt https://$(oc get route hello-node-custom -ojsonpath='{.spec.host}') hello-node-8dd54cb99-27j5h 3. Expose the console on a custom route but default domain, test and undo again: $ oc patch ingress.config.openshift.io/cluster --type=merge -p '{"spec":{"componentRoutes": [{"name": "console", "namespace": "openshift-console", "hostname": "console.apps.name.example.net"}]}}' -n openshift-ingress-operator ingress.config.openshift.io/cluster patched $ curl --cacert rootCA.crt -Lv console.apps.name.example.net >/dev/null <...> * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN: server did not agree on a protocol. Uses default. * Server certificate: * subject: CN=*.apps.name.example.net * start date: Feb 24 10:34:18 2025 GMT * expire date: Feb 24 10:34:18 2026 GMT * subjectAltName: host "console.apps.name.example.net" matched cert's "*.apps.name.example.net" * issuer: CN=MyRootCa * SSL certificate verify ok. * using HTTP/1.x } [5 bytes data] > GET / HTTP/1.1 > Host: console.apps.name.example.net <...> $ oc patch ingress.config.openshift.io/cluster --type=merge -p '{"spec":{"componentRoutes": []}}' -n openshift-ingress-operator $ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.name.example.net console https reencrypt/Redirect None downloads downloads-openshift-console.apps.name.example.net downloads http edge/Redirect None 4. Expose the console on the custom domain without specifying servingCertKeyPairSecret $ oc patch ingress.config.openshift.io/cluster --type=merge -p '{"spec":{"componentRoutes": [{"name": "console", "namespace": "openshift-console", "hostname": "console.apps.name.example.org"}]}}' -n openshift-ingress-operator ingress.config.openshift.io/cluster patched $ oc get route -n openshift-console NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.name.example.net console https reencrypt/Redirect None downloads downloads-openshift-console.apps.name.example.net downloads http edge/Redirect None $ oc logs -n openshift-console-operator deployment/console-operator <...> E0224 15:45:30.836226 1 base_controller.go:268] ConsoleRouteController reconciliation failed: secret reference for custom route TLS secret is not defined
Actual results:
console-operator rejects the route: ConsoleRouteController reconciliation failed: secret reference for custom route TLS secret is not defined
Expected results:
As an admin I expect the defaultCertificate with valid SAN's to be used for a custom console route. Currently I need to maintain the certificate in 2 different secret's (namespaces openshift-config & openshift-ingress).
Additional info:
Description of problem:
openshift virt allows hotplugging block volumes into it's pods, which relies on the fact that changing the cgroup corresponding to the pid of the container suffices. crun is test driving some changes it integrated recently; it's configuring two cgroups, `*.scope` and sub cgroup called `container` while before, the parent existed as sort of a no op (wasn't configured, so, all devices were allowed, for example) This results in the volume hotplug breaking since applying the device filter to the sub cgroup is not enough anymore
Version-Release number of selected component (if applicable):
4.18.0 RC2
How reproducible:
100%
Steps to Reproduce:
1. Block volume hotplug to VM 2. 3.
Actual results:
Failure
Expected results:
Success
Additional info:
https://kubevirt.io/user-guide/storage/hotplug_volumes/
Description of problem:
install private cluster with CAPI failed by MissingNodeRef
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-13-083421
How reproducible:
Always
Steps to Reproduce:
1.install private cluster based on bastion host 2.prepare the install-config.yaml: publish: Internal featureSet: CustomNoUpgrade featureGates: ["ClusterAPIInstall=true"] baseDomain: private-ibmcloud-2.qe.devcluster.openshift.com credentialsMode: Manual platform: ibmcloud: region: jp-tok networkResourceGroupName: ci-op-0yrhzx7l-ac1a3-rg vpcName: ci-op-0yrhzx7l-ac1a3-vpc controlPlaneSubnets: - ci-op-0yrhzx7l-ac1a3-control-plane-jp-tok-3-0 - ci-op-0yrhzx7l-ac1a3-control-plane-jp-tok-2-0 - ci-op-0yrhzx7l-ac1a3-control-plane-jp-tok-1-0 computeSubnets: - ci-op-0yrhzx7l-ac1a3-compute-jp-tok-3-0 - ci-op-0yrhzx7l-ac1a3-compute-jp-tok-2-0 - ci-op-0yrhzx7l-ac1a3-compute-jp-tok-1-0 resourceGroupName: ci-op-0yrhzx7l-ac1a3-rg controlPlane: name: master platform: ibmcloud: type: bx2-4x16 zones: [jp-tok-1, jp-tok-2, jp-tok-3] replicas: 3 compute: - name: worker platform: ibmcloud: type: bx2-4x16 zones: [jp-tok-1, jp-tok-2, jp-tok-3] replicas: 3 proxy: httpProxy: http://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@10.244.128.4:3128 httpsProxy: http://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@10.244.128.4:3128 networking: machineNetwork: - cidr: 10.244.0.0/16 3. install cluster failed
Actual results:
install failed in bootkube.json of log-bundle "stage": "resolve-api-url", "phase": "stage end", "result": "failure", "errorLine": "76 resolve_url /usr/local/bin/bootstrap-verify-api-server-urls.sh", "errorMessage": "Checking if api.maxu-capi9p.private-ibmcloud.qe.devcluster.openshift.com of type API_URL is resolvable\nStarting stage resolve-api-url\nUnable to resolve API_URL api.maxu-capi9p.private-ibmcloud.qe.devcluster.openshift.com" in Cluster-openshift-cluster-api-guests-ci-op-0yrhzx7l-ac1a3-2vc6j.yaml - type: ControlPlaneInitialized status: "False" severity: Info lasttransitiontime: "2024-12-17T09:39:31Z" reason: MissingNodeRef message: Waiting for the first control plane machine to have its status.nodeRef set
Expected results:
install succeed.
Additional info:
install External cluster with proxy succeed. install Internal cluster failed, ref Prow Job: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/59392/rehearse-59392-periodic-ci-openshift-verification-tests-master-installer-rehearse-4.19-installer-rehearse-ibmcloud-proxy-private-capi/1868948420899639296 https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/59392/rehearse-59392-periodic-ci-openshift-verification-tests-master-installer-rehearse-4.19-cucushift-installer-rehearse-ibmcloud-ipi-private-capi/1871051791659962368
Description of problem:
When building console, it is possible to include exposedModules that are not used by console-extensions.json
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Always
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Go to Nodes > Node details > Logs at mobile resolution and note the toolbar above the log is off the screen.
https://drive.google.com/file/d/1WDCSlEmbVqvrJ_QcXUQC_jXi_C1imDAt/view?usp=sharing
In the agent-based installer, the user must provide an apiVIP and ingressVIP on the baremetal and vsphere platforms.
In IPI, the VIPs must be provided for baremetal, but they are optional for vsphere.
There is currently no validation that checks that the VIPs are provided on vsphere, and if an install-config is provided that does not set platform.vsphere.apiVIP or platform.vsphere.ingressVIP, the installer crashes when trying to generate the agent ISO:
panic: runtime error: index out of range [0] with length 0 goroutine 1 [running]: github.com/openshift/installer/pkg/asset/agent/manifests.(*AgentClusterInstall).Generate(0xc0004ad350, 0x5?) /home/zbitter/openshift/installer/pkg/asset/agent/manifests/agentclusterinstall.go:182 +0xd79
The following test is failing with the updated 1.32 Kubernetes in OCP 4.19:
[It] [sig-node] PodRejectionStatus Kubelet should reject pod when the node didn't have enough resource
This test will be disabled temporarily to not block the rebase progress. This bug ticket is used to track the work to enable this test in OCP 4.19 again.
Description of problem:
When PowerVS deletes a cluster, it does it via pattern matching in the name. Limit the searches by resource group ID to prevent collisions.
Description of problem:
DEBUG Creating ServiceAccount for control plane nodes DEBUG Service account created for XXXXX-gcp-r4ncs-m DEBUG Getting policy for openshift-dev-installer DEBUG adding roles/compute.instanceAdmin role, added serviceAccount:XXXXX-gcp-r4ncs-m@openshift-dev-installer.iam.gserviceaccount.com member DEBUG adding roles/compute.networkAdmin role, added serviceAccount:XXXXX-gcp-r4ncs-m@openshift-dev-installer.iam.gserviceaccount.com member DEBUG adding roles/compute.securityAdmin role, added serviceAccount:XXXXX-gcp-r4ncs-m@openshift-dev-installer.iam.gserviceaccount.com member DEBUG adding roles/storage.admin role, added serviceAccount:XXXXX-gcp-r4ncs-m@openshift-dev-installer.iam.gserviceaccount.com member ERROR failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed during pre-provisioning: failed to add master roles: failed to set IAM policy, unexpected error: googleapi: Error 400: Service account XXXXX-gcp-r4ncs-m@openshift-dev-installer.iam.gserviceaccount.com does not exist., badRequest It appears that the Service account was created correctly. The roles are assigned to the service account. It is possible that there needs to be a "wait for action to complete" on the server side to ensure that this will all be ok.
Version-Release number of selected component (if applicable):
How reproducible:
Random. Appears to be a sync issue
Steps to Reproduce:
1. Run the installer for a normal GCP basic install 2. 3.
Actual results:
Installer fails saying that the Service Account that the installer created does not have the permissions to perform an action. Sometimes it takes numerous tries for this to happen (very intermittent).
Expected results:
Successful install
Additional info:
// NOTE: The nested `Context` containers inside the following `Describe` container are used to group certain tests based on the environments they demand.// NOTE: When adding a test-case, ensure that the test-case is placed in the appropriate `Context` container.// NOTE: The containers themselves are guaranteed to run in the order in which they appear.var _ = g.Describe("[sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set", g.Ordered, func() { defer g.GinkgoRecover()
o.SetDefaultEventuallyTimeout(15 * time.Minute) o.SetDefaultEventuallyPollingInterval(5 * time.Second)
r := &runner{}
In OCP Origin we have the above test playing with global variables for poll interval and poll timeout which is causing all other tests in origin to have flakes.
a networking test is failing because we are not polling correctly, this above test overrode the default poll value of 10ms and instead made it poll 5sec which caused the test to fail because out poll timeout was itself only 5seconds
Please don't use the global variables or maybe we can unset them after the test run is over?
Please note that this causes flakes that are hard to debug, we didn't know what was causing the poll interval to be 5seconds instead of the default 10ms.
3 payloads in a row have failed due to service-load-balancer-with-pdb-reused-connections mean disruption should be less than historical plus five standard deviations begining with 4.19.0-0.nightly-2025-02-07-133207
Tested the payload before the issue appeared as well as the noted payload via gangway and the issue was reproduced only in the jobs (started between 7:45 && 7:55 2/8/25)using the noted payload
Noted that cloud-provider-aws#98 is present in the payload the issues appeared to start. Will test revert to see if the issue clears or continues to reproduce.
Description of problem:
oc adm node-image create --pxe does not generate only pxe artifacts, but copies everything from the node-joiner pod. Also, the name of the pxe artifacts are not corrected (prefixed with agent, instead of node)
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. oc adm node-image create --pxe
Actual results:
All the node-joiner pods are copied. PXE artifacts name are wrong.
Expected results:
In the target folder, there should be only the following artifacts:
* node.x86_64-initrd.img * node.x86_64-rootfs.img * node.x86_64-vmlinuz
Additional info:
Description of problem:
byo-kms instlal failed with CAPI
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-24-213048
How reproducible:
always
Steps to Reproduce:
1.create a key 2.use the key in install-config.yaml publish: External featureSet: CustomNoUpgrade featureGates: ["ClusterAPIInstall=true"] baseDomain: ibmcloud.qe.devcluster.openshift.com credentialsMode: Manual platform: ibmcloud: region: jp-tok networkResourceGroupName: ci-op-7hcfbzfy-142dd-rg vpcName: ci-op-7hcfbzfy-142dd-vpc controlPlaneSubnets: - ci-op-7hcfbzfy-142dd-control-plane-jp-tok-3-0 - ci-op-7hcfbzfy-142dd-control-plane-jp-tok-2-0 - ci-op-7hcfbzfy-142dd-control-plane-jp-tok-1-0 computeSubnets: - ci-op-7hcfbzfy-142dd-compute-jp-tok-3-0 - ci-op-7hcfbzfy-142dd-compute-jp-tok-2-0 - ci-op-7hcfbzfy-142dd-compute-jp-tok-1-0 resourceGroupName: ci-op-7hcfbzfy-142dd defaultMachinePlatform: bootVolume: encryptionKey: "crn:v1:bluemix:public:kms:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861:4a6c67ca-7708-44c0-87fe-9eff2c111c00:key:4cf691f0-9cb1-4011-80b5-02aed0bbae60" or publish: External featureSet: CustomNoUpgrade featureGates: ["ClusterAPIInstall=true"] baseDomain: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX credentialsMode: Manual platform: ibmcloud: region: jp-tok networkResourceGroupName: ci-op-3py46711-142dd-rg vpcName: ci-op-3py46711-142dd-vpc controlPlaneSubnets: - ci-op-3py46711-142dd-control-plane-jp-tok-3-0 - ci-op-3py46711-142dd-control-plane-jp-tok-2-0 - ci-op-3py46711-142dd-control-plane-jp-tok-1-0 computeSubnets: - ci-op-3py46711-142dd-compute-jp-tok-3-0 - ci-op-3py46711-142dd-compute-jp-tok-2-0 - ci-op-3py46711-142dd-compute-jp-tok-1-0 resourceGroupName: ci-op-3py46711-142dd controlPlane: name: master platform: ibmcloud: type: bx2-4x16 zones: [jp-tok-1, jp-tok-2, jp-tok-3] bootVolume: encryptionKey: "crn:v1:bluemix:public:kms:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861:2aa5aefd-1168-4191-a525-c9dce0da520e:key:a95a2abe-c566-43f9-b523-b06698465601" replicas: 3 compute: - name: worker platform: ibmcloud: type: bx2-4x16 zones: [jp-tok-1, jp-tok-2, jp-tok-3] bootVolume: encryptionKey: "crn:v1:bluemix:public:kms:jp-tok:a/fdc2e14cf8bc4d53a67f972dc2e2c861:95afa81f-7486-49ce-a84f-f7b491d85c8c:key:2efdff99-4fdf-46ab-b424-b30f171094df" replicas: 3 3.install cluster with CAPI
Actual results:
install failed. in kube-apiserver.log rejected by webhook "vibmvpcmachine.kb.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"vibmvpcmachine.kb.io\" denied the request: IBMVPCMachine.infrastructure.cluster.x-k8s.io \"ci-op-7hcfbzfy-142dd-8vxvb-bootstrap\" is invalid: spec.bootVolume.sizeGiB: Invalid value: v1beta2.IBMVPCMachineSpec{Name:\"ci-op-7hcfbzfy-142dd-8vxvb-master-0\", CatalogOffering:(*v1beta2.IBMCloudCatalogOffering)(nil), PlacementTarget:(*v1beta2.VPCMachinePlacementTarget)(nil), Image:(*v1beta2.IBMVPCResourceReference)(0xc000d8a2c0), LoadBalancerPoolMembers:[]v1beta2.VPCLoadBalancerBackendPoolMember{v1beta2.VPCLoadBalancerBackendPoolMember{LoadBalancer:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a2e0)}, Pool:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a2f0)}, Port:6443, Weight:(*int64)(nil)}, v1beta2.VPCLoadBalancerBackendPoolMember{LoadBalancer:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a300)}, Pool:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a310)}, Port:22623, Weight:(*int64)(nil)}, v1beta2.VPCLoadBalancerBackendPoolMember{LoadBalancer:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a320)}, Pool:v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a330)}, Port:6443, Weight:(*int64)(nil)}}, Zone:\"jp-tok-1\", Profile:\"bx2-4x16\", BootVolume:(*v1beta2.VPCVolume)(0xc0014f20f0), ProviderID:(*string)(nil), PrimaryNetworkInterface:v1beta2.NetworkInterface{SecurityGroups:[]v1beta2.VPCResource{v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a340)}, v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a350)}, v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a360)}, v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a370)}, v1beta2.VPCResource{ID:(*string)(nil), Name:(*string)(0xc000d8a380)}}, Subnet:\"ci-op-7hcfbzfy-142dd-control-plane-jp-tok-1-0\"}, SSHKeys:[]*v1beta2.IBMVPCResourceReference(nil)}: valid Boot VPCVolume size is 10 - 250 GB", Reason:"Invalid", Details:(*v1.StatusDetails)(0xc0044719e0), Code:422}}
Expected results:
install succeed with CAPI using byo-kms
Additional info:
ref: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/59392/rehearse-59392-periodic-ci-openshift-verification-tests-master-installer-rehearse-4.19-cucushift-installer-rehearse-ibmcloud-ipi-byo-kms-capi/1871830778820694016
Description of problem:
Currently we are creating HFC for all BMH, doesn't mater if they are using ipmi or redfish, this can lead to misunderstanding.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
4.18 HyperShift operator's NodePool controller fails to serialize NodePool ConfigMaps that contain ImageDigestMirrorSet. Inspecting the code, it fails on NTO reconciliation logic, where only machineconfiguration API schemas are loaded into the YAML serializer: https://github.com/openshift/hypershift/blob/f7ba5a14e5d0cf658cf83a13a10917bee1168011/hypershift-operator/controllers/nodepool/nto.go#L415-L421
Version-Release number of selected component (if applicable):
4.18
How reproducible:
100%
Steps to Reproduce:
1. Install 4.18 HyperShift operator 2. Create NodePool with configuration ConfigMap that includes ImageDigestMirrorSet 3. HyperShift operator fails to reconcile NodePool
Actual results:
HyperShift operator fails to reconcile NodePool
Expected results:
HyperShift operator to successfully reconcile NodePool
Additional info:
Regression introduced by https://github.com/openshift/hypershift/pull/4717
Description of problem:
A new format library was introduced for CEL in 4.18, but, it is not usable in 4.18 due to upgrade checks put in place (to allow version skew between API servers and rollbacks). This means that the library is actually only presently usable in 4.19 once 1.32 ships. However, there are some issues we may face. We have a number of APIs in flight currently that would like to use this new library, we cannot get started on those features until this library is enabled. Some of those features would also like to be backported to 4.18. We also have risks on upgrades. If we decide to use this format library in any API that is upgraded prior to KAS, then during an upgrade, the CRD will be applied to the older version of the API server, blocking the upgrade as it will fail. By backporting the library (pretending it was introduced earlier, and then introducing it directly into 4.17), we can enable anything that installs post KAS upgrade to leverage this from 4.18 (solving those features asking for backports), and enable anything that upgrades pre-kas to actually leverage this in 4.19. API approvers will be responsible for making sure the libraries and upgrade compatibility are considered as new APIs are introduced. Presently, the library has had no bug fixes applied to the release-1.31 or release-1.32 branches upstream. The backport from 4.18 to 4.17 was clean bar some conflict in the imports that was easily resolved. So I'm confident that if we do need to backport any bug fixes, this should be straight forward. Any bugs in these libraries can be assigned to me (jspeed)
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When Swift is not available (from any manner: 403, 404, etc), Cinder is the backend for the Cluster Image Registry Operator with a PVC. The problem here is that we see an error that Swift is not available but then no PVC is being created.
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
Disable swiftoperator role for your user and no PVC will be created
Actual results:
E1122 15:37:26.301213 1 controller.go:379] unable to sync: unable to sync storage configuration: persistentvolumeclaims "image-registry-storage" not found, requeuing E1122 15:37:50.851275 1 swift.go:84] error listing swift containers: Expected HTTP response code [200 204 300[] when accessing [GET https://10.8.1.135:13808/v1/AUTH_6640775c6b5d4e5fa997fb9b85254da1/], but got 403 instead: <html><h1>Forbidden</h1><p>Access was denied to this resource.</p></html> I1122 15:37:50.858381 1 controller.go:294] object changed: *v1.Config, Name=cluster (metadata=false, spec=true): added:spec.storage.pvc.claim="image-registry-storage", changed:status.conditions.2.lastTransitionTime={"2024-11-22T15:37:26Z" -> "2024-11-22T15:37:50Z"} I1122 15:37:50.873526 1 controller.go:340] object changed: *v1.Config, Name=cluster (status=true): changed:metadata.generation={"12.000000" -> "11.000000"}, removed:metadata.managedFields.2.apiVersion="imageregistry.operator.openshift.io/v1", removed:metadata.managedFields.2.fieldsType="FieldsV1", removed:metadata.managedFields.2.manager="cluster-image-registry-operator", removed:metadata.managedFields.2.operation="Update", removed:metadata.managedFields.2.time="2024-11-22T15:37:50Z", changed:status.conditions.2.lastTransitionTime={"2024-11-22T15:37:26Z" -> "2024-11-22T15:37:50Z"}, changed:status.observedGeneration={"10.000000" -> "11.000000"} E1122 15:37:50.885488 1 controller.go:379] unable to sync: unable to sync storage configuration: persistentvolumeclaims "image-registry-storage" not found, requeuing
Expected results:
PVC should be created and therefore the operator to become healthy.
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
After the SMCP is automatically created by the controller, it can be manually deleted but is not automatically recreated if the default gatewayclass exists. If you also manually delete the default gatewayclass, you can automatically recreate both by recreating the default gatewayclass. Perhaps if SMCP is deleted, the gatewayclass should also be automatically deleted? Or, if the SMCP is deleted, check into why it doesn't get reconciled and recreated if the default gatewayclass exists.
Version-Release number of selected component (if applicable):
4.16
How reproducible:
Always
Steps to Reproduce:
1. Create the default gatewayclass, which automatically creates the SMCP 2. Delete the SMCP 3. Try to create a httpRoute
Actual results:
SMCP is not recreated and gatewayAPI function gets broken, e.g. httpRoute never attaches to gatewayClass and never works.
Expected results:
SMCP gets recreated if it is missing when the gatewayAPI controller reconciles.
Additional info:
If you delete the SMCP and default gatewayClass at the same time, and then create a new gatewayClass, it will also recreate the SMCP at that time.
Description of the problem:
ImageClusterInstall is timing out due to ibi-monitor-cm configmap missing. Seems this is a result of the installation-configuration.service failing on the spoke cluster when attempting to unmarshal the image-digest-sources.json file containing IDMS information for the spoke.
How reproducible:
100%
Steps to reproduce:
1. Configure a spoke with IBIO using IDMS
Additional information:
Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Post pivot operation has started" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="waiting for block device with label cluster-config or for configuration folder /opt/openshift/cluster-configuration" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Reading seed image info" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Reading seed reconfiguration info" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="/opt/openshift/setSSHKey.done already exists, skipping" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="/opt/openshift/pull-secret.done already exists, skipping" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Copying nmconnection files if they were provided" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="/opt/openshift/apply-static-network.done already exists, skipping" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Running systemctl restart [NetworkManager.service]" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Executing systemctl with args [restart NetworkManager.service]" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Setting new hostname target-0-0" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Executing hostnamectl with args [set-hostname target-0-0]" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Writing machine network cidr 192.168.126.0 into /etc/default/nodeip-configuration" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Setting new dnsmasq and forcedns dispatcher script configuration" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Running systemctl restart [dnsmasq.service]" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Executing systemctl with args [restart dnsmasq.service]" Jan 22 10:32:05 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:05" level=info msg="Executing bash with args [-c update-ca-trust]" Jan 22 10:32:07 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:07" level=info msg="/opt/openshift/recert.done already exists, skipping" Jan 22 10:32:07 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:07" level=info msg="No server ssh keys were provided, fresh keys already regenerated by recert, skipping" Jan 22 10:32:07 target-0-0 lca-cli[1216989]: 2025-01-22T10:32:07Z INFO post-pivot-dynamic-client Setting up retry middleware Jan 22 10:32:07 target-0-0 lca-cli[1216989]: 2025-01-22T10:32:07Z INFO post-pivot-dynamic-client Successfully created dynamic client Jan 22 10:32:07 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:07" level=info msg="Running systemctl enable [kubelet --now]" Jan 22 10:32:07 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:07" level=info msg="Executing systemctl with args [enable kubelet --now]" Jan 22 10:32:08 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:08" level=info msg="Start waiting for api" Jan 22 10:32:08 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:08" level=info msg="waiting for api" Jan 22 10:32:08 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:08" level=info msg="Deleting ImageContentSourcePolicy and ImageDigestMirrorSet if they exist" Jan 22 10:32:08 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:08" level=info msg="Deleting default catalog sources" Jan 22 10:32:08 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:08" level=info msg="Applying manifests from /opt/openshift/cluster-configuration/manifests" Jan 22 10:32:09 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:09" level=info msg="manifest applied: /opt/openshift/cluster-configuration/manifests/99-master-ssh.json" Jan 22 10:32:09 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:09" level=info msg="manifest applied: /opt/openshift/cluster-configuration/manifests/99-worker-ssh.json" Jan 22 10:32:09 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:09" level=error msg="failed apply manifests: failed to decode manifest image-digest-sources.json: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go value of type map[string]interface {}" Jan 22 10:32:09 target-0-0 lca-cli[1216989]: time="2025-01-22 10:32:09" level=fatal msg="Post pivot operation failed" Jan 22 10:32:09 target-0-0 systemd[1]: installation-configuration.service: Main process exited, code=exited, status=1/FAILURE
Seems to stem from the way imagedigestsources are being handled in the installer.
vs
Please review the following PR: https://github.com/openshift/operator-framework-operator-controller/pull/86
The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.
Differences in upstream and downstream builds impact the fidelity of your CI signal.
If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.
Closing this issue without addressing the difference will cause the issue to
be reopened automatically.
Description of problem:
Previously, failed task rus did not emit results, now they do but the UI still shows "No TaskRun results available due to failure" even though task run's status contains a result.
Version-Release number of selected component (if applicable):
4.14.3
How reproducible:
Always with a task run producing a result but failing afterwards
Steps to Reproduce:
1. Create the pipelinerun below 2. have a look on its task run
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: hello-pipeline
spec:
tasks:
- name: hello
taskSpec:
results:
- name: greeting1
steps:
- name: greet
image: registry.access.redhat.com/ubi8/ubi-minimal
script: |
#!/usr/bin/env bash
set -e
echo -n "Hello World!" | tee $(results.greeting1.path)
exit 1
results:
- name: greeting2
value: $(tasks.hello.results.greeting1)
Actual results:
No results in UI
Expected results:
One result should be displayed even though task run failed
Additional info:
Pipelines 1.13.0
Description of problem:
When running 4.18 installer QE full function test, following arm64 instances types are detected and tested passed, so append them in installer doc[1]: * StandardDpdsv6Family * StandardDpldsv6Family * StandardDplsv6Family * StandardDpsv6Family * StandardEpdsv6Family * StandardEpsv6Family [1] https://github.com/openshift/installer/blob/main/docs/user/azure/tested_instance_types_aarch64.md
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
"Cannot read properties of undefined (reading 'state')" Error in search tool when filtering Subscriptions while adding new Subscriptions
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. As an Administrator, go to Home -> Search and filter by Subscription component 2. Start creating subscriptions (bulk)
Actual results:
The filtered results will turn in "Oh no! Something went wrong" view
Expected results:
Get updated results every few seconds
Additional info:
If the view is reloaded -> Fix
Stack Trace:
TypeError: Cannot read properties of undefined (reading 'state') at L (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/subscriptions-chunk-89fe3c19814d1f6cdc84.min.js:1:3915) at na (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:58879) at Hs (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:111315) at Sc (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:98327) at Cc (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:98255) at _c (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:98118) at pc (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:95105) at https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:44774 at t.unstable_runWithPriority (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:289:3768) at Uo (https://console-openshift-console.apps.ods-qe-psi-02.osp.rh-ods.com/static/vendors~main-chunk-c416c917452592bcdcba.min.js:281:44551)
Description of problem:
CNO doesn't propagate HCP labels to 2nd level operands
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create hostedCluster with .spec.Labels
Actual results:
cloud-network-config-controller, multus-admission-controller, network-node-identity, ovnkube-control-plane-6dd8775f97-75f89 pods don't have the specified labels.
Expected results:
cloud-network-config-controller, multus-admission-controller, network-node-identity, ovnkube-control-plane-6dd8775f97-75f89 pods have the specified labels.
Additional info:
Description of problem:
Improving tests to remove the issue in the following helm test case Perform the helm chart upgrade for already upgraded helm chart : HR-08-TC02: Helm Release Perform the helm chart upgrade for already upgraded helm chart : HR-08-TC02 expand_less 37s {The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection. > Cannot read properties of undefined (reading 'repoName') When Cypress detects uncaught errors originating from your application it will automatically fail the current test.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Workloads - DeploymentConfigs - Add storage : 'container from image ' is in English.
Version-Release number of selected component (if applicable):
4.18.0-rc.6
How reproducible:
always
Steps to Reproduce:
1. Change web console UI into non en_US locale 2. Navigate to Workloads - DeploymentConfigs - <click on deployment config' - Actions - Add storage - <click on 'select specific container') 3. 'Container from image ' is in English
Actual results:
content is in English
Expected results:
content should be in selected language
Additional info:
Reference screenshot added
Description of problem:
1. when there is no UDNs, it just an button to create UDN from form 2. when tehre are UDNs, there are two options: create Cluster UDN and UDN
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Due to fundamental Kubernetes design, all OpenShift Container Platform updates between minor versions must be serialized. You must update from OpenShift Container Platform <4.y> to <4.y+1>, and then to <4.y+2>. You cannot update from OpenShift Container Platform <4.y> to <4.y+2> directly. However, administrators who want to update between two even-numbered minor versions can do so incurring only a single reboot of non-control plane hosts.
We should add a new precondition that enforces that policy, so cluster admins who run --to-image ... don't hop straight from 4.y.z to 4.(y+2).z' or similar without realizing that they were outpacing testing and policy.
The policy and current lack-of guard both date back to all OCP 4 releases, and since they're Kube-side constraints, they may date back to the start of Kube.
Every time.
1. Install a 4.y.z cluster.
2. Use --to-image to request an update to a 4.(y+2).z release.
3. Wait a few minutes for the cluster-version operator to consider the request.
4. Check with oc adm upgrade.
Update accepted.
Update rejected (unless it was forced), complaining about the excessively long hop.
Description of problem:
Upgrade to 4.18 is not working, because the machine-config update is stuck:
$ oc get co/machine-config
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
machine-config 4.17.0-0.nightly-2025-01-13-120007 True True True 133m Unable to apply 4.18.0-rc.4: error during syncRequiredMachineConfigPools: [context deadline exceeded, MachineConfigPool master has not progressed to latest configuration: controller version mismatch for rendered-master-ef1c06aa9aeedcebfa50569c3aa9472a expected a964f19a214946f0e5f1197c545d3805393d0705 has 3594c4b2eb42d8c9e56a146baea52d9c147721b0: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-826ddf793cf0a677228234437446740f, retrying]
The machine-config-controller shows the responsible for that:
$ oc logs -n openshift-machine-config-operator machine-config-controller-69f59598f7-57lkv [...] I0116 13:54:16.605692 1 drain_controller.go:183] node ostest-xgjnz-master-0: Drain failed. Drain has been failing for more than 10 minutes. Waiting 5 minutes then retrying. Error message from drain: error when evicting pods/"openstack-manila-csi-controllerplugin-6754c7589f-dwjtm" -n "openshift-manila-csi-driver": This pod has more than one PodDisruptionBudget, which the eviction subresource does not support.
There are 2 PDBs on the manila namespace:
$ oc get pdb -n openshift-manila-csi-driver NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE manila-csi-driver-controller-pdb N/A 1 1 80m openstack-manila-csi-controllerplugin-pdb N/A 1 1 134m
So a workaround is to remove the pdb openstack-manila-csi-controllerplugin-pdb.
Version-Release number of selected component (if applicable):
From 4.17.0-0.nightly-2025-01-13-120007 to 4.18.0-rc.4 on top of RHOS-17.1-RHEL-9-20241030.n.1
How reproducible:
Always
Steps to Reproduce:
1. Install latest 4.17
2. Update to 4.18, for example:
oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.18.0-rc.4 --allow-explicit-upgrade --force
Additional info: must-gather on private comment.
Description of problem:
All OCP components should be included in OCP's monitoring.
OLMV1 does not have this configuration.
References
Acceptence Criteria:
PS.: We should also check if we cannot add an e2e downstream test that covers it.
Description of problem:
Common cli flags are redefined for the copy and delete commands, causing inconsistencies. For example, `--log-level` for copy and `--loglevel` for delete. We should make it so common flags are shared and set only once.
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
delete accepts a `--log-level` flag. Other than that, no change in the available flags for each command
Additional info:
Description of problem:
During the EUS to EUS upgrade of a MNO cluster from 4.14.16 to 4.16.11 on baremetal, we have seen that depending on the custom configuration, like performance profile or container runtime config, one or more control plane nodes are rebooted multiple times. Seems that this is a race condition. When the first MachineConfig rendered is generated, the first Control Plane node start the reboot(the maxUnavailable is set to 1 on the master MCP), and at this moment a new MachineConfig render is generated, what means a second reboot. Once this first node is rebooted the second time, the rest of the Control Plane nodes are rebooted just once, because no more new MachineConfig renders are generated.
Version-Release number of selected component (if applicable):
OCP 4.14.16 > 4.15.31 > 4.16.11
How reproducible:
Perform the upgrade of a Multi Node OCP with a custom configuration like a performance profile or container runtime configuration (like force cgroups v1, or update runc to crun)
Steps to Reproduce:
1. Deploy on baremetal a MNO OCP 4.14 with a custom manifest, like the below: --- apiVersion: config.openshift.io/v1 kind: Node metadata: name: cluster spec: cgroupMode: v1 2. Upgrade the cluster to the next minor version available, for instance 4.15.31, make a partial upgrade pausing the worker Machine Config Pool. 3. Monitoring the upgrade process (cluster operators, Machine Configs, Machine Config Pools and nodes)
Actual results:
You will see that once almost all the Cluster Operators are in the 4.15.31 version, except the Machine Config Operator, at this moment review the MachineConfig reders that are generated for the master Machine Config Pool, and also monitor the nodes, to see that new MachineConfig render is generated once the first Control Plane node has been rebooted.
Expected results:
What is expected is that in a upgrade only one Machine Config Render is generated per Machine Config Pool, and only one reboot per node to finish the upgrade.
Additional info:
Any upgrade up to 4.15.{current-z}
Any non-Microshift cluster with an operator installed via OLM before upgrade to 4.15. After upgrading to 4.15, re-installing a previously uninstalled operator may also cause this issue.
OLM Operators can't be upgraded and may incorrectly report failed status.
Delete the resources associated with the OLM installation related to the failure message in the olm-operator.
A failure message similar to this may appear on the CSV:
InstallComponentFailed install strategy failed: rolebindings.rbac.authorization.k8s.io "openshift-gitops-operator-controller-manager-service-auth-reader" already exists |
The following resource types have been observed to encounter this issue and should be safe to delete:
Under no circumstances should a user delete a CustomResourceDefinition (CRD) if the same error occurs and names such a resource as data loss may occur. Note that we have not seen this type of resource named in the error from any of our users so far.
Labeling the problematic resources with olm.managed: "true" then restarting the olm-operator pod in the openshift-operator-lifecycle-manager namespace may also resolve the issue if the resource appears risky to delete.
Yes, functionality which worked in 4.14 may break after upgrading to 4.15.Not a regression, this is a new issue related to performance improvements added to OLM in 4.15
https://issues.redhat.com/browse/OCPBUGS-24009
Description of problem:
Missing translation for ""Read write once pod (RWOP)" ja and zh
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
We can input invalid value to zone field in gcpproviderSpec
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-10-16-094159
How reproducible:
Always
Steps to Reproduce:
1.Edit machineset with invalid zone value , scale machineset
Actual results:
Machineset edited successfully Machines stuck with blank status and do not fail miyadav@miyadav-thinkpadx1carbongen8:~/multifieldsgcp$ oc get machines NAME PHASE TYPE REGION ZONE AGE miyadav-1809g-7bdh4-master-0 Running n2-standard-4 us-central1 us-central1-a 62m miyadav-1809g-7bdh4-master-1 Running n2-standard-4 us-central1 us-central1-b 62m miyadav-1809g-7bdh4-master-2 Running n2-standard-4 us-central1 us-central1-c 62m miyadav-1809g-7bdh4-worker-a-9kmdv Running n2-standard-4 us-central1 us-central1-a 57m miyadav-1809g-7bdh4-worker-b-srj28 Running n2-standard-4 us-central1 us-central1-b 57m miyadav-1809g-7bdh4-worker-c-828v9 Running n2-standard-4 us-central1 us-central1-c 57m miyadav-1809g-7bdh4-worker-f-7d9bx 11m miyadav-1809g-7bdh4-worker-f-bcr7v Running n2-standard-4 us-central1 us-central1-f 20m miyadav-1809g-7bdh4-worker-f-tjfjk 7m3s
Expected results:
machines status can report failed status and the reason , may be timeout instead of waiting continuously .
Additional info:
logs are present in machine-controller "E1018 03:55:39.735293 1 controller.go:316] miyadav-1809g-7bdh4-worker-f-7d9bx: failed to check if machine exists: unable to verify project/zone exists: openshift-qe/us-central1-in; err: googleapi: Error 400: Invalid value for field 'zone': 'us-central1-in'. Unknown zone., invalid" the machines will be stuck in deletion also because of no status. for Invalid ProjectID - Errors in logs - urce project OPENSHIFT-QE. Details: [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developers console", "url": "https://console.developers.google.com" } ] }, { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "domain": "googleapis.com", "metadatas": { "consumer": "projects/OPENSHIFT-QE", "service": "compute.googleapis.com" }, "reason": "CONSUMER_INVALID" } ] , forbidden E1018 08:59:40.405238 1 controller.go:316] "msg"="Reconciler error" "error"="unable to verify project/zone exists: OPENSHIFT-QE/us-central1-f; err: googleapi: Error 403: Permission denied on resource project OPENSHIFT-QE.\nDetails:\n[\n {\n \"@type\": \"type.googleapis.com/google.rpc.Help\",\n \"links\": [\n {\n \"description\": \"Google developers console\",\n \"url\": \"https://console.developers.google.com\"\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"domain\": \"googleapis.com\",\n \"metadatas\": {\n \"consumer\": \"projects/OPENSHIFT-QE\",\n \"service\": \"compute.googleapis.com\"\n },\n \"reason\": \"CONSUMER_INVALID\"\n }\n]\n, forbidden" "controller"="machine-controller" "name"="miyadav-1809g-7bdh4-worker-f-dcnf5" "namespace"="openshift-machine-api" "object"={"name":"miyadav-1809g-7bdh4-worker-f-dcnf5","namespace":"openshift-machine-api"} "reconcileID"="293f9d09-1387-4702-8b67-2d209316585e"
must-gather- https://drive.google.com/file/d/1N--U8V3EfdEYgQUvK-fcrGxBYRDnzK1G/view?usp=sharing
ProjectID issue must-gather -https://drive.google.com/file/d/1lKNOu4eVmJJbo23gbieD5uVNtw_qF7p6/view?usp=sharing
Description of problem:
Konnectivity introduced a smarter readiness check with kubernetes-sigs/apiserver-network-proxy#485. It would be nice to do some better readiness and liveness check on startup.
Version-Release number of selected component (if applicable):
How reproducib
Steps to Reproduce:
-
Actual results
Expected results:
Additional info: Implementation in https://github.com/openshift/hypershift/pull/4829
Description of problem:
If hostDevices.deviceName has multiple types, the generated hostDevices.name may have duplicates.
Version-Release number of selected component (if applicable):
4.19 4.18 4.17
How reproducible:
100%
Steps to Reproduce:
apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool spec: platform: kubevirt: hostDevices: - count: 8 deviceName: nvidia.com/H20 - count;4 deviceName: nvidia.com/NVSwitch
Actual results:
kubevirtmachines yaml hostDevices: - deviceName: nvidia.com/H20 name :hostdevice-1 - deviceName: nvidia.com/H20 name :hostdevice-2 - deviceName: nvidia.com/H20 name :hostdevice-3 - deviceName: nvidia.com/H20 name :hostdevice-4 - deviceName: nvidia.com/NVSwitch name :hostdevice-1 - deviceName: nvidia.com/NVSwitch name :hostdevice-2
Expected results:
kubevirtmachines yaml hostDevices: - deviceName: nvidia.com/H20 name :hostdevice-1 - deviceName: nvidia.com/H20 name :hostdevice-2 - deviceName: nvidia.com/H20 name :hostdevice-3 - deviceName: nvidia.com/H20 name :hostdevice-4 - deviceName: nvidia.com/NVSwitch name :hostdevice-5 - deviceName: nvidia.com/NVSwitch name :hostdevice-6
Additional info:
Description of problem:
Tests are failing because {{oc image info quay.io/coreos/etcd:latest}} does not work anymore e.g. https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.17-upgrade-from-stable-4.16-e2e-gcp-ovn-rt-upgrade/1897807117629263872 [sig-imageregistry][Feature:ImageInfo] Image info should display information about images [apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
when delete the logs are wrong, still saying mirror is ongoing: Mirroring is ongoing. No errors
Version-Release number of selected component (if applicable):
./oc-mirror.rhel8 version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202411090338.p0.g0a7dbc9.assembly.stream.el9-0a7dbc9", GitCommit:"0a7dbc90746a26ddff3bd438c7db16214dcda1c3", GitTreeState:"clean", BuildDate:"2024-11-09T08:33:46Z", GoVersion:"go1.22.7 (Red Hat 1.22.7-1.module+el8.10.0+22325+dc584f75) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
when execute the delete, the logs still say mirror is ongoing: oc mirror delete --delete-yaml-file test/yinzhou/debug72708/working-dir/delete/delete-images.yaml docker://my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com --v2 --dest-tls-verify=false --force-cache-delete=true envar TEST_E2E detected - bypassing unshare2024/11/12 03:10:04 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/12 03:10:04 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/12 03:10:04 [INFO] : ⚙️ setting up the environment for you... 2024/11/12 03:10:04 [INFO] : 🔀 workflow mode: diskToMirror / delete 2024/11/12 03:10:04 [INFO] : 👀 Reading delete file... 2024/11/12 03:10:04 [INFO] : 🚀 Start deleting the images... 2024/11/12 03:10:04 [INFO] : images to delete 396 ✓ 1/396 : (0s) docker://registry.redhat.io/devworkspace/devworkspace-operator-bundle@sha256:5689ad3d80dea99cd842992523debcb1aea17b6db8dbd80e412cb2e… 2024/11/12 03:10:04 [INFO] : Mirroring is ongoing. No errors.
Actual results:
oc mirror delete --delete-yaml-file test/yinzhou/debug72708/working-dir/delete/delete-images.yaml docker://my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com --v2 --dest-tls-verify=false --force-cache-delete=true envar TEST_E2E detected - bypassing unshare2024/11/12 03:10:04 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/12 03:10:04 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/12 03:10:04 [INFO] : ⚙️ setting up the environment for you... 2024/11/12 03:10:04 [INFO] : 🔀 workflow mode: diskToMirror / delete 2024/11/12 03:10:04 [INFO] : 👀 Reading delete file... 2024/11/12 03:10:04 [INFO] : 🚀 Start deleting the images... 2024/11/12 03:10:04 [INFO] : images to delete 396 ✓ 1/396 : (0s) docker://registry.redhat.io/devworkspace/devworkspace-operator-bundle@sha256:5689ad3d80dea99cd842992523debcb1aea17b6db8dbd80e412cb2e… 2024/11/12 03:10:04 [INFO] : Mirroring is ongoing. No errors.
Expected results:
Show correct delete logs
Additional info:
Description of problem:
clusters running on OpenShift Virt (Agent Based Install) where I see `metal3-ramdisk-logs` container eating up a core, but the logs are empty:
oc adm top pod --sort-by=cpu --sum -n openshift-machine-api --containers POD NAME CPU(cores) MEMORY(bytes) metal3-55c9bc8ff4-nh792 metal3-ramdisk-logs 988m 1Mi metal3-55c9bc8ff4-nh792 metal3-httpd 1m 20Mi metal3-55c9bc8ff4-nh792 metal3-ironic 0m 121Mi cluster-baremetal-operator-5bf8bcbbdd-jvhq7 cluster-baremetal-operator 1m 25Mi
Version-Release number of selected component (if applicable):
4.17.12
How reproducible:
always
Steps to Reproduce:
Cluster is reachable on Red Hat VPN - reach out on slack to get access
Actual results:
logs are empty, but a core is consumed
Expected results:
container should be more or less idle
Additional info:
Description of problem:
When deploying Metallb the validatingwebhookconfigurations.admissionregistration is being deleted. It can take several minutes to come up. During this time it is possible to configure invalid frrconfigurations.
Version-Release number of selected component (if applicable):
How reproducible: Easily
Steps to Reproduce:
1. Verify validatingwebhookconfigurations.admissionregistration is deployed
2. Deploy metallb
3. Verify that validatingwebhookconfigurations.admissionregistration has been removed.
Actual results:
validatingwebhookconfigurations.admissionregistration is being removed
Expected results:
When metallb is deployed it should not remove the validatingwebhookconfigurations.admissionregistration
Additional info:
Description of problem:
The installation failed in the disconnected environment due to a failure to get controlPlaneOperatorImageLabels: failed to look up image metadata.
Version-Release number of selected component (if applicable):
4.19 4.18
How reproducible:
100%
Steps to Reproduce:
1.disconnected env 2.create agent hostedcluster
Actual results:
cluster can be ready
Expected results:
- lastTransitionTime: "2025-01-05T13:55:14Z" message: 'failed to get controlPlaneOperatorImageLabels: failed to look up image metadata for registry.ci.openshift.org/ocp/4.18-2025-01-04-031500@sha256:ba93b7791accfb38e76634edbc815d596ebf39c3d4683a001f8286b3e122ae69: failed to obtain root manifest for registry.ci.openshift.org/ocp/4.18-2025-01-04-031500@sha256:ba93b7791accfb38e76634edbc815d596ebf39c3d4683a001f8286b3e122ae69: manifest unknown: manifest unknown' observedGeneration: 2 reason: ReconciliationError status: "False" type: ReconciliationSucceeded
Additional info:
- mirrors: - virthost.ostest.test.metalkube.org:5000/localimages/local-release-image source: registry.build01.ci.openshift.org/ci-op-p2mqdwjp/release - mirrors: - virthost.ostest.test.metalkube.org:5000/localimages/local-release-image source: registry.ci.openshift.org/ocp/4.18-2025-01-04-031500 - mirrors: - virthost.ostest.test.metalkube.org:6001/openshifttest source: quay.io/openshifttest - mirrors: - virthost.ostest.test.metalkube.org:6001/openshift-qe-optional-operators source: quay.io/openshift-qe-optional-operators - mirrors: - virthost.ostest.test.metalkube.org:6001/olmqe source: quay.io/olmqe - mirrors: - virthost.ostest.test.metalkube.org:6002 source: registry.redhat.io - mirrors: - virthost.ostest.test.metalkube.org:6002 source: brew.registry.redhat.io - mirrors: - virthost.ostest.test.metalkube.org:6002 source: registry.stage.redhat.io - mirrors: - virthost.ostest.test.metalkube.org:6002 source: registry-proxy.engineering.redhat.com
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Console reports its internal version back in segment.io telemetry. This version is opaque and cannot easily be correlated back to a particular OpenShift version. We should use an OpenShift versions like 4.17.4 instead in segment.io events.
Description of problem:
Node was created today with worker label. It was labeled as a loadbalancer to match mcp selector. MCP saw the selector and moved to Updating but the machine-config-daemon pod isn't responding. We tried deleting the pod and it still didn't pick up that it needed to get a new config. Manually editing the desired config appears to workaround the issue but shouldn't be necessary.
Node created today: [dasmall@supportshell-1 03803880]$ oc get nodes worker-048.kub3.sttlwazu.vzwops.com -o yaml | yq .metadata.creationTimestamp '2024-04-30T17:17:56Z' Node has worker and loadbalancer roles: [dasmall@supportshell-1 03803880]$ oc get node worker-048.kub3.sttlwazu.vzwops.com NAME STATUS ROLES AGE VERSION worker-048.kub3.sttlwazu.vzwops.com Ready loadbalancer,worker 1h v1.25.14+a52e8df MCP shows a loadbalancer needing Update and 0 nodes in worker pool: [dasmall@supportshell-1 03803880]$ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE loadbalancer rendered-loadbalancer-1486d925cac5a9366d6345552af26c89 False True False 4 3 3 0 87d master rendered-master-47f6fa5afe8ce8f156d80a104f8bacae True False False 3 3 3 0 87d worker rendered-worker-a6be9fb3f667b76a611ce51811434cf9 True False False 0 0 0 0 87d workerperf rendered-workerperf-477d3621fe19f1f980d1557a02276b4e True False False 38 38 38 0 87d Status shows mcp updating: [dasmall@supportshell-1 03803880]$ oc get mcp loadbalancer -o yaml | yq .status.conditions[4] lastTransitionTime: '2024-04-30T17:33:21Z' message: All nodes are updating to rendered-loadbalancer-1486d925cac5a9366d6345552af26c89 reason: '' status: 'True' type: Updating Node still appears happy with worker MC: [dasmall@supportshell-1 03803880]$ oc get node worker-048.kub3.sttlwazu.vzwops.com -o yaml | grep rendered- machineconfiguration.openshift.io/currentConfig: rendered-worker-a6be9fb3f667b76a611ce51811434cf9 machineconfiguration.openshift.io/desiredConfig: rendered-worker-a6be9fb3f667b76a611ce51811434cf9 machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-worker-a6be9fb3f667b76a611ce51811434cf9 machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-worker-a6be9fb3f667b76a611ce51811434cf9 machine-config-daemon pod appears idle: [dasmall@supportshell-1 03803880]$ oc logs -n openshift-machine-config-operator machine-config-daemon-wx2b8 -c machine-config-daemon 2024-04-30T17:48:29.868191425Z I0430 17:48:29.868156 19112 start.go:112] Version: v4.12.0-202311220908.p0.gef25c81.assembly.stream-dirty (ef25c81205a65d5361cfc464e16fd5d47c0c6f17) 2024-04-30T17:48:29.871340319Z I0430 17:48:29.871328 19112 start.go:125] Calling chroot("/rootfs") 2024-04-30T17:48:29.871602466Z I0430 17:48:29.871593 19112 update.go:2110] Running: systemctl daemon-reload 2024-04-30T17:48:30.066554346Z I0430 17:48:30.066006 19112 rpm-ostree.go:85] Enabled workaround for bug 2111817 2024-04-30T17:48:30.297743470Z I0430 17:48:30.297706 19112 daemon.go:241] Booted osImageURL: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20b4937e8d107af19d8e39329e1767471b78ba6abd07b5a3e328dafd7b146858 (412.86.202311271639-0) 828584d351fcb58e4d799cebf271094d5d9b5c1a515d491ee5607b1dcf6ebf6b 2024-04-30T17:48:30.324852197Z I0430 17:48:30.324543 19112 start.go:101] Copied self to /run/bin/machine-config-daemon on host 2024-04-30T17:48:30.325677959Z I0430 17:48:30.325666 19112 start.go:188] overriding kubernetes api to https://api-int.kub3.sttlwazu.vzwops.com:6443 2024-04-30T17:48:30.326381479Z I0430 17:48:30.326368 19112 metrics.go:106] Registering Prometheus metrics 2024-04-30T17:48:30.326447815Z I0430 17:48:30.326440 19112 metrics.go:111] Starting metrics listener on 127.0.0.1:8797 2024-04-30T17:48:30.327835814Z I0430 17:48:30.327811 19112 writer.go:93] NodeWriter initialized with credentials from /var/lib/kubelet/kubeconfig 2024-04-30T17:48:30.327932144Z I0430 17:48:30.327923 19112 update.go:2125] Starting to manage node: worker-048.kub3.sttlwazu.vzwops.com 2024-04-30T17:48:30.332123862Z I0430 17:48:30.332097 19112 rpm-ostree.go:394] Running captured: rpm-ostree status 2024-04-30T17:48:30.332928272Z I0430 17:48:30.332909 19112 daemon.go:1049] Detected a new login session: New session 1 of user core. 2024-04-30T17:48:30.332935796Z I0430 17:48:30.332926 19112 daemon.go:1050] Login access is discouraged! Applying annotation: machineconfiguration.openshift.io/ssh 2024-04-30T17:48:30.368619942Z I0430 17:48:30.368598 19112 daemon.go:1298] State: idle 2024-04-30T17:48:30.368619942Z Deployments: 2024-04-30T17:48:30.368619942Z * ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20b4937e8d107af19d8e39329e1767471b78ba6abd07b5a3e328dafd7b146858 2024-04-30T17:48:30.368619942Z Digest: sha256:20b4937e8d107af19d8e39329e1767471b78ba6abd07b5a3e328dafd7b146858 2024-04-30T17:48:30.368619942Z Version: 412.86.202311271639-0 (2024-04-30T17:05:27Z) 2024-04-30T17:48:30.368619942Z LayeredPackages: kernel-devel kernel-headers 2024-04-30T17:48:30.368619942Z 2024-04-30T17:48:30.368619942Z ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20b4937e8d107af19d8e39329e1767471b78ba6abd07b5a3e328dafd7b146858 2024-04-30T17:48:30.368619942Z Digest: sha256:20b4937e8d107af19d8e39329e1767471b78ba6abd07b5a3e328dafd7b146858 2024-04-30T17:48:30.368619942Z Version: 412.86.202311271639-0 (2024-04-30T17:05:27Z) 2024-04-30T17:48:30.368619942Z LayeredPackages: kernel-devel kernel-headers 2024-04-30T17:48:30.368907860Z I0430 17:48:30.368884 19112 coreos.go:54] CoreOS aleph version: mtime=2023-08-08 11:20:41.285 +0000 UTC build=412.86.202308081039-0 imgid=rhcos-412.86.202308081039-0-metal.x86_64.raw 2024-04-30T17:48:30.368932886Z I0430 17:48:30.368926 19112 coreos.go:71] Ignition provisioning: time=2024-04-30T17:03:44Z 2024-04-30T17:48:30.368938120Z I0430 17:48:30.368931 19112 rpm-ostree.go:394] Running captured: journalctl --list-boots 2024-04-30T17:48:30.372893750Z I0430 17:48:30.372884 19112 daemon.go:1307] journalctl --list-boots: 2024-04-30T17:48:30.372893750Z -2 847e119666d9498da2ae1bd89aa4c4d0 Tue 2024-04-30 17:03:13 UTC—Tue 2024-04-30 17:06:32 UTC 2024-04-30T17:48:30.372893750Z -1 9617b204b8b8412fb31438787f56a62f Tue 2024-04-30 17:09:06 UTC—Tue 2024-04-30 17:36:39 UTC 2024-04-30T17:48:30.372893750Z 0 3cbf6edcacde408b8979692c16e3d01b Tue 2024-04-30 17:39:20 UTC—Tue 2024-04-30 17:48:30 UTC 2024-04-30T17:48:30.372912686Z I0430 17:48:30.372891 19112 rpm-ostree.go:394] Running captured: systemctl list-units --state=failed --no-legend 2024-04-30T17:48:30.378069332Z I0430 17:48:30.378059 19112 daemon.go:1322] systemd service state: OK 2024-04-30T17:48:30.378069332Z I0430 17:48:30.378066 19112 daemon.go:987] Starting MachineConfigDaemon 2024-04-30T17:48:30.378121340Z I0430 17:48:30.378106 19112 daemon.go:994] Enabling Kubelet Healthz Monitor 2024-04-30T17:48:31.486786667Z I0430 17:48:31.486747 19112 daemon.go:457] Node worker-048.kub3.sttlwazu.vzwops.com is not labeled node-role.kubernetes.io/master 2024-04-30T17:48:31.491674986Z I0430 17:48:31.491594 19112 daemon.go:1243] Current+desired config: rendered-worker-a6be9fb3f667b76a611ce51811434cf9 2024-04-30T17:48:31.491674986Z I0430 17:48:31.491603 19112 daemon.go:1253] state: Done 2024-04-30T17:48:31.495704843Z I0430 17:48:31.495617 19112 daemon.go:617] Detected a login session before the daemon took over on first boot 2024-04-30T17:48:31.495704843Z I0430 17:48:31.495624 19112 daemon.go:618] Applying annotation: machineconfiguration.openshift.io/ssh 2024-04-30T17:48:31.503165515Z I0430 17:48:31.503052 19112 update.go:2110] Running: rpm-ostree cleanup -r 2024-04-30T17:48:32.232728843Z Bootloader updated; bootconfig swap: yes; bootversion: boot.1.1, deployment count change: -1 2024-04-30T17:48:35.755815139Z Freed: 92.3 MB (pkgcache branches: 0) 2024-04-30T17:48:35.764568364Z I0430 17:48:35.764548 19112 daemon.go:1563] Validating against current config rendered-worker-a6be9fb3f667b76a611ce51811434cf9 2024-04-30T17:48:36.120148982Z I0430 17:48:36.120119 19112 rpm-ostree.go:394] Running captured: rpm-ostree kargs 2024-04-30T17:48:36.179660790Z I0430 17:48:36.179631 19112 update.go:2125] Validated on-disk state 2024-04-30T17:48:36.182434142Z I0430 17:48:36.182406 19112 daemon.go:1646] In desired config rendered-worker-a6be9fb3f667b76a611ce51811434cf9 2024-04-30T17:48:36.196911084Z I0430 17:48:36.196879 19112 config_drift_monitor.go:246] Config Drift Monitor started
Version-Release number of selected component (if applicable):
4.12.45
How reproducible:
They can reproduce in multiple clusters
Actual results:
Node stays with rendered-worker config
Expected results:
machineconfigpool updating should prompt a change to the desired config which the machine-config-daemon pod then updates node to
Additional info:
here is the latest must-gather where this issue is occuring: https://attachments.access.redhat.com/hydra/rest/cases/03803880/attachments/3fd0cf52-a770-4525-aecd-3a437ea70c9b?usePresignedUrl=true
Description of problem:
Console plugin details page is throwing error on some specific YAML
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-30-141716
How reproducible:
Always
Steps to Reproduce:
1. Create a ConsolePlugin with minimum required fields apiVersion: console.openshift.io/v1 kind: ConsolePlugin metadata: name: console-demo-plugin-two spec: backend: type: Service displayName: OpenShift Console Demo Plugin 2. Visit consoleplugin details page at /k8s/cluster/console.openshift.io~v1~ConsolePlugin/console-demo-plugin
Actual results:
2. We will see an error page
Expected results:
2. we should not show an error page since ConsolePlugin YAML has every required fields although they are not complete
Additional info:
Description of problem:
On "OCP 4.18.0-0.nightly-2024-12-14-152515" tried adding the spec but it was removed on reconcile with an "Admission Webhook" warning popup: ``` ConsolePlugin odf-console violates policy 299 - "unknown field \"spec.contentSecurityPolicy)"'' ```
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Try adding "contentSecurityPolicy" to any ConsolePlugin CR. 2. 3.
Actual results:
Spec is getting removed.
Expected results:
Spec should be supported by ConsolePlugin "v1".
Additional info:
Refer https://redhat-internal.slack.com/archives/C011BL0FEKZ/p1734339650501379 for more details.
Security Tracking Issue
Do not make this issue public.
Flaw:
Non-linear parsing of case-insensitive content in golang.org/x/net/html
https://bugzilla.redhat.com/show_bug.cgi?id=2333122
An attacker can craft an input to the Parse functions that would be processed non-linearly with respect to its length, resulting in extremely slow parsing. This could cause a denial of service.
Please review the following PR: https://github.com/openshift/operator-framework-operator-controller/pull/26
The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.
Differences in upstream and downstream builds impact the fidelity of your CI signal.
If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.
Closing this issue without addressing the difference will cause the issue to
be reopened automatically.
Description of problem:
Some permissions are missing when edge zones are specified in the install-config.yaml, probably those related to Carrier Gateways (but maybe more)
Version-Release number of selected component (if applicable):
4.16+
How reproducible:
always with minimal permissions
Steps to Reproduce:
1. 2. 3.
Actual results:
time="2024-11-20T22:40:58Z" level=debug msg="\tfailed to describe carrier gateways in vpc \"vpc-0bdb2ab5d111dfe52\": UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::460538899914:user/ci-op-girt7h2j-4515a-minimal-perm is not authorized to perform: ec2:DescribeCarrierGateways because no identity-based policy allows the ec2:DescribeCarrierGateways action"
Expected results:
All required permissions are listed in pkg/asset/installconfig/aws/permissions.go
Additional info:
See https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_installer/9222/pull-ci-openshift-installer-master-e2e-aws-ovn-edge-zones/1859351015715770368 for a failed min-perms install
Description of problem:
A-06-TC02, A-06-TC05, A-06-TC10 test cases are failing for create-from-git.feature file. The file requires an update
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Tests are failing with timeout error
Expected results:
Test should run green
Additional info:
Description of problem:
Alongside users which are updating resources often audit log analyzer should find resources updated often. Existing tests don't trigger when resource is being updated by different users or not namespaced
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
In the Assisted-Installer, discovery hosts use reverse DNS (rDNS) to set their hostname (in case the hostname is not in the DHCP). If the hostname cannot be resolved via DNS, the default hostname localhost.localdomain is assigned. During the discovery phase, there is a validation check that prevents installation if the hostname is localhost. To bypass this check, the system replaces unresolved hostnames with the MAC address of the NIC associated with the IP address. However, this replacement does not work for NICs configured with bonding. As a result, users must manually change the hostname to proceed with the installation.
Part of DEI efforts, may need input from testplatform and CRT.
Justin mentions there is a transparent way to do this with github and he has perms to do it. Update branch config in release repo.
Description of problem:
When a developer sandbox user logs in for the first time, OpenShift Console ignores the path in the URL the user clicked on (eg."/add/ns/krana-dev") and navigates the user to 'all projects' view instead of the namespace from the user URL.
This also happens in any other OCP instance, the behavior is apparently tied to the user console settings ConfigMap. When a user logs in for the first time, the console user setting ConfigMap is not created yet. When the CM is not created (and doesn't have the console.lastNamespace field populated yet), the UI overrides the namespace navigation from "/add/ns/krana-dev" to "add/all-namespaces". For the second login, when the CM with the field is present already, the URL navigation works as expected.
Version-Release number of selected component (if applicable):
not sure if previous or future versions are impacted. we are currently on 4.17, so can confirm that Openshift 4.17 is impacted.
How reproducible:
reproducible for first time user.
Steps to Reproduce:
In any OCP cluster 1. Provision new cluster (or use any cluster you haven't logged in before) 2. Navigate to <cluster-url>/add/ns/default (you can also use any other namespace already present in the cluster) 3. You will land in the "all-namespaces" view and the URL will be changed to /add/all-namespaces
In Developer Sandbox 1. make sure you're not active user on https://developers.redhat.com/developer-sandbox 2. if you are already active, you can either create a new user or ask anyone in #forum-dev-sandbox to deactivate your user. 3. Go to https://console.redhat.com/openshift/sandbox click on "Get Started" and then click on "launch" button in the Red Hat OpenShift tile. This redirects you to your namespace /add/ns/<username>-dev 4. OpenShift console doesn't navigate you to the "*-dev" namespace, but navigates you to all-namespaces view. The change is also visible in the url.
Actual results:
User is navigated to all-namespaces.
This behavior has negative impact on the UX in the Developer Sandbox. A good percentage of users of DevSandbox are trying to learn more about openshift and often don't know what to do and where to go. Not landing in the right namespace can discourage users from experimenting with OpenShift and can also break any redirection from learning paths or other RH materials.
Expected results:
The path in the URL is not ignored and not overwritten, and the user is navigated to the namespace from the URL
Additional info:
Description of problem:
It appears that during cluster creation, when MAPI starts up and begins to manage Machines in an IPI deployed cluster on IBM Cloud, it can detect an unhealthy CP node and attempt a one time replacement of that node, effectively destroying the cluster.
Version-Release number of selected component (if applicable):
4.19
How reproducible:
< 10%
Steps to Reproduce:
A potential way to reproduce, but is a relatively small timing window to meet 1. Create a new IPI cluster on IBM Cloud 2. Attempt to Stop a CP node once MAPI starts deploying, to allow MAPI to believe a CP node needs replacement. (This is an extremely tight window) Replication may not be possible manually, and only just by luck.
Actual results:
One or more CP nodes get replaced during cluster creation, destroying etcd and other deployment of critical CP workloads, effectively breaking the cluster.
Expected results:
Successful cluster deployment.
Additional info:
Back when OCP was using RHEL 8 (RHCOS base), a known bug with NetworkManager caused the loss of the assigned IP a new IBM Cloud Instance (VSI), resulting in the new Instance never being able to start up with dracut and Ignition to work with the MCO. Because of this bug with NetworkManager, a fix was created to force a one time replacement of that VSI by MAPI, to try to resolve this issue, and allow the VSI to complete bringup and report into the cluster. https://issues.redhat.com/browse/OCPBUGS-1327 Unfortunately at that time, this appeared to only affect worker nodes, but in a few cases, it appears it is now affecting CP nodes as well, which was not the intention. I will add some logs and details with what I think is proof that MAPI is performing this same replacement on CP nodes.
Description of problem:
MCC pod reports a panic when we install a cluster using the PR that makes OCL v1 GA. The panic doesn't seem related to OCL it self, but with the ManagedBootImages functionality. Nevertheless, we only have been able to reproduce it when we use the mentioned PR to build the cluster.
Version-Release number of selected component (if applicable):
We only have been able to reproduce the issue while using this PR to create the cluster https://github.com/openshift/api/pull/2192
How reproducible:
2 out of 2. Not sure if always.
Steps to Reproduce:
1. Use https://github.com/openshift/api/pull/2192 to create payload image 2. Install the cluster using the image generated in step 1
Actual results:
MCC reports a panic in its 'previous' logs $ oc logs -p machine-config-controller-5b4b8d7d94-bmhdh ...... I0206 09:55:53.678676 1 kubelet_config_controller.go:222] Re-syncing all kubelet config controller generated MachineConfigs due to apiServer cluster change E0206 09:55:53.678746 1 template_controller.go:245] "Unhandled Error" err="couldn't get ControllerConfig on dependency callback &%!w(errors.StatusError=errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ListMeta:v1.ListMeta{SelfLink:\"\", ResourceVersion:\"\", Continue:\"\", RemainingItemCount:(*int64)(nil)}, Status:\"Failure\", Message:\"controllerconfig.machineconfiguration.openshift.io \\\"machine-config-controller\\\" not found\", Reason:\"NotFound\", Details:(*v1.StatusDetails)(0xc0009a0de0), Code:404}})" I0206 09:55:53.679532 1 reflector.go:368] Caches populated for *v1.ClusterVersion from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0206 09:55:53.680703 1 template_controller.go:198] Re-syncing ControllerConfig due to apiServer cluster change I0206 09:55:53.680747 1 reflector.go:368] Caches populated for *v1.ConfigMap from k8s.io/client-go/informers/factory.go:160 I0206 09:55:53.686268 1 machine_set_boot_image_controller.go:221] configMap coreos-bootimages added, reconciling enrolled machine resources panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3a87c1a] goroutine 372 [running]: github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image.(*Controller).syncMAPIMachineSets(0xc000dc8900, {0x43aeec5, 0x17}) /go/src/github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image/machine_set_boot_image_controller.go:329 +0xba github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image.(*Controller).addConfigMap.func1() /go/src/github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image/machine_set_boot_image_controller.go:225 +0x25 created by github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image.(*Controller).addConfigMap in goroutine 347 /go/src/github.com/openshift/machine-config-operator/pkg/controller/machine-set-boot-image/machine_set_boot_image_controller.go:225 +0x145
Expected results:
No panic should be reported
Additional info:
Please review the following PR: https://github.com/openshift/operator-framework-operator-controller/pull/51
The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.
Differences in upstream and downstream builds impact the fidelity of your CI signal.
If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.
Closing this issue without addressing the difference will cause the issue to
be reopened automatically.
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following test:
operator conditions kube-apiserver
Significant regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 99.46% to 90.48%.
Sample (being evaluated) Release: 4.19
Start Time: 2025-02-27T00:00:00Z
End Time: 2025-03-06T12:00:00Z
Success Rate: 90.48%
Successes: 57
Failures: 6
Flakes: 0
Base (historical) Release: 4.18
Start Time: 2025-02-04T00:00:00Z
End Time: 2025-03-06T12:00:00Z
Success Rate: 99.46%
Successes: 183
Failures: 1
Flakes: 0
View the test details report for additional context.
A selection of baremetal jobs now fail due to the following error:
time="2025-03-06T06:08:48Z" level=info msg="Extracted /usr/bin/k8s-tests-ext.gz for tag hyperkube from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c21b9b37064cdf3297b44e747ea86ca92269e1e133f5042f8bd3ef9aaeda9e6e (disk size 119671560, extraction duration 7.038773908s)" time="2025-03-06T06:08:48Z" level=info msg="Listing images for \"k8s-tests-ext\"" error: encountered errors while listing tests: failed running '/alabama/.cache/openshift-tests/registry_ci_openshift_org_ocp_release_4_19_0-0_nightly-2025-03-06-043021_f1a1994f6536/k8s-tests-ext list': exit status 1 Output: E0306 06:08:48.426861 116 test_context.go:584] Unknown provider "baremetal". The following providers are known: aws azure gce kubemark local openstack skeleton vsphere {"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:84","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun","level":"error","msg":"Error executing test process","severity":"error","time":"2025-03-06T06:08:48Z"} error: failed to execute wrapped command: exit status 1 INFO[2025-03-06T06:08:50Z] Step e2e-metal-ipi-serial-ovn-ipv6-baremetalds-e2e-test failed after 22s.
This surfaces in the operator conditions tests for kube and openshift APIs as we fail fast, before these can stabilize, and thus the cluster does not appear ready when we expect it to be.
It is believed to have been caused by https://github.com/openshift/kubernetes/pull/2229
We should be using marketplace images when testing aks as that's what will be used in production.
Description of problem:
Installing 4.17 agent-based hosted cluster on bare-metal with IPv6 stack in disconnected environment. We cannot install MetalLB operator on the hosted cluster to expose openshift router and handle ingress because the openshift-marketplace pods that extract the operator bundle and the relative pods are in Error state. They try to execute the following command but cannot reach the cluster apiserver: opm alpha bundle extract -m /bundle/ -n openshift-marketplace -c b5a818607a7a162d7f9a13695046d44e47d8127a45cad69c0d8271b2da945b1 -z INFO[0000] Using in-cluster kube client config Error: error loading manifests from directory: Get "https://[fd02::1]:443/api/v1/namespaces/openshift-marketplace/configmaps/b5a818607a7a162d7f9a13695046d44e47d8127a45cad69c0d8271b2da945b1": dial tcp [fd02::1]:443: connect: connection refused In our hosted cluster fd02::1 is the clusterIP of the kubernetes service and the endpoint associated to the service is [fd00::1]:6443. By debugging the pods we see that connection to clusterIP is refused but if we try to connect to its endpoint the connection is established and we get 403 Forbidden: sh-5.1$ curl -k https://[fd02::1]:443 curl: (7) Failed to connect to fd02::1 port 443: Connection refused sh-5.1$ curl -k https://[fd00::1]:6443 { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"", "reason": "Forbidden", "details": {}, "code": 403 This issue is happening also in other pods in the hosted cluster which are in Error or in CrashLoopBackOff, we have similar error in their logs, e.g.: F1011 09:11:54.129077 1 cmd.go:162] failed checking apiserver connectivity: Get "https://[fd02::1]:443/apis/coordination.k8s.io/v1/namespaces/openshift-service-ca-operator/leases/service-ca-operator-lock": dial tcp [fd02::1]:443: connect: connection refused IPv6 disconnected 4.16 hosted cluster with same configuration was installed successfully and didn't show this issue, and neither IPv4 disconnected 4.17. So the issue is with IPv6 stack only.
Version-Release number of selected component (if applicable):
Hub cluster: 4.17.0-0.nightly-2024-10-10-004834 MCE 2.7.0-DOWNANDBACK-2024-09-27-14-52-56 Hosted cluster: version 4.17.1 image: registry.ci.openshift.org/ocp/release@sha256:e16ac60ac6971e5b6f89c1d818f5ae711c0d63ad6a6a26ffe795c738e8cc4dde
How reproducible:
100%
Steps to Reproduce:
1. Install MCE 2.7 on 4.17 IPv6 disconnected BM hub cluster 2. Install 4.17 agent-based hosted cluster and scale up the nodepool 3. After worker nodes are installed, attempt to install MetalLB operator to hanlde ingress
Actual results:
MetalLB operator cannot be installed because pods cannot connect to the cluster apiserver.
Expected results:
Pods in the cluster can connect to apiserver.
Additional info:
Description of problem:
In Agent-Base Installation, storage network on all nodes will be also configured. If both VLAN interfaces in the same L3 switch used as gateways for compute cluster management and storage networks of the OCP cluster have arp-proxy enabled, then the IP collisions validation will report errors. The reason why the IP collision validation fails is that the validation seems to send the arp-request from both interfaces bond0.4082 and bond1.2716 for all addresses used for compute cluster management and storage networks.
Version-Release number of selected component (if applicable):
4.14
How reproducible:
Always
Steps to Reproduce:
1. Install cluster using Agent-base with proxy enabled. 2. Configur both VLAN interfaces in the same L3 switch, used as gateways for compute cluster management and storage networks of the OCP cluster have arp-proxy enabled
Actual results:
IP collision validation fails, the validation seems to send the arp-request from both interfaces bond0.4082 and bond1.2716 for all addresses used for compute cluster management and storage networks. The validation seems to trigger the arp-requests sent out from all NICs of the nodes. If the gateways connected to the different NICs of the node have arp-proxy configured, then the IP collision failure will be observed.
Expected results:
Validation should pass in Arp Porxy Scenario
Additional info:
In arp-proxy scenario, the arp reply is not needed from the NIC which IP address is not in the same subnet as the destination IP address of the arp-request. Generally speaking, when the node tries to communicate to any destination IP address in the same subnet as the IP address of one NIC, it will only send out the arp-request from this NIC. So even arp-proxy in this case, it will cause any issue.
Description of problem:
The installation with aws installation fails when the SCP has the value for AssociatePublicIpAddress set to False. The IAM user is not able to create new EC2 instances i.e. the worker nodes are not getting created. However the bootstrap and Master nodes gets created. The below logs can be observed in the machine-api controller logs : 2024/10/31 16:05:28 failed to create instance: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::<account-id>:assumed-role/<role-name> is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:ap-southeast-1:<account-id>:network-interface/* with an explicit deny in a service control policy. Encoded authorization failure message: <encoded-message>
Version-Release number of selected component (if applicable):
4.17
How reproducible:
Always
Steps to Reproduce:
1. Set the value of AssociatePublicIpAddress: False inside SCP. 2. Perform a normal IPI aws installation with IAM user which has the above SCP applied. 3. Observe that the workers are not getting created.
Actual results:
Expected results:
Additional info:
Description of problem:
When switching the container runtime using a ContainerRuntimeConfig, conmon using the wrong,unupdated --root path for the runtime.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-20-221830
Steps to Reproduce:
1.Apply containerRuntimeConfig requesting runc as the default runtime. apiVersion: machineconfiguration.openshift.io/v1 kind: ContainerRuntimeConfig metadata: name: enable-runc-worker spec: containerRuntimeConfig: defaultRuntime: runc machineConfigPoolSelector: matchLabels: machineconfiguration.openshift.io/role: worker 2. Wait for MCP to turn updated 3. oc debug the worker node 4. ps -ef | grep -E 'crun|runc'
Actual results:
The last command will yield a list of processes like:
root 1230048 1 0 08:33 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/9917f380f2e7a88cd8cf09023f13a7500014cd81cfd97213b29bcd35258f886d/userdata -c ........../userdata -r /usr/bin/runc --runtime-arg --root=/run/crun --socket-dir-path /var/run/crio --syslog -u 9917f380f2e7a88cd8cf09023f13a7500014cd81cfd97213b29bcd35258f886d -s -t
Upon closer inspection, the --root=/run/crun value does not align with the -r /usr/bin/runc runtime specification.
Expected results:
All processes should be using runc with correct --root value: root 1230048 1 0 08:33 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/9917f380f2e7a88cd8cf09023f13a7500014cd81cfd97213b29bcd35258f886d/userdata -c ........../userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 9917f380f2e7a88cd8cf09023f13a7500014cd81cfd97213b29bcd35258f886d -s -t
Additional info:
Description of problem:
node-joiner --pxe does not rename pxe artifacts
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. node-joiner --pxe
Actual results:
agent*.* artifacts are generated in the working dir
Expected results:
In the target folder, there should be only the following artifacts:
* node.x86_64-initrd.img * node.x86_64-rootfs.img * node.x86_64-vmlinuz * node.x86_64.ipxe (if required)
Additional info:
Description of problem:
The CUDN creation view doesnt prevent namespace-selector with no rules
Version-Release number of selected component (if applicable):
4.18
How reproducible:
100%
Steps to Reproduce:
1.In the UI, go to CUDN creation view, create CDUN with empty namespace-selector. 2. 3.
Actual results:
The CUDN will select all namespaces exist in the cluster, including openshift-* namespaces. Affecting the cluster system components including the api-server, etcd.
Expected results:
I expect the UI to block creating CUDN with namespace-selector that has zero rules.
Additional info:
Description of problem:
We’re unable to find a stable and accessible OAuth-proxy image, which is causing a bug that we haven’t fully resolved yet. Krzys made a PR to address this , but it’s not a complete solution since the image path doesn’t seem consistently available. Krzys tried referencing the OAuth-proxy image from the OpenShift openshift namespace, but it didn’t work reliably.There’s an imagestream for OAuth-proxy in the openshift namespace, which we might be able to reference in tests, but not certain of the correct Docker URL format for it. Also, it’s possible that there are permission issues, which could be why the image isn’t accessible when referenced this way.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
The monitoring-plugin is still using Patternfly v4; it needs to be upgraded to Patternfly v5. This major version release deprecates components in the monitoring-plugin. These components will need to be replaced/removed to accommodate the version update.
We need to remove the deprecated components from the monitoring plugin, extending the work from CONSOLE-4124
Work to be done:
Description of problem:
Finding the Console plugins list can be challenging as it is not in the primary nav. We should add it to the primary nav so it is easier to find.
Description of problem:
In an effort to ensure all HA components are not degraded by design during normal e2e test or upgrades, we are collecting all operators that are blipping Degraded=True during any payload job run. This card captures machine-config operator that blips Degraded=True during some ci job runs. Example Job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.18-e2e-aws-ovn-techpreview-serial/1843561357304139776 Reasons associated with the blip: MachineConfigDaemonFailed or MachineConfigurationFailed For now, we put an exception in the test. But it is expected that teams take action to fix those and remove the exceptions after the fix go in. Exception is defined here: https://github.com/openshift/origin/blob/e5e76d7ca739b5699639dd4c500f6c076c697da6/pkg/monitortests/clusterversionoperator/legacycvomonitortests/operators.go#L109 See linked issue for more explanation on the effort.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
If the install is performed with an AWS user missing the `ec2:DescribeInstanceTypeOfferings`, the installer will use a hardcoded instance type from the set of non-edge machine pools. This can potentially cause the edge node to fail during provisioning, since the instance type doesn't take into account edge/wavelength zones support. Because edge nodes are not needed for the installation to complete, the issue is not noticed by the installer, only by inspecting the status of the edge nodes.
Version-Release number of selected component (if applicable):
4.16+ (since edge nodes support was added)
How reproducible:
always
Steps to Reproduce:
1. Specify an edge machine pool in the install-config without an instance type 2. Run the install with an user without `ec2:DescribeInstanceTypeOfferings` 3.
Actual results:
In CI the `node-readiness` test step will fail and the edge nodes will show errorMessage: 'error launching instance: The requested configuration is currently not supported. Please check the documentation for supported configurations.' errorReason: InvalidConfiguration
Expected results:
Either 1. the permission is always required when instance type is not set for an edge pool; or 2. a better instance type default is used
Additional info:
Example CI job: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_installer/9230/pull-ci-openshift-installer-master-e2e-aws-ovn-edge-zones/1862140149505200128
Description of problem:
When try to a full catalog will failed with error: 2024/11/21 02:55:48 [ERROR] : unable to rebuild catalog docker://registry.redhat.io/redhat/redhat-operator-index:v4.17: filtered declarative config not found
Version-Release number of selected component (if applicable):
oc-mirror version W1121 02:59:58.748933 61010 mirror.go:102] ⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.2.0-alpha.1-324-gbae91d5", GitCommit:"bae91d55", GitTreeState:"clean", BuildDate:"2024-11-20T02:06:04Z", GoVersion:"go1.23.0", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. mirror ocp with full==true for catalog : cat config.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 full: true oc-mirror -c config.yaml docker://localhost:5000 --workspace file://full-catalog --v2
Actual results:
oc-mirror -c config.yaml docker://localhost:5000 --workspace file://full-catalog --v22024/11/21 02:55:27 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/21 02:55:27 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/21 02:55:27 [INFO] : ⚙️ setting up the environment for you... 2024/11/21 02:55:27 [INFO] : 🔀 workflow mode: mirrorToMirror 2024/11/21 02:55:27 [INFO] : 🕵️ going to discover the necessary images... 2024/11/21 02:55:27 [INFO] : 🔍 collecting release images... 2024/11/21 02:55:27 [INFO] : 🔍 collecting operator images... ⠦ (20s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.17 2024/11/21 02:55:48 [WARN] : error parsing image registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9 : registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9 unable to parse image co ✓ (20s) Collecting catalog registry.redhat.io/redhat/redhat-operator-index:v4.17 2024/11/21 02:55:48 [WARN] : registry.redhat.io/openshift4/ose-kube-rbac-proxy-rhel9 unable to parse image correctly : tag and digest are empty : SKIPPING 2024/11/21 02:55:48 [WARN] : [OperatorImageCollector] gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1@sha256:d4883d7c622683b3319b5e6b3a7edfbf2594c18060131a8bf64504805f875522 has both tag and digest : using digest to pull, but tag only for mirroring 2024/11/21 02:55:48 [INFO] : 🔍 collecting additional images... 2024/11/21 02:55:48 [INFO] : 🔍 collecting helm images... 2024/11/21 02:55:48 [INFO] : 🔂 rebuilding catalogs 2024/11/21 02:55:48 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/11/21 02:55:48 [ERROR] : unable to rebuild catalog docker://registry.redhat.io/redhat/redhat-operator-index:v4.17: filtered declarative config not found
Expected results:
no error
Additional info:
Description of problem:
Need to bump k8s to v0.31.1 in 4.18
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Due to internal conversion of network name, which replaces '-' with '.', non-conflicting networks get conflicting internal names, e.g. for node switches. https://github.com/ovn-kubernetes/ovn-kubernetes/blob/cb682053aafcbe8e35dd8bb705c3c9d2bd72b821/go-controller/pkg/util/multi_network.go#L725-L728
For example, NADs with network name 'tenant-blue' and 'tenant.blue' will have such conflict. In UDN case, network name is built as <namespace>.<name>, therefore 2 UDNs with namespace+name => network will have a conflict:
'test' + 'tenant-blue' => test.tenant-blue
'test-tenant' + 'blue' => test-tenant.blue
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
Installations on Google Cloud require the constraints/compute.vmCanIpForward to not be enforced. Error: time=\"2024-12-16T10:20:27Z\" level=debug msg=\"E1216 10:20:27.538990 97 reconcile.go:155] \\"Error creating an instance\\" err=\\"googleapi: Error 412: Constraint constraints/compute.vmCanIpForward violated for projects/ino-paas-tst. Enabling IP forwarding is not allowed for projects/ino-paas-tst/zones/europe-west1-b/instances/paas-osd-tst2-68r4m-master-0., conditionNotMet\\" controller=\\"gcpmachine\\" controllerGroup=\\"infrastructure.cluster.x-k8s.io\\" controllerKind=\\"GCPMachine\\" GCPMachine=\\"openshift-cluster-api-guests/paas-osd-tst2-68r4m-master-0\\" namespace=\\"openshift-cluster-api-guests\\" reconcileID=\\"3af74f44-96fe-408a-a0ad-9d63f023d2ee\\" name=\\"paas-osd-tst2-68r4m-master-0\\" zone=\\"europe-west1-b\\"\"
Version-Release number of selected component (if applicable):
4.17
How reproducible:
Every Time
Steps to Reproduce:
1. Enable constraints/compute.vmCanIpForward on a project 2. Install OSD 4.17 on that project 3. Installation fails
Actual results:
Installation fails
Expected results:
Installation does not fail
Additional info:
More info in the attachments
Description of problem:
The current "description" annotation of the CoreDNSErrorsHigh alert doesn't provide much context about what's happening and what to do next.
Version-Release number of selected component (if applicable):
4.14.39
How reproducible:
always
Steps to Reproduce:
Deploy OCP and use unhealthy nameservers as upstream for CoreDNS ! Or control DNS pods placement without complying with the standard architecture...
Actual results:
CoreDNSErrorsHigh but the description annotation doesn't provide lots of information about what to do next.
Expected results:
Detailed instructions.
Additional info:
https://access.redhat.com/solutions/5917331
Description of problem:
egress-router-binary-copy, an init-container of multus-additional-cni-plugins runs on the egress-router-cni container image. This image is still based on the 4.16 image.
I'm not sure about other possible implications, but when run on confidential TDX instances, this causes a segfault that 4.19 images do not cause. This is possibly due to a known glibc issue.
Version-Release number of selected component (if applicable): 4.19
How reproducible: 100%
Steps to Reproduce:
1. Install a cluster on TDX instances (being developed).
2. egress-router-binary-copy segfaults.
3.
Actual results:
egress-router-binary-copy container image is based on 4.16
It segfaults when running on a TDX confidential cluster node.
Expected results:
egress-router-binary-copy container image is based on 4.19.
It runs normally on a TDX confidential cluster node.
Additional info:
Fix proposal available in https://github.com/openshift/egress-router-cni/pull/89
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
Some control plane pods are not receiving the tolerations specified using the hypershift create cluster azure --toleration command.
Steps to Reproduce:
1. Create Azure HC with hypershift create cluster azure --toleration key=foo-bar.baz/quux,operator=Exists --toleration=key=fred,operator=Equal,value=foo,effect=NoSchedule --toleration key=waldo,operator=Equal,value=bar,effect=NoExecute,tolerationSeconds=3600 ...
2. Run the following script against the MC NAMESPACE="clusters-XXX" PODS="$(oc get pods -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}')" for POD in $PODS; do echo "Checking pod: $POD" tolerations="$(oc get po -n $NAMESPACE $POD -o jsonpath='{.spec.tolerations}' | jq -c --sort-keys)" failed="false" if ! grep -q '"key":"foo-bar.baz/quux","operator":"Exists"' <<< "$tolerations"; then echo "No foo-bar.baz/quux key found" >&2 failed="true" fi if ! grep -q '"effect":"NoSchedule","key":"fred","operator":"Equal","value":"foo"' <<< "$tolerations"; then echo "No fred key found" >&2 failed="true" fi if ! grep -q '"effect":"NoExecute","key":"waldo","operator":"Equal","tolerationSeconds":3600,"value":"bar"' <<< "$tolerations"; then echo "No waldo key found" >&2 failed="true" fi if [[ $failed == "true" ]]; then echo "Tolerations: " echo "$tolerations" | jq --sort-keys fi echo done
3. Take note of the results
Actual results (and dump files):
https://drive.google.com/drive/folders/1MQYihLSaK_9WDq3b-H7vx-LheSX69d2O?usp=sharing
Expected results:
All specified tolerations are propagated to all control plane pods.
Description of problem:
OWNERS file updated to include prabhakar and Moe as owners and reviewers
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
This is to fecilitate easy backport via automation
See https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_api/1997/pull-ci-openshift-api-master-e2e-aws-ovn-hypershift/1886643435440443392 and https://redhat-internal.slack.com/archives/C01CQA76KMX/p1738645154113239?thread_ts=1738642163.306719&cid=C01CQA76KMX for example
: TestCreateCluster/Main/EnsureCustomLabels expand_less 0s {Failed === RUN TestCreateCluster/Main/EnsureCustomLabels util.go:1954: expected pods [aws-cloud-controller-manager-67bfd8cbc5-8dvwp, aws-ebs-csi-driver-controller-6f8786f899-9mjgz, aws-ebs-csi-driver-operator-6c5b795565-gv8dh, capi-provider-76b766f99-2gjhh, catalog-operator-5b478677d5-thwqt, certified-operators-catalog-84cb448c4d-zn4h9, cloud-credential-operator-68bb55c657-2fxm8, cloud-network-config-controller-589ff54f97-7sbp6, cluster-api-75855d4758-9pnq9, cluster-image-registry-operator-58475b676c-6xsrf, cluster-network-operator-56c777dbc4-26gng, cluster-node-tuning-operator-5697596c7d-z7gt8, cluster-policy-controller-c8d6d6b6c-lllbn, cluster-storage-operator-75ddcdd454-94sb7, cluster-version-operator-5c5677754d-hqxxq, community-operators-catalog-75cb5645bd-5zz89, control-plane-operator-7886f776d4-sth5w, control-plane-pki-operator-5f6fb9f6fd-pqv9z, csi-snapshot-controller-7479d75445-wchhd, csi-snapshot-controller-operator-f657b76f8-9g7fl, dns-operator-7bbfbd7568-jj8zt, etcd-0, hosted-cluster-config-operator-866bc4b498-4m7nc, ignition-server-54bd9b464-5vwsb, ignition-server-proxy-7966cfbcf-vx4qc, ingress-operator-74cdbc59f8-xvmvs, konnectivity-agent-5775fbfd6f-kwx85, kube-apiserver-6f4f79b98c-qj7vs, kube-controller-manager-5999d8597b-hbnnp, kube-scheduler-745d45554b-6gn6c, machine-approver-566b56d5cc-9xflj, multus-admission-controller-698f686986-2dj4r, network-node-identity-5b995c6748-pnc7f, oauth-openshift-649d7467c5-nszbt, olm-operator-6cfc78d86f-xspl9, openshift-apiserver-6678b9d68-rt77m, openshift-controller-manager-59dc766b95-nm8xs, openshift-oauth-apiserver-68b8f7fbdc-54sj4, openshift-route-controller-manager-fdc875484-bqnlf, ovnkube-control-plane-654b979d95-54zgf, packageserver-57dfb7b586-mmdpt, redhat-marketplace-catalog-79f86885f-lxdt5, redhat-operators-catalog-5764567c54-mhfg9, router-7799588d9b-jrnq4] to have label hypershift-e2e-test-label=test --- FAIL: TestCreateCluster/Main/EnsureCustomLabels (0.02s)
this test seems to be failing across all PRs?
Description of problem:
When using oc-mirror version 2, it creates idms-oc-mirror.yaml and itms-oc-mirror.yaml after a successful mirroring process(mirror to mirror). However, On a second attempt, I've added an operator in the imageset-config.yaml and then re-run the same command with a --dry-run, I noticed that these resource files got deleted.
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. Create a imageset-config.yaml 2. Run oc-mirror --v2, mirror images 3. Run with a --dry-run option
Actual results:
the existing yaml in the ocp_mirror/working-dir/cluster-resources/ directory when running with a --dry-run option
Expected results:
--dry-run supposed not to delete files
Additional info:
tested with 4.17 oc-mirror version
Description of problem:
When hosted clusters are delayed in deleting, their dedicated request serving nodes may have already been removed, but the configmap indicating that the node pair label is in use remains. Placeholder pods are currently getting scheduled on new nodes that have these pair labels. When the scheduler tries to use these new nodes, it says it can't because there is a configmap associating the pair label with a cluster that is in the process of deleting.
Version-Release number of selected component (if applicable):
4.19
How reproducible:
sometimes
Steps to Reproduce:
1. In a size tagging dedicated request serving architecture, create hosted cluster(s). 2. Place an arbitrary finalizer on the hosted cluster(s) so it cannot be deleted. 3. Delete the hosted clusters 4. Look at placeholder pods in hypershift-request-serving-node-placeholders
Actual results:
some placeholder pods are scheduled on nodes that correspond to fleet manager pairs taken up by the deleting clusters
Expected results:
no placeholder pods are scheduled on nodes that correspond to hosted clusters.
Additional info:
https://github.com/openshift/oc-mirror/blob/main/v2/docs/operator-filtering-investigation.md#annex---acceptance-criteria-set-for-v2 is obsolete since work on https://github.com/sherine-k/catalog-filter/pull/7 was done as part of https://issues.redhat.com/browse/OCPBUGS-43731 and https://issues.redhat.com/browse/CLID-235
We are aiming to find containers that are restarting more than 3 times in the progress of a test.
If this link code rots, you just need to search for "restarted .* times" in the openshift CI for 4.18.
PS I took a guess at how owns the openshift/ingress-operator so please assign once you find the correct owner.
We are adding an exclusion for this container but we ask that you look into fixing this.
Description of problem:
"kubernetes.io/cluster/${infra_id}:shared" should be attached to the private subnets, but it's missing now.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-02-11-060301
How reproducible:
Always
Steps to Reproduce:
1. Create a cluster with byo-VPC 2. 3.
Actual results:
"kubernetes.io/cluster/${infra_id}:shared" tag was not attached to private subnets.
Expected results:
"kubernetes.io/cluster/${infra_id}:shared" tag presents in private subnets.
Additional info:
Checking the recent change, it looks like caused by the typo issue [1] in: https://github.com/openshift/installer/pull/9430 (4.18.0-0.nightly-2025-02-07-182732) https://github.com/openshift/installer/pull/9445 (4.17.0-0.nightly-2025-02-11-131208) [1] https://github.com/openshift/installer/pull/9430/files#diff-703b72d7af46ab11b2fd79c7073598468fdf038db0666628521f9e6923dc78daR72 This issue blocks C2S/SC2S cluster testing: level=info msg=Credentials loaded from the "default" profile in file "/tmp/secret/aws_temp_creds" level=info msg=Creating infrastructure resources... level=fatal msg=failed to fetch Cluster: failed to generate asset "Cluster": could not add tags to subnets: MissingParameter: The request must contain the parameter resourceIdSet level=fatal msg= status code: 400, request id: d6ffad31-f2a7-4a1d-883f-c44bbc1ee1c7
Description of problem:
In some cases, installer may need to call ReplaceRoute action, but with minimum permission, this is not allowed: ... time="2025-02-22T06:44:35Z" level=debug msg="E0222 06:44:35.976720 218 awscluster_controller.go:319] \"failed to reconcile network\" err=<" time="2025-02-22T06:44:35Z" level=debug msg="\tfailed to replace outdated route on route table \"rtb-0f3322786d2a7a9fc\": UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::301721915996:user/ci-op-n3z38rfl-21543-minimal-perm-installer is not authorized to perform: ec2:ReplaceRoute on resource: arn:aws:ec2:us-east-1:301721915996:route-table/rtb-0f3322786d2a7a9fc because no identity-based policy allows the ec2:ReplaceRoute action. Encoded authorization failure message: HIDDEN" time="2025-02-22T06:44:35Z" level=debug msg="\t\tstatus code: 403, request id: 405cded7-daae-49b6-aa38-ed2d2fbceb75" time="2025-02-22T06:44:35Z" level=debug msg=" > controller=\"awscluster\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"AWSCluster\" AWSCluster=\"openshift-cluster-api-guests/ci-op-n3z38rfl-21543-6h797\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-n3z38rfl-21543-6h797\" reconcileID=\"24aae75e-bd3e-4705-a88a-e69bfa0b4974\" cluster=\"openshift-cluster-api-guests/ci-op-n3z38rfl-21543-6h797\"" time="2025-02-22T06:44:35Z" level=debug msg="I0222 06:44:35.976749 218 recorder.go:104] \"Operation ReplaceRoute failed with a credentials or permission issue\" logger=\"events\" type=\"Warning\" object={\"kind\":\"AWSCluster\",\"namespace\":\"openshift-cluster-api-guests\",\"name\":\"ci-op-n3z38rfl-21543-6h797\",\"uid\":\"dfdcd50e-f0d5-4456-b5d6-c26de8f2c2ce\",\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1beta2\",\"resourceVersion\":\"448\"} reason=\"UnauthorizedOperation\"" time="2025-02-22T06:44:35Z" level=debug msg="I0222 06:44:35.976773 218 recorder.go:104] \"Failed to replace outdated route on managed RouteTable \\\"rtb-0f3322786d2a7a9fc\\\": UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::301721915996:user/ci-op-n3z38rfl-21543-minimal-perm-installer is not authorized to perform: ec2:ReplaceRoute on resource: arn:aws:ec2:us-east-1:301721915996:route-table/rtb-0f3322786d2a7a9fc because no identity-based policy allows the ec2:ReplaceRoute action. Encoded authorization failure message: [HIDDEN]\\n\\tstatus code: 403, request id: 405cded7-daae-49b6-aa38-ed2d2fbceb75\" logger=\"events\" type=\"Warning\" object={\"kind\":\"AWSCluster\",\"namespace\":\"openshift-cluster-api-guests\",\"name\":\"ci-op-n3z38rfl-21543-6h797\",\"uid\":\"dfdcd50e-f0d5-4456-b5d6-c26de8f2c2ce\",\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1beta2\",\"resourceVersion\":\"448\"} reason=\"FailedReplaceRoute\"" time="2025-02-22T06:44:36Z" level=debug msg="E0222 06:44:36.035951 218 controller.go:324] \"Reconciler error\" err=<" ...
Version-Release number of selected component (if applicable):
4.18.1
How reproducible:
Occasionally
Steps to Reproduce:
it's not always reproducible, in this case, the install-config is like: ... fips: true controlPlane: platform: aws: zones: - us-east-1c - us-east-1b type: m6i.xlarge architecture: amd64 name: master replicas: 3 compute: - platform: aws: zones: - us-east-1c - us-east-1b type: m5.xlarge architecture: amd64 name: worker replicas: 3 - name: edge architecture: amd64 hyperthreading: Enabled replicas: 1 platform: aws: zones: [us-east-1-atl-1a] baseDomain: qe.devcluster.openshift.com platform: aws: region: us-east-1 ...
Actual results:
Install failed.
Expected results:
Install succeeds.
Additional info:
It looks like the issue comes from the upstream CAPA [1], so all CAPI installs (4.16+) might be affected. [1] https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/4e912b4e4d1f855abf9b5194acaf9f31b5763c57/pkg/cloud/services/network/routetables.go#L160
Description of problem:
During bootstrapping we're running into the following scenario: 4 members: master 0, 1 and 2 (are full voting) and bootstrap (torn down/dead member) revision rollout causes 0 to restart and leaves you with 2/4 healthy, which means quorum-loss. This causes apiserver unavailability during the installation and should be avoided.
Version-Release number of selected component (if applicable):
4.17, 4.18 but is likely a longer standing issue
How reproducible:
rarely
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
apiserver should not return any errors
Additional info:
Beginning with payloads 4.19.0-0.ci-2025-02-12-111027 and 4.19.0-0.nightly-2025-02-13-083804 aggregate azure jobs started failing on test [sig-network][Feature:EgressFirewall] when using openshift ovn-kubernetes should ensure egressfirewall is created [Suite:openshift/conformance/parallel] - not all the time, but enough to fail aggregation.
We believe this is due to an added step in https://github.com/openshift/origin/pull/29414/files and are testing the revert in https://github.com/openshift/origin/pull/29540
Description of problem:
Missing translations for "PodDisruptionBudget violated" string
Code:
"count PodDisruptionBudget violated_one": "count PodDisruptionBudget violated_one", "count PodDisruptionBudget violated_other": "count PodDisruptionBudget violated_other", | |
Code:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
ConsolePlugin CRD is missing connect-src CSP directives, which need to be added to its API and ported into both console-operator and console itself.
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Set connect-src CSP directive in the ConsolePlugin CR. 2. Save changes 3.
Actual results:
API server will error out with unknown DirectiveType type
Expected results:
Added CSP directives should be saved as part of the updated ConsolePlugin CR, and aggregated CSP directives should be set as part of the bridge server response header, containing the added CSP directives
Additional info:
As noted in https://github.com/openshift/api/pull/1963#discussion_r1910598226, we are currently ignoring tags set on a port in a MAPO Machine or MachineSet. This appears to be a mistake that we should correct.
All versions of MAPO that use CAPO under the hood are affected.
n/a
n/a
n/a
n/a
See https://github.com/openshift/api/pull/1963#discussion_r1910598226
Description of problem:
Once Machine Config Pool goes into degraded state due to incorrect machine config, pool doesn't recover from this state even after updating machine config with correct config.
Version-Release number of selected component (if applicable):
4.16.0, applicable for previous versions as well.
How reproducible:
Always
Steps to Reproduce:
1. Create Machine Config with invalid extension name. 2. Wait for Machine Config Pool goes into degraded state. 3. Update Machine Config with correct extension name or delete Machine Config.
Actual results:
Machine Config Pool doesn't recover and always in degraded state.
Expected results:
Machine Config Pool must be restored and degraded condition must be set with false.
Additional info:
conditions: - lastTransitionTime: "2024-05-16T11:15:51Z" message: "" reason: "" status: "False" type: RenderDegraded - lastTransitionTime: "2024-05-27T15:05:50Z" message: "" reason: "" status: "False" type: Updated - lastTransitionTime: "2024-05-27T15:07:41Z" message: 'Node worker-1 is reporting: "invalid extensions found: [ipsec11]"' reason: 1 nodes are reporting degraded status on sync status: "True" type: NodeDegraded - lastTransitionTime: "2024-05-27T15:07:41Z" message: "" reason: "" status: "True" type: Degraded - lastTransitionTime: "2024-05-27T15:05:50Z" message: All nodes are updating to MachineConfig rendered-worker-c585a5140738aa0a2792cf5f25b4eb20 reason: "" status: "True" type: Updating
Description of problem:
Improving the OpenShift installer for Azure Deployments to comply PCI-DSS/BAFIN regluations. The OpenShift installer utilizes thegithub.com/hashicorp/terraform-provider-azurermmodule which in versions < 4 have the cross_tenant_replication_enabled parameter set to true. Two options available to fix this are: 1. adjust the OpenShift installer to create the resourceStorageAccount [1] as requested with the default set to FALSE 2. upgrade the OpenShift installer module version used of terraform-provider-azurerm to 4.x were this parameter now defaults to FALSE [1] https://github.com/hashicorp/terraform-provider-azurerm/blob/57cd1c81d557a49e18b2f49651a4c741b465937b/internal/services/storage/storage_account_resource.go#L212 This security voilation blocks using and scaling Clusters in Public cloud environments for the Banking and Financial industry which need to comply to BAFIN and PCI-DSS regulations.4. List any affected packages or components.OpenShift Installer 4.xCompliance Policy Azure https://learn.microsoft.com/en-us/azure/storage/common/security-controls-policy.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Automate Configuration related Test cases of AMD Last Level Cache Feature.
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
User Story:
As an OpenShift Engineer I want Create PR for machine-api refactoring of feature gate parameters so We need to pull out the logic from Neil's PR that removes individual feature gate parameters to use the new FeatureGate mutable map.
Description:
< Record any background information >
Acceptance Criteria:
< Record how we'll know we're done >
Other Information:
< Record anything else that may be helpful to someone else picking up the card >
issue created by splat-bot
Description of problem:
In 4.19 HyperShift hosted cluster, kubeadmin login always fails. 4.18 HyperShift hosted cluster (MGTM cluster and hosted cluster both are 4.18) doesn't have the issue.
Version-Release number of selected component (if applicable):
MGMT cluster version and hosted cluster version both are 4.19.0-0.nightly-2025-02-11-161912
How reproducible:
Always
Steps to Reproduce:
1. Launch 4.19 HyperShift management cluster and a hosted cluster on it. 2. Run kubeadmin login against HCP: $ export KUBECONFIG=/path/to/mgmt/kubeconfig $ oc get secret kubeadmin-password -n clusters-hypershift-ci-334742 -o 'jsonpath={ .data.password }' | base64 -d WJt9r-xxxxx-xxxxx-fpAMT $ export KUBECONFIG=/path/to/hosted-cluster/kubeconfig $ oc login -u kubeadmin -p "WJt9r-xxxxx-xxxxx-fpAMT" Login failed (401 Unauthorized) Verify you have provided the correct credentials.
Actual results:
HyperShift hosted cluster kubeadmin login always fails.
Expected results:
Success.
Additional info:
If I then configured htpasswd IDP for the hosted cluster, htpasswd user can login successfully.
Description of problem:
The LB name should be yunjiang-ap55-sk6jl-ext-a6aae262b13b0580, rather than ending with ELB service endpoint (elb.ap-southeast-5.amazonaws.com): failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed provisioning resources after infrastructure ready: failed to find HostedZone ID for NLB: failed to list load balancers: ValidationError: The load balancer name 'yunjiang-ap55-sk6jl-ext-a6aae262b13b0580.elb.ap-southeast-5.amazonaws.com' cannot be longer than '32' characters\n\tstatus code: 400, request id: f8adce67-d844-4088-9289-4950ce4d0c83 Checking the tag value, the value of Name key is correct: yunjiang-ap55-sk6jl-ext
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-30-141716
How reproducible:
always
Steps to Reproduce:
1. Deploy a cluster on ap-southeast-5 2. 3.
Actual results:
The LB can not be created
Expected results:
Create a cluster successfully.
Additional info:
No such issues on other AWS regions.
Description of problem:
This is a "clone" of https://issues.redhat.com/browse/OCPBUGS-38647 Multus CNI's delete doesn't delete Pods if the API server is not up.
Version-Release number of selected component (if applicable):
MicroShift ships Multus in 4.16+
How reproducible:
100%
Steps to Reproduce:
1. Start MicroShift 2. Cleanup microshift (sudo microshift-cleanup-data --all) 3. Run `sudo crictl pods`
Actual results:
There are Pods running
Expected results:
There should be no Pods running
Additional info:
Primary problem seems to be that func GetPod() [https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/pkg/multus/multus.go#L510] doesn't return - it takes too long and perhaps CRI-O doesn't wait that long: 2024-11-26T08:45:51Z [debug] CmdDel: &{8cc9b938cc29474eeca8593c1c22a2f258a4794bdc1e2bfa2ffb2572a1d2671a /var/run/netns/c5786f46-2faf-43b7-84d5-8f1e1cf5d3b4 eth0 IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-service-ca;K8S_POD_NAME=service-ca-7cf6f558c6-7b4t7;K8S_POD_INFRA_CONTAINER_ID=8cc9b938cc29474eeca8593c1c22a2f258a4794bdc1e2bfa2ffb2572a1d2671a;K8S_POD_UID=1f986d36-414b-4fb6-892c-06ebb86e6f19 /run/cni/bin:/usr/libexec/cni [...]}, <nil>, <nil> 2024-11-26T08:45:51Z [debug] GetPod for [openshift-service-ca/service-ca-7cf6f558c6-7b4t7] starting
Full log: https://drive.google.com/file/d/1zUyJ7DXdwV0sogSxMZ66BgGiMgQYSmMp/view?usp=sharing
Description of problem:
When "users in ns/openshift-... must not produce too many applies" test flakes it doesn't output a useful output: it `{ details in audit log}`. Instead it should be ``` {user system:serviceaccount:openshift-infra:serviceaccount-pull-secrets-controller had 43897 applies, check the audit log and operator log to figure out why user system:serviceaccount:openshift-infra:podsecurity-admission-label-syncer-controller had 1034 applies, check the audit log and operator log to figure out why details in audit log} ```
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When doing deployments on baremetal with assisted installer, it is not possible to use nmstate-configuration because it is only enabled for platform baremetal, and AI uses platform none. Since we have many baremetal users on assisted we should enable it there as well.
Version-Release number of selected component (if applicable):
4.16
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Missing metrics - example: cluster_autoscaler_failed_scale_ups_total
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
#curl the autoscalers metrics endpoint: $ oc exec deployment/cluster-autoscaler-default -- curl -s http://localhost:8085/metrics | grep cluster_autoscaler_failed_scale_ups_total
Actual results:
the metrics does not return a value until an event has happened
Expected results:
The metric counter should be initialized at start up providing a zero value
Additional info:
I have been through the file: https://raw.githubusercontent.com/openshift/kubernetes-autoscaler/master/cluster-autoscaler/metrics/metrics.go and checked off the metrics that do not appear when scraping the metrics endpoint straight after deployment. the following metrics are in metrics.go but are missing from the scrape ~~~ node_group_min_count node_group_max_count pending_node_deletions errors_total scaled_up_gpu_nodes_total failed_scale_ups_total failed_gpu_scale_ups_total scaled_down_nodes_total scaled_down_gpu_nodes_total unremovable_nodes_count skipped_scale_events_count ~~~
Description of problem:
checked in 4.18.0-0.nightly-2024-12-05-103644/4.19.0-0.nightly-2024-12-04-031229, OCPBUGS-34533 is reproduced on 4.18+, no such issue with 4.17 and below.
steps: login admin console or developer console(admin console go to "Observe -> Alerting -> Silences" tab, developer console go to "Observe -> Silences" tab), to create silence, edit the "Until" option, even with a valid timestamp or invalid stamp, will get error "[object Object]" in the "Until" field. see screen recording: https://drive.google.com/file/d/14JYcNyslSVYP10jFmsTaOvPFZSky1eg_/view?usp=drive_link
checked 4.17 fix for OCPBUGS-34533 is already in 4.18+ code
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always
Steps to Reproduce:
1. see the descriptions
Actual results:
Unable to edit "until" filed in silences
Expected results:
able to edit "until" filed in silences
Description of problem:
Currently if we try to create a cluster with assigServicePrincipals set to false it wont return the marshalled data plane identities and will fail to create a cluster. We should still return this without assigning service principal roles
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When user start a Pipeline, in Pipeline visualization, it will show as Failed for Tasks and after that it will show as Running state.
Version-Release number of selected component (if applicable):
4.17.z
How reproducible:
Not always but more frequently
Steps to Reproduce:
1. Create a Pipeline and start it 2. Observe Pipeline visualization in details page
Actual results:
Pipeline visualisation shows all tasks as Failed and after that goes to Running state
Expected results:
Pipeline visualisation should not shows all tasks as Failed before it goes to Running state
Additional info:
Description of problem:
The proposed name for services in the ui has an additional 'asd' after example: `exampleasd`.
4.18.0-0.nightly-multi-2025-01-15-030049
How reproducible: Always
Steps to Reproduce:
1. Go to the UI -> Networking -> Services 2. Click create a new service
Actual results:
--- apiVersion: v1 kind: Service metadata: name: exampleasd namespace: test spec: selector: app: name spec: ...
Expected results:
--- apiVersion: v1 kind: Service metadata: name: example namespace: test spec: ....
Additional info:
Description of problem:
The initial set of default endpoint overrides we specified in the installer are missing a v1 at the end of the DNS services override.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Starting with OpenShift Container Platform 4.16, it was observed that cluster-network-operator is stuck in CrashLoopBackOff state because of the below error reported. 2024-09-17T16:32:46.503056041Z I0917 16:32:46.503016 1 controller.go:242] "All workers finished" controller="pod-watcher" 2024-09-17T16:32:46.503056041Z I0917 16:32:46.503045 1 internal.go:526] "Stopping and waiting for caches" 2024-09-17T16:32:46.503209536Z I0917 16:32:46.503189 1 internal.go:530] "Stopping and waiting for webhooks" 2024-09-17T16:32:46.503209536Z I0917 16:32:46.503206 1 internal.go:533] "Stopping and waiting for HTTP servers" 2024-09-17T16:32:46.503217413Z I0917 16:32:46.503212 1 internal.go:537] "Wait completed, proceeding to shutdown the manager" 2024-09-17T16:32:46.503231142Z F0917 16:32:46.503221 1 operator.go:130] Failed to start controller-runtime manager: failed to start metrics server: failed to create listener: listen tcp :8080: bind: address already in use That problem seems to be related to the change done in https://github.com/openshift/cluster-network-operator/pull/2274/commits/acd67b432be4ef2efb470710aebba2e3551bc00d#diff-99c0290799daf9abc6240df64063e20bfaf67b371577b67ac7eec6f4725622ff, where it was missed to pass BindAddress with 0 https://github.com/openshift/cluster-network-operator/blob/master/vendor/sigs.k8s.io/controller-runtime/pkg/metrics/server/server.go#L70 to keep previous functionality. With the current code in place, cluster-network-operator will expose a metrics server on port 8080 which was not the case and can create conflicts with custom application. This is especially true in environment, where compact OpenShift Container Platform 4 - Clusters are running (three-node cluster)
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.16
How reproducible:
Always
Steps to Reproduce:
1. Install OpenShift Container Platform 4.15 (three-node cluster) and create a service that is listening on HostNetwork with port 8080 2. Update to OpenShift Container Platform 4.16 3. Watch cluster-network-operator being stuck in CrashLoopBackOff state because port 8080 is already bound
Actual results:
2024-09-17T16:32:46.503056041Z I0917 16:32:46.503016 1 controller.go:242] "All workers finished" controller="pod-watcher" 2024-09-17T16:32:46.503056041Z I0917 16:32:46.503045 1 internal.go:526] "Stopping and waiting for caches" 2024-09-17T16:32:46.503209536Z I0917 16:32:46.503189 1 internal.go:530] "Stopping and waiting for webhooks" 2024-09-17T16:32:46.503209536Z I0917 16:32:46.503206 1 internal.go:533] "Stopping and waiting for HTTP servers" 2024-09-17T16:32:46.503217413Z I0917 16:32:46.503212 1 internal.go:537] "Wait completed, proceeding to shutdown the manager" 2024-09-17T16:32:46.503231142Z F0917 16:32:46.503221 1 operator.go:130] Failed to start controller-runtime manager: failed to start metrics server: failed to create listener: listen tcp :8080: bind: address already in use
Expected results:
In previous version BindAddress was set to 0 for the Metrics server, meaning it would not start respectively expose on port 8080. Therefore the same should be done with OpenShift Container Platform 4.16 to keep backward compatability and prevent port conflicts.
Additional info:
see upstream issue https://github.com/metal3-io/ironic-image/issues/630
that looks like it's not getting the correct values from the read command, also the colorized output is interfering with the parsing
Reviving rhbz#1948087, the kube-storage-version-migrator ClusterOperator occasionally goes Available=False with reason=KubeStorageVersionMigrator_Deploying. For example, this run includes:
: [bz-kube-storage-version-migrator] clusteroperator/kube-storage-version-migrator should not change condition/Available expand_less 1h34m30s { 1 unexpected clusteroperator state transitions during e2e test run. These did not match any known exceptions, so they cause this test-case to fail: Oct 03 22:09:07.933 - 33s E clusteroperator/kube-storage-version-migrator condition/Available reason/KubeStorageVersionMigrator_Deploying status/False KubeStorageVersionMigratorAvailable: Waiting for Deployment
But that is a node rebooting into newer RHCOS, and do not warrant immediate admin intervention. Teaching the KSVM operator to stay Available=True for this kind of brief hiccup, while still going Available=False for issues where least part of the component is non-functional, and that the condition requires immediate administrator intervention would make it easier for admins and SREs operating clusters to identify when intervention was required.
4.8 and 4.15. Possibly all supported versions of the KSVM operator have this exposure.
Looks like many (all?) 4.15 update jobs have near 100% reproducibility for some kind of issue with KSVM going Available=False, see Actual results below. These are likely for reasons that do not require admin intervention, although figuring that out is tricky today, feel free to push back if you feel that some of these do warrant admin immediate admin intervention.
w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=48h&type=junit&search=clusteroperator/kube-storage-version-migrator+should+not+change+condition/Available' | grep '^periodic-.*4[.]15.*failures match' | sort
periodic-ci-openshift-hypershift-release-4.15-periodics-e2e-kubevirt-conformance (all) - 2 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-multiarch-master-nightly-4.15-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 19 runs, 42% failed, 163% of failures match = 68% impact periodic-ci-openshift-multiarch-master-nightly-4.15-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 18 runs, 61% failed, 118% of failures match = 72% impact periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-nightly-4.14-ocp-e2e-aws-sdn-arm64 (all) - 1 runs, 100% failed, 100% of failures match = 100% impact periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-nightly-4.14-ocp-ovn-remote-libvirt-ppc64le (all) - 4 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-aws-ovn-heterogeneous-upgrade (all) - 19 runs, 47% failed, 189% of failures match = 89% impact periodic-ci-openshift-multiarch-master-nightly-4.15-upgrade-from-stable-4.14-ocp-e2e-aws-sdn-arm64 (all) - 9 runs, 78% failed, 86% of failures match = 67% impact periodic-ci-openshift-release-master-ci-4.15-e2e-aws-ovn-upgrade (all) - 11 runs, 64% failed, 114% of failures match = 73% impact periodic-ci-openshift-release-master-ci-4.15-e2e-azure-ovn-upgrade (all) - 65 runs, 45% failed, 169% of failures match = 75% impact periodic-ci-openshift-release-master-ci-4.15-e2e-azure-sdn-upgrade (all) - 6 runs, 50% failed, 133% of failures match = 67% impact periodic-ci-openshift-release-master-ci-4.15-e2e-gcp-ovn-upgrade (all) - 75 runs, 24% failed, 361% of failures match = 87% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-aws-ovn-upgrade (all) - 75 runs, 29% failed, 277% of failures match = 81% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-aws-sdn-upgrade (all) - 8 runs, 50% failed, 175% of failures match = 88% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-azure-sdn-upgrade (all) - 74 runs, 36% failed, 185% of failures match = 68% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-rt-upgrade (all) - 69 runs, 49% failed, 156% of failures match = 77% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-ovn-upgrade (all) - 7 runs, 57% failed, 175% of failures match = 100% impact periodic-ci-openshift-release-master-ci-4.15-upgrade-from-stable-4.14-e2e-gcp-sdn-upgrade (all) - 6 runs, 33% failed, 250% of failures match = 83% impact periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-ovn-single-node-serial (all) - 6 runs, 100% failed, 17% of failures match = 17% impact periodic-ci-openshift-release-master-nightly-4.15-e2e-aws-sdn-upgrade (all) - 60 runs, 38% failed, 187% of failures match = 72% impact periodic-ci-openshift-release-master-nightly-4.15-e2e-gcp-sdn-upgrade (all) - 6 runs, 33% failed, 200% of failures match = 67% impact periodic-ci-openshift-release-master-nightly-4.15-e2e-metal-ipi-sdn-bm-upgrade (all) - 7 runs, 29% failed, 300% of failures match = 86% impact periodic-ci-openshift-release-master-nightly-4.15-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 7 runs, 71% failed, 80% of failures match = 57% impact periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-aws-sdn-upgrade (all) - 6 runs, 50% failed, 200% of failures match = 100% impact periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-metal-ipi-sdn-bm-upgrade (all) - 6 runs, 100% failed, 83% of failures match = 83% impact periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-metal-ipi-upgrade-ovn-ipv6 (all) - 6 runs, 100% failed, 83% of failures match = 83% impact periodic-ci-openshift-release-master-okd-4.15-e2e-aws-ovn-upgrade (all) - 13 runs, 54% failed, 71% of failures match = 38% impact periodic-ci-openshift-release-master-okd-scos-4.15-e2e-aws-ovn-upgrade (all) - 16 runs, 63% failed, 70% of failures match = 44% impact
KSVM goes Available=False if and only if immediate admin intervention is appropriate.
Component Readiness has found a potential regression in the following test:
[sig-arch][Late] clients should not use APIs that are removed in upcoming releases [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]
Probability of significant regression: 99.98%
Sample (being evaluated) Release: 4.17
Start Time: 2024-08-01T00:00:00Z
End Time: 2024-08-07T23:59:59Z
Success Rate: 94.59%
Successes: 105
Failures: 6
Flakes: 0
Base (historical) Release: 4.16
Start Time: 2024-05-31T00:00:00Z
End Time: 2024-06-27T23:59:59Z
Success Rate: 100.00%
Successes: 36
Failures: 0
Flakes: 302
Description of problem:
This bug is filed a result of https://access.redhat.com/support/cases/#/case/03977446 ALthough both nodes topologies are equavilent, PPC reported a false negative: Error: targeted nodes differ: nodes host1.development.lab and host2.development.lab have different topology: the CPU corres differ: processor core #20 (2 threads), logical processors [2 66] vs processor core #20 (2 threads), logical processors [2 66]
Version-Release number of selected component (if applicable):
4.14
How reproducible:
always
Steps to Reproduce:
1.TBD 2. 3.
Actual results:
Error: targeted nodes differ: nodes host1.development.lab and host2.development.lab have different topology: the CPU corres differ: processor core #20 (2 threads), logical processors [2 66] vs processor core #20 (2 threads), logical processors [2 66]
Expected results:
topologies matches, the PPC should work fine
Additional info:
Description of problem:
Test pxe boot in ABI day2 install, day2 host can not reboot from the disk properly, but reboot from pxe again. This is not reproduced in all hosts and reboots.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-26-075648
How reproducible:
Not always, about 70% in amd64 and 100% in arm64
Steps to Reproduce:
1. Run ABI day1 install booting from pxe 2. After day1 cluster is installed, run ABI day2 install booting from pxe 3. Day2 host hasn't reboot from disk as expected, but reboot from pxe again. From agent.service log, we can see the error: level=info msg=\"SetBootOrder, runtime.GOARCH: amd64, device: /dev/sda\"\ntime=\"2024-11-27T06:48:15Z\" level=info msg=\"Setting efibootmgr to boot from disk\"\ntime=\"2024-11-27T06:48:15Z\" level=error msg=\"failed to find EFI directory\" error=\"failed to mount efi device: failed executing /usr/bin/nsenter [--target 1 --cgroup --mount --ipc --pid -- mount /dev/sda2 /mnt], Error exit status 32, LastOutput \\\"mount: /var/mnt: special device /dev/sda2 does not exist.\\\"\"\ntime=\"2024-11-27T06:48:15Z\" level=warning msg=\"Failed to set boot order\" error=\"failed to mount efi device: failed executing /usr/bin/nsenter [--target 1 --cgroup --mount --ipc --pid -- mount /dev/sda2 /mnt], Error exit status 32
Actual results:
Day2 host hasn't reboot from disk as expected
Expected results:
Day2 host should reboot from disk to complete the installation
Additional info:
Description of problem:
When a clusternetwork entry has an invalid hostPrefix, which is <= CIDR mask and the custom IPv4 join subnet is provided in the install-config, the installer gives a runtime panic error: negative shift amount The error is expected to occur as it is an invalid configuration, but a more descriptive error should be returned instead of panics.
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
Create a cluster with the following install-config (i.e. some fields are intentionally not included): ``` additionalTrustBundlePolicy: Proxyonly apiVersion: v1 baseDomain: devcluster.openshift.com compute: - architecture: amd64 hyperthreading: Enabled name: worker platform: {} replicas: 3 controlPlane: architecture: amd64 hyperthreading: Enabled name: master platform: {} replicas: 3 metadata: creationTimestamp: null name: thvo-dev networking: clusterNetwork: - cidr: 10.128.0.0/19 hostPrefix: 18 # Bad because hostPrefix must be >= 19 - cidr: 10.128.32.0/19 hostPrefix: 23 machineNetwork: - cidr: 10.0.0.0/16 networkType: OVNKubernetes serviceNetwork: - 172.30.0.0/16 ovnKubernetesConfig: ipv4: internalJoinSubnet: 101.64.0.0/16 platform: aws: region: us-east-1 publish: External ```
Actual results:
panic: runtime error: negative shift amount
Expected results:
A descriptive user-friendly error that points out why it is invalid
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Application of PerformanceProfile with invalid cpuset in one of the reserved/isolated/shared/offlined cpu fields causing webhook validation to panic instead of returning an informant error.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-22-231049
How reproducible:
Apply a PerformanceProfile with invalid cpu values
Steps to Reproduce:
Apply the following PerformanceProfile with invalid cpu values: apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: pp spec: cpu: isolated: 'garbage' reserved: 0-3 machineConfigPoolSelector: pools.operator.machineconfiguration.openshift.io/worker-cnf: "" nodeSelector: node-role.kubernetes.io/worker-cnf: ""
Actual results:
On OCP >= 4.18 the error is:
Error from server: error when creating "pp.yaml": admission webhook "vwb.performance.openshift.io" denied the request: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
On OCP <= 4.17 the error is:
Validation webhook passes without any errors. Invalid configuration propogates to the cluster and breaks it.
Expected results:
We expect to pushback an informant error when invalid cpuset has been entered, without panicking or accepting it!
Description of problem:
When internal serving certificates expire (and are renewed), the new certificates are not picked up automatically by control plane components, resulting in an unstable control plane.
Version-Release number of selected component (if applicable):
All
How reproducible:
Always
Steps to Reproduce:
1. Create a HostedCluster with annotations for a short certificate expiration time: hypershift.openshift.io/certificate-validity: "1h" hypershift.openshift.io/certificate-renewal: "0.3" 2. Wait for initial certificates to expire
Actual results:
Cluster becomes degraded, apiservices in hosted cluster API become unavailable. To test this, obtain a kubeconfig for the hosted cluster and list apiservices: $ oc get apiservices API services that are external to the kube-apiserver appear as unavailable.
Expected results:
Cluster continues to function as expected
Additional info:
The aks-e2e test keeps failing on the CreateClusterV2 test because the `ValidReleaseInfo` condition is not set. The patch that sets this status keeps failing. Investigate why & provide a fix.
See https://github.com/openshift/api/pull/2127 and when verify ran here: https://storage.googleapis.com/test-platform-results/pr-logs/pull/openshift_api/2127/pull-ci-openshift-api-release-4.18-verify/1886269516866916352/build-log.txt it pulled sippy results from 4.19 instead of 4.18, sippy shows nothing red for https://sippy.dptools.openshift.org/sippy-ng/tests/4.18/details?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22%5BOCPFeatureGate%3ANetworkSegmentation%5D%22%7D%5D%7D vsphere on 4.18 which is what fails on that verify PR
See https://redhat-internal.slack.com/archives/C01CQA76KMX/p1738586200370409 for more details
Description of problem:
Azure installer panics if there's insufficient permissions to check for ip availability when creating a load balancer
Version-Release number of selected component (if applicable):
How reproducible:
Frequent if there's a permissions issue to get available IPs
Steps to Reproduce:
1. Create cluster on Azure
Actual results:
Full on panic [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3c11064] goroutine 1 [running]: github.com/openshift/installer/pkg/asset/manifests/azure.getNextAvailableIP({0x22548bc0, 0x272aab80}, 0xc002056840) /go/src/github.com/openshift/installer/pkg/asset/manifests/azure/cluster.go:270 +0x124 github.com/openshift/installer/pkg/asset/manifests/azure.GenerateClusterAssets(0xc002056840, 0xc0010b7e40) /go/src/github.com/openshift/installer/pkg/asset/manifests/azure/cluster.go:120 +0xbcb github.com/openshift/installer/pkg/asset/manifests/clusterapi.(*Cluster).Generate(0x27263570, {0x5?, 0x8aa87ee?}, 0xc001fcf530) /go/src/github.com/openshift/installer/pkg/asset/manifests/clusterapi/cluster.go:103 +0x57b github.com/openshift/installer/pkg/asset/store.(*storeImpl).fetch(0xc000eed200, {0x22548c70, 0xc001182000}, {0x2251ea10, 0x27263570}, {0x0, 0x0}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:227 +0x6e2 github.com/openshift/installer/pkg/asset/store.(*storeImpl).Fetch(0xc000eed200, {0x22548c70?, 0xc001182000?}, {0x2251ea10, 0x27263570}, {0x272259c0, 0x6, 0x6}) /go/src/github.com/openshift/installer/pkg/asset/store/store.go:77 +0x4e github.com/openshift/installer/pkg/asset/store.(*fetcher).FetchAndPersist(0xc00119a3e0, {0x22548c70, 0xc001182000}, {0x272259c0, 0x6, 0x6}) /go/src/github.com/openshift/installer/pkg/asset/store/assetsfetcher.go:47 +0x16b main.newCreateCmd.runTargetCmd.func3({0x7ffcc595a72a?, 0xe?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:321 +0x6a main.newCreateCmd.runTargetCmd.func4(0x27231000, {0xc00119a1d0?, 0x4?, 0x8a5d385?}) /go/src/github.com/openshift/installer/cmd/openshift-install/create.go:335 +0x102 github.com/spf13/cobra.(*Command).execute(0x27231000, {0xc00119a1b0, 0x1, 0x1}) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:989 +0xa91 github.com/spf13/cobra.(*Command).ExecuteC(0xc000ee0f08) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /go/src/github.com/openshift/installer/vendor/github.com/spf13/cobra/command.go:1041 main.installerMain() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:67 +0x390 main.main() /go/src/github.com/openshift/installer/cmd/openshift-install/main.go:39 +0x168
Expected results:
No panic and provide error messages.
Additional info:
Slack link: https://redhat-internal.slack.com/archives/C01V1DP387R/p1738888488097589
Description of problem:
When instance types are not specified in the machine pool, the installer checks which instance types (from a list) are available in a given az. If the ec2:DescribeInstanceType permission is not present, the check will fail gracefully and default to using the m6i instance type. This instance type is not available in all regions (e.g. ap-southeast-4 and eu-south-2), so those installs will fail. OCPBUGS-45218 describes a similar issue with edge nodes. ec2:DescribeInstanceTypeOfferings is not a controversial permission and should be required by default for all installs to avoid this type of issue.
Version-Release number of selected component (if applicable):
Affects all versions, but we will just fix in main (4.19)
How reproducible:
Always
Steps to Reproduce:
See OCPBUGS-45218 for one example. Another example (unverified) 1. Use permissions without ec2:DescribeInstanceTypeOfferings 2. Install config: set region to eu-south-2 or ap-southeast-4. Do not set instance types 3. Installer should default to m6i instance type (can be confirmed from machine manifests). 4. Install will fail as m6i instances are not available in those regions: https://docs.aws.amazon.com/ec2/latest/instancetypes/ec2-instance-regions.html
Actual results:
Install fails due to unavailable m6i instance
Expected results:
Installer should select different instance type, m5
Additional info:
Description of problem:
Create a cluster with custom DNS enabled, the coredns.yaml file was created on bootstrap but no coredns pod: [core@ip-10-0-98-49 ~]$ sudo crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD 50264b0c68ffd registry.ci.openshift.org/ocp/release@sha256:c7b642a1fdc7c2bd99d33695c57f7c45401f422a08996640a1eb8b7a9a50a983 14 minutes ago Running cluster-version-operator 2 afafcb955fef5 bootstrap-cluster-version-operator-ip-10-0-98-49 b1410c782f7aa 7691fe6c8036ecba066c3cfa9865455cb714dacf10bb6bef6b0aca10dc37f96e 15 minutes ago Running kube-controller-manager 1 485370aaad3b5 bootstrap-kube-controller-manager-ip-10-0-98-49 369341e2b5bdc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e5d868208c8ee95b2973406833be802bca7ef220eea6ad4273384990abe10f3 15 minutes ago Running cluster-policy-controller 0 485370aaad3b5 bootstrap-kube-controller-manager-ip-10-0-98-49 a390d13a3847b 934889bfd49afe7a91680889af526bc5722c813d8fa76c23b7544951a0300a66 15 minutes ago Running kube-apiserver-insecure-readyz 0 2a3112486283a bootstrap-kube-apiserver-ip-10-0-98-49 cd370437856c6 7691fe6c8036ecba066c3cfa9865455cb714dacf10bb6bef6b0aca10dc37f96e 15 minutes ago Running kube-apiserver 0 2a3112486283a bootstrap-kube-apiserver-ip-10-0-98-49 4c7b7d343c38f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1dbd4ef2dddd9a124a128754580bfcbe50de106b7a503f3df59da58bd88bf0c9 15 minutes ago Running kube-scheduler 0 f26b53a65bd94 bootstrap-kube-scheduler-ip-10-0-98-49 b88105f79dfe1 936d4cb44a2471bb928caf54c80597686160b3fff5c49a793cf7677e9dbb3e48 15 minutes ago Running cloud-credential-operator 0 33371da90c541 cloud-credential-operator-ip-10-0-98-49 d66c2126bf646 64f5aaf4e366de0d2e3301def8a9c38e4db99d7bcc578f0a50bfba0d73918e4f 15 minutes ago Running machine-config-server 0 050647bcc3ee3 bootstrap-machine-config-operator-ip-10-0-98-49 75ced844a6524 c35e3479a3b694513582d6ede502b1472c908ee1b7ae87a5f705d09570ed21ac 16 minutes ago Running etcd 0 970037306ed33 etcd-bootstrap-member-ip-10-0-98-49 eb50696dda431 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:598847ee4f002b98d9251e34d91041881837464aca652acc8952f2c647cf49af 16 minutes ago Running etcdctl 0 970037306ed33 etcd-bootstrap-member-ip-10-0-98-49 install-config: platform: aws: userProvisionedDNS: Enabled featureSet: TechPreviewNoUpgrade
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-16-065305
How reproducible:
Always
Steps to Reproduce:
1. Config install-config.yaml as described in the description 2. Create cluster. 3.
Actual results:
No coredns pod on bootstrap machine.
Expected results:
Like on GCP, coredns pod should be created: sudo crictl ps | grep coredns a6122cfb07b2f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8677b12826bd9d33e8f6bb96d70bbcdaf3c5dec9e28a8b72af2c2e60620b6b19 33 minutes ago Running coredns 0 09622bf07804a coredns-yunjiang-dnsgcp9-4xfj4-bootstrap
Additional info:
Description of problem:
Observing these e2e failures consistently [sig-builds][Feature:Builds][subscription-content] builds installing subscription content [apigroup:build.openshift.io] should succeed for RHEL 7 base images [Suite:openshift/conformance/parallel] [sig-builds][Feature:Builds][subscription-content] builds installing subscription content [apigroup:build.openshift.io] should succeed for RHEL 8 base images [Suite:openshift/conformance/parallel] [sig-builds][Feature:Builds][subscription-content] builds installing subscription content [apigroup:build.openshift.io] should succeed for RHEL 9 base images [Suite:openshift/conformance/parallel]
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Fails consistently and fails when run steps are run manually
Steps to Reproduce:
1. Setup 4.18 cluster 2. Run e2e 3. Test cases in file - origin/test/extended/builds/subscription_content.go raw - https://raw.githubusercontent.com/openshift/origin/f7e4413793877efb24be86de05319dad00d05897/test/extended/builds/subscription_content.go
Actual results:
Test case fails
Expected results:
Additional info:
Failures were observed in both OCP-4.17 as well as OCP-4.18. Following are the logs.
Upgrade alertmanager to 0.28.0 downstream
https://github.com/openshift/prometheus-alertmanager
syncbot update failed: https://github.com/rhobs/syncbot/commit/c06d1d22eba68699bf218705b76f53e180eaf940/checks
This will need to update syncbot repo as well to fix the build
Description of problem:
Font size of `BuildSpec details` on BuildRun details page is larger than other title on the page
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Navigate to Shipwright BuildRun details page
Actual results:
Font size of `BuildSpec details` on BuildRun details page is larger than other title on the page
Expected results:
All title should be of same size
Additional info:
Screenshots
https://github.com/user-attachments/assets/74853838-1fff-46d5-9ed6-5b605caebbf0
Description of problem:
We're seeing the following error in the ovnkube-controller container log on RHEL-8 worker which leads to the network is not ready of the node
F0206 03:40:21.953369 12091 ovnkube.go:137] failed to run ovnkube: [failed to start network controller: failed to start default network controller - while waiting for any node to have zone: "ip-10-0-75-250.ec2.internal", error: context canceled, failed to start node network controller: failed to start default node network controller: failed to find kubelet cgroup path: %!w(<nil>)]
The full log of the ovnkube-controller container:
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-02-05-033447/4.18.0-0.nightly-2025-02-04-192134
How reproducible:
Always
Steps to Reproduce:
1. Add RHEL-8 worker to 4.18 OCP cluster, but the RHEL workers can't be ready, found the following error about ovnkube-controller in the kublet.log
Feb 06 11:38:34 ip-10-0-50-48.us-east-2.compute.internal kubenswrapper[15267]: E0206 11:38:34.798490 15267 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"ovnkube-controller\" with CrashLoopBackOff: \"back-off 20s restarting failed container=ovnkube-controller pod=ovnkube-node-txkkp_openshift-ovn-kubernetes(c22474ab-6f0b-4403-93a6-eb80766934e6)\"" pod="openshift-ovn-kubernetes/ovnkube-node-txkkp" podUID="c22474ab-6f0b-4403-93a6-eb80766934e6"
An example failure job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-workers-rhel8/1887322975150018560
2.
3.
Actual results:
Expected results:
Additional info:
Based on the test history, it's working for 4.18.0-0.nightly-2025-02-04-114552, but start failing for 4.18.0-0.nightly-2025-02-05-033447.
(Update: confirmed it's also failed on 4.18.0-0.nightly-2025-02-04-192134)
Here's the daily 4.18 rhel8 job history link:
https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-workers-rhel8
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
PF "-v5-" classes are breaking for the plugin on console's latest main branch. Ref: https://redhat-internal.slack.com/archives/C011BL0FEKZ/p1740475225829469
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Run console latest "main" locally. 2. Run odf-console plugin latest "master" locally (https://github.com/red-hat-storage/odf-console), or any demo/actual plugin using PatternFly version 5. 3. Navigate to pages/components using classes like: "pf-v5-u-mt-md", "pf-v5-u-pt-sm" etc.
Actual results:
pf-v5-* classes are not getting applied.
Expected results:
PF v5 classes should work as expected.
Additional info:
Description of problem:
In order to test OCL we run e2e automated test cases in a cluster that has OCL enabled in master and worker pools. We have seen that rarely a new machineconfig is rendered but no MOSB resource is created.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Rare
Steps to Reproduce:
We don't have any steps to reproduce it. It happens eventually when we run a regression in a cluster with OCL enabled in master and worker pools.
Actual results:
We see that in some scenarios a new MC is created, then a new rendered MC is created too, but now MOSB is created and the pool is stuck forever.
Expected results:
Whenever a new rendered MC is created, a new MOSB sould be created too to build the new image.
Additional info:
In the comments section we will add all the must-gather files that are related to this issue. In some scenarios we can see this error reported by the os-builder pod: 2024-12-03T16:44:14.874310241Z I1203 16:44:14.874268 1 request.go:632] Waited for 596.269343ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-machine-config-operator/secrets?labelSelector=machineconfiguration.openshift.io%2Fephemeral-build-object%2Cmachineconfiguration.openshift.io%2Fmachine-os-build%3Dmosc-worker-5fc70e666518756a629ac4823fc35690%2Cmachineconfiguration.openshift.io%2Fon-cluster-layering%2Cmachineconfiguration.openshift.io%2Frendered-machine-config%3Drendered-worker-7c0a57dfe9cd7674b26bc5c030732b35%2Cmachineconfiguration.openshift.io%2Ftarget-machine-config-pool%3Dworker Nevertheless, we only see this error in some of them, not in all of them.
Description of problem:
When CCO does not provide credentials for CSI driver operators, the CSI driver operators are Progressing=true forever, with a message:
Operator unavailable (GCPPDCSIDriverOperatorCR_GCPPDDriverControllerServiceController_Deploying): GCPPDCSIDriverOperatorCRAvailable: GCPPDDriverControllerServiceControllerAvailable: Waiting for Deployment Operator unavailable (GCPPDCSIDriverOperatorCR_GCPPDDriverControllerServiceController_Deploying): GCPPDCSIDriverOperatorCRAvailable: GCPPDDriverControllerServiceControllerAvailable: Waiting for Deployment
(full job run).
This will be further emphasized by CCO being an optional component in OCP 4.15. CSO / CSI driver operators should provide a more useful error and even degrade the cluster when Secret is not available in X minutes. The message should point to CCO, so users know where to look for details.
The following test is failing more than expected:
Undiagnosed panic detected in pod
See the sippy test details for additional context.
Observed in 4.18-e2e-vsphere-ovn-upi-serial/1861922894817267712
Undiagnosed panic detected in pod { pods/openshift-machine-config-operator_machine-config-daemon-4mzxf_machine-config-daemon_previous.log.gz:E1128 00:28:30.700325 4480 panic.go:261] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<}
Description of problem:
openshift-install is always raise WANING when install a cluster on AWS Region us-east-1 with default configuration, no zones is set. ~~~ WARNING failed to find default instance type for worker pool: no instance type found for the zone constraint WARNING failed to find default instance type: no instance type found for the zone constraint ~~~ The process to discover zones list all zones in the region, then tries to describe instance type offerings across all zones from a list of supported instance types by installer. The problem that there is an "dead" zone in this region, the us-east-1e (ID use1-az3), which does not support any instance type we support, leading in creating infra resources in one zone which isn't useful as may not able to launch supported instance types there.
Version-Release number of selected component (if applicable):
* (?)
How reproducible:
always
Steps to Reproduce:
1. create install-config targeting AWS region us-east-1, without setting zones (default) 2. create manifests, or create cluster
Actual results:
~~~ WARNING failed to find default instance type for worker pool: no instance type found for the zone constraint WARNING failed to find default instance type: no instance type found for the zone constraint ~~~ The WARNING is raised, the install does not fail because the fallback instance type is supported across zones used to make Control Plane and Worker nodes
Expected results:
No WARNINGS/failures
Additional info:
Description of problem:
Openshift cluster upgrade from 4.12.10 to 4.12.30 failing because pod version-4.12.30-xxx is in CreateContainerConfigError. Also tested in 4.14
Steps to Reproduce:
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.10 True False 12m Cluster version is 4.12.10
--- allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: null apiVersion: security.openshift.io/v1 defaultAddCapabilities: null fsGroup: type: RunAsAny groups: [] kind: SecurityContextConstraints metadata: name: scc-hostpath-cnf-cat-1 priority: null readOnlyRootFilesystem: false requiredDropCapabilities: - KILL - MKNOD - SETUID - SETGID runAsUser: type: MustRunAsNonRoot seLinuxContext: type: MustRunAs supplementalGroups: type: RunAsAny users: [] volumes: - configMap - downwardAPI - emptyDir - hostPath - persistentVolumeClaim - projected - secret
$ oc adm upgrade --to=4.12.30 $ oc get pod -n openshift-cluster-version NAME READY STATUS RESTARTS AGE cluster-version-operator-85db98885c-jt25z 1/1 Running 0 41m version-4.12.30-vw4pm-l2nng 0/1 Init:CreateContainerConfigError 0 42s $ oc get events | grep Failed 10s Warning Failed pod/version-4.12.30-p6k4r-nmn6m Error: container has runAsNonRoot and image will run as root (pod: "version-4.12.30-p6k4r-nmn6m_openshift-cluster-version(4d1704d9-ca34-4aa3-86e1-1742e8cead0c)", container: cleanup) $ oc get pod version-4.12.30-97nbr-88mxp -o yaml |grep scc openshift.io/scc: scc-hostpath-cnf-cat-1
As a workaround, we can remove the scc "scc-hostpath-cnf-cat-1" and the pod version-xxx and the upgrade worked. Customer has created custom scc for use of applications.
$ oc get pod version-4.12.30-nmskz-d5x2c -o yaml | grep scc
openshift.io/scc: node-exporter
$ oc get pod
NAME READY STATUS RESTARTS AGE
cluster-version-operator-6cb5557f8f-v65vb 1/1 Running 0 54s
version-4.12.30-nmskz-d5x2c 0/1 Completed 0 67s
There's an old bug https://issues.redhat.com/browse/OCPBUGSM-47192 which was fixed setting readOnlyRootFilesystem to false, but in this case the scc is still failing.
--- container.SecurityContext = &corev1.SecurityContext{ Privileged: pointer.BoolPtr(true), ReadOnlyRootFilesystem: pointer.BoolPtr(false), } ---
Component Readiness has found a potential regression in the following test:
install should succeed: infrastructure
installer fails with:
time="2024-10-20T04:34:57Z" level=error msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: control-plane machines were not provisioned within 15m0s: client rate limiter Wait returned an error: context deadline exceeded"
Significant regression detected.
Fishers Exact probability of a regression: 99.96%.
Test pass rate dropped from 98.94% to 89.29%.
Sample (being evaluated) Release: 4.18
Start Time: 2024-10-14T00:00:00Z
End Time: 2024-10-21T23:59:59Z
Success Rate: 89.29%
Successes: 25
Failures: 3
Flakes: 0
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 98.94%
Successes: 93
Failures: 1
Flakes: 0
Description of problem: EIP UDN Layer 3 pre-merge testing] In SGW and LGW modes, after restarting ovnkube-node pod of client host of local EIP pod, EIP traffic from remote EIP pod can not be captured on egress node ovs-if-phys0
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. labeled a node to be egress node, created an egressIP object
2. Created a namespace, applied layer3 UDN CRD to it
3. Created two test pods, one local to egress node, the other one is remote to egress node
4. Restarted the ovnkube-node pod of the local EIP pod's client host (or egress node), waited till the ovnkube-node pod recreated and ovnkube-node ds rollout succeeded
5. Curl external from both test pods
Actual results: egressing packets from remote EIP pod can not be captured on egress node ovs-if-phys0 after restarting the ovnkube-node pod of egress node
Expected results: egressing packets from either EIP pod can be captured on egress node ovs-if-phys0 after restarting the ovnkube-node pod of egress node
Additional info:
egressing packets from local EIP pod can not be captured on egress node ovs-if-phys0 after restarting the ovnkube-node pod of egress node
must-gather: https://drive.google.com/file/d/12aonBDHMPsmoGKmM47yGTBFzqyl1IRlx/view?usp=drive_link
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
When a primary UDN or CUDN is created, it creates what is known as a secondary zone network controller that handles configuring OVN and getting the network created so that pods can be attached. The time it takes for this to happen can be up to a minute if namespaces are being deleted on the cluster while the UDN controller is starting.
This is because if the namespace is deleted, GetActiveNetworkForNamespace will fail, and the pod will be retried for up to a minute.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create a bunch of namespaces with pods
2. Create a primary CUDN or UDN
3. Very quickly start deleting namespaces that have pods in them while OVNK is starting up its network controller
Actual results:
Logs show time to start the controller taking a minute:
I0131 16:15:30.383221 5583 secondary_layer2_network_controller.go:365] Starting controller for secondary network e2e-network-segmentation-e2e-8086.blue I0131 16:16:30.390813 5583 secondary_layer2_network_controller.go:369] Starting controller for secondary network e2e-network-segmentation-e2e-8086.blue took 1m0.007579788s
Expected results:
Once started the controller should only take a few seconds (depending on cluster size and load) to finish starting.
Description of problem:
There is "Learn more about Route" doc link on routes page if there is not route in the project, it points to 4.16 doc link "https://docs.redhat.com/en/documentation/openshift_container_platform/4.16/html/networking/configuring-routes". It should points to 4.17 for 4.17 cluster and 4.18 for 4.18 cluster.
Version-Release number of selected component (if applicable):
4.17.0-0.nightly-2024-09-18-003538 4.18.0-0.nightly-2024-09-17-060032
How reproducible:
Always
Steps to Reproduce:
1. Check "Learn more about Route" doc link on routes page. 2. 3.
Actual results:
1. It points to 4.16 link.
Expected results:
1. It should point to 4.17 for 4.17 cluster and 4.18 for 4.18 cluster.
Additional info:
Description of problem:
Pull support from upstream kubernetes (see KEP 4800: https://github.com/kubernetes/enhancements/issues/4800) for LLC alignment support in cpumanager
Version-Release number of selected component (if applicable):
4.19
How reproducible:
100%
Steps to Reproduce:
1. try to schedule a pod which requires exclusive CPU allocation and whose CPUs should be affine to the same LLC block 2. observe random and likely wrong (not LLC-aligned) allocation 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Issue can be observed on Hypershift e2e-powervs CI (reference : https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.18-periodics-e2e-powervs-ovn/1879800904798965760) HostedCluster is deployed but still getting incorrect condition for status,"HostedCluster is deploying, upgrading, or reconfiguring)" This is happening because of following issue observed on cluster-version Logs reference : https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.18-periodics-e2e-powervs-ovn/1879800904798965760/artifacts/e2e-powervs-ovn/run-e2e/build-log.txt ``` eventually.go:226: - incorrect condition: wanted Progressing=False, got Progressing=True: Progressing(HostedCluster is deploying, upgrading, or reconfiguring) eventually.go:226: - wanted HostedCluster to desire image registry.build01.ci.openshift.org/ci-op-cxr9zifq/release@sha256:7e40dc5dace8cb816ce91829517309e3609c7f4f6de061bf12a8b21ee97bb713, got registry.build01.ci.openshift.org/ci-op-cxr9zifq/release@sha256:e17cb3eab53be67097dc9866734202cb0f882afc04b2972c02997d9bc1a6e96b eventually.go:103: Failed to get *v1beta1.HostedCluster: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline ```
Note : Issue oberved on 4.19.0 Hypershift e2e-aws-multi CI as well (reference : https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-multi/1880072687829651456)
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
100%
Steps to Reproduce:
1. Create PowerVS hypershift cluster with 4.18.0 release 2. 3.
Actual results:
HostedCluster getting deployed but still getting incorrect condition for cluster-version
Expected results:
HostedCluster should get deployed successfully with all conditions met
Additional info:
The issue was first observed on Dec 25, 2024. Currently reproducible on 4.19.0 (reference : https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-multi/1880072687829651456/artifacts/e2e-aws-multi/hypershift-aws-run-e2e-external/build-log.txt)
During review of ARO MiWi permissions, some permissions in the MAPI CredentialsRequest for Azure having other permissions identified through a linked action that are missing.
A linked access check is an action performed by Azure Resource Manager during a incoming request. For example, when you issue a create operation to a network interface ( Microsoft.Network/networkInterfaces/write ) you specify a subnet in the payload. ARM parses the payload, sees you're setting a subnet property, and as a result requires the linked access check Microsoft.Network/virtualNetworks/subnets/join/action to the subnet resource specified in the network interface. If you update a resource but don't include the property in the payload, it will not perform the permission check.
The following permissions were identified as possibly needed in MAPI CredsRequest as they are specified as linked action of one of MAPI's existing permissions
Microsoft.Compute/disks/beginGetAccess/action Microsoft.KeyVault/vaults/deploy/action Microsoft.ManagedIdentity/userAssignedIdentities/assign/action Microsoft.Network/applicationGateways/backendAddressPools/join/action Microsoft.Network/applicationSecurityGroups/joinIpConfiguration/action Microsoft.Network/applicationSecurityGroups/joinNetworkSecurityRule/action Microsoft.Network/ddosProtectionPlans/join/action Microsoft.Network/gatewayLoadBalancerAliases/join/action Microsoft.Network/loadBalancers/backendAddressPools/join/action Microsoft.Network/loadBalancers/frontendIPConfigurations/join/action Microsoft.Network/loadBalancers/inboundNatPools/join/action Microsoft.Network/loadBalancers/inboundNatRules/join/action Microsoft.Network/networkInterfaces/join/action Microsoft.Network/networkSecurityGroups/join/action Microsoft.Network/publicIPAddresses/join/action Microsoft.Network/publicIPPrefixes/join/action Microsoft.Network/virtualNetworks/subnets/join/action
Each permission needs to be validated as to whether it is needed by MAPI through any of its code paths.
Description of problem:
Registry storage alerts did not link a runbook
Version-Release number of selected component (if applicable):
4.18
How reproducible:
always
Steps to Reproduce:
According to the doc: https://github.com/openshift/enhancements/blob/master/enhancements/monitoring/alerting-consistency.md#documentation-required, I think should add runhook link to the registry storage alert pr: https://github.com/openshift/cluster-image-registry-operator/pull/1147/files, thanks
Actual results:
Expected results:
Additional info:
Description of problem:
In Azure, there are 2 regions that don't have availability zones or availability set fault domains (centraluseuap, eastusstg). They are test regions, one of which is in-use by the ARO team. Machine API provider seems to be hardcoding an availability set fault domain count of 2 in creation of the machineset: https://github.com/openshift/machine-api-provider-azure/blob/main/pkg/cloud/azure/services/availabilitysets/availabilitysets.go#L32, so if there is not at least a fault domain count of 2 in the target region, the install will fail because worker nodes get a Failed status. This is the error from Azure, reported by the machine API: `The specified fault domain count 2 must fall in the range 1 to 1.` Because of this, the regions are not able to support OCP clusters.
Version-Release number of selected component (if applicable):
Observed in 4.15
How reproducible:
Very
Steps to Reproduce:
1. Attempt creation of an OCP cluster in centraluseuap or eastusstg regions 2. Observe worker machine failures
Actual results:
Worker machines get a failed state
Expected results:
Worker machines are able to start. I am guessing that this would happen via dynamic setting of the availability set fault domain count rather than hardcoding it to 2, which right now just happens to work in most regions in Azure because the fault domain counts are typically at least 2. In upstream, it looks like we're dynamically setting this by querying the amount of fault domains in a region: https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/40f0fabc264388de02a88de7fbe400c21d22e7e2/azure/services/availabilitysets/spec.go#L70
Additional info:
The image tests are currently failing on presubs as they cannot build our hypershift-tests image. This is caused by the fact that in 4.19 dnf used in CI is really a wrapper and we should now install the Microsoft repositories in /etc/yum/repos.art/ci/ so that the azure-cli can be found when attempting to install it with dnf.
Slack Thread: https://redhat-internal.slack.com/archives/CB95J6R4N/p1737060704212009
Description of problem:
We have been asked by PowerVS cloud team to change how we destory DHCP networks. Because after they have been deleted, there are still sub-resources still in the process of being deleted. So, we are also check to make sure there are no subnets still open as they will be deleted also.
Description of problem:
[azure-disk-csi-driver] ARO HCP uses UserAssignedIdentityCredentials could not provision volume
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-02-26-050226
How reproducible:
Always
Steps to Reproduce:
1. Install ARO hypershift cluster uses UserAssignedIdentityCredentials mode. 2. Create pvc using the managed-csi(azure disk csi provisioner) storageclass, and create pod consume the pvc. 3. Check the pvc could provision successfully and pod could start running.
Actual results:
In step3: the pvc provision failed of -> I0225 13:44:52.851851 1 controllerserver.go:281] begin to create azure disk(pvc-99f969ab-3629-4729-87c7-e796e081f27e) account type(Premium_LRS) rg(generic-managed-rg) location(eastus) size(5) diskZone() maxShares(0) panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x21ced59] goroutine 516 [running]: sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*ManagedDiskController).CreateManagedDisk(0xc000624668, {0x2e5c030, 0xc0004fef30}, 0xc00030d040) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/azuredisk/azure_managedDiskController.go:271 +0x1759 sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*Driver).CreateVolume(0xc00033c800, {0x2e5c030, 0xc0004fef30}, 0xc00029ec80) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/azuredisk/controllerserver.go:332 +0x36c5 ...
Expected results:
In step3: the pvc could provision successfully and pod could start running.
Additional info:
From the csi driver controller logs we could see -> azure_auth.go:175] "No valid auth method found" logger="GetServicePrincipalToken" It seems current cloud-provider-azure(https://github.com/openshift/azure-disk-csi-driver/blob/master/go.mod#L46) depeddency does not contains the support for UserAssignedIdentityCredentials.
Description of problem:
Tracking per-operator fixes for the following related issues static pod node, installer, and revision controllers: https://issues.redhat.com/browse/OCPBUGS-45924 https://issues.redhat.com/browse/OCPBUGS-46372 https://issues.redhat.com/browse/OCPBUGS-48276
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
While working on ARO-13685 I (accidentally) crashed the CVO payload init containers. I found that the removal logic based on plain "rm" is not idempotent, so if any of the init containers crash mid-way, the restart will never be able to succeed. The fix is to use "rm -f" in all places instead.
Version-Release number of selected component (if applicable):
4.18 / main, but existed in prior versions
How reproducible:
always
Steps to Reproduce:
1. inject a crash in the bootstrap init container https://github.com/openshift/hypershift/blob/99c34c1b6904448fb065cd65c7c12545f04fb7c9/control-plane-operator/controllers/hostedcontrolplane/cvo/reconcile.go#L353 2. the restarting previous init container "prepare-payload" will crash loop on "rm" not succeeding as the previous invocation already deleted all manifests
Actual results:
the prepare-payload init container will crash loop forever, preventing the container CVO from running
Expected results:
a crashing init container should be able to restart gracefully without getting stuck on file removal and eventually run the CVO container
Additional info:
based off the work in https://github.com/openshift/hypershift/pull/5315
Description of problem:
when delete platform images, oc-mirror failed with error: Unable to delete my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com/openshift/release:4.15.37-s390x-alibaba-cloud-csi-driver. Image may not exist or is not stored with a v2 Schema in a v2 registry
Version-Release number of selected component (if applicable):
./oc-mirror.rhel8 version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202411090338.p0.g0a7dbc9.assembly.stream.el9-0a7dbc9", GitCommit:"0a7dbc90746a26ddff3bd438c7db16214dcda1c3", GitTreeState:"clean", BuildDate:"2024-11-09T08:33:46Z", GoVersion:"go1.22.7 (Red Hat 1.22.7-1.module+el8.10.0+22325+dc584f75) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. imagesetconfig as follow : kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: quay.io/openshifttest/hello-openshift@sha256:61b8f5e1a3b5dbd9e2c35fd448dc5106337d7a299873dd3a6f0cd8d4891ecc27 operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15 packages: - name: devworkspace-operator platform: architectures: - "s390x" channels: - name: stable-4.15 type: ocp 2. run the mirror2disk and disk2mirror command : `oc mirror -c /home/fedora/yinzhou/openshift-tests-private/test/extended/testdata/workloads/config-72708.yaml file://test/yinzhou/debug72708 --v2` `oc mirror -c /home/fedora/yinzhou/openshift-tests-private/test/extended/testdata/workloads/config-72708.yaml --from file://test/yinzhou/debug72708 --v2 docker://my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com --dest-tls-verify=false` 3. generate delete image list: `oc mirror delete --config /home/fedora/yinzhou/openshift-tests-private/test/extended/testdata/workloads/delete-config-72708.yaml docker://my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com --v2 --workspace file://test/yinzhou/debug72708 --generate` 4. execute the delete command : `oc mirror delete --delete-yaml-file test/yinzhou/debug72708/working-dir/delete/delete-images.yaml docker://my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com --v2 --dest-tls-verify=false --force-cache-delete=true`
Actual results:
4. delete hit error : ⠋ 21/396 : (0s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd62cc631a6bf6e13366d29da5ae64088d3b42410f9b52579077cc82d2ea2ab9 2024/11/12 03:10:07 [ERROR] : Unable to delete my-route-zhouy.apps.yinzhou-1112.qe.devcluster.openshift.com/openshift/release:4.15.37-s390x-alibaba-cloud-csi-driver. Image may not exist or is not stored with a v2 Schema in a v2 registry
Expected results:
4. no error
Additional info:
Description of problem:
network-tools pod-run-netns-command failed due to "ERROR: Can't get netns pid". Seems containerRuntime changed from runc to crun, so we need to update network-tools utils: https://github.com/openshift/network-tools/blob/1df82dfade80ce31b325dab703b37bf7e8924e99/debug-scripts/utils#L108
Version-Release number of selected component (if applicable):
4.18.0-0.test-2024-11-27-013900-ci-ln-s87rfh2-latest
How reproducible:
always
Steps to Reproduce:
1. create test pod in namespace test $ oc get pod -n test NAME READY STATUS RESTARTS AGE hello-pod2 1/1 Running 0 22s 2.run command "ip a" with network-tools script pod-run-netns-command
Actual results:
$ ./network-tools pod-run-netns-command test hello-pod2 "ip route show" Temporary namespace openshift-debug-btzwc is created for debugging node... Starting pod/qiowang-120303-zb568-worker-0-5phll-debug ... To use host binaries, run `chroot /host` Removing debug pod ... Temporary namespace openshift-debug-btzwc was removed. error: non-zero exit code from debug container ERROR: Can't get netns pid <--- Failed INFO: Running ip route show in the netns of pod hello-pod2 Temporary namespace openshift-debug-l7xv4 is created for debugging node... Starting pod/qiowang-120303-zb568-worker-0-5phll-debug ... To use host binaries, run `chroot /host` nsenter: failed to parse pid: 'parse' Removing debug pod ... Temporary namespace openshift-debug-l7xv4 was removed. error: non-zero exit code from debug container ERROR: Command returned non-zero exit code, check output or logs.
Expected results:
run command with network-tools script pod-run-netns-command successfuly
Additional info:
There is no container running: $ oc debug node/qiowang-120303-zb568-worker-0-5phll Temporary namespace openshift-debug-hrr94 is created for debugging node... Starting pod/qiowang-120303-zb568-worker-0-5phll-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.2.190 If you don't see a command prompt, try pressing enter. sh-5.1# chroot /host sh-5.1# runc list ID PID STATUS BUNDLE CREATED OWNER sh-5.1#
Description of problem:
HorizontalNav component of @openshift-console/dynamic-plugin-sdk doest not have customData prop which is available in console repo. This prop is needed to pass any value between tabs in details page
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Create a UDN network, check NAD list, the UDN network is also presenting there
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
UDN network only presents under UDN, not presents in NAD
Expected results:
Additional info:
Remove legacy, unused annotation 'exclude.release.openshift.io/internal-openshift-hosted'
Description of problem:
on Hypershift guest cluster, 'Filter by Node type' list on Cluster utilization is empty
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-11-221222
How reproducible:
Always
Steps to Reproduce:
1. click on 'Filter by Node type' dropdown in Cluster utilization card on Home -> Overview
Actual results:
1. The dropdown list is empty
Expected results:
1. Hypershift guest cluster only has Worker nodes, so we should probably need only present `worker` option $ oc get node --kubeconfig ./kubeconfig-guest NAME STATUS ROLES AGE VERSION ip-10-0-141-150.us-east-2.compute.internal Ready worker 4h21m v1.31.3 ip-10-0-148-36.us-east-2.compute.internal Ready worker 4h21m v1.31.3 ip-10-0-167-238.us-east-2.compute.internal Ready worker 4h22m v1.31.3
Additional info:
This is a clone of issue OCPBUGS-40772. The following is the description of the original issue:
—
Please review the following PR: https://github.com/openshift/service-ca-operator/pull/246
The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.
Differences in upstream and downstream builds impact the fidelity of your CI signal.
If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.
Closing this issue without addressing the difference will cause the issue to
be reopened automatically.
Description of problem:
I was going through the operator hub and searching for "apache zookeeper operator" and "stackable common operator." When I clicked on those operators, I'm getting, `Oh no, something went wrong` error message.
Version-Release number of selected component (if applicable):
4.17.0
How reproducible:
100%
Steps to Reproduce:
1. Go to the web console and then go to operator hub. 2. Search for "apache zookeeper operator" and "stackable common operator" 3. And then click on that operator to install, and you will see that error message.
Actual results:
Getting Oh no something went wrong message
Expected results:
Should show the next page to install the operator
Additional info:
Description of problem:
After the hostpath-provisioner installation the csi-provisioner should create the CSIstoragecapacity objects. But the creation is failing with the following error.
I0120 04:14:26.724089 1 controller.go:873] "Started provisioner controller" component="kubevirt.io.hostpath-provisioner_hostpath-provisioner-csi-smrh4_6264f065-eaed-47aa-90a9-3a2563c965c2" W0120 04:14:28.020372 1 reflector.go:547] k8s.io/client-go/informers/factory.go:160: failed to list v1.CSIStorageCapacity: unable to parse requirement: values[0][csi.storage.k8s.io/managed-by]: Invalid value: "-53f40b57-worker1.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?') E0120 04:14:28.020406 1 reflector.go:150] k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIStorageCapacity: failed to list v1.CSIStorageCapacity: unable to parse requirement: values[0][csi.storage.k8s.io/managed-by]: Invalid value: "-53f40b57-worker1.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?')
This issue is coming with the worker nodes have long node names.
worker0.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com
worker1.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com
This is coming if the worker node name length is exactly 53 characters.
For more details refer this:
https://github.com/kubernetes-csi/external-provisioner/issues/1333
Version-Release number of selected component (if applicable):
How reproducible:
corner case.
Steps to Reproduce:
1. Create the openshift cluster with long worker node names. The worker name should match exactly 53 characters. 2. Install the hostpath provisoner in waitForfirstconsumer mode. 3. Try to create a pvc using the hostpath-provisioner storage class.
Actual results:
csi-provisoner side car is throwing these errors. I0120 04:14:26.724089 1 controller.go:873] "Started provisioner controller" component="kubevirt.io.hostpath-provisioner_hostpath-provisioner-csi-smrh4_6264f065-eaed-47aa-90a9-3a2563c965c2" W0120 04:14:28.020372 1 reflector.go:547] k8s.io/client-go/informers/factory.go:160: failed to list v1.CSIStorageCapacity: unable to parse requirement: values[0][csi.storage.k8s.io/managed-by]: Invalid value: "-53f40b57-worker1.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?') E0120 04:14:28.020406 1 reflector.go:150] k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSIStorageCapacity: failed to list v1.CSIStorageCapacity: unable to parse requirement: values[0][csi.storage.k8s.io/managed-by]: Invalid value: "-53f40b57-worker1.ocp-virt1-s390x.s390g.lab.eng.rdu2.redhat.com": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?')
Expected results:
It should successfully create the CSI storage capacities without any errors.
Additional info:
Description of problem:
OpenShift - the log in button can be clicked repeatably
Version-Release number of selected component (if applicable):
4.16
How reproducible:
Always
Steps to Reproduce:
1. Login to OpenShift 2. Enter a valid Username and Password 3. Click the 'Log in' button repeatably
Actual results:
The login button can be clicked repeatedly
Expected results:
The login button can be clicked several times which could have negative consequences elsewhere in the application. Ideally, buttons should be unclickable once clicked initially to prevent against this behaviour.
Additional info:
Description of problem:
When no value passed to annotation cloud.google.com/network-tier service remain stuck in pending
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-01-15-060507
How reproducible:
Always
Steps to Reproduce:
1.Create service use below yaml apiVersion: v1 kind: Service metadata: labels: name: lb-service-unsecure name: lb-service-unsecure annotations: cloud.google.com/network-tier: "" spec: ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: name: web-server-rc type: LoadBalancer 2. Service gets created successfully 3. miyadav@miyadav-thinkpadx1carbongen8:~/annotations$ oc get svc lb-service1 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE lb-service1 LoadBalancer 172.30.233.126 <pending> 80:30502/TCP 9m36s miyadav@miyadav-thinkpadx1carbongen8:~/annotations$
Actual results:
service stuck in pending state
Expected results:
If possible it could produce error ( admission webhook may be to pass value to annotation )
Additional info:
describe svc does give info like below -
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 33s (x8 over 10m) service-controller Ensuring load balancer Warning SyncLoadBalancerFailed 32s (x8 over 10m) service-controller Error syncing load balancer: failed to ensure load balancer: unsupported network tier: ""
Description of problem:
Note this is without UDN
While tracking nightly runs we see that there is a big jump in CPU utilization between 4.18.0-0.nightly-2025-01-28-114040 and 4.18.0-0.nightly-2025-01-28-165333 nightlies
and
4.19.0-0.nightly-2025-01-25-021606 and 4.19.0-0.nightly-2025-01-27-130640
nightlies
which corresponds to https://github.com/openshift/ovn-kubernetes/pull/2426 https://github.com/openshift/ovn-kubernetes/pull/2420 PRs being merged
Version-Release number of selected component (if applicable): 4.18 and 4.19
How reproducible:
Always
Steps to Reproduce:
1. Run node-density and check ovnkube-controller cpu usage
Actual results:
Worse CPU utilization
Expected results:
Similar or better CPU utilization
Description of problem:
Non admin users cannot create UserDefinedNetwork instances.
Version-Release number of selected component (if applicable):
How reproducible:
100%
Steps to Reproduce:
1. Create UDN instance as non-admin users.
2.
3.
Actual results:
In the UI, openning the UserDefinedPage fails with the following error
```
userdefinednetworks.k8s.ovn.org is forbidden: User "test" cannot list resource "userdefinednetworks" in API group "k8s.ovn.org" at the cluster scope
```
We get similar error trying to create one.
Expected results:
As a non-admin user I want to be able to create UDN CR w/o cluster-admin intervention.
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
See https://github.com/kubernetes/kubernetes/issues/127352
Version-Release number of selected component (if applicable):
See https://github.com/kubernetes/kubernetes/issues/127352
How reproducible:
See https://github.com/kubernetes/kubernetes/issues/127352
Steps to Reproduce:
See https://github.com/kubernetes/kubernetes/issues/127352
Actual results:
See https://github.com/kubernetes/kubernetes/issues/127352
Expected results:
See https://github.com/kubernetes/kubernetes/issues/127352
Additional info:
See https://github.com/kubernetes/kubernetes/issues/127352
Description of problem:
https://github.com/openshift/api/pull/1997 is the PR where we plan to promote the FG for UDNs in early Jan - the week of 6th
I need 95% pass rate for overlapping IPs test which is currently at 92% threshold,
See the verify jobs: https://storage.googleapis.com/test-platform-results/pr-logs/pull/openshift_api/1997/pull-ci-openshift-api-master-verify/1864369939800920064/build-log.txt that are failing on that PR
Talk with TRT and understand the flakes and improve the tests to get good pass results.
This is blocker for GA
Fix:
INSUFFICIENT CI testing for "NetworkSegmentation". F1204 18:09:40.736814 181286 root.go:64] Error running codegen: error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using NetworkAttachmentDefinitions is isolated from the default network with L2 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {gcp amd64 ha } error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using NetworkAttachmentDefinitions isolates overlapping CIDRs with L3 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {gcp amd64 ha } error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using UserDefinedNetwork is isolated from the default network with L3 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {gcp amd64 ha } error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using UserDefinedNetwork isolates overlapping CIDRs with L3 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {gcp amd64 ha } error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using NetworkAttachmentDefinitions isolates overlapping CIDRs with L3 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {metal amd64 ha ipv4} error: "[sig-network][OCPFeatureGate:NetworkSegmentation][Feature:UserDefinedPrimaryNetworks] when using openshift ovn-kubernetes created using UserDefinedNetwork isolates overlapping CIDRs with L3 primary UDN [Suite:openshift/conformance/parallel]" only passed 92%, need at least 95% for "NetworkSegmentation" on {metal amd64 ha ipv4}
Description of problem:
The 'Plus' button in the 'Edit Pod Count' popup window overlaps the input field, which is incorrect.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-05-103644
How reproducible:
Always
Steps to Reproduce:
1.Navigate to Workloads -> ReplicaSets page, choose one resource, and click the Keban list buton, choose ‘Edit Pod count’ 2. 3.
Actual results:
The Layout is incorrect
Expected results:
The 'Plus' button in the 'Edit Pod Count' popup window should not overlaps the input field
Additional info:
Snapshot: https://drive.google.com/file/d/1mL7xeT7FzkdsM1TZlqGdgCP5BG6XA8uh/view?usp=drive_link https://drive.google.com/file/d/1qmcal_4hypEPjmG6PTG11AJPwdgt65py/view?usp=drive_link
When updating a 4.13 cluster to 4.14, the new-in-4.14 ImageRegistry capability will always be enabled, because capabilities cannot be uninstalled.
4.14 oc should learn about this, so they will appropriately extract registry CredentialsRequests when connecting to 4.13 clusters for 4.14 manifests. 4.15 oc will get OTA-1010 to handle this kind of issue automatically, but there's no problem with getting an ImageRegistry hack into 4.15 engineering candidates in the meantime.
100%
1. Connect your oc to a 4.13 cluster.
2. Extract manifests for a 4.14 release.
3. Check for ImageRegistry CredentialsRequests.
$ oc adm upgrade | head -n1 Cluster version is 4.13.12 $ oc adm release extract --included --credentials-requests --to credentials-requests quay.io/openshift-release-dev/ocp-release:4.14.0-rc.0-x86_64 $ grep -r ImageRegistry credentials-requests ...no hits...
$ oc adm upgrade | head -n1 Cluster version is 4.13.12 $ oc adm release extract --included --credentials-requests --to credentials-requests quay.io/openshift-release-dev/ocp-release:4.14.0-rc.0-x86_64 $ grep -r ImageRegistry credentials-requests credentials-requests/0000_50_cluster-image-registry-operator_01-registry-credentials-request.yaml: capability.openshift.io/name: ImageRegistry
We already do this for MachineAPI. The ImageRegistry capability landed later, and this is us catching the oc-extract hack up with that change.
Description of problem:
When a HostedCluster is initially deployed with a NodePool that does not autoscale, but then the NodePool is later changed to using autoscaling, the autoscaler deployment on the HCP is not scaled up and thus the NodePool does not autoscale. We observe this as failures in the e2e TestAutoscaling
Version-Release number of selected component (if applicable):
How reproducible:
Occasionally
Steps to Reproduce:
1. 2. 3.
Actual results:
When NodePool spec changes to enabling autoscaling, the annotation disabling the autoscaler deployment is not being removed from the HCP
Expected results:
When NodePool spec changes to enabling autoscaling, the annotation disabling the autoscaler deployment in the HCP should be removed
Additional info:
Description of problem:
The Reason for the Available condition should propagate from HostedControlPlane to HostedCluster together with Status and Message. Currently, only Status and Message is propagated: link. In this case we end up with KASLoadBalanderNotReachable in HCP and WaitForAvailable in HC, but we propagate the detailed message:
ᐅ oc get hc $CLUSTER_NAME -n $NAMESPACE -oyaml - lastTransitionTime: "2025-02-13T15:13:55Z" message: 'Get "https://ad470e4971ffe4f24bb0085802628868-46f6d7fdaaca476a.elb.us-east-1.amazonaws.com:6443/healthz": dial tcp: lookup ad470e4971ffe4f24bb0085802628868-46f6d7fdaaca476a.elb.us-east-1.amazonaws.com on 172.31.0.10:53: no such host' observedGeneration: 3 reason: WaitingForAvailable status: "False" type: Available ᐅ oc get hcp hc1 -n clusters-hc1 -oyaml - lastTransitionTime: "2025-02-13T15:14:09Z" message: 'Get "https://ad470e4971ffe4f24bb0085802628868-46f6d7fdaaca476a.elb.us-east-1.amazonaws.com:6443/healthz": dial tcp: lookup ad470e4971ffe4f24bb0085802628868-46f6d7fdaaca476a.elb.us-east-1.amazonaws.com on 172.31.0.10:53: no such host' observedGeneration: 1 reason: KASLoadBalancerNotReachable status: "False" type: Available
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Reproduced as part of https://issues.redhat.com/browse/OCPBUGS-49913 which uses cluster-wide proxy for the management cluster. In this case, the HCP and HC do not become available and show the errors above.
Steps to Reproduce:
Steps described in this JIRA comment
Actual results:
The Reason for HCP is KASLoadBalancerNotReachable while the reason for HC is WaitingForAvailable, but the Message is same in both cases.
Expected results:
The reason KASLoadBalancerNotReachable is propagated to HostedCluster.
Additional info:
Description of problem:
Increase the verbosity of logging for the CVO when applying the manifests in"/etc/cvo/" directory. We have a case when the error is "Permission Denied" but this is very vague. The error in the CVO logs is: ~~~ failure=Unable to download and prepare the update: stat /etc/cvo/updatepayloads/7WNaprXJNWTsPAepCHJ00Q/release-manifests/release-metadata: permission denied. ~~~ At this moment we are unable to figure out the reason why it fails. The CVO mounts the "/etc/cvo" directory as hostPath in Read-Only mode, and the files are in 444 with container_file_t selinux, but the CVO runs with spc_t so selinux should not cause the issue. With that, it would be good to add additional context for the problem, why the CVO can't read the manifests.
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.15
How reproducible:
n/a
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
If resolv-prepender is triggered by the path unit before the dispatcher script has populated the env file it fails because the env file is mandatory. We should make it optional by using EnvironmentFile=-
Version-Release number of selected component (if applicable):
4.16
How reproducible:
$ systemctl cat on-prem-resolv-prepender.service # /etc/systemd/system/on-prem-resolv-prepender.service [Unit] Description=Populates resolv.conf according to on-prem IPI needs # Per https://issues.redhat.com/browse/OCPBUGS-27162 there is a problem if this is started before crio-wipe After=crio-wipe.service StartLimitIntervalSec=0 [Service] Type=oneshot # Would prefer to do Restart=on-failure instead of this bash retry loop, but # the version of systemd we have right now doesn't support it. It should be # available in systemd v244 and higher. ExecStart=/bin/bash -c " \ until \ /usr/local/bin/resolv-prepender.sh; \ do \ sleep 10; \ done" EnvironmentFile=/run/resolv-prepender/env
$ systemctl cat crio-wipe.service
No files found for crio-wipe.service.
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The "now" constant defined at the beginning of the Timestamp component is not a valid Date value.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Create security group rules in batches to reduce the number of calls to the OpenStack API. This is a performance improvement that is not expected to result in any functional change.
Description of problem:
Additional network is not correctly configured on the secondary interface inside the masters and the workers.
With install-config.yaml with this section:
# This file is autogenerated by infrared openshift plugin apiVersion: v1 baseDomain: "shiftstack.local" compute: - name: worker platform: openstack: zones: [] additionalNetworkIDs: ['26a751c3-c316-483c-91ed-615702bcbba9'] type: "worker" replicas: 3 controlPlane: name: master platform: openstack: zones: [] additionalNetworkIDs: ['26a751c3-c316-483c-91ed-615702bcbba9'] type: "master" replicas: 3 metadata: name: "ostest" networking: clusterNetworks: - cidr: fd01::/48 hostPrefix: 64 serviceNetwork: - fd02::/112 machineNetwork: - cidr: "fd2e:6f44:5dd8:c956::/64" networkType: "OVNKubernetes" platform: openstack: cloud: "shiftstack" region: "regionOne" defaultMachinePlatform: type: "master" apiVIPs: ["fd2e:6f44:5dd8:c956::5"] ingressVIPs: ["fd2e:6f44:5dd8:c956::7"] controlPlanePort: fixedIPs: - subnet: name: "subnet-ssipv6" pullSecret: | {"auths": {"installer-host.example.com:8443": {"auth": "ZHVtbXkxMjM6ZHVtbXkxMjM="}}} sshKey: <hidden> additionalTrustBundle: <hidden> imageContentSources: - mirrors: - installer-host.example.com:8443/registry source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - installer-host.example.com:8443/registry source: registry.ci.openshift.org/ocp/release
The installation works. However, the additional network is not configured on the masters or the workers, which leads in our case to faulty manila integration.
In journal of all OCP nodes, it's observed logs repeteadly like below one from the master-0:
Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <info> [1731590504.9667] device (enp4s0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed') Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <warn> [1731590504.9672] device (enp4s0): Activation: failed for connection 'Wired connection 1' Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <info> [1731590504.9674] device (enp4s0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed') Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <info> [1731590504.9768] dhcp4 (enp4s0): canceled DHCP transaction Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <info> [1731590504.9768] dhcp4 (enp4s0): activation: beginning transaction (timeout in 45 seconds) Nov 14 13:21:44 ostest-kmmtt-master-0 NetworkManager[1126]: <info> [1731590504.9768] dhcp4 (enp4s0): state changed no lease
Where that server has specifically an interface connected to the subnet "StorageNFSSubnet":
$ openstack server list | grep master-0 | da23da4a-4af8-4e54-ac60-88d6db2627b6 | ostest-kmmtt-master-0 | ACTIVE | StorageNFS=fd00:fd00:fd00:5000::fb:d8; network-ssipv6=fd2e:6f44:5dd8:c956::2e4 | ostest-kmmtt-rhcos | master |
That subnet is defined in openstack as dhcpv6-stateful:
$ openstack subnet show StorageNFSSubnet +----------------------+-------------------------------------------------------+ | Field | Value | +----------------------+-------------------------------------------------------+ | allocation_pools | fd00:fd00:fd00:5000::fb:10-fd00:fd00:fd00:5000::fb:fe | | cidr | fd00:fd00:fd00:5000::/64 | | created_at | 2024-11-13T12:34:41Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | True | | gateway_ip | None | | host_routes | | | id | 480d7b2a-915f-4f0c-9717-90c55b48f912 | | ip_version | 6 | | ipv6_address_mode | dhcpv6-stateful | | ipv6_ra_mode | dhcpv6-stateful | | name | StorageNFSSubnet | | network_id | 26a751c3-c316-483c-91ed-615702bcbba9 | | prefix_length | None | | project_id | 4566c393806c43b9b4e9455ebae1cbb6 | | revision_number | 0 | | segment_id | None | | service_types | None | | subnetpool_id | None | | tags | | | updated_at | 2024-11-13T12:34:41Z | +----------------------+-------------------------------------------------------+
I also compared with ipv4 installation, and the storageNFSsubnet IP is successfully configured on enp4s0.
Version-Release number of selected component (if applicable):
How reproducible: Always
Additional info: must-gather and journal of the OCP nodes provided in private comment.
Description of problem:
When doing firmware updates we saw cases where the update is successful but the newer information wasn't stored in the HFC, the root cause was that ironic didn't save the newer information in the DB.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
We are in the process of supporting the use of tags on PowerVS resources. So the destroy code needs the ability to query either by name or tag.
Description of problem:
Bootstrapping currently waits to observe 2 endpoints in the "kubernetes" service in HA topologies. The bootstrap kube-apiserver instance itself appears to be included in that number. Soon after observing 2 (bootstrap instance plus one permanent instance), the bootstrap instance is torn down and leaves the cluster with only 1 instance. Each rollout to that instance causes disruption to kube-apiserver availability until the second permanent instance is started for the first time, easily totaling multiple minutes of 0% kube-apiserver availability.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Integrate codespell into Make Verify so that things are spell correctly in our upstream docs and codebase.
Description of problem:
We identified a discrepancy in the cluster-reader ClusterRole between OpenShift 4.14 and OpenShift 4.16. Specifically, the cluster-reader role in OpenShift 4.16 includes permissions for delete, create, update, and patch verbs, which are unexpected for this role. We identified that the cluster-reader ClusterRole in OpenShift 4.16 uses an aggregationRule to pull rules from other ClusterRoles matching the following labels: rbac.authorization.k8s.io/aggregate-to-cluster-reader: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" Further investigation revealed that the system:openshift:machine-config-operator:cluster-reader ClusterRole contributes specific rules under the machineconfiguration.openshift.io API group. These permissions include: Resources: machineconfignodes, machineconfignodes/status, machineosconfigs, machineosconfigs/status, machineosbuilds, machineosbuilds/status Verbs: get, list, watch, delete, create, update, patch The identified permissions appear to originate from the MCO and are linked to the following pull requests: PR 4062 (OCPBUGS-24416) PR 4327 (MCO-1131) Request: Can the MCO team confirm if these additional permissions are intentional? If not, adjustments may be required as the cluster-reader role should not include delete, create, update, or patch verbs.
Version-Release number of selected component (if applicable):
4.16
How reproducible:
100%
Steps to Reproduce:
1. Deploy a fresh OpenShift 4.16 environment. 2. Inspect the rules under the cluster-reader ClusterRole. 3. Observe the inclusion of delete, create, update, and patch verbs for resources under the machineconfiguration.openshift.io API group.
Actual results:
The cluster-reader ClusterRole in OpenShift 4.16 includes unexpected permissions for the above-mentioned verbs.
Expected results:
The cluster-reader ClusterRole in OpenShift 4.16 should not have delete, create, update, and patch verbs.
Additional info:
This behavior deviates from the expected permissions in earlier versions (e.g., OpenShift 4.14) and could lead to potential security or operational concerns.
In 4.8's installer#4760, the installer began passing oc adm release new ... a manifest so the cluster-version operator would manage a coreos-bootimages ConfigMap in the openshift-machine-config-operator namespace. installer#4797 reported issues with the 0.0.1-snapshot placeholder not getting substituted, and installer#4814 attempted to fix that issue by converting the manifest from JSON to YAML to align with the replacement rexexp. But for reasons I don't understand, that manifest still doesn't seem to be getting replaced.
From 4.8 through 4.15.
100%
With 4.8.0:
$ oc adm release extract --to manifests quay.io/openshift-release-dev/ocp-release:4.8.0-x86_64 $ grep releaseVersion manifests/0000_50_installer_coreos-bootimages.yaml
releaseVersion: 0.0.1-snapshot
releaseVersion: 4.8.0
or other output that matches the extracted release. We just don't want the 0.0.1-snapshot placeholder.
Reproducing in the latest 4.14 RC:
$ oc adm release extract --to manifests quay.io/openshift-release-dev/ocp-release:4.14.0-rc.2-x86_64 $ grep releaseVersion manifests/0000_50_installer_coreos-bootimages.yaml releaseVersion: 0.0.1-snapshot
Description of problem:
When user projects monitoring feature is turned off the operator is cleaning up resources for user project monitoring, running multiple delete requests to apiserver. This has several drawbacks: * API Server can't cache DELETE requests, so it has to request etcd every time * Audit log is flooded with "delete failed: object 'foo' not found" record The function should first check that the object exists (GET requests are cachable) before issuing a DELETE request
Version-Release number of selected component (if applicable):
4.16.0
How reproducible:
Always
Steps to Reproduce:
1. Start 4.16.0 2. Check audit log 3.
Actual results:
Audit log has messages like: configmaps "serving-certs-ca-bundle" not found servicemonitors.monitoring.coreos.com "thanos-ruler" not found printed every few minutes
Expected results:
No failed delete requests
Additional info:
The OpenShift installer includes a autoscaling:DescribeAutoScalingGroups IAM permission which I believe its not used and carried over from something that may have existed in Hive a long time ago or maybe not at all.
Reference installer commit
I don't see references for API calls for it Hive in the code and I chatting with the Hive team in slack they too don't see it.
Done criteria:
To do
Description of problem:
Due to the workaround / solution of https://issues.redhat.com/browse/OCPBUGS-42609, namespaces must be created with a specific label to allow the use of primary UDN. This label must be added by the cluster admin - making it impossible for regular users to self-provision their network. With this, the dialog we introduced to the UI where a UDN can be created while defining a Project is no longer functioning (labels cannot be set through Projects). Until a different solution will be introduced, primary UDNs will not be self-service and therefore we should remove the Network tab from Create project dialog.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
1. Open the new project dialog
Actual results:
The "Network" tab is there
Expected results:
It should be removed for now
Additional info:
I'm marking this as critical, since this UI element is very visible and would easily confuse users.
Description of problem:
While generating delete-images.yaml for the pruning of images using oc-mirror v2, the manifest which are generated under working-dir/cluster-resources (IDMS,ITMS etc) are getting deleted automatically
Version-Release number of selected component (if applicable):
4.17
How reproducible:
100% reproducible
Steps to Reproduce:
1- Create a DeleteImageSetConfiguration file like below
apiVersion: mirror.openshift.io/v2alpha1 kind: DeleteImageSetConfiguration delete: platform: channels: - name: stable-4.17 minVersion: 4.17.3 maxVersion: 4.17.3 operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 packages: - name: aws-load-balancer-operator - name: node-observability-operator - name: 3scale-operator additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0
2- ensure that the manifest are generated by the oc-mirror are present in the working-dir/cluster-resources
ls -lrth /opt/417ocmirror/working-dir/cluster-resources/
total 16K
-rw-r--r--. 1 root root 491 Nov 18 21:57 itms-oc-mirror.yaml
-rw-r--r--. 1 root root 958 Nov 18 21:57 idms-oc-mirror.yaml
-rw-r--r--. 1 root root 322 Nov 18 21:57 updateService.yaml
-rw-r--r--. 1 root root 268 Nov 18 21:57 cs-redhat-operator-index-v4-17.yaml
3- Generate the delete-images.yaml using below command
./oc-mirror delete --config ./deleteimageset.yaml --workspace file:///opt/417ocmirror --v2 --generate docker://bastionmirror.amuhamme.upi:8443/417images2024/11/18 23:53:12 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/11/18 23:53:12 [INFO] : 👋 Hello, welcome to oc-mirror 2024/11/18 23:53:12 [INFO] : ⚙️ setting up the environment for you... 2024/11/18 23:53:12 [INFO] : 🔀 workflow mode: diskToMirror / delete 2024/11/18 23:53:12 [INFO] : 🕵️ going to discover the necessary images... 2024/11/18 23:53:12 [INFO] : 🔍 collecting release images... 2024/11/18 23:53:12 [INFO] : 🔍 collecting operator images... 2024/11/18 23:53:13 [INFO] : 🔍 collecting additional images... 2024/11/18 23:53:13 [INFO] : 📄 Generating delete file... 2024/11/18 23:53:13 [INFO] : /opt/417ocmirror/working-dir/delete file created 2024/11/18 23:53:13 [INFO] : delete time : 712.42082ms 2024/11/18 23:53:13 [INFO] : 👋 Goodbye, thank you for using oc-mirror
4- Verify after generating the delete-images.yaml the manifests present in the working-dir/cluster-resources/ got deleted.
# ls -lrth /opt/417ocmirror/working-dir/cluster-resources/ total 0 # ls -lrth /opt/417ocmirror/working-dir/delete total 72K -rwxr-xr-x. 1 root root 65K Nov 18 23:53 delete-images.yaml -rwxr-xr-x. 1 root root 617 Nov 18 23:53 delete-imageset-config.yaml
Actual results:
Generating delete-images.yaml is deleting the manifest under working-dir/cluster-resources/
Expected results:
Generating delete-images.yaml should not delete the manifest under working-dir/cluster-resources/
Additional info:
// This is [Serial] because it modifies ClusterCSIDriver.var _ = g.Describe("[sig-storage][FeatureGate:VSphereDriverConfiguration][Serial][apigroup:operator.openshift.io] vSphere CSI Driver Configuration", func() { defer g.GinkgoRecover() var ( ctx = context.Background() oc = exutil.NewCLI(projectName) originalDriverConfigSpec *opv1.CSIDriverConfigSpec )
o.SetDefaultEventuallyTimeout(5 * time.Minute) o.SetDefaultEventuallyPollingInterval(5 * time.Second)
In OCP Origin we have the above test playing with global variables for poll interval and poll timeout which is causing all other tests in origin to have flakes.
a networking test is failing because we are not polling correctly, this above test overrode the default poll value of 10ms and instead made it poll 5sec which caused the test to fail because out poll timeout was itself only 5seconds
Please don't use the global variables or maybe we can unset them after the test run is over?
Please note that this causes flakes that are hard to debug, we didn't know what was causing the poll interval to be 5seconds instead of the default 10ms.
Description of problem:
checked in 4.18.0-0.nightly-2024-12-05-103644/4.19.0-0.nightly-2024-12-04-03122, admin console go to "Observe -> Metrics", execute one query, make sure there is result for it, for example "cluster_version", click the kebab menu, "Show all series" under the list, it's wrong, should be "Hide all series", click "Show all series" will unselect all series, then "Hide all series" always show under the menu, click it, the series would be changed from selected and unselected, but always see "Hide all series", see recording: https://drive.google.com/file/d/1kfwAH7FuhcloCFdRK--l01JYabtzcG6e/view?usp=drive_link
same issue for developer console for 4.18+, 4.17 and below does not have such issue
Version-Release number of selected component (if applicable):
4.18+
How reproducible:
always with 4.18+
Steps to Reproduce:
see the description
Actual results:
Hide/Show all series status under"Observe -> Metrics" kebab menu is wrong
Expected results:
should be right
Description of problem:
On the Azure HCP cluster when creating internal ingress controller we are getting authorization error
Version-Release number of selected component (if applicable):
4.19 and may be further versions
How reproducible:
create internal ingress controller in cluster bot or prowci created Azure HCP cluster
Steps to Reproduce:
1.Create a internal ingress controller mjoseph@mjoseph-mac Downloads % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.19.0-0.nightly-2025-01-21-163021 True False False 107m csi-snapshot-controller 4.19.0-0.nightly-2025-01-21-163021 True False False 120m dns 4.19.0-0.nightly-2025-01-21-163021 True False False 107m image-registry 4.19.0-0.nightly-2025-01-21-163021 True False False 107m ingress 4.19.0-0.nightly-2025-01-21-163021 True False False 108m insights 4.19.0-0.nightly-2025-01-21-163021 True False False 109m kube-apiserver 4.19.0-0.nightly-2025-01-21-163021 True False False 121m kube-controller-manager 4.19.0-0.nightly-2025-01-21-163021 True False False 121m kube-scheduler 4.19.0-0.nightly-2025-01-21-163021 True False False 121m kube-storage-version-migrator 4.19.0-0.nightly-2025-01-21-163021 True False False 109m monitoring 4.19.0-0.nightly-2025-01-21-163021 True False False 102m network 4.19.0-0.nightly-2025-01-21-163021 True False False 120m node-tuning 4.19.0-0.nightly-2025-01-21-163021 True False False 112m openshift-apiserver 4.19.0-0.nightly-2025-01-21-163021 True False False 121m openshift-controller-manager 4.19.0-0.nightly-2025-01-21-163021 True False False 121m openshift-samples 4.19.0-0.nightly-2025-01-21-163021 True False False 107m operator-lifecycle-manager 4.19.0-0.nightly-2025-01-21-163021 True False False 121m operator-lifecycle-manager-catalog 4.19.0-0.nightly-2025-01-21-163021 True False False 121m operator-lifecycle-manager-packageserver 4.19.0-0.nightly-2025-01-21-163021 True False False 121m service-ca 4.19.0-0.nightly-2025-01-21-163021 True False False 109m storage 4.19.0-0.nightly-2025-01-21-163021 True False False 109m mjoseph@mjoseph-mac Downloads % oc get ingresses.config/cluster -o jsonpath={.spec.domain} apps.93499d233a19644b81ad.qe.azure.devcluster.openshift.com% mjoseph@mjoseph-mac Downloads % oc create -f New\ Folder\ With\ Items/internal_ingress_controller.yaml ingresscontroller.operator.openshift.io/internal created mjoseph@mjoseph-mac Downloads % mjoseph@mjoseph-mac Downloads % mjoseph@mjoseph-mac Downloads % mjoseph@mjoseph-mac Downloads % cat New\ Folder\ With\ Items/internal_ingress_controller.yaml kind: IngressController apiVersion: operator.openshift.io/v1 metadata: name: internal namespace: openshift-ingress-operator spec: domain: internal.93499d233a19644b81ad.qe.azure.devcluster.openshift.com replicas: 1 endpointPublishingStrategy: loadBalancer: scope: Internal type: LoadBalancerService 2. Check the controller status mjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator get ingresscontroller NAME AGE default 139m internal 29s mjoseph@mjoseph-mac Downloads % oc get po -n openshift-ingress NAME READY STATUS RESTARTS AGE router-default-5c4db6659b-7cq46 1/1 Running 0 128m router-internal-6b6547cb9-hhtzq 1/1 Running 0 39s mjoseph@mjoseph-mac Downloads % mjoseph@mjoseph-mac Downloads % mjoseph@mjoseph-mac Downloads % oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE ingress 4.19.0-0.nightly-2025-01-21-163021 True True False 127m Not all ingress controllers are available. 3. Check the internal ingress controller status mjoseph@mjoseph-mac Downloads % oc -n openshift-ingress-operator get ingresscontroller internal -oyaml apiVersion: operator.openshift.io/v1 kind: IngressController metadata: creationTimestamp: "2025-01-23T07:46:15Z" finalizers: - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller generation: 2 name: internal namespace: openshift-ingress-operator resourceVersion: "29755" uid: 29244558-4d19-4ea4-a5b8-e98b9c07edb3 spec: clientTLS: clientCA: name: "" clientCertificatePolicy: "" domain: internal.93499d233a19644b81ad.qe.azure.devcluster.openshift.com endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Managed scope: Internal type: LoadBalancerService httpCompression: {} httpEmptyRequestsPolicy: Respond httpErrorCodePages: name: "" replicas: 1 tuningOptions: reloadInterval: 0s unsupportedConfigOverrides: null status: availableReplicas: 1 conditions: - lastTransitionTime: "2025-01-23T07:46:15Z" reason: Valid status: "True" type: Admitted - lastTransitionTime: "2025-01-23T07:46:50Z" message: The deployment has Available status condition set to True reason: DeploymentAvailable status: "True" type: DeploymentAvailable - lastTransitionTime: "2025-01-23T07:46:50Z" message: Minimum replicas requirement is met reason: DeploymentMinimumReplicasMet status: "True" type: DeploymentReplicasMinAvailable - lastTransitionTime: "2025-01-23T07:46:50Z" message: All replicas are available reason: DeploymentReplicasAvailable status: "True" type: DeploymentReplicasAllAvailable - lastTransitionTime: "2025-01-23T07:46:50Z" message: Deployment is not actively rolling out reason: DeploymentNotRollingOut status: "False" type: DeploymentRollingOut - lastTransitionTime: "2025-01-23T07:46:16Z" message: The endpoint publishing strategy supports a managed load balancer reason: WantedByEndpointPublishingStrategy status: "True" type: LoadBalancerManaged - lastTransitionTime: "2025-01-23T07:46:16Z" message: |- The service-controller component is reporting SyncLoadBalancerFailed events like: Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '51b4e7f0-f41b-4b52-9bfc-412366b68308' with object id '51b4e7f0-f41b-4b52-9bfc-412366b68308' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-ln-wqg34k2-c04e6-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-ln-wqg34k2-c04e6-vnet/subnets/ci-ln-wqg34k2-c04e6-subnet' or the scope is invalid. If access was recently granted, please refresh your credentials."}} The cloud-controller-manager logs may contain more details. reason: SyncLoadBalancerFailed status: "False" type: LoadBalancerReady - lastTransitionTime: "2025-01-23T07:46:16Z" message: LoadBalancer is not progressing reason: LoadBalancerNotProgressing status: "False" type: LoadBalancerProgressing - lastTransitionTime: "2025-01-23T07:46:16Z" message: DNS management is supported and zones are specified in the cluster DNS config. reason: Normal status: "True" type: DNSManaged - lastTransitionTime: "2025-01-23T07:46:16Z" message: The wildcard record resource was not found. reason: RecordNotFound status: "False" type: DNSReady - lastTransitionTime: "2025-01-23T07:46:16Z" message: |- One or more status conditions indicate unavailable: LoadBalancerReady=False (SyncLoadBalancerFailed: The service-controller component is reporting SyncLoadBalancerFailed events like: Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '51b4e7f0-f41b-4b52-9bfc-412366b68308' with object id '51b4e7f0-f41b-4b52-9bfc-412366b68308' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-ln-wqg34k2-c04e6-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-ln-wqg34k2-c04e6-vnet/subnets/ci-ln-wqg34k2-c04e6-subnet' or the scope is invalid. If access was recently granted, please refresh your credentials."}} The cloud-controller-manager logs may contain more details.) reason: IngressControllerUnavailable status: "False" type: Available - lastTransitionTime: "2025-01-23T07:46:50Z" status: "False" type: Progressing - lastTransitionTime: "2025-01-23T07:47:46Z" message: |- One or more other status conditions indicate a degraded state: LoadBalancerReady=False (SyncLoadBalancerFailed: The service-controller component is reporting SyncLoadBalancerFailed events like: Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '51b4e7f0-f41b-4b52-9bfc-412366b68308' with object id '51b4e7f0-f41b-4b52-9bfc-412366b68308' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/ci-ln-wqg34k2-c04e6-vnet-rg/providers/Microsoft.Network/virtualNetworks/ci-ln-wqg34k2-c04e6-vnet/subnets/ci-ln-wqg34k2-c04e6-subnet' or the scope is invalid. If access was recently granted, please refresh your credentials."}} The cloud-controller-manager logs may contain more details.) reason: DegradedConditions status: "True" type: Degraded - lastTransitionTime: "2025-01-23T07:46:16Z" message: IngressController is upgradeable. reason: Upgradeable status: "True" type: Upgradeable - lastTransitionTime: "2025-01-23T07:46:16Z" message: No evaluation condition is detected. reason: NoEvaluationCondition status: "False" type: EvaluationConditionsDetected domain: internal.93499d233a19644b81ad.qe.azure.devcluster.openshift.com endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Managed scope: Internal type: LoadBalancerService observedGeneration: 2 selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=internal tlsProfile: ciphers: - ECDHE-ECDSA-AES128-GCM-SHA256 - ECDHE-RSA-AES128-GCM-SHA256 - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384 - ECDHE-ECDSA-CHACHA20-POLY1305 - ECDHE-RSA-CHACHA20-POLY1305 - DHE-RSA-AES128-GCM-SHA256 - DHE-RSA-AES256-GCM-SHA384 - TLS_AES_128_GCM_SHA256 - TLS_AES_256_GCM_SHA384 - TLS_CHACHA20_POLY1305_SHA256 minTLSVersion: VersionTLS12 mjoseph@mjoseph-mac Downloads %
Actual results:
mjoseph@mjoseph-mac Downloads % oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE ingress 4.19.0-0.nightly-2025-01-21-163021 True True False 127m Not all ingress controllers are available.
Expected results:
the internal controller should come up
Additional info:
One more test scenario which is causing the similar error in the HCP cluster in internal LB 1. Create a web server with two services mjoseph@mjoseph-mac Downloads % oc create -f New\ Folder\ With\ Items/webrc.yaml replicationcontroller/web-server-rc created service/service-secure created service/service-unsecure created mjoseph@mjoseph-mac Downloads % oc get po NAME READY STATUS RESTARTS AGE web-server-rc-q87rv 1/1 Running 0 40s mjoseph@mjoseph-mac Downloads % oc get svc oc geNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 172.31.0.1 <none> 443/TCP 152m openshift ExternalName <none> kubernetes.default.svc.cluster.local <none> 147m openshift-apiserver ClusterIP 172.31.165.239 <none> 443/TCP 150m openshift-oauth-apiserver ClusterIP 172.31.254.44 <none> 443/TCP 150m packageserver ClusterIP 172.31.131.10 <none> 443/TCP 150m service-secure ClusterIP 172.31.6.17 <none> 27443/TCP 46s service-unsecure ClusterIP 172.31.199.11 <none> 27017/TCP 46s 2. Add two lb services mjoseph@mjoseph-mac Downloads % oc create -f ../Git/openshift-tests-private/test/extended/testdata/router/bug2013004-lb-services.yaml service/external-lb-57089 created service/internal-lb-57089 created mjoseph@mjoseph-mac Downloads % cat ../Git/openshift-tests-private/test/extended/testdata/router/bug2013004-lb-services.yaml apiVersion: v1 kind: List items: - apiVersion: v1 kind: Service metadata: name: external-lb-57089 spec: ports: - name: https port: 28443 protocol: TCP targetPort: 8443 selector: name: web-server-rc type: LoadBalancer - apiVersion: v1 kind: Service metadata: name: internal-lb-57089 annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" spec: ports: - name: https port: 29443 protocol: TCP targetPort: 8443 selector: name: web-server-rc type: LoadBalancer 3. Check the external ip of the internal service, which is not yet assigned mjoseph@mjoseph-mac Downloads % oc get svc -owide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR external-lb-57089 LoadBalancer 172.31.248.177 20.83.73.54 28443:30437/TCP 44s name=web-server-rc internal-lb-57089 LoadBalancer 172.31.156.88 <pending> 29443:31885/TCP 44s name=web-server-rc kubernetes ClusterIP 172.31.0.1 <none> 443/TCP 153m <none> openshift ExternalName <none> kubernetes.default.svc.cluster.local <none> 148m <none> openshift-apiserver ClusterIP 172.31.165.239 <none> 443/TCP 151m <none> openshift-oauth-apiserver ClusterIP 172.31.254.44 <none> 443/TCP 151m <none> packageserver ClusterIP 172.31.131.10 <none> 443/TCP 151m <none> service-secure ClusterIP 172.31.6.17 <none> 27443/TCP 112s name=web-server-rc service-unsecure ClusterIP 172.31.199.11 <none> 27017/TCP 112s name=web-server-rc
Description of problem:
https://access.redhat.com/errata/RHSA-2024:5422 did not seemingly fix the issue https://issues.redhat.com/browse/OCPBUGS-37060 in ROSA HCP so opening new bug. The builds installed in the hosted clusters are having issues to git-clone repositories from external URLs where their CA are configured in the ca-bundle.crt from trsutedCA section: spec: configuration: apiServer: [...] proxy: trustedCA: name: user-ca-bundle <--- In traditional OCP implementations, the *-global-ca configmap is installed in the same namespace from the build and the ca-bundle.crt is injected into this configmap. In hosted clusters the configmap is being created empty: $ oc get cm -n <app-namespace> <build-name>-global-ca -oyaml apiVersion: v1 data: ca-bundle.crt: "" As mentioned, the user-ca-bundle has the certificates configured: $ oc get cm -n openshift-config user-ca-bundle -oyaml apiVersion: v1 data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- <---
Version-Release number of selected component (if applicable):
4.16.17
How reproducible:
Steps to Reproduce:
1. Install hosted cluster with trustedCA configmap 2. Run a build in the hosted cluster 3. Check the global-ca configmap
Actual results:
global-ca is empty
Expected results:
global-ca injects the ca-bundle.crt properly
Additional info:
Created a new ROSA HCP cluster behind a transparent proxy at version 4.16.8 as it was mentioned as fixed in the above errata and the issue still exists. The transparent proxy certificate provided at cluster installation time is referenced in proxy/cluster as "user-ca-bundle-abcdefgh" and both "user-ca-bundle" and "user-ca-bundle-abcdefgh" configmaps in the "openshift-config" contain the certificate. However starting a template build for example such as "oc new-app cakephp-mysql-persistent" still results in the certificate not being injected into the "cakephp-mysql-persistent-1-global-ca" configmap and the build failing unlike the same scenario in an OCP cluster. oc logs build.build.openshift.io/cakephp-mysql-persistent-1 Cloning "https://github.com/sclorg/cakephp-ex.git" ... error: fatal: unable to access 'https://github.com/sclorg/cakephp-ex.git/': SSL certificate problem: unable to get local issuer certificate Also upgraded the cluster to 4.16.17 and still the issue persists.
Description of problem:
Textarea resizing causes it to overflow outside the window.
Version-Release number of selected component (if applicable):
4.19.0-0.test-2025-02-14-032806
How reproducible:
Always
Steps to Reproduce:
1. Find a resizable textarea, for example on Create Project page 2. Resizing the textarea for the description 3.
Actual results:
the textarea is out of windows
Expected results:
the textarea should not out of windows
Additional info:
https://drive.google.com/file/d/1m_UCHju1FXYGbMJBa4P4R1SJDoOUsIkG/view?usp=drive_link
Security Tracking Issue
Do not make this issue public.
Flaw:
Non-linear parsing of case-insensitive content in golang.org/x/net/html
https://bugzilla.redhat.com/show_bug.cgi?id=2333122
An attacker can craft an input to the Parse functions that would be processed non-linearly with respect to its length, resulting in extremely slow parsing. This could cause a denial of service.
Original bug title:
cert-manager [v1.15 Regression] Failed to issue certs with ACME Route53 dns01 solver in AWS STS env
Description of problem:
When using Route53 as the dns01 solver to create certificates, it fails in both automated and manual tests. For the full log, please refer to the "Actual results" section.
Version-Release number of selected component (if applicable):
cert-manager operator v1.15.0 staging build
How reproducible:
Always
Steps to Reproduce: also documented in gist
1. Install the cert-manager operator 1.15.0 2. Follow the doc to auth operator with AWS STS using ccoctl: https://docs.openshift.com/container-platform/4.16/security/cert_manager_operator/cert-manager-authenticate.html#cert-manager-configure-cloud-credentials-aws-sts_cert-manager-authenticate 3. Create a ACME issuer with Route53 dns01 solver 4. Create a cert using the created issuer
OR:
Refer by running `/pj-rehearse pull-ci-openshift-cert-manager-operator-master-e2e-operator-aws-sts` on https://github.com/openshift/release/pull/59568
Actual results:
1. The certificate is not Ready. 2. The challenge of the cert is stuck in the pending status: PresentError: Error presenting challenge: failed to change Route 53 record set: operation error Route 53: ChangeResourceRecordSets, get identity: get credentials: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region
Expected results:
The certificate should be Ready. The challenge should succeed.
Additional info:
The only way to get it working again seems to be injecting the "AWS_REGION" environment variable into the controller pod. See upstream discussion/change:
I couldn't find a way to inject the env var into our operator-managed operands, so I only verified this workaround using the upstream build v1.15.3. After applying the patch with the following command, the challenge succeeded and the certificate became Ready.
oc patch deployment cert-manager -n cert-manager \ --patch '{"spec": {"template": {"spec": {"containers": [{"name": "cert-manager-controller", "env": [{"name": "AWS_REGION", "value": "aws-global"}]}]}}}}'
Creating clusters in which machines are created in a public subnet and use a public IP makes it possible to avoid creating NAT gateways (or proxies) for AWS clusters. While not applicable for every test, this configuration will save us money and cloud resources.
Description of problem:
Bonded network configurations with mode=active-backup and fail_over_mac=follow are not functioning due to a race in /var/usrlocal/bin/configure-ovs.sh
This race condition results in flapping.
The customer who encountered the issue, in July, worked with the IBM LTC Power team to track the issue through the Linux Kernel to OVN-Kube and into the MCO configuration. The customer details can be shared in slack.
The corresponding BZ https://bugzilla.linux.ibm.com/show_bug.cgi?id=210291 could not be mirrored.
The GH issue is in https://github.com/openshift/machine-config-operator/issues/4605
The fix is in https://github.com/openshift/machine-config-operator/pull/4609
From Dave Wilder... the interfaces are setup as described in the issue...
At this point the MACs of the bond's slaves (enP32807p1s0,enP49154p1s0) are the same. The purpose of fail_over_mac=follow is to insure the MACs will not be the same. This is preventing the bond from functioning. This initially appeared to be a problem with the bonding driver, after tracing the calls NetworkManager is making to the bonding driver I discovered the root of the problem is in configure-ovs.sh.
The function: activate_nm_connections() attempts to activate all its generated profiles that are not currently in the "active" state. In my case the following profiles are activated one at a time in this order:
br-ex, ovs-if-phys0, enP32807p1s0-slave-ovs-clone, enP49154p1s0-slave-ovs-clone, ovs-if-br-ex
However the generated profiles have autoconnect-slaves set, therefore when br-ex is activated ovs-if-phys0, enP32807p1s0-slave-ovs-clone and enP49154p1s0-slave-ovs-clone's state changes to "activating", as we are only checking for the "activated" state these profiles may be activated again. As the list is walked, some of the profile's state will automatically go from activating to active. These interfaces are not activated a second time leaving the state of the bond in an unpredictable state. I am able to see in the bonding traces why both slave interface have the same MAC.
My fix is to check for either activating or active states.
— configure-ovs.sh 2024-09-20 15:29:03.160536239 -0700
+++ configure-ovs.sh.patched 2024-09-20 15:33:38.040336032 -0700
@@ -575,8 +575,8 @@
- But set the entry in master_interfaces to true if this is a slave
- Also set autoconnect to yes
local active_state=$(nmcli -g GENERAL.STATE conn show "$conn")
Version-Release number of selected component (if applicable): First seen in 4.14 OVN-Kube
How reproducible: Specific OVN-Kube configuration with network bonding set for fail_over_mac=follow. This is the ideal setting for the SR-IOV/Network setup at the customer site where they rely on high availability.
Steps to Reproduce:
1. Setup the interfaces as described.
Actual results: Failed Bonding
Expected results: No flapping and the failover workers
Additional info:
https://github.com/openshift/machine-config-operator/issues/4605
https://github.com/openshift/machine-config-operator/pull/4609
#rhel-netorking-subsystem https://redhat-internal.slack.com/archives/C04NN96F1S4/p1719943109040989
Description of problem:
clustercatalog is a cluster scoped resource and it does not need to have namespace mentioned in its yaml file. Currently below is how it is been seen. {code:java} apiVersion: olm.operatorframework.io/v1 kind: ClusterCatalog metadata: name: cc-redhat-operator-index-v4-17 namespace: openshift-marketplace spec: priority: 0 source: image: ref: ec2-3-137-182-27.us-east-2.compute.amazonaws.com:5000/cc/redhat/redhat-operator-index:v4.17 type: Image status: {}
Version-Release number of selected component (if applicable):{code:none}
4.18
How reproducible:
Always
Steps to Reproduce:
1. Mirror an imageSetConfig as shown below with the PR which has fix for clusterCatlaog {code:java} kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 packages: - name: security-profiles-operator
2. Run command `oc-mirror -c isc.yaml --workspace file://test docker://localhost:5000 --v2 --dest-tls-verify=false
3.
Actual results:{code:none} Generating ClusterCatalog yaml has namespace as shown in the description.
Expected results:
There should not be any namespace in the yaml file since clustercatalog is a cluster scoped resource.
Additional info:
https://redhat-internal.slack.com/archives/C050P27C71S/p1738324252074669?thread_ts=1738322462.210089&cid=C050P27C71S
Description of problem:
the go to arrow and new doc link icon not aligned with text any more
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-12-144418
How reproducible:
Always
Steps to Reproduce:
1. goes to Home -> Overview page 2. 3.
Actual results:
the go to arrow and new doc link icon are not horizontal aligned with their text any more
Expected results:
icon and text should be aligned
Additional info:
screenshot https://drive.google.com/file/d/1S61XY-lqmmJgGbwB5hcR2YU_O1JSJPtI/view?usp=drive_link
Description of problem:
After kubevirt vm live migration, VM egress traffic still going over the previous node wher the VM was running before live migration.
Version-Release number of selected component (if applicable): 4.19
How reproducible: Always
Steps to Reproduce:
1. Create a vm with, primary UDN, l2 topology and ipv4 2. Send egress traffic 3 . Live migrate the VM 4. Send egress traffic
Actual results:
IPv4 egress traffic goes over the node was running before live migration
Expected results:
IPv4 Egress traffic should go over the node where the VM is running after the live migration
Additional info:
The problem is related to VM ipv4 neighbors cache not being updated, to fix this a GARP should be send by ovn-kubernetes to update that cache after live migration.
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a regression in the following test:
install should succeed: overall
Extreme regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 98.88% to 57.89%.
Overrode base stats using release 4.17
Sample (being evaluated) Release: 4.19
Start Time: 2025-03-03T00:00:00Z
End Time: 2025-03-10T08:00:00Z
Success Rate: 57.89%
Successes: 22
Failures: 16
Flakes: 0
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T00:00:00Z
Success Rate: 98.88%
Successes: 88
Failures: 1
Flakes: 0
View the test details report for additional context.
gcp installs seem to be failing frequently with the error:
These cluster operators were not stable: [openshift-samples]
The samples operator reports:
status: conditions: - lastTransitionTime: "2025-03-09T19:56:05Z" status: "False" type: Degraded - lastTransitionTime: "2025-03-09T19:56:17Z" message: Samples installation successful at 4.19.0-0.nightly-2025-03-09-190956 status: "True" type: Available - lastTransitionTime: "2025-03-09T20:43:02Z" message: "Samples installed at 4.19.0-0.nightly-2025-03-09-190956, with image import failures for these imagestreams: java,kube-root-ca.crt,openshift-service-ca.crt,nodejs; last import attempt 2025-03-09 19:57:39 +0000 UTC" reason: FailedImageImports status: "False" type: Progressing
I'm confused how this is failing install given available=true and degraded=false, and yet there does appear to be a problem reported in the message. It is possible this artifact was collected a few minutes after the install failed, is it possible the operator stabilizes (ignores these errors) in that time? Note that not all installs are failing this way, but a good chunk.
Problem appears limited to 4.19 gcp, I do see one hit for vsphere though.
Description of problem:
Disable delete button for UDN if the UDN cannot be deleted
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create a UDN and then delete it 2. 3.
Actual results:
The UDN is not deleted
Expected results:
Disable the delete button if UDN cannot be removed
Additional info:
Description of problem:
when deleting operator images with v2 for operator images mirrored by v1 from oci catalog, oc-mirror doesn' t find the same tags to delete and fails to delete the images
Version-Release number of selected component (if applicable):
GitCommit:"affa0177"
How reproducible:
always
Steps to Reproduce:
1. Mirror to mirror with v1: ./bin/oc-mirror -c config_logs/bugs.yaml docker://localhost:5000/437311 --dest-skip-tls --dest-use-http kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 mirror: operators: - catalog: oci:///home/skhoury/redhat-index-all targetCatalog: "ocicatalog73452" targetTag: "v16" packages: - name: cluster-kube-descheduler-operator 2. mirror to disk with v2, and almost same ISC (but v2alpha1): ./bin/oc-mirror --v2 -c config_logs/bugs.yaml file:///home/skhoury/43774v2 3. delete with ./bin/oc-mirror delete --generate --delete-v1-images --v2 -c config_logs/bugs.yaml --workspace file:///home/skhoury/43774v2 docker://sherinefedora:5000/437311 kind: DeleteImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 delete: operators: - catalog: oci:///home/skhoury/redhat-index-all targetCatalog: "ocicatalog73452" targetTag: "v16" packages: - name: cluster-kube-descheduler-operator
Actual results:
mapping.txt of v1: registry.redhat.io/openshift-sandboxed-containers/osc-cloud-api-adaptor-webhook-rhel9@sha256:4da2fe27ef0235afcac1a1b5e90522d072426f58c0349702093ea59c40e5ca68=localhost:5000/437311/openshift-sandboxed-containers/osc-cloud-api-adaptor-webhook-rhel9:491be520 registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:cb836456974e510eb4bccbffadbc6d99d5f57c36caec54c767a158ffd8a025d5=localhost:5000/437311/openshift4/ose-kube-rbac-proxy:d07492b2 registry.redhat.io/openshift-sandboxed-containers/osc-operator-bundle@sha256:7465f4e228cfc44a3389f042f7d7b68d75cbb03f2adca1134a7ec417bbd89663=localhost:5000/437311/openshift-sandboxed-containers/osc-operator-bundle:a2f35fa7 registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator@sha256:c6c589d5e47ba9564c66c84fc2bc7e5e046dae1d56a3dc99d7343f01e42e4d31=localhost:5000/437311/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator:d7b79dea registry.redhat.io/openshift-sandboxed-containers/osc-operator-bundle@sha256:ff2bb666c2696fed365df55de78141a02e372044647b8031e6d06e7583478af4=localhost:5000/437311/openshift-sandboxed-containers/osc-operator-bundle:695e2e19 registry.redhat.io/openshift-sandboxed-containers/osc-rhel8-operator@sha256:5d2b03721043e5221dfb0cf164cf59eba396ba3aae40a56c53aa3496c625eea0=localhost:5000/437311/openshift-sandboxed-containers/osc-rhel8-operator:204cb113 registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator@sha256:b1e824e126c579db0f56d04c3d1796d82ed033110c6bc923de66d95b67099611=localhost:5000/437311/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator:1957f330 registry.redhat.io/openshift-sandboxed-containers/osc-rhel8-operator@sha256:a660f0b54b9139bed9a3aeef3408001c0d50ba60648364a98a09059b466fbcc1=localhost:5000/437311/openshift-sandboxed-containers/osc-rhel8-operator:ab38b9d5 registry.redhat.io/openshift-sandboxed-containers/osc-operator-bundle@sha256:8da62ba1c19c905bc1b87a6233ead475b047a766dc2acb7569149ac5cfe7f0f1=localhost:5000/437311/openshift-sandboxed-containers/osc-operator-bundle:1adce9f registry.redhat.io/redhat/redhat-operator-index:v4.15=localhost:5000/437311/redhat/redhat-operator-index:v4.15 registry.redhat.io/openshift-sandboxed-containers/osc-monitor-rhel9@sha256:03381ad7a468abc1350b229a8a7f9375fcb315e59786fdacac8e5539af4a3cdc=localhost:5000/437311/openshift-sandboxed-containers/osc-monitor-rhel9:53bbc3cb registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-operator-bundle@sha256:2808a0397495982b4ea0001ede078803a043d5c9b0285662b08044fe4c11f243=localhost:5000/437311/openshift-sandboxed-containers-tech-preview/osc-operator-bundle:c30c7861 registry.redhat.io/openshift-sandboxed-containers/osc-podvm-payload-rhel9@sha256:4bca24d469a41be77db7450e02fa01660a14f4c68e829cba4a8ae253d427bbfd=localhost:5000/437311/openshift-sandboxed-containers/osc-podvm-payload-rhel9:d25beb31 registry.redhat.io/openshift-sandboxed-containers/osc-cloud-api-adaptor-rhel9@sha256:7185c1b6658147e2cfbb0326e6b5f59899f14f5de73148ef9a07aa5c7b9ead74=localhost:5000/437311/openshift-sandboxed-containers/osc-cloud-api-adaptor-rhel9:18ba6d86 registry.redhat.io/openshift-sandboxed-containers/osc-rhel8-operator@sha256:8f30a9129d817c3f4e404d2c43fb47e196d8c8da3badba4c48f65d440a4d7584=localhost:5000/437311/openshift-sandboxed-containers/osc-rhel8-operator:17b81cfd registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator@sha256:051bd7f1dad8cc3251430fee32184be8d64077aba78580184cef0255d267bdcf=localhost:5000/437311/openshift-sandboxed-containers-tech-preview/osc-rhel8-operator:6a87f996 registry.redhat.io/openshift-sandboxed-containers/osc-rhel9-operator@sha256:3e3b8849f8a0c8cd750815e6bde7eb2006e5a2b4ea898c9d3ea27f2bfed635d9=localhost:5000/437311/openshift-sandboxed-containers/osc-rhel9-operator:4c46a1f7 registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-operator-bundle@sha256:a91cee14f47824ce49759628d06bf4e48276e67dae00b50123d3233d78531720=localhost:5000/437311/openshift-sandboxed-containers-tech-preview/osc-operator-bundle:d22b8cff registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:7efeeb8b29872a6f0271f651d7ae02c91daea16d853c50e374c310f044d8c76c=localhost:5000/437311/openshift4/ose-kube-rbac-proxy:5574585a registry.redhat.io/openshift-sandboxed-containers/osc-podvm-builder-rhel9@sha256:a4099ea5ad907ad1daee3dc2c9d659b5a751adf2da65f8425212e82577b227e7=localhost:5000/437311/openshift-sandboxed-containers/osc-podvm-builder-rhel9:36a60f3f delete-images.yaml of v2 apiVersion: mirror.openshift.io/v2alpha1 items: - imageName: docker://registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator-bundle@sha256:b473fba287414d3ccb09aaabc64f463af2c912c322ca2c41723020b216d98d14 imageReference: docker://sherinefedora:5000/437311/openshift4/ose-cluster-kube-descheduler-operator-bundle:52836815 type: operatorBundle - imageName: docker://registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator-bundle@sha256:b148d5cf4943d0341781a0f7c6f2a7116d315c617f8beb65c9e7a24ac99304ff imageReference: docker://sherinefedora:5000/437311/openshift4/ose-cluster-kube-descheduler-operator-bundle:bd7c9abe type: operatorBundle - imageName: docker://registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator@sha256:c7b198e686dc7117994d71027710ebc6ac0bf21afa436a79794d2e64970c8003 imageReference: docker://sherinefedora:5000/437311/openshift4/ose-cluster-kube-descheduler-operator:223f8a32 type: operatorRelatedImage - imageName: docker://registry.redhat.io/openshift4/ose-cluster-kube-descheduler-operator@sha256:ba0b71ff2a30a069b4a8a8f3c1e0898aaadc6db112e4cc12aff7c77ced7a0405 imageReference: docker://sherinefedora:5000/437311/openshift4/ose-cluster-kube-descheduler-operator:b0b2a0ab type: operatorRelatedImage - imageName: docker://registry.redhat.io/openshift4/ose-descheduler@sha256:257b69180cc667f2b8c1ce32c60fcd23a119195ad9ba2fdd6a6155ec5290f8cf imageReference: docker://sherinefedora:5000/437311/openshift4/ose-descheduler:6585e5e1 type: operatorRelatedImage - imageName: docker://registry.redhat.io/openshift4/ose-descheduler@sha256:45dc69ad93ab50bdf9ce1bb79f6d98f849e320db68af30475b10b7f5497a1b13 imageReference: docker://sherinefedora:5000/437311/openshift4/ose-descheduler:7ac5ce2 type: operatorRelatedImage kind: DeleteImageList
Expected results:
same tags found for destination images
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
4.16 and newer
How reproducible:
Always
Steps to Reproduce:
Actual results:
Expected results:
Additional info:
$ oc --context sharedocp416-sbr patch clustercsidriver csi.vsphere.vmware.com --type merge -p "{\"spec\":{\"managementState\":\"Unmanaged\"}}" $ oc --context sharedocp416-sbr -n openshift-cluster-csi-drivers get deploy/vmware-vsphere-csi-driver-controller -o json | jq -r '.spec.template.spec.containers[] | select(.name == "csi-attacher").args' [ "--csi-address=$(ADDRESS)", "--timeout=300s", "--http-endpoint=localhost:8203", "--leader-election", "--leader-election-lease-duration=137s", "--leader-election-renew-deadline=107s", "--leader-election-retry-period=26s", "--v=2" "--reconcile-sync=10m" <<----------------- ADD THE INCREASED RSYNC INTERVAL ]
Security Tracking Issue
Do not make this issue public.
Flaw:
Non-linear parsing of case-insensitive content in golang.org/x/net/html
https://bugzilla.redhat.com/show_bug.cgi?id=2333122
An attacker can craft an input to the Parse functions that would be processed non-linearly with respect to its length, resulting in extremely slow parsing. This could cause a denial of service.
~~~
Description of problem:
Ratcheting validation was implemented and made beta in 1.30. Validation ratcheting works for changes to the main resource, but does not work when applying updates to a status subresource. Details in https://github.com/kubernetes/kubernetes/issues/129503
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Install 4.17 2. Set powervs serviceEndpoints in the platformStatus to a valid lowercase string 3. Upgrade to 4.18 - validation has changed 4. Attempt to update an adjacent status field
Actual results:
Validation fails and rejects the update
Expected results:
Ratcheting should kick in and accept the object
Additional info:
Description of problem:
Cluster-reader couldn't able to view controlplancemachineset resources
Version-Release number of selected component (if applicable):
4.19.0-0.ci-2024-12-15-181719
How reproducible:
Always
Steps to Reproduce:
1. Add cluster-reader role to a common user $ oc adm policy add-cluster-role-to-user cluster-reader testuser-48 --as system:admin 3. Login in the cluster with the common user $ oc login -u testuser-48 Authentication required for https://api.zhsungcp58.qe.gcp.devcluster.openshift.com:6443 (openshift) Username: testuser-48 Password: Login successful. 4. Check cluster-reader could view controlplancemachineset resources.
Actual results:
cluster-reader couldn't view controlplancemachineset resources $ oc get controlplanemachineset Error from server (Forbidden): controlplanemachinesets.machine.openshift.io is forbidden: User "testuser-48" cannot list resource "controlplanemachinesets" in API group "machine.openshift.io" in the namespace "openshift-machine-api"
Expected results:
cluster-reader could view controlplanemachinesets resources
Additional info:
Description of problem:
A similar testing scenario to OCPBUGS-38719, but with the pre-existing dns private zone is not a peering zone, instead it is a normal dns zone which binds to another VPC network. And the installation will fail finally, because the dns record-set "*.apps.<cluster name>.<base domain>" is added to the above dns private zone, rather than the cluster's dns private zone.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-multi-2024-10-24-093933
How reproducible:
Always
Steps to Reproduce:
Please refer to the steps told in https://issues.redhat.com/browse/OCPBUGS-38719?focusedId=25944076&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-25944076
Actual results:
The installation failed, due to the cluster operator "ingress" degraded
Expected results:
The installation should succeed.
Additional info:
The following tests are failing with the updated 1.32 Kubernetes in OCP 4.19:
[sig-storage] CSI Mock volume expansion Expansion with recovery [Feature:RecoverVolumeExpansionFailure] recovery should be possible for node-only expanded volumes with final error [sig-storage] CSI Mock volume expansion Expansion with recovery [Feature:RecoverVolumeExpansionFailure] should record target size in allocated resources [sig-storage] CSI Mock volume expansion Expansion with recovery [Feature:RecoverVolumeExpansionFailure] should allow recovery if controller expansion fails with infeasible error [sig-storage] CSI Mock volume expansion Expansion with recovery [Feature:RecoverVolumeExpansionFailure] recovery should not be possible in partially expanded volumes [sig-storage] CSI Mock volume expansion Expansion with recovery [Feature:RecoverVolumeExpansionFailure] recovery should be possible for node-only expanded volumes with infeasible error
They will be disabled temporarily to not block the rebase progress. This bug ticket is used to track the work to enable this test in OCP 4.19 again.
Description of problem:
When Applying profile with isolated field containing huge cpu list, profile doesn't apply and no errors is reported
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-26-075648
How reproducible:
Everytime.
Steps to Reproduce:
1. Create a profile as specified below: apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: annotations: kubeletconfig.experimental: '{"topologyManagerPolicy":"restricted"}' creationTimestamp: "2024-11-27T10:25:13Z" finalizers: - foreground-deletion generation: 61 name: performance resourceVersion: "3001998" uid: 8534b3bf-7bf7-48e1-8413-6e728e89e745 spec: cpu: isolated: 25,281,117,373,127,383,3,259,30,286,77,333,23,279,21,277,66,322,12,268,15,271,61,317,120,376,35,291,62,318,93,349,126,382,19,275,52,308,110,366,50,306,92,348,124,380,119,375,2,258,29,285,60,316,115,371,118,374,104,360,108,364,70,326,72,328,76,332,96,352,99,355,64,320,80,336,97,353,8,264,11,267,38,294,53,309,57,313,103,359,14,270,87,343,7,263,40,296,51,307,94,350,116,372,39,295,46,302,90,346,101,357,107,363,26,282,67,323,98,354,106,362,113,369,6,262,10,266,20,276,33,289,112,368,85,341,121,377,68,324,71,327,79,335,81,337,83,339,88,344,9,265,89,345,91,347,100,356,54,310,31,287,58,314,59,315,22,278,47,303,105,361,17,273,114,370,111,367,28,284,49,305,55,311,84,340,27,283,95,351,5,261,36,292,41,297,43,299,45,301,75,331,102,358,109,365,37,293,56,312,63,319,65,321,74,330,125,381,13,269,42,298,44,300,78,334,122,378,4,260,16,272,34,290,123,379,18,274,48,304,69,325,82,338,24,280,32,288,73,329,86,342,220,476,251,507,206,462,226,482,229,485,221,477,253,509,255,511,135,391,144,400,183,439,233,489,137,393,186,442,198,454,190,446,234,490,147,403,163,419,172,428,148,404,149,405,250,506,151,407,217,473,238,494,131,387,230,486,235,491,246,502,145,401,194,450,199,455,143,399,169,425,231,487,245,501,129,385,142,398,179,435,225,481,236,492,152,408,203,459,214,470,166,422,207,463,212,468,130,386,155,411,215,471,188,444,201,457,210,466,193,449,200,456,248,504,141,397,167,423,191,447,181,437,222,478,252,508,128,384,139,395,174,430,164,420,168,424,187,443,232,488,133,389,157,413,208,464,140,396,185,441,241,497,219,475,175,431,184,440,213,469,154,410,197,453,249,505,209,465,218,474,227,483,244,500,134,390,153,409,178,434,160,416,195,451,196,452,211,467,132,388,136,392,146,402,138,394,150,406,239,495,173,429,192,448,202,458,205,461,216,472,158,414,159,415,176,432,189,445,237,493,242,498,177,433,182,438,204,460,240,496,254,510,162,418,171,427,180,436,243,499,156,412,165,421,170,426,228,484,247,503,161,417,223,479,224,480 reserved: 0,256,1,257 hugepages: defaultHugepagesSize: 1G pages: - count: 20 size: 2M machineConfigPoolSelector: machineconfiguration.openshift.io/role: worker-cnf net: userLevelNetworking: true nodeSelector: node-role.kubernetes.io/worker-cnf: "" numa: topologyPolicy: restricted realTimeKernel: enabled: false workloadHints: highPowerConsumption: true perPodPowerManagement: false realTime: true 2. The worker-cnf node doesn't contain any kernel args associated with the above profile. 3.
Actual results:
System doesn't boot with kernel args associated with above profile
Expected results:
System should boot with Kernel args presented from Performance Profile.
Additional info:
We can see MCO gets the details and creates the mc: Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: machine-config-daemon[9550]: "Running rpm-ostree [kargs --delete=systemd.unified_cgroup_hierarchy=1 --delete=cgroup_no_v1=\"all\" --delete=psi=0 --delete=skew_tick=1 --delete=tsc=reliable --delete=rcupda> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: cbs=25,281,117,373,127,383,3,259,30,286,77,333,23,279,21,277,66,322,12,268,15,271,61,317,120,376,35,291,62,318,93,349,126,382,19,275,52,308,110,366,50,306,92,348,124,380,119,375,2,258,29,285,60,316,115,3> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: 4,190,446,234,490,147,403,163,419,172,428,148,404,149,405,250,506,151,407,217,473,238,494,131,387,230,486,235,491,246,502,145,401,194,450,199,455,143,399,169,425,231,487,245,501,129,385,142,398,179,435,2> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: systemd.cpu_affinity=0,1,256,257 --append=iommu=pt --append=amd_pstate=guided --append=tsc=reliable --append=nmi_watchdog=0 --append=mce=off --append=processor.max_cstate=1 --append=idle=poll --append=is> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: ,78,334,122,378,4,260,16,272,34,290,123,379,18,274,48,304,69,325,82,338,24,280,32,288,73,329,86,342,220,476,251,507,206,462,226,482,229,485,221,477,253,509,255,511,135,391,144,400,183,439,233,489,137,393> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: 510,162,418,171,427,180,436,243,499,156,412,165,421,170,426,228,484,247,503,161,417,223,479,224,480 --append=nohz_full=25,281,117,373,127,383,3,259,30,286,77,333,23,279,21,277,66,322,12,268,15,271,61,317> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: ,476,251,507,206,462,226,482,229,485,221,477,253,509,255,511,135,391,144,400,183,439,233,489,137,393,186,442,198,454,190,446,234,490,147,403,163,419,172,428,148,404,149,405,250,506,151,407,217,473,238,49> Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com root[18779]: ppend=nosoftlockup --append=skew_tick=1 --append=rcutree.kthread_prio=11 --append=default_hugepagesz=1G --append=hugepagesz=2M --append=hugepages=20]" Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com rpm-ostree[18750]: client(id:machine-config-operator dbus:1.336 unit:crio-36c845a9c9a58a79a0e09dab668f8b21b5e46e5734a527c269c6a5067faa423b.scope uid:0) added; new total=1 Dec 02 08:59:43 cnfdd11.t5g-dev.eng.rdu2.dc.redhat.com rpm-ostree[18750]: Loaded sysroot Actual Kernel args: BOOT_IMAGE=(hd1,gpt3)/boot/ostree/rhcos-854dd632224b34d5f4df1884c4ba8c2f9527422b37744b83e7b1b98172586ff4/vmlinuz-5.14.0-427.44.1.el9_4.x86_64 rw ostree=/ostree/boot.0/rhcos/854dd632224b34d5f4df1884c4ba8c2f9527422b37744b83e7b1b98172586ff4/0 ignition.platform.id=metal ip=dhcp root=UUID=0068e804-432c-409d-aabc-260aa71e3669 rw rootflags=prjquota boot=UUID=7797d927-876e-426b-9a30-d1e600c1a382 systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all psi=0 skew_tick=1 tsc=reliable rcupdate.rcu_normal_after_boot=1 nohz=on
Description of problem:
The deprovision CI step[1] e2e-aws-ovn-shared-vpc-edge-zones-ipi-deprovision-deprovision is missing the permission ec2:ReleaseAddress in the installer user to remove the custom IPv4 Address (EIP) allocated in the cluster creation. The BYO IPv4 is default on CI jobs, and enabled when the pool has IP address. Error: level=warning msg=UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::460538899914:user/ci-op-rxxt8srv-bf840-minimal-perm-installer is not authorized to perform: ec2:ReleaseAddress on resource: arn:aws:ec2:us-east-1:[redacted]:elastic-ip/eipalloc-0f4b652b702e73204 because no identity-based policy allows the ec2:ReleaseAddress action. Job: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_installer/9413/pull-ci-openshift-installer-main-e2e-aws-ovn-shared-vpc-edge-zones/1884340831955980288
Version-Release number of selected component (if applicable):
4.19
How reproducible:
always when BYO Public IPv4 pool is activated in the install-config
Steps to Reproduce:
1. install a cluster with byo IPv4 pool set on install-config 2. 3.
Actual results:
level=warning msg=UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::460538899914:user/ci-op-rxxt8srv-bf840-minimal-perm-installer is not authorized to perform: ec2:ReleaseAddress on resource: arn:aws:ec2:us-east-1:[Redacted]:elastic-ip/eipalloc-0f4b652b702e73204 because no identity-based policy allows the ec2:ReleaseAddress action.
Expected results:
Permissions granted, EIP released.
Additional info:
Description of problem:
[Azure-Disk-CSI-Driver] allocatable volumes count incorrect in csinode for D family v6 instance types
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2025-02-20-131651
How reproducible:
Always
Steps to Reproduce:
1. Install openshift cluster with instance type "Standard_D8lds_v6". 2. Check the csinode object max allocatable volumes count should be consistent with cloud supports.
Actual results:
In step2 the csi node max allocatable volumes count is 16(correct value is 24) which is not consistent with cloud. $ oc get no/pewang-dxv6-a-qf9ck-worker-eastus2-l49ft -oyaml|grep 'instance' beta.kubernetes.io/instance-type: Standard_D8lds_v6 node.kubernetes.io/instance-type: Standard_D8lds_v6 $ oc get csinode pewang-dxv6-a-qf9ck-worker-eastus2-l49ft -oyaml|yq .spec drivers: - name: file.csi.azure.com nodeID: pewang-dxv6-a-qf9ck-worker-eastus2-l49ft topologyKeys: null - allocatable: count: 16 name: disk.csi.azure.com nodeID: pewang-dxv6-a-qf9ck-worker-eastus2-l49ft topologyKeys: - topology.disk.csi.azure.com/zone - topology.kubernetes.io/zone $ az vm list-sizes -l eastus -o tsv | awk -F '\t' '{print "\""$3"\":"$1","}' | sort | uniq|grep 'Standard_D8lds_v6' "Standard_D8lds_v6":24,
Expected results:
In step2 the csi node max allocatable volumes count is 24
Additional info:
Dlsv6, Dldsv6, Dsv6, Ddsv6 are not set in -> https://github.com/openshift/azure-disk-csi-driver/blob/master/pkg/azuredisk/azure_dd_max_disk_count.go
Note: also notify the Hive team we're doing these bumps.
We have a number of a11y rules that are disabled. Once axe is upgraded via https://github.com/openshift/console/pull/14311, we should check to see if these rules can be enabled.
Description of problem:
Sippy complains about pathological events in ns/openshift-cluster-csi-drivers in vsphere-ovn-serial jobs. See this job as one example.
Jan noticed that the DaemonSet generation is 10-12, while in 4.17 is 2. Why is our operator updating the DaemonSet so often?
I wrote a quick "one-liner" to generate json diffs from the vmware-vsphere-csi-driver-operator logs:
prev=''; grep 'DaemonSet "openshift-cluster-csi-drivers/vmware-vsphere-csi-driver-node" changes' openshift-cluster-csi-drivers_vmware-vsphere-csi-driver-operator-5b79c58f6f-hpr6g_vmware-vsphere-csi-driver-operator.log | sed 's/^.*changes: //' | while read -r line; do diff <(echo $prev | jq .) <(echo $line | jq .); prev=$line; echo "####"; done
It really seems to be only operator.openshift.io/spec-hash and operator.openshift.io/dep-* fields changing in the json diffs:
#### 4,5c4,5 < "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "MZ-w-Q==", < "operator.openshift.io/spec-hash": "fb274874404ad6706171c6774a369876ca54e037fcccc200c0ebf3019a600c36" --- > "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "AFeN-A==", > "operator.openshift.io/spec-hash": "27a1bab0c00ace8ac21d95a5fe9a089282e7b2b3ec042045951bd5e26ae01a09" 13c13 < "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "MZ-w-Q==" --- > "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "AFeN-A==" #### 4,5c4,5 < "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "AFeN-A==", < "operator.openshift.io/spec-hash": "27a1bab0c00ace8ac21d95a5fe9a089282e7b2b3ec042045951bd5e26ae01a09" --- > "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "MZ-w-Q==", > "operator.openshift.io/spec-hash": "fb274874404ad6706171c6774a369876ca54e037fcccc200c0ebf3019a600c36" 13c13 < "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "AFeN-A==" --- > "operator.openshift.io/dep-1b5c921175cca7ab09ea7d1d58e35428291b8": "MZ-w-Q==" ####
The deployment is also changing in the same way. We need to find what is causing the spec-hash and dep-* fields to change and avoid the unnecessary churn that causes new daemonset / deployment rollouts.
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
~20% failure rate in 4.18 vsphere-ovn-serial jobs
Steps to Reproduce:
Actual results:
operator rolls out unnecessary daemonset / deployment changes
Expected results:
don't roll out changes unless there is a spec change
Additional info:
Description of problem:
HyperShift CEL validation blocks ARM64 NodePool creation for non-AWS/Azure platforms Can't add a Bare Metal worker node to the hosted cluster. This was discussed on #project-hypershift Slack channel.
Version-Release number of selected component (if applicable):
MultiClusterEngine v2.7.2 HyperShift Operator image: registry.redhat.io/multicluster-engine/hypershift-rhel9-operator@sha256:56bd0210fa2a6b9494697dc7e2322952cd3d1500abc9f1f0bbf49964005a7c3a
How reproducible:
Always
Steps to Reproduce:
1. Create a HyperShift HostedCluster on a non-AWS/non-Azure platform 2. Try to create a NodePool with ARM64 architecture specification
Actual results:
- CEL validation blocks creating NodePool with arch: arm64 on non-AWS/Azure platforms - Receive error: "The NodePool is invalid: spec: Invalid value: "object": Setting Arch to arm64 is only supported for AWS and Azure" - Additional validation in NodePool spec also blocks arm64 architecture
Expected results:
- Allow ARM64 architecture specification for NodePools on BareMetal platform - Remove or update the CEL validation to support this use case
Additional info:
NodePool YAML: apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata: name: nodepool-doca5-1 namespace: doca5 spec: arch: arm64 clusterName: doca5 management: autoRepair: false replace: rollingUpdate: maxSurge: 1 maxUnavailable: 0 strategy: RollingUpdate upgradeType: InPlace platform: agent: agentLabelSelector: {} type: Agent release: image: quay.io/openshift-release-dev/ocp-release:4.16.21-multi replicas: 1
Description of problem:
If a customer populates the serviceEndpoints for powervs via the install config in 4.17, the validation is incorrect and persists lowercase values. status: platformStatus: type: PowerVS powervs: serviceEndpoints: - name: dnsservices url: ... On upgrade, the schema is currently updated to an enum, courtesy of https://github.com/openshift/api/pull/2076 The validation upgrade and ratcheting was tested, but only for the `spec` version of the field. It was assumed that spec and status validation behaved the same. However, https://issues.redhat.com/browse/OCPBUGS-48077, has recently been found, and this means that on upgrade, all writes to the status subresource of the infrastructure object fail, until the serviceEndpoints are fixed. In a steady state, this may not cause general cluster degredation, writing to the status of the infrastructure object is not common. However, any controller that does attempt to write to it, will fail, and end up erroring until the value has been fixed. There are several possible approaches to resolve this: 1. Revert https://github.com/openshift/api/pull/2076 and anything else that depended on it 2. Merge and backport the fix for https://issues.redhat.com/browse/OCPBUGS-48077 3. Introduce something in 4.18 to fix invalid values in the status (eg convert dnsservices to DNSServices) Until one of these three (or perhaps other fixes) is taken, I think this needs to be considered a PowerVS upgrade blocker, and then management can decide if this is enough to block 4.18
Version-Release number of selected component (if applicable):
4.17 to 4.18
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The CEL for AWSNetworkLoadBalancerParameters that ensures Subnets and EIPs are equal, should be "feature gated" by both SetEIPForNLBIngressController and IngressControllerLBSubnetsAWS. Meaning, the CEL should only be present/executed if both feature gates are enabled.
At the time we released this feature, there wasn't a way to do "AND" for the FeatureGateAwareXValidation marker, but recently https://github.com/openshift/kubernetes-sigs-controller-tools/pull/21 has been merged which now supports that.
However, nothing is currently broken since both feature gates are now enabled by default, but if the IngressControllerLBSubnetsAWS feature gate was disabled for any reason, the IngressController CRD would become invalid and unable to install. You'd get an error message similar to:
ERROR: <input>:1:157: undefined field 'subnets'
Version-Release number of selected component (if applicable):
4.17 and 4.18
How reproducible:
100%?
Steps to Reproduce:
1. Disable IngressControllerLBSubnetsAWS feature gate
Actual results:
IngressController CRD is now broken
Expected results:
IngressController shouldn't be broken.
Additional info:
To be clear, this is not a bug with an active impact, but this is more of an inconsistency that could cause problems in the future.
Description of problem:
There is no clipValue function for the annotation router.openshift.io/haproxy.health.check.interval. Once any value with abnormal values, the router-default starts to report the following messages: [ALERT] (50) : config : [/var/lib/haproxy/conf/haproxy.config:13791] : 'server be_secure:xxx:httpd-gateway-route/pod:xxx:xxx-gateway-service:pass-through-https:10.129.xx.xx:8243' : timer overflow in argument <50000d> to <inter> of server pod:xxx:xxx:pass-through-https:10.129.xx.xx:8243, maximum value is 2147483647 ms (~24.8 days).. In the above case, the value 50000d was passed to the route annotation router.openshift.io/haproxy.health.check.interval accidentally
Version-Release number of selected component (if applicable):
How reproducible:
Easily
Steps to Reproduce:
1. Run the following script and this will break the cluster oc get routes -A | awk '{print $1 " " $2}' | tail -n+2 | while read line; do read -r namespace routename <<<$(echo $line) echo -n "NS: $namespace | " echo "ROUTENAME: $routename" CMD="oc annotate route -n $namespace $routename --overwrite router.openshift.io/haproxy.health.check.interval=50000d" echo "Annotating route with:" echo $CMD ; eval "$CMD" echo "---" done
Actual results:
The alert messages are reported and the router-default pod never reaches the ready state.
Expected results:
Clip the value in order to prevent the issue
Additional info:
Related to OCPBUGS-33891, sometimes degraded nodes are accompanied by a message like this:
failed to set annotations on node: unable to update node "&Node{ObjectMeta:{ 0 0001-01-01 00:00:00 +00 00 UTC <nil> <nil> map[] map[] [] [] []},Spec:NodeSpec{PodCIDR:,DoNotUseExternalID:,ProviderID:,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[],},Status:NodeStatus{Capacit y:ResourceList{},Allocatable:ResourceList{},Phase:,Conditions:[]NodeCondition{},Addresses:[]NodeAddress{},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:0,},},NodeI nfo:NodeSystemInfo{MachineID:,SystemUUID:,BootID:,KernelVersion:,OSImage:,ContainerRuntimeVersion:,KubeletVersion:,KubeProxyVersion:,OperatingSystem:,Architecture:,},Images:[]ContainerImage {},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}": Patch "https://api-int.evakhoni-1215.qe.devcluster.openshift.com:6443/api/v1/nodes/<node>": read tcp 10.0.26.198:41196 ->10.0.15.142:6443: read: connection reset by peer
The reason annotation is potentially user-facing (or am I wrong?) so dumping the full &Node... structure in there is probably not useful, I would expect such info only in the log or in clearly non-user-facing location.
4.16+
Like OCPBUGS-33891
Like OCPBUGS-33891
Go struct dump in the message
No Go struct dump in the message
flag: --metrics-bind-addr was deprecated and is now removed, we need to update how we deploy CAPO.
Description of problem:
For various reasons, Pods may get evicted. Once they are evicted, the owner of the Pod should recreate the Pod so it is scheduled again.
With OLM, we can see that evicted Pods owned by Catalogsources are not rescheduled. The outcome is that all subscriptions have a "ResolutionFailed=True" condition, which hinders an upgrade of the operator. Specifically the customer is seeing an affected CatalogSource is "multicluster-engine-CENSORED_NAME-redhat-operator-index "in openshift-marketplace namespace, pod name: "multicluster-engine-CENSORED_NAME-redhat-operator-index-5ng9j"
Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.16.21
How reproducible:
Sometimes, when Pods are evicted on the cluster
Steps to Reproduce:
1. Set up an OpenShift Container Platform 4.16 cluster, install various Operators
2. Create a condition that a Node will evict Pods (for example by creating DiskPressure on the Node)
3. Observe if any Pods owned by CatalogSources are being evicted
Actual results:
If Pods owned by CatalogSources are being evicted, they are not recreated / rescheduled.
Expected results:
When Pods owned by CatalogSources are being evicted, they are being recreacted / rescheduled.
Additional info:
Description of problem:
=== RUN TestNodePool/HostedCluster2/EnsureHostedCluster/EnsureSATokenNotMountedUnlessNecessary util.go:1943: Expected <string>: kube-api-access-5jlcn not to have prefix <string>: kube-api-access-
Pod spec:
name: manila-csi-driver-operator resources: requests: cpu: 10m memory: 50Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true runAsUser: 1000690000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/guest-kubeconfig name: guest-kubeconfig - mountPath: /etc/openstack-ca/ name: cacert - mountPath: /etc/openstack/ name: cloud-credentials - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-5jlcn readOnly: true
Description of problem:
For upgrades and migrations we introduced a check to avoid them when overlapping CIDRs exist - https://github.com/openshift/ovn-kubernetes/pull/2313. However the installation allows it which ends with the installation broken and no information from anywhere to easily lead to the issue.For example if I create a install-config with clusternetwork CIDR overlapping with OVNK-I transit switch, everything goes on and at certain point the install will just get stuck and only by accessing one of the masters and seeing the ovnkube-controller logs we can see what is the issue. And this is because I know what I was doing wrong since I configure this on purpose. Our installation documentation has no mention of this CIDR (only the join subnet) and we can't expect customers to read all other documents prior to install the clusters as we can see in the example below:https://docs.openshift.com/container-platform/4.15/installing/installing_bare_metal/installing-bare-metal-network-customizations.html
Version-Release number of selected component (if applicable):
OCP 4.14 and 4.15
How reproducible:
every time
Steps to Reproduce:
1. Install a cluster with clusternetwork CIDR that overlaps with the OVN internal transit switch subnet 2. 3.
Actual results:
Expected results:
Get an error warning about overlapping CIDRs and asking user to change.
Additional info:
I had extracted this code, but I broke the case where we're getting the pull secret from the cluster – because it was immediately getting deleted on return. In-line the code instead to prevent that, so it gets deleted when we're done using it.
The error you get when this happens looks like:error running options:
could not create external binary provider: couldn't extract release payload image stream: failed extracting image-references from "quay.io/openshift-release-dev/ocp-release-nightly@sha256:856d044a4f97813fb31bc4edda39b05b2b7c02de1327b9b297bdf93edc08fa95": error during image extract: exit status 1 (error: unable to load --registry-config: stat /tmp/external-binary2166428580/.dockerconfigjson: no such file or directory
If ESXi host is powered off in a cluster that an OVA is being imported to the ova import will fail.
Description of problem:
On the page /command-line-tools, the oc and virtctl sort the links differently
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
1.Login in and browse page /command-line-tools 2.Check the download links in oc and virtctl 3.
Actual results:
oc and virtctl sort the links differently oc sort them via x86_64, ARM 64, IBM blabla, while virtctl sort them via Linux, Mac, Windows Both sort are fine. But putting them together, it's a little wired.
Expected results:
All download links on this page sort in the same way.
Additional info:
Description of problem:
Incorrect capitalization for `Lightspeed` to capitalized `LightSpeed` in ja and zh langs
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When attempting to install a specific version of an operator from the web console, the install plan of the latest version of that operator is created if the operator version had a + in it.
Version-Release number of selected component (if applicable):
4.17.6 (Tested version)
How reproducible:
Easily reproducible
Steps to Reproduce:
1. Under Operators > Operator Hub, install an operator with a + character in the version. 2. On the next screen, note that the + in the version text box is missing. 3. Make no changes to the default options and proceed to install the operator. 4. An install plan is created to install the operator with the latest version from the channel.
Actual results:
The install plan is created for the latest version from the channel.
Expected results:
The install plan is created for the requested version.
Additional info:
Notes on the reproducer: - For step 1: the selected version shouldn't be the latest version from the channel for the purposes of this bug. - For step 1: The version will need to be selected from the version dropdown to reproduce the bug. If the default version that appears in the dropdown is used, then the bug won't reproduce. Other Notes: - This might also happen with other special characters in the version string other than +, but this is not something that I tested.
Description of problem:
When creating a kubevirt hosted cluster with the following apiserver publishing configuration - service: APIServer servicePublishingStrategy: type: NodePort nodePort: address: my.hostna.me port: 305030 Shows following error: "failed to reconcile virt launcher policy: could not determine if amy.hostna.me is an IPv4 or IPv6 address" And network policies and not propertly deployed at the virtual machine namespaces.
Version-Release number of selected component (if applicable):
4.17
How reproducible:
Always
Steps to Reproduce:
1.Create a kubevirt hosted cluster with apiserver nodeport publish with a hostname 2. Wait for hosted cluster creation.
Actual results:
Following error pops up and network policies are not created "failed to reconcile virt launcher policy: could not determine if amy.hostna.me is an IPv4 or IPv6 address"
Expected results:
No error pops ups and network policies are created.
Additional info:
This is where the error get originated -> https://github.com/openshift/hypershift/blob/ef8596d4d69a53eb60838ae45ffce2bca0bfa3b2/hypershift-operator/controllers/hostedcluster/network_policies.go#L644 That error should prevent network policies creation.
There's a possible flake in openshift-tests because the upstream test "Should recreate evicted statefulset" occasionally causes kubelet to emit a "failed to bind hostport" event because it tries to recreate a deleted pod too quickly, and this gets flagged by openshift-tests as a bad thing, even though it's not (because it retries and succeeds).
I filed a PR to fix this a long time ago, it just needs review.
Description of problem:
[control-plane-operator] azure-file-csi using nfs protocal provision volume failed of "vnetName or location is empty"
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-01-21-070749
How reproducible:
Always
Steps to Reproduce:
1. Create aro hosted cluster on azure. 2. Create a new storageclass using azure file csi provisioner and nfs protocol and create pvc with the created storageclass, create pod comsume the pvc. $ oc apply -f - <<EOF allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: azurefile-csi-nfs parameters: protocol: nfs provisioner: file.csi.azure.com reclaimPolicy: Delete volumeBindingMode: Immediate --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: mypvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: azurefile-csi-nfs volumeMode: Filesystem --- apiVersion: v1 kind: Pod metadata: name: mypod spec: containers: - name: hello-app image: quay.io/openshifttest/hello-openshift@sha256:56c354e7885051b6bb4263f9faa58b2c292d44790599b7dde0e49e7c466cf339 volumeMounts: - mountPath: /mnt/storage name: data volumes: - name: data persistentVolumeClaim: claimName: mypvc EOF 3. Check the volume should be provisioned and pod could read and write inside the file volume.
Actual results:
In step 3: the volume provision failed of vnetName or location is empty $ oc describe pvc mypvc Name: mypvc Namespace: default StorageClass: azurefile-csi-nfs Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-provisioner: file.csi.azure.com volume.kubernetes.io/storage-provisioner: file.csi.azure.com Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: mypod Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 7s (x3 over 10s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. Normal Provisioning 3s (x4 over 10s) file.csi.azure.com_azure-file-csi-driver-controller-7cb9b5f788-n9ztr_85399802-95c4-468e-814d-2c4df5140069 External provisioner is provisioning volume for claim "default/mypvc" Warning ProvisioningFailed 3s (x4 over 10s) file.csi.azure.com_azure-file-csi-driver-controller-7cb9b5f788-n9ztr_85399802-95c4-468e-814d-2c4df5140069 failed to provision volume with StorageClass "azurefile-csi-nfs": rpc error: code = Internal desc = update service endpoints failed with error: vnetName or location is empty
Expected results:
In step 3: the volume should be provisioned and pod could read and write inside the file volume.
Additional info:
# aro hcp missed the vnetName/vnetResourceGroup will caused using nfs protocol provision volumes failed oc extract secret/azure-file-csi-config --to=- # cloud.conf { "cloud": "AzurePublicCloud", "tenantId": "XXXXXXXXXX", "useManagedIdentityExtension": false, "subscriptionId": "XXXXXXXXXX", "aadClientId": "XXXXXXXXXX", "aadClientSecret": "", "aadClientCertPath": "/mnt/certs/ci-op-gcprj1wl-0a358-azure-file", "resourceGroup": "ci-op-gcprj1wl-0a358-rg", "location": "centralus", "vnetName": "", "vnetResourceGroup": "", "subnetName": "", "securityGroupName": "", "securityGroupResourceGroup": "", "routeTableName": "", "cloudProviderBackoff": false, "cloudProviderBackoffDuration": 0, "useInstanceMetadata": false, "loadBalancerSku": "", "disableOutboundSNAT": false, "loadBalancerName": "" }
Description of problem:
Dialog creating the primary namespaced UDN does not need "name" field. Users can only use one primary UDN per namespace. We can make the flow smoother by generating (or hardcoding) the name on the UI. This should be static (not random). A side effect of this would be, that it would prevent users from creating multiple primary UDNs by mistake.
Version-Release number of selected component (if applicable):
rc.4
How reproducible:
Always
Steps to Reproduce:
1. Go to the create UDN dialog 2. 3.
Actual results:
It asks for a name
Expected results:
It should not ask for a name, using "primary-udn" as the hardcoded value OR It should still give the option to set it, but use "primary-udn" as the default pre-filled in the textbox
Additional info:
Description of problem:
Table layout missing on Metrics page. After PR https://github.com/openshift/console/pull/14615 change, based on the changes the PatternFly 4 shared modules has been removed
Version-Release number of selected component (if applicable):
pre-merge
How reproducible:
Always
Steps to Reproduce:
1. Navigate to Observe -> Metrics page 2. Click 'Insert example query' button 3. Check the layout for Query results table
Actual results:
The results table layout issue
Expected results:
Layout should as same as OCP 4.18
Additional info:
More infomation: could be check the PR https://github.com/openshift/console/pull/14615
Tracking https://github.com/distribution/distribution/issues/4112 and/or our own fixes.
user specified tls skip verify which triggers a bug that do not respect proxy values.
short term fix: if self signed cert is used, specify cacert accordingly instead of skipping verification.
Description of problem:
when I run the e2e locally and specify the image to be registry.ci.openshift.org/ocp/release:4.19.0-0.nightly-2025-01-30-091858, The e2e never validates the cluster due to eventually.go:226: - wanted HostedCluster to desire image registry.ci.openshift.org/ocp/release:4.19.0-0.nightly-2025-01-30-091858, got registry.ci.openshift.org/ocp/release@sha256:daccaa3c0223e23bfc6d9890d7f8e52faa8a5071b2e80d5f753900f16584e3f0even though the hc.status is completed and the desired image matches the history desired: image: registry.ci.openshift.org/ocp/release@sha256:daccaa3c0223e23bfc6d9890d7f8e52faa8a5071b2e80d5f753900f16584e3f0 version: 4.19.0-0.nightly-2025-01-30-091858 history: - completionTime: "2025-01-31T12:45:37Z" image: registry.ci.openshift.org/ocp/release@sha256:daccaa3c0223e23bfc6d9890d7f8e52faa8a5071b2e80d5f753900f16584e3f0 startedTime: "2025-01-31T12:35:07Z" state: Completed verified: false version: 4.19.0-0.nightly-2025-01-30-091858 observedGeneration: 1
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
During the collection, the KubeVirt image and the OSUS (graph image) are being skipped in case of failure to find them.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
Check the local_store_collector.go on the release package
Actual results:
Images are skipped
Expected results:
If the image was requested in the image set config, since these are release images, they should not be skipped, the workflow should fail
Additional info:
backporting https://github.com/prometheus/prometheus/pull/16174
release note entry is to be added.
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Day2 monitoring is not handling api server temporarily disconnection
Version-Release number of selected component (if applicable):
4.17.0-0.ci-2024-08-26-170911
How reproducible:
always in libvirt manually run
Steps to Reproduce:
1. Run agent install in libvirt env manually 2. Run day2 install after cluster is installed succeed 3. Run 'oc adm node-image monitor' to track the day2 install, when there is api server temporarily disconnection , monitoring program will run into error/EOF. 4, Only reproduced in libvirt env, baremetal platform is working fine.
Actual results:
Day2 monitoring should run without break to track day2 install in libvirt
Expected results:
Day2 monitoring run into error/EOF
Additional info:
Monitoring output link: https://docs.google.com/spreadsheets/d/17cOCfYvqxLHlhzBHkwCnFZDUatDRcG1Ej-HQDTDin0c/edit?gid=0#gid=0
Description of problem:
when more than one release is added to ImageSetConfig.yaml the images number is double and incorrect, check the log we could see duplications. ImageSetConfig.yaml: ================= [fedora@preserve-fedora-yinzhou test]$ cat /tmp/clid-232.yaml apiVersion: mirror.openshift.io/v2alpha1 kind: ImageSetConfiguration mirror: platform: channels: - name: stable-4.16 minVersion: 4.16.0 maxVersion: 4.16.0 - name: stable-4.15 minVersion: 4.15.0 maxVersion: 4.15.0 images to copy 958 cat /tmp/sss |grep 1fd628f40d321354832b0f409d2bf9b89910de27bc6263a4fb5a55c25e160a99 ✓ 178/958 : (8s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1fd628f40d321354832b0f409d2bf9b89910de27bc6263a4fb5a55c25e160a99 ✓ 945/958 : (8s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1fd628f40d321354832b0f409d2bf9b89910de27bc6263a4fb5a55c25e160a99 cat /tmp/sss |grep x86_64 ✓ 191/958 : (3s) quay.io/openshift-release-dev/ocp-release:4.16.0-x86_64 ✓ 383/958 : (2s) quay.io/openshift-release-dev/ocp-release:4.15.0-x86_64 ✓ 575/958 : (1s) quay.io/openshift-release-dev/ocp-release:4.15.0-x86_64 ✓ 767/958 : (11s) quay.io/openshift-release-dev/ocp-release:4.15.35-x86_64 ✓ 958/958 : (5s) quay.io/openshift-release-dev/ocp-release:4.16.0-x86_64
Version-Release number of selected component (if applicable):
./oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-unknown-68e608e2", GitCommit:"68e608e2", GitTreeState:"clean", BuildDate:"2024-10-14T05:57:17Z", GoVersion:"go1.23.0", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. clone oc-mirror repo, cd oc-mirror, run make build 2. Now use the imageSetConfig.yaml present above and run mirror2disk & disk2mirror commands 3. oc-mirror -c /tmp/clid-232.yaml file://CLID-232 --v2 ; oc-mirror -c /tmp/clid-232.yaml --from file://CLID-232 docker://localhost:5000/clid-232 --dest-tls-verify=false --v2
Actual results:
1. see mirror duplication
Expected results:
no dup.
Additional info:
Description of problem:
While setting userTags in the install-config file for AWS does not support all AWS valid characters as per [1]. platform: aws: region: us-east-1 propagateUserTags: true userTags: key1: "Test Space" key2: value2 ERROR failed to fetch Metadata: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: platform.aws.userTags[key1]: Invalid value: "Test Space": value contains invalid characters The documentation at: https://docs.openshift.com/container-platform/4.16/installing/installing_aws/installation-config-parameters-aws.html#installation-configuration-parameters-optional-aws_installation-config-parameters-aws does not refer to any restrictions. However: Validation is done here: https://github.com/openshift/installer/blob/74ee94f2a34555a41107a5a7da627ab5de0c7373/pkg/types/aws/validation/platform.go#L106 Which in turn refers to a regex here: https://github.com/openshift/installer/blob/74ee94f2a34555a41107a5a7da627ab5de0c7373/pkg/types/aws/validation/platform.go#L17 Which allows these characters: `^[0-9A-Za-z_.:/=+-@]*$` [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions).
Version-Release number of selected component (if applicable):
How reproducible:
100 %
Steps to Reproduce:
1. Create a install-config with a value usertags as mention in description. 2. Run the installer.
Actual results:
Command failed with below error: ERROR failed to fetch Metadata: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: platform.aws.userTags[key1]: Invalid value: "Test Space": value contains invalid characters
Expected results:
Installer should run successfully.
Additional info:
In userTags when the value with space is set then the installer failed to compile the install-config.
Description of problem:
When there are no images for copying or size of images to copy is zero oc-mirror enters into infinite loop as shown below {code:java} [fedora@knarra-fedora knarra]$ ./oc-mirror -c /tmp/iscoci.yaml file://test --v2 --cache-dir /test/yinzhou 2025/01/31 17:25:34 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/31 17:25:34 [INFO] : :gear: setting up the environment for you... 2025/01/31 17:25:34 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/31 17:25:34 [INFO] : 🕵 going to discover the necessary images... 2025/01/31 17:25:34 [INFO] : :mag: collecting release images... 2025/01/31 17:25:34 [INFO] : :mag: collecting operator images... ✗ () Collecting catalog oci:///home/fedora/knarra/openshift-tests-private/redhat-operator-index 2025/01/31 17:25:34 [WARN] : [OperatorImageCollector] catalog invalid source name oci:///home/fedora/knarra/openshift-tests-private/redhat-operator-index: lstat /home/fedora/knarra/openshift-tests-private: no such file or directory : SKIPPING 2025/01/31 17:25:34 [INFO] : :mag: collecting additional images... 2025/01/31 17:25:34 [INFO] : :mag: collecting helm images... 2025/01/31 17:25:34 [INFO] : :repeat_one: rebuilding catalogs 2025/01/31 17:25:34 [INFO] : :rocket: Start copying the images... 207712134 / 0 (7m56s) [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0 %
Version-Release number of selected component (if applicable):{code:none}
4.18
How reproducible:
Always
Steps to Reproduce:
1. Run the command to copy redhat-operator-index as an oci file by using command skopeo copy --all docker://registry.redhat.io/redhat/redhat-operator-index:v4.16 --remove-signatures --insecure-policy oci:///home/fedora/knarra/redhat-operator-index 2. Now create imageSetConfig.yaml as shown below {code:java} kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: oci:///home/fedora/knarra/openshift-tests-private/redhat-operator-index targetCatalog: "ocicatalog73452" targetTag: "v16" packages: - name: cluster-kube-descheduler-operator
3. Run command oc-mirror -c /tmp/iscoci.yaml file://test --v2 --cache-dir /test/yinzhou
Actual results:{code:none} You can see that mirroring enters into infinite loop
Expected results:
Mirroring should fail since there is no such directory and come out.
Additional info:
https://redhat-internal.slack.com/archives/C050P27C71S/p1738344713804659
Description of problem:
Pipeline repository overview page close button have always loading
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Navigate to create Repository form 2. Fill all details and submit the form
Actual results:
Close button on Reposiotry overview page have a loading icon
Expected results:
Loading icon should not be there.
Additional info:
Description of problem:
oc-mirror failed with error: the manifest type *ocischema.DeserializedImageIndex is not supported
Version-Release number of selected component (if applicable):
./oc-mirror.rhel8 version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202410251041.p0.g95f0611.assembly.stream.el9-95f0611", GitCommit:"95f0611c1dc9584a4a9e857912b9eaa539234bbc", GitTreeState:"clean", BuildDate:"2024-10-25T11:28:19Z", GoVersion:"go1.22.7 (Red Hat 1.22.7-1.module+el8.10.0+22325+dc584f75) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. imagesetconfig as follow : kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 mirror: operators: - catalog: registry.redhat.io/redhat/certified-operator-index:v4.16 packages: - name: nginx-ingress-operator 2. run the mirror2mirror command : ./oc-mirror.rhel8 -c config-66870.yaml docker://localhost:5000/18 --dest-skip-tls --dest-use-http
Actual results:
2. hit error : ./oc-mirror.rhel8 -c config-66870.yaml docker://localhost:5000/18 --dest-skip-tls --dest-use-http Checking push permissions for localhost:5000 Creating directory: oc-mirror-workspace/src/publish Creating directory: oc-mirror-workspace/src/v2 Creating directory: oc-mirror-workspace/src/charts Creating directory: oc-mirror-workspace/src/release-signatures backend is not configured in config-66870.yaml, using stateless mode backend is not configured in config-66870.yaml, using stateless mode No metadata detected, creating new workspace .... manifests: sha256:dea36b1dde70a17369d775cbabe292e7173abcef426dfc21b8a44896544a30da -> ae3ddd14 stats: shared=0 unique=27 size=139.1MiB ratio=1.00error: the manifest type *ocischema.DeserializedImageIndex is not supported error: an error occurred during planning
Expected results:
3. no error
Additional info:
compared with 4.17 oc-mirror, no such issue
The virtualization perspective wants to have the observe section so that they can be a fully independent perspective.
The prerequisite functionality is added to the monitoring-plugin without showing regressions in the admin and developer perspectives.
Description of problem:
OCP cluster upgrade is stuck with image registry pod in degraded state. The image registry co shows the below error message. - lastTransitionTime: "2024-09-13T03:15:05Z" message: "Progressing: All registry resources are removed\nNodeCADaemonProgressing: The daemon set node-ca is deployed\nAzurePathFixProgressing: Migration failed: I0912 18:18:02.117077 1 main.go:233] Azure Stack Hub environment variables not present in current environment, skipping setup...\nAzurePathFixProgressing: panic: Get \"https://xxxxximageregistry.blob.core.windows.net/xxxxcontainer?comp=list&prefix=docker&restype=container\": dial tcp: lookup xxxximageregistry.blob.core.windows.net on 192.168.xx.xx. no such host\nAzurePathFixProgressing: \nAzurePathFixProgressing: goroutine 1 [running]:\nAzurePathFixProgressing: main.main()\nAzurePathFixProgressing: \t/go/src/github.com/openshift/cluster-image-registry-operator/cmd/move-blobs/main.go:53 +0x12a\nAzurePathFixProgressing: " reason: AzurePathFixFailed::Removed status: "False" type: Progressing
Version-Release number of selected component (if applicable):
4.14.33
How reproducible:
Steps to Reproduce:
1. configure azure storage in configs.imageregistry.operator.openshift.io/cluster 2. then mark the managementState as Removed 3. check the operator status
Actual results:
CO image-registry remain is degraded state
Expected results:
Operator should not be in degraded state
Additional info:
Description of problem:
CSI Operator doesn't propagate HCP labels to 2nd level operands
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
100%
Steps to Reproduce:
1. Create hostedCluster with .spec.Labels
Actual results:
aws-ebs-csi-driver-controller, aws-ebs-csi-driver-operator, csi-snapshot-controller, csi-snapshot-webhook pods don't have the specified labels.
Expected results:
aws-ebs-csi-driver-controller, aws-ebs-csi-driver-operator, csi-snapshot-controller, csi-snapshot-webhook pods have the specified labels.
Additional info:
Description of problem:
[Azure disk csi driver]on ARO HCP could not provision volume succeed
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-13-083421
How reproducible:
Always
Steps to Reproduce:
1.Install AKS cluster on azure. 2.Install hypershift operator on the AKS cluster. 3.Use hypershift CLI create hosted cluster with the Client Certificate mode. 4.Check the azure disk/file csi dirver work well on hosted cluster.
Actual results:
In step 4: the the azure disk csi dirver provision volume failed on hosted cluster # azure disk pvc provision failed $ oc describe pvc mypvc ... Normal WaitForFirstConsumer 74m persistentvolume-controller waiting for first consumer to be created before binding Normal Provisioning 74m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_2334468f-9d27-4bdd-a53c-27271ee60073 External provisioner is provisioning volume for claim "default/mypvc" Warning ProvisioningFailed 74m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_2334468f-9d27-4bdd-a53c-27271ee60073 failed to provision volume with StorageClass "managed-csi": rpc error: code = Unavailable desc = error reading from server: EOF Warning ProvisioningFailed 71m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_28ba5ad9-c4f8-4dc8-be40-c80c546b7ef8 failed to provision volume with StorageClass "managed-csi": rpc error: code = Unavailable desc = error reading from server: EOF Normal Provisioning 71m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_28ba5ad9-c4f8-4dc8-be40-c80c546b7ef8 External provisioner is provisioning volume for claim "default/mypvc" ... $ oc logs azure-disk-csi-driver-controller-74d944bbcb-7zz89 -c csi-driver W1216 08:07:04.282922 1 main.go:89] nodeid is empty I1216 08:07:04.290689 1 main.go:165] set up prometheus server on 127.0.0.1:8201 I1216 08:07:04.291073 1 azuredisk.go:213] DRIVER INFORMATION: ------------------- Build Date: "2024-12-13T02:45:35Z" Compiler: gc Driver Name: disk.csi.azure.com Driver Version: v1.29.11 Git Commit: 4d21ae15d668d802ed5a35068b724f2e12f47d5c Go Version: go1.23.2 (Red Hat 1.23.2-1.el9) X:strictfipsruntime Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone I1216 08:09:36.814776 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I1216 08:09:36.814803 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316","csi.storage.k8s.io/pvc/name":"mypvc","csi.storage.k8s.io/pvc/namespace":"default","skuname":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}}]} I1216 08:09:36.815338 1 controllerserver.go:208] begin to create azure disk(pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316) account type(Premium_LRS) rg(ci-op-zj9zc4gd-12c20-rg) location(centralus) size(1) diskZone() maxShares(0) panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x190c61d] goroutine 153 [running]: sigs.k8s.io/cloud-provider-azure/pkg/provider.(*ManagedDiskController).CreateManagedDisk(0x0, {0x2265cf0, 0xc0001285a0}, 0xc0003f2640) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_managedDiskController.go:127 +0x39d sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*Driver).CreateVolume(0xc000564540, {0x2265cf0, 0xc0001285a0}, 0xc000272460) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/azuredisk/controllerserver.go:297 +0x2c59 github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x2265cf0?, 0xc0001285a0?}, {0x1e5a260?, 0xc000272460?}) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6420 +0xcb sigs.k8s.io/azuredisk-csi-driver/pkg/csi-common.logGRPC({0x2265cf0, 0xc0001285a0}, {0x1e5a260, 0xc000272460}, 0xc00017cb80, 0xc00014ea68) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/csi-common/utils.go:80 +0x409 github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x1f3e440, 0xc000564540}, {0x2265cf0, 0xc0001285a0}, 0xc00029a700, 0x2084458) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6422 +0x143 google.golang.org/grpc.(*Server).processUnaryRPC(0xc00059cc00, {0x2265cf0, 0xc000128510}, {0x2270d60, 0xc0004f5980}, 0xc000308480, 0xc000226a20, 0x31c8f80, 0x0) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1379 +0xdf8 google.golang.org/grpc.(*Server).handleStream(0xc00059cc00, {0x2270d60, 0xc0004f5980}, 0xc000308480) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1790 +0xe8b google.golang.org/grpc.(*Server).serveStreams.func2.1() /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1029 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 16 /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1040 +0x125
Expected results:
In step 4: the the azure disk csi dirver should provision volume succeed on hosted cluster
Additional info:
Description of problem:
OWNERS file updated to include prabhakar and Moe as owners and reviewers
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
This is to fecilitate easy backport via automation
Description of problem:
CO olm Degraded.
jiazha-mac:~ jiazha$ omg get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version False True 1h7m Unable to apply 4.19.0-0.nightly-multi-2025-02-26-050012: the cluster operator olm is not available jiazha-mac:~ jiazha$ omg get co olm -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: ... spec: {} status: conditions: - lastTransitionTime: '2025-02-26T16:25:34Z' message: 'CatalogdDeploymentCatalogdControllerManagerDegraded: Deployment was progressing too long OperatorcontrollerDeploymentOperatorControllerControllerManagerDegraded: Deployment was progressing too long' reason: CatalogdDeploymentCatalogdControllerManager_SyncError::OperatorcontrollerDeploymentOperatorControllerControllerManager_SyncError status: 'True' type: Degraded - lastTransitionTime: '2025-02-26T16:08:34Z' message: 'CatalogdDeploymentCatalogdControllerManagerProgressing: Waiting for Deployment to deploy pods OperatorcontrollerDeploymentOperatorControllerControllerManagerProgressing: Waiting for Deployment to deploy pods' reason: CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying status: 'True' type: Progressing - lastTransitionTime: '2025-02-26T16:08:34Z' message: 'CatalogdDeploymentCatalogdControllerManagerAvailable: Waiting for Deployment OperatorcontrollerDeploymentOperatorControllerControllerManagerAvailable: Waiting for Deployment' reason: CatalogdDeploymentCatalogdControllerManager_Deploying::OperatorcontrollerDeploymentOperatorControllerControllerManager_Deploying status: 'False' type: Available
However, the `catalogd` and `operator-controller` deployment worked well at that time.
jiazha-mac:~ jiazha$ omg get deploy NAME READY UP-TO-DATE AVAILABLE AGE catalogd-controller-manager 1/1 1 1 1h1m jiazha-mac:~ jiazha$ omg get deploy -n openshift-operator-controller NAME READY UP-TO-DATE AVAILABLE AGE operator-controller-controller-manager 1/1 1 1 1h1m jiazha-mac:~ jiazha$ omg get deploy catalogd-controller-manager -o yaml apiVersion: apps/v1 kind: Deployment ... status: availableReplicas: '1' conditions: - lastTransitionTime: '2025-02-26T16:24:35Z' lastUpdateTime: '2025-02-26T16:24:35Z' message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: 'True' type: Available - lastTransitionTime: '2025-02-26T16:22:42Z' lastUpdateTime: '2025-02-26T16:24:35Z' message: ReplicaSet "catalogd-controller-manager-7f855d8d48" has successfully progressed. reason: NewReplicaSetAvailable status: 'True' type: Progressing observedGeneration: '1' readyReplicas: '1' replicas: '1' updatedReplicas: '1' jiazha-mac:~ jiazha$ omg get deploy -n openshift-operator-controller operator-controller-controller-manager -o yaml apiVersion: apps/v1 kind: Deployment ... status: availableReplicas: '1' conditions: - lastTransitionTime: '2025-02-26T16:23:49Z' lastUpdateTime: '2025-02-26T16:23:49Z' message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: 'True' type: Available - lastTransitionTime: '2025-02-26T16:22:54Z' lastUpdateTime: '2025-02-26T16:23:49Z' message: ReplicaSet "operator-controller-controller-manager-57f648fb64" has successfully progressed. reason: NewReplicaSetAvailable status: 'True' type: Progressing observedGeneration: '1' readyReplicas: '1' replicas: '1' updatedReplicas: '1'
Version-Release number of selected component (if applicable):
How reproducible:
Not always
Steps to Reproduce:
encountered this issues twice:
1. 2. 3.
Actual results:
CO olm Degraded.
Expected results:
CO olm availabel.
Additional info:
jiazha-mac:~ jiazha$ omg project openshift-cluster-olm-operator Now using project openshift-cluster-olm-operator jiazha-mac:~ jiazha$ omg get pods NAME READY STATUS RESTARTS AGE cluster-olm-operator-5c6b8c4959-swxtt 0/1 Running 0 38m jiazha-mac:~ jiazha$ omg logs cluster-olm-operator-5c6b8c4959-swxtt -c cluster-olm-operator 2025-02-26T16:31:52.648371813Z I0226 16:31:52.643085 1 cmd.go:253] Using service-serving-cert provided certificates 2025-02-26T16:31:52.648662533Z I0226 16:31:52.648619 1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. ... 2025-02-26T16:32:05.467351366Z E0226 16:32:05.467298 1 base_controller.go:279] "Unhandled Error" err="CatalogdDeploymentCatalogdControllerManager reconciliation failed: Deployment was progressing too long" 2025-02-26T16:32:06.059681614Z I0226 16:32:06.059629 1 builder.go:224] "ProxyHook updating environment" logger="builder" deployment="operator-controller-controller-manager" 2025-02-26T16:32:06.059769494Z I0226 16:32:06.059758 1 featuregates_hook.go:33] "updating environment" logger="feature_gates_hook" deployment="operator-controller-controller-manager" 2025-02-26T16:32:06.066149493Z E0226 16:32:06.066095 1 base_controller.go:279] "Unhandled Error" err="OperatorcontrollerDeploymentOperatorControllerControllerManager reconciliation failed: Deployment was progressing too long"
Payload 4.19.0-0.nightly-2025-02-27-081354 sees console operator regression reintroduced
The original revert included steps to validate the fix
`/payload-aggregate periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-ovn-conformance 10`
Unfortunately this did not appear to happen and payloads are blocked again.
This also impacts metal-ipi-ovn-bm
and could be tested via
`/payload-job periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm`
New revert is up.
Description of problem:
Intermittently, when the MOSC resource is deleted, MOSBS resources aren't removed.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-14-090045
How reproducible:
Intermittent
Steps to Reproduce:
1.Enable techpreview 2.Create a MOSC resource to enable OCL in a pool 3.Wait until the build pod finishes and the first node starts updating 4.Remove the MOSC resource
Actual results:
The MOSB resource is not removed when the the MOSC is deleted. It is leaked.
Expected results:
When a MOSC resource is removed, all its MOSBs should be removed too.
Additional info:
Description of problem:
The release signature configmap file is invalid with no name defined
Version-Release number of selected component (if applicable):
oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202410011141.p0.g227a9c4.assembly.stream.el9-227a9c4", GitCommit:"227a9c499b6fd94e189a71776c83057149ee06c2", GitTreeState:"clean", BuildDate:"2024-10-01T20:07:43Z", GoVersion:"go1.22.5 (Red Hat 1.22.5-1.module+el8.10.0+22070+9237f38b) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
100%
Steps to Reproduce:
1) with isc : cat /test/yinzhou/config.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: channels: - name: stable-4.16 2) do mirror2disk + disk2mirror 3) use the signature configmap to create resource
Actual results:
3) failed to create resource with error: oc create -f signature-configmap.json The ConfigMap "" is invalid: metadata.name: Required value: name or generateName is required oc create -f signature-configmap.yaml The ConfigMap "" is invalid: metadata.name: Required value: name or generateName is required
Expected results:
No error
Description of problem:
Machine scale failed for GCP Marketplace cluster after upgrade from 4.12 to 4.13
Version-Release number of selected component (if applicable):
Upgrade from 4.12.26 to 4.13.0-0.nightly-2023-07-27-013427
How reproducible:
Always
Steps to Reproduce:
1.Install a 4.12 GCP Marketplace cluster liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.26 True False 24m Cluster version is 4.12.26 liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 48m huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 48m huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 48m huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 46m huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 46m 2.Upgrade to 4.13 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.0-0.nightly-2023-07-27-013427 True False 15m Cluster version is 4.13.0-0.nightly-2023-07-27-013427 liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 175m huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 175m huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 175m huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 172m huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 172m 3.Scale a machineset liuhuali@Lius-MacBook-Pro huali-test % oc scale machineset huliu-41142-4cd9z-worker-a --replicas=2 machineset.machine.openshift.io/huliu-41142-4cd9z-worker-a scaled liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-41142-4cd9z-master-0 Running n2-standard-4 us-central1 us-central1-a 5h35m huliu-41142-4cd9z-master-1 Running n2-standard-4 us-central1 us-central1-b 5h35m huliu-41142-4cd9z-master-2 Running n2-standard-4 us-central1 us-central1-c 5h35m huliu-41142-4cd9z-worker-a-pdzg2 Failed 113s huliu-41142-4cd9z-worker-a-z772h Running n2-standard-4 us-central1 us-central1-a 5h33m huliu-41142-4cd9z-worker-b-7vb9n Running n2-standard-4 us-central1 us-central1-b 5h33m liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-41142-4cd9z-worker-a-pdzg2 -oyaml apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: annotations: machine.openshift.io/instance-state: Unknown creationTimestamp: "2023-07-31T07:42:44Z" finalizers: - machine.machine.openshift.io generateName: huliu-41142-4cd9z-worker-a- generation: 1 labels: machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a name: huliu-41142-4cd9z-worker-a-pdzg2 namespace: openshift-machine-api ownerReferences: - apiVersion: machine.openshift.io/v1beta1 blockOwnerDeletion: true controller: true kind: MachineSet name: huliu-41142-4cd9z-worker-a uid: 43046eac-5ff5-4810-8e20-f0332128410f resourceVersion: "163107" uid: 1cd7d4d2-f231-457c-b21b-4ebc2d27363e spec: lifecycleHooks: {} metadata: {} providerSpec: value: apiVersion: machine.openshift.io/v1beta1 canIPForward: false credentialsSecret: name: gcp-cloud-credentials deletionProtection: false disks: - autoDelete: true boot: true image: projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145 labels: null sizeGb: 128 type: pd-ssd kind: GCPMachineProviderSpec machineType: n2-standard-4 metadata: creationTimestamp: null networkInterfaces: - network: huliu-41142-4cd9z-network subnetwork: huliu-41142-4cd9z-worker-subnet projectID: openshift-qe region: us-central1 serviceAccounts: - email: huliu-41142-4cd9z-w@openshift-qe.iam.gserviceaccount.com scopes: - https://www.googleapis.com/auth/cloud-platform shieldedInstanceConfig: {} tags: - huliu-41142-4cd9z-worker userDataSecret: name: worker-user-data zone: us-central1-a status: conditions: - lastTransitionTime: "2023-07-31T07:42:44Z" status: "True" type: Drainable - lastTransitionTime: "2023-07-31T07:42:44Z" message: Instance has not been created reason: InstanceNotCreated severity: Warning status: "False" type: InstanceExists - lastTransitionTime: "2023-07-31T07:42:44Z" status: "True" type: Terminable errorMessage: 'error launching instance: googleapi: Error 400: Invalid value for field ''resource.shieldedInstanceConfig'': ''{ "enableVtpm": true, "enableIntegrityMonitoring": true}''. Shielded VM Config can only be set when using a UEFI-compatible disk., invalid' errorReason: InvalidConfiguration lastUpdated: "2023-07-31T07:42:50Z" phase: Failed providerStatus: conditions: - lastTransitionTime: "2023-07-31T07:42:50Z" message: 'googleapi: Error 400: Invalid value for field ''resource.shieldedInstanceConfig'': ''{ "enableVtpm": true, "enableIntegrityMonitoring": true}''. Shielded VM Config can only be set when using a UEFI-compatible disk., invalid' reason: MachineCreationFailed status: "False" type: MachineCreated metadata: {} liuhuali@Lius-MacBook-Pro huali-test % oc get machineset huliu-41142-4cd9z-worker-a -oyaml apiVersion: machine.openshift.io/v1beta1 kind: MachineSet metadata: annotations: machine.openshift.io/GPU: "0" machine.openshift.io/memoryMb: "16384" machine.openshift.io/vCPU: "4" creationTimestamp: "2023-07-31T02:09:14Z" generation: 2 labels: machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z name: huliu-41142-4cd9z-worker-a namespace: openshift-machine-api resourceVersion: "163067" uid: 43046eac-5ff5-4810-8e20-f0332128410f spec: replicas: 2 selector: matchLabels: machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a template: metadata: labels: machine.openshift.io/cluster-api-cluster: huliu-41142-4cd9z machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: huliu-41142-4cd9z-worker-a spec: lifecycleHooks: {} metadata: {} providerSpec: value: apiVersion: machine.openshift.io/v1beta1 canIPForward: false credentialsSecret: name: gcp-cloud-credentials deletionProtection: false disks: - autoDelete: true boot: true image: projects/redhat-marketplace-public/global/images/redhat-coreos-ocp-48-x86-64-202210040145 labels: null sizeGb: 128 type: pd-ssd kind: GCPMachineProviderSpec machineType: n2-standard-4 metadata: creationTimestamp: null networkInterfaces: - network: huliu-41142-4cd9z-network subnetwork: huliu-41142-4cd9z-worker-subnet projectID: openshift-qe region: us-central1 serviceAccounts: - email: huliu-41142-4cd9z-w@openshift-qe.iam.gserviceaccount.com scopes: - https://www.googleapis.com/auth/cloud-platform tags: - huliu-41142-4cd9z-worker userDataSecret: name: worker-user-data zone: us-central1-a status: availableReplicas: 1 fullyLabeledReplicas: 2 observedGeneration: 2 readyReplicas: 1 replicas: 2
Actual results:
Machine scale Failed
Expected results:
Machine should get Running, it shouldn’t validation when Shielded VM Config is not set.
Additional info:
Although we found this bug https://issues.redhat.com/browse/OCPBUGS-7367, but for the upgrade, the users didn’t set the parameter (shieldedInstanceConfig), didn’t want to use the feature either, but they cannot scale up the old machineset. That’s not convenient.
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following test:
operator conditions etcd
Significant regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 99.72% to 93.55%.
Sample (being evaluated) Release: 4.18
Start Time: 2025-02-04T00:00:00Z
End Time: 2025-02-11T16:00:00Z
Success Rate: 93.55%
Successes: 87
Failures: 6
Flakes: 0
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 99.72%
Successes: 355
Failures: 1
Flakes: 0
View the test details report for additional context.
The issue is currently being discussed via https://redhat-internal.slack.com/archives/C01CQA76KMX/p1739211100312459. It seems to specifically impact periodic-ci-openshift-cluster-control-plane-machine-set-operator-release-4.18-periodics-e2e-azure jobs which in part change instances / instance types during the job and appears to be impacting static pods.
Test is failing due to oddness with oc adm logs.
We think it is related to PodLogsQuery feature that went into 1.32.
Description of problem:
The manila controller[1] defines labels that are not based on the asset prefix defined in the manila config[2], consequently when assets that selects this resource are generated they use the asset prefix as a base to define the label, resulting in them not being selected. For example in the pod antifinity[3] and controller pbd[4]. We need to change the labels used in the selectors to match the actual labels of the controller.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
machine-approver logs
E0221 20:29:52.377443 1 controller.go:182] csr-dm7zr: Pending CSRs: 1871; Max pending allowed: 604. Difference between pending CSRs and machines > 100. Ignoring all CSRs as too many recent pending CSRs seen
.
oc get csr |wc -l
3818
oc get csr |grep "node-bootstrapper" |wc -l
2152
By approving the pending CSR manually I can get the cluster to scaleup.
We can increase the maxPending to a higher number https://github.com/openshift/cluster-machine-approver/blob/2d68698410d7e6239dafa6749cc454272508db19/pkg/controller/controller.go#L330
Description of problem:
We are currently using node 18, but our types are for node 10
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Always
Steps to Reproduce:
1. Open frontend/package.json 2. Observe @types/node and engine version 3.
Actual results:
They are different
Expected results:
They are the same
Additional info:
The edit route action shows an "Edit" button in order to save the changes, instead of a "Save" button.
The button label is "Save" on other forms e.g. Deployment.
Description of problem:
Routes with SHA1 CA certificates (spec.tls.caCertificate) break HAProxy preventing reload
Version-Release number of selected component (if applicable):
4.16
How reproducible:
Always
Steps to Reproduce:
1. create Route with SHA1 CA certificates 2. 3.
Actual results:
HAProxy router fails to reload
Expected results:
HAProxy router should either reject Routes with SHA1 CA certificates, or reload successfully
Additional info:
[ALERT] (312) : config : parsing [/var/lib/haproxy/conf/haproxy.config:131] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : 'crt-list' : error processing line 1 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load chain certificate into SSL Context '/var/lib/haproxy/router/certs/test:test.pem': ca md too weak. [ALERT] (312) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] (312) : config : Fatal errors found in configuration.
This is a continuation/variance of https://issues.redhat.com/browse/OCPBUGS-26498
Description of problem:
When destroying an azure cluster, if the main resource group has already been destroyed, DNS entries are not scrubbed, despite BaseDomainResourceGroup being provided via metadata.
Version-Release number of selected component (if applicable):
Probably many. I think I have been reproducing in 4.16 and/or 4.17.
How reproducible:
Easy, 100%
Steps to Reproduce:
1. Install azure cluster with distinct BaseDomainResourceGroup . Save metadata.json. Confirm it contains BaseDomainResourceGroup. 2. Manually (via cloud API/CLI) destroy main resource group. 3. Destroy cluster, providing metadata.json from step 1.
Actual results:
DNS records still exist.
Expected results:
DNS records scrubbed.
Additional info:
thread
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
Issue found in prow ci
periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-gcp-ipi-ovn-ipsec-arm-mixarch-f14 #1890061783440297984
periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-gcp-ipi-ovn-ipsec-amd-mixarch-f28-destructive #1890035862469611520
periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-gcp-ipi-ovn-ipsec-arm-mixarch-f14 #1890279505117843456
must-gather logs for second one https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-o[…]r-must-gather/artifacts/must-gather.tar
% omg get nodes NAME STATUS ROLES AGE VERSION ci-op-9pmd0iim-3eaf1-dcw66-master-0 Ready control-plane,master 1h12m v1.32.1 ci-op-9pmd0iim-3eaf1-dcw66-master-1 Ready control-plane,master 1h13m v1.32.1 ci-op-9pmd0iim-3eaf1-dcw66-master-2 Ready control-plane,master 1h11m v1.32.1 ci-op-9pmd0iim-3eaf1-dcw66-worker-a-d6sw7 Ready worker 1h0m v1.32.1 ci-op-9pmd0iim-3eaf1-dcw66-worker-b-97qfp Ready worker 58m v1.32.1 % omg get pods -n openshift-ovn-kubernetes -o wide NAME READY STATUS RESTARTS AGE IP NODE ovn-ipsec-host-2qfqh 2/2 Running 0 33m 10.0.0.4 ci-op-9pmd0iim-3eaf1-dcw66-master-2 ovn-ipsec-host-bqh5n 0/2 Pending 0 33m 10.0.128.3 ci-op-9pmd0iim-3eaf1-dcw66-worker-b-97qfp ovn-ipsec-host-hdjtx 2/2 Running 0 33m 10.0.0.3 ci-op-9pmd0iim-3eaf1-dcw66-master-1 ovn-ipsec-host-jwn8s 2/2 Running 0 33m 10.0.0.6 ci-op-9pmd0iim-3eaf1-dcw66-master-0 ovn-ipsec-host-n4cpv 0/2 Pending 0 33m 10.0.128.2 ci-op-9pmd0iim-3eaf1-dcw66-worker-a-d6sw7 ovnkube-control-plane-85cbb47f9d-n6rps 2/2 Running 1 55m 10.0.0.6 ci-op-9pmd0iim-3eaf1-dcw66-master-0 ovnkube-control-plane-85cbb47f9d-slb94 2/2 Running 0 47m 10.0.0.3 ci-op-9pmd0iim-3eaf1-dcw66-master-1 ovnkube-node-2hwb6 8/8 Running 0 1h0m 10.0.128.2 ci-op-9pmd0iim-3eaf1-dcw66-worker-a-d6sw7 ovnkube-node-9nhj6 8/8 Running 1 53m 10.0.0.4 ci-op-9pmd0iim-3eaf1-dcw66-master-2 ovnkube-node-h2fd2 8/8 Running 2 53m 10.0.0.3 ci-op-9pmd0iim-3eaf1-dcw66-master-1 ovnkube-node-hwng4 8/8 Running 0 56m 10.0.0.6 ci-op-9pmd0iim-3eaf1-dcw66-master-0 ovnkube-node-k6rfl 8/8 Running 0 58m 10.0.128.3 ci-op-9pmd0iim-3eaf1-dcw66-worker-b-97qfp
% omg get pod ovn-ipsec-host-n4cpv -n openshift-ovn-kubernetes -o yaml apiVersion: v1 kind: Pod metadata: annotations: cluster-autoscaler.kubernetes.io/enable-ds-eviction: 'false' creationTimestamp: '2025-02-13T14:54:05Z' generateName: ovn-ipsec-host- labels: app: ovn-ipsec component: network controller-revision-hash: 8b4dd5dc7 kubernetes.io/os: linux openshift.io/component: network pod-template-generation: '1' type: infra managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:cluster-autoscaler.kubernetes.io/enable-ds-eviction: {} f:target.workload.openshift.io/management: {} f:generateName: {} f:labels: .: {} f:app: {} f:component: {} f:controller-revision-hash: {} f:kubernetes.io/os: {} f:openshift.io/component: {} f:pod-template-generation: {} f:type: {} f:ownerReferences: .: {} k:{"uid":"61870386-d205-465b-832c-061c3bf7366e"}: {} f:spec: f:affinity: .: {} f:nodeAffinity: .: {} f:requiredDuringSchedulingIgnoredDuringExecution: {} f:containers: k:{"name":"ovn-ipsec"}: .: {} f:command: {} f:env: .: {} k:{"name":"K8S_NODE"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} f:image: {} f:imagePullPolicy: {} f:lifecycle: .: {} f:preStop: .: {} f:exec: .: {} f:command: {} f:livenessProbe: .: {} f:exec: .: {} f:command: {} f:failureThreshold: {} f:initialDelaySeconds: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:name: {} f:resources: .: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/etc"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/cni/net.d"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/openvswitch"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/usr/libexec/ipsec"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/usr/sbin/ipsec"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/lib"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/log/openvswitch/"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/run"}: .: {} f:mountPath: {} f:name: {} k:{"name":"ovn-ipsec-cleanup"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: .: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/etc"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/ovn/"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/run"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:hostNetwork: {} f:hostPID: {} f:initContainers: .: {} k:{"name":"ovn-keys"}: .: {} f:command: {} f:env: .: {} k:{"name":"K8S_NODE"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: .: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:privileged: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/etc"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/openvswitch"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/etc/ovn/"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/signer-ca"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/run"}: .: {} f:mountPath: {} f:name: {} f:nodeSelector: {} f:priorityClassName: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:serviceAccount: {} f:serviceAccountName: {} f:terminationGracePeriodSeconds: {} f:tolerations: {} f:volumes: .: {} k:{"name":"etc-openvswitch"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"etc-ovn"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"host-cni-netd"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"host-etc"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"host-var-lib"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"host-var-log-ovs"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"host-var-run"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"ipsec-bin"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"ipsec-lib"}: .: {} f:hostPath: .: {} f:path: {} f:type: {} f:name: {} k:{"name":"signer-ca"}: .: {} f:configMap: .: {} f:defaultMode: {} f:name: {} f:name: {} manager: kube-controller-manager operation: Update time: '2025-02-13T14:54:04Z' - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"PodReadyToStartContainers"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:hostIPs: {} f:initContainerStatuses: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"10.0.128.2"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: '2025-02-13T14:54:05Z' name: ovn-ipsec-host-n4cpv namespace: openshift-ovn-kubernetes ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: DaemonSet name: ovn-ipsec-host uid: 61870386-d205-465b-832c-061c3bf7366e resourceVersion: '38812' uid: ce7f6619-3015-414d-9de4-5991d74258fd spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - ci-op-9pmd0iim-3eaf1-dcw66-worker-a-d6sw7 containers: - command: - /bin/bash - -c - "#!/bin/bash\nset -exuo pipefail\n\n# Don't start IPsec until ovnkube-node has\ \ finished setting up the node\ncounter=0\nuntil [ -f /etc/cni/net.d/10-ovn-kubernetes.conf\ \ ]\ndo\n counter=$((counter+1))\n sleep 1\n if [ $counter -gt 300 ];\n \ \ then\n echo \"ovnkube-node pod has not started after $counter seconds\"\ \n exit 1\n fi\ndone\necho \"ovnkube-node has configured node.\"\n\ \nif ! pgrep pluto; then\n echo \"pluto is not running, enable the service\ \ and/or check system logs\"\n exit 2\nfi\n\n# The ovs-monitor-ipsec doesn't\ \ set authby, so when it calls ipsec auto --start\n# the default ones defined\ \ at Libreswan's compile time will be used. On restart,\n# Libreswan will use\ \ authby from libreswan.config. If libreswan.config is\n# incompatible with\ \ the Libreswan's compiled-in defaults, then we'll have an\n# authentication\ \ problem. But OTOH, ovs-monitor-ipsec does set ike and esp algorithms,\n# so\ \ those may be incompatible with libreswan.config as well. Hence commenting\ \ out the\n# \"include\" from libreswan.conf to avoid such conflicts.\ndefaultcpinclude=\"\ include \\/etc\\/crypto-policies\\/back-ends\\/libreswan.config\"\nif ! grep\ \ -q \"# ${defaultcpinclude}\" /etc/ipsec.conf; then\n sed -i \"/${defaultcpinclude}/s/^/#\ \ /\" /etc/ipsec.conf\n # since pluto is on the host, we need to restart it\ \ after changing connection\n # parameters.\n chroot /proc/1/root ipsec restart\n\ \n counter=0\n until [ -r /run/pluto/pluto.ctl ]; do\n counter=$((counter+1))\n\ \ sleep 1\n if [ $counter -gt 300 ];\n then\n echo \"ipsec has\ \ not started after $counter seconds\"\n exit 1\n fi\n done\n echo\ \ \"ipsec service is restarted\"\nfi\n\n# Workaround for https://github.com/libreswan/libreswan/issues/373\n\ ulimit -n 1024\n\n/usr/libexec/ipsec/addconn --config /etc/ipsec.conf --checkconfig\n\ # Check kernel modules\n/usr/libexec/ipsec/_stackmanager start\n# Check nss\ \ database status\n/usr/sbin/ipsec --checknss\n\n# Start ovs-monitor-ipsec which\ \ will monitor for changes in the ovs\n# tunnelling configuration (for example\ \ addition of a node) and configures\n# libreswan appropriately.\n# We are running\ \ this in the foreground so that the container will be restarted when ovs-monitor-ipsec\ \ fails.\n/usr/libexec/platform-python /usr/share/openvswitch/scripts/ovs-monitor-ipsec\ \ \\\n --pidfile=/var/run/openvswitch/ovs-monitor-ipsec.pid --ike-daemon=libreswan\ \ --no-restart-ike-daemon \\\n --ipsec-conf /etc/ipsec.d/openshift.conf --ipsec-d\ \ /var/lib/ipsec/nss \\\n --log-file --monitor unix:/var/run/openvswitch/db.sock\n" env: - name: K8S_NODE valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /bin/bash - -c - '#!/bin/bash set -exuo pipefail # In order to maintain traffic flows during container restart, we # need to ensure that xfrm state and policies are not flushed. # Don''t allow ovs monitor to cleanup persistent state kill "$(cat /var/run/openvswitch/ovs-monitor-ipsec.pid 2>/dev/null)" 2>/dev/null || true ' livenessProbe: exec: command: - /bin/bash - -c - "#!/bin/bash\nif [[ $(ipsec whack --trafficstatus | wc -l) -eq 0 ]]; then\n\ \ echo \"no ipsec traffic configured\"\n exit 10\nfi\n" failureThreshold: 3 initialDelaySeconds: 15 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 1 name: ovn-ipsec resources: requests: cpu: 10m memory: 100Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/cni/net.d name: host-cni-netd - mountPath: /var/run name: host-var-run - mountPath: /var/log/openvswitch/ name: host-var-log-ovs - mountPath: /etc/openvswitch name: etc-openvswitch - mountPath: /var/lib name: host-var-lib - mountPath: /etc name: host-etc - mountPath: /usr/sbin/ipsec name: ipsec-bin - mountPath: /usr/libexec/ipsec name: ipsec-lib - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true - command: - /bin/bash - -c - "#!/bin/bash\n\n# When NETWORK_NODE_IDENTITY_ENABLE is true, use the per-node\ \ certificate to create a kubeconfig\n# that will be used to talk to the API\n\ \n\n# Wait for cert file\nretries=0\ntries=20\nkey_cert=\"/etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\"\ \nwhile [ ! -f \"${key_cert}\" ]; do\n (( retries += 1 ))\n if [[ \"${retries}\"\ \ -gt ${tries} ]]; then\n echo \"$(date -Iseconds) - ERROR - ${key_cert}\ \ not found\"\n return 1\n fi\n sleep 1\ndone\n\ncat << EOF > /var/run/ovnkube-kubeconfig\n\ apiVersion: v1\nclusters:\n - cluster:\n certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n\ \ server: https://api-int.ci-op-9pmd0iim-3eaf1.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443\n\ \ name: default-cluster\ncontexts:\n - context:\n cluster: default-cluster\n\ \ namespace: default\n user: default-auth\n name: default-context\n\ current-context: default-context\nkind: Config\npreferences: {}\nusers:\n -\ \ name: default-auth\n user:\n client-certificate: /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\n\ \ client-key: /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\n\ EOF\nexport KUBECONFIG=/var/run/ovnkube-kubeconfig\n\n\n# It is safe to flush\ \ xfrm states and policies and delete openshift.conf\n# file when east-west\ \ ipsec is disabled. This fixes a race condition when\n# ovs-monitor-ipsec is\ \ not fast enough to notice ipsec config change and\n# delete entries before\ \ it's being killed.\n# Since it's cleaning up all xfrm states and policies,\ \ it may cause slight\n# interruption until ipsec is restarted in case of external\ \ ipsec config.\n# We must do this before killing ovs-monitor-ipsec script,\ \ otherwise\n# preStop hook doesn't get a chance to run it because ovn-ipsec\ \ container\n# is abruptly terminated.\n# When east-west ipsec is not disabled,\ \ then do not flush xfrm states and\n# policies in order to maintain traffic\ \ flows during container restart.\nipsecflush() {\n if [ \"$(kubectl get networks.operator.openshift.io\ \ cluster -ojsonpath='{.spec.defaultNetwork.ovnKubernetesConfig.ipsecConfig.mode}')\"\ \ != \"Full\" ] && \\\n [ \"$(kubectl get networks.operator.openshift.io\ \ cluster -ojsonpath='{.spec.defaultNetwork.ovnKubernetesConfig.ipsecConfig}')\"\ \ != \"{}\" ]; then\n ip x s flush\n ip x p flush\n rm -f /etc/ipsec.d/openshift.conf\n\ \ # since pluto is on the host, we need to restart it after the flush\n \ \ chroot /proc/1/root ipsec restart\n fi\n}\n\n# Function to handle SIGTERM\n\ cleanup() {\n echo \"received SIGTERM, flushing ipsec config\"\n # Wait upto\ \ 15 seconds for ovs-monitor-ipsec process to terminate before\n # cleaning\ \ up ipsec entries.\n counter=0\n while kill -0 \"$(cat /var/run/openvswitch/ovs-monitor-ipsec.pid\ \ 2>/dev/null)\"; do\n counter=$((counter+1))\n sleep 1\n if [ $counter\ \ -gt 15 ];\n then\n echo \"ovs-monitor-ipsec has not terminated after\ \ $counter seconds\"\n break\n fi\n done\n ipsecflush\n exit 0\n\ }\n\n# Trap SIGTERM and call cleanup function\ntrap cleanup SIGTERM\n\ncounter=0\n\ until [ -r /var/run/openvswitch/ovs-monitor-ipsec.pid ]; do\n counter=$((counter+1))\n\ \ sleep 1\n if [ $counter -gt 300 ];\n then\n echo \"ovs-monitor-ipsec\ \ has not started after $counter seconds\"\n exit 1\n fi\ndone\necho \"\ ovs-monitor-ipsec is started\"\n\n# Monitor the ovs-monitor-ipsec process.\n\ while kill -0 \"$(cat /var/run/openvswitch/ovs-monitor-ipsec.pid 2>/dev/null)\"\ ; do\n sleep 1\ndone\n\n# Once the ovs-monitor-ipsec process terminates, execute\ \ the cleanup command.\necho \"ovs-monitor-ipsec is terminated, flushing ipsec\ \ config\"\nipsecflush\n\n# Continue running until SIGTERM is received (or exit\ \ naturally)\nwhile true; do\n sleep 1\ndone\n" image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imagePullPolicy: IfNotPresent name: ovn-ipsec-cleanup resources: requests: cpu: 10m memory: 50Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/ovn/ name: etc-ovn - mountPath: /var/run name: host-var-run - mountPath: /etc name: host-etc - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true dnsPolicy: Default enableServiceLinks: true hostNetwork: true hostPID: true imagePullSecrets: - name: ovn-kubernetes-node-dockercfg-sds8g initContainers: - command: - /bin/bash - -c - "#!/bin/bash\nset -exuo pipefail\n\n# When NETWORK_NODE_IDENTITY_ENABLE is true,\ \ use the per-node certificate to create a kubeconfig\n# that will be used to\ \ talk to the API\n\n\n# Wait for cert file\nretries=0\ntries=20\nkey_cert=\"\ /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\"\nwhile [ ! -f \"${key_cert}\"\ \ ]; do\n (( retries += 1 ))\n if [[ \"${retries}\" -gt ${tries} ]]; then\n\ \ echo \"$(date -Iseconds) - ERROR - ${key_cert} not found\"\n return\ \ 1\n fi\n sleep 1\ndone\n\ncat << EOF > /var/run/ovnkube-kubeconfig\napiVersion:\ \ v1\nclusters:\n - cluster:\n certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n\ \ server: https://api-int.ci-op-9pmd0iim-3eaf1.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443\n\ \ name: default-cluster\ncontexts:\n - context:\n cluster: default-cluster\n\ \ namespace: default\n user: default-auth\n name: default-context\n\ current-context: default-context\nkind: Config\npreferences: {}\nusers:\n -\ \ name: default-auth\n user:\n client-certificate: /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\n\ \ client-key: /etc/ovn/ovnkube-node-certs/ovnkube-client-current.pem\n\ EOF\nexport KUBECONFIG=/var/run/ovnkube-kubeconfig\n\n\n# Every time we restart\ \ this container, we will create a new key pair if\n# we are close to key expiration\ \ or if we do not already have a signed key pair.\n#\n# Each node has a key\ \ pair which is used by OVS to encrypt/decrypt/authenticate traffic\n# between\ \ each node. The CA cert is used as the root of trust for all certs so we need\n\ # the CA to sign our certificate signing requests with the CA private key. In\ \ this way,\n# we can validate that any signed certificates that we receive\ \ from other nodes are\n# authentic.\necho \"Configuring IPsec keys\"\n\ncert_pem=/etc/openvswitch/keys/ipsec-cert.pem\n\ \n# If the certificate does not exist or it will expire in the next 6 months\n\ # (15770000 seconds), we will generate a new one.\nif ! openssl x509 -noout\ \ -dates -checkend 15770000 -in $cert_pem; then\n # We use the system-id as\ \ the CN for our certificate signing request. This\n # is a requirement by\ \ OVN.\n cn=$(ovs-vsctl --retry -t 60 get Open_vSwitch . external-ids:system-id\ \ | tr -d \"\\\"\")\n\n mkdir -p /etc/openvswitch/keys\n\n # Generate an SSL\ \ private key and use the key to create a certitificate signing request\n umask\ \ 077 && openssl genrsa -out /etc/openvswitch/keys/ipsec-privkey.pem 2048\n\ \ openssl req -new -text \\\n -extensions v3_req \\\n \ \ -addext \"subjectAltName = DNS:${cn}\" \\\n -subj \"/C=US/O=ovnkubernetes/OU=kind/CN=${cn}\"\ \ \\\n -key /etc/openvswitch/keys/ipsec-privkey.pem \\\n \ \ -out /etc/openvswitch/keys/ipsec-req.pem\n\n csr_64=$(base64 -w0 /etc/openvswitch/keys/ipsec-req.pem)\ \ # -w0 to avoid line-wrap\n\n # Request that our generated certificate signing\ \ request is\n # signed by the \"network.openshift.io/signer\" signer that\ \ is\n # implemented by the CNO signer controller. This will sign the\n #\ \ certificate signing request using the signer-ca which has been\n # set up\ \ by the OperatorPKI. In this way, we have a signed certificate\n # and our\ \ private key has remained private on this host.\n cat <<EOF | kubectl create\ \ -f -\n apiVersion: certificates.k8s.io/v1\n kind: CertificateSigningRequest\n\ \ metadata:\n generateName: ipsec-csr-$(hostname)-\n labels:\n k8s.ovn.org/ipsec-csr:\ \ $(hostname)\n spec:\n request: ${csr_64}\n signerName: network.openshift.io/signer\n\ \ usages:\n - ipsec tunnel\nEOF\n # Wait until the certificate signing\ \ request has been signed.\n counter=0\n until [ -n \"$(kubectl get csr -lk8s.ovn.org/ipsec-csr=\"\ $(hostname)\" --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].status.certificate}'\ \ 2>/dev/null)\" ]\n do\n counter=$((counter+1))\n sleep 1\n if [\ \ $counter -gt 60 ];\n then\n echo \"Unable to sign certificate\ \ after $counter seconds\"\n exit 1\n fi\n done\n\n # Decode\ \ the signed certificate.\n kubectl get csr -lk8s.ovn.org/ipsec-csr=\"$(hostname)\"\ \ --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1:].status.certificate}'\ \ | base64 -d | openssl x509 -outform pem -text -out $cert_pem\n\n # kubectl\ \ delete csr/$(hostname)\n\n # Get the CA certificate so we can authenticate\ \ peer nodes.\n openssl x509 -in /signer-ca/ca-bundle.crt -outform pem -text\ \ -out /etc/openvswitch/keys/ipsec-cacert.pem\nfi\n\n# Configure OVS with the\ \ relevant keys for this node. This is required by ovs-monitor-ipsec.\n#\n#\ \ Updating the certificates does not need to be an atomic operation as\n# the\ \ will get read and loaded into NSS by the ovs-monitor-ipsec process\n# which\ \ has not started yet.\novs-vsctl --retry -t 60 set Open_vSwitch . other_config:certificate=$cert_pem\ \ \\\n other_config:private_key=/etc/openvswitch/keys/ipsec-privkey.pem\ \ \\\n other_config:ca_cert=/etc/openvswitch/keys/ipsec-cacert.pem\n" env: - name: K8S_NODE valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imagePullPolicy: IfNotPresent name: ovn-keys resources: requests: cpu: 10m memory: 100Mi securityContext: privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /etc/ovn/ name: etc-ovn - mountPath: /var/run name: host-var-run - mountPath: /signer-ca name: signer-ca - mountPath: /etc/openvswitch name: etc-openvswitch - mountPath: /etc name: host-etc - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true nodeName: ci-op-9pmd0iim-3eaf1-dcw66-worker-a-d6sw7 nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: ovn-kubernetes-node serviceAccountName: ovn-kubernetes-node terminationGracePeriodSeconds: 10 tolerations: - operator: Exists volumes: - hostPath: path: /var/lib/ovn-ic/etc type: '' name: etc-ovn - hostPath: path: /var/log/openvswitch type: DirectoryOrCreate name: host-var-log-ovs - configMap: defaultMode: 420 name: signer-ca name: signer-ca - hostPath: path: /var/lib/openvswitch/etc type: DirectoryOrCreate name: etc-openvswitch - hostPath: path: /var/run/multus/cni/net.d type: '' name: host-cni-netd - hostPath: path: /var/run type: DirectoryOrCreate name: host-var-run - hostPath: path: /var/lib type: DirectoryOrCreate name: host-var-lib - hostPath: path: /etc type: Directory name: host-etc - hostPath: path: /usr/sbin/ipsec type: File name: ipsec-bin - hostPath: path: /usr/libexec/ipsec type: Directory name: ipsec-lib - name: kube-api-access-7rvbc projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace - configMap: items: - key: service-ca.crt path: service-ca.crt name: openshift-service-ca.crt status: conditions: - lastProbeTime: null lastTransitionTime: '2025-02-13T14:54:05Z' status: 'False' type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: '2025-02-13T14:54:05Z' message: 'containers with incomplete status: [ovn-keys]' reason: ContainersNotInitialized status: 'False' type: Initialized - lastProbeTime: null lastTransitionTime: '2025-02-13T14:54:05Z' message: 'containers with unready status: [ovn-ipsec ovn-ipsec-cleanup]' reason: ContainersNotReady status: 'False' type: Ready - lastProbeTime: null lastTransitionTime: '2025-02-13T14:54:05Z' message: 'containers with unready status: [ovn-ipsec ovn-ipsec-cleanup]' reason: ContainersNotReady status: 'False' type: ContainersReady - lastProbeTime: null lastTransitionTime: '2025-02-13T14:54:05Z' status: 'True' type: PodScheduled containerStatuses: - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imageID: '' lastState: {} name: ovn-ipsec ready: false restartCount: 0 started: false state: waiting: reason: PodInitializing volumeMounts: - mountPath: /etc/cni/net.d name: host-cni-netd - mountPath: /var/run name: host-var-run - mountPath: /var/log/openvswitch/ name: host-var-log-ovs - mountPath: /etc/openvswitch name: etc-openvswitch - mountPath: /var/lib name: host-var-lib - mountPath: /etc name: host-etc - mountPath: /usr/sbin/ipsec name: ipsec-bin - mountPath: /usr/libexec/ipsec name: ipsec-lib - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true recursiveReadOnly: Disabled - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imageID: '' lastState: {} name: ovn-ipsec-cleanup ready: false restartCount: 0 started: false state: waiting: reason: PodInitializing volumeMounts: - mountPath: /etc/ovn/ name: etc-ovn - mountPath: /var/run name: host-var-run - mountPath: /etc name: host-etc - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true recursiveReadOnly: Disabled hostIP: 10.0.128.2 hostIPs: - ip: 10.0.128.2 initContainerStatuses: - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e262b9ed22e74a3a8d7a345b775645267acfbcd571b510e1ace519cc2f658bf imageID: '' lastState: {} name: ovn-keys ready: false restartCount: 0 started: false state: waiting: reason: PodInitializing volumeMounts: - mountPath: /etc/ovn/ name: etc-ovn - mountPath: /var/run name: host-var-run - mountPath: /signer-ca name: signer-ca - mountPath: /etc/openvswitch name: etc-openvswitch - mountPath: /etc name: host-etc - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-7rvbc readOnly: true recursiveReadOnly: Disabled phase: Pending podIP: 10.0.128.2 podIPs: - ip: 10.0.128.2 qosClass: Burstable startTime: '2025-02-13T14:54:05Z' 1.7.1 PrivateBin is a minimalist, open source online pastebin where the server has zero knowledge of pasted data. Data is encrypted/decrypted in the browser using 256 bits AES. More information on the project page. Red Hat Employee Privacy Statement
1.
2.
3.
Actual results:
Expected results:
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Please review the following PR: https://github.com/openshift/service-ca-operator/pull/246
The PR has been automatically opened by ART (#forum-ocp-art) team automation and indicates
that the image(s) being used downstream for production builds are not consistent
with the images referenced in this component's github repository.
Differences in upstream and downstream builds impact the fidelity of your CI signal.
If you disagree with the content of this PR, please contact @release-artists
in #forum-ocp-art to discuss the discrepancy.
Closing this issue without addressing the difference will cause the issue to
be reopened automatically.
CVO manifests contain some feature-gated ones:
We observed HyperShift CI jobs to fail when adding DevPreview-gated deployment manifests to CVO, which was unexpected. Investigating further, we discovered that HyperShift applies them:
error: error parsing /var/payload/manifests/0000_00_update-status-controller_03_deployment-DevPreviewNoUpgrade.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{".ReleaseImage":interface {}(nil)}
But even without these added manifests, this happens for existing ClusterVersion CRD manifests present in the payload:
$ ls -1 manifests/*clusterversions*crd.yaml manifests/0000_00_cluster-version-operator_01_clusterversions-CustomNoUpgrade.crd.yaml manifests/0000_00_cluster-version-operator_01_clusterversions-Default.crd.yaml manifests/0000_00_cluster-version-operator_01_clusterversions-DevPreviewNoUpgrade.crd.yaml manifests/0000_00_cluster-version-operator_01_clusterversions-TechPreviewNoUpgrade.crd.yaml
In a passing HyperShift CI job, the same log shows that all four manifests are applied instead of just one:
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
4.18
Always
1. inspect the cluster-version-operator-*-bootstrap.log of a HyperShift CI job
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io configured
= all four ClusterVersion CRD manifests are applied
customresourcedefinition.apiextensions.k8s.io/clusterversions.config.openshift.io created
= ClusterVersion CRD manifest is applied just once
I'm filing this card so that I can link it to the "easy" fix https://github.com/openshift/hypershift/pull/5093 which is not the perfect fix, but allows us to add featureset-gated manifests to CVO without breaking HyperShift. It is desirable to improve this even further and actually correctly select the manifests to be applied for CVO bootstrap, but that involves non-trivial logic similar to one used by CVO and it seems to be better approached as a feature to be properly assessed and implemented, rather than a bugfix, so I'll file a separate HOSTEDCP card for that.
Description of problem:
Install cluster in existing resource group, After bootstrap server is destroyed, inboundNatRule ssh_in in external load balancer is not deleted. ssh nsg rule is also leftover in nsg. $ az network lb list -g ci-op-vq47c2zq-11f79-rg -otable Location Name ProvisioningState ResourceGroup ResourceGuid ---------- ----------------------------------- ------------------- ----------------------- ------------------------------------ centralus ci-op-vq47c2zq-11f79-xhl4q Succeeded ci-op-vq47c2zq-11f79-rg 282960e6-014e-4abe-8f61-2782cd82ca82 centralus ci-op-vq47c2zq-11f79-xhl4q-internal Succeeded ci-op-vq47c2zq-11f79-rg 0e3afbf2-f2b2-4f59-8771-ccef9457fd90 $ az network lb inbound-nat-rule list --lb-name ci-op-vq47c2zq-11f79-xhl4q -g ci-op-vq47c2zq-11f79-rg -otable BackendPort EnableFloatingIP EnableTcpReset FrontendPort IdleTimeoutInMinutes Name Protocol ProvisioningState ResourceGroup ------------- ------------------ ---------------- -------------- ---------------------- --------------------------------- ---------- ------------------- ----------------------- 22 False False 22 4 ci-op-vq47c2zq-11f79-xhl4q_ssh_in Tcp Succeeded ci-op-vq47c2zq-11f79-rg $ az network nsg rule list --nsg-name ci-op-vq47c2zq-11f79-xhl4q-nsg -g ci-op-vq47c2zq-11f79-rg -otable Name ResourceGroup Priority SourcePortRanges SourceAddressPrefixes SourceASG Access Protocol Direction DestinationPortRanges DestinationAddressPrefixes DestinationASG -------------------------------------------------------- ----------------------- ---------- ------------------ ----------------------- ----------- -------- ---------- ----------- ----------------------- ---------------------------- ---------------- apiserver_in ci-op-vq47c2zq-11f79-rg 101 * * None Allow Tcp Inbound 6443 * None ci-op-vq47c2zq-11f79-xhl4q_ssh_in ci-op-vq47c2zq-11f79-rg 220 * * None Allow Tcp Inbound 22 * None k8s-azure-lb_allow_IPv4_556f7044ec033071ec0dfcf7cd85bc93 ci-op-vq47c2zq-11f79-rg 500 * Internet None Allow Tcp Inbound 443 80 48.214.241.65 None
Version-Release number of selected component (if applicable):
4.18 nightly build
How reproducible:
Always
Steps to Reproduce:
1. Specify platform.azure.resourceGroupName to pre-created resource group name in install-config 2. Install cluster 3.
Actual results:
InboundNatRule in external load balancer and ssh nsg rule in nsg are leftover after bootstrap server is deleted.
Expected results:
All resources associated with bootstrap should be removed after bootstrap server is destroyed.
Additional info:
Look like that resource group name is hard-coded as "<infrad-id>-rg" in post destroy, see code: https://github.com/openshift/installer/blob/master/pkg/infrastructure/azure/azure.go#L717
Description of problem:
It's the testing scenario of QE test case OCP-24405, i.e. after a successful IPI installation, add an additional compute/worker node without infra_id as name prefix. The expectation is, "destroy cluster" could delete the additional compute/worker machine smoothly. But the testing results is, "destroy cluster" seems unaware of the machine.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-multi-2024-12-17-192034
How reproducible:
Always
Steps to Reproduce:
1. install an IPI cluster on GCP and make sure it succeeds (see [1]) 2. add the additional compute/worker node, and ensure the node's name doesn't have the cluster infra ID (see [2]) 3. wait for the node ready and all cluster operators available 4. (optional) scale ingress operator replica into 3 (see [3]), and wait for ingress operator finishing progressing 5. check the new machine on GCP (see [4]) 6. "destroy cluster" (see [5])
Actual results:
The additional compute/worker node is not deleted, which seems also leading to k8s firewall-rules / forwarding-rule / target-pool / http-health-check not deleted.
Expected results:
"destroy cluster" should be able to detect the additional compute/worker node by the label "kubernetes-io-cluster-<infra id>: owned" and delete it along with all resources of the cluster.
Additional info:
Alternatively, we also tested with creating the additional compute/worker machine by a machineset YAML (rather than a machine YAML), and we got the same issue in such case.
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
If a user updates a deployment config using Form view instead of yaml, the image pull secret is getting duplicated. ~~~ $ oc get pods ubi9-2-deploy message: | error: couldn't assign source annotation to deployment ubi9-2: failed to create manager for existing fields: failed to convert new object (testdc-dup-sec/ubi9-2; /v1, Kind=ReplicationController) to smd typed: .spec.template.spec.imagePullSecrets: duplicate entries for key [name="test-pull-secret"] reason: Error ~~~
Version-Release number of selected component (if applicable):
4.13.z,4.14.z,4.15.z
How reproducible:
Steps to Reproduce:
1. Edit DeploymentConfig in Form view 2. Update image version 3. Save
Actual results:
Expected results:
Additional info:
Issue is not reproducible on OCP 4.16.7+ version.
Description of problem:
Status card has some styling issues on some browsers
Version-Release number of selected component (if applicable):
For example Firefox ESR 128.7.0esr (64-bit) Firefox 135.0 (aarch64) Safari 18.1.1 (20619.2.8.11.12)
How reproducible:
Always
Steps to Reproduce:
1. navigate to Home -> Overview page, observe Status card format
Actual results:
issue on Safari https://drive.google.com/file/d/1lfHgS-B_bWGN4YnerDPrawN13ujTGKCt/view?usp=drive_link issue on Firefox and Firefox ESR https://drive.google.com/file/d/1dP-ZZ-11EIdcquoZ3_XSKaYLo__O5_ge/view?usp=drive_link
Expected results:
consistent format across all browsers
Additional info:
OSD-26887: managed services taints several nodes as infrastructure. This taint appears to be applied after some of the platform DS are scheduled there, causing this alert to fire. Managed services rebalances the DS after the taint is added, and the alert clears, but origin fails this test. Allowing this alert to fire while we investigate why the taint is not added at node birth.
Description of problem:
The --report and --pxe flags were introduced in 4.18. It should be marked as experimental until 4.19.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
It would be good to fail the build if the rt rpm does not match the kernel. Since 9.4+ based releases, rt comes from the same package as kernel. With this change, ARTs consistency check lost an ability. This bug is to "shift-left" that test, and have the build fail at build time.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When deploying the disconnected scenario in PowerVS (SNAT disabled network), components that rely on NTP fail. IBM Cloud has an NTP server that we can use internally so we need to point to that through chrony.conf
Version-Release number of selected component (if applicable):
How reproducible:
As easy as deploying a disconnected cluster
Steps to Reproduce:
1. Deploy a disconnected cluster 2. image-registry will fail due to NTP mismatch 3.
Actual results:
Expected results:
Additional info:
Description of problem:
when mirror ocp payload with digest will failed with error : invalid destination name docker://ci-op-n2k1twzy-c1a88-bastion-mirror-registry-xxxxxxxx-zhouy.apps.yinzhou-1031.qe.devcluster.openshift.com/ci-op-n2k1twzy/release/openshift/release-images:: invalid reference format
Version-Release number of selected component (if applicable):
./oc-mirror.rhel8 version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.18.0-202410251041.p0.g95f0611.assembly.stream.el9-95f0611", GitCommit:"95f0611c1dc9584a4a9e857912b9eaa539234bbc", GitTreeState:"clean", BuildDate:"2024-10-25T11:28:19Z", GoVersion:"go1.22.7 (Red Hat 1.22.7-1.module+el8.10.0+22325+dc584f75) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
How reproducible:
Always
Steps to Reproduce:
1. imagesetconfig with digest for ocp payload : cat config.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: release: registry.ci.openshift.org/ocp/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4 2. run the mirror2mirror command : ./oc-mirror.rhel8.18 -c config.yaml docker://ci-op-n2k1twzy-c1a88-bastion-mirror-registry-xxxxxxxx-zhouy.apps.yinzhou-1031.qe.devcluster.openshift.com/ci-op-n2k1twzy/release --dest-tls-verify=false --v2 --workspace file://out1 --authfile auth.json
Actual results:
2. hit error : ✗ 188/188 : (2s) registry.ci.openshift.org/ocp/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4 2024/10/31 06:20:03 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/10/31 06:20:03 [ERROR] : invalid destination name docker://ci-op-n2k1twzy-c1a88-bastion-mirror-registry-xxxxxxxx-zhouy.apps.yinzhou-1031.qe.devcluster.openshift.com/ci-op-n2k1twzy/release/openshift/release-images:: invalid reference format
Expected results:
3. no error
Additional info:
compared with 4.17 oc-mirror, no such issue : ./oc-mirror -c config.yaml docker://ci-op-n2k1twzy-c1a88-bastion-mirror-registry-xxxxxxxx-zhouy.apps.yinzhou-1031.qe.devcluster.openshift.com/ci-op-n2k1twzy/release --dest-tls-verify=false --v2 --workspace file://out1 --authfile auth.json 2024/10/31 06:23:04 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/10/31 06:23:04 [INFO] : 👋 Hello, welcome to oc-mirror ... ✓ 188/188 : (1s) registry.ci.openshift.org/ocp/release@sha256:e87cdacdf5c575ff99d4cca7ec38758512a408ac1653fef8dd7b2c4b85e295f4 2024/10/31 06:27:58 [INFO] : === Results === 2024/10/31 06:27:58 [INFO] : ✅ 188 / 188 release images mirrored successfully 2024/10/31 06:27:58 [INFO] : 📄 Generating IDMS file... 2024/10/31 06:27:58 [INFO] : out1/working-dir/cluster-resources/idms-oc-mirror.yaml file created 2024/10/31 06:27:58 [INFO] : 📄 No images by tag were mirrored. Skipping ITMS generation. 2024/10/31 06:27:58 [INFO] : 📄 No catalogs mirrored. Skipping CatalogSource file generation. 2024/10/31 06:27:58 [INFO] : mirror time : 4m54.452548695s 2024/10/31 06:27:58 [INFO] : 👋 Goodbye, thank you for using oc-mirror
golang.org/x/net is a CVE-prone dependency, and even if we are not actually exposed to some issues, carrying an old dep exposes us to version-based vulnerability scanners.
When building a container image using Dockerfile.dev, the resulting image does not include the necessary font files provided by PatternFly (e.g., RedHatText). As a result, the console renders with a system fallback. The root cause of this issue is an overly broad ignore introduced with https://github.com/openshift/console/pull/12538.
Description of problem:
The current `oc adm inspect --all-namespaces` command line results in something like this: oc adm inspect --dest-dir must-gather --rotated-pod-logs csistoragecapacities ns/assisted-installer leases --all-namespaces Which is wrong because of 2 reasons: - The `ns/assisted-installer` is there although a namespace is not namespaced, so it should go to the `named_resources` variable (this happens only in 4.16+) - The rest of the items on `all_ns_resources` variable are group resources, but they are not separated by `,` like we do with `group_resources` (this happens on 4.14+) As a result, we never collect what is intended with this command.
Version-Release number of selected component (if applicable):
Any 4.14+ version
How reproducible:
Always
Steps to Reproduce:
1. Get a must-gather 2. 3.
Actual results:
Data from "oc adm inspect --all-namespaces" missing
Expected results:
No data missing
Additional info:
Description of problem:
Layout incorrect for Service weight on Create Route page,
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-05-103644
How reproducible:
Always
Steps to Reproduce:
1. Navigate to ‘Create Route’ page, eg: /k8s/ns/default/route.openshift.io~v1~Route/~new/form 2. Check the field od 'Service weight' 3.
Actual results:
the input field for 'Service weight' is too long
Expected results:
Compared to a similar component in OpenShift, the input field should be shorter
Additional info:
Description of problem:
Observed in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn/1866088107347021824/artifacts/e2e-gcp-ovn/ipi-install-install/artifacts/.openshift_install-1733747884.log Distinct issues occurring in this job caused the "etcd bootstrap member to be removed from cluster" gate to take longer than its 5 minute timeout, but there was plenty of time left to complete bootstrapping successfully. It doesn't make sense to have a narrow timeout here because progress toward removal of the etcd bootstrap member begins the moment the etcd cluster starts for the first time, not when the installer starts waiting to observe it.
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Sometimes
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When running oc-mirror V2 (either 4.16 or 4.17 has been tested) on a RHEL 9 FIPS enabled and STIG Security profile enforced system, oc-mirror fails due to a hard coded PGP key in oc-mirror V2.
Version-Release number of selected component (if applicable):
At least 4.16-4.17
How reproducible:
Very reproducible
Steps to Reproduce:
ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: channels: - name: stable-4.16 minVersion: 4.16.18 maxVersion: 4.16.24 shortestPath: true
3. run oc-mirror with the following flags:
[cnovak@localhost ocp4-disconnected-config]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v2 2024/12/18 14:40:01 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/12/18 14:40:01 [INFO] : 👋 Hello, welcome to oc-mirror 2024/12/18 14:40:01 [INFO] : ⚙️ setting up the environment for you... 2024/12/18 14:40:01 [INFO] : 🔀 workflow mode: mirrorToDisk 2024/12/18 14:40:01 [INFO] : 🕵️ going to discover the necessary images... 2024/12/18 14:40:01 [INFO] : 🔍 collecting release images... 2024/12/18 14:40:02 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure 2024/12/18 14:40:02 [ERROR] : generate release signatures: error list invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 2024/12/18 14:40:02 [INFO] : 🔍 collecting operator images... 2024/12/18 14:40:02 [INFO] : 🔍 collecting additional images... 2024/12/18 14:40:02 [INFO] : 🚀 Start copying the images... 2024/12/18 14:40:02 [INFO] : images to copy 0 2024/12/18 14:40:02 [INFO] : === Results === 2024/12/18 14:40:02 [INFO] : 📦 Preparing the tarball archive... 2024/12/18 14:40:02 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/12/18 14:40:02 [ERROR] : unable to add cache repositories to the archive : lstat /home/cnovak/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
Expected results/immediate workaround:
[cnovak@localhost ~]$ curl -s https://raw.githubusercontent.com/openshift/cluster-update-keys/d44fca585d081a72cb2c67734556a27bbfc9470e/manifests.rhel/0000_90_cluster-update-keys_configmap.yaml | sed -n '/openshift[.]io/d;s/Comment:.*//;s/^ //p' > /tmp/pgpkey [cnovak@localhost ~]$ export OCP_SIGNATURE_VERIFICATION_PK=/tmp/pgpkey [cnovak@localhost ~]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v22024/12/19 08:54:42 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/12/19 08:54:42 [INFO] : 👋 Hello, welcome to oc-mirror 2024/12/19 08:54:42 [INFO] : ⚙️ setting up the environment for you... 2024/12/19 08:54:42 [INFO] : 🔀 workflow mode: mirrorToDisk 2024/12/19 08:54:42 [INFO] : 🕵️ going to discover the necessary images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting release images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting operator images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting additional images... 2024/12/19 08:54:42 [INFO] : 🚀 Start copying the images... 2024/12/19 08:54:42 [INFO] : images to copy 382 ⠸ 1/382 : (7s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb ✓ 2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace ✓ 3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 ✓ 4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 ⠋ 1/382 : (19s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb ✓ 2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace ✓ 3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 ✓ 4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 ✓ 5/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e54fc21197c341fe257d2f2f2ad14b578483c4450474dc2cf876a885f11e745 ✓ 6/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c934b4d95545e29f9cb7586964fd43cdb7b8533619961aaa932fe2923ab40db ✓ 7/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:851ba9ac5219a9f11e927200715e666ae515590cd9cc6dde9631070afb66b5d7 ✓ 8/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f614ef855220f2381217c31b8cb94c05ef20edf3ca23b5efa0be1b957cdde3a4
Additional info:
The reason this is a critical issue, is Red Hat has a relatively large footprint within the DoD/U.S Government space, and anyone who is working in a disconnected environment, with a STIG Policy enforced on a RHEL 9 machine, will run into this problem. Additionally, below is output from oc-mirror version [cnovak@localhost ~]$ oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.17.0-202411251634.p0.g07714b7.assembly.stream.el9-07714b7", GitCommit:"07714b7c836ec3ad1b776f25b44c3b2c2f083aa2", GitTreeState:"clean", BuildDate:"2024-11-26T08:28:42Z", GoVersion:"go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
Description of problem:
When Cluster Settings tab is opened on the console the below error is displayed: Oh no! Something went wrong Type Error Description: Cannot read properties of null (reading "major')
Version-Release number of selected component (if applicable):
OCP Version 4.14.34
How reproducible:
Steps to Reproduce:
1. Go on console. 2. Go to Cluster Settings.
Actual results:
Oh no! Something went wrong
Expected results:
Cluster settings should be visible.
Additional info:
Description of problem:
When the master MCP is paused below alert are triggered Failed to resync 4.12.35 because: Required MachineConfigPool 'master' is paused The node have been rebooted to make sure there is no pending MC rollout
Affects version
4.12
How reproducible:
Steps to Reproduce:
1. Create a MC and apply it to master 2. use below mc apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master name: 99-master-cgroupsv2 spec: kernelArguments: - systemd.unified_cgroup_hierarchy=1 3.Wait until the nodes are rebooted and running 4. pause the mcp Actual results:{code:none} MCP pausing causing the alert
Expected results:
Alerts should not be fired Additional info:{code:none}
Description of problem: I'm trying to import this function https://github.com/pierDipi/node-func-logger using the import function UI flow
Version-Release number of selected component (if applicable): 4.14 OCP and Serverless (current development branch)
How reproducible: Always
Steps to Reproduce:
1. Import this function https://github.com/pierDipi/node-func-logger using the import function UI flow
2. Click create
Actual results: An error occurred Cannot read properties of undefined (reading 'filter')
UI error image: https://drive.google.com/file/d/1GrhX2LUNSzvVuhUmeFYEeZwZ1X58LBAB/view?usp=drive_link
Expected results: No errors
Additional info: As noted above I'm using Serverless development branch, I'm not sure if it's reproducible with a released Serverless release, however, either way we would need to fix it
Description of problem:
with user defined tags, sometimes "create cluster" got panic
Version-Release number of selected component (if applicable):
4.18.0-rc.9 for example
How reproducible:
Sometimes (Easy to reproduce in PROW CI, at least today)
Steps to Reproduce:
1. "create install-config", and then insert interested settings (see [1]) 2. activate the IAM service account which has the required permissions 3. (optional)"create manifests" 4. "create cluster"
Actual results:
Sometimes "create manifests" or "create cluster" got panic (see [2]).
Expected results:
The installation should either succeed, or tell clear error messages. In any case, there should be no panic.
Additional info:
The panic looks being caused by either PROW System flake or GCP flake, because of below reasons: (1) We tried manually installation locally, of 4.18.0-0.nightly-multi-2025-02-17-042334 and 4.17.0-0.nightly-multi-2025-02-15-095503, both succeeded. (2) As for PROW CI tests, both with 4.18.0-rc.9, the Feb. 14's installation succeeded, but today's installation got the panic issue (see [3]). FYI the PROW CI debug PR: https://github.com/openshift/release/pull/61698
Description of problem:
console is showing view release notes on several places, but the current link only point to Y release main release note
Version-Release number of selected component (if applicable):
4.17.2
How reproducible:
Always
Steps to Reproduce:
1. set up 4.17.2 cluster 2. navigate to Cluster Settings page, check 'View release note' link in 'Update history' table
Actual results:
the link only point user to Y release main release note
Expected results:
the link should point to release note of a specific version the correct link should be https://access.redhat.com/documentation/en-us/openshift_container_platform/${major}.${minor}/html/release_notes/ocp-${major}-${minor}-release-notes#ocp-${major}-${minor}-${patch}_release_notes
Additional info:
Description of problem:
The resource-controller endpoint override is not honored in all parts of the machine API provider for power vs.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
The temp folder for extracting the ISO is not cleared in some cases.
Version-Release number of selected component (if applicable):
How reproducible:
Run the installer
Steps to Reproduce:
1. 2. 3.
Actual results:
ls -ld /tmp/agent\* 06 drwx------. 6 root root 4096 Sep 3 18:25 /tmp/agent1486745391 06 drwx------. 6 root root 4096 Sep 3 19:11 /tmp/agent2382395525 06 drwx------. 6 root root 4096 Sep 3 16:21 /tmp/agent2495758451 06 drwx------. 6 root root 4096 Sep 3 18:44 /tmp/agent2810534235 06 drwx------. 6 root root 4096 Sep 3 16:41 /tmp/agent2862979295 06 drwx------. 6 root root 4096 Sep 3 17:31 /tmp/agent2935357941 06 drwx------. 6 root root 4096 Sep 3 17:00 /tmp/agent2952470601 06 drwx------. 6 root root 4096 Sep 3 17:12 /tmp/agent4019363474 06 drwx------. 6 root root 4096 Sep 3 19:03 /tmp/agent589005812
Expected results:
Temp folder to be removed once the iso is created.
Additional info:
Description of problem:
Multi-NetworkPolicy when defined with protocol defined and port undefined, we expect it to match all ports with defined protocol (as per documentation). But policy is not being applied and it allows all traffic. Error Message in multus-networkpolicy logs: E1127 12:12:22.098844 1 server.go:661] sync rules failed for pod [policy-ns1/pod1]: exit status 2: iptables-restore v1.8.10 (nf_tables): invalid port/service `<nil>' specified Error occurred at line: 30 https://docs.openshift.com/container-platform/4.17/rest_api/network_apis/multinetworkpolicy-k8s-cni-cncf-io-v1beta1.html#spec-egress-ports-2
Version-Release number of selected component (if applicable):
4.18.ec2
How reproducible:
--> Apply below policy. ports array should have only protocol defined but not port. apiVersion: k8s.cni.cncf.io/v1beta1 kind: MultiNetworkPolicy metadata: annotations: k8s.v1.cni.cncf.io/policy-for: policy-test-ns1/bond-nad,policy-test-ns2/bond-nad name: egress-port namespace: policy-test-ns1 spec: podSelector: matchLabels: app: pod1 policyTypes: - Egress egress: - ports: - protocol: TCP
Steps to Reproduce:
1. Create SRIOV VFs, bond NAD and create pods that attach to bond NAD 2. Apply MultiNetworkPolicy as mentioned above. 3. Test egress traffic.
Actual results:
Egress works as if no policy is applied.
Expected results:
Egress should work only for TCP protocol to all ports
Additional info:
Must gather : https://drive.google.com/drive/folders/1Le1PdIGiOt965Hqr2xTUXyeDAUGhYYiN?usp=sharing
Description of problem:
When we enable OCL in the master and the worker pool at the same time, it may happen that one of the MOSB resources are not updated with the osImage value.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2025-01-21-015441
How reproducible:
Intermittent
Steps to Reproduce:
1. Enable OCL at the same time in worker and master pool (or with a couple of seconds between creating the MOSC resources)
Actual results:
One of the MOSB resources may not be updated with the osImage and the image will never be applied.
Expected results:
The image should be built and applied without problems
Additional info:
More information in this slack conversation: https://redhat-internal.slack.com/archives/GH7G2MANS/p1737652042188709 This scenario may look like unlikely but we need to take into account that, when we upgrade clusters with OCL enabled in worker and master pools, both pools will always start updating at the same time.
Description of problem:
When I use the UI to check the status of a primary UDN, it does not correctly report the value of the lifecycle attribute.
Version-Release number of selected component (if applicable):
4.99
How reproducible:
Always
Steps to Reproduce:
1. create a project 2. create a UDN in said project using the CLI 3. in the UI, access the newly created UDN
Actual results:
The UDN ipam.lifecycle attribute will *not* be properly presented to the user - it'll say "Not available".
Expected results:
The UDN ipam.lifecycle attribute is presented to the user with the correct value.
Additional info:
The API changed recently, and while before the attribute was present in UDN.spec.layer2.ipamLifecycle, it now is present on UDN.spec.layer2.ipam.lifecycle
In payloads 4.18.0-0.ci-2024-11-01-110334 and 4.18.0-0.nightly-2024-11-01-101707 we observed GCP install failures
Container test exited with code 3, reason Error --- ails: level=error msg=[ level=error msg= { level=error msg= "@type": "type.googleapis.com/google.rpc.ErrorInfo", level=error msg= "domain": "googleapis.com", level=error msg= "metadatas": { level=error msg= "consumer": "projects/711936183532", level=error msg= "quota_limit": "ListRequestsFilterCostOverheadPerMinutePerProject", level=error msg= "quota_limit_value": "75", level=error msg= "quota_location": "global", level=error msg= "quota_metric": "compute.googleapis.com/filtered_list_cost_overhead", level=error msg= "service": "compute.googleapis.com" level=error msg= }, level=error msg= "reason": "RATE_LIMIT_EXCEEDED" level=error msg= }, level=error msg= { level=error msg= "@type": "type.googleapis.com/google.rpc.Help", level=error msg= "links": [ level=error msg= { level=error msg= "description": "The request exceeds API Quota limit, please see help link for suggestions.", level=error msg= "url": "https://cloud.google.com/compute/docs/api/best-practices#client-side-filter" level=error msg= } level=error msg= ] level=error msg= } level=error msg=] level=error msg=, rateLimitExceeded
Patrick Dillon Noted ListRequestsFilterCostOverheadPerMinutePerProject can not have it's quota limit increased.
The problem subsided over the weekend presumably with fewer jobs run but has started to appear again. opening to track ongoing issue and potential work arounds.
This contributes to the following test failures for GCP
install should succeed: configuration install should succeed: overall
The following test is failing with the updated 1.32 Kubernetes in OCP 4.19:
[It] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: (delete policy)] volumegroupsnapshottable [Feature:volumegroupsnapshot] VolumeGroupSnapshottable should create snapshots for multiple volumes in a pod
Since the VolumeGroupSnapshot feature is disabled by default but will become GA in 4.19, the test has been disabled temporarily to unblock the Kubernetes 1.32 rebase.
This ticket tracks the re-enabling of the test once the feature is GA and enabled by default.
Description of problem:
Home - Search : 'Label' is in English
Version-Release number of selected component (if applicable):
4.18.0-rc.6
How reproducible:
always
Steps to Reproduce:
1. Change ocp web console UI into non en_US locale 2. Navigate to Home - Search 3. 'Label' drop down menu name is in English.
Actual results:
Content is in English.
Expected results:
Content should be in selected language.
Additional info:
Reference screenshot added
EnsureLimitedEgressTrafficToManagementKAS frequently flakes on TestUpgradeControlPlane ever since https://github.com/openshift/hypershift/pull/5168
Description of problem:
Currently both the nodepool controller and capi controller set the updatingConfig condition on nodepool upgrades. We should only use one to set the condition to avoid constant switching between conditions and to ensure the logic used for setting this condition is the same.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
CAPI and Nodepool controller set a different status because their logic is not consistent.
Expected results:
CAPI and Nodepool controller set the same status because their logic is not cosolidated.
Additional info:
Description of problem:
When we upgrade an OCL cluster from 4.18 -> 4.18, and we configure a machineset so that its base cloud Image is automatically updated in the upgrade, the machine-config CO may become degraded with this message: - lastTransitionTime: "2025-01-24T19:58:23Z" message: 'Unable to apply 4.18.0-0.nightly-2025-01-24-014549: bootimage update failed: 1 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments | Error(s): error syncing MAPI MachineSet cloned-tc-70813-label: timed out waiting for coreos-bootimages config map: mismatch between MCO hash version stored in configmap and current MCO version; sync will exit to wait for the MCO upgrade to complete' reason: MachineConfigurationFailed status: "True" type: Degraded Link to the prow execution: https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-4.18-upgrade-from-stable-4.18-aws-ipi-ocl-fips-f60/1882788481910968320
Version-Release number of selected component (if applicable):
Upgrading from 4.18.0-rc.6 to 4.18.0-0.nightly-2025-01-24-014549
How reproducible:
Rarely
Steps to Reproduce:
1. install 4.18.0-rc.6 2. Clone an existing machineset 3. Configure the new machineset so that its base cloud image is updated automatically in the upgraded. Use a label configuration, so that only this machineset is updated. 4. Upgrade to 4.18.0-0.nightly-2025-01-24-014549
Actual results:
The machine-config CO is degraded with this message: - lastTransitionTime: "2025-01-24T19:58:23Z" message: 'Unable to apply 4.18.0-0.nightly-2025-01-24-014549: bootimage update failed: 1 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments | Error(s): error syncing MAPI MachineSet cloned-tc-70813-label: timed out waiting for coreos-bootimages config map: mismatch between MCO hash version stored in configmap and current MCO version; sync will exit to wait for the MCO upgrade to complete' reason: MachineConfigurationFailed status: "True" type: Degraded
Expected results:
Additional info:
It looks like the coreos-bootimages configma was never updated with the new MCOVersionHash It may not be related to OCL at all.
Description of problem:
No zone for master machines
Version-Release number of selected component (if applicable):
4.17.0-0.nightly-2024-08-15-153405
How reproducible:
Always
Steps to Reproduce:
1. Install an azure cluster 2. Run "oc get machine" 3.
Actual results:
No zone info for master machine $ oc get machine NAME PHASE TYPE REGION ZONE AGE yingwang-0816-tvqdc-master-0 Running Standard_D8s_v3 eastus 104m yingwang-0816-tvqdc-master-1 Running Standard_D8s_v3 eastus 104m yingwang-0816-tvqdc-master-2 Running Standard_D8s_v3 eastus 104m yingwang-0816-tvqdc-worker-eastus1-54ckq Running Standard_D4s_v3 eastus 1 96m yingwang-0816-tvqdc-worker-eastus2-dwr2j Running Standard_D4s_v3 eastus 2 96m yingwang-0816-tvqdc-worker-eastus3-7wchl Running Standard_D4s_v3 eastus 3 96m $ oc get machine --show-labels NAME PHASE TYPE REGION ZONE AGE LABELS yingwang-0816-tvqdc-master-0 Running Standard_D8s_v3 eastus 104m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=master,machine.openshift.io/cluster-api-machine-type=master,machine.openshift.io/instance-type=Standard_D8s_v3,machine.openshift.io/region=eastus yingwang-0816-tvqdc-master-1 Running Standard_D8s_v3 eastus 104m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=master,machine.openshift.io/cluster-api-machine-type=master,machine.openshift.io/instance-type=Standard_D8s_v3,machine.openshift.io/region=eastus yingwang-0816-tvqdc-master-2 Running Standard_D8s_v3 eastus 104m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=master,machine.openshift.io/cluster-api-machine-type=master,machine.openshift.io/instance-type=Standard_D8s_v3,machine.openshift.io/region=eastus yingwang-0816-tvqdc-worker-eastus1-54ckq Running Standard_D4s_v3 eastus 1 96m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=worker,machine.openshift.io/cluster-api-machine-type=worker,machine.openshift.io/cluster-api-machineset=yingwang-0816-tvqdc-worker-eastus1,machine.openshift.io/instance-type=Standard_D4s_v3,machine.openshift.io/interruptible-instance=,machine.openshift.io/region=eastus,machine.openshift.io/zone=1 yingwang-0816-tvqdc-worker-eastus2-dwr2j Running Standard_D4s_v3 eastus 2 96m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=worker,machine.openshift.io/cluster-api-machine-type=worker,machine.openshift.io/cluster-api-machineset=yingwang-0816-tvqdc-worker-eastus2,machine.openshift.io/instance-type=Standard_D4s_v3,machine.openshift.io/interruptible-instance=,machine.openshift.io/region=eastus,machine.openshift.io/zone=2 yingwang-0816-tvqdc-worker-eastus3-7wchl Running Standard_D4s_v3 eastus 3 96m machine.openshift.io/cluster-api-cluster=yingwang-0816-tvqdc,machine.openshift.io/cluster-api-machine-role=worker,machine.openshift.io/cluster-api-machine-type=worker,machine.openshift.io/cluster-api-machineset=yingwang-0816-tvqdc-worker-eastus3,machine.openshift.io/instance-type=Standard_D4s_v3,machine.openshift.io/interruptible-instance=,machine.openshift.io/region=eastus,machine.openshift.io/zone=3
Expected results:
Zone info can be shown when run "oc get machine"
Additional info:
Description of problem:
When using PublicIPv4Pool, CAPA will try to allocate IP address in the supplied pool which requires the `ec2:AllocateAddress` permission
Version-Release number of selected component (if applicable):
4.16+
How reproducible:
always
Steps to Reproduce:
1. Minimal permissions and publicIpv4Pool set 2. 3.
Actual results:
time="2024-11-21T05:39:49Z" level=debug msg="E1121 05:39:49.352606 327 awscluster_controller.go:279] \"failed to reconcile load balancer\" err=<" time="2024-11-21T05:39:49Z" level=debug msg="\tfailed to allocate addresses to load balancer: failed to allocate address from Public IPv4 Pool \"ipv4pool-ec2-0768267342e327ea9\" to role lb-apiserver: failed to allocate Elastic IP for \"lb-apiserver\": UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:iam::460538899914:user/ci-op-2cr41ill-663fd-minimal-perm is not authorized to perform: ec2:AllocateAddress on resource: arn:aws:ec2:us-east-1:460538899914:ipv4pool-ec2/ipv4pool-ec2-0768267342e327ea9 because no identity-based policy allows the ec2:AllocateAddress action. Encoded authorization failure message: Iy1gCtvfPxZ2uqo-SHei1yJQvNwaOBl5F_8BnfeEYCLMczeDJDdS4fZ_AesPLdEQgK7ahuOffqIr--PWphjOUbL2BXKZSBFhn3iN9tZrDCnQQPKZxf9WaQmSkoGNWKNUGn6rvEZS5KvlHV5vf5mCz5Bk2lk3w-O6bfHK0q_dphLpJjU-sTGvB6bWAinukxSYZ3xbirOzxfkRfCFdr7nDfX8G4uD4ncA7_D-XriDvaIyvevWSnus5AI5RIlrCuFGsr1_3yEvrC_AsLENZHyE13fA83F5-Abpm6-jwKQ5vvK1WuD3sqpT5gfTxccEqkqqZycQl6nsxSDP2vDqFyFGKLAmPne8RBRbEV-TOdDJphaJtesf6mMPtyMquBKI769GW9zTYE7nQzSYUoiBOafxz6K1FiYFoc1y6v6YoosxT8bcSFT3gWZWNh2upRJtagRI_9IRyj7MpyiXJfcqQXZzXkAfqV4nsJP8wRXS2vWvtjOm0i7C82P0ys3RVkQVcSByTW6yFyxh8Scoy0HA4hTYKFrCAWA1N0SROJsS1sbfctpykdCntmp9M_gd7YkSN882Fy5FanA" time="2024-11-21T05:39:49Z" level=debug msg="\t\tstatus code: 403, request id: 27752e3c-596e-43f7-8044-72246dbca486"
Expected results:
Additional info:
Seems to happen consistently with shared-vpc-edge-zones CI job: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_installer/9230/pull-ci-openshift-installer-master-e2e-aws-ovn-shared-vpc-edge-zones/1860015198224519168
they conflict with our owns and they produce different builds between CI and production actually making the tests completely unreliable
Description of the problem:
From our CI, reproduced several time lately - trying to install 4.17 +ODF + CNV.
Getting those messages after 40 minutes :
Operator odf status: progressing message: installing: waiting for deployment odf-operator-controller-manager to become ready: deployment "odf-operator-controller-manager" not available: Deployment does not have minimum availability.
"Operator odf status: progressing message: installing: waiting for deployment odf-console to become ready: deployment "odf-console" not available: Deployment does not have minimum availability."
CI job waits for cluster to complete for 2.5h. Cluster link - https://console.dev.redhat.com/openshift/assisted-installer/clusters/74b62ea4-61ce-4fde-acbe-cc1cf41f1fb8 Attached the installation logs and a video of installation.
How reproducible:
Still checking
Steps to reproduce:
1.
2.
3.
Actual results:
Expected results:
Description of problem:
The status controller creates a ClusterOperator when one does not exist. In the test case verifying behavior with an already present ClusterOperator, it is requried to wait until the ClusterOperator created by the test is ready. Failing to do so can result in the controller attempting to create a duplicate ClusterOperator, causing the test to fail with an "already exists" error.
Version-Release number of selected component (if applicable):
How reproducible:
Sometimes, race condition
Steps to Reproduce:
1. Run ci/prow/unit presubmit job
Actual results:
Test fails with: clusteroperators.config.openshift.io \"machine-approver\" already exists",
Expected results:
Test passes
Additional info:
Unit test only issue. No custommer impact.
After adding new CRs we should update https://github.com/openshift/cluster-baremetal-operator/blob/main/manifests/0000_31_cluster-baremetal-operator_07_clusteroperator.cr.yaml
Description of problem:
As reported in https://issues.redhat.com/browse/OCPBUGS-52256, there are memory issues at scale and the use of a development logger setting is contributing to the issue. Production code should not be using the development logger settings.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Bring the downstream operator-controller repo up-to-date with the v0.7.0 upstream release.
Summary: OLMv0 should avoid setting Pod Security Admission (PSA) labels with a fixed value. Instead, it should pin the PSA version to match the Kubernetes API version specified in the go.mod file as agreed and discussed at: https://redhat-internal.slack.com/archives/C06KP34REFJ/p1739880491760029
Thit Jira is orginated from: https://issues.redhat.com/browse/OCPBUGS-42526
As a developer looking to contribute to OCP BuildConfig I want contribution guidelines that make it easy for me to build and test all the components.
Much of the contributor documentation for openshift/builder is either extremely out of date or buggy. This hinders the ability for newcomers to contribute.
We should expand upon our current pre-commit hooks:
This will help prevent errors before code makes it on GitHub and CI.
Description of problem:
dynamic plugin in Pending status will block console plugins tab page loading
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-11-27-162407
How reproducible:
Always
Steps to Reproduce:
1. Create a dynamic plugin which will be in Pending status, we can create from file https://github.com/openshift/openshift-tests-private/blob/master/frontend/fixtures/plugin/pending-console-demo-plugin-1.yaml 2. Enable the 'console-demo-plugin-1' plugin and navigate to Console plugins tab at /k8s/cluster/operator.openshift.io~v1~Console/cluster/console-plugins
Actual results:
2. page will always be loading
Expected results:
2. console plugins list table should be displayed
Additional info:
Description of problem:
Tracking per-operator fixes for the following related issues static pod node, installer, and revision controllers: https://issues.redhat.com/browse/OCPBUGS-45924 https://issues.redhat.com/browse/OCPBUGS-46372 https://issues.redhat.com/browse/OCPBUGS-48276
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
We had bugs like https://issues.redhat.com/browse/OCPBUGS-44324 from the payload tests in vsphere and gcp, and this was fixed by https://github.com/openshift/api/commit/ec9bf3faa1aa2f52805c44b7b13cd7ab4b984241 There are a few operators which are missing that openshift/api bump. These operators do not have blocking payload jobs but we still need this fix before 4.18 is released. It affects the following operators: https://github.com/openshift/ibm-vpc-block-csi-driver-operator/blob/master/go.mod#L11 https://github.com/openshift/ibm-powervs-block-csi-driver-operator/blob/main/go.mod#L6 https://github.com/openshift/gcp-filestore-csi-driver-operator/blob/main/go.mod#L8 https://github.com/openshift/secrets-store-csi-driver-operator/blob/main/go.mod#L8
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
All but 4 csi driver operators have the fix
Expected results:
All csi driver operators have this fix vendored: https://github.com/openshift/api/commit/ec9bf3faa1aa2f52805c44b7b13cd7ab4b984241
Additional info:
Description of problem:
When deploying with an Internal publishing strategy, it is required that you specify a pre existing VPC via 'platform.powervs.vpcName'. Currently, if you don't, the installer will fail and say "VPC not found", when a more accurate error would be "A pre-existing VPC is required when deploying with an Internal publishing strategy".
Version-Release number of selected component (if applicable):
How reproducible:
Easily
Steps to Reproduce:
1. Set strategy to Internal, do not specify vpcName 2. Deploy 3. Observe error
Actual results:
Confusing error
Expected results:
Accurate error
Additional info:
Description of problem:
once https://github.com/openshift/kubernetes/pull/2055 merges, the kubelet will excessively print the warning "WARNING: the kernel version is incompatible with the feature gate, which needs as a minimum kernel version" kernelVersion="5.14.0" feature="UserNamespacesSupport" minKernelVersion="6.3" when the cluster is in tech preview
Version-Release number of selected component (if applicable):
4.18.0
How reproducible:
100%
Steps to Reproduce:
1. launch openshift 2. look at node logs 3.
Actual results:
"WARNING: the kernel version is incompatible with the feature gate, which needs as a minimum kernel version" kernelVersion="5.14.0" feature="UserNamespacesSupport" minKernelVersion="6.3" warning appears
Expected results:
"WARNING: the kernel version is incompatible with the feature gate, which needs as a minimum kernel version" warning only appears when the node actually doesn't support idmapped mounts (RHEL 8 or RHEL 9.2)
Additional info:
Description of problem:
[Azure disk/file csi driver]on ARO HCP could not provision volume succeed
Version-Release number of selected component (if applicable):
4.19.0-0.nightly-2024-12-13-083421
How reproducible:
Always
Steps to Reproduce:
1.Install AKS cluster on azure. 2.Install hypershift operator on the AKS cluster. 3.Use hypershift CLI create hosted cluster with the Client Certificate mode. 4.Check the azure disk/file csi dirver work well on hosted cluster.
Actual results:
In step 4: the the azure disk/file csi dirver provision volume failed on hosted cluster # azure disk pvc provision failed $ oc describe pvc mypvc ... Normal WaitForFirstConsumer 74m persistentvolume-controller waiting for first consumer to be created before binding Normal Provisioning 74m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_2334468f-9d27-4bdd-a53c-27271ee60073 External provisioner is provisioning volume for claim "default/mypvc" Warning ProvisioningFailed 74m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_2334468f-9d27-4bdd-a53c-27271ee60073 failed to provision volume with StorageClass "managed-csi": rpc error: code = Unavailable desc = error reading from server: EOF Warning ProvisioningFailed 71m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_28ba5ad9-c4f8-4dc8-be40-c80c546b7ef8 failed to provision volume with StorageClass "managed-csi": rpc error: code = Unavailable desc = error reading from server: EOF Normal Provisioning 71m disk.csi.azure.com_azure-disk-csi-driver-controller-74d944bbcb-7zz89_28ba5ad9-c4f8-4dc8-be40-c80c546b7ef8 External provisioner is provisioning volume for claim "default/mypvc" ... $ oc logs azure-disk-csi-driver-controller-74d944bbcb-7zz89 -c csi-driver W1216 08:07:04.282922 1 main.go:89] nodeid is empty I1216 08:07:04.290689 1 main.go:165] set up prometheus server on 127.0.0.1:8201 I1216 08:07:04.291073 1 azuredisk.go:213] DRIVER INFORMATION: ------------------- Build Date: "2024-12-13T02:45:35Z" Compiler: gc Driver Name: disk.csi.azure.com Driver Version: v1.29.11 Git Commit: 4d21ae15d668d802ed5a35068b724f2e12f47d5c Go Version: go1.23.2 (Red Hat 1.23.2-1.el9) X:strictfipsruntime Platform: linux/amd64 Topology Key: topology.disk.csi.azure.com/zone I1216 08:09:36.814776 1 utils.go:77] GRPC call: /csi.v1.Controller/CreateVolume I1216 08:09:36.814803 1 utils.go:78] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":""}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316","parameters":{"csi.storage.k8s.io/pv/name":"pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316","csi.storage.k8s.io/pvc/name":"mypvc","csi.storage.k8s.io/pvc/namespace":"default","skuname":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}}]} I1216 08:09:36.815338 1 controllerserver.go:208] begin to create azure disk(pvc-d6af3900-ec5b-4e09-83d6-d0e112b02316) account type(Premium_LRS) rg(ci-op-zj9zc4gd-12c20-rg) location(centralus) size(1) diskZone() maxShares(0) panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x190c61d] goroutine 153 [running]: sigs.k8s.io/cloud-provider-azure/pkg/provider.(*ManagedDiskController).CreateManagedDisk(0x0, {0x2265cf0, 0xc0001285a0}, 0xc0003f2640) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/sigs.k8s.io/cloud-provider-azure/pkg/provider/azure_managedDiskController.go:127 +0x39d sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*Driver).CreateVolume(0xc000564540, {0x2265cf0, 0xc0001285a0}, 0xc000272460) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/azuredisk/controllerserver.go:297 +0x2c59 github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1({0x2265cf0?, 0xc0001285a0?}, {0x1e5a260?, 0xc000272460?}) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6420 +0xcb sigs.k8s.io/azuredisk-csi-driver/pkg/csi-common.logGRPC({0x2265cf0, 0xc0001285a0}, {0x1e5a260, 0xc000272460}, 0xc00017cb80, 0xc00014ea68) /go/src/github.com/openshift/azure-disk-csi-driver/pkg/csi-common/utils.go:80 +0x409 github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x1f3e440, 0xc000564540}, {0x2265cf0, 0xc0001285a0}, 0xc00029a700, 0x2084458) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go:6422 +0x143 google.golang.org/grpc.(*Server).processUnaryRPC(0xc00059cc00, {0x2265cf0, 0xc000128510}, {0x2270d60, 0xc0004f5980}, 0xc000308480, 0xc000226a20, 0x31c8f80, 0x0) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1379 +0xdf8 google.golang.org/grpc.(*Server).handleStream(0xc00059cc00, {0x2270d60, 0xc0004f5980}, 0xc000308480) /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1790 +0xe8b google.golang.org/grpc.(*Server).serveStreams.func2.1() /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1029 +0x7f created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 16 /go/src/github.com/openshift/azure-disk-csi-driver/vendor/google.golang.org/grpc/server.go:1040 +0x125 # azure file pvc provision failed $ oc describe pvc mypvc Name: mypvc Namespace: openshift-cluster-csi-drivers StorageClass: azurefile-csi Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-provisioner: file.csi.azure.com volume.kubernetes.io/storage-provisioner: file.csi.azure.com Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 14s (x2 over 14s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. Normal Provisioning 7s (x4 over 14s) file.csi.azure.com_azure-file-csi-driver-controller-879f56577-5hjn8_38c8218e-e52c-4248-ada7-268742afaac0 External provisioner is provisioning volume for claim "openshift-cluster-csi-drivers/mypvc" Warning ProvisioningFailed 7s (x4 over 14s) file.csi.azure.com_azure-file-csi-driver-controller-879f56577-5hjn8_38c8218e-e52c-4248-ada7-268742afaac0 failed to provision volume with StorageClass "azurefile-csi": rpc error: code = Internal desc = failed to ensure storage account: could not list storage accounts for account type Standard_LRS: StorageAccountClient is nil
Expected results:
In step 4: the the azure disk/file csi dirver should provision volume succeed on hosted cluster
Additional info:
Description of problem:
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.19.0-0.nightly-2025-02-07-024732 True False 133m Cluster version is 4.19.0-0.nightly-2025-02-07-024732 $ oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 | grep "Error on ingesting results from rule evaluation with different value but same timestamp" ts=2025-02-07T06:50:32.371Z caller=group.go:599 level=warn name=sum:apiserver_request:burnrate5m index=11 component="rule manager" file=/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4c40cd93-505e-4e93-a53c-fdbb47f77d9d.yaml group=kube-apiserver.rules msg="Error on ingesting results from rule evaluation with different value but same timestamp" num_dropped=1 ts=2025-02-07T06:51:02.376Z caller=group.go:599 level=warn name=sum:apiserver_request:burnrate5m index=11 component="rule manager" file=/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-kube-apiserver-kube-apiserver-slos-basic-4c40cd93-505e-4e93-a53c-fdbb47f77d9d.yaml group=kube-apiserver.rules msg="Error on ingesting results from rule evaluation with different value but same timestamp" num_dropped=1 ....
checked, there are 2 "sum:apiserver_request:burnrate5m" recording rule for 4.19, the second one should be "sum:apiserver_request:burnrate6h", not "sum:apiserver_request:burnrate5m"
$ oc -n openshift-kube-apiserver get prometheusrules kube-apiserver-slos-basic -oyaml ... - expr: | sum(apiserver_request:burn5m) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET|POST|PUT|PATCH|DELETE"}[5m])) record: sum:apiserver_request:burnrate5m ... - expr: | sum(apiserver_request:burn6h) / sum by (cluster) (rate(apiserver_request_total{job="apiserver",verb=~"LIST|GET|POST|PUT|PATCH|DELETE"}[6h])) record: sum:apiserver_request:burnrate5m
Version-Release number of selected component (if applicable):
4.19+
How reproducible:
always
Steps to Reproduce:
1. see the descriptions
Actual results:
2 "sum:apiserver_request:burnrate5m" recording rule for 4.19
Expected results:
only one "sum:apiserver_request:burnrate5m" recording rule for 4.19
Additional info:
issue is only with 4.19
Description of problem:
"Match Labels" is on the same level of 'Projects" while creating CUDN
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
When running the delete command on oc-mirror after a mirrorToMirror, the graph-image is not being deleted.
Version-Release number of selected component (if applicable):
How reproducible:
With the following ImageSetConfiguration (use the same for the DeleteImageSetConfiguration only changing the kind and the mirror to delete)
kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: channels: - name: stable-4.13 minVersion: 4.13.10 maxVersion: 4.13.10 graph: true
Steps to Reproduce:
1. Run mirror to mirror ./bin/oc-mirror -c ./alex-tests/alex-isc/isc.yaml --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-230 docker://localhost:6000 --v2 --dest-tls-verify=false 2. Run the delete --generate ./bin/oc-mirror delete -c ./alex-tests/alex-isc/isc-delete.yaml --generate --workspace file:///home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-230 --delete-id clid-230-delete-test docker://localhost:6000 --v2 --dest-tls-verify=false 3. Run the delete ./bin/oc-mirror delete --delete-yaml-file /home/aguidi/go/src/github.com/aguidirh/oc-mirror/alex-tests/clid-230/working-dir/delete/delete-images-clid-230-delete-test.yaml docker://localhost:6000 --v2 --dest-tls-verify=false
Actual results:
During the delete --generate the graph-image is not being included in the delete file 2024/10/25 09:44:21 [WARN] : unable to find graph image in local cache: SKIPPING. %!v(MISSING) 2024/10/25 09:44:21 [WARN] : reading manifest latest in localhost:55000/openshift/graph-image: manifest unknown Because of that the graph-image is not being deleted from the target registry [aguidi@fedora oc-mirror]$ curl http://localhost:6000/v2/openshift/graph-image/tags/list | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 51 100 51 0 0 15577 0 --:--:-- --:--:-- --:--:-- 17000 { "name": "openshift/graph-image", "tags": [ "latest" ] }
Expected results:
graph-image should be deleted even after mirrorToMirror
Additional info:
Description of problem:
Clicking "Don't show again" won't spot "Hide Lightspeed" if current page is on Language/Notifications/Applications tab of "user-preferences"
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-16-065305
How reproducible:
Always
Steps to Reproduce:
1.User goes to one of Language/Notifications/Applications tabs on "user-preferences" page. 2.Open "Lightspeed" modal at the right bottom and click "Don't show again". 3.
Actual results:
2. The url changes to "/user-preferences/general?spotlight=[data-test="console.hideLightspeedButton%20field"]", but still stays at the original tab.
Expected results:
2. Should jump to "Hide Lightspeed" part on General tab of "user-preferences" page.
Additional info:
During the cluster bootstrap, disruption can occur when a kube-apiserver instance doesn't have access to any live etcd endpoints. This happens in one very specific scenario:
When both of these scenarios happens whilst a kube-apiserver instance is still on revision 1, its readyz probe will fail
The suggested solution to fix this issue is to add a check in cluster-bootstrap that makes sure that we have at least 2 etcd-servers that are not bootstrap and localhost for each kube-apiserver pods before getting rid of the bootstrap node.
Job where this is happening: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-etcd-operator/1387/pull-ci-openshift-cluster-etcd-operator-master-e2e-aws-ovn-serial/1880358740390055936
Description of problem:
How reproducible:
Steps to Reproduce:
Actual results:
Expected results:
As a maintainer of the SNO CI lane, I would like to ensure that the following test doesn't failure regularly as part of SNO CI.
[sig-architecture] platform pods in ns/openshift-e2e-loki should not exit an excessive amount of times
This issue is a symptom of a greater problem with SNO where there is downtime in resolving DNS after the upgrade reboot where the DNS operator has an outage while its deploying the new DNS pods. During that time, loki exists after hitting the following error:
2024/10/23 07:21:32 OIDC provider initialization failed: Get "https://sso.redhat.com/auth/realms/redhat-external/.well-known/openid-configuration": dial tcp: lookup sso.redhat.com on 172.30.0.10:53: read udp 10.128.0.4:53104->172.30.0.10:53: read: connection refused
This issue is important because it can contribute to payload rejection in our blocking CI jobs.
Acceptance Criteria:
The monitoring documentation is going through the restructuring. For this reason, the console links need to be updated to reflect that change.
Link for more information: https://issues.redhat.com/browse/OBSDOCS-1673
Description of problem:
SetUp: Set browser's(e.g. google-chrome) default language as French or Spanish Issue: Goto console login page, content is in English, not in French or Spanish.
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-06-01-063526
How reproducible:
Always
Steps to Reproduce:
1. Set Browser's(e.g. Google-chrome) default language as French or Spanish 2. Goto console login page, content is in English, not in French or Spanish. 3. Same setup is working fine with rest of the supported locales.
Actual results:
Login page is in English
Expected results:
Login page should be available in browser's default language for supported OCP languages.
Additional info:
Reference screencast attached
The story is to track i18n upload/download routine tasks which are perform every sprint.
A.C.
- Upload strings to Memosource at the start of the sprint and reach out to localization team
- Download translated strings from Memsource when it is ready
- Review the translated strings and open a pull request
- Open a followup story for next sprint
This is a recurrence task that needs to be performed from time to time to keep the dependencies updated.
Beginning with 4.19.0-0.nightly-2024-11-27-025041 this job failed with a pattern I don't recognize.
I'll note some other aws jobs failed on the same payload which looked like infra issues; however this test re-ran in full and so its timing was very different.
Then it failed with much the same pattern on the next payload too.
The failures are mainly on tests like these:
[sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set initially, in a homogeneous default environment, should expose default metrics [Suite:openshift/conformance/parallel] expand_more [sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set in a heterogeneous environment, should revert to default collection profile when an empty collection profile value is specified [Suite:openshift/conformance/parallel] expand_more [sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set in a heterogeneous environment, should expose information about the applied collection profile using meta-metrics [Suite:openshift/conformance/parallel] expand_more [sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set in a heterogeneous environment, should have at least one implementation for each collection profile [Suite:openshift/conformance/parallel] expand_more [sig-instrumentation][OCPFeatureGate:MetricsCollectionProfiles] The collection profiles feature-set in a homogeneous minimal environment, should hide default metrics [Suite:openshift/conformance/parallel] expand_more
Each has a run where it looks like something timed out:
fail [github.com/openshift/origin/test/extended/prometheus/collection_profiles.go:99]: Interrupted by User Ginkgo exit error 1: exit with code 1
and a second run failing to update configmap cluster-monitoring-config
{ fail [github.com/openshift/origin/test/extended/prometheus/collection_profiles.go:197]: Expected <*errors.StatusError | 0xc006738280>: Operation cannot be fulfilled on configmaps "cluster-monitoring-config": the object has been modified; please apply your changes to the latest version and try again { ErrStatus: code: 409 details: kind: configmaps name: cluster-monitoring-config message: 'Operation cannot be fulfilled on configmaps "cluster-monitoring-config": the object has been modified; please apply your changes to the latest version and try again' metadata: {} reason: Conflict status: Failure, } to be nil Ginkgo exit error 1: exit with code 1}
The parseIPList function currently fails to handle IP lists that contain both valid and invalid IPs or CIDRs. When the function encounters an invalid entry, it immediately returns an empty string, which prevents any valid IPs from being processed or returned.
Modify the parseIPList function to:
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following test:
[sig-arch] events should not repeat pathologically for ns/openshift-machine-api
Extreme regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 100.00% to 20.00%.
Sample (being evaluated) Release: 4.19
Start Time: 2025-02-20T00:00:00Z
End Time: 2025-02-27T08:00:00Z
Success Rate: 20.00%
Successes: 3
Failures: 12
Flakes: 0
Base (historical) Release: 4.18
Start Time: 2025-01-28T00:00:00Z
End Time: 2025-02-27T08:00:00Z
Success Rate: 100.00%
Successes: 60
Failures: 0
Flakes: 0
View the test details report for additional context.
This has been regressed for 5 days with a permafail on gcp serial jobs, techpreview and default both.
[sig-arch] events should not repeat pathologically for ns/openshift-machine-api expand_less 0s { 3 events happened too frequently event happened 24 times, something is wrong: namespace/openshift-machine-api hmsg/30080f8454 machineset/ci-op-qinhsdt8-e68fb-4xk6j-worker-c - reason/ReconcileError error fetching disk information: unable to retrieve image "rhcos-9-6-20250121-0-gcp-x86-64-fake-update" in project "rhcos-cloud": googleapi: Error 403: Required 'compute.images.get' permission for 'projects/rhcos-cloud/global/images/rhcos-9-6-20250121-0-gcp-x86-64-fake-update', forbidden (08:35:32Z) result=reject event happened 25 times, something is wrong: namespace/openshift-machine-api hmsg/30080f8454 machineset/ci-op-qinhsdt8-e68fb-4xk6j-worker-f - reason/ReconcileError error fetching disk information: unable to retrieve image "rhcos-9-6-20250121-0-gcp-x86-64-fake-update" in project "rhcos-cloud": googleapi: Error 403: Required 'compute.images.get' permission for 'projects/rhcos-cloud/global/images/rhcos-9-6-20250121-0-gcp-x86-64-fake-update', forbidden (08:35:43Z) result=reject event happened 30 times, something is wrong: namespace/openshift-machine-api hmsg/30080f8454 machineset/ci-op-qinhsdt8-e68fb-4xk6j-worker-a - reason/ReconcileError error fetching disk information: unable to retrieve image "rhcos-9-6-20250121-0-gcp-x86-64-fake-update" in project "rhcos-cloud": googleapi: Error 403: Required 'compute.images.get' permission for 'projects/rhcos-cloud/global/images/rhcos-9-6-20250121-0-gcp-x86-64-fake-update', forbidden (08:35:08Z) result=reject }
There is a number of various inconsistencies while using the observe section in the virtualization perspective.
Description of problem:
2024-05-07 17:21:59 level=debug msg=baremetal: getting master addresses 2024-05-07 17:21:59 level=warning msg=Failed to extract host addresses: open ocp/ostest/.masters.json: no such file or directory
Description of problem:
HyperShift CEL validation blocks ARM64 NodePool creation for non-AWS/Azure platforms Can't add a Bare Metal worker node to the hosted cluster. This was discussed on #project-hypershift Slack channel.
Version-Release number of selected component (if applicable):
MultiClusterEngine v2.7.2 HyperShift Operator image: registry.redhat.io/multicluster-engine/hypershift-rhel9-operator@sha256:56bd0210fa2a6b9494697dc7e2322952cd3d1500abc9f1f0bbf49964005a7c3a
How reproducible:
Always
Steps to Reproduce:
1. Create a HyperShift HostedCluster on a non-AWS/non-Azure platform 2. Try to create a NodePool with ARM64 architecture specification
Actual results:
- CEL validation blocks creating NodePool with arch: arm64 on non-AWS/Azure platforms - Receive error: "The NodePool is invalid: spec: Invalid value: "object": Setting Arch to arm64 is only supported for AWS and Azure" - Additional validation in NodePool spec also blocks arm64 architecture
Expected results:
- Allow ARM64 architecture specification for NodePools on BareMetal platform - Remove or update the CEL validation to support this use case
Additional info:
NodePool YAML: apiVersion: hypershift.openshift.io/v1beta1 kind: NodePool metadata: name: nodepool-doca5-1 namespace: doca5 spec: arch: arm64 clusterName: doca5 management: autoRepair: false replace: rollingUpdate: maxSurge: 1 maxUnavailable: 0 strategy: RollingUpdate upgradeType: InPlace platform: agent: agentLabelSelector: {} type: Agent release: image: quay.io/openshift-release-dev/ocp-release:4.16.21-multi replicas: 1
Description of problem:
Creating CUDN with mismatch spec.topology and the topology config succeed, but it should failed because its invalid.
See below examples.
Version-Release number of selected component (if applicable):
4.18
How reproducible:
100%
Steps to Reproduce:
1. Create CUDN CR with spec.topology mismatch topology configuration:
Example 1:
apiVersion: k8s.ovn.org/v1 kind: ClusterUserDefinedNetwork metadata: name: mynet spec: namespaceSelector: matchLabels: "kubernetes.io/metadata.name": "red" network: topology: Layer2 # <--- spec.topology should match layer3: # <------------ topology configuration type role: Primary subnets: [{cidr: 192.168.112.12/24}]
Example 2:
apiVersion: k8s.ovn.org/v1 kind: ClusterUserDefinedNetwork metadata: name: mynet spec: namespaceSelector: matchLabels: "kubernetes.io/metadata.name": "red" network: topology: Layer3 # <--- spec.topology should match layer2: # <------------ topology configuration type role: Secondary subnets: [192.168.112.12/24]
Actual results:
The CUDN is created successfully.
ovn-kubernetes control-plane pod get into crush looping due to the following panic:
panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1dd8415] goroutine 12154 [running]: github.com/ovn-org/ovn-kubernetes/go-controller/pkg/util/udn.IsPrimaryNetwork({0x2d8dfd0, 0xc0059746a8}) /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/util/udn/udn.go:17 +0x55 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork.(*Controller).updateNAD(0xc0007340f0, {0x2dcad90, 0xc005974580}, {0xc000012480, 0x3}) /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork/controller_helper.go:24 +0x94 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork.(*Controller).syncClusterUDN(0xc0007340f0, 0xc005974420) /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork/controller.go:604 +0xa10 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork.(*Controller).reconcileCUDN(0xc0007340f0, {0xc00651c2d6, 0x5}) /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/clustermanager/userdefinednetwork/controller.go:519 +0xff github.com/ovn-org/ovn-kubernetes/go-controller/pkg/controller.(*controller[...]).processNextQueueItem(0x19a93e0) /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/controller/controller.go:253 +0xd7 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/controller.(*controller[...]).startWorkers.func1() /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/controller/controller.go:163 +0x6f created by github.com/ovn-org/ovn-kubernetes/go-controller/pkg/controller.(*controller[...]).startWorkers in goroutine 7794 /home/omergi/workspace/github.com/ovn-kubernetes/go-controller/pkg/controller/controller.go:160 +0x185
Expected results:
Creating CUDN with mismatch spec.topology and topology configuration should fail - at the API level.
Additional info:
Please fill in the following template while reporting a bug and provide as much relevant information as possible. Doing so will give us the best chance to find a prompt resolution.
Affected Platforms:
Is it an
If it is an internal RedHat testing failure:
If it is a CI failure:
If it is a customer / SD issue:
Description of problem:
Failed to create a disconnected cluster using HCP/HyperShift CLI
Version-Release number of selected component (if applicable):
4.19 4.18
How reproducible:
100%
Steps to Reproduce:
1. create disconnected hostedcluster with hcp cli 2. The environment where the command is executed cannot access the payload.
Actual results:
/tmp/hcp create cluster agent --cluster-cidr fd03::/48 --service-cidr fd04::/112 --additional-trust-bundle=/tmp/secret/registry.2.crt --network-type=OVNKubernetes --olm-disable-default-sources --name=b2ce1d5218a2c7b561d6 --pull-secret=/tmp/.dockerconfigjson --agent-namespace=hypershift-agents --namespace local-cluster --base-domain=ostest.test.metalkube.org --api-server-address=api.b2ce1d5218a2c7b561d6.ostest.test.metalkube.org --image-content-sources /tmp/secret/mgmt_icsp.yaml --ssh-key=/tmp/secret/id_rsa.pub --release-image virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:7acdfad179f4571cbf211a87bce87749a1576b72f1d57499e6d9be09b0c4d31d1422024-12-31T08:01:05Z ERROR Failed to create cluster {"error": "failed to retrieve manifest virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:7acdfad179f4571cbf211a87bce87749a1576b72f1d57499e6d9be09b0c4d31d: failed to create repository client for https://virthost.ostest.test.metalkube.org:5000: Get \"https://virthost.ostest.test.metalkube.org:5000/v2/\": Internal Server Error"}143github.com/openshift/hypershift/product-cli/cmd/cluster/agent.NewCreateCommand.func1144 /remote-source/app/product-cli/cmd/cluster/agent/create.go:32145github.com/spf13/cobra.(*Command).execute146 /remote-source/app/vendor/github.com/spf13/cobra/command.go:985147github.com/spf13/cobra.(*Command).ExecuteC148 /remote-source/app/vendor/github.com/spf13/cobra/command.go:1117149github.com/spf13/cobra.(*Command).Execute150 /remote-source/app/vendor/github.com/spf13/cobra/command.go:1041151github.com/spf13/cobra.(*Command).ExecuteContext152 /remote-source/app/vendor/github.com/spf13/cobra/command.go:1034153main.main154 /remote-source/app/product-cli/main.go:59155runtime.main156 /usr/lib/golang/src/runtime/proc.go:272157Error: failed to retrieve manifest virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:7acdfad179f4571cbf211a87bce87749a1576b72f1d57499e6d9be09b0c4d31d: failed to create repository client for https://virthost.ostest.test.metalkube.org:5000: Get "https://virthost.ostest.test.metalkube.org:5000/v2/": Internal Server Error158failed to retrieve manifest virthost.ostest.test.metalkube.org:5000/localimages/local-release-image@sha256:7acdfad179f4571cbf211a87bce87749a1576b72f1d57499e6d9be09b0c4d31d: failed to create repository client for https://virthost.ostest.test.metalkube.org:5000: Get "https://virthost.ostest.test.metalkube.org:5000/v2/": Internal Server Error
Expected results:
can be created successful
Additional info:
Description of problem:
Add E2E test cases for PPC related to PerPodPowermanagement
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
Steps to Reproduce:
E2E Test cases were missing in PPC related to PerPodPowerManagment workload hint
Actual results:
Expected results:
Additional info:
Description of problem:
When updating cypress-axe, new changes and bugfixes in the axe-core accessibility auditing package have surfaced various accessibility violations that have to be addressed
Version-Release number of selected component (if applicable):
openshift4.18.0
How reproducible:
always
Steps to Reproduce:
1. Update axe-core and cypress-axe to the latest versions 2. Run test-cypress-console and run a cypress test, I used other-routes.cy.ts
Actual results:
The tests fail with various accessibility violations
Expected results:
The tests pass without accessibility violations
Additional info:
Description of problem:
IBM Cloud relies on a static hard coded list of support Regions for IPI. However, whenever a new Region becomes available, it cannot be used until it is added to this list of available Regions. https://github.com/openshift/installer/blob/13932601852174e4294e16ff9cfca7df082f1ce0/pkg/types/ibmcloud/validation/platform.go#L15-L30
Version-Release number of selected component (if applicable):
4.19.0
How reproducible:
100%
Steps to Reproduce:
1. Create install-config for IBM Cloud 2. Specify a newly available Region (e.g., ca-mon) 3. Attempt to create manifests
Actual results:
ERROR failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: platform.ibmcloud.region: Unsupported value: "ca-mon": supported values: "eu-gb", "jp-tok", "au-syd", "ca-tor", "jp-osa", "br-sao", "us-south", "us-east", "eu-de", "eu-es"
Expected results:
Successful cluster creation
Additional info:
Rather than keep updating this static list, it would be better if the Regions could be looked up dynamically, to provide to the install-config creation path. Thus, validation for the Region would not be necessary in that case. For instance current regions: Listing regions... Name Show name au-syd Sydney (au-syd) in-che Chennai (in-che) jp-osa Osaka (jp-osa) jp-tok Tokyo (jp-tok) eu-de Frankfurt (eu-de) eu-es Madrid (eu-es) eu-gb London (eu-gb) ca-mon Montreal (ca-mon) ca-tor Toronto (ca-tor) us-south Dallas (us-south) ...
Need to update the openshift/api project to contain correct disk size limit
when building machine-os-images container https://github.com/openshift/machine-os-images we see an error stating that the coreos-installer package can't be installed
No match for argument: coreos-installer Error: Unable to find a match: coreos-installer error: build error: building at STEP "RUN dnf install -y jq wget coreos-installer": while running runtime: exit status 1
after a quick troubleshooting with the help of ART team it seems that the issue is related to the installer image used as base image that does not have the correct repositories
Description of problem:
Applying a performance profile on an ARM cluster, results with the tuned profile to turn degraded
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Label a worker node with a worker-cnf label 2. Create an mcp referring to that label 3. Apply the below performance profile apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: cpu: isolated: "1-3,4-6" reserved: "0,7" hugepages: defaultHugepagesSize: 512M pages: - count: 1 node: 0 size: 512M - count: 128 node: 1 size: 2M machineConfigPoolSelector: machineconfiguration.openshift.io/role: worker-cnf net: userLevelNetworking: true nodeSelector: node-role.kubernetes.io/worker-cnf: '' kernelPageSize: 64k numa: topologyPolicy: single-numa-node realTimeKernel: enabled: false workloadHints: highPowerConsumption: true perPodPowerManagement: false realTime: true
Actual results:
Expected results:
Additional info:
[root@ampere-one-x-04 ~]# oc get profiles -A NAMESPACE NAME TUNED APPLIED DEGRADED MESSAGE AGE openshift-cluster-node-tuning-operator ocp-ctlplane-0.libvirt.lab.eng.tlv2.redhat.com openshift-control-plane True False TuneD profile applied. 22h openshift-cluster-node-tuning-operator ocp-ctlplane-1.libvirt.lab.eng.tlv2.redhat.com openshift-control-plane True False TuneD profile applied. 22h openshift-cluster-node-tuning-operator ocp-ctlplane-2.libvirt.lab.eng.tlv2.redhat.com openshift-control-plane True False TuneD profile applied. 22h openshift-cluster-node-tuning-operator ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com openshift-node-performance-performance False True The TuneD daemon profile not yet applied, or application failed. 22h openshift-cluster-node-tuning-operator ocp-worker-1.libvirt.lab.eng.tlv2.redhat.com openshift-node True False TuneD profile applied. 22h openshift-cluster-node-tuning-operator ocp-worker-2.libvirt.lab.eng.tlv2.redhat.com openshift-node True False TuneD profile applied. 22h [root@ampere-one-x-04 ~]# oc describe performanceprofile Name: performance Namespace: Labels: <none> Annotations: <none> API Version: performance.openshift.io/v2 Kind: PerformanceProfile Metadata: Creation Timestamp: 2025-03-04T15:28:44Z Finalizers: foreground-deletion Generation: 1 Resource Version: 74234 UID: 0d9c1817-c12f-4ea8-9c4b-b37badc232e9 Spec: Cpu: Isolated: 1-3,4-6 Reserved: 0,7 Hugepages: Default Hugepages Size: 512M Pages: Count: 1 Node: 0 Size: 512M Count: 128 Node: 1 Size: 2M Kernel Page Size: 64k Machine Config Pool Selector: machineconfiguration.openshift.io/role: worker-cnf Net: User Level Networking: true Node Selector: node-role.kubernetes.io/worker-cnf: Numa: Topology Policy: single-numa-node Real Time Kernel: Enabled: false Workload Hints: High Power Consumption: true Per Pod Power Management: false Real Time: true Status: Conditions: Last Heartbeat Time: 2025-03-04T15:28:45Z Last Transition Time: 2025-03-04T15:28:45Z Status: False Type: Available Last Heartbeat Time: 2025-03-04T15:28:45Z Last Transition Time: 2025-03-04T15:28:45Z Status: False Type: Upgradeable Last Heartbeat Time: 2025-03-04T15:28:45Z Last Transition Time: 2025-03-04T15:28:45Z Status: False Type: Progressing Last Heartbeat Time: 2025-03-04T15:28:45Z Last Transition Time: 2025-03-04T15:28:45Z Message: Tuned ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com Degraded Reason: TunedError. Tuned ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com Degraded Message: TuneD daemon issued one or more error message(s) during profile application. TuneD stderr: . Reason: TunedProfileDegraded Status: True Type: Degraded Runtime Class: performance-performance Tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-performance Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Creation succeeded 112m (x9 over 17h) performance-profile-controller Succeeded to create all components [root@ampere-one-x-04 ~]# oc logs pod/tuned-kjc8j I0304 15:35:50.346412 3259 controller.go:1666] starting in-cluster ocp-tuned v4.19.0-202502262344.p0.gf166846.assembly.stream.el9-0-g0d9dd16-dirty I0304 15:35:50.401840 3259 controller.go:671] writing /var/lib/ocp-tuned/image.env I0304 15:35:50.418669 3259 controller.go:702] tunedRecommendFileRead(): read "openshift-node-performance-performance" from "/etc/tuned/recommend.d/50-openshift.conf" I0304 15:35:50.419585 3259 controller.go:1728] starting: profile unpacked is "openshift-node-performance-performance" fingerprint "ab0d99d8009d6539b91ed1aeff3e4fa1c629c1cd4e9a32bdc132dcc9737e4fc9" I0304 15:35:50.419646 3259 controller.go:1424] recover: no pending deferred change I0304 15:35:50.419666 3259 controller.go:1734] starting: no pending deferred update I0304 15:36:06.074575 3259 controller.go:382] disabling system tuned... I0304 15:36:06.121045 3259 controller.go:1546] started events processors I0304 15:36:06.121492 3259 controller.go:359] set log level 0 I0304 15:36:06.121850 3259 controller.go:1567] monitoring filesystem events on "/etc/tuned/bootcmdline" I0304 15:36:06.121886 3259 controller.go:1570] started controller I0304 15:36:06.122603 3259 controller.go:692] tunedRecommendFileWrite(): written "/etc/tuned/recommend.d/50-openshift.conf" to set TuneD profile openshift-node-performance-performance I0304 15:36:06.122634 3259 controller.go:417] profilesExtract(): extracting 6 TuneD profiles (recommended=openshift-node-performance-performance) I0304 15:36:06.210862 3259 controller.go:462] profilesExtract(): recommended TuneD profile openshift-node-performance-performance content unchanged [openshift] I0304 15:36:06.211950 3259 controller.go:462] profilesExtract(): recommended TuneD profile openshift-node-performance-performance content unchanged [openshift-node-performance-performance] I0304 15:36:06.212311 3259 controller.go:478] profilesExtract(): fingerprint of extracted profiles: "ab0d99d8009d6539b91ed1aeff3e4fa1c629c1cd4e9a32bdc132dcc9737e4fc9" I0304 15:36:06.212389 3259 controller.go:818] tunedReload() I0304 15:36:06.212493 3259 controller.go:745] starting tuned... I0304 15:36:06.212547 3259 run.go:121] running cmd... 2025-03-04 15:36:06,335 INFO tuned.daemon.application: TuneD: 2.25.1, kernel: 5.14.0-570.el9.aarch64+64k 2025-03-04 15:36:06,335 INFO tuned.daemon.application: dynamic tuning is globally disabled 2025-03-04 15:36:06,340 INFO tuned.daemon.daemon: using sleep interval of 1 second(s) 2025-03-04 15:36:06,340 INFO tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration. 2025-03-04 15:36:06,341 INFO tuned.daemon.daemon: Using 'openshift-node-performance-performance' profile 2025-03-04 15:36:06,342 INFO tuned.profiles.loader: loading profile: openshift-node-performance-performance 2025-03-04 15:36:06,460 ERROR tuned.daemon.daemon: Cannot set initial profile. No tunings will be enabled: Cannot load profile(s) 'openshift-node-performance-performance': Cannot find profile 'openshift-node-performance--aarch64-performance' in '['/var/lib/ocp-tuned/profiles', '/usr/lib/tuned', '/usr/lib/tuned/profiles']'. 2025-03-04 15:36:06,461 INFO tuned.daemon.controller: starting controller sh-5.1# systemctl --no-pager | grep hugepages dev-hugepages.mount loaded active mounted Huge Pages File System ● hugepages-allocation-2048kB-NUMA1.service loaded failed failed Hugepages-2048kB allocation on the node 1 hugepages-allocation-524288kB-NUMA0.service loaded active exited Hugepages-524288kB allocation on the node 0 sh-5.1# systemctl status hugepages-allocation-2048kB-NUMA1.service × hugepages-allocation-2048kB-NUMA1.service - Hugepages-2048kB allocation on the node 1 Loaded: loaded (/etc/systemd/system/hugepages-allocation-2048kB-NUMA1.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Tue 2025-03-04 15:32:33 UTC; 17h ago Main PID: 1002 (code=exited, status=1/FAILURE) CPU: 6ms Mar 04 15:32:33 ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com systemd[1]: Starting Hugepages-2048kB allocation on the node 1... Mar 04 15:32:33 ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com hugepages-allocation.sh[1002]: ERROR: /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages does not exist Mar 04 15:32:33 ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com systemd[1]: hugepages-allocation-2048kB-NUMA1.service: Main process exited, code=exited, status=1/FAILURE Mar 04 15:32:33 ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com systemd[1]: hugepages-allocation-2048kB-NUMA1.service: Failed with result 'exit-code'. Mar 04 15:32:33 ocp-worker-0.libvirt.lab.eng.tlv2.redhat.com systemd[1]: Failed to start Hugepages-2048kB allocation on the node 1. sh-5.1# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt3)/boot/ostree/rhcos-e032e3de5cffeccaf88bc5dc1945da35b4273c5f5b758a6ca1d0d78344b55e7f/vmlinuz-5.14.0-570.el9.aarch64+64k rw ostree=/ostree/boot.0/rhcos/e032e3de5cffeccaf88bc5dc1945da35b4273c5f5b758a6ca1d0d78344b55e7f/0 ignition.platform.id=openstack console=ttyAMA0,115200n8 console=tty0 root=UUID=96763b3b-e217-4879-a03e-56568ca84bf9 rw rootflags=prjquota boot=UUID=d98055a6-2355-40d3-8e87-98eedd0e8c91 systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all psi=0
bash-5.1# ls /var/lib/ocp-tuned/profiles/ openshift openshift-node-performance-intel-x86-performance openshift-node-performance-amd-x86-performance openshift-node-performance-performance openshift-node-performance-arm-aarch64-performance openshift-node-performance-rt-performance bash-5.1# cat /var/lib/ocp-tuned/profiles/openshift-node-performance-performance/tuned.conf [main] summary=Openshift node optimized for deterministic performance at the cost of increased power consumption, focused on low latency network performance. Based on Tuned 2.11 and Cluster node tuning (oc 4.5) # The final result of the include depends on cpu vendor, cpu architecture, and whether the real time kernel is enabled # The first line will be evaluated based on the CPU vendor and architecture # This has three possible results: # include=openshift-node-performance-amd-x86; # include=openshift-node-performance-arm-aarch64; # include=openshift-node-performance-intel-x86; # The second line will be evaluated based on whether the real time kernel is enabled # This has two possible results: # openshift-node,cpu-partitioning # openshift-node,cpu-partitioning,openshift-node-performance-rt-<PerformanceProfile name> include=openshift-node,cpu-partitioning${f:regex_search_ternary:${f:exec:uname:-r}:rt:,openshift-node-performance-rt-performance:}; openshift-node-performance-${f:lscpu_check:Vendor ID\:\s*GenuineIntel:intel:Vendor ID\:\s*AuthenticAMD:amd:Vendor ID\:\s*ARM:arm}-${f:lscpu_check:Architecture\:\s*x86_64:x86:Architecture\:\s*aarch64:aarch64}-performance # Inheritance of base profiles legend: # cpu-partitioning -> network-latency -> latency-performance # https://github.com/redhat-performance/tuned/blob/master/profiles/latency-performance/tuned.conf # https://github.com/redhat-performance/tuned/blob/master/profiles/network-latency/tuned.conf # https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/tuned.conf # All values are mapped with a comment where a parent profile contains them. # Different values will override the original values in parent profiles. [variables] #> isolated_cores take a list of ranges; e.g. isolated_cores=2,4-7 isolated_cores=1-6 not_isolated_cores_expanded=${f:cpulist_invert:${isolated_cores_expanded}} [cpu] #> latency-performance #> (override) force_latency=cstate.id:1|3 governor=performance energy_perf_bias=performance min_perf_pct=100 [service] service.stalld=start,enable [vm] #> network-latency transparent_hugepages=never [irqbalance] # Disable the plugin entirely, which was enabled by the parent profile `cpu-partitioning`. # It can be racy if TuneD restarts for whatever reason. #> cpu-partitioning enabled=false [scheduler] runtime=0 group.ksoftirqd=0:f:11:*:ksoftirqd.* group.rcuc=0:f:11:*:rcuc.* group.ktimers=0:f:11:*:ktimers.* default_irq_smp_affinity = ignore irq_process=false [sysctl] #> cpu-partitioning #RealTimeHint kernel.hung_task_timeout_secs=600 #> cpu-partitioning #RealTimeHint kernel.nmi_watchdog=0 #> RealTimeHint kernel.sched_rt_runtime_us=-1 #> cpu-partitioning #RealTimeHint vm.stat_interval=10 # cpu-partitioning and RealTimeHint for RHEL disable it (= 0) # OCP is too dynamic when partitioning and needs to evacuate #> scheduled timers when starting a guaranteed workload (= 1) kernel.timer_migration=1 #> network-latency net.ipv4.tcp_fastopen=3 # If a workload mostly uses anonymous memory and it hits this limit, the entire # working set is buffered for I/O, and any more write buffering would require # swapping, so it's time to throttle writes until I/O can catch up. Workloads # that mostly use file mappings may be able to use even higher values. # # The generator of dirty data starts writeback at this percentage (system default # is 20%) #> latency-performance vm.dirty_ratio=10 # Start background writeback (via writeback threads) at this percentage (system # default is 10%) #> latency-performance vm.dirty_background_ratio=3 # The swappiness parameter controls the tendency of the kernel to move # processes out of physical memory and onto the swap disk. # 0 tells the kernel to avoid swapping processes out of physical memory # for as long as possible # 100 tells the kernel to aggressively swap processes out of physical memory # and move them to swap cache #> latency-performance vm.swappiness=10 # also configured via a sysctl.d file # placed here for documentation purposes and commented out due # to a tuned logging bug complaining about duplicate sysctl: # https://issues.redhat.com/browse/RHEL-18972 #> rps configuration # net.core.rps_default_mask=${not_isolated_cpumask} [selinux] #> Custom (atomic host) avc_cache_threshold=8192 [net] channels=combined 2 nf_conntrack_hashsize=131072 [bootloader] # !! The names are important for Intel and are referenced in openshift-node-performance-intel-x86 # set empty values to disable RHEL initrd setting in cpu-partitioning initrd_remove_dir= initrd_dst_img= initrd_add_dir= # overrides cpu-partitioning cmdline cmdline_cpu_part=+nohz=on rcu_nocbs=${isolated_cores} tuned.non_isolcpus=${not_isolated_cpumask} systemd.cpu_affinity=${not_isolated_cores_expanded} # No default value but will be composed conditionally based on platform cmdline_iommu= cmdline_isolation=+isolcpus=managed_irq,${isolated_cores} cmdline_realtime_nohzfull=+nohz_full=${isolated_cores} cmdline_realtime_nosoftlookup=+nosoftlockup cmdline_realtime_common=+skew_tick=1 rcutree.kthread_prio=11 # No default value but will be composed conditionally based on platform cmdline_power_performance= # No default value but will be composed conditionally based on platform cmdline_idle_poll= [rtentsk]
Trying to solve the root issue from this bug: https://issues.redhat.com/browse/OCPBUGS-39199?focusedId=26104570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-26104570
To fix this, we need each of the sync functions to be able to individually clear a CO degrade that they have set earlier. Our current flow only clears a CO degrade when the all of sync functions are successful and that tends to be problematic if they happen to get stuck in one of the sync functions. We typically see this for syncRequiredMachineConfigPools, which waits until the master nodes have finished updating during an upgrade.
Description of problem:
After https://github.com/openshift/api/pull/2076, the validation in the image registry operator for the Power VS platform does not match the API's expectations.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
This function https://github.com/openshift/hypershift/blame/c34a1f6cef0cb41c8a1f83acd4ddf10a4b9e8532/support/util/util.go#L391 does not checks the IDMS/ICSP overrides during the reconciliation, so it breaks the disconnected deployments.
Description of problem:
'Channel' and 'Version' dropdowns do not collapse if the user does not select an option
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-04-113014
How reproducible:
Always
Steps to Reproduce:
1. Naviage to Operator Insatallation page OR Operator Install details page eg: /operatorhub/ns/openshift-console?source=["Red+Hat"]&details-item=datagrid-redhat-operators-openshift-marketplace&channel=stable&version=8.5.4 /operatorhub/subscribe?pkg=datagrid&catalog=redhat-operators&catalogNamespace=openshift-marketplace&targetNamespace=openshift-console&channel=stable&version=8.5.4&tokenizedAuth= 2. Click the Channel/Update channel OR 'Version' dropdown list 3. Click the dropdow again
Actual results:
The dropdown list cannot collapse, only if user selected an option OR click other area
Expected results:
the dropdown can collapse after click
Additional info:
Azure File tests are perma-failing with the updated 1.32 Kubernetes in OCP 4.19:
Slack thread: https://redhat-internal.slack.com/archives/C07V32J0YKF/p1734112832694339
At "-v=2" Says nothing about telemetry being disabled on the cluster https://docs.openshift.com/container-platform/4.17/support/remote_health_monitoring/opting-out-of-remote-health-reporting.html
maybe it does in v>2, check that.
See https://issues.redhat.com/browse/OCPBUGS-45683
Description of problem:
s2i conformance test appears to fail permanently on OCP 4.16.z
Version-Release number of selected component (if applicable):
4.16.z
How reproducible:
Since 2024-11-04 at least
Steps to Reproduce:
Run OpenShift build test suite in PR
Actual results:
Test fails - root cause appears to be that a built/deployed pod crashloops
Expected results:
Test succeeds
Additional info:
Job history https://prow.ci.openshift.org/job-history/gs/test-platform-results/pr-logs/directory/pull-ci-openshift-openshift-controller-manager-release-4.16-e2e-gcp-ovn-builds
time="2025-02-11T04:45:39Z" level=debug msg="I0211 04:45:39.801815 670 machine_controller.go:359] \"Skipping deletion of Kubernetes Node associated with Machine as it is not allowed\" controller=\"machine\" controllerGroup=\"cluster.x-k8s.io\" controllerKind=\"Machine\" Machine=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-bootstrap\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-il9yv8px-da9f5-526v6-bootstrap\" reconcileID=\"1cc19254-df46-4194-a020-c87f9b6eebb5\" Cluster=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-0\" Cluster=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-0\" Node=\"\" cause=\"noderef is nil\"" time="2025-02-11T04:45:39Z" level=debug msg="I0211 04:45:39.809203 712 vimmachine.go:384] \"Updated VSphereVM\" controller=\"vspheremachine\" controllerGroup=\"infrastructure.cluster.x-k8s.io\" controllerKind=\"VSphereMachine\" VSphereMachine=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-bootstrap\" namespace=\"openshift-cluster-api-guests\" name=\"ci-op-il9yv8px-da9f5-526v6-bootstrap\" reconcileID=\"57d3cf64-4277-4b55-9f98-bfdf6030f4b7\" Machine=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-bootstrap\" Cluster=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-0\" Cluster=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-0\" VSphereCluster=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-0\" VSphereVM=\"openshift-cluster-api-guests/ci-op-il9yv8px-da9f5-526v6-bootstrap\""
time="2025-02-11T04:50:39Z" level=warning msg="Timeout deleting bootstrap machine: context deadline exceeded" time="2025-02-11T04:50:39Z" level=info msg="Shutting down local Cluster API controllers..."
Description of problem:
Every import of react-redux useSelector and useDispatch hooks must have ts and eslint override comments because @types/react-redux is out of sync with the react-redux and there are no type definitions for these hooks.
Version-Release number of selected component (if applicable):
4.16.0
How reproducible:
Steps to Reproduce:
1. Import useSelector or useDispatch hook
Actual results:
A typescript error is shown: "react-redux"' has no exported member named 'useSelector'
Expected results:
These modules can be imported without typescript errors
Was disabled here https://issues.redhat.com/browse/MON-3959
It'll need a rewrite/some mocking. (it's good to test that MCO detects changes and sync, but maybe we could do that in a dry run way)
Description of problem:
OpenShift internal registry panics when deploying OpenShift on AWS ap-southeast-5 region
Version-Release number of selected component (if applicable):
OpenShift 4.15.29
How reproducible:
Always
Steps to Reproduce:
1. Deploy OpenShift 4.15.29 on AWS ap-southeast-5 region 2. The cluster gets deployed but the image-registry Operator is not available and image-registry pods get in CrashLoopBackOff state
Actual results:
panic: invalid region provided: ap-southeast-5goroutine 1 [running]: github.com/distribution/distribution/v3/registry/handlers.NewApp({0x2983cd0?, 0xc00005c088?}, 0xc000640c00) /go/src/github.com/openshift/image-registry/vendor/github.com/distribution/distribution/v3/registry/handlers/app.go:130 +0x2bf1 github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware.NewApp({0x2983cd0, 0xc00005c088}, 0x0?, {0x2986620?, 0xc000377560}) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/supermiddleware/app.go:96 +0xb9 github.com/openshift/image-registry/pkg/dockerregistry/server.NewApp({0x2983cd0?, 0xc00005c088}, {0x296fa38?, 0xc0008e4148}, 0xc000640c00, 0xc000aa6140, {0x0?, 0x0}) /go/src/github.com/openshift/image-registry/pkg/dockerregistry/server/app.go:138 +0x485 github.com/openshift/image-registry/pkg/cmd/dockerregistry.NewServer({0x2983cd0, 0xc00005c088}, 0xc000640c00, 0xc000aa6140) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:212 +0x38a github.com/openshift/image-registry/pkg/cmd/dockerregistry.Execute({0x2968300, 0xc000666408}) /go/src/github.com/openshift/image-registry/pkg/cmd/dockerregistry/dockerregistry.go:166 +0x86b main.main() /go/src/github.com/openshift/image-registry/cmd/dockerregistry/main.go:93 +0x496
Expected results:
image-resgistry Operator and pods are available
Additional info:
We can asume the results will be the same while deploying on 4.16 and 4.17 but can't be tested yet as only 4.15 is working in this region. Will open another bug for the Installer to solve the issues while deploying on this region
Description of problem:
From the ARO team evaluating oc-mirror. Setting --parallel-images processes one more image than specified
Version-Release number of selected component (if applicable):
4.18
How reproducible:
Always
Steps to Reproduce:
Use --parallel-images=1
Actual results:
See attached image
Expected results:
Should use only 1 layer
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
4.19 CI payloads have now failed mutiple times in a row on hypershift-e2e, same two Karpenter tests.
: TestKarpenter/Main expand_less 0s {Failed === RUN TestKarpenter/Main util.go:153: Successfully waited for kubeconfig to be published for HostedCluster e2e-clusters-t8sw8/example-vr6sz in 25ms util.go:170: Successfully waited for kubeconfig secret to have data in 25ms util.go:213: Successfully waited for a successful connection to the guest API server in 25ms karpenter_test.go:52: Expected success, but got an error: <*meta.NoKindMatchError | 0xc002d931c0>: no matches for kind "NodePool" in version "karpenter.sh/v1" { GroupKind: { Group: "karpenter.sh", Kind: "NodePool", }, SearchedVersions: ["v1"], } --- FAIL: TestKarpenter/Main (0.10s) } : TestKarpenter expand_less 27m15s {Failed === RUN TestKarpenter === PAUSE TestKarpenter === CONT TestKarpenter hypershift_framework.go:316: Successfully created hostedcluster e2e-clusters-t8sw8/example-vr6sz in 24s hypershift_framework.go:115: Summarizing unexpected conditions for HostedCluster example-vr6sz util.go:1699: Successfully waited for HostedCluster e2e-clusters-t8sw8/example-vr6sz to have valid conditions in 25ms hypershift_framework.go:194: skipping postTeardown() hypershift_framework.go:175: skipping teardown, already called --- FAIL: TestKarpenter (1635.11s) }
https://github.com/openshift/hypershift/pull/5404 is in the first payload and looks extremely related.
Description of problem:
The PowerVS installer uses a hard coded list of supported machine types. However, this does not keep up as new types are added. Therefore switch to querying the datacenter for the current supported types.
Description of problem:
platform.powervs.clusterOSImage is no longer supported but is still supported in the install configuration YAML file.
Version-Release number of selected component (if applicable):
4.19.0
Expected results:
In case the platform.powervs.clusterOSImage key is specified, a warning "The value of platform.powervs.clusterOSImage will be ignored." should be shown and the value of the key should be ignored.
Description of the problem:
How reproducible:
Steps to reproduce:
1.
2.
3.
Actual results:
Expected results:
Description of problem:
Azure MAG cluster install failed. azure-cloud-controller-manager pod stuck in CrashLoopBackOff state, seems endpoint is not correct, should be https://management.usgovcloudapi.net/ The code is here https://github.com/openshift/cloud-provider-azure/blob/main/pkg/provider/azure.go#L490 2025-02-17T23:49:41.894495557Z E0217 23:49:41.894412 1 azure.go:490] InitializeCloudFromConfig: failed to sync regional zones map for the first time: list zones: GET https://management.azure.com/subscriptions/8fe0c1b4-8b05-4ef7-8129-7cf5680f27e7/providers/Microsoft.Compute 2025-02-17T23:49:41.894495557Z -------------------------------------------------------------------------------- 2025-02-17T23:49:41.894495557Z RESPONSE 404: 404 Not Found 2025-02-17T23:49:41.894495557Z ERROR CODE: SubscriptionNotFound 2025-02-17T23:49:41.894495557Z -------------------------------------------------------------------------------- 2025-02-17T23:49:41.894495557Z { 2025-02-17T23:49:41.894495557Z "error": { 2025-02-17T23:49:41.894495557Z "code": "SubscriptionNotFound", 2025-02-17T23:49:41.894495557Z "message": "The subscription '8fe0c1b4-8b05-4ef7-8129-7cf5680f27e7' could not be found." 2025-02-17T23:49:41.894495557Z } 2025-02-17T23:49:41.894495557Z } 2025-02-17T23:49:41.894495557Z -------------------------------------------------------------------------------- 2025-02-17T23:49:41.894495557Z F0217 23:49:41.894467 1 controllermanager.go:353] Cloud provider azure could not be initialized: could not init cloud provider azure: list zones: GET https://management.azure.com/subscriptions/8fe0c1b4-8b05-4ef7-8129-7cf5680f27e7/providers/Microsoft.Compute 2025-02-17T23:49:41.894495557Z -------------------------------------------------------------------------------- 2025-02-17T23:49:41.894495557Z RESPONSE 404: 404 Not Found 2025-02-17T23:49:41.894495557Z ERROR CODE: SubscriptionNotFound 2025-02-17T23:49:41.894495557Z -------------------------------------------------------------------------------- 2025-02-17T23:49:41.894495557Z { 2025-02-17T23:49:41.894495557Z "error": { 2025-02-17T23:49:41.894495557Z "code": "SubscriptionNotFound", 2025-02-17T23:49:41.894495557Z "message": "The subscription '8fe0c1b4-8b05-4ef7-8129-7cf5680f27e7' could not be found." 2025-02-17T23:49:41.894495557Z } 2025-02-17T23:49:41.894495557Z } 2025-02-17T23:49:41.894495557Z --------------------------------------------------------------------------------
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. Install cluster on azure mag 2. 3.
Actual results:
Cluster install failed
Expected results:
Cluster install succeed
Additional info:
This only found on 4.19 azure mag clusters
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-azure-mag-ipi-fips-f7/1890522382758580224
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-azure-mag-ipi-fullyprivate-f7/1891606065993224192
Description of problem:
In tech preview, the cluster-capi-operator is unable to automatically generate the core Cluster on powerVS clusters Failing with the following error: `failed to get infra cluster` This is due to a wrong infraclusterkind string for the IBMPowerVSCluster InfraCluster Kind.
Version-Release number of selected component (if applicable):
4.19 4.18
How reproducible:
always
Steps to Reproduce:
1. stand up a TechPreview powervs cluster 2. Run e2e tests from the cluster-capi-operator repo 3. check the cluster-capi-operator logs
Actual results:
failed to get infra cluster
Expected results:
should get the infra cluster and create a core cluster
Additional info:
Description of problem:
Destroying a private cluster doesn't delete the forwarding-rule/backend-service/health-check/firewall-rules created by ingress operator.
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-multi-2024-10-23-202329
How reproducible:
Always
Steps to Reproduce:
1. pre-create vpc network/subnets/router and a bastion host 2. "create install-config", and then insert the network settings under platform.gcp, along with "publish: Internal" (see [1]) 3. "create cluster" (use the above bastion host as http proxy) 4. "destroy cluster" (see [2])
Actual results:
Although "destroy cluster" completes successfully, the forwarding-rule/backend-service/health-check/firewall-rules created by ingress operator are not deleted (see [3]), which leads to deleting the vpc network/subnets failure.
Expected results:
The forwarding-rule/backend-service/health-check/firewall-rules created by ingress operator should also be deleted during "destroy cluster".
Additional info:
FYI one history bug https://issues.redhat.com/browse/OCPBUGS-37683
This section includes Jira cards that are not linked to either an Epic or a Feature. These tickets were not completed when this image was assembled
Description of problem:
When installing a 4.19 OCP on OSP, the afterburn-hostname.service is failing the installation progresses, but that services fails and timeout. Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: Jan 27 21:19:10.475 WARN failed to locate config-drive, using the metadata service API instead Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: Jan 27 21:19:10.489 INFO Fetching http://169.254.169.254/latest/meta-data/hostname: Attempt #1 Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: Jan 27 21:19:10.498 INFO Fetch successful Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: Error: failed to run Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: Caused by: Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: 0: writing hostname Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: 1: failed to create file "/dev/stdout" Jan 27 21:19:10 ostest-qjbdz-master-1 bash[3245]: 2: Permission denied (os error 13) Jan 27 21:19:10 ostest-qjbdz-master-1 hostnamectl[3249]: Too few arguments.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
USER STORY:
As an OpenShift administrator, I want to be able to configure thin provisioned for my new data disks so that adjust the behavior that may be different than my default storage policy.
DESCRIPTION:
Currently, we have the machine api changes forcing the thin provisioned flag to true. We need to add a flag to allow admin to configure this. The default behavior will be to not set the flag and use default storage policy.
ACCEPTANCE CRITERIA:
The UDN CRD is only installed in the OpenShift cluster when the `NetworkSegmentation feature gate is enabled.
Hence, the tests must reflect this.
Description of problem:
openstack-manila-csi-controllerplugin-csi-driver container is not functional in the first run, it needs to restart once and then it's good. This causes HCP e2e to fail on the EnsureNoCrashingPods test.
Version-Release number of selected component (if applicable):
4.19, 4.18
How reproducible:
Deploy Shift on Stack with Manila available in the cloud.
Actual results:
The openstack-manila-csi-controllerplugin pod will restart once and then it'll be functional.
Expected results:
No restart should be needed. This is likely an orchestration issue.
In the OpenShift Web Console, under Storage → VolumeSnapshots, VolumeSnapshots are not displayed when selecting "All Projects" as the project.
However, when selecting a specific project, the VolumeSnapshots are displayed correctly.
This issue only occurs when "All Projects" is selected.
CMO should create and deploy a configmap that contains the data for accelerators monitoring. When CMO creates node-exporter daemonsets and mounts the the configmap into the node-exporter's pods
Currently, in most of assisted installer components CI images we don't have a way to tell from which commit reference the image was built. Since We use an image stream for each component, and we import these streams from one CI component configuration to another, we might end up with images to are not up-to-date. In this case, we would like to have the ability to check if this is actually the case.
We should add golangci-lint to our `make verify`. This will catch common golang errors. There are a number of linters we can include if we want - https://golangci-lint.run/usage/linters/.
In addition, we can introduce GCI through golangci-lint so that all imports are sorted in a proper format.
The Cluster API provider Azure has a deployment manifest that deploys Azure service operator from mcr.microsoft.com/k8s/azureserviceoperator:v2.6.0 image.
We need to set up OpenShift builds of the operator and update the manifest generator to use the OpenShift image.
Azure have split the API calls out of their provider so that they now use the service operator. We now need to ship service operator as part of the CAPI operator to make sure that we can support CAPZ.
Description of problem:
Use the /livez/ping endpoint to proxy unauthenticated health checks on the master. /livez/ping is a faster and more reliable endpoint, so it should be used over /version.
Version-Release number of selected component (if applicable):
Impacts all releases, fix can be limited to 4.18
How reproducible:
Always
Steps to Reproduce:
1. https://github.com/openshift/hypershift/blob/6356dab0c28b77cca1a74d911f7154f70a3cb68d/hypershift-operator/controllers/nodepool/apiserver-haproxy/haproxy.cfg#L26 2. https://github.com/openshift/hypershift/blob/6356dab0c28b77cca1a74d911f7154f70a3cb68d/hypershift-operator/controllers/nodepool/haproxy.go#L381
Actual results:
/version
Expected results:
/livez/ping
Additional info:
One of our customers observed this issue. In order to reproduce, In my test cluster, I intentionally increased the overall CPU limits to over 200% and monitored the cluster for more than 2 days. However, I did not see the KubeCPUOvercommit alert, which ideally should trigger after 10 minutes of overcommitment.
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 2654m (75%) 8450m (241%)
memory 5995Mi (87%) 12264Mi (179%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
OCP console --> Observe --> alerting --> alert rule and select for the `KubeCPUOvercommit` alert.
Expression:
sum by (cluster) (namespace_cpu:kube_pod_container_resource_requests:sum{job="kube-state-metrics"}) - (sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) - max by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})) > 0 and (sum by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) - max by (cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})) > 0
Description of problem:
While testing on https://github.com/openshift/machine-config-operator/pull/4800#issuecomment-2628801965, we found the worker can't fetch Ignition successfully so it couldn't be started well. The worker machine console serial log shows: �[K[�[0m�[0;31m* �[0m] A start job is running for Ignition (fetch) (50min 30s / no limit) [ 3034.007742] ignition[823]: GET https://api-int.gpei-0201-gcpdns.qe.gcp.devcluster.openshift.com:22623/config/worker: attempt #607 [ 3034.044185] ignition[823]: GET error: Get "https://api-int.gpei-0201-gcpdns.qe.gcp.devcluster.openshift.com:22623/config/worker": dial tcp: lookup api-int.gpei-0201-gcpdns.qe.gcp.devcluster.openshift.com on 169.254.169.254:53: no such host
Version-Release number of selected component (if applicable):
Payload built based on https://github.com/openshift/machine-config-operator/pull/4800, https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/1885498038630223872
How reproducible:
Always
Steps to Reproduce:
1. Create a 4.19 cluster with customDNS feature enabled: featureSet: CustomNoUpgrade featureGates: - GCPClusterHostedDNS=true
Actual results:
Expected results:
Additional info:
Description of problem:
Instead of synthesised "operator conditions" test we should use real create / schedule pods via deployment controller to test kube-apiserver / kcm / scheduler functionality
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
The Installer does not seem to be checking the server group quota before installing.
With OSASINFRA-2570, each cluster will need two server groups. Additionally, it might be worth checking that server groups are set to accept at least n members (where n is the number of Compute replicas).
Refer to https://github.com/okd-project/okd/discussions/2060
Upgrade to scos-release:4.16.0-okd-scos.0 from 4.15.0-0.okd-scos-2024-01-18-223523 for me stuck on network-operator rolling out the DaemonSet "/openshift-multus/whereabouts-reconciler".
The whereabouts-reconciler Pod is crashlooping with:
{{ [entrypoint.sh] FATAL ERROR: Unsupported OS ID=centos}}
Indeed, is image is based on:
{{cat /etc/os-release:
NAME="CentOS Stream"
VERSION="9"
ID="centos"
...}}
But entrypoint.sh does not handle the centos name:
{{# Collect host OS information
. /etc/os-release
rhelmajor=
We've seen a high rate of failure for this test since last Thursday.
(Feel free to update this bug's summary to be more specific.)
Component Readiness has found a potential regression in the following test:
[sig-arch][Late] operators should not create watch channels very often [apigroup:apiserver.openshift.io] [Suite:openshift/conformance/parallel]
Extreme regression detected.
Fishers Exact probability of a regression: 100.00%.
Test pass rate dropped from 100.00% to 78.33%.
Sample (being evaluated) Release: 4.18
Start Time: 2024-11-04T00:00:00Z
End Time: 2024-11-18T23:59:59Z
Success Rate: 78.33%
Successes: 47
Failures: 13
Flakes: 0
Base (historical) Release: 4.17
Start Time: 2024-09-01T00:00:00Z
End Time: 2024-10-01T23:59:59Z
Success Rate: 100.00%
Successes: 133
Failures: 0
Flakes: 0
View the test details report for additional context.
Description of problem:
I am getting error while generating PXE images(s390x) through Agent based installer and it seems the error is hit when the base ISO image used is ISO 9660 CD-ROM filesystem data 'rhcos-9.6.20250121-0 # ./openshift-install agent create pxe-files INFO Configuration has 3 master replicas and 2 worker replicas INFO The rendezvous host IP (node0 IP) is 172.23.231.117 INFO Extracting base ISO from release payload INFO Verifying cached file INFO Base ISO obtained from release and cached at [/root/.cache/agent/image_cache/coreos-s390x.iso] INFO Consuming Install Config from target directory INFO Consuming Agent Config from target directory FATAL failed to write asset (Agent Installer PXE Files) to disk: open /tmp/agent1716972189/images/pxeboot/kernel.img: no such file or directory is there any change gone in which is causing this issue. I am not facing it with older version - ISO 9660 CD-ROM filesystem data 'rhcos-418.94.202410090804-0' (bootable)
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Create a pxe image with latest 4.19 for ABI on s390x. 2. 3.
Actual results:
Image creation fails with error (FATAL failed to write asset (Agent Installer PXE Files) to disk: open /tmp/agent1716972189/images/pxeboot/kernel.img: no such file or directory )
Expected results:
PXE images should get created successfully
Additional info:
Currently there are some cases which have high failed ratio, we need to identify the root cause as soon as possible and fix the automation issues as soon as possible, so that our automatioin could be more stable and more efficent for our fast release tests.
With each release, each sprint, it's an ongoing work.
Job results: https://prow.ci.openshift.org/?job=*master-e2e-aws-ovn-single-node, pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-ovn-single-node always runs in to failures, especially for repo openshift/cluster-kube-apiserver-operator.
For example, https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-kube-apiserver-operator/1786/pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-ovn-single-node/1882243871929274368
Description of problem:
IPI deployment of OCP on vsphere fails to complete. Network operator in degraded state: spec: {} status: conditions: - lastTransitionTime: "2024-07-04T10:30:43Z" message: 'Error while synchronizing spec and status of infrastructures.config.openshift.io/cluster: Error on validating API VIPs and Machine Networks: VIP ''10.xx.xx.xx'' cannot be found in any machine network' oc get infrastructure shows the following: spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: VSphere vsphere: apiServerInternalIPs: - 10.xx.xx.xx failureDomains: - name: generated-failure-domain region: generated-region server: vcenter.url.com topology: computeCluster: /OpenShift-DC/host/OCP datacenter: OpenShift-DC datastore: /OpenShift-DC/datastore/OCP-PNQ-Datastore networks: - network-id resourcePool: /OpenShift-DC/host/OCP/Resources template: /OpenShift-DC/vm/ocp4ipi07-txnr6-rhcos-generated-region-generated-zone zone: generated-zone ingressIPs: - 10.xx.xx.xx machineNetworks: - 10.0.0.0/16 The machine network is added as default value `10.0.0.0/16` which was not provided when running create cluster command.
Version-Release number of selected component (if applicable):
How reproducible:
100 %
Steps to Reproduce:
1. openshift-installer create cluster 2. post installation check network operator status 3. oc get infrastructure -o yaml
Actual results:
Network operator in degraded state
Expected results:
All operator should be available.
Additional info:
Description of problem:
4.18: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cluster-etcd-operator-1384-ci-4.18-e2e-gcp-ovn/1877471255607644160 40e0ff5ee27c98d0 = bootstrap 1edfd46c2c62c92c = master-0 a0dabab5ae6f967b = master-1 7a509502206916e3 = master-2 22:20:03 (term 2) 1 bootstrap_teardown_controller.go:144] Removing bootstrap member [40e0ff5ee27c98d0] 22:20:03 (term 2) 1 bootstrap_teardown_controller.go:151] Successfully removed bootstrap member [40e0ff5ee27c98d0] 22:20:04 (term 2) leader lost 22:20:04 (term 3) leader=master-1 {"level":"warn","ts":"2025-01-09T22:20:19.459912Z","logger":"raft","caller":"etcdserver/zap_raft.go:85","msg":"a0dabab5ae6f967b stepped down to follower since quorum is not active"} 4.19: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-upgrade/1877300145469526016 26129ddf465384ed = bootstrap 10.0.0.3 fb4ff9e6f7280fcb = master-0 10.0.0.5 283343bd1d4e0df2 = master-1 10.0.0.6 896ca4df9c7807c1 = master-2 10.0.0.4 bootstrap_teardown_controller.go:144] Removing bootstrap member [26129ddf465384ed] I0109 10:48:33.201639 1 bootstrap_teardown_controller.go:151] Successfully removed bootstrap member [26129ddf465384ed] {"level":"info","ts":"2025-01-09T10:48:34.588799Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"896ca4df9c7807c1 became leader at term 3"} {"level":"warn","ts":"2025-01-09T10:48:51.583385Z","logger":"raft","caller":"etcdserver/zap_raft.go:85","msg":"896ca4df9c7807c1 stepped down to follower since quorum is not active"}
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
ap-southeast-5 is supported by https://issues.redhat.com/browse/CORS-3640, but it is missing in the installer's survey ? Region [Use arrows to move, type to filter, ? for more help] ap-southeast-1 (Asia Pacific (Singapore)) ap-southeast-2 (Asia Pacific (Sydney)) ap-southeast-3 (Asia Pacific (Jakarta)) > ap-southeast-4 (Asia Pacific (Melbourne)) ca-central-1 (Canada (Central)) ca-west-1 (Canada West (Calgary)) eu-central-1 (Europe (Frankfurt))
Version-Release number of selected component (if applicable):
4.18.0-0.nightly-2024-12-24-145308
How reproducible:
Always
Steps to Reproduce:
1. Run openshift-install create cluster --dir instdir1 2. 3.
Actual results:
Expected results:
Additional info:
Description of problem:
Version-Release number of selected component (if applicable):
In 4.19, whereabouts-reconciler pods are CrashLoopBackOff after creating additionalNetworks
How reproducible:
Always
Steps to Reproduce:
1. After a OCP cluster up, oc edit networks.operator.openshift.io cluster
additionalNetworks: - name: whereabouts-shim namespace: default type: Raw rawCNIConfig: |- { "cniVersion": "0.3.0", "type": "bridge", "name": "cnitest0", "ipam": { "type": "whereabouts", "subnet": "192.0.2.0/24" } }
Actual results:
whereabouts-reconciler pods are created and CrashLoopBackOff in 4.19
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.19.0-0.nightly-2025-03-04-213423 True False 3h22m Error while reconciling 4.19.0-0.nightly-2025-03-04-213423: the cluster operator network is degraded $ oc get pod | grep whereabouts-reconciler whereabouts-reconciler-dd5w8 0/1 CrashLoopBackOff 14 (5m5s ago) 51m whereabouts-reconciler-lbtmz 0/1 CrashLoopBackOff 14 (5m4s ago) 51m whereabouts-reconciler-m6qx5 0/1 CrashLoopBackOff 14 (4m47s ago) 51m whereabouts-reconciler-njkhm 0/1 CrashLoopBackOff 14 (4m54s ago) 51m whereabouts-reconciler-pg5hr 0/1 CrashLoopBackOff 14 (5m5s ago) 51m whereabouts-reconciler-q24fw 0/1 CrashLoopBackOff 14 (4m53s ago) 51m
Expected results:
whereabouts-reconciler pods are created and Running in 4.18
$ oc get pod | grep whereabouts-reconciler whereabouts-reconciler-689jq 1/1 Running 0 33s whereabouts-reconciler-8nzcv 1/1 Running 0 33s whereabouts-reconciler-cr8qp 1/1 Running 0 33s whereabouts-reconciler-qqv6k 1/1 Running 0 33s whereabouts-reconciler-w48w7 1/1 Running 0 33s whereabouts-reconciler-xxl2f 1/1 Running 0 33s $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.nightly-2025-03-05-050145 True False 40m Cluster version is 4.18.0-0.nightly-2025-03-05-050145
Additional info:
This is a 4.19 regression bug.
Description of problem:
The UI to create a new project is asking for a "Subnet", wile expecting the CIDR for the subnet
Version-Release number of selected component (if applicable):
4.18.0-rc.4
How reproducible:
100%
Steps to Reproduce:
1. Change to dev console 2. Create new project 3. got to network tab
Actual results:
"Subnet" expects a CIDRO
Expected results:
"Subnet CIDR" expects a CIDR
Additional info:
https://ibb.co/CQX7RTJ
Containerized docs build:
We should make this podman rootless friendly and keep a tighter control on versions.
Fix opentelemetry metrics to use regexp on label match
slack thread: https://redhat-internal.slack.com/archives/C0VMT03S5/p1737575715571469
Adding new device for accelerator monitoring in order to start inventory data flowing for the current GPUs that we have on the cluster
Overview
This task is to revisit and refactor the "fairly complex code and logic"
Outcome
Simplify and refactor.
Keep separation of concerns.
Use common function - reduce duplicated code
Check why the performance slowed down when oc-mirror started to use the olm logic to filter the catalog.
As a (user persona), I want to be able to:
so that I can achieve
Description of criteria:
Detail about what is specifically not being delivered in the story
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Description of problem:
Starting OCPBUGS-41849 , "pod-security.kubernetes.io/*-version" is set to "latest". But the openshift-operator-lifecycle-manager and openshift-marketplace namespaces still use old pod-security.kubernetes.io/*-version v1.24 and v1.25 respectively. Therefore creating this Jira tracker.
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-09-26-011209 4.17.0-rc.6 4.18.0-0.nightly-2024-09-26-222528
How reproducible:
Always
Steps to Reproduce:
Check `oc get ns -o yaml` in 4.16 / 4.17 / 4.18 envs.
Actual results:
All envs show the openshift-operator-lifecycle-manager and openshift-marketplace namespaces still use old pod-security.kubernetes.io/*-version v1.24 and v1.25 respectively: - apiVersion: v1 kind: Namespace metadata: ... labels: ... pod-security.kubernetes.io/audit: baseline pod-security.kubernetes.io/audit-version: v1.25 pod-security.kubernetes.io/enforce: baseline pod-security.kubernetes.io/enforce-version: v1.25 pod-security.kubernetes.io/warn: baseline pod-security.kubernetes.io/warn-version: v1.25 name: openshift-marketplace ... - apiVersion: v1 kind: Namespace metadata: ... labels: ... pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: v1.24 name: openshift-operator-lifecycle-manager ... - apiVersion: v1 kind: Namespace metadata: ... labels: kubernetes.io/metadata.name: openshift-operators openshift.io/scc: "" pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/enforce-version: v1.24 name: openshift-operators ...
Expected results:
Like OCPBUGS-41849 sets "pod-security.kubernetes.io/*-version" to "latest" starting 4.17, the openshift-operator-lifecycle-manager and openshift-marketplace namespaces should not still use old pod-security.kubernetes.io/*-version v1.24 and v1.25 respectively.
For the openshift-operators namespace, let's mention it too here, it still uses v1.24. In despite of https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/README.md#versioning saying "The privileged profile always means fully unconstrained and is effectively unversioned (specifying a version is allowed but ignored)", it is better to not specify v1.24.
Additional info:
As an action item from our most recent retrospective, we would like to investigate how to have a better signal on unit test coverage. One possibility is Codecov.
USER STORY:
As a developer, I need to create e2e tests for the new vSphere Data Disk feature so that we have proper code coverage and meet the required metrics to allow the feature to become GA some point in the future
Required:
Need to create e2e tests that meet the metrics defined in the openshift/api project
Nice to have:
Move these e2e tests into the projects that should own them using the new external tests feature being added to origin.
ACCEPTANCE CRITERIA:
ENGINEERING DETAILS:
Lets use this opportunity to start migrating e2e tests to the project where the new features are being added which will remove the dependency of having to stage merging code in one repo and then updating e2e in origin.