Back to index

4.10.51

Jump to: Complete Features | Incomplete Features | Complete Epics | Incomplete Epics | Other Complete | Other Incomplete |

Changes from 4.9.59

Note: this page shows the Feature-Based Change Log for a release

Complete Features

These features were completed when this image was assembled

Feature CCX-137: Insights Advisor: Insights Advisor for OpenShift as standalone app

Feature Overview
Insights Advisor for OpenShift is integrated within OpenShift Cluster Manager. This has some limitations for adding new features and also for sharing codebase between RHEL Advisor and OCM Insights Advisor tab. Insights Advisor for OpenShift lacks certain features from the RHEL UI, the codebase is not 1:1 clone.
As a customer of Insights I will have same/very similar user experience with Insights for OpenShift and Insights for RHEL. The workflows will share the main concepts, the UI elements will be same and features introduced to Advisor will be automatically considered for both all supported platforms.
As OpenShift users I will still see integrations of Insights Advisor within OpenShift Cluster Manager that shows aggregated information for customer account and single cluster view on Advisor data. These integration will point to new Insights Advisor for OpenShift app that will be tightly integrated into OpenShift Cluster Manager.

  • Note: The application will be reusing the codebase but will run as a separate app for OpenShift. THere's no intent to merge RHEL and OpenShift workflows into a single app.

Goals

  • Q2CY21: Explore possibility to unify codebase between RHEL Advisor and OCM Insights Advisor tab. Identify architecture misalignments, create UI mockups to merge the two existing UIs.
  • Q3CY21: Integrate OpenShift into Advisor codebase, standup the Insights Advisor for OpenShift application and change integration in OpenShift Cluster manager to point at the new app
  • Q4CY21: Deliver missing screen of Insights Advisor for OpenShift (Systems and Recommendations views)

Requirements

  • UX overview of UI elements in both UIs - Marie Doruskova
  • Architecture overview/misalignments for both UIs - Jan Zeleny [~fjansen]

Benefits

  • Feature parity between RHEL and OpenShift
  • Adopting new features developed by RHEL Advisor team quicker
  • Smaller maintenance cost

Questions to answer...

  • Possible deviations between OpenShift and RHEL
  • Remediation workflow different between OpenShift and RHEL

Out of Scope

  • Single app that combines RHEL hosts and OpenShift clusters. Goal is still to differentiate between platforms and offer view only for a single platform.
  • Direct/Supervised remediations and integration of remediations with Advanced Cluster Manager (as a Service)

Background, and strategic fit

  • Insights Advisor for OpenShift follows the goal to introduce multiple applications that add value for OpenShift customers under the Insights brand. The current UI and integration of Advisor into OpenShift cluster manager doesn't follow pattern that other Insights for OpenShift applications can/will follow.

Documentation Considerations

  • OCM documentation is impacted, existing workflows described in OCM documentation will persist. The placement of the application within OCM will be different.

 

Epic CCXDEV-6500: OCP Advisor (frontend, CY22Q1)

TBA

Task CCXDEV-7039: Redirect OCP WebConsole users to Advisor through the links in the widget

OCP WebConsole, in the main dashboard, has an Insights Advisor widget, which has been redirecting users to OCM. Due to the Insights Advisor tab decommission in OCM, the links should point to Advisor instead.

4.10 code freeze = 28 January (marking the task as urgent)

Feature OBSDA-3: Allow non-admin users to configure individual notification settings

Problem Alignment

The Problem

Today, all configuration for setting individual, for example, routing configuration is done via a single configuration file that only admins have access to. If an environment uses multiple tenants and each tenant, for example, has different systems that they are using to notify teams in case of an issue, then someone needs to file a request w/ an admin to add the required settings.

That can be bothersome for individual teams, since requests like that usually disappear in the backlog of an administrator. At the same time, administrators might get tons of requests that they have to look at and prioritize, which takes them away from more crucial work.

We would like to introduce a more self service approach whereas individual teams can create their own configuration for their needs w/o the administrators involvement.

Last but not least, since Monitoring is deployed as a Core service of OpenShift there are multiple restrictions that the SRE team has to apply to all OSD and ROSA clusters. One restriction is the ability for customers to use the central Alertmanager that is owned and managed by the SRE team. They can't give access to the central managed secret due to security concerns so that users can add their own routing information.

High-Level Approach

Provide a new API (based on the Operator CRD approach) as part of the Prometheus Operator that allows creating a subset of the Alertmanager configuration without touching the central Alertmanager configuration file.

Please note that we do not plan to support additional individual webhooks with this work. Customers will need to deploy their own version of the third party webhooks.

Goal & Success

  • Allow users to deploy individual configurations that allow setting up Alertmanager for their needs without an administrator.

Solution Alignment

Key Capabilities

  • As an OpenShift administrator, I want to control who can CRUD individual configuration so that I can make sure that any unknown third person can touch the central Alertmanager instance shipped within OpenShift Monitoring.
  • As a team owner, I want to deploy a routing configuration to push notifications for alerts to my system of choice.

Key Flows

Team A wants to send all their important notifications to a specific Slack channel.

  • Administrator gives permission to Team A to allow creating a new configuration CR in their individual namespace.
  • Team A creates a new configuration CR.
  • Team A configures what alerts should go into their Slack channel.
  • Open Questions & Key Decisions (optional)
  • Do we want to improve anything inside the developer console to allow configuration?
Feature RHDP-291: Maintain existing portfolio priorities

Feature Overview

This Feature is a general "catch all" for the time being. There are a number of existing priorities from Q1 that should be aligned with existing priorities below but if not, assign to this feature as needed.

Goals

In order to get a better overall portfolio view, we'll leverage this Feature to gather work that doesn't fall into other existing priorities on this board. As this list grows, the portfolio priority grooming team will look to split out or handle appropriately.

Requirements

A list of specific needs or objectives that a Feature must deliver to satisfy the Feature. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts.  If a non MVP requirement slips, it does not shift the feature.

 

requirement                                                                        Notes                                                              isMvp
     
     
     
     

 

 

(Optional) Use Cases

< How will the user interact with this feature? >

< Which users will use this and when will they use it? >

< Is this feature used as part of current user interface? >

Out of Scope

 

Background, and strategic fit

< What does the person writing code, testing, documenting need to know? >

Assumptions

< Are there assumptions being made regarding prerequisites and dependencies?>

< Are there assumptions about hardware, software or people resources?>

Customer Considerations

< Are there specific customer environments that need to be considered (such as working with existing h/w and software)?>

Documentation Considerations

< What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)? >

<What does success look like?>

< Does this feature have doc impact? Possible values are: New Content, Updates to existing content,  Release Note, or No Doc Impact?>

 <If unsure and no Technical Writer is available, please contact Content Strategy. If yes, complete the following.>

  • <What concepts do customers need to understand to be successful in [action]?>
  • <How do we expect customers will use the feature? For what purpose(s)?>
  • <What reference material might a customer want/need to complete [action]?>
  • <Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available. >
  • <What is the doc impact (New Content, Updates to existing content, or Release Note)?>

Questions

Question Outcome
   

 

Epic ODC-6351: Dynamic Plugins - Round 3

Problem:

Console provides support UI for operators which is dynamically enabled when the operator is installed; by using feature flags against presence of CRDs. While operators have their own release cadence separately from OpenShift which makes for alignment of UI to API difficult. As new features are released for the operator, the UI becomes out of sync with APIs and customers must wait till the following OpenShift release to get any new UI.

Goal:

  • Create an extensibility mechanism which allows Red Hat operators to build and package their own UI that extends the console.
  • Make console extensible in areas required to support the needs of contributing plugins.

Why is it important?

  • Allows an operator to maintain their own UI and release at their own cadence.
  • Alleviates the pressure on console to deliver UI features for multiple operators within a release.

Use cases:

  1. Serverless / Pipelines / Helm to contribute resource details pages, import flows, topology visuals etc...

Acceptance criteria:

  1. Red Hat Operator can build their own UI which is deployed alongside the operator and extend the dev-console
    1. objective is to get to a point where it is possible to accomplish this however code will not be moved to a separate repository, nor deployed by an operator
  2. New extensions for console to allow operators to extend the various areas of console needed in order to provide the proper user experience.
  3. Enable operators to override the static built in support, and supply their own UI

Dependencies (External/Internal):

Design Artifacts:

Console extensions:
https://docs.google.com/document/d/1HW5_cl6cOX5P14PQN-1_8c60o9dMY6HbFDRftH6aTno/edit

Dynamic Plugins:
https://docs.google.com/document/d/19BAFo_8BtMZVvKsU-bE61bZpSydeYONkCMWntMU9NgE/edit

Enhancement proposal:
https://github.com/openshift/enhancements/pull/441

Exploration:

Note:

  • plugin framework covered by another epic
  • out of scope:
    • moving plugins to separate git repository
Story ODC-6219: Override static plugin contribution with dynamic plugin contribution

Description

As a developer, I want to be able to contribute a dynamic plugin extension and override the same extension contributed by static plugin.

Acceptance Criteria

  1. Should replace static plugin contribution of same name by dynamic plugin contribution

Additional Details:

Incomplete Features

When this image was assembled, these features were not yet completed. Therefore, only the Jira Cards included here are part of this release

Feature OCPPLAN-5652: The details of this Jira Card are restricted (Only Red Hat employees and contractors)
The details of this Jira Card are restricted (Only Red Hat employees and contractors)
Feature OCPPLAN-7878: NetEdge - Maintainability and Debugability & Tech Backlog

tldr: three basic claims, the rest is explanation and one example

  1. We cannot improve long term maintainability solely by fixing bugs.
  2. Teams should be asked to produce designs for improving maintainability/debugability.
  3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

Feature OCPPLAN-8029: Console: Dynamic Plugin Framework

Feature Overview

Plugin teams need a mechanism to extend the OCP console that is decoupled enough so they can deliver at the cadence of their projects and not be forced in to the OCP Console release timelines.

The OCP Console Dynamic Plugin Framework will enable all our plugin teams to do the following:

  • Extend the Console
  • Deliver UI code with their Operator
  • Work in their own git Repo
  • Deliver at their own cadence

Goals

    • Operators can deliver console plugins separate from the console image and update plugins when the operator updates.
    • The dynamic plugin API is similar to the static plugin API to ease migration.
    • Plugins can use shared console components such as list and details page components.
    • Shared components from core will be part of a well-defined plugin API.
    • Plugins can use Patternfly 4 components.
    • Cluster admins control what plugins are enabled.
    • Misbehaving plugins should not break console.
    • Existing static plugins are not affected and will continue to work as expected.

Out of Scope

    • Initially we don't plan to make this a public API. The target use is for Red Hat operators. We might reevaluate later when dynamic plugins are more mature.
    • We can't avoid breaking changes in console dependencies such as Patternfly even if we don't break the console plugin API itself. We'll need a way for plugins to declare compatibility.
    • Plugins won't be sandboxed. They will have full JavaScript access to the DOM and network. Plugins won't be enabled by default, however. A cluster admin will need to enable the plugin.
    • This proposal does not cover allowing plugins to contribute backend console endpoints.

 

Requirements

 

Requirement Notes isMvp?
 UI to enable and disable plugins    YES 
 Dynamic Plugin Framework in place    YES 
Testing Infra up and running   YES 
 Docs and read me for creating and testing Plugins    YES 
CI - MUST be running successfully with test automation This is a requirement for ALL features. YES
Release Technical Enablement Provide necessary release enablement details and documents. YES

 
 Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?  
  • New Content, Updates to existing content,  Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic CONSOLE-2907: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story CONSOLE-2946: Expose all of core PatternFly for dynamic plugin use

Currently, webpack tree shakes PatternFly and only includes the components used by console in its vendor bundle. We need to expose all of the core PatternFly components for use in dynamic plugin, which means we have to disable tree shaking for PatternFly. We should expose this as a separate bundle. This will allow browsers to cache more efficiently and only need to load the PF bundle again when we upgrade PatternFly.

Open Questions

What parts of PatternFly do we consider core?

Acceptance Criteria

  • All PatternFly core components are exposed to dynamic plugins
  • PatternFly is exposed as a separate bundle that is not part of the main vendor bundle

cc Christian Vogt Vojtech Szocs Joseph Caiani James Talton

Feature OCPPLAN-8030: Console: Customer Happiness (RFEs) for 4.8-4.12

Feature Overview

  • This Section:* High-Level description of the feature ie: Executive Summary
  • Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

 

Goals

  • This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature
  • …

 

Requirements

  • This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

 

Requirement Notes isMvp?
CI - MUST be running successfully with test automation This is a requirement for ALL features. YES
Release Technical Enablement Provide necessary release enablement details and documents. YES

 

(Optional) Use Cases

This Section: 

  • Main success scenarios - high-level user stories
  • Alternate flow/scenarios - high-level user stories
  • ...

 

Questions to answer…

  • ...

 

Out of Scope

  • …

 

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

 

Assumptions

  • ...

 

Customer Considerations

  • ...

 

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?  
  • New Content, Updates to existing content,  Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?
Epic CONSOLE-2893: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story CONSOLE-2360: Run Pod in Debug mode

As a user, I want the ability to run a pod in debug mode.

This should be the equivalent of running:  oc debug pod

Acceptance Criteria for MVP

  • Build off of the crash-loop back off popover from https://github.com/openshift/console/pull/7302 to include a description of what crash-loop back off is, a link to view logs, a link to view events and a link to debug (container-name) in terminal. If more than one container is crash-looping list them individually.
  • Create a debug container page that includes breadcrumbs as well as the terminal to debug. Add an informational alert at the top to make it clear that this is a temporary Pod and closing this page will delete the temporary pod.
  • Add debug in terminal as an action to the logs tool bar. Only enable the action when the crash-loop back off status occurs for the selected container. Add a tool tip to explain when the action is disabled.

Assets
Designs (WIP): https://docs.google.com/document/d/1b2n9Ox4xDNJ6AkVsQkXc5HyG8DXJIzU8tF6IsJCiowo/edit#

Feature OCPSTRAT-469: Install and upgrade OpenShift with GCP Workload Identity

OCP/Telco Definition of Done
Feature Template descriptions and documentation.
Feature Overview

  • Connect OpenShift workloads to Google services with Google Workload Identity

Goals

  • Customers want to be able to manage and operate OpenShift on Google Cloud Platform with workload identity, much like they do with AWS + STS or Azure + workload identity.
  • Customers want to be able to manage and operate operators and customer workloads on top of OCP on GCP with workload identity.

Requirements

  • Add support to CCO for the Installation and Upgrade using both UPI and IPI methods with GCP workload identity.
  • Support install and upgrades for connected and disconnected/restriction environments.
  • Support the use of Operators with GCP workload identity with minimal friction.
  • Support for HyperShift and non-HyperShift clusters.
  • This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.
Requirement Notes isMvp?
CI - MUST be running successfully with test automation This is a requirement for ALL features. YES
Release Technical Enablement Provide necessary release enablement details and documents. YES

(Optional) Use Cases

This Section:

  • Main success scenarios - high-level user stories
  • Alternate flow/scenarios - high-level user stories
  • ...

Questions to answer…

  • ...

Out of Scope

  • …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

  • ...

Customer Considerations

  • ...

Documentation Considerations

Questions to be addressed:

  • What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
  • Does this feature have doc impact?
  • New Content, Updates to existing content, Release Note, or No Doc Impact
  • If unsure and no Technical Writer is available, please contact Content Strategy.
  • What concepts do customers need to understand to be successful in [action]?
  • How do we expect customers will use the feature? For what purpose(s)?
  • What reference material might a customer want/need to complete [action]?
  • Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
  • What is the doc impact (New Content, Updates to existing content, or Release Note)?

 

Epic CCO-114: Support GCP workload identity

Epic Goal

  • Complete the implementation for GCP  workload identity, including support and documentation.

Why is this important?

  • Many customers want to follow best security practices for handling credentials.

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.

Dependencies (internal and external)

Open questions:

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story CCO-123: Update openshift operators to consume new 'external_account' type credentials

We need to ensure following things in the openshift operators

1)  Make sure to operator uses v0.0.0-20210218202405-ba52d332ba99 or later version of the golang.org/x/oauth2 module

2) Mount the oidc token in the operator pod, this needs to go in the deployment. We have done it for cluster-image-registry-operator here

3) For workload identity to work, gco credentials that the operator pod uses should be of external_account type (not service_account). The external_account credentials type have path to oidc token along, url of the service account to impersonate along with other details. These type of credentials can be generated from gcp console or programmatically (supported by ccoctl). The operator pod can then consume it from a kube secret. Make appropriate code changes to the operators so that can consume these new credentials 

 

Following repos need one or more of above changes

Feature OCPSTRAT-475: Enable sharing ConfigMaps and Secrets across namespaces [Tech Preview]

Feature Overview

Enable sharing ConfigMap and Secret across namespaces

Requirements

Requirement Notes isMvp?
Secrets and ConfigMaps can get shared across namespaces   YES

Questions to answer…

NA

Out of Scope

NA

Background, and strategic fit

Consumption of RHEL entitlements has been a challenge on OCP 4 since it moved to a cluster-based entitlement model compared to the node-based (RHEL subscription manager) entitlement mode. In order to provide a sufficiently similar experience to OCP 3, the entitlement certificates that are made available on the cluster (OCPBU-93) should be shared across namespaces in order to prevent the need for cluster admin to copy these entitlements in each namespace which leads to additional operational challenges for updating and refreshing them. 

Documentation Considerations

Questions to be addressed:
 * What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
 * Does this feature have doc impact?
 * New Content, Updates to existing content, Release Note, or No Doc Impact
 * If unsure and no Technical Writer is available, please contact Content Strategy.
 * What concepts do customers need to understand to be successful in [action]?
 * How do we expect customers will use the feature? For what purpose(s)?
 * What reference material might a customer want/need to complete [action]?
 * Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
 * What is the doc impact (New Content, Updates to existing content, or Release Note)?

Epic BUILD-293: Tech Preview Shared Resource CSI Driver

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • Deliver the Projected Resources CSI driver via the OpenShift Payload

Why is this important?

  • Projected resource shares will be a core feature of OpenShift. The share and CSI driver have multiple use cases that are important to users and cluster administrators.
  • The use of projected resources will be critical to distributing Simple Content Access (SCA) certificates to workloads, such as Deployments, DaemonSets, and OpenShift Builds.

Scenarios

As a developer using OpenShift
I want to mount a Simple Content Access certificate into my build
So that I can access RHEL content within a Docker strategy build.

As a application developer or administrator
I want to share credentials across namespaces
So that I don't need to copy credentials to every workspace

Acceptance Criteria

  • OCP conformance suite must ensure that the projected resource CSI driver is installed on every OpenShift deployment.
  • OCP build suite tests that projected resource CSI driver volumes can be added to builds. Only if builds support inline CSI volumes.
  • Release Technical Enablement - Docs and demos on how to create a Projected Resource share and add it as a volume to workloads. A special use case for adding RHEL entitlements to builds should be included.

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story BUILD-345: Expose CSI driver metrics to Telemetry

User Story

As an OpenShift engineer
I want to know which clusters are using the Shared Resource CSI Driver
So that I can be proactive in supporting customers who are using this tech preview feature

Acceptance Criteria

  • Key metrics for the shared resource CSI driver are exported to Telemeter via the cluster monitoring operator.

Docs Impact

None - metrics exported to telemetry are not formally documented.

QE Impact

QE can verify that the query/recording rule for cluster monitoring operator returns data if the cluster has the Shared Resource CSI driver installed and utilizes a SharedSecret or SharedConfigMap in a pod/workload.

PX Impact

Insights rules can potentially be created off of these exported metrics. This would allow CEE to identify which clusters are using SharedSecrets or SharedConfigMaps, especially if we are exporting mount failure metrics.

Notes

To implement, a prometheus query/recording rule needs to be added to the cluster monitoring operator. Once approved by the monitoring team, the metric data will be available on DataHub once 4.10 clusters are installed with the updated version of the monitoring operator.

Story BUILD-284: Integrate Shared Resources Operator with Cluster Storage Operator

User Story

As a cluster admin
I want the cluster storage operator to install the shared resources CSI driver
So that I can test the shared resources CSI driver on my cluster

Acceptance Criteria

  • Cluster storage operator uses image references to resolve the csi-driver-shared-resource-operator and all images needed to deploy the csi driver.
  • Shared resources CSI driver is installed when the cluster enables the CSIDriverSharedResources feature gate, OR
  • Shared resource CSI driver is installed when the cluster enables the TechPreviewNoUpgrade feature set
  • CI ensures that if the TechPreviewNoUpgrade feature set is enabled on the cluster, the shared resource CSI driver is deployed and functions correctly.

Docs Impact

Docs will need to identify how to install the shared resources CSI driver (by enabling the tech preview feature set)

Notes

Tasks:

  • Add the Share APIs (SharedSecret, SharedConfigMap) to openshift/api
  • Generate clients in openshift/client-go for Share APIs
  • Update the CSI driver name used in the enum for the ClusterCSIDriver custom resource.
  • Generate custom resource definitions and include it in the deployment YAMLs for the shared resource operator
  • Add YAML deployment manifests for the shared resource operator to the cluster storage operator (include necessary RBAC)
  • Ensure cluster storage operator has permission to create custom resource definitions
  • Enhance the cluster storage operator to install the shared resource CSI driver only when the cluster enables the CSIDriverSharedResources feature gate

Note that to be able to test all of this on any cloud provider, we need STOR-616 to be implemented. We can work around this by making the CSI driver installable on AWS or GCP for testing purposes.

The cluster storage operator has cluster-admin permissions. However, no other CSI driver managed by the operator includes a CRD for its API.

See https://issues.redhat.com/browse/BUILD-159?focusedCommentId=16360509&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16360509

Feature OCPSTRAT-526: Cloud Controller Managers: Final Testing and GA tasks - Phase 1

Feature Overview (aka. Goal Summary)  

Upstream Kuberenetes is following other SIGs by moving it's intree cloud providers to an out of tree plugin format, Cloud Controller Manager, at some point in a future Kubernetes release. OpenShift needs to be ready to action this change  

Goals (aka. expected user outcomes)

Bring together all the cloud controller managers (AWS, GCP, Azure), complete testing and prepare for final GA

Requirements (aka. Acceptance Criteria):

A list of specific needs or objectives that a feature must deliver in order to be considered complete.  Be sure to include nonfunctional requirements such as security, reliability, performance, maintainability, scalability, usability, etc.  Initial completion during Refinement status.

 

Use Cases (Optional):

Include use case diagrams, main success scenarios, alternative flow scenarios.  Initial completion during Refinement status.

 

Questions to Answer (Optional):

Include a list of refinement / architectural questions that may need to be answered before coding can begin.  Initial completion during Refinement status.

 

Out of Scope

High-level list of items that are out of scope.  Initial completion during Refinement status.

 

Background

Provide any additional context is needed to frame the feature.  Initial completion during Refinement status.

 

Customer Considerations

Provide any additional customer-specific considerations that must be made when designing and delivering the Feature.  Initial completion during Refinement status.

 

Documentation Considerations

Provide information that needs to be considered and planned so that documentation will meet customer needs.  Initial completion during Refinement status.

 

Interoperability Considerations

Which other projects and versions in our portfolio does this feature impact?  What interoperability test scenarios should be factored by the layered products?  Initial completion during Refinement status.

Epic OCPCLOUD-1224: Prepare CCCMO for General Availability

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • Prepare the Cluster Cloud Controller Manager Operator (CCCMO) component, introduced in 4.9 for GA

Why is this important?

  • We must ensure that the component is stable before we can declare the product GA

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story OCPCLOUD-1189: CCCMO: isolate provider specific logic within operator

Initial work was started there: https://github.com/lobziik/cluster-cloud-controller-manager-operator/pull/1/files

Need to isolate provider specific code in respective packages and introduce interface to leverage it (regular and bootstrap manifests rendering should be there atm)

DoD:

  • Introduce templating logic to replace existing substitution mixture
  • Isolate templating logic so that this is transparent to the core of the CCCMO
  • Improve testing of the substitution

Complete Epics

This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were completed when this image was assembled

Epic CONSOLE-2065: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Epic CONSOLE-2966: 4.10 Console Dependencies & Tech Debt

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Story CONSOLE-2985: Replace all instances of old variables controlling global grid widths and breakpoints with Patternfly variables for more consistency of spacing between elements and behaviors

The console has many instances of old variables, $grid-float-breakpoint and $grid-gutter-width, controlling margins/padding and responsive breakpoints throughout the Admin and Dev Console. These do not provide spacing and behaviors consistent with Patternfly components which use their own variables, $pf-global-gutter-md, $pf-global-gutter, and $pf-global-breakpoint-{size}. By replacing these, the intent it to bring the console closer to a pure Patternfly structure and behavior, requiring less overrides and customizations.

Story CONSOLE-2979: Upgrade Cypress to 8.5.0

Update console from Cypress 6.0.0 to 8.5.0. Changes that impact us:

  • cypress run is headless by default
  • cy.intercept URL matching is more strict
  • Uncaught exception and unhandled promise rejection checks are more strict

https://docs.cypress.io/guides/references/migration-guide#Migrating-to-Cypress-8-0

Task CONSOLE-2964: Dynamic Plugin needs to be externally consumable - Update ts-node

As an adopter of the @openshift-console/dynamic-plugin-sdk I want to easily integrate into my development pipeline so that I can extend the OCP console.

Trying to pull in the dynamic-plugin-sdk into ACM is proving to be problematic. We would have to move to older dependencies. Integrating with webpack and typescript requires a very specific setup.

The dynamic-plugin-sdk has only really been used internally by OCP and is strongly tied to the setup and dependencies of OCP. For the dynamic-plugin-sdk to be externally consumable by adopters, it should be as easy to use as other webpack plugins such as HtmlWebpackPlugin or CompressionPlugin.

Acceptance Criteria

  • Uses up to date dependencies - not tied to specific versions OCP console uses
  • Includes it's own dependencies - does not require adopters to include those dependencies
  • The dynamic demo plugin should be updated to use newer dependencies and use the plugin without a bunch of tweaks to tsconfig paths. 

Currently

  • requires old dependencies 
    • ts-node 5.0.1 → 10.2.1

 

Epic IR-208: Continuous Improvement of Maintainability 4.10

Epic Goal

  • Improve CI testing of the image registry components.

Why is this important?

  • The image registry, image API and the image pruner had a lot of tests removed during transition 4.0. This may make the platform less stable and/or slow down the team.

Scenarios

  1. ...

Acceptance Criteria

  • CI - tests should be more stable and have broader coverage

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
Story IR-104: Use library-go in image-registry

In the image-registry, we have packages origin-common and kubernetes-common. The problem is that this code doesn't get updates. We can replace them with more supported library-go.

Epic IR-210: Update k8s to 1.23

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

  • ...

Why is this important?

  • …

Scenarios

  1. ...

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • ...

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story IR-211: Bump k8s to 1.23 in image-registry repo

As a OpenShift engineer
I want image-registry to use the latest k8s libraries
so that image-registry can benefit from new upstream features.

Acceptance criteria

  • image-registry uses k8s.io/api v1.23.z
  • image-registry uses latest openshift/api, openshift/library-go, openshift/client-go
Epic JKNS-132: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story JKNS-257: Use sidecar pattern for Jenkins pod templates

User Story

As a developer using Jenkins to build my application
I want to use the base Jenkins agent image as a sidecar in my PodTemplate
So that I can use any s2i builder image in my Jenkins pipelines

Acceptance Criteria

  • Provide new Kubernetes Plugin Pod Templates which uses the sidecar pattern for NodeJS and Maven.
  • Add documentation on how to use the new pod template in a Jenkinsfile (need to specify the container where the build occurs).
  • Add documentation on how developers can provide an inline pod template within a Jenkinsfile. Documentation should have the following formats:
    • New YAML declarative format
    • Deprecated Groovy format
  • Existing pipelines that use the default Kubernetes Plugin Pod Templates do not break.
  • End to end testing (for client or sync plugin) verifies that the new pod templates work.

QE Impact

QE will need to verify that the new pod templates can successfully execute a JenkinsPipeline build.

Docs Impact

Documentation needs to be updated to explain how to use the new template.

PX Impact

Unclear if we need new CEE/PX materials beyond doc updates.

Notes

We currently have built-in pod templates for NodeJS and Maven, which use specialized agent images with NodeJS/Maven image.
Blog post here outlines the process: https://developers.redhat.com/blog/2020/06/04/an-easier-way-to-create-custom-jenkins-containers/

The Groovy style of declaring in-line pod templates is deprecated in favor of a YAML-style format.

Existing documentation for the Jenkin pod templates: https://docs.openshift.com/container-platform/4.9/openshift_images/using_images/images-other-jenkins.html#images-other-jenkins-config-kubernetes_images-other-jenkins

Epic MON-1988: Enable audit and query logging for all prometheus read paths

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

Epic Goal

  • As a CFE team, we would like to enable query logging for all Prometheus read paths
  • As part of this, we would like to enable audit & query logging for Prometheus Adapter(aggregated server audit log), Prometheus(query log) and ThanosQuerier(query log)

Why is this important?

  • This would help all parties(customers, app-sres, CCX, monitoring team,..) to debug an overloaded Prometheus instance.

Scenarios

  1. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable audit logging in Prometheus Adapter to see which component is calling metrics API
  2. When a customer faces a high cpu consumption in any of the Prometheus instance, they can enable query logging in all Prometheus instances(PM & UWM) and ThanosQuerier to see which query is frequently executed
  3. https://bugzilla.redhat.com/show_bug.cgi?id=1982302

Acceptance Criteria

  • CI - MUST be running successfully with tests automated
  • Release Technical Enablement - Provide necessary release enablement details and documents.
  • Prometheus Adapter audit logs must be enabled by default
  • Prometheus Adapter audit logs must be preserved after each CI run

Open questions::

  1. Should we enable ThanosRuler query logs?

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Task MON-1786: Allow OpenShift users to configure audit logs for prometheus-adapter

After investigating a complex Bugzilla involving many applications making queries to prometheus-adapter, we've noticed that we were lacking insights on the requests made to prometheus-adapter. To have such information for an aggregated API, the best would be to have audit logs for prometheus-adapter. This wasn't configurable before, but with https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/92, upstream users should now be able to configure it.

Since this would greatly help in investigating prometheus-adapter Bugzilla in the future, it would be great if we allowed OpenShift users to configure the audit logs so that they could provide them to us.

Note for the assignee: as of the time of the creation of this ticket, the upstream PR hasn't been merged in custom-metrics-apiserver and thus wasn't synced in prometheus-adapter. So we will have to wait a bit before starting looking into this ticket.

 

DoD:

  • Allow OpenShift users to configure audit logs for prometheus-adapter
  • Integrate with must-gather
  • Document how to configure audit logs in the official OpenShift documentation
  • Upstream jsonnet patch that enables this feature through a configuration
Epic NETOBSERV-16: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story NETOBSERV-31: Expose CNI type features as a config-map

The console requires to know the network type capabilities to show/hide some Network Policy form fields.

As a result of https://issues.redhat.com/browse/NETOBSERV-27, this logic is implemented as a features document inside the console code. The console fetches the network type from the network operator and checks the supported features towards this document.

However, this limits the feature to admin users, as other logged-in users do not have permissions to fetch the network type.

This task aims to modify the current Cluster Network Operator to expose the network capabilities as an `sdn-public` Config Map, writeable only by the SDN, readable by any `system:authenticated` user.

Enhancement Proposal PR: https://github.com/openshift/enhancements/pull/875

Epic OCPCLOUD-1256: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story OCPCLOUD-1252: Set values for validation weebhook for GuestAccelerators field in GCPProviderSpec

We want to configure 'default' and 'allowed' values in validation webhook for Guest Accelerators field in GCPProviderSpec. Also revendor it to include newly added Guest Accelerators field.

This can be done after https://github.com/openshift/cluster-api-provider-gcp/pull/172  is merged.

DoD:

  • Make sure that validations return errors on issues with GPU configuration
  • Ensure the unit tests for the webhooks are updated
Epic OCPRHV-594: [Refactor] Migrate OCP on RHV subprojects to go-ovirt-client, go-ovirt-client-log and k8sOVirtCredentialsMonitor

Description:

Openshift on RHV is composed of the following subproject the team maintains:

Each of those projects currently uses the generated oVirt API project go-ovirt.

This leads to a number of issues:

  1. Duplicated code between the subprojects: Since the go-ovirt is a thin layer around the API then a lot of the code which interacts with oVirt is duplicated between the projects, which leads to all the classic duplication problems such as maintaining the project, lack of clear conventions, and so on.
  2. Bad error handling and unclear errors:
    1. Since the go-ovirt is a thin layer there is a lot of error handling and checking which needs to be done, since a lot of the times it looks like a certain error should be ignored, it is never checked which could lead to unexpected situations.
    2. Since the errors which are returned from the oVirt Engine are sometimes unclear, when we return those errors to the users or log them is hard to understand what is the actual issue.
  3. Lack of retries: sometimes an operation can take some time due to some condition that needs to be met, or an operation can fail due to infrastructure issues, the go-ovirt library doesn't contain any retry logic which means each client needs to implement its own retry logic which is not done at the moment and will cause more duplicated code.
  4. Poor logging: The current go-ovirt library doesn't log anything, and all the logs come from the subprojects, this leads to:
    1. Inconsistent logging between the projects.
    2. Lack of logs.
  5. Almost no test coverage:
    1. It's very hard to mock and write tests with go-ovirt since there are so many calls, but will be much easier to mock and write tests with go-ovirt-clent.
    2. go-ovirt only has rudimentary tests.

Then came go-ovirt-client, go-ovirt-client-log, go-ovirt-client-log-klog and k8sOVirtCredentialsMonitor to the rescue!

The go-ovirt-client is a wrapper around the go-ovirt which contains all the error handling/retry logic/logs/tests needed to provide a decent user experience and an easy-to-use API to the oVirt engine.

go-ovirt-client-log is a library to unify the logging logic between the projects, it is used by go-ovirt-client and should be used by all the sub-projects.

go-ovirt-client-log-klog is a companion library to go-ovirt-client-log enabling logging via the Kubernetes "klog" facility.

k8sOVirtCredentialsMonitor is a utility for monitoring the oVirt credentials secret, which will automatically update the ovirt credentials is they are changed. 

We aim to move all projects which are using the go-ovirt to use go-ovirt-client, go-ovirt-client-log and k8sOVirtCredentialsMonitor instead.

Benefits for the eng:

  • Possible to write unit tests.
  • Easier to maintain since less code duplication - reduce the amount of code.
  • Test coverage exists on the ovirt-client as well.
  • No(Less) bugs regarding operations that needed a retry or polling logic.
  • Solves a number of existing bugs

Benefits for the customers:

  • Clearer error messages and logs.
  • Fewer bugs.

Acceptance criteria:

  1. All sub-projects are not using go-ovirt directly - at least 90% of the calls to go-ovirt should be migrated to go-ovirt-client.
  2. All sub-projects should use the corresponding go-ovirt-client-log for logging.
  3. All csi-driver and cluster provide use k8sOVirtCredentialsMonitor.
  4. CI tests are green for all components.

How to test:

  1. QE regression - make sure all flows are still working.
  2. Green CI on all jobs.
  3. Keep an eye out for log messages that might confuse customers.
Task OCPRHV-596: Migrate ovirt-csi-driver to go-ovirt-client

Description:

  1. Identify all the communication between ovirt-csi-driver and the go-ovirt.
  2. Port all the logic to go-ovirt-client.
  3. Port all calls on ovirt-csi-driver to go-ovirt-client.

Acceptance:

ovirt-csi-driver uses go-ovirt-client for 95% percent of all oVirt related logic.

Epic ODC-6266: Improve DevExp for front end developers

T-shirt size: M

Goal:

Provide an easy and successful experience for front end developers to build and deploy their applications

Why is it important?

Currently, the front end dev experience is not positive. It's much easier for them to use other platforms. Improving the front end dev experience will enable us to gain more marketshare

Use cases:

  1. Need to be able to override the npm command when using Node Builder Image
  2. Need to expose target port
  3. Need access to the URL to access my application

Although we provide the ability for 2 & 3 today, the current journey does not match with the mental model of the front end developer

Acceptance criteria:

  1. When importing an app, I should be able to easily provide the npm build and run commands
  2. When opting in to create a route, the target port should be exposed without having to open any Advanced Options
  3. After importing my app, if a route is exposed, I should be able to access/copy that URL

Dependencies (External/Internal):

Design Artifacts:

Desired UX experience

  • enable user to provide the *Build Command* when Node Builder image is being used
  • enable user to provide the *Run Command* when Node Builder image is being used
  • expose the Target Port under the *Create a route to the Application *rather than inside Show advanced Routing options
  • NEED TO FINALIZE HOW TO PROVIDE THE ROUTE TO EASILY COPY – Inline Notification maybe? As well as side panel?

Note:

Story ODC-6443: Add an option to add additional labels for just the Route and move the target port before the route checkbox

Description

As a user, I want have the option to add additional labels to a Route, as I could do in OCP3. See RFE-622

The additional labels should only be added to the route, not the service or other components. The advanced option "Labels" should not be touched and these labels are added to all components.

As an small additional we should also show always the "Target port" since it also defines the Service port and to make this more clear, the "Target port" should be shown before the "Create a route to the Application" checkbox.

Acceptance Criteria

The following changes should be applied to the Import flow (from Git, from Container, ...) and to the Edit page as well:

  1. Move the option "Target port" before the checkbox "Create a route to the Application" and do not hide the "Target port" when the checkbox is disabled
  2. Add a new "Additional route labels" option, with a label input field to the "Advanced Routing options"
  3. Save (Import) and update (Edit) the labels to the Route resource. When editing a Deployment with a Route the route labels should not show the shared labels.

Additional Details:

Epic ODC-6322: Automation Test plan for 4.10 Release

Problem:

This epic is mainly focused on the 4.10 Release QE activities

Goal:

1. Identify the scenarios for automation
2. Segregate the test Scenarios into smoke, Regression and other user stories
a. Update the https://docs.jboss.org/display/ODC/Automation+Status+Report
3. Align with layered operator teams for updating scripts
3. Work closely with dev team for epic automation
4. Create the automation scripts using cypress
5. Implement CI for nightly builds
6. Execute scripts on sprint basis

Why is it important?

To the track the QE progress at one place in 4.10 Release Confluence page

Use cases:

  1. <case>

Acceptance criteria:

  1. <criteria>

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Task ODC-6455: Add page tests should use latest UI labels like "Import from Git" instead of mapping "From Devfile" strings

There are different code spots which maps the old action items "From Git", "From Dockerfile" and "From Devfile" to the new action "Import from Git".

We should avoid mapping different strings to the new version and instead update our tests so that the feature and page object files matches the latest frontend code.

Code areas I found are marked with

      // TODO (ODC-6455): Tests should use latest UI labels like "Import from Git" instead of mapping strings

Incomplete Epics

This section includes Jira cards that are linked to an Epic, but the Epic itself is not linked to any Feature. These epics were not completed when this image was assembled

Epic CONSOLE-2848: Port all Protractor tests to Cypress

Epic Goal

  • Port all remaining Protractor tests to Cypress

Why is this important?

  • Protractor is very hard to debug when tests fail/flake
  • Once all protractor tests are ported we can remove all Protractor dependencies, scripts, and configuration files.
  • Cypress has better debugging, plug-ins, and reporting tools

Acceptance Criteria

  • CI - MUST be running successfully with tests automated

Dependencies (internal and external)

  1. ...

Previous Work (Optional):

  1. …

Open questions::

  1. …

Done Checklist

  • CI - CI is running, tests are automated and merged.
  • Release Enablement <link to Feature Enablement Presentation>
  • DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
  • DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
  • DEV - Downstream build attached to advisory: <link to errata>
  • QE - Test plans in Polarion: <link or reference to Polarion>
  • QE - Automated tests merged: <link or reference to automated tests>
  • DOC - Downstream documentation merged: <link to meaningful PR>
Story CONSOLE-2867: Cypress: port protractor OAuth tests

Please read: migrating-protractor-tests-to-cypress

Protractor test to migrate:  `frontend/integration-tests/tests/oauth.scenario.ts`
Large but straight forward

47) OAuth

   48) BasicAuth IDP
      ✔ creates a Basic Authentication IDP
      ✔ shows the BasicAuth IDP on the OAuth settings page

   49) GitHub IDP
      ✔ creates a GitHub IDP
      ✔ shows the GitHub IDP on the OAuth settings page

   50) GitLab IDP
      ✔ creates a GitLab IDP
      ✔ shows the GitLab IDP on the OAuth settings page

   51) Google IDP
      ✔ creates a Google IDP
      ✔ shows the Google IDP on the OAuth settings page

   52) Keystone IDP
      ✔ creates a Keystone IDP
      ✔ shows the Keystone IDP on the OAuth settings page

   53) LDAP IDP
      ✔ creates a LDAP IDP
      ✔ shows the LDAP IDP on the OAuth settings page

   54) OpenID IDP
      ✔ creates a OpenID IDP
      ✔ shows the OpenID IDP on the OAuth settings page

 Accpetance Criteria

  • Protractor test ported to cypress
  • Remove any unused legacy data-test-id`s
  • Protractor test deleted, and non longer referenced in `frontend/integration-tests/protractor.conf.ts`
Epic OCPCLOUD-737: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story OCPCLOUD-1263: Integrate openshift/API machine definitions into components

 

Background

As a follow up to OCPCLOUD-693, we need to, once all of the API definitions are present in openshift/api, migrate the existing code bases to use the new API locations.

 

This will include:

  • Machine API Operator
  • Cluster Machine Approver
  • Cluster API Provider AWS|Azure|GCP|IBM|Alibaba|OpenStack|Kubevirt
  • Cluster API actuator pkg
  • Installer
  • WMCO
  • MCO
  • Hive
  • Grep OpenShift for other references to our old APIs

Steps

  • Replace the Machine API imports with the new openshift/API MAPI locations

Stakeholders

  • Cluster Infra
  • Owners of the repos listed above

Definition of Done

  • The openshift/API defintions are used across components in the MAPI ecosystem
  • Docs
  • Generated docs for API types should now come from openshift/API
  • Testing
  • Regular regression testing should be sufficient, this is a copy paste for the most part and we expect the code won't compile if we break this
Epic ODC-6381: 4.9 Epics Automation stories tech debt

Problem:

Complete all the 4.9 epic features automation user stories and merge it to master branch.

Goal:

4.9 epics automation completion

Why is it important?

Tech debt should be completed

Use cases:

  1. <case>

Acceptance criteria:

Create the pr's for 4.9 epic user stories automation
Review it
Merge it to 4.10 master branch and 4.9 master branch

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Story ODC-6364: Epic Automation for ODC-5149 "Pipeline as Code"

Description

As a user, I want to store my delivery pipelines in a Git repository as the source of truth and execute the pipeline on OpenShift on Git events, so that I can version and trace changes to the delivery pipelines in Git.

Use Cases

  • Developer can see the list of Git repositories that are added to the namespace for pipeline-as-code execution
  • Developer can navigate from the Console to the Git repository on the Git provider
  • For each Git repository, developer can see the details of the last pipeline execution and the commit id that triggered it with possibility to navigating to the Git commit in the Git provider
  • Developer can see the list of pipelinerun executions related to a Git repository in a chronological order and the commit id that triggered each

Acceptance Criteria

  1. As a user, looking at the Pipelines page in the Developer Console, I should be able to see a list of (a) Git repositories that are added to the namespace for PAC execution AND (b) all pipelines in the namespace
  2. As a user, I should be able to navigate to a details page of the git repo.
    1. This details page should provide access to (a) details of the git repo and (b) a list of pipeline runs.
    2. This PLR tab should show additional information than the typical PLR List view, including SHA (commit id), commit message, branch & trigger type
  3. As a user, when looking at a Pipeline Run Details page, if associate with a git repo (PAC),
    1. Indicate that it's from a specific git repo rather than a PL resource
    2. Include the SHA (commit id), commit message, branch & trigger type

Other Complete

This section includes Jira cards that are not linked to either an Epic or a Feature. These tickets were completed when this image was assembled

Task IR-224: Bump openshift/api package

Acceptance criteria:

  • All tests (including e2e) pass
  • No regressions are introduced
  • openshift/api points to a recent commit on the master branch
Bug OCPBUGS-473: Various Jenkins CVEs for August 2022 [openshift-4.10.z]
  • CVE-2022-36882
  • CVE-2022-29047
  • CVE-2022-30945
  • CVE-2022-30946
  • CVE-2022-30948
  • CVE-2022-30952
  • CVE-2022-30953
  • CVE-2022-30954
  • CVE-2022-34174
  • CVE-2022-36883
  • CVE-2022-36884
  • CVE-2022-36885
  • CVE-2022-34177
  • CVE-2022-34176
  • CVE-2022-36881
Bug OCPBUGS-5345: WriteRequestBodies audit profile records routes/status events at RequestResponse level

This bug is a backport clone of [Bugzilla Bug 2073220](https://bugzilla.redhat.com/show_bug.cgi?id=2073220). The following is the description of the original bug:
—
Description of problem:

https://docs.openshift.com/container-platform/4.10/security/audit-log-policy-config.html#about-audit-log-profiles_audit-log-policy-config

Version-Release number of selected component (if applicable): 4.*

How reproducible: always

Steps to Reproduce:
1. Set audit profile to WriteRequestBodies
2. Wait for api server rollout to complete
3. tail -f /var/log/kube-apiserver/audit.log | grep routes/status

Actual results:

Write events to routes/status are recorded at the RequestResponse level, which often includes keys and certificates.

Expected results:

Events involving routes should always be recorded at the Metadata level, per the documentation at https://docs.openshift.com/container-platform/4.10/security/audit-log-policy-config.html#about-audit-log-profiles_audit-log-policy-config

Additional info:

Story CONSOLE-2892: Allow dynamic plugins to proxy to services on the cluster

Goal

We have several use cases where dynamic plugins need to proxy to another service on the cluster. One example is the Helm plugin. We would like to move the backend code for Helm to a separate service on the cluster, and the Helm plugin could proxy to that service for its requests. This is required to make Helm a dynamic plugin. Similarly if we want to have ACM contribute any views through dynamic plugins, we will need a way for ACM to proxy to its services (e.g., for Search).

It's possible for plugins to make requests to services exposed through routes today, but that has several problems:

  1. It requires that the service be exposed outside the cluster, which is not always desired.
  2. It requires the service support CORS headers for the console.
  3. There is no way to specify a CA file for the route if it's not trusted by the browser.
  4. Plugins will not have access to the user's access token on the client, which means that there is no simple way to handle auth.

Plugins need a way to declare in-cluster services that they need to connect to. The console backend will need to set up proxies to those services on console load. This also requires that the console operator be updated to pass the configuration to the console backend.

 

This work will apply only to single clusters.

 

Open Questions

  • What happens when a multitenant isolated network policy is configured on the cluster?

https://docs.openshift.com/container-platform/4.7/networking/network_policy/multitenant-network-policy.html

  • How do we (and can we?) support this for multi-cluster where console is running on a different hub cluster?
  • Do we need to auth for all requests?

Acceptance Criteria

  • Plugins can declare a service to proxy to in the ConsolePlugin resource
  • Plugins can specify a CA cert for the service
  • Console falls back to the service signing CA if none is specified
  • Plugins have a way of specifying whether the user's authentication token is included in requests through the service proxy
  • Dynamic plugin enhancement is updated with the implementation details
  • Support for server-side events (SSE) for ACM
  • Add support, or a flag, if auth is needed for each request.  

cc Ali Mobrem [~christianmvogt]

Bug OCPBUGS-3968: must-gather namespace should have “privileged” warn and audit pod security labels besides enforce

Description of problem:

https://bugzilla.redhat.com/show_bug.cgi?id=2103126

Version-Release number of selected component (if applicable):

 

How reproducible:

 

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-3996: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-4134: configure-ovs: delays boot waiting for patch port to be unmanaged

This bug is a backport clone of [Bugzilla Bug 2072040](https://bugzilla.redhat.com/show_bug.cgi?id=2072040). The following is the description of the original bug:
—
Description of problem:

configure-ovs resets network configuration on boot, and while doing so waits for all devices to become unmanaged in NetworkManager. Currently the patch port between br-int and br-ex created by ovn-kubernetes is managed by NetworkManager, never becomes unmanaged and causes an accumulated delay of 2 minutes on boot.

This patct port should never be managed by NetworkManager.

Bug OCPBUGS-4810: MCO does not sync kubeAPIServerServingCAData to controllerconfig if there are not ready nodes

This is a clone of issue OCPBUGS-3882. The following is the description of the original issue:
—
This bug is a backport clone of [Bugzilla Bug 2034883](https://bugzilla.redhat.com/show_bug.cgi?id=2034883). The following is the description of the original bug:
—
Description of problem:

Situation (starting point):

  • There is an ongoing change to the machine-config-daemon daemonset being applied by the machine-config-operator pod. It is waiting for the daemonset to roll out.
  • There are some nodes not ready, so daemonset rollout never ends and waiting on that ends in timeout error.

Problem:

  • Machine-config-operator pods stops trying to reconcile stuff whenever it finds timeout error in waiting for the machine-config-daemon rollout
  • This implies that the `spec.kubeAPIServerServingCAData` field of controllerconfig/machine-config-controller object is not updated when the kube-apiserver-operator updates kube-apiserver-to-kubelet-client-ca configmap.
  • Without that field updated, a kube-apiserver-to-kubelet-client-ca change is never rolled out to the nodes.
  • That ultimately leads to cluster-wide unavailability of "oc logs", "oc rsh" etc. commands when the kube-apiserver-operator starts using a client cert signed by the new kube-apiserver-to-kubelet-client-ca to access kubelet ports.

Version-Release number of MCO (Machine Config Operator) (if applicable):

4.7.21

Platform (AWS, VSphere, Metal, etc.): (not relevant)

Are you certain that the root cause of the issue being reported is the MCO (Machine Config Operator)?
(Y/N/Not sure): Y

How reproducible:

Always if the said conditions are met.

Steps to Reproduce:
1. Have some nodes not ready
2. Force a change that requires machine-config-daemon daemonset rollout (I think that changing proxy settings would work for this)
3. Wait until a new kube-apiserver-to-kubelet-client-ca is rolled out by kube-apiserver-operator

Actual results:

New kube-apiserver-to-kubelet-client-ca not forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca not deployed on nodes

Expected results:

kube-apiserver-to-kubelet-client-ca forwarded to controllerconfig, kube-apiserver-to-kubelet-client-ca deployed to nodes.

Additional info:

In comments

Bug OCPBUGS-5294: Backport fix for OCPBUGSM-36848 to 4.10

A bug prevents the package server cert from being rotated. This was fixed for 4.11 with the release of RHSA-2022:5069 but not fixed in 4.10 or earlier. See bz 2020484 or OCPBUGSM-36848 for details.

Bug OCPBUGS-6645: [release-4.10] openshift4/ose-jenkins:v4.10.0 run script throws too many arguments error

This is a clone of issue OCPBUGS-5947. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4833. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4819. The following is the description of the original issue:
—
Description of problem:

s2i/run script has a bug - /usr/libexec/s2i/run: line 578: [: too many arguments

Version-Release number of selected component (if applicable):

v4.10

How reproducible:

Start jenkins from the container image using /usr/libexec/s2i/run while having a route that contains a certificate or key that includes special characters.

Steps to Reproduce:

1. create a route that contains a TLS certificate
2. start a pod using openshift4/ose-jenkins:v4.10.0
3. view the log

Actual results:

2022/12/12 17:30:33 [go-init] No pre-start command defined, skip
2022/12/12 17:30:33 [go-init] Main command launched : /usr/libexec/s2i/run
CONTAINER_MEMORY_IN_MB='12288', using /usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.el8_4.x86_64/bin/java and /usr/lib/jvm/java-11-openjdk-11.0.16.0.8-1.el8_4.x86_64/bin/javac
Administrative monitors that contact the update center will remain active
Migrating slave image configuration to current version tag ...
/usr/libexec/s2i/run: line 578: [: too many arguments
Using JENKINS_SERVICE_NAME=jenkins
Generating jenkins.model.JenkinsLocationConfiguration.xml using (/var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml.tpl) ...
Jenkins URL set to: https://bojenkinsdev.micron.com/ in file: /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml
+ exec java -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:+ParallelRefProcEnabled -XX:+UseG1GC -XX:+UseStringDeduplication -XX:HeapDumpPath=/var/log/jenkins/ '-Xlog:gc*=debug:file=/var/log/jenkins-engserv/gc-%t.log:utctime:filecount=2,filesize=100m' -Xms2g -Xmx8g -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Djava.util.logging.config.file=/var/lib/jenkins/logging.properties -Djavax.net.ssl.trustStore=/var/lib/jenkins/ca-anchors-keystore -Dcom.redhat.fips=false -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= -Duser.home=/var/lib/jenkins -Djavamelody.application-name=jenkins -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true -Djenkins.install.runSetupWizard=false -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=false -XX:+AlwaysPreTouch -XX:ErrorFile=/var/log/jenkins-engserv -Dhudson.model.ParametersAction.keepUndefinedParameters=false -jar /usr/lib/jenkins/jenkins.war --prefix=/je...
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true

Expected results:

not have the following error:
/usr/libexec/s2i/run: line 578: [: too many arguments

Additional info:

 
Bug OCPBUGS-2628: Worker creation fails within provider networks (as primary and secondary)

This is a clone of issue OCPBUGS-2508. The following is the description of the original issue:
—
Description of problem:

Installer fails due to Neutron policy error when creating Openstack servers for OCP master nodes.

$ oc get machines -A
NAMESPACE               NAME                          PHASE          TYPE   REGION   ZONE   AGE
openshift-machine-api   ostest-kwtf8-master-0         Running                               23h
openshift-machine-api   ostest-kwtf8-master-1         Running                               23h
openshift-machine-api   ostest-kwtf8-master-2         Running                               23h
openshift-machine-api   ostest-kwtf8-worker-0-g7nrw   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-lrkvb   Provisioning                          23h
openshift-machine-api   ostest-kwtf8-worker-0-vwrsk   Provisioning                          23h

$ oc -n openshift-machine-api logs machine-api-controllers-7454f5d65b-8fqx2 -c machine-controller
[...]
E1018 10:51:49.355143       1 controller.go:317] controller/machine_controller "msg"="Reconciler error" "error"="error creating Openstack instance: Failed to create port err: Request forbidden: [POST https://overcloud.redhat.local:13696/v2.0/ports], error message: {\"NeutronError\": {\"type\": \"PolicyNotAuthorized\", \"message\": \"(rule:create_port and (rule:create_port:allowed_address_pairs and (rule:create_port:allowed_address_pairs:ip_address and rule:create_port:allowed_address_pairs:ip_address))) is disallowed by policy\", \"detail\": \"\"}}" "name"="ostest-kwtf8-worker-0-lrkvb" "namespace"="openshift-machine-api"

Version-Release number of selected component (if applicable):

4.10.0-0.nightly-2022-10-14-023020

How reproducible:

Always

Steps to Reproduce:

1. Install 4.10 within provider networks (in primary or secondary interface)

Actual results:

Installation failure:
4.10.0-0.nightly-2022-10-14-023020: some cluster operators have not yet rolled out

Expected results:

Successful installation

Additional info:

Please find must-gather for installation on primary interface link here and for installation on secondary interface link here.

 

Bug OCPBUGS-2717: Devfile Catalog and Import a Devfile on a fully disconnected cluster should fail directly instead of timeout after 30sec

Description of problem:
Users on a fully-disconnected cluster could not see Devfiles in the developer catalog or import a Devfiles. That's fine.

But the API calls /api/devfile/samples/ and /api/devfile/ takes 30 seconds until they fail with a 504 Gateway timeout error.

If possible they should fail immediately.

Version-Release number of selected component (if applicable):
This might happen since 4.8

Tested this yet only on 4.12.0-0.nightly-2022-09-07-112008

How reproducible:
Always

Steps to Reproduce:

  1. Start a disconnected cluster with a proxy
  2. Open the browser network inspector and filter for /api/devfile
  3. Switch to Developer perspective
    1. Navigate to Add > Developer Catalog (All Services) > Devfiles
    2. Or Add > Import from Git > and enter https://github.com/devfile-samples/devfile-sample-go-basic.git

Actual results:

  • Network call fails after 30 seconds
  • Import doesn't work

Expected results:

  • Network calls should fail immediately
  • We doesn't expect that the import will work

Additional info:
The console Pod log contains this error:

E0909 10:28:18.448680 1 devfile-handler.go:74] Failed to parse devfile: failed to populateAndParseDevfile: Get "https://registry.devfile.io/devfiles/go": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Bug OCPBUGS-3103: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-4086: Kube-State-metrics pod fails to start due to panic

The kube-state-metric pod inside the openshift-monitoring namespace is not running as expected.

On checking the logs I am able to see that there is a memory panic

~~~
2022-11-22T09:57:17.901790234Z I1122 09:57:17.901768 1 main.go:199] Starting kube-state-metrics self metrics server: 127.0.0.1:8082
2022-11-22T09:57:17.901975837Z I1122 09:57:17.901951 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.902389844Z I1122 09:57:17.902291 1 main.go:210] Starting metrics server: 127.0.0.1:8081
2022-11-22T09:57:17.903191857Z I1122 09:57:17.903133 1 main.go:66] levelinfomsgTLS is disabled.http2false
2022-11-22T09:57:17.906272505Z I1122 09:57:17.906224 1 builder.go:191] Active resources: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
2022-11-22T09:57:17.917758187Z E1122 09:57:17.917560 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2022-11-22T09:57:17.917758187Z goroutine 24 [running]:
2022-11-22T09:57:17.917758187Z k8s.io/apimachinery/pkg/util/runtime.logPanic(

{0x1635600, 0x2696e10})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d
2022-11-22T09:57:17.917758187Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
2022-11-22T09:57:17.917758187Z panic({0x1635600, 0x2696e10}

)
2022-11-22T09:57:17.917758187Z /usr/lib/golang/src/runtime/panic.go:1038 +0x215
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.ingressMetricFamilies.func6(0x40)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/internal/store/ingress.go:136 +0x189
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/internal/store.wrapIngressFunc.func1(

{0x17fe520, 0xc00063b590})
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/internal/store/ingress.go:175 +0x49
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/pkg/metric_generator.(*FamilyGenerator).Generate(...)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:67
2022-11-22T09:57:17.917758187Z k8s.io/kube-state-metrics/v2/pkg/metric_generator.ComposeMetricGenFuncs.func1({0x17fe520, 0xc00063b590}

)
2022-11-22T09:57:17.917758187Z /go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:107 +0xd8
~~~

Logs are attached to the support case

Bug OCPBUGS-550: Input values in Instantiate Template are disappeared randomly in the developer console

#Description of problem:

Developer Console > +ADD > Develoeper Catalog > Service > select Types Templates > Initiate Template

Input values in Instantiate Template are disappeared randomly.

#Version-Release number of selected component (if applicable):

  • Customer ENV
  • OCP4.10.9
  • Developer Console
  • Edge 88x. / Edge 85.0.x / Chrome 97.x /Chrome 88.x
  • Internet Disconnected OCP cluster
  • quicklab test ENV
  • Developer Console
  • OCP4.10.12 
  • Chrome 100

#How reproducible:

I reproduced this issue in ocp410ovn shared cluster in the quicklab

Select Apache HTTP Server > Input name "test" in Application Hostname box
After several seconds, the value has disappeared in the web console.

#Steps to Reproduce:

0. Developer Console > +ADD > Develoeper Catalog > Service > select Types Templates > Initiate Template

1. Input values in the box of template menu.

2. The values are disappeared after several seconds later. (20s~ or randomly)

3. Many users have experienced this issue.

  • The web browser version of users experiencing this issue.
  • Customer: Edge 88x. / Edge 85.0.x / Chrome 97.x /Chrome 88.x
  • My browser: Chrome 102.x

==> the browser version doesn't matter.

#Actual results:

Input values in "Instantiate Template" are disappeared randomly.
Users can't use the Initiate Template feature in the Dev console.

#Expected results:
Input values remain in the web console and users creat the object by the "Instantiate Template"

#Additional info:

See "Application Name" has disappeared in the video I attached.

Bug OCPBUGS-2037: OpenShift webconsole , metrics tab does not display data

Description of problem:

After upgrading the cluster to version 4.10.32 the web console and metrics tab  does not display any data.

we have to restart the thanos pods to get the functionality back to running state


Version-Release number of selected component (if applicable):

4.10.32

How reproducible:

NA

Steps to Reproduce:

NA

Actual results:

The data is not getting displayed and monitoring of the cluster is hampered

Expected results:

The data should be displayed properly without having to restart the pods

Additional info:

 

Bug OCPBUGS-2802: [release-4.10] Add required parameter --credentials-requests-dir for ccoctl gcp delete

In order to delete the correct GCP cloud resources, the "--credentials-requests-dir" parameter must be passed to "ccoctl gcp delete". This was fixed for 4.12 as part of https://github.com/openshift/cloud-credential-operator/pull/489 but must be backported for previous releases. See https://github.com/openshift/cloud-credential-operator/pull/489#issuecomment-1248733205 for discussion regarding this bug.

To reproduce, create GCP infrastructure with a name parameter that is a subset of another set of GCP infrastructure's name parameter. I will "ccoctl gcp create all" with "name=abutcher-gcp" and "name=abutcher-gcp1".

$ ./ccoctl gcp create-all \
--name=abutcher-gcp \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

$ ./ccoctl gcp create-all \
--name=abutcher-gcp1 \
--region=us-central1 \
--project=openshift-hive-dev \
--credentials-requests-dir=./credrequests

Running "ccoctl gcp delete --name=abutcher-gcp" will result in GCP infrastructure for both "abutcher-gcp" and "abutcher-gcp1" being deleted. 

$ ./ccoctl gcp delete --name abutcher-gcp --project openshift-hive-dev
2022/10/24 11:30:06 Credentials loaded from file "/home/abutcher/.gcp/osServiceAccount.json"
2022/10/24 11:30:06 Deleted object .well-known/openid-configuration from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 Deleted object keys.json from bucket abutcher-gcp-oidc
2022/10/24 11:30:07 OIDC bucket abutcher-gcp-oidc deleted
2022/10/24 11:30:09 IAM Service account abutcher-gcp-openshift-image-registry-gcs deleted
2022/10/24 11:30:10 IAM Service account abutcher-gcp-openshift-gcp-ccm deleted
2022/10/24 11:30:11 IAM Service account abutcher-gcp1-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:12 IAM Service account abutcher-gcp-openshift-machine-api-gcp deleted
2022/10/24 11:30:13 IAM Service account abutcher-gcp-openshift-ingress-gcp deleted
2022/10/24 11:30:15 IAM Service account abutcher-gcp-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:16 IAM Service account abutcher-gcp1-openshift-ingress-gcp deleted
2022/10/24 11:30:17 IAM Service account abutcher-gcp1-openshift-image-registry-gcs deleted
2022/10/24 11:30:19 IAM Service account abutcher-gcp-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:20 IAM Service account abutcher-gcp1-openshift-gcp-pd-csi-driver-operator deleted
2022/10/24 11:30:21 IAM Service account abutcher-gcp1-openshift-gcp-ccm deleted
2022/10/24 11:30:22 IAM Service account abutcher-gcp1-cloud-credential-operator-gcp-ro-creds deleted
2022/10/24 11:30:24 IAM Service account abutcher-gcp1-openshift-machine-api-gcp deleted
2022/10/24 11:30:25 IAM Service account abutcher-gcp-openshift-cloud-network-config-controller-gcp deleted
2022/10/24 11:30:25 Workload identity pool abutcher-gcp deleted

 

Bug OCPBUGS-1936: [4.10] Bootimage bump tracker

Tracker bug for bootimage bump in 4.10. This bug should block bugs which need a bootimage bump to fix.

Bug OCPBUGS-1538: Make northd probe interval default to 10 seconds

Description of problem:

Tracking this for backport of https://bugzilla.redhat.com/show_bug.cgi?id=2072710

Version-Release number of selected component (if applicable):

 

How reproducible:

 

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-205: [release-4.10] Add support for BUILDAH_QUIET environment variable

Description of problem:
Previously we had a bug opened for "Reduce buildah log level for default build log level [NEEDINFO]":

https://bugzilla.redhat.com/show_bug.cgi?id=1996883

We suggested customer using secrets, however, customer confirmed that they are using secrets as per:
https://docs.openshift.com/container-platform/4.8/cicd/builds/creating-build-inputs.html#builds-input-secrets-configmaps_creating-build-inputs

But not able to use "--quiet" build argument since they are using openshift s2i config:

strategy:
type: Source
sourceStrategy:
from:
kind: ImageStreamTag
namespace: xxxxxxx
name: 's2i-xxxxxx-xxxxxxx:v1.0.0'

Actual results:
Under this setting, the secret as well as every other openshift secret are printed.

Expected results:
Sensitive information (ENV) should not appear in build logs

Additional info:
Maybe pass the --quiet option via the buildconfig fir s2i?

— Additional comment from taxu@redhat.com on 2022-07-01 03:42:46 UTC —

Hi Build team,

I see that the target release for this bug is 4.11.0

Please let us know where to add the LOG_LEVEL for s2i and/or passing "--quiet" options once the fix is deployed.

Kind regards,

Tao Xu

— Additional comment from aos-team-art-private@redhat.com on 2022-07-13 06:44:30 UTC —

Elliott changed bug status from MODIFIED to ON_QA.
This bug is expected to ship in the next 4.12 release.

— Additional comment from rdlugyhe@redhat.com on 2022-07-26 20:34:06 UTC —

Please approve the updated Doc Text.

— Additional comment from cdaley@redhat.com on 2022-07-26 20:49:05 UTC —

Approved

— Additional comment from jitsingh@redhat.com on 2022-07-27 04:48:18 UTC —

verified

— Additional comment from oarribas@redhat.com on 2022-08-15 12:11:23 UTC —

@Corey, I can see this BZ now verified for 4.12. Can this be backported to previous OCP versions?

— Additional comment from cdaley@redhat.com on 2022-08-15 13:57:02 UTC —

What version are you interested in getting it backported to?

— Additional comment from oarribas@redhat.com on 2022-08-15 15:46:10 UTC —

@Corey, I can see only 4.10 and 4.11 as "Full Support" in [1], so at least to those versions.

Bug OCPBUGS-2553: [release-4.10] member loses rights after some other user login in openid / group sync

This is a clone of issue OCPBUGS-533. The following is the description of the original issue:
—
Description of problem:

customer is using Azure AD as openid provider and groups synchronization from the provider.

The scenario is the following:

1)

  • user A login.
    groups are created with the membership.
    User A is member of a group with admin rights and it's cluster-admin

2)

  • user B login:
    groups are updated with membership
    UserB is also member of the group with admin rights and it's cluster admin

3)

  • user A login:
    groups are identical as in the former step.
    user A has no administration rights.

The groups memberships are the same in step 2 and 3.
The cluster role bindings of the groups have never changed.

the only way to have user A again the admin rights is to delete the membership from the group and have user A login again.

I have not managed to reproduce this using RH SSO. Neither Azure AD.

But my configuration is not exactly the same yet.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

 

Bug OCPBUGS-5291: Default to floating automaticRestart for new GCP instances

This is a clone of issue OCPBUGS-4579. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-4504. The following is the description of the original issue:
—
This is a clone of issue OCPBUGS-1557. The following is the description of the original issue:
—
Seen in an instance created recently by a 4.12.0-ec.2 GCP provider:

  "scheduling": {
    "automaticRestart": false,
    "onHostMaintenance": "MIGRATE",
    "preemptible": false,
    "provisioningModel": "STANDARD"
  },

From GCP's docs, they may stop instances on hardware failures and other causes, and we'd need automaticRestart: true to auto-recover from that. Also from GCP docs, the default for automaticRestart is true. And on the Go provider side, we doc:

If omitted, the platform chooses a default, which is subject to change over time, currently that default is "Always".

But the implementing code does not actually float the setting. Seems like a regression here, which is part of 4.10:

$ git clone https://github.com/openshift/machine-api-provider-gcp.git
$ cd machine-api-provider-gcp
$ git log --oneline origin/release-4.10 | grep 'migrate to openshift/api'
44f0f958 migrate to openshift/api

But that's not where the 4.9 and earlier code is located:

$ git branch -a | grep origin/release
  remotes/origin/release-4.10
  remotes/origin/release-4.11
  remotes/origin/release-4.12
  remotes/origin/release-4.13

Hunting for 4.9 code:

$ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.9.48-x86_64 | grep gcp
  gcp-machine-controllers                        https://github.com/openshift/cluster-api-provider-gcp                       c955c03b2d05e3b8eb0d39d5b4927128e6d1c6c6
  gcp-pd-csi-driver                              https://github.com/openshift/gcp-pd-csi-driver                              48d49f7f9ef96a7a42a789e3304ead53f266f475
  gcp-pd-csi-driver-operator                     https://github.com/openshift/gcp-pd-csi-driver-operator                     d8a891de5ae9cf552d7d012ebe61c2abd395386e

So looking there:

$ git clone https://github.com/openshift/cluster-api-provider-gcp.git
$ cd cluster-api-provider-gcp
$ git log --oneline | grep 'migrate to openshift/api'
...no hits...
$ git grep -i automaticRestart origin/release-4.9  | grep -v '"description"\|compute-gen.go'
origin/release-4.9:vendor/google.golang.org/api/compute/v1/compute-api.json:        "automaticRestart": {

Not actually clear to me how that code is structured. So 4.10 and later GCP machine-API providers are impacted, and I'm unclear on 4.9 and earlier.

Bug OCPBUGS-912: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-2660: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-900: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Story SPLAT-246: [vsphere] Ensure clear user agent strings set for components calling to vSphere API

*USER STORY:*

 As a customer or OpenShift engineer, I want to see the user agent for anything calling from OpenShift -> vSphere to eliminate troubleshooting guesswork.

*DESCRIPTION:*

A question in #forum-vmware was raised where we identified that the user-agent may not be configured for all OpenShift components calling to vSphere API.

https://coreos.slack.com/archives/CH06KMDRV/p1627368902058800

*Required:*

Audit of OpenShift components calling to vSphere API to make sure user agent strings are set appropriately.

*Nice to have:*

How can this be prevented in the future? How can we minimize maintenance costs added by new PRs/bugs reported from this spike?

*ACCEPTANCE CRITERIA:*

New PRs or bug reports for each effected component.

Bug OCPBUGS-286: The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
Bug OCPBUGS-3161: "CatalogSource not found" in the virtualization overview

Description of problem:

In the WebUI for the Virtualization Overview, the details of "Service name", "Provider", and "Update Channel" have no value displayed. The "OpenShift Virtulization version" is showing "Cannot update CatalogSource not found"

Version-Release number of selected component (if applicable):

v4.10.4

How reproducible:

In 3 environments that have recently been deployed all show the same thing. 100%

Steps to Reproduce:

1. Install the OpenShift Virtualization operator from the WedUI
2. Use the suggested options
3.

Actual results:

The Details card is showing a warning that it cannot update

Expected results:

The Details card should have all values provided.

Additional info:

 
Bug OCPBUGS-3619: List pages in pipelines is taking more time to load when there are too many items

This is a clone of issue OCPBUGS-2077. The following is the description of the original issue:
—
Description of problem:

Pipeline list page fetches all the pipelineruns to find the last pipeline run and which results in more load time. This performance issue needs to be addressed in all the pieplines list pages wherever applicable.

Version-Release number of selected component (if applicable):
4.9

How reproducible:
Always

Steps to Reproduce:
1. Create 10+ pipelines in a namespace
2. Create more number of pipelineruns under each pipeline
3. navigate to piplines list page.

Actual results:
Pipelines list will take a long time to load the list.

Expected results:

Pipeline list should not take more time to load the list.

Additional info:

Reduce the amount to data fetched to find the last pipelinerun, maybe use PartialMetadata to find the latest pipeline run and to improve the performance.

Bug OCPBUGS-377: downloading govc is impacted by github rate limiting

This is a clone of issue OCPBUGS-262. The following is the description of the original issue:
—
github rate limit failures for upi image downloading govc.

https://search.ci.openshift.org/?search=build+error%3A+error+building+at+STEP+%22RUN+curl+-L+-o&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Bug OCPBUGS-1758: [4.11] ETCD Operator goes degraded when a second internal node ip is added

This is a clone of issue OCPBUGS-1354. The following is the description of the original issue:
—
This was originally reported in BZ as https://bugzilla.redhat.com/show_bug.cgi?id=2046335

—

Description of problem:

The issue reported here https://bugzilla.redhat.com/show_bug.cgi?id=1954121 still occur (tested on OCP 4.8.11, the CU also verified that the issue can happen even with OpenShift 4.7.30, 4.8.17 and 4.9.11)

How reproducible:

Attach a NIC to a master node will trigger the issue

Steps to Reproduce:
1. Deploy an OCP cluster (I've tested it IPI on AWS)
2. Attach a second NIC to a running master node (in my case "ip-10-0-178-163.eu-central-1.compute.internal")

Actual results:

~~~
$ oc get node ip-10-0-178-163.eu-central-1.compute.internal -o json | jq ".status.addresses"
[

{ "address": "10.0.178.163", "type": "InternalIP" }

,

{ "address": "10.0.187.247", "type": "InternalIP" }

,

{ "address": "ip-10-0-178-163.eu-central-1.compute.internal", "type": "Hostname" }

,

{ "address": "ip-10-0-178-163.eu-central-1.compute.internal", "type": "InternalDNS" }

]

$ oc get co etcd
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
etcd 4.8.11 True False True 31h

$ oc get co etcd -o json | jq ".status.conditions[0]"

{ "lastTransitionTime": "2022-01-26T15:47:42Z", "message": "EtcdCertSignerControllerDegraded: [x509: certificate is valid for 10.0.178.163, not 10.0.187.247, x509: certificate is valid for ::1, 10.0.178.163, 127.0.0.1, ::1, not 10.0.187.247]", "reason": "EtcdCertSignerController_Error", "status": "True", "type": "Degraded" }

~~~

Expected results:

To have the certificate valid also for the second IP (the newly created one "10.0.187.247")

Additional info:

Deleting the following secrets seems to solve the issue:
~~~
$ oc get secret n openshift-etcd | grep kubernetes.io/tls | grep ^etcd
etcd-client kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 61s
etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 60s
etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal kubernetes.io/tls 2 58s
etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal kubernetes.io/tls 2 59s
etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal kubernetes.io/tls 2 58s

$ oc get secret n openshift-etcd | grep kubernetes.io/tls | grep ^etcd | awk '

{print $1}

' | xargs -I {} oc delete secret {} -n openshift-etcd
secret "etcd-client" deleted
secret "etcd-peer-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-peer-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-ip-10-0-202-187.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-132-49.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-178-163.eu-central-1.compute.internal" deleted
secret "etcd-serving-metrics-ip-10-0-202-187.eu-central-1.compute.internal" deleted

$ oc get co etcd -o json | jq ".status.conditions[0]"

{ "lastTransitionTime": "2022-01-26T15:52:21Z", "message": "NodeControllerDegraded: All master nodes are ready\nEtcdMembersDegraded: No unhealthy members found", "reason": "AsExpected", "status": "False", "type": "Degraded" }

~~~

Bug OCPBUGS-2132: openshift template cakephp-mysql-persistent has a typo in apiVersion for DeploymentConfig

Description of problem:

openshift template cakephp-mysql-persistent has a typo in apiVersion for DeploymentConfig

Version-Release number of selected component (if applicable):

4.10

How reproducible:

This bug was resolved for ocp 4.11, Now there is requirement to backport this fix to ocp4.10

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-2623: Backport Request for BZ 2037513

Description of problem:

View https://bugzilla.redhat.com/show_bug.cgi?id=2037513

Version-Release number of selected component (if applicable):

 

How reproducible:

 

Steps to Reproduce:

1.
2.
3.

Actual results:

 

Expected results:

 

Additional info:

 
Bug OCPBUGS-4861: [4.10] Network Policy executes duplicate transactions for every pod update

Description of problem:

With every pod update we are executing a mutate operation to add the pod port to the port group or add the pod IP to an address set. This functionally doesn't hurt, since mutate will not add duplicate values to the same set. However, this is bad for performance. For example, with a 730 network policies affecting a pod, and issuing 7 pod updates would result in over 5k transactions.
Bug OCPBUGS-1440: [release-4.10] Jenkins install-plugins script does not ignore updates to locked plugin versions

Description of problem:

Jenkins install-plugins.sh script does not ignore update requests for locked versions of plugins, and does not verify that the locked version was actually included in the bundle-plugins.txt file.

Version-Release number of selected component (if applicable):





How reproducible:




Run make plugins-list





Steps to Reproduce:




1.
2.
3.





Actual results:








Expected results:








Additional info:







            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2202:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            IR-227:
            
             Remove legacy code for platformStatus
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Before platformStatus, the operator used to get information about AWS and GCP from the install-config config map. This code can be removed.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            CONSOLE-2768:
            
             console-operator should use bindata instead of inlining manifests
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
console-operator codebase contains a lot of inline manifests. Instead we should put those manifests into a `/bindata` folder, from which they will be read and then updated per purpose.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4136:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5950:
            
             wal: max entry size limit exceeded
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-5876. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5761. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5458. The following is the description of the original issue: 

—

reported in https://coreos.slack.com/archives/C027U68LP/p1673010878672479




Description of problem:




Hey guys, I have a openshift cluster that was upgraded to version 4.9.58 from version 4.8. After the upgrade was done, the etcd pod on master1 isn't coming up and is crashlooping. and it gives the following error: {"level":"fatal","ts":"2023-01-06T12:12:58.709Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"wal: max entry size limit exceeded, recBytes: 13279, fileSize(313430016) - offset(313418480) - padBytes(1) = entryLimit(11535)","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/remote-source/cachito-gomod-with-deps/app/server/etcdmain/main.go:40\nmain.main\n\t/remote-source/cachito-gomod-with-deps/app/server/main.go:32\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}





Version-Release number of selected component (if applicable):










How reproducible:










Steps to Reproduce:




1.
2.
3.





Actual results:










Expected results:










Additional info:









            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1987:
            
             Insight Operator keeps rebooting on Openshift 4.10.18 in disconnected environment
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Insight Operator keeps rebooting on Openshift 4.10.18 in disconnected environment




Version-Release number of selected component (if applicable):




Openshift Container Platform 4.10.18




How reproducible:




Install Openshift 4.10.18 in disconnected environment




Steps to Reproduce:




1. Download the global cluster pull secret to your local file system.
 2. In a text editor, edit the .dockerconfigjson file that was downloaded. 3. Remove the cloud.openshift.com JSON entry
4. Observe that insight operator keeps rebooting 




Actual results:




Insight Operator keeps rebooting




Expected results:




Insight Operator should be stable




Additional info:




Disconnected Environment 




 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2279:
            
             [release-4.10] Add ccoctl option to create private s3 bucket with OIDC configurations served through public CloudFront URL
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug represents a backport of CCO-222 to release-4.10.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3743:
            
             Installer fails to build on ppc64le with golang 1.17.12
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Building the installer image started to fail on ppc64le




Version-Release number of selected component (if applicable):




4.10.0




How reproducible:




Always on the brew builders




Steps to Reproduce:




1.
2.
3.





Actual results:




2022-11-14 11:55:00,819 - atomic_reactor.tasks.binary_container_build - INFO - + go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=v4.10.0 -X github.com/openshift/installer/pkg/version.Commit=4e8922c14379dce4845f362ac3a83ff80f1dc655 -X github.com/openshift/installer/pkg/version.defaultArch=ppc64le -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install
2022-11-14 11:58:01,556 - atomic_reactor.tasks.binary_container_build - INFO - # github.com/openshift/installer/cmd/openshift-install
2022-11-14 11:58:01,556 - atomic_reactor.tasks.binary_container_build - INFO - github.com/aliyun/alibaba-cloud-sdk-go/services/cms.(*Client).DescribeMonitoringAgentHostsWithChan.func1: direct call too far: runtime.duffzero+1f0-tramp0 -20000e8




Expected results:




Successful build




Additional info:




This could potentially be a golang issue. After some preliminary investigation, we tried setting `-linkmode=external` in the ldflags which seems to allow the build to finish.
So while we investigate the real cause of the issue, changing the link mode will serve as a workaround to unblock the build pipeline.



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4160:
            
             Fails to deprovision cluster when swift omits 'content-type'
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem: This is a follow-up to OCPBUGS-2795 and OCPBUGS-2941.



The installer fails to destroy the cluster when the OpenStack object storage omits 'content-type' from responses. This can happen on responses with HTTP status code 204, where a reverse proxy is truncating content-related headers (see this nginX bug report). In such cases, the Installer errors with:




level=error msg=Bulk deleting of container "5ifivltb-ac890-chr5h-image-registry-fnxlmmhiesrfvpuxlxqnkoxdbl" objects failed: Cannot extract names from response with content-type: []





Listing container object suffers from the same issue as listing the containers and this one isn't fixed in latest versions of gophercloud. I've reported https://github.com/gophercloud/gophercloud/issues/2509 and fixing it with https://github.com/gophercloud/gophercloud/issues/2510, however we likely won't be able to backport the bump to gophercloud master back to release-4.8 so we'll have to look for alternatives.



I'm setting the priority to critical as it's causing all our jobs to fail in master.



Version-Release number of selected component (if applicable):




4.8.z





How reproducible:




Likely not happening in customer environments where Swift is exposed directly. We're seeing the issue in our CI where we're using a non-RHOSP managed cloud.





Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5112:
            
             Fails to deprovision cluster when swift omits 'content-type' and there are empty containers
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-5078. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5019. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-4941. The following is the description of the original issue: 

—

Description of problem: This is a follow-up to OCPBUGS-3933.



The installer fails to destroy the cluster when the OpenStack object storage omits 'content-type' from responses, and a container is empty.



Version-Release number of selected component (if applicable):




4.8.z





How reproducible:




Likely not happening in customer environments where Swift is exposed directly. We're seeing the issue in our CI where we're using a non-RHOSP managed cloud.





Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5961:
            
             Add support for API version v1beta1 for knativeServing and knativeEventing
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-5258. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5191. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5164. The following is the description of the original issue: 

—

Description of problem:



It looks like the ODC doesn't register KNATIVE_SERVING and KNATIVE_EVENTING flags. Those are based on KnativeServing and KnativeEventing CRs, but they are looking for v1alpha1 version of those: https://github.com/openshift/console/blob/f72519fdf2267ad91cc0aa51467113cc36423a49/frontend/packages/knative-plugin/console-extensions.json#L6-L8

This PR https://github.com/openshift-knative/serverless-operator/pull/1695 moved the CRs to v1beta1, and that breaks that ODC discovery.



Version-Release number of selected component (if applicable):



Openshift 4.8, Serverless Operator 1.27



Additional info:



https://coreos.slack.com/archives/CHGU4P8UU/p1671634903447019



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            MON-1949:
            
             Improve prometheus-adapter consistency
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
The current integration of prometheus-adapter in OpenShift uses the platform Prometheus as a backend to get metrics. The problem with this design is that we are getting metrics from 2 different Prometheus instances which don't have replicated data, so two queries sent at the same time to prometheus-adapter might yield different results since the underlying promQL queries executed by prometheus-adapter might be on different Prometheus servers. The consequence is that we end up having inconsistent data across multiple autoscaling requests.



This can be easily tested by running:




$ while true ; do date; oc adm top pod -n openshift-monitoring  prometheus-k8s-0 ; echo; sleep 1 ;done 

Mon Jul 26 03:55:07 EDT 2021
NAME               CPU(cores)   MEMORY(bytes)   
prometheus-k8s-0   208m         4879Mi          

Mon Jul 26 03:55:08 EDT 2021                               
NAME               CPU(cores)   MEMORY(bytes)   
prometheus-k8s-0   246m         4877Mi          

Mon Jul 26 03:55:09 EDT 2021                               
NAME               CPU(cores)   MEMORY(bytes)   
prometheus-k8s-0   208m         4879Mi          

Mon Jul 26 03:55:10 EDT 2021
NAME               CPU(cores)   MEMORY(bytes)   
prometheus-k8s-0   246m         4877Mi          






This isn't a bug in itself since it was designed that way, but we could do better by using thanos-querier as a backend instead of the platform Prometheus because it will duplicate the metrics from both instances and serve one consistent result based on the data that it will get from the Prometheuses.



DoD:


    
	
  • Use thanos-querier as a backend for prometheus-adapter
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2902:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1737:
            
             [OCPonOpenstack] Remove clustername length limitation
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Currently when installing Openshift on the Openstack cluster name length limit is allowed to  14 characters.
Customer wants to know if is it possible to override this validation when installing Openshift on Openstack and create a cluster name that is greater than 14 characters.

Version : OCP 4.8.5 UPI Disconnected 
Environment : Openstack 16 

Issue:
User reports that they are getting error for OCP cluster in Openstack UPI, where the name of the cluster is > 14 characters.

Error events :
~~~
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/openshift-install", "create", "manifests", "--dir=/home/gitlab-runner/builds/WK8mkokN/0/CPE/SKS/pipelines/non-prod/ocp4-openstack-build/ocpinstaller/install-upi"], "delta": "0:00:00.311397", "end": "2022-09-03 21:38:41.974608", "msg": "non-zero return code", "rc": 1, "start": "2022-09-03 21:38:41.663211", "stderr": "level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters", "stderr_lines": ["level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: metadata.name: Invalid value: \"sks-osp-inf-cpe-1-cbr1a\": cluster name is too long, please restrict it to 14 characters"], "stdout": "", "stdout_lines": []}
~~~




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




 




Actual results:




Users are getting error "cluster name is too long" when clustername contains more than 14 characters for OCP on Openstack




Expected results:




The 14 characters limit should be change for the OCP clustername on Openstack




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2448:
            
             Downward API (annotations) is missing PCI information when using the tuning metaPlugin on SR-IOV Networks
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2092839](https://bugzilla.redhat.com/show_bug.cgi?id=2092839). The following is the description of the original bug:

—

Description of problem:



Application of customer uses the Downward API (/etc/podnetinfo/annotations) to get the pci-addresses for SR-IOV interfaces. However, when it adds the tuning metaplugin for sysctl configuration on the SR-IOV Networks, it does not get the pci-addresses for the SR-IOV interfaces.basically customer want to fetch the PCI-ID to bind the interface with the application and dpdk driver.




Version-Release number of selected component (if applicable):



OCP 4.7




Additional info:



OCP : 4.7

Customer: Ericsson

Case: 03221272

Customer has shared all the logs attached with the case.



Please do let us know, if you are able to access the attached log in the case.



For more information, please refer the case.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2464:
            
             Add unit-test and gofmt support for ovn-kubernetes
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Add the ability to run utests and linter jobs in downstream ovn-kubernetes




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3485:
            
             Console shouldn't try to install dynamic plugins if permissions aren't available
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3300. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-3265. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-3172. The following is the description of the original issue: 

—

Customer is trying to install the Logging operator, which appears to attempt to install a dynamic plugin. The operator installation fails in the console because permissions aren't available to "patch resource consoles".



We shouldn't block operator installation if permission issues prevent dynamic plugin installation.



This is an OSD cluster, presumably for a customer with "cluster-admin", although it may be a paired down permission set called "dedicated-admin".



See https://docs.google.com/document/d/1hYS-bm6aH7S6z7We76dn9XOFcpi9CGYcGoJys514YSY/edit for permissions investigation work on OSD

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1638:
            
             Increase default value of cache TTL to 6 hours
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Running the discovery cache every 10 minutes has a significant productivity impact on using kubectl on clusters with many CRDs as it is taking time to run these unnecessary requests.
The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers.





Version-Release number of selected component (if applicable):




4.10.z





How reproducible:




Cluster with a lot of CRDs





Steps to Reproduce:




https://github.com/kubernetes/kubernetes/issues/107130





Actual results:




Significant grow of kubectl request completion every 10 minutes






Expected results:




Significant grow of kubectl request completion every 24 hours






Additional info:




https://github.com/kubernetes/kubernetes/issues/107130




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2973:
            
             `oc explain` output of default overlaySize is outdated
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2081119](https://bugzilla.redhat.com/show_bug.cgi?id=2081119). The following is the description of the original bug:

—

Description of problem:



`oc explain` output of default overlaySize is 10G. The outdated doc confused the customer with default overlaySize. The description should be updated since the default was removed from the implementation of storage.conf.



Version-Release number of selected component (if applicable):




How reproducible:




Actual results:



$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.logLevel

KIND:     ContainerRuntimeConfig

VERSION:  machineconfiguration.openshift.io/v1



FIELD:    logLevel <string>



DESCRIPTION:

     logLevel specifies the verbosity of the logs based on the level it is set

     to. Options are fatal, panic, error, warn, info, and debug.

[qiwan@qiwan ~]$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.overlaySize

KIND:     ContainerRuntimeConfig

VERSION:  machineconfiguration.openshift.io/v1



FIELD:    overlaySize <string>



DESCRIPTION:

     overlaySize specifies the maximum size of a container image. This flag can

     be used to set quota on the size of container images. (default: 10GB).




Expected results:



$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.logLevel

KIND:     ContainerRuntimeConfig

VERSION:  machineconfiguration.openshift.io/v1



FIELD:    logLevel <string>



DESCRIPTION:

     logLevel specifies the verbosity of the logs based on the level it is set

     to. Options are fatal, panic, error, warn, info, and debug.

[qiwan@qiwan ~]$ oc explain containerRuntimeConfig.spec.containerRuntimeConfig.overlaySize

KIND:     ContainerRuntimeConfig

VERSION:  machineconfiguration.openshift.io/v1



FIELD:    overlaySize <string>



DESCRIPTION:

     overlaySize specifies the maximum size of a container image. This flag can

     be used to set quota on the size of container images.





Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-466:
            
             PDB warning alert when CR replica count is set to zero (edit)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
copy of BZ https://bugzilla.redhat.com/show_bug.cgi?id=2053622



Description of problem:



PodDisruptionBudgetAtLimit Warning alert when CR replica count is zero.



Version-Release number of selected component (if applicable):

4.7



How reproducible: Everytime




Steps to Reproduce:

1. oc new-project test



2. oc new-app httpd



3. oc create -f pdb



$ cat pdb.yaml



~~~

apiVersion: policy/v1beta1 

kind: PodDisruptionBudget

metadata:

  name: my-pdb

spec:

  maxUnavailable: 1

  selector:  

    matchLabels:

      deployment: httpd

~~~



$ oc get pdb

NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE

my-pdb   N/A             0                 0                     3h27m




4. oc scale deployment httpd --replicas=0



5. Wait for some time alert will be triggered at the console.



Actual results: unexpected warning alert




Expected results: As we are intentionally dropping down the replicas it should not generate an alert.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3778:
            
             [4.10] ovn-k network policy races
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Network policy code has some problems, most of them are races, therefore it can be difficult to reproduce and verify, here is the list

1. all kinds of add/delete port to/from default deny port group failures, possible symptoms:
  - port should’ve been added to default deny port group, but wasn’t: connections that should’ve been dropped are allowed
  - port should’ve been deleted from default deny port group, but wasn’t: connections that should be allowed are dropped
  - db ops failures when an attempt to add/delete port to/from default deny port group fails, e.g. because this operation already was done
2. default deny port group was overwritten when 2 network policies are created in a namespace at the same time. Can lead to ports not being added to the default deny port group => denied connections will be allowed
3. handle error when getting local pod from the cache fails, possible symptoms
  - "Failed to get LSP after multiple retries for pod %s/%s for networkPolicy" log message
  - pod is not added to netpol port groups, network policy is not applied
4. creating deleted namespace via ensureNamespaceLocked, symptoms:
  - namespace was deleted, but address set is present in the db
5. policy acl loglevel update wasn’t applied, possible symptoms:
  - netpol acl log level isn’t set/updated to namespace loglevel
6. netpol cleanup failures, symptoms:
  - network policy failed to be deleted, something is still left in the db, error messages like
  - "failed to destroy network policy"
  - "Rollback of default port groups and acls for policy: %s/%s failed, Unable to ensure namespace for network policy"
7. concurrent write to sets.String - this will panic, you won’t miss
8. retry for network policy handler after network policy was deleted, you should see failures saying that some network policy related object is nil or doesn’t exist, e.g.
  - "peer AddressSet is nil, cannot add <object>"
9. host network and completed pods selected by network policy can produce error logs, no real harm
  - "Failed to get LSP for pod <namespace>/<name> for networkPolicy %s refetching err"
10. namespace pod handlers are never stopped, can affect memory usage and look like a memory leak
11. add local pod failure, since netpol port group is not committed to db yet, error looks like
  - "Failed to create *factory.localPodSelector <name>, error: object not found"






Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




Example 1
1. Create network policy with [in/e]gress selector that applies to a namespace labeled project: myproject
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: test
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              project: myproject

2. Use oc apply to delete network policy and crate a pod in project: myproject namespace at the same time
3. check ovnkube-master logs for "peer AddressSet is nil, cannot add peer pod(s)", this should retry with the same error 15 times
4. This may not work from the first try, since we need to hit specific order of network policy delete and pod add handling
5. With the new version no error messages should be present

Example 2
1. create network policy that applies to a namespace test
piVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: test
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
2. Create host network pod in namespace test
3. Check 15 logs saying "Failed to get LSP for pod %s/%s for networkPolicy %s refetching err: "
4. check final log "Failed to get LSP after multiple retries for pod %s/%s for networkPolicy"
5. With the new version no error message should be present

All the other cases are difficult to reproduce, maybe just running some standard network policy tests and making sure everything works will be a good verification.





Actual results:




 




Expected results:




 




Additional info:




Only some parts were backported to 4.10 due to significant releases differences.
The problems that are fixed, and perf improvement:
1. don't retry unscheduled pod, wait for update event instead. 
2.  Cleanup pod handler for namespaceAndPod handler on namespace delete
event.
3. Only update localPods after successful db transaction, 
return fast from localPod handlers based on `np.localPods`
4. Don't retry fetching lsp from the lspCache, that "stops" the handler for 1 second
5.  Use stored portUUID to delete local pods instead of getting that info
from lspCache



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4839:
            
             [4.10] Pod LSP missing from PortGroup
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




For some reason, the LSP of a pod is not properly added to the port group where the ACL of a NetworkPolicy is applied. This results on the networkpolicy not being applied to the pod and communication not possible.





Version-Release number of selected component (if applicable):




4.10





How reproducible:




Always with a concrete pod at customer environment.





Steps to Reproduce:




(not known exactly yet)





Actual results:




LSP not in port group. ACL not applied. Netpol not in effect.





Expected results:




LSP in port group. ACL applied. Netpol in effect.





Additional info:




Details in private comments, as they involve sensitive data.

Deleting the pod does nothing, but it is possible that this has something to do with the pod being recreated with the same name (although the LSPs UUIDs are different in each incarnation).




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2898:
            
             Backport fix(openshift): use env var instead of clusterversion status to OCP 4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is for the OCP 4.10 backport of the bugfix for this issue.



--------------------------------------------------------------

Description of problem:

Cluster attempted to upgrade from 4.9.29 -> 4.10.15. The upgrade could not progress because of:



Last Transition Time: 2022-06-15T16:08:33Z

Message: ClusterServiceVersions blocking cluster upgrade: redhat-rhoam-operator/managed-api-service.v1.22.0 is incompatible with OpenShift minor versions greater than 4.10

Reason: IncompatibleOperatorsInstalled

Status: False

Type: Upgradeable



The mentioned CSV has olm.maxOpenShiftVersion of 4.10:



*

{ "type": "olm.maxOpenShiftVersion", "value": "4.10" }

*



Actual results:

The upgrade is blocked on this olm.maxOpenShiftVersion



Expected results:

The upgrade should not be blocked on this olm.maxOpenShiftVersion



Additional info:

The particular customer has successfully upgraded twice on different clusters on the same OCP edge and with managed-api-service.v1.22.0 installed



must-gather will be attached in a private comment



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1676:
            
             Improve prometheus-adapter consistency
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



This bugs purpose is to enable a feature backport of https://issues.redhat.com/browse/MON-1949.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2185:
            
             Bump Jenkins version to 2.361.1
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1942. The following is the description of the original issue: 

—

Description of problem:




Bump Jenkins version to 2.361.1 and also test the images built by running verify-jenkins.sh script. This script verifies the jenkins versions and plugin in an image. Verify script is present at https://gist.githubusercontent.com/coreydaley/fbf11d3b1a7a567f8c494da6a07bad41/raw/80e569131479c212d5e023bc41ce26fb15a17752/verify-jenkins.sh 




Version-Release number of selected component (if applicable):




2.361.1




Additional info:



Verify script is present at https://gist.githubusercontent.com/coreydaley/fbf11d3b1a7a567f8c494da6a07bad41/raw/80e569131479c212d5e023bc41ce26fb15a17752/verify-jenkins.sh 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1873:
            
             Tag all resources created by CMO e2e tests
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
The CMO e2e tests create a bunch of resources. These should be cleaned up on a successful run. However:


    
	
  • Some test failures leave the create resource behind, which have to be cleaned up before a re-run.
    • 
	
  • There have been developer reports that even successful runs don't tidy up everything.
    • 





In a CI context this is rarely a problem, however running the tests locally can be made quite awkward, especially repeated runs on the same cluster.



We should tag all resources created by the e2e tests with a label (app.kubernetes.io/created-by: cmo-e2e-test).

This will allow easy cleanup by deleting all resources with that label and will allow for checking proper clean-up.



DoD:

All e2e resources get properly tagged.

It is straight forward to ensure that future code changes don't skip adding this tag.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-761:
            
             [OCPonRHV] CSI provisioned disks are effectively preallocated due to go-ovirt-client setting Provisioned and Initial size of the disk to the same value
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-515. The following is the description of the original issue: 

—

When a thin provisioned COW format disk is created on OCP on RHV via CSI driver (a PVC - 

https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml


    
	
  • with for example requested storage 100 GB), the go-ovirt-client behaviour makes it so that the created disk has virtual size 100 GB and it's actual size is 110 GB.
    • 





But this is thin provisioned disk, so the initial size of the disk should be default of the engine and then grow as needed, it shouldn't be this big. 



This causes all the disks created this way to be functionally preallocated (since it eats all that space), which is a real waste of space. 



How reproducible: 100%



Steps to Reproduce:

1. Create a storage claim (PVC) in Openshift (

https://github.com/openshift/ovirt-csi-driver/blob/master/deploy/example/storage-claim.yaml

) using the default storage class (or any other storage class with thinProvisioning: "true") and with requested storage i.e. 100Gi



$ oc create -f storage-claim.yaml



2. In the RHV web console navigate to Storage -> Disks and check Virtual size and Actual size of the created disk (PVC) 



Actual results:

Disk from our example with requested storage 100GB reports virtual size 100GB and actual size 110 GB. 



Expected results:

Thin provisioned disks should start with small initial size and then grow as needed, so its actual size should be considerably smaller (the default initial size set by the engine should be 2.5 GB if I'm not mistaken). 



Note: The extra 10GB in the actual size are caused by overhead for the qcow2 disk format, which is 10%, and this was tracked here as a separate issue: 

https://bugzilla.redhat.com/show_bug.cgi?id=2097139

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-642:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2800:
            
             [4.10] etcd and kube-apiserver pods get restarted due to failed liveness probes while deleting/re-creating pods on SNO
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-2113. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-1329. The following is the description of the original issue: 

—

Description of problem:




etcd and kube-apiserver pods get restarted due to failed liveness probes while deleting/re-creating pods on SNO




Version-Release number of selected component (if applicable):




4.10.32




How reproducible:




Not always, after ~10 attempts




Steps to Reproduce:




1. Deploy SNO with Telco DU profile applied
2. Create multiple pods with local storage volumes attached(attaching yaml manifest)
3. Force delete and re-create pods 10 times





Actual results:




etcd and kube-apiserver pods get restarted, making to cluster unavailable for a period of time




Expected results:




etcd and kube-apiserver do not get restarted




Additional info:




Attaching must-gather.

Please let me know if any additional info is required. Thank you!



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-724:
            
             Update blueocean-autofavorite to 1.2.5
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Update blueocean-autofavorite to version 1.2.5, which does not bundle the groovy-sandbox library

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-938:
            
             machineConfigPool without nodes assigned stuck at 0% progress
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




When a custom machineConfigPool is created and no node is associated with it, the mcp remains at 0% progress.




Version-Release number of selected component (if applicable):




 




How reproducible:




100%




Steps to Reproduce:




1. Create a custom mcp:
~~~
cat << EOF | oc create -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: custom
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,custom]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/custom: "" 
EOF
~~~





Actual results:




The mcp is visible from the from "Administrator view > Cluster Settings > Details" at 0% progress




Expected results:




It shouldn't be stuck at 0% 




Additional info:




 




 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1446:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-233:
            
             Upgrade failing because restrictive scc is injected into version pod
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



OCP Upgrade failing



Version-Release number of the following components:



oc version

Client Version: 4.8.0-202108312109.p0.git.0d10c3f.assembly.stream-0d10c3f

Server Version: 4.10.13

Kubernetes Version: v1.23.5+b463d71



How reproducible: Always



Steps to Reproduce:

1. Create the following SCC (that has `with readOnlyRootFilesystem: true`):

~~~

cat << EOF | oc create -f -

allowHostDirVolumePlugin: true

allowHostIPC: false

allowHostNetwork: false

allowHostPID: false

allowHostPorts: false

allowPrivilegeEscalation: true

allowPrivilegedContainer: true

allowedCapabilities: []

apiVersion: security.openshift.io/v1

defaultAddCapabilities: []

fsGroup:

  type: MustRunAs

groups: []

kind: SecurityContextConstraints

metadata:

  annotations:

    meta.helm.sh/release-name: azure-arc

    meta.helm.sh/release-namespace: default

  labels:

    app.kubernetes.io/managed-by: Helm

  name: kube-aad-proxy-scc

priority: null

readOnlyRootFilesystem: true

requiredDropCapabilities: []

runAsUser:

  type: RunAsAny

seLinuxContext:

  type: MustRunAs

supplementalGroups:

  type: RunAsAny

users:


    
	
  • system:serviceaccount:azure-arc:azure-arc-kube-aad-proxy-sa
    
volumes:
    • 
	
  • configMap
    • 
	
  • hostPath
    • 
	
  • secret
    
EOF
    
~~~
    • 





2. oc adm upgrade --to=4.10.20




Actual results:



SCC kube-aad-proxy-scc, which has readOnlyRootFilesystem is injected inside the pod version-4.10.20-smvt9-6vqwc, causing it to fail.

~~~


    
	
  1. oc get po -n openshift-cluster-version
    
NAME                                        READY   STATUS    RESTARTS   AGE
    
cluster-version-operator-6b5c8ff5c8-4bmxx   1/1     Running   0          33m
    
version-4.10.20-smvt9-6vqwc                 0/1     Error     0          10s
    2. 
	
  2. oc logs version-4.10.20-smvt9-6vqwc -n openshift-cluster-version
    
oc logs version-4.10.20-smvt9-6vqwc 
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_00_namespace.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_adminack_configmap.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_admingate_configmap.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusteroperator.crd.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusterversion.crd.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_02_roles.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_00_cluster-version-operator_03_deployment.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_90_cluster-version-operator_00_prometheusrole.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_90_cluster-version-operator_01_prometheusrolebinding.yaml': Read-only file system
    
mv: cannot remove '/manifests/0000_90_cluster-version-operator_02_servicemonitor.yaml': Read-only file system
    
mv: cannot remove '/manifests/0001_00_cluster-version-operator_03_service.yaml': Read-only file system
    
~~~
    3. 





Expected results:



Pod version-4.10.20-smvt9-6vqwc should run fine



Additional info:



I don't know why, but SCC kube-aad-proxy-scc is injected inside pod version-4.10.20-smvt9-6vqwc:

~~~

apiVersion: v1

kind: Pod

metadata:

  annotations:

    k8s.v1.cni.cncf.io/network-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.129.0.70"

          ],

          "default": true,

          "dns": {}

      }]

    k8s.v1.cni.cncf.io/networks-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.129.0.70"

          ],

          "default": true,

          "dns": {}

      }]

    openshift.io/scc: kube-aad-proxy-scc             ### HERE

  creationTimestamp: "2022-07-25T16:47:39Z"

  generateName: version-4.10.20-5xqtv-

  labels:

    controller-uid: ba707bbe-1825-4f80-89ce-f6bf2301a812

    job-name: version-4.10.20-5xqtv

  name: version-4.10.20-5xqtv-9gcwk

  namespace: openshift-cluster-version

  ownerReferences:


    
	
  • apiVersion: batch/v1
    
    blockOwnerDeletion: true
    
    controller: true
    
    kind: Job
    
    name: version-4.10.20-5xqtv
    
    uid: ba707bbe-1825-4f80-89ce-f6bf2301a812
    
  resourceVersion: "40040"
    
  uid: 0d668d3d-7452-463f-a421-4dfee9c89c23
    
spec:
    
  containers:
    • 
	
  • args:
    • 
	
  • -c
    • 
	
  • mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /manifests /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/manifests
    
      && mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /release-manifests
    
      /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/release-manifests
    
    command:
    • 
	
  • /bin/sh
    
    image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
    
    imagePullPolicy: IfNotPresent
    
    name: payload
    
    resources:
    
      requests:
    
        cpu: 10m
    
        ephemeral-storage: 2Mi
    
        memory: 50Mi
    
    securityContext:
    
      privileged: true
    
      readOnlyRootFilesystem: true
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: File
    
    volumeMounts:
    • 
	
  • mountPath: /etc/cvo/updatepayloads
    
      name: payloads
    • 
	
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
      name: kube-api-access-fwblb
    
      readOnly: true
    
  dnsPolicy: ClusterFirst
    
  enableServiceLinks: true
    
  imagePullSecrets:
    • 
	
  • name: default-dockercfg-smmf4
    
  nodeName: ip-10-0-215-206.eu-central-1.compute.internal
    
  nodeSelector:
    
    node-role.kubernetes.io/master: ""
    
  preemptionPolicy: PreemptLowerPriority
    
  priority: 1000000000
    
  priorityClassName: openshift-user-critical
    
  restartPolicy: OnFailure
    
  schedulerName: default-scheduler
    
  securityContext:
    
    fsGroup: 1000030000
    
    seLinuxOptions:
    
      level: s0:c6,c0
    
  serviceAccount: default
    
  serviceAccountName: default
    
  terminationGracePeriodSeconds: 30
    
  tolerations:
    • 
	
  • key: node-role.kubernetes.io/master
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/not-ready
    
    operator: Exists
    
    tolerationSeconds: 300
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/unreachable
    
    operator: Exists
    
    tolerationSeconds: 300
    • 
	
  • effect: NoSchedule
    
    key: node.kubernetes.io/memory-pressure
    
    operator: Exists
    
  volumes:
    • 
	
  • hostPath:
    
      path: /etc/cvo/updatepayloads
    
      type: ""
    
    name: payloads
    • 
	
  • name: kube-api-access-fwblb
    
    projected:
    
      defaultMode: 420
    
      sources:
    • 
	
  • serviceAccountToken:
    
          expirationSeconds: 3607
    
          path: token
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: ca.crt
    
            path: ca.crt
    
          name: kube-root-ca.crt
    • 
	
  • downwardAPI:
    
          items:
    • 
	
  • fieldRef:
    
              apiVersion: v1
    
              fieldPath: metadata.namespace
    
            path: namespace
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: service-ca.crt
    
            path: service-ca.crt
    
          name: openshift-service-ca.crt
    
status:
    
  conditions:
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-07-25T16:47:39Z"
    
    status: "True"
    
    type: Initialized
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-07-25T16:47:39Z"
    
    message: 'containers with unready status: [payload]'
    
    reason: ContainersNotReady
    
    status: "False"
    
    type: Ready
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-07-25T16:47:39Z"
    
    message: 'containers with unready status: [payload]'
    
    reason: ContainersNotReady
    
    status: "False"
    
    type: ContainersReady
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-07-25T16:47:39Z"
    
    status: "True"
    
    type: PodScheduled
    
  containerStatuses:
    • 
	
  • containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
    
    image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
    
    imageID: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
    
    lastState:
    
      terminated:
    
        containerID: cri-o://fdac85e975eb00a3abd08e18061ae3673a857769ddfc87ca94a3527a8c7b83f3
    
        exitCode: 1
    
        finishedAt: "2022-07-25T16:47:42Z"
    
        reason: Error
    
        startedAt: "2022-07-25T16:47:42Z"
    
    name: payload
    
    ready: false
    
    restartCount: 2
    
    started: false
    
    state:
    
      terminated:
    
        containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
    
        exitCode: 1
    
        finishedAt: "2022-07-25T16:47:56Z"
    
        reason: Error
    
        startedAt: "2022-07-25T16:47:56Z"
    
  hostIP: 10.0.215.206
    
  phase: Running
    
  podIP: 10.129.0.70
    
  podIPs:
    • 
	
  • ip: 10.129.0.70
    
  qosClass: Burstable
    
  startTime: "2022-07-25T16:47:39Z"
    
~~~
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4128:
            
             catalog-operator fatal error: concurrent map writes
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2117324](https://bugzilla.redhat.com/show_bug.cgi?id=2117324). The following is the description of the original bug:

—

+++ This bug was initially created as a clone of Bug #2101357 +++



Description of problem:



        message: "her.go:105 +0xe5\ncreated by k8s.io/apimachinery/pkg/watch.NewStreamWatcher\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76

          +0x130\n\ngoroutine 5545 [select, 7 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc00240a780,

          0xc003321a00)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345

          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0xc002efea80?,

          0xc0009cc7a0?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207

          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136

          +0x30a\n\ngoroutine 5678 [select, 3 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc000b70480,

          0xc0035d4500)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345

          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0x6e5326?, 0xc002999e90?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207

          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136

          +0x30a\n\ngoroutine 5836 [select, 1 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc003b00180,

          0xc003ff8a00)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345

          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0x6e5326?, 0xc003a1c8d0?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207

          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136

          +0x30a\n\ngoroutine 5905 [chan receive, 1 minutes]:\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver.(*sourceInvalidator).GetValidChannel.func1()\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver/source_registry.go:51

          +0x85\ncreated by github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver.(*sourceInvalidator).GetValidChannel\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver/source_registry.go:50

          +0x231\n"

        reason: Error

        startedAt: "2022-06-27T00:00:59Z"




Version-Release number of selected component (if applicable):

mac:~ jianzhang$ oc exec catalog-operator-66cb8fd8c5-j7vkx – olm --version

OLM version: 0.19.0

git commit: 8c2bd46147a90d58e98de73d34fd79477769f11f

mac:~ jianzhang$ oc get clusterversion

NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS

version   4.11.0-0.nightly-2022-06-25-081133   True        False         10h     Cluster version is 4.11.0-0.nightly-2022-06-25-081133



How reproducible:

always



Steps to Reproduce:

1. Install OCP 4.11

2. Check OLM pods




Actual results:



mac:~ jianzhang$ oc get pods

NAME                                      READY   STATUS      RESTARTS     AGE

catalog-operator-66cb8fd8c5-j7vkx         1/1     Running     2 (8h ago)   10h

collect-profiles-27605340-wgsvf           0/1     Completed   0            42m

collect-profiles-27605355-ffgxd           0/1     Completed   0            27m

collect-profiles-27605370-w7ds7           0/1     Completed   0            12m

olm-operator-6cfd444b8f-r5q4t             1/1     Running     0            10h

package-server-manager-66589d4bf8-csr7j   1/1     Running     0            10h

packageserver-59977db6cf-nkn5w            1/1     Running     0            10h

packageserver-59977db6cf-nxbnx            1/1     Running     0            10h




mac:~ jianzhang$ oc get pods catalog-operator-66cb8fd8c5-j7vkx  -o yaml

apiVersion: v1

kind: Pod

metadata:

  annotations:

    k8s.v1.cni.cncf.io/network-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.26"

          ],

          "default": true,

          "dns": {}

      }]

    k8s.v1.cni.cncf.io/networks-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.26"

          ],

          "default": true,

          "dns": {}

      }]

    openshift.io/scc: nonroot-v2

    seccomp.security.alpha.kubernetes.io/pod: runtime/default

  creationTimestamp: "2022-06-26T23:12:45Z"

  generateName: catalog-operator-66cb8fd8c5-

  labels:

    app: catalog-operator

    pod-template-hash: 66cb8fd8c5

  name: catalog-operator-66cb8fd8c5-j7vkx

  namespace: openshift-operator-lifecycle-manager

  ownerReferences:


    
	
  • apiVersion: apps/v1
    
    blockOwnerDeletion: true
    
    controller: true
    
    kind: ReplicaSet
    
    name: catalog-operator-66cb8fd8c5
    
    uid: bcf173be-97bc-4152-8cec-d45f820e167c
    
  resourceVersion: "67395"
    
  uid: bb34c37b-b22d-4412-bbc0-1e57a1b2bd3a
    
spec:
    
  containers:
    • 
	
  • args:
    • 
	
  • --namespace
    • 
	
  • openshift-marketplace
    • 
	
  • --configmapServerImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:033bd57b54bb2e827b0ae95a03cb6f7ceeb65422e85de635185cfed88c17b2b4
    • 
	
  • --opmImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:033bd57b54bb2e827b0ae95a03cb6f7ceeb65422e85de635185cfed88c17b2b4
    • 
	
  • --util-image
    • 
	
  • quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d42dfe153a5bfa3a66e5e9d4664b4939aed376b4afbd9e098291c259daca2f8
    • 
	
  • --writeStatusName
    • 
	
  • operator-lifecycle-manager-catalog
    • 
	
  • --tls-cert
    • 
	
  • /srv-cert/tls.crt
    • 
	
  • --tls-key
    • 
	
  • /srv-cert/tls.key
    • 
	
  • --client-ca
    • 
	
  • /profile-collector-cert/tls.crt
    
    command:
    • 
	
  • /bin/catalog
    
    env:
    • 
	
  • name: RELEASE_VERSION
    
      value: 4.11.0-0.nightly-2022-06-25-081133
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d42dfe153a5bfa3a66e5e9d4664b4939aed376b4afbd9e098291c259daca2f8
    
    imagePullPolicy: IfNotPresent
    
    livenessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    name: catalog-operator
    
    ports:
    • 
	
  • containerPort: 8443
    
      name: metrics
    
      protocol: TCP
    
    readinessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    resources:
    
      requests:
    
        cpu: 10m
    
        memory: 80Mi
    
    securityContext:
    
      allowPrivilegeEscalation: false
    
      capabilities:
    
        drop:
    • 
	
  • ALL
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: FallbackToLogsOnError
    
    volumeMounts:
    • 
	
  • mountPath: /srv-cert
    
      name: srv-cert
    
      readOnly: true
    • 
	
  • mountPath: /profile-collector-cert
    
      name: profile-collector-cert
    
      readOnly: true
    • 
	
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
      name: kube-api-access-7txv8
    
      readOnly: true
    
  dnsPolicy: ClusterFirst
    
  enableServiceLinks: true
    
  nodeName: ip-10-0-173-128.ap-southeast-1.compute.internal
    
  nodeSelector:
    
    kubernetes.io/os: linux
    
    node-role.kubernetes.io/master: ""
    
  preemptionPolicy: PreemptLowerPriority
    
  priority: 2000000000
    
  priorityClassName: system-cluster-critical
    
  restartPolicy: Always
    
  schedulerName: default-scheduler
    
  securityContext:
    
    runAsNonRoot: true
    
    runAsUser: 65534
    
    seLinuxOptions:
    
      level: s0:c20,c0
    
    seccompProfile:
    
      type: RuntimeDefault
    
  serviceAccount: olm-operator-serviceaccount
    
  serviceAccountName: olm-operator-serviceaccount
    
  terminationGracePeriodSeconds: 30
    
  tolerations:
    • 
	
  • effect: NoSchedule
    
    key: node-role.kubernetes.io/master
    
    operator: Exists
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/unreachable
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/not-ready
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoSchedule
    
    key: node.kubernetes.io/memory-pressure
    
    operator: Exists
    
  volumes:
    • 
	
  • name: srv-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: catalog-operator-serving-cert
    • 
	
  • name: profile-collector-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: pprof-cert
    • 
	
  • name: kube-api-access-7txv8
    
    projected:
    
      defaultMode: 420
    
      sources:
    • 
	
  • serviceAccountToken:
    
          expirationSeconds: 3607
    
          path: token
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: ca.crt
    
            path: ca.crt
    
          name: kube-root-ca.crt
    • 
	
  • downwardAPI:
    
          items:
    • 
	
  • fieldRef:
    
              apiVersion: v1
    
              fieldPath: metadata.namespace
    
            path: namespace
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: service-ca.crt
    
            path: service-ca.crt
    
          name: openshift-service-ca.crt
    
status:
    
  conditions:
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-26T23:16:59Z"
    
    status: "True"
    
    type: Initialized
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T01:19:09Z"
    
    status: "True"
    
    type: Ready
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T01:19:09Z"
    
    status: "True"
    
    type: ContainersReady
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-26T23:16:59Z"
    
    status: "True"
    
    type: PodScheduled
    
  containerStatuses:
    • 
	
  • containerID: cri-o://f3fce12480556b5ab279236c290acdf15e1bc850426078730dcfd333ecda6795
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d42dfe153a5bfa3a66e5e9d4664b4939aed376b4afbd9e098291c259daca2f8
    
    imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8d42dfe153a5bfa3a66e5e9d4664b4939aed376b4afbd9e098291c259daca2f8
    
    lastState:
    
      terminated:
    
        containerID: cri-o://5e3ce04f1cf538f679af008dcb62f9477131aae1772f527d6207fdc7bb247a55
    
        exitCode: 2
    
        finishedAt: "2022-06-27T01:19:08Z"
    
        message: "her.go:105 +0xe5\ncreated by k8s.io/apimachinery/pkg/watch.NewStreamWatcher\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76
    
          +0x130\n\ngoroutine 5545 [select, 7 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc00240a780,
    
          0xc003321a00)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345
    
          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0xc002efea80?,
    
          0xc0009cc7a0?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207
    
          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136
    
          +0x30a\n\ngoroutine 5678 [select, 3 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc000b70480,
    
          0xc0035d4500)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345
    
          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0x6e5326?, 0xc002999e90?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207
    
          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136
    
          +0x30a\n\ngoroutine 5836 [select, 1 minutes]:\ngolang.org/x/net/http2.(*clientStream).writeRequest(0xc003b00180,
    
          0xc003ff8a00)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1345
    
          +0x9c9\ngolang.org/x/net/http2.(*clientStream).doRequest(0x6e5326?, 0xc003a1c8d0?)\n\t/build/vendor/golang.org/x/net/http2/transport.go:1207
    
          +0x1e\ncreated by golang.org/x/net/http2.(*ClientConn).RoundTrip\n\t/build/vendor/golang.org/x/net/http2/transport.go:1136
    
          +0x30a\n\ngoroutine 5905 [chan receive, 1 minutes]:\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver.(*sourceInvalidator).GetValidChannel.func1()\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver/source_registry.go:51
    
          +0x85\ncreated by github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver.(*sourceInvalidator).GetValidChannel\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver/source_registry.go:50
    
          +0x231\n"
    
        reason: Error
    
        startedAt: "2022-06-27T00:00:59Z"
    
    name: catalog-operator
    
    ready: true
    
    restartCount: 2
    
    started: true
    
    state:
    
      running:
    
        startedAt: "2022-06-27T01:19:08Z"
    
  hostIP: 10.0.173.128
    
  phase: Running
    
  podIP: 10.130.0.26
    
  podIPs:
    • 
	
  • ip: 10.130.0.26
    
  qosClass: Burstable
    
  startTime: "2022-06-26T23:16:59Z"
    • 






Expected results:

catalog-operator works well.



Additional info:

Operators can be subscribed successfully.

mac:~ jianzhang$ oc get sub -A

NAMESPACE                    NAME                     PACKAGE                  SOURCE            CHANNEL

jian                         learn                    learn                    qe-app-registry   beta

openshift-logging            cluster-logging          cluster-logging          qe-app-registry   stable

openshift-operators-redhat   elasticsearch-operator   elasticsearch-operator   qe-app-registry   stable

mac:~ jianzhang$ 

mac:~ jianzhang$ oc get pods -n jian

NAME                                                              READY   STATUS      RESTARTS   AGE

552b4660850a7fe1e1f142091eb5e4305f18af151727c56f70aa5dffc1dg8cg   0/1     Completed   0          54m

learn-operator-666b687bfb-7qppm                                   1/1     Running     0          54m

qe-app-registry-hbzxg                                             1/1     Running     0          58m

mac:~ jianzhang$ oc get csv -n jian

NAME                            DISPLAY                            VERSION   REPLACES                PHASE

elasticsearch-operator.v5.5.0   OpenShift Elasticsearch Operator   5.5.0                             Succeeded

learn-operator.v0.0.3           Learn Operator                     0.0.3     learn-operator.v0.0.2   Succeeded



— Additional comment from jiazha@redhat.com on 2022-06-27 09:58:18 UTC —



Created attachment 1892927

olm must-gather



— Additional comment from jiazha@redhat.com on 2022-06-27 09:59:01 UTC —



Created attachment 1892928

marketplace project must-gather



— Additional comment from jiazha@redhat.com on 2022-06-28 02:05:39 UTC —




mac:~ jianzhang$ oc get clusterversion

NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS

version   4.11.0-0.nightly-2022-06-25-132614   True        False         145m    Cluster version is 4.11.0-0.nightly-2022-06-25-132614



mac:~ jianzhang$ oc get pods

NAME                                      READY   STATUS      RESTARTS        AGE

catalog-operator-869fb4bd4d-lbhgj         1/1     Running     3 (9m25s ago)   170m

collect-profiles-27606330-4wg5r           0/1     Completed   0               33m

collect-profiles-27606345-lmk4q           0/1     Completed   0               18m

collect-profiles-27606360-mksv6           0/1     Completed   0               3m17s

olm-operator-5f485d9d5f-wczjc             1/1     Running     0               170m

package-server-manager-6cf996b4cc-79lrw   1/1     Running     2 (156m ago)    170m

packageserver-5f668f98d7-2vjdn            1/1     Running     0               165m

packageserver-5f668f98d7-mb2wc            1/1     Running     0               165m

mac:~ jianzhang$ oc get pods catalog-operator-869fb4bd4d-lbhgj -o yaml

apiVersion: v1

kind: Pod

metadata:

  annotations:

    k8s.v1.cni.cncf.io/network-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.34"

          ],

          "default": true,

          "dns": {}

      }]

    k8s.v1.cni.cncf.io/networks-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.34"

          ],

          "default": true,

          "dns": {}

      }]

    openshift.io/scc: nonroot-v2

    seccomp.security.alpha.kubernetes.io/pod: runtime/default

  creationTimestamp: "2022-06-27T23:13:12Z"

  generateName: catalog-operator-869fb4bd4d-

  labels:

    app: catalog-operator

    pod-template-hash: 869fb4bd4d

  name: catalog-operator-869fb4bd4d-lbhgj

  namespace: openshift-operator-lifecycle-manager

  ownerReferences:


    
	
  • apiVersion: apps/v1
    
    blockOwnerDeletion: true
    
    controller: true
    
    kind: ReplicaSet
    
    name: catalog-operator-869fb4bd4d
    
    uid: 3a1a8cd3-2151-4650-a96b-c1951d461c67
    
  resourceVersion: "75671"
    
  uid: 2f06d663-0697-4e88-9e9a-f802c6641efe
    
spec:
    
  containers:
    • 
	
  • args:
    • 
	
  • --namespace
    • 
	
  • openshift-marketplace
    • 
	
  • --configmapServerImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40552deca32531d2fe754f703613f24b514e27ffaa660b57d760f9cd984d9000
    • 
	
  • --opmImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40552deca32531d2fe754f703613f24b514e27ffaa660b57d760f9cd984d9000
    • 
	
  • --util-image
    • 
	
  • quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    • 
	
  • --writeStatusName
    • 
	
  • operator-lifecycle-manager-catalog
    • 
	
  • --tls-cert
    • 
	
  • /srv-cert/tls.crt
    • 
	
  • --tls-key
    • 
	
  • /srv-cert/tls.key
    • 
	
  • --client-ca
    • 
	
  • /profile-collector-cert/tls.crt
    
    command:
    • 
	
  • /bin/catalog
    
    env:
    • 
	
  • name: RELEASE_VERSION
    
      value: 4.11.0-0.nightly-2022-06-25-132614
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imagePullPolicy: IfNotPresent
    
    livenessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    name: catalog-operator
    
    ports:
    • 
	
  • containerPort: 8443
    
      name: metrics
    
      protocol: TCP
    
    readinessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    resources:
    
      requests:
    
        cpu: 10m
    
        memory: 80Mi
    
    securityContext:
    
      allowPrivilegeEscalation: false
    
      capabilities:
    
        drop:
    • 
	
  • ALL
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: FallbackToLogsOnError
    
    volumeMounts:
    • 
	
  • mountPath: /srv-cert
    
      name: srv-cert
    
      readOnly: true
    • 
	
  • mountPath: /profile-collector-cert
    
      name: profile-collector-cert
    
      readOnly: true
    • 
	
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
      name: kube-api-access-sdjns
    
      readOnly: true
    
  dnsPolicy: ClusterFirst
    
  enableServiceLinks: true
    
  nodeName: ip-10-0-190-130.ap-south-1.compute.internal
    
  nodeSelector:
    
    kubernetes.io/os: linux
    
    node-role.kubernetes.io/master: ""
    
  preemptionPolicy: PreemptLowerPriority
    
  priority: 2000000000
    
  priorityClassName: system-cluster-critical
    
  restartPolicy: Always
    
  schedulerName: default-scheduler
    
  securityContext:
    
    runAsNonRoot: true
    
    runAsUser: 65534
    
    seLinuxOptions:
    
      level: s0:c20,c0
    
    seccompProfile:
    
      type: RuntimeDefault
    
  serviceAccount: olm-operator-serviceaccount
    
  serviceAccountName: olm-operator-serviceaccount
    
  terminationGracePeriodSeconds: 30
    
  tolerations:
    • 
	
  • effect: NoSchedule
    
    key: node-role.kubernetes.io/master
    
    operator: Exists
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/unreachable
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/not-ready
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoSchedule
    
    key: node.kubernetes.io/memory-pressure
    
    operator: Exists
    
  volumes:
    • 
	
  • name: srv-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: catalog-operator-serving-cert
    • 
	
  • name: profile-collector-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: pprof-cert
    • 
	
  • name: kube-api-access-sdjns
    
    projected:
    
      defaultMode: 420
    
      sources:
    • 
	
  • serviceAccountToken:
    
          expirationSeconds: 3607
    
          path: token
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: ca.crt
    
            path: ca.crt
    
          name: kube-root-ca.crt
    • 
	
  • downwardAPI:
    
          items:
    • 
	
  • fieldRef:
    
              apiVersion: v1
    
              fieldPath: metadata.namespace
    
            path: namespace
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: service-ca.crt
    
            path: service-ca.crt
    
          name: openshift-service-ca.crt
    
status:
    
  conditions:
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:16:47Z"
    
    status: "True"
    
    type: Initialized
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-28T01:53:53Z"
    
    status: "True"
    
    type: Ready
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-28T01:53:53Z"
    
    status: "True"
    
    type: ContainersReady
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:16:47Z"
    
    status: "True"
    
    type: PodScheduled
    
  containerStatuses:
    • 
	
  • containerID: cri-o://2a83d3ba503aa760d27aeef28bf03af84b9ff134ea76c6545251f3253507c22b
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    lastState:
    
      terminated:
    
        containerID: cri-o://2766bb3bad38c02f531915aac0321ffb18c7a993a7d84b27e8c7199935f7c858
    
        exitCode: 2
    
        finishedAt: "2022-06-28T01:53:52Z"
    
        message: "izer/streaming/streaming.go:77 +0xa7\nk8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b12500)\n\t/build/vendor/k8s.io/client-go/rest/watch/decoder.go:49
    
          +0x4f\nk8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0036178c0)\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105
    
          +0xe5\ncreated by k8s.io/apimachinery/pkg/watch.NewStreamWatcher\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76
    
          +0x130\n\ngoroutine 3479 [sync.Cond.Wait]:\nsync.runtime_notifyListWait(0xc003a987c8,
    
          0x8)\n\t/usr/lib/golang/src/runtime/sema.go:513 +0x13d\nsync.(*Cond).Wait(0xc001f9bf10?)\n\t/usr/lib/golang/src/sync/cond.go:56
    
          +0x8c\ngolang.org/x/net/http2.(*pipe).Read(0xc003a987b0, 
{0xc0036e4001,
          0xdff, 0xdff}

    )\n\t/build/vendor/golang.org/x/net/http2/pipe.go:76 +0xeb\ngolang.org/x/net/http2.transportResponseBody.Read(
    
{0x0?}

    ,
    
          
    
{0xc0036e4001?, 0x2?, 0x203511d?}

    )\n\t/build/vendor/golang.org/x/net/http2/transport.go:2407
    
          +0x85\nencoding/json.(*Decoder).refill(0xc002fc0640)\n\t/usr/lib/golang/src/encoding/json/stream.go:165
    
          +0x17f\nencoding/json.(*Decoder).readValue(0xc002fc0640)\n\t/usr/lib/golang/src/encoding/json/stream.go:140
    
          +0xbb\nencoding/json.(*Decoder).Decode(0xc002fc0640, 
    
{0x1d377c0, 0xc003523098}

    )\n\t/usr/lib/golang/src/encoding/json/stream.go:63
    
          +0x78\nk8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc003127770,
    
          
    
{0xc0036dd500, 0x1000, 0x1500}

    )\n\t/build/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152
    
          +0x19c\nk8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc003502aa0,
    
          0xc001f9bf10?, 
    
{0x2366870, 0xc0044dca80}

    )\n\t/build/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77
    
          +0xa7\nk8s.io/client-go/rest/watch.(*Decoder).Decode(0xc00059f700)\n\t/build/vendor/k8s.io/client-go/rest/watch/decoder.go:49
    
          +0x4f\nk8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0044dcd40)\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105
    
          +0xe5\ncreated by k8s.io/apimachinery/pkg/watch.NewStreamWatcher\n\t/build/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76
    
          +0x130\n"
    
        reason: Error
    
        startedAt: "2022-06-28T01:06:59Z"
    
    name: catalog-operator
    
    ready: true
    
    restartCount: 3
    
    started: true
    
    state:
    
      running:
    
        startedAt: "2022-06-28T01:53:53Z"
    
  hostIP: 10.0.190.130
    
  phase: Running
    
  podIP: 10.130.0.34
    
  podIPs:
    • 
	
  • ip: 10.130.0.34
    
  qosClass: Burstable
    
  startTime: "2022-06-27T23:16:47Z"
    • 





— Additional comment from jiazha@redhat.com on 2022-06-28 02:09:23 UTC —



mac:~ jianzhang$ oc get pods package-server-manager-6cf996b4cc-79lrw -o yaml

apiVersion: v1

kind: Pod

metadata:

  annotations:

    k8s.v1.cni.cncf.io/network-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.13"

          ],

          "default": true,

          "dns": {}

      }]

    k8s.v1.cni.cncf.io/networks-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.13"

          ],

          "default": true,

          "dns": {}

      }]

    openshift.io/scc: nonroot-v2

    seccomp.security.alpha.kubernetes.io/pod: runtime/default

  creationTimestamp: "2022-06-27T23:13:10Z"

  generateName: package-server-manager-6cf996b4cc-

  labels:

    app: package-server-manager

    pod-template-hash: 6cf996b4cc

  name: package-server-manager-6cf996b4cc-79lrw

  namespace: openshift-operator-lifecycle-manager

  ownerReferences:


    
	
  • apiVersion: apps/v1
    
    blockOwnerDeletion: true
    
    controller: true
    
    kind: ReplicaSet
    
    name: package-server-manager-6cf996b4cc
    
    uid: 191cb627-a1f3-452b-aed2-336de7871004
    
  resourceVersion: "19056"
    
  uid: 63cb7200-32d6-4b0d-bfb2-ba725cda7fbd
    
spec:
    
  containers:
    • 
	
  • args:
    • 
	
  • --name
    • 
	
  • $(PACKAGESERVER_NAME)
    • 
	
  • --namespace
    • 
	
  • $(PACKAGESERVER_NAMESPACE)
    
    command:
    • 
	
  • /bin/psm
    • 
	
  • start
    
    env:
    • 
	
  • name: PACKAGESERVER_NAME
    
      value: packageserver
    • 
	
  • name: PACKAGESERVER_IMAGE
    
      value: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    • 
	
  • name: PACKAGESERVER_NAMESPACE
    
      valueFrom:
    
        fieldRef:
    
          apiVersion: v1
    
          fieldPath: metadata.namespace
    • 
	
  • name: RELEASE_VERSION
    
      value: 4.11.0-0.nightly-2022-06-25-132614
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imagePullPolicy: IfNotPresent
    
    livenessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8080
    
        scheme: HTTP
    
      initialDelaySeconds: 30
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    name: package-server-manager
    
    readinessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8080
    
        scheme: HTTP
    
      initialDelaySeconds: 30
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    resources:
    
      requests:
    
        cpu: 10m
    
        memory: 50Mi
    
    securityContext:
    
      allowPrivilegeEscalation: false
    
      capabilities:
    
        drop:
    • 
	
  • ALL
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: FallbackToLogsOnError
    
    volumeMounts:
    • 
	
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
      name: kube-api-access-k9hh2
    
      readOnly: true
    
  dnsPolicy: ClusterFirst
    
  enableServiceLinks: true
    
  nodeName: ip-10-0-190-130.ap-south-1.compute.internal
    
  nodeSelector:
    
    kubernetes.io/os: linux
    
    node-role.kubern
    • 





— Additional comment from jiazha@redhat.com on 2022-06-28 02:10:02 UTC —



preemptionPolicy: PreemptLowerPriority

  priority: 2000000000

  priorityClassName: system-cluster-critical

  restartPolicy: Always

  schedulerName: default-scheduler

  securityContext:

    runAsNonRoot: true

    runAsUser: 65534

    seLinuxOptions:

      level: s0:c20,c0

    seccompProfile:

      type: RuntimeDefault

  serviceAccount: olm-operator-serviceaccount

  serviceAccountName: olm-operator-serviceaccount

  terminationGracePeriodSeconds: 30

  tolerations:


    
	
  • effect: NoSchedule
    
    key: node-role.kubernetes.io/master
    
    operator: Exists
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/unreachable
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/not-ready
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoSchedule
    
    key: node.kubernetes.io/memory-pressure
    
    operator: Exists
    
  volumes:
    • 
	
  • name: kube-api-access-k9hh2
    
    projected:
    
      defaultMode: 420
    
      sources:
    • 
	
  • serviceAccountToken:
    
          expirationSeconds: 3607
    
          path: token
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: ca.crt
    
            path: ca.crt
    
          name: kube-root-ca.crt
    • 
	
  • downwardAPI:
    
          items:
    • 
	
  • fieldRef:
    
              apiVersion: v1
    
              fieldPath: metadata.namespace
    
            path: namespace
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: service-ca.crt
    
            path: service-ca.crt
    
          name: openshift-service-ca.crt
    
status:
    
  conditions:
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:16:47Z"
    
    status: "True"
    
    type: Initialized
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:27:28Z"
    
    status: "True"
    
    type: Ready
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:27:28Z"
    
    status: "True"
    
    type: ContainersReady
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-27T23:16:47Z"
    
    status: "True"
    
    type: PodScheduled
    
  containerStatuses:
    • 
	
  • containerID: cri-o://2c56089fabbaa6d7067feb231750ad20422e299dc33dabc0cd19ceca5951ed3a
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    lastState:
    
      terminated:
    
        containerID: cri-o://76cc5448de346ec636070a71868432c8f3aa213e16fe211c8fe18a3fee112d23
    
        exitCode: 1
    
        finishedAt: "2022-06-27T23:26:36Z"
    
        message: "d/vendor/github.com/spf13/cobra/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/build/vendor/github.com/spf13/cobra/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/build/vendor/github.com/spf13/cobra/command.go:902\nmain.main\n\t/build/cmd/package-server-manager/main.go:36\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\n1.6563723963629913e+09\tERROR\tFailed
    
          to get API Group-Resources\t
{\"error\": \"Get \\\"https://172.30.0.1:443/api?timeout=32s\\\":
          dial tcp 172.30.0.1:443: connect: connection refused\"}

    \nsigs.k8s.io/controller-runtime/pkg/cluster.New\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/cluster/cluster.go:160\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/manager/manager.go:322\nmain.run\n\t/build/cmd/package-server-manager/main.go:67\ngithub.com/spf13/cobra.(*Command).execute\n\t/build/vendor/github.com/spf13/cobra/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/build/vendor/github.com/spf13/cobra/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/build/vendor/github.com/spf13/cobra/command.go:902\nmain.main\n\t/build/cmd/package-server-manager/main.go:36\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\n1.6563723963631017e+09\tERROR\tsetup\tfailed
    
          to setup manager instance\t
    
{\"error\": \"Get \\\"https://172.30.0.1:443/api?timeout=32s\\\":
          dial tcp 172.30.0.1:443: connect: connection refused\"}

    \ngithub.com/spf13/cobra.(*Command).execute\n\t/build/vendor/github.com/spf13/cobra/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/build/vendor/github.com/spf13/cobra/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\t/build/vendor/github.com/spf13/cobra/command.go:902\nmain.main\n\t/build/cmd/package-server-manager/main.go:36\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250\nError:
    
          Get \"https://172.30.0.1:443/api?timeout=32s\": dial tcp 172.30.0.1:443:
    
          connect: connection refused\nencountered an error while executing the binary:
    
          Get \"https://172.30.0.1:443/api?timeout=32s\": dial tcp 172.30.0.1:443:
    
          connect: connection refused\n"
    
        reason: Error
    
        startedAt: "2022-06-27T23:26:11Z"
    
    name: package-server-manager
    
    ready: true
    
    restartCount: 2
    
    started: true
    
    state:
    
      running:
    
        startedAt: "2022-06-27T23:26:54Z"
    
  hostIP: 10.0.190.130
    
  phase: Running
    
  podIP: 10.130.0.13
    
  podIPs:
    • 
	
  • ip: 10.130.0.13
    
  qosClass: Burstable
    
  startTime: "2022-06-27T23:16:47Z"
    • 





— Additional comment from jiazha@redhat.com on 2022-06-29 08:43:51 UTC —



Observed the error restarts:



mac:~ jianzhang$ oc get clusterversion

NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS

version   4.11.0-0.nightly-2022-06-28-160049   True        False         5h57m   Cluster version is 4.11.0-0.nightly-2022-06-28-160049



mac:~ jianzhang$ oc get pods

NAME                                     READY   STATUS      RESTARTS      AGE

catalog-operator-7b88dddfbc-rsfhz        1/1     Running     6 (26m ago)   5h51m

collect-profiles-27608160-6m7r6          0/1     Completed   0             37m

collect-profiles-27608175-94n56          0/1     Completed   0             22m

collect-profiles-27608190-nbzcf          0/1     Completed   0             7m55s

olm-operator-5977ffb855-lgfn8            1/1     Running     0             9h

package-server-manager-75db6dcfc-hql4v   1/1     Running     0             9h

packageserver-5955fb79cd-9n56n           1/1     Running     0             9h

packageserver-5955fb79cd-xf6f6           1/1     Running     0             9h



mac:~ jianzhang$ oc get pods catalog-operator-7b88dddfbc-rsfhz -o yaml

apiVersion: v1

kind: Pod

metadata:

  annotations:

    k8s.v1.cni.cncf.io/network-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.121"

          ],

          "default": true,

          "dns": {}

      }]

    k8s.v1.cni.cncf.io/networks-status: |-

      [{

          "name": "openshift-sdn",

          "interface": "eth0",

          "ips": [

              "10.130.0.121"

          ],

          "default": true,

          "dns": {}

      }]

    openshift.io/scc: nonroot-v2

    seccomp.security.alpha.kubernetes.io/pod: runtime/default

  creationTimestamp: "2022-06-29T02:46:23Z"

  generateName: catalog-operator-7b88dddfbc-

  labels:

    app: catalog-operator

    pod-template-hash: 7b88dddfbc

  name: catalog-operator-7b88dddfbc-rsfhz

  namespace: openshift-operator-lifecycle-manager

  ownerReferences:


    
	
  • apiVersion: apps/v1
    
    blockOwnerDeletion: true
    
    controller: true
    
    kind: ReplicaSet
    
    name: catalog-operator-7b88dddfbc
    
    uid: 4c796f4d-b2f2-4e0d-b2de-6e8ab5f45ce2
    
  resourceVersion: "278732"
    
  uid: 10e8c6e9-8f1c-4484-b934-c5208016c426
    
spec:
    
  containers:
    • 
	
  • args:
    • 
	
  • --debug
    • 
	
  • --namespace
    • 
	
  • openshift-marketplace
    • 
	
  • --configmapServerImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40552deca32531d2fe754f703613f24b514e27ffaa660b57d760f9cd984d9000
    • 
	
  • --opmImage=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40552deca32531d2fe754f703613f24b514e27ffaa660b57d760f9cd984d9000
    • 
	
  • --util-image
    • 
	
  • quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    • 
	
  • --writeStatusName
    • 
	
  • operator-lifecycle-manager-catalog
    • 
	
  • --tls-cert
    • 
	
  • /srv-cert/tls.crt
    • 
	
  • --tls-key
    • 
	
  • /srv-cert/tls.key
    • 
	
  • --client-ca
    • 
	
  • /profile-collector-cert/tls.crt
    
    command:
    • 
	
  • /bin/catalog
    
    env:
    • 
	
  • name: RELEASE_VERSION
    
      value: 4.11.0-0.nightly-2022-06-28-160049
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imagePullPolicy: IfNotPresent
    
    livenessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    name: catalog-operator
    
    ports:
    • 
	
  • containerPort: 8443
    
      name: metrics
    
      protocol: TCP
    
    readinessProbe:
    
      failureThreshold: 3
    
      httpGet:
    
        path: /healthz
    
        port: 8443
    
        scheme: HTTPS
    
      periodSeconds: 10
    
      successThreshold: 1
    
      timeoutSeconds: 1
    
    resources:
    
      requests:
    
        cpu: 10m
    
        memory: 80Mi
    
    securityContext:
    
      allowPrivilegeEscalation: false
    
      capabilities:
    
        drop:
    • 
	
  • ALL
    
    terminationMessagePath: /dev/termination-log
    
    terminationMessagePolicy: FallbackToLogsOnError
    
    volumeMounts:
    • 
	
  • mountPath: /srv-cert
    
      name: srv-cert
    
      readOnly: true
    • 
	
  • mountPath: /profile-collector-cert
    
      name: profile-collector-cert
    
      readOnly: true
    • 
	
  • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    
      name: kube-api-access-cv642
    
      readOnly: true
    
  dnsPolicy: ClusterFirst
    
  enableServiceLinks: true
    
  imagePullSecrets:
    • 
	
  • name: olm-operator-serviceaccount-dockercfg-f26bf
    
  nodeName: ip-10-0-130-83.ap-south-1.compute.internal
    
  nodeSelector:
    
    kubernetes.io/os: linux
    
    node-role.kubernetes.io/master: ""
    
  preemptionPolicy: PreemptLowerPriority
    
  priority: 2000000000
    
  priorityClassName: system-cluster-critical
    
  restartPolicy: Always
    
  schedulerName: default-scheduler
    
  securityContext:
    
    runAsNonRoot: true
    
    runAsUser: 65534
    
    seLinuxOptions:
    
      level: s0:c20,c0
    
    seccompProfile:
    
      type: RuntimeDefault
    
  serviceAccount: olm-operator-serviceaccount
    
  serviceAccountName: olm-operator-serviceaccount
    
  terminationGracePeriodSeconds: 30
    
  tolerations:
    • 
	
  • effect: NoSchedule
    
    key: node-role.kubernetes.io/master
    
    operator: Exists
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/unreachable
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoExecute
    
    key: node.kubernetes.io/not-ready
    
    operator: Exists
    
    tolerationSeconds: 120
    • 
	
  • effect: NoSchedule
    
    key: node.kubernetes.io/memory-pressure
    
    operator: Exists
    
  volumes:
    • 
	
  • name: srv-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: catalog-operator-serving-cert
    • 
	
  • name: profile-collector-cert
    
    secret:
    
      defaultMode: 420
    
      secretName: pprof-cert
    • 
	
  • name: kube-api-access-cv642
    
    projected:
    
      defaultMode: 420
    
      sources:
    • 
	
  • serviceAccountToken:
    
          expirationSeconds: 3607
    
          path: token
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: ca.crt
    
            path: ca.crt
    
          name: kube-root-ca.crt
    • 
	
  • downwardAPI:
    
          items:
    • 
	
  • fieldRef:
    
              apiVersion: v1
    
              fieldPath: metadata.namespace
    
            path: namespace
    • 
	
  • configMap:
    
          items:
    • 
	
  • key: service-ca.crt
    
            path: service-ca.crt
    
          name: openshift-service-ca.crt
    
status:
    
  conditions:
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-29T02:46:23Z"
    
    status: "True"
    
    type: Initialized
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-29T08:11:56Z"
    
    status: "True"
    
    type: Ready
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-29T08:11:56Z"
    
    status: "True"
    
    type: ContainersReady
    • 
	
  • lastProbeTime: null
    
    lastTransitionTime: "2022-06-29T02:46:23Z"
    
    status: "True"
    
    type: PodScheduled
    
  containerStatuses:
    • 
	
  • containerID: cri-o://64a99805cca11e9af66452d739a91259566a35a9e580bfac81dd7205f9272e18
    
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac69cd6ac451019bc6060627a1cceb475d2b6b521803addb31b47e501d5e5935
    
    lastState:
    
      terminated:
    
        containerID: cri-o://938e2eb90465caf8ac99ff4405cfbb9a0f03b16598474f944cca44c3e70af9df
    
        exitCode: 2
    
        finishedAt: "2022-06-29T08:11:54Z"
    
        message: "ub.com/operator-framework/operator-lifecycle-manager/pkg/lib/kubestate/kubestate.go:128
    
          +0xc3 fp=0xc003c11180 sp=0xc003c11118 pc=0x1a36603\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*subscriptionSyncer).Sync(0xc000a2f420,
    
          
{0x237a238, 0xc0004ce4c0}, {0x236a448, 0xc004993ce0})\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/syncer.go:77
    
          +0x535 fp=0xc003c11648 sp=0xc003c11180 pc=0x1a4f535\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*QueueInformer).Sync(...)\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer.go:35\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).processNextWorkItem(0xc0001e3ad0,
    
          {0x237a238, 0xc0004ce4c0}

    , 0xc000cee360)\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:287
    
          +0x57c fp=0xc003c11f70 sp=0xc003c11648 pc=0x1a3ca7c\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).worker(0x10000c0008fd6e0?,
    
          
    
{0x237a238, 0xc0004ce4c0}

    , 0xc0004837b8?)\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:231
    
          +0x45 fp=0xc003c11fb0 sp=0xc003c11f70 pc=0x1a3c4a5\ngithub.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).start.func3()\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:221
    
          +0x32 fp=0xc003c11fe0 sp=0xc003c11fb0 pc=0x1a3c152\nruntime.goexit()\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571
    
          +0x1 fp=0xc003c11fe8 sp=0xc003c11fe0 pc=0x4719c1\ncreated by github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).start\n\t/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:221
    
          +0x557\n"
    
        reason: Error
    
        startedAt: "2022-06-29T07:56:16Z"
    
    name: catalog-operator
    
    ready: true
    
    restartCount: 6
    
    started: true
    
    state:
    
      running:
    
        startedAt: "2022-06-29T08:11:55Z"
    
  hostIP: 10.0.130.83
    
  phase: Running
    
  podIP: 10.130.0.121
    
  podIPs:
    • 
	
  • ip: 10.130.0.121
    
  qosClass: Burstable
    
  startTime: "2022-06-29T02:46:23Z"
    • 





— Additional comment from jiazha@redhat.com on 2022-07-04 03:50:38 UTC —



Please ignore comment 4, 5, they are nothing with this issue.



— Additional comment from jiazha@redhat.com on 2022-07-04 06:57:24 UTC —



Check the `previous` log.



mac:~ jianzhang$ oc logs catalog-operator-f8ddcb57b-j5rf2 --previous

time="2022-07-03T23:49:00Z" level=info msg="log level info"

...

...

time="2022-07-04T03:43:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=

time="2022-07-04T03:43:25Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=

fatal error: concurrent map writes

fatal error: concurrent map writes



goroutine 559 [running]:

runtime.throw(

{0x2048546?, 0x0?}

)

	/usr/lib/golang/src/runtime/panic.go:992 +0x71 fp=0xc001f9c508 sp=0xc001f9c4d8 pc=0x43e9f1

runtime.mapassign_faststr(0x1d09880, 0xc0031847b0, 

{0x2079799, 0x2e}

)

	/usr/lib/golang/src/runtime/map_faststr.go:295 +0x38b fp=0xc001f9c570 sp=0xc001f9c508 pc=0x419b4b

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler.Pod(0xc001f4a900, 

{0x203dd0f, 0xf}

, 

{0xc00132ccc0, 0x38}

, 

{0xc003582d50, 0x13}, 0xc00452c1e0, 0xc0031847b0, 0x5, ...)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler/reconciler.go:227 +0xbdf fp=0xc001f9cbb0 sp=0xc001f9c570 pc=0x1a475ff

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler.(*grpcCatalogSourceDecorator).Pod(0xc001f9ce50, {0xc003582d50, 0x13}

)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler/grpc.go:125 +0xf9 fp=0xc001f9cc30 sp=0xc001f9cbb0 pc=0x1a42c99

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler.(*GrpcRegistryReconciler).currentPodsWithCorrectImageAndSpec(0xc001f9ce68?, 

{0xc001f4a900}

, 

{0xc003582d50, 0x13}

)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler/grpc.go:190 +0x198 fp=0xc001f9ce48 sp=0xc001f9cc30 pc=0x1a437b8

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler.(*GrpcRegistryReconciler).CheckRegistryServer(0xc000bcbf80?, 0x493b77?)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/reconciler/grpc.go:453 +0x4c fp=0xc001f9ce88 sp=0xc001f9ce48 pc=0x1a45fcc

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*catalogHealthReconciler).healthy(0x38ca8453?, 0xc001f4a900)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/reconciler.go:196 +0x7e fp=0xc001f9ced0 sp=0xc001f9ce88 pc=0x1a4ae1e

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*catalogHealthReconciler).health(0x1bc37c0?, 0xc003e7e7e0, 0x8?)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/reconciler.go:159 +0x2a fp=0xc001f9cf10 sp=0xc001f9ced0 pc=0x1a4ac8a

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*catalogHealthReconciler).catalogHealth(0xc000a59a90, 

{0xc003356a50, 0x11}

)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/reconciler.go:137 +0x387 fp=0xc001f9d040 sp=0xc001f9cf10 pc=0x1a4a827

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*catalogHealthReconciler).Reconcile(0xc000a59a90, 

{0x237a238, 0xc0003fe5c0}, {0x7f9f6e5b3900?, 0xc0050f64f0?})

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/reconciler.go:82 +0x21e fp=0xc001f9d118 sp=0xc001f9d040 pc=0x1a4a1be

github.com/operator-framework/operator-lifecycle-manager/pkg/lib/kubestate.ReconcilerChain.Reconcile({0xc0008e43c0?, 0x3, 0xc001f9d258?}, {0x237a238, 0xc0003fe5c0}

, 

{0x7f9f6e5b3328?, 0xc0050f6490?}

)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/kubestate/kubestate.go:128 +0xc3 fp=0xc001f9d180 sp=0xc001f9d118 pc=0x1a36603

github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription.(*subscriptionSyncer).Sync(0xc0004dfd50, 

{0x237a238, 0xc0003fe5c0}, {0x236a448, 0xc003cbb760})

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/controller/operators/catalog/subscription/syncer.go:77 +0x535 fp=0xc001f9d648 sp=0xc001f9d180 pc=0x1a4f535

github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*QueueInformer).Sync(...)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer.go:35

github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).processNextWorkItem(0xc00057a580, {0x237a238, 0xc0003fe5c0}

, 0xc000954720)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:287 +0x57c fp=0xc001f9df70 sp=0xc001f9d648 pc=0x1a3ca7c

github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).worker(0x0?, 

{0x237a238, 0xc0003fe5c0}

, 0x0?)

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:231 +0x45 fp=0xc001f9dfb0 sp=0xc001f9df70 pc=0x1a3c4a5

github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).start.func3()

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:221 +0x32 fp=0xc001f9dfe0 sp=0xc001f9dfb0 pc=0x1a3c152

runtime.goexit()

	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001f9dfe8 sp=0xc001f9dfe0 pc=0x4719c1

created by github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer.(*operator).start

	/build/vendor/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/queueinformer/queueinformer_operator.go:221 +0x557




Seems like it failed at: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/registry/reconciler/reconciler.go#L227



— Additional comment from agreene@redhat.com on 2022-07-05 16:01:22 UTC —



As Jian pointed out, the catalog operator is failing due to a concurrent write at https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/registry/reconciler/reconciler.go#L227.




This is happening because:





Line 227 in the reconciler.go directly mutates the catalogSource's annotations. The grpcCatalogSourceDecorator's Annotations function should be returning a copy of the annotations or it should be created with a deepcopy of the catalogSource to avoid mutating an object in the lister cache.



This doesn't seem to be a blocker, but we should get a fix in swiftly.



— Additional comment from jiazha@redhat.com on 2022-07-13 05:02:04 UTC —



1, Create a cluster with the fixed PR via the Cluster-bot.

mac:~ jianzhang$ oc get clusterversion

NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS

version   4.11.0-0.ci.test-2022-07-13-022646-ci-ln-41fvni2-latest   True        False         126m    Cluster version is 4.11.0-0.ci.test-2022-07-13-022646-ci-ln-41fvni2-latest



2, Subscribe some operators.

mac:~ jianzhang$ oc get sub -A

NAMESPACE                    NAME                     PACKAGE                  SOURCE                CHANNEL

default                      etcd                     etcd                     community-operators   singlenamespace-alpha

openshift-logging            cluster-logging          cluster-logging          redhat-operators      stable

openshift-operators-redhat   elasticsearch-operator   elasticsearch-operator   redhat-operators      stable



mac:~ jianzhang$ oc get sub -A

NAMESPACE                    NAME                     PACKAGE                  SOURCE                CHANNEL

default                      etcd                     etcd                     community-operators   singlenamespace-alpha

openshift-logging            cluster-logging          cluster-logging          redhat-operators      stable

openshift-operators-redhat   elasticsearch-operator   elasticsearch-operator   redhat-operators      stable

mac:~ jianzhang$ 

mac:~ jianzhang$ 

mac:~ jianzhang$ oc get csv -n openshift-operators-redhat

NAME                           DISPLAY                            VERSION   REPLACES   PHASE

elasticsearch-operator.5.4.2   OpenShift Elasticsearch Operator   5.4.2                Succeeded

mac:~ jianzhang$ oc get csv -n openshift-logging 

NAME                           DISPLAY                            VERSION   REPLACES   PHASE

cluster-logging.5.4.2          Red Hat OpenShift Logging          5.4.2                Succeeded

elasticsearch-operator.5.4.2   OpenShift Elasticsearch Operator   5.4.2                Succeeded

mac:~ jianzhang$ oc get csv -n default

NAME                           DISPLAY                            VERSION   REPLACES              PHASE

elasticsearch-operator.5.4.2   OpenShift Elasticsearch Operator   5.4.2                           Succeeded

etcdoperator.v0.9.4            etcd                               0.9.4     etcdoperator.v0.9.2   Succeeded



3, Check OLM catalog-operator pods status.

mac:~ jianzhang$ oc get pods

NAME                                      READY   STATUS      RESTARTS   AGE

catalog-operator-546db7cdf5-7pldg         1/1     Running     0          145m

collect-profiles-27628110-lr2nv           0/1     Completed   0          30m

collect-profiles-27628125-br8b8           0/1     Completed   0          15m

collect-profiles-27628140-m64gp           0/1     Completed   0          38s

olm-operator-754d7f6f56-26qhw             1/1     Running     0          145m

package-server-manager-77d5cbf696-v9w4p   1/1     Running     0          145m

packageserver-6884994d98-2smtw            1/1     Running     0          143m

packageserver-6884994d98-5d7jg            1/1     Running     0          143m




mac:~ jianzhang$ oc logs catalog-operator-546db7cdf5-7pldg  --previous

Error from server (BadRequest): previous terminated container "catalog-operator" in pod "catalog-operator-546db7cdf5-7pldg" not found



No terminated container. catalog-operator works well. Verify it.



— Additional comment from aos-team-art-private@redhat.com on 2022-07-13 22:50:04 UTC —



Elliott changed bug status from MODIFIED to ON_QA.

This bug is expected to ship in the next 4.12 release.



— Additional comment from jiazha@redhat.com on 2022-07-18 07:23:08 UTC —



Changed the status to VERIFIED based on comment 10.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4604:
            
             fix https://github.com/kubernetes/kubernetes/pull/109137 in OpenShift 4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Custonmer is facing this problem in OCP 4.10.40.
https://github.com/coredns/coredns/issues/5593

 Its root cause seems to be this bug in k8s code:

https://github.com/kubernetes/kubernetes/issues/109115
https://github.com/kubernetes/kubernetes/pull/109137

The issue seems to be fixed in OpenShift 4.11, but the customer can't update at this moment.

Can this fix be backported to OpenShift 4.10?






Version-Release number of selected component (if applicable):




4.10.40




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 




 



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5990:
            
             [4.10] unit test data race with egress ip tests
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-5766. The following is the description of the original issue: 

—

Description of problem:




Data race seen in unit tests:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/1448/pull-ci-openshift-ovn-kubernetes-release-4.11-unit/1604898712423763968/artifacts/test/build-log.txt






 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2583:
            
             User settings resources (ConfigMap, Role, RB) should be deleted when a user is deleted
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2019564](https://bugzilla.redhat.com/show_bug.cgi?id=2019564). The following is the description of the original bug:

—

Description of problem:

On OpenShift Dev Sandbox users are automatically deleted after a month. Also on other clusters we should cleanup resources when the user are deleted.



Version-Release number of selected component (if applicable):

4.8+



How reproducible:

Always



Steps to Reproduce:

1. Create a user

2. Use console (this should automatically create 3 resources in the namespace "openshift-console-user-settings" (a Role, a RoleBinding and a ConfigMap)

3. Delete the user



Actual results:

The three created resources in namespace "openshift-console-user-settings" stays forever



Expected results:

The three created resources in namespace "openshift-console-user-settings" should be automatically removed



Additional info:

None

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3288:
            
             OCP 4.10.33 uses a weak 3DES cipher in the VMWare CSI Operator for communication and provides no method to disable it
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-2083. The following is the description of the original issue: 

—

Description of problem:

Currently we are running VMWare CSI Operator in OpenShift 4.10.33. After running vulnerability scans, the operator was discovered to be running a known weak cipher 3DES. We are attempting to upgrade or modify the operator to customize the ciphers available. We were looking at performing a manual upgrade via Quay.io but can't seem to pull the image and was trying to steer away from performing a custom install from scratch. Looking for any suggestions into mitigated the weak cipher in the kube-rbac-proxy under VMware CSI Operator.




Version-Release number of selected component (if applicable):

 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3858:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-450:
            
             KubeDaemonSetRolloutStuck alert using incorrect metric in 4.9 and 4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



kube_daemonset_updated_number_scheduled got renamed to kube_daemonset_status_updated_number_scheduled in 4.9, but the definition of KubeDaemonSetRolloutStuck didn't get updated up until 4.11 https://github.com/openshift/cluster-monitoring-operator/commit/35ffa690cec23d6a708aafea36a2d8b77f8a8556 . 



Version-Release number of selected component (if applicable):



How reproducible:



Always



We suggest backporting the fix at least to 4.10, as it can affect both customers and ours ability to detect and troubleshoot certain issues, both from single-cluster as  well as from the fleet-wide longer-term trends perspective point of view.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5344:
            
             [4.10] virtual media provisioning fails when iLO Ironic driver is used
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-5145. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5100. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-5068. The following is the description of the original issue: 

—

Description of problem:




virtual media provisioning fails when iLO Ironic driver is used




Version-Release number of selected component (if applicable):




4.13




How reproducible:




Always




Steps to Reproduce:




1. attempt virtual media provisioning on a node configured with ilo-virtualmedia:// drivers
2.
3.





Actual results:




Provisioning fails with "An auth plugin is required to determine endpoint URL" error




Expected results:




Provisioning succeeds




Additional info:




Relevant log snippet:

3742 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector [None req-e58ac1f2-fac6-4d28-be9e-983fa900a19b - - - - - -] Unable to start managed inspection for node e4445d43-3458-4cee-9cbe-6da1de75      78cd: An auth plugin is required to determine endpoint URL: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL
 3743 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector Traceback (most recent call last):
 3744 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/inspector.py", line 210, in _start_managed_inspection
 3745 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     task.driver.boot.prepare_ramdisk(task, ramdisk_params=params)
 3746 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic_lib/metrics.py", line 59, in wrapped
 3747 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     result = f(*args, **kwargs)
 3748 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/ilo/boot.py", line 408, in prepare_ramdisk
 3749 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     iso = image_utils.prepare_deploy_iso(task, ramdisk_params,
 3750 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 624, in prepare_deploy_iso
 3751 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     return prepare_iso_image(inject_files=inject_files)
 3752 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 537, in _prepare_iso_image
 3753 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     image_url = img_handler.publish_image(
 3754 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/image_utils.py", line 193, in publish_image
 3755 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     swift_api = swift.SwiftAPI()
 3756 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector   File "/usr/lib/python3.9/site-packages/ironic/common/swift.py", line 66, in __init__
 3757 2022-12-19T19:02:05.997747170Z 2022-12-19 19:02:05.995 1 ERROR ironic.drivers.modules.inspector     endpoint = keystone.get_endpoint('swift', session=session)



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            CONSOLE-2975:
            
             Migrate from Node Sass to Dart Sass
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Node Sass is deprecated. See https://github.com/sass/node-sass

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1218:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1759:
            
             Dev Catalog taking too much time to load in a complete disconnected cluster
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1523. The following is the description of the original issue: 

—

Description of problem:

In a complete disconnected cluster, the dev catalog is taking too much time in loading 



Version-Release number of selected component (if applicable):




How reproducible:

Always



Steps to Reproduce:

1. A complete disconnected cluster

2. In add page go to the All services page

3.



Actual results:

Taking too much time too load



Expected results:

Time taken should be reduced



Additional info:

Attached a gif for reference

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1696:
            
             All Nodes overview in console are showing "Something went wrong"
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




All Nodes overview in console are showing "Something went wrong"




Version-Release number of selected component (if applicable):




4.10




How reproducible:




I have tested in test-lab and i am able see node overviews




Steps to Reproduce:




1.
2.
3.





Actual results:




In the Console, Under compute tab nodes overview showing error "Oh no! something went wrong"




Expected results:




It should show the node overview




Additional info:




It is happening for all users with admin roles 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2781:
            
             Import: Advanced option sentence is splited into two parts and headlines has no padding
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a manual clone of https://bugzilla.redhat.com/show_bug.cgi?id=2093597 to backport this to 4.10. 



Description of problem:

When importing a component from git or from a container image, and open one or more advanced options, the sentence "Click on the names to access advanced options for ..." is splited into two parts. And the headlines have no padding and everything looks squashed.



Version-Release number of selected component (if applicable):

4.10+



How reproducible:

Always



Steps to Reproduce:

1. Switch to dev perspective

2. Navigate to the add page > Import from container

3. Scroll down and open one or more of the advanced options



Actual results:

1. The sentence "Click on the names to access advanced options for ..." is shown before the opened option. The other available options are shown below the selected option.

2. The headline is displayed directly below "Click on the names to access advanced options for"

3. Another section is also shown directly under the first one.



Expected results:

1. The sentence "Click on the names to access advanced options for ..." and the options should be "one sentence" again.

2+3. Some padding for the header and/or between the sections, similar to 4.9. It must not look exactly as in 4.9, but there should be some padding between independent sections.



Additional info:

none

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-382:
            
             [OSD]Should disable the ability to add idp from console and make ClusterVersion YAML editor should be read onl
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:

OSD cluster, cluster admin is not allowed to update ClusterVersion details, however console is rendering an editable YAML editor 



Version-Release number of selected component (if applicable):

4.10.18



How reproducible:

Always



Steps to Reproduce:

1. navigate to ClusterVersion YAML page /k8s/cluster/config.openshift.io~v1~ClusterVersion/version, click on YAML tab

2. cluster-admin is able to do some changes in YAML editor, however when saving the changes it will report 

An error occurred

admission webhook "regular-user-validation.managed.openshift.io" denied the request: Prevented from accessing Red Hat managed resources. This is in an effort to prevent harmful actions that may cause unintended consequences or affect the stability of the cluster. If you have any questions about this, please reach out to Red Hat support at https://access.redhat.com/support



Actual results:

2. cluster admin user is able to edit but not allowed to save the changes



Expected results:



 



ISSUE 2:



 



Steps to Reproduce:

1.On OSD console, cluster admin user adds idp from "Administration">"Cluster Settings">"Configuration"->"OAuth",

2.

3.



Actual results:

1.Could add idp successfully.



Expected results:

1. Should disable the function to add idp from OSD console.



 



Created from:



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4799:
            
             OLM generates invalid component selector labels
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
[This issue is for a backport to 4.10.z for our CI.  This issue was already addressed for 4.11+ in https://github.com/openshift/operator-framework-olm/pull/285]



Description of problem:




When installing an operator, OLM creates an "operator" Custom Resource whose status will be updated to contain a list of resources associated with the operator. This is done by labeling each resource associated with an operator with a label based off this code: https://github.com/operator-framework/operator-lifecycle-manager/blob/7eccf5342199b88f4657b6c996d4e66d9fa978fa/pkg/controller/operators/decorators/operator.go#L92-L105




Version-Release number of selected component (if applicable):




4.8




How reproducible:




Always




Steps to Reproduce:




1. Create a subscription named managed-node-metadata-operator in the openshift-managed-node-metadata-operator namespace, which causes the truncated label to end on `-`, which is an illegal character.
2. Watch the OLM Operator logs.




Actual results:




The adoption controller within OLM continuously fails to adopt the subscription due to an illegal label value:

{"level":"error","ts":1670862754.2096953,"logger":"controllers.adoption","msg":"Error adopting Subscription","request":"openshift-managed-node-metadata-operator/managed-node-metadata-operator","error":"Subscription.operators.coreos.com \"managed-node-metadata-operator\" is invalid: metadata.labels: Invalid value: \"operators.coreos.com/managed-node-metadata-operator.openshift-managed-node-metadata-\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1670862754.2097518,"logger":"controller.subscription","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"Subscription","name":"managed-node-metadata-operator","namespace":"openshift-managed-node-metadata-operator","error":"Subscription.operators.coreos.com \"managed-node-metadata-operator\" is invalid: metadata.labels: Invalid value: \"operators.coreos.com/managed-node-metadata-operator.openshift-managed-node-metadata-\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')","errorCauses":[{"error":"Subscription.operators.coreos.com \"managed-node-metadata-operator\" is invalid: metadata.labels: Invalid value: \"operators.coreos.com/managed-node-metadata-operator.openshift-managed-node-metadata-\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')"}
],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}




 



Expected results:




The adoption control creates a label that can be applied to the subscription so it may be "adopted" by the controller.




Additional info:




This was orginially fixed in 4.11 here: https://github.com/operator-framework/operator-lifecycle-manager/pull/2731




 



 



 



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-6498:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1454:
            
             [release-4.10] Unable to create Network Policies: error: unexpectedly found multiple equivalent ACLs (arp v/s arp||nd) (ns_netpol1 v/s ns_netpol2)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Intended to backport the corresponding https://bugzilla.redhat.com/show_bug.cgi?id=2095852 which has been fixed already for this version.




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5831:
            
             Empty/missing node-sizing SYSTEM_RESERVED_ES parameter can result in kubelet not starting
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-4945. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-4805. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-4101. The following is the description of the original issue: 

—

Description of problem:




We experienced two separate upgrade failures relating to the introduction of the SYSTEM_RESERVED_ES node sizing parameter, causing kubelet to stop running.

One cluster (clusterA) upgraded from 4.11.14 to 4.11.17. It experienced an issue whereby 
   /etc/node-sizing.env 
on its master nodes contained an empty SYSTEM_RESERVED_ES value:

---
cat /etc/node-sizing.env 
SYSTEM_RESERVED_MEMORY=5.36Gi
SYSTEM_RESERVED_CPU=0.11
SYSTEM_RESERVED_ES=
---

causing the kubelet to not start up. To restore service, this file was manually updated to set a value (1Gi), and kubelet was restarted.

We are uncertain what conditions led to this occuring on the clusterA master nodes as part of the upgrade.

A second cluster (clusterB) upgraded from 4.11.16 to 4.11.17. It experienced an issue whereby worker nodes were impacted by a similar problem, however this was because a custom node-sizing-enabled.env MachineConfig which did not set SYSTEM_RESERVED_ES

This caused existing worker nodes to go into a NotReady state after the ugprade, and additionally new nodes did not join the cluster as their kubelet would become impacted. 

For clusterB the conditions are more well-known of why the value is empty.

However, for both clusters, if SYSTEM_RESERVED_ES ends up as empty on a node it can cause the kubelet to not start. 

We have some asks as a result:
- Can MCO be made to recover from this situation if it occurs, perhaps  through application of a safe default if none exists, such that kubelet would start correctly?
- Can there possibly be alerting that could indicate and draw attention to the misconfiguration?




Version-Release number of selected component (if applicable):




4.11.17




How reproducible:




Have not been able to reproduce it on a fresh cluster upgrading from 4.11.16 to 4.11.17




Expected results:




If SYSTEM_RESERVED_ES is empty in /etc/node-sizing*env then a default should be applied and/or kubelet able to continue running.




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-255:
            
             lifecycle.posStart hook does not have network connectivity.
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of Bug #2081562 +++



Description of problem:



lifecycle.posStart does not have network connectivity on OpenShiftSDN CNI. (OVNKubernetes does not have the issue)




Version-Release number of selected component (if applicable):

4.10



How reproducible:

always



Steps to Reproduce:

1. create statefulset

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

$ oc create -f statefulset.yaml

$ cat statefulset.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: httpd

spec:

  serviceName: "httpd"

  replicas: 1

  selector:

    matchLabels:

      app: httpd

  template:

    metadata:

      labels:

        app: httpd

    spec:

      containers:


    
	
  • name: httpd
    
        image: registry.redhat.io/rhel8/httpd-24:1-191
    
        ports:
    • 
	
  • containerPort: 80
    
          name: web
    
        lifecycle:
    
          postStart:
    
            exec:
    
              command:
    • 
	
  • /bin/sh
    • 
	
  • -c
    • 
	
  • curl -k https://<IP:PORT> > /tmp/urltest.txt
    
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    • 





Actual results:



PostStartHook fails 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

36s         Normal    Killing               pod/httpd-0         FailedPostStartHook

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




Expected results:



PostStartHook should not fail.





Additional info:



by adding a dummy initContainers, you can workaround the issue. 

something like this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    spec:

      initContainers:


    
	
  • name: init-myservice
    
        image: busybox:1.28
    
        command: ['sh', '-c', 'sleep 2']
    
      containers:
    • 
	
  • name: httpd
    
        image: registry.redhat.io/rhel8/httpd-24:1-191
    
        ports:
    • 
	
  • containerPort: 80
    
          name: web
    
        lifecycle:
    
          postStart:
    
            exec:
    
              command:
    • 
	
  • /bin/sh
    • 
	
  • -c
    • 
	
  • curl -k <IP:PORT> > /tmp/urltest.txt
    
        ....
    
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    • 





— Additional comment from rphillips@redhat.com on 2022-05-11 19:48:10 UTC —



crio's contract with networking is to have networking up when the container starts. Moving to the openshift-sdn team to help triage what is going on.



— Additional comment from hyoskim@redhat.com on 2022-06-09 00:40:33 UTC —



Hello,



Is there any update on this issue?



— Additional comment from npinaeva@redhat.com on 2022-06-09 07:53:38 UTC —



Hello, yeah we found the root cause and working on the fix now - PR should be ready by the end of the week



— Additional comment from aos-team-art-private@bot.bugzilla.redhat.com on 2022-07-24 15:21:48 UTC —



Elliott changed bug status from MODIFIED to ON_QA.

This bug is expected to ship in the next 4.11 release.



— Additional comment from errata-xmlrpc@redhat.com on 2022-07-27 00:18:40 UTC —



This bug has been added to advisory RHSA-2022:5069 by OpenShift Release Team Bot (ocp-build/buildvm.openshift.eng.bos.redhat.com@REDHAT.COM)



— Additional comment from swasthan@redhat.com on 2022-07-27 05:38:55 UTC —



Hello Team, thank you for the help so far!



May we know if this is going to backport in v4.10.z as well?



Regards,

Swadeep



— Additional comment from zzhao@redhat.com on 2022-07-27 06:40:30 UTC —



this fixed PR is merged to build 4.12.0-0.nightly-2022-07-24-180529

So I update the target version to 4.12 version.



— Additional comment from zzhao@redhat.com on 2022-07-27 06:48:33 UTC —




still failed on build 4.12.0-0.nightly-2022-07-26-131732



Creating above statefulset and pod still cannot be worked with same error 



27s         Warning   FailedPostStartHook                                pod/httpd-0                                       Exec lifecycle hook ([/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt]) for Container "httpd" in Pod "httpd-0_default(7e519841-7092-4513-928b-03c7783ddc7d)" failed - error: command '/bin/sh -c curl -k https://<IP:PORT> > /tmp/urltest.txt' exited with 1: /bin/sh: -c: line 0: syntax error near unexpected token `>'...

85s         Normal    Killing                                            pod/httpd-0                                       FailedPostStartHook



— Additional comment from npinaeva@redhat.com on 2022-07-27 12:50:53 UTC —



Hey @zzhao@redhat.com can you share full statefulset yaml you're running?

Doesn't "line 0: syntax error near unexpected token `>'..." mean bash command is wrong?



— Additional comment from zzhao@redhat.com on 2022-07-27 13:36:43 UTC —



(In reply to Nadia Pinaeva from comment #9)

> Hey @zzhao@redhat.com can you share full statefulset yaml you're running?

> Doesn't "line 0: syntax error near unexpected token `>'..." mean bash

> command is wrong?



I'm using the statefulset from comment 0



$ cat statefulset.yaml

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: httpd

spec:

  serviceName: "httpd"

  replicas: 1

  selector:

    matchLabels:

      app: httpd

  template:

    metadata:

      labels:

        app: httpd

    spec:

      containers:


    
	
  • name: httpd
    
        image: registry.redhat.io/rhel8/httpd-24:1-191
    
        ports:
    • 
	
  • containerPort: 80
    
          name: web
    
        lifecycle:
    
          postStart:
    
            exec:
    
              command:
    • 
	
  • /bin/sh
    • 
	
  • -c
    • 
	
  • curl -k https://<IP:PORT> > /tmp/urltest.txt
    • 





— Additional comment from npinaeva@redhat.com on 2022-07-27 14:02:54 UTC —



Did you replace <IP:PORT> here "curl -k https://<IP:PORT> > /tmp/urltest.txt"?



— Additional comment from zzhao@redhat.com on 2022-07-28 07:43:25 UTC —



(In reply to Nadia Pinaeva from comment #11)

> Did you replace <IP:PORT> here "curl -k https://<IP:PORT> >

> /tmp/urltest.txt"?



oh my bad



Tested again after replacing the ip and port with following




apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: httpd

spec:

  serviceName: "httpd"

  replicas: 1

  selector:

    matchLabels:

      app: httpd

  template:

    metadata:

      labels:

        app: httpd

    spec:

      containers:


    
	
  • name: httpd
    
        image: registry.redhat.io/rhel8/httpd-24:1-191
    
        ports:
    • 
	
  • containerPort: 80
    
          name: web
    
        lifecycle:
    
          postStart:
    
            exec:
    
              command:
    • 
	
  • /bin/sh
    • 
	
  • -c
    • 
	
  • curl -k https://172.30.0.1:443 > /tmp/urltest.txt
    • 





—



on 4.12.0-0.nightly-2022-07-27-133042



$ oc get pod

NAME      READY   STATUS    RESTARTS   AGE

httpd-0   1/1     Running   0          2m28s



— Additional comment from npinaeva@redhat.com on 2022-07-29 13:02:53 UTC —



@swasthan@redhat.com yes, we are going to backport it to 4.10 (hopefully it will be faster than the fix itself  )

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-235:
            
             ClusterVersion availableUpdates is stale: PromQL conditional risks vs. slow/stuck Thanos
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:

We are seeing customer's upgrade cannot kickoff due to the availableUpdates is null in clusterversion CR

Version-Release number of the following components:




How reproducible:

sometime

Steps to Reproduce:

1.

2.

3.



Actual results:

Please include the entire output from the last TASK line through the end of output if an error is generated



Expected results:



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4467:
            
             [4.10] Improve ironic logging configuration in metal3
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-4410. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-4311. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-4305. The following is the description of the original issue: 

—

Description of problem:



Please add an option to DISABLE debug in ironic-api. Presently it is enabled by default and there is no way to disable it or reduce log level 



https://github.com/metal3-io/ironic-image/blob/main/ironic-config/ironic.conf.j2#L3








Version-Release number of selected component (if applicable): none



How reproducible: Every time



Steps to Reproduce:



Please check source code here: https://github.com/metal3-io/ironic-image/blob/main/ironic-config/ironic.conf.j2#L3 



It is enabled by default and there is no way to disable it or reduce log level 



Actual results:



Please check Case: 03371411, the log file grew to 409 GB 



Expected results: Need a way to disable debug




Additional info: Case 03371411. A cluster must gather and log file can be found in the case.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-920:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1964:
            
             Make Telemeter receive endpoint request limit configurable
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Currently, Telemeter is not equipped with configurable request limit for receive endpoint (for full context see: https://github.com/openshift/cluster-monitoring-operator/pull/1416). It is using the default limit defined in the code base, however it seems this limit might not be suitable for our usage.



As a part of this ticket, it should be:



1) Understood what is the appropriate limit for request size for our use cases



2) Make the limit configurable in Telemeter via a flag



3) Deploy the changes, initially to the staging environment, to enable our team to test it.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2282:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2622:
            
             CI: Backend unit tests fails because devfile registry was updated (mock response)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1678. The following is the description of the original issue: 

—

Description of problem:

pkg/devfile/sample_test.go fails after devfile registry was updated (https://github.com/devfile/registry/pull/126)



OCPBUGS-1677 is about updating our assertion so that the CI job runs successfully again. We might want to backport this as well.



This is about updating the code that the test should use a mock response instead of the latest registry content OR check some specific attributes instead of comparing the full JSON response.



Version-Release number of selected component (if applicable):

4.12



How reproducible:

Always



Steps to Reproduce:

1. Clone openshift/console

2. Run ./test-backend.sh



Actual results:

Unit tests fail



Expected results:

Unit tests should pass again



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1465:
            
             E2E: intermittent failure is seen on tests for devfile
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1410. The following is the description of the original issue: 

—

Clone of https://bugzilla.redhat.com/show_bug.cgi?id=2106803 to backport the e2e fix to 4.11 and 4.10.



Description of problem: E2E: intermittent failure is seen on tests for devfile due to network call to devfile registry




Deploy git workload with devfile from topology page: A-04-TC01



https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_console/11726/pull-ci-openshift-console-master-e2e-gcp-console/1547046629775773696/artifacts/e2e-gcp-console/test/artifacts/gui_test_screenshots/cypress/screenshots/e2e/add-flow-ci.feature/Deploy%20git%20workload%20with%20devfile%20from%20topology%20page%20A-04-TC01%20--%20after%20each%20hook%20(failed).png



https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_console/11768/pull-ci-openshift-console-master-e2e-gcp-console/1547061730478133248




Version-Release number of selected component (if applicable):




How reproducible: Intermittent




Steps to Reproduce:

1. Run test for add-flow-ci.feature to test Deploy git workload with devfile from topology page: A-04-TC01




Actual results: 




Expected results: Show always pass




Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1225:
            
             [4.10] Supermicro server FirmwareSchema CR does not contain allowable_values, attribute_type and read_only flag
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




See https://bugzilla.redhat.com/show_bug.cgi?id=2104275




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1951:
            
             noAllowedAddressPairs set to true causes all the ports to be added without the allowed address pairs
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1890. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-1765. The following is the description of the original issue: 

—

Description of problem:




If a customer creates a machine with a networks section like this

networks:
- filter: {}
  noAllowedAddressPairs: false
  subnets:
  - filter: {}
    uuid: primary-subnet-uuid
- filter: {}
  noAllowedAddressPairs: true
  subnets:
  - filter: {}
    uuid: other-subnet-uuid
primarySubnet: primary-subnet-uuid

Then all the ports are created without the allowed address pairs.

Doing some research in the source code, I have found that:
- For each entry on the networks: section, networks are filtered as per its filter: section[1]
- Then, if the subnets: section of the network entry is not empty, for each of the network IDs found above[2], 2 things are done that are relevant for this situatoin:
  - The net ID is saved on a netsWithoutAllowedAddressPairs[3]. That map is later checked while creating any port[4].
  - For each subnet entry that matches the network ID, a port is created[5].

So, the problematic behavior happens due to the following:

- Both entries in the networks array have empty filters. This means that both entries selected all the neutron networks.
- This configuration results in one port per subnet as expected because, in the later traversal of the subnets array of each entry[5], it is filtering by subnet and creating a single port as expected.
- However, the entry with "noAllowedAddressPairs: true" is selecting all the neutron networks, so it adds all of them to the netsWithoutAllowedAddressPairs map[3], regardless of the subnets filtering.
- As all the networks are in noAllowedAddressPairs: true array, all the ports created for the VM have their allowed address pairs removed[4].

Why do we consider this behavior undesired?

I understand that, if we create a port for a network that has no allowed pairs, we create all the other ports in the same networks without the pairs. However, it is surprising that a port in a network is removed the allowed address pairs due to a setting in an entry that yielded no port on that network. In other words, one would expect that the same subnet filtering that happens on each network entry in what regards yielding ports for the VM would also work for the noAllowedPairs parameter.






Version-Release number of selected component (if applicable):




4.10.30





How reproducible:




Always





Steps to Reproduce:




1. Create a machineset like in the description
2.
3.





Actual results:




All ports have no address pairs





Expected results:




Only the port on the secondary subnet has no address pairs.





Additional info:




A simple workaround would be to just fill the filter so that a single network is selected for each network entry.

References:
[1] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L576
[2] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L580
[3] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L581-L583
[4] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L658-L660
[5] - https://github.com/openshift/cluster-api-provider-openstack/blob/f6b51710d4f395ded401347589447f5f41dd5c4c/pkg/cloud/openstack/clients/machineservice.go#L610-L625




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-924:
            
             alertmanager-main pods failing to start due to startupprobe timeout
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:

During a fresh installation on a BareMetal platform, the monitoring cluster operator fails and becomes degraded. Further troubleshooting shows that the "alertmanagers" are not in a ready state (5/6).



Logs from the alertmanager:



level=info ts=2022-05-03T07:18:08.011Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=rhaos-4.10-rhel-8, revision=0993e91aab7afce476de5c45bead4ebb8d1295a7)"

level=info ts=2022-05-03T07:18:08.011Z caller=main.go:226 build_context="(go=go1.17.5, user=root@df86d88450ef, date=20220409-10:25:31)"



alertmanager-main pods are failing to start due to startupprobe timeout, it seems related to BZ 2037073

We tried to manually increase the timers in the startupprobe, but it was not possible.



Version-Release number of selected component (if applicable):

OCP 4.10.10



How reproducible:

OCP IPI Baremetal Install on HPE ProLiant BL460c Gen10, CU tried several time to redeploy always with the same outcome.



Actual results:

CMO is not being deployed



Expected results:

CMO deploys without errors



Additional info:


    
	
  • CU is deploying OCP 4.10 IPI on a baremetal disconnected cluster
    • 
	
  • cluster is 3 nodes with masters schedulable
    • 





 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            OCPCLOUD-1278:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1740:
            
             [4.10] ironic clear_job_queue and reset_idrac pending issues
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




I checked the log of the ironic-conductor container to get the shared lock of iDRAC, but the next installation does not proceed.

Provision Node
 ㄴ bootstrap
    ㄴ ironic-conductor    --- pending --> iDRAC8 (Baremetal master1,2,3)




Version-Release number of selected component (if applicable):




4.10.31




How reproducible:




1) baremetal inventory
   - Dell R630 firmware 2.75.75
   - iDRAC8

2) nodes
provisioning node, master 1,2,3

3) network : static IP(no DHCP) and bond/vlan
             disconnected installation




Steps to Reproduce:




1. Preparing to configure ipi on the provisioning node
   - RHEL 8 ( haproxy, named, mirror registry, rhcos_cache_server ..)

2. configuring the install-config.yaml (attached)
   
3. deploy the cluster





Actual results:




ironic does an iso mount to baremetal via idrac and starts coreos booting




Expected results:




ironic has established a session with idrac, but the installation does not proceed.




Additional info:




## curl <idrac api>
{
    "error": {
        "@Message.ExtendedInfo": [
            {
                "Message": "The request failed due to an internal service error.  The service is still operational.",
                "MessageArgs": [],
                "MessageArgs@odata.count": 0,
                "MessageId": "Base.1.2.InternalError",
                "RelatedProperties": [],
                "RelatedProperties@odata.count": 0,
                "Resolution": "Resubmit the request.  If the problem persists, consider resetting the service.",
                "Severity": "Critical"
            }
        ],
        "code": "Base.1.2.GeneralError",
        "message": "A general error has occurred. See ExtendedInfo for more information"
    }
}



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2746:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3336:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-620:
            
             oVirt CSI driver should use latest go-ovirt-client
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




To avoid any potential bugs, the oVirt CSI driver should use the latest go-ovirt-client, preferably the tagged 1.0.0 version.




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-200:
            
             Exits with a non-zero code in case Etcd backup fails
          

          
          
          

            
            
            
            
            
            
            
          

          
          
          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2523:
            
             e2e tests: Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-2451. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-2181. The following is the description of the original issue: 

—

Description of problem:




E2E test Installs Red Hat Integration - 3scale operator test is failing due to change of Operator name




CI Search: https://search.ci.openshift.org/?search=Installs+Red+Hat+Integration+-+3scale+operator+in+test+namespace+and+creates+3scale+Backend+Schema+operand+instance&maxAge=24h&context=1&type=bug%2Bissue%2Bjunit&name=pull-ci-openshift-console-master-e2e-gcp-console&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-930:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1525:
            
             package-server-manager does not migrate packageserver CSV from v0.17.0 to v0.18.3 on OCP 4.8 -> 4.9 upgrade
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1104. The following is the description of the original issue: 

—

Description of problem:



In OCP 4.9, the package-server-manager was introduced to manage the packageserver CSV. However, when OCP 4.8 in upgraded to 4.9, the packageserver stays stuck in v0.17.0, which is the version in OCP 4.8, and v0.18.3 does not roll out, which is the version in OCP 4.9



Version-Release number of selected component (if applicable):




 




How reproducible:




Always




Steps to Reproduce:




1. Install OCP 4.8

2. Upgrade to OCP 4.9 

$ oc get clusterversion 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2022-08-31-160214   True        True          50m     Working towards 4.9.47: 619 of 738 done (83% complete)

$ oc get clusterversion 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.47    True        False         4m26s   Cluster version is 4.9.47
 




Actual results:




Check packageserver CSV. It's in v0.17.0 

$ oc get csv  NAME            DISPLAY          VERSION   REPLACES   PHASE packageserver   Package Server   0.17.0               Succeeded 




Expected results:




packageserver CSV is at 0.18.3 




Additional info:




packageserver CSV version in 4.8: https://github.com/openshift/operator-framework-olm/blob/release-4.8/manifests/0000_50_olm_15-packageserver.clusterserviceversion.yaml#L12

packageserver CSV version in 4.9: https://github.com/openshift/operator-framework-olm/blob/release-4.9/pkg/manifests/csv.yaml#L8




 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2829:
            
             [IPI on Baremetal] ipv6 support issue in metal3-httpd
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1972. The following is the description of the original issue: 

—

Description of problem:




The metal3 Pod of openshift-machine-api is in CrashLoopBackOff status.




Version-Release number of selected component (if applicable):




4.10.31




How reproducible:




Always reproduce in IPv6




Steps to Reproduce:




1. Preparing to configure ipi on the provisioning node
   - RHEL 8 ( haproxy, named, mirror registry, rhcos_cache_server ..)

2. configuring the install-config.yaml (attached)
   - provisioningNetwork: disabled
   - machine network: only IPv6
   - disconnected installation
   
3. deploy the cluster 




Actual results:




It is not possible to add worker nodes because metal3 does not start normally.




Expected results:




metal3 starts normally in IPv6 environment




Additional info:




1. attached must-gather
https://drive.google.com/file/d/1GKxj3syROIMnURx_PYzOYhJdEuXBNXVW/view?usp=sharing

2. pod status
[kni@prov ~]$ oc get pod
NAME                                           READY   STATUS             RESTARTS          AGE
cluster-autoscaler-operator-6656bfd7b9-bt4j8   2/2     Running            0                 35h
cluster-baremetal-operator-6bbdd6758-rmxgq     2/2     Running            0                 35h
machine-api-controllers-55fb545b56-kl5sj       7/7     Running            0                 34h
machine-api-operator-845b6cf855-q7gdd          2/2     Running            0                 35h
metal3-574876cfdb-98fmz                        6/7     CrashLoopBackOff   179 (3m17s ago)   14h
metal3-image-cache-5mq2w                       1/1     Running            0                 14h
metal3-image-cache-nftpj                       1/1     Running            0                 14h
metal3-image-cache-t7whh                       1/1     Running            0                 14h
metal3-image-customization-68d4d6d99b-dbqgn    1/1     Running            0                 15h

[kni@prov ~]$ oc logs metal3-574876cfdb-98fmz -c metal3-httpd
AH00526: Syntax error on line 8 of /etc/httpd/conf.d/vmedia.conf:
The port number "2001:feed:101::102" is outside the appropriate range (i.e., 1..65535).



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5895:
            
             [vsphere-csi-driver-operator] CSI cloud.conf doesn't list multiple datacenters when specified 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1805. The following is the description of the original issue: 

—

The vSphere CSI cloud.conf lists the single datacenter from platform workspace config but in a multi-zone setup (https://github.com/openshift/enhancements/pull/918 ) there may be more than the one datacenter.



This issue is resulting in PVs failing to attach because the virtual machines can't be find in any other datacenter. For example:




0s Warning FailedAttachVolume pod/image-registry-85b5d5db54-m78vp AttachVolume.Attach failed for volume "pvc-ab1a0611-cb3b-418d-bb3b-1e7bbe2a69ed" : rpc error: code = Internal desc = failed to find VirtualMachine for node:"rbost-zonal-ghxp2-worker-3-xm7gw". Error: virtual machine wasn't found  




The machine above lives in datacenter-2 but the CSI cloud.conf is only aware of the datacenter IBMCloud. 




$ oc get cm vsphere-csi-config -o yaml  -n openshift-cluster-csi-drivers | grep datacenters
    datacenters = "IBMCloud" 




 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1323:
            
             Cluster Autoscaler not scaling down nodes which seem to qualify for scale-down [NEEDINFO]
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



The `clusterautoscaler` in his cluster to scale up and down the nodes automatically as per the requirement and load.

However, it has been noticed that when removing test load Pods, nodes do not scale down.



Version-Release number of selected component (if applicable):

OpenShift Version: 4.10.20



How reproducible:

Easily reproducible: https://docs.openshift.com/container-platform/4.10/rest_api/autoscale_apis/clusterautoscaler-autoscaling-openshift-io-v1.html#specification



Expected results:



When there is no load, I waited a very long time for the ClusterAutoscaler to scale down, but this never occurs. The ClusterAutoscaler controller Pod's logs keep saying no nodes are eligible to scale down despite the nodes very quite idle.



Additional info:



This Bugzilla is looking similar:

https://bugzilla.redhat.com/show_bug.cgi?id=2053343



Interestingly, I could not find the OLM and redhat-operator pod scheduled on any node other than the master.

Kindly have a look at the attached file for pod details.



 



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2546:
            
             Remove policy/v1beta1 in 4.10 and later
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
OCPBUGS-1251 landed an admin-ack gate in 4.11.z to help admins prepare for Kubernetes 1.25 API removals which are coming in OpenShift 4.12.  Poking around in a 4.12.0-ec.2 cluster where APIRemovedInNextReleaseInUse is firing:





$ oc --as system:admin adm must-gather -- /usr/bin/gather_audit_logs
$ zgrep -h v1beta1/poddisruptionbudget must-gather.local.1378724704026451055/quay*/audit_logs/kube-apiserver/*.log.gz | jq -r '.verb + " " + (.user | .username + " " + (.extra["authentication.kubernetes.io/pod-name"] | tostr
ing))' | sort | uniq -c
parse error: Invalid numeric literal at line 29, column 6
     28 watch system:serviceaccount:openshift-machine-api:cluster-autoscaler ["cluster-autoscaler-default-5cf997b8d6-ptgg7"]






Finding the source for that container:





$ oc --as system:admin -n openshift-machine-api get -o json pod cluster-autoscaler-default-5cf997b8d6-ptgg7 | jq -r '.status.containerStatuses[].image'
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f81ab7ce0c851ba5e5169bba717cb54716ce5457cbe89d159c97a5c25fd820ed
$ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f81ab7ce0c851ba5e5169bba717cb54716ce5457cbe89d159c97a5c25fd820ed | grep github
             SOURCE_GIT_URL=https://github.com/openshift/kubernetes-autoscaler
             io.openshift.build.commit.url=https://github.com/openshift/kubernetes-autoscaler/commit/1dac0311b9842958ec630273428b74703d51c1c9
             io.openshift.build.source-location=https://github.com/openshift/kubernetes-autoscaler






Poking about in the source:





$ git clone --depth 30 --branch master https://github.com/openshift/kubernetes-autoscaler.git
$ cd kubernetes-autoscaler
$ find . -name vendor
./addon-resizer/vendor
./cluster-autoscaler/vendor
./vertical-pod-autoscaler/e2e/vendor
./vertical-pod-autoscaler/vendor






Lots of vendoring.  I haven't checked to see how new the client code is in the various vendor packages.  But the main issue seems to be the v1beta1 in:





$ git grep policy cluster-autoscaler/core cluster-autoscaler/utils | grep policy.*v1beta1
cluster-autoscaler/core/scaledown/actuation/actuator_test.go:   policyv1beta1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/actuation/actuator_test.go:                                   eviction := createAction.GetObject().(*policyv1beta1.Eviction)
cluster-autoscaler/core/scaledown/actuation/drain.go:   policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/actuation/drain_test.go:      policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/legacy/legacy.go:     policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/legacy/wrapper.go:    policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/scaledown/scaledown.go: policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/core/static_autoscaler_test.go:      policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/drain/drain.go:        policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/drain/drain_test.go:   policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/kubernetes/listers.go: policyv1 "k8s.io/api/policy/v1beta1"
cluster-autoscaler/utils/kubernetes/listers.go: v1policylister "k8s.io/client-go/listers/policy/v1beta1"






The main change from v1beta1 to v1 involves spec.selector; I dunno if that's relevant to the autoscaler use-case or not.



Do we run autoscaler CI?  I was poking around a bit, but did not find a 4.12 periodic excercising the autoscaler that might have turned up this alert and issue.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3434:
            
             Pod logs: Long lines are corrupted when using timestamps=true
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3174. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-3117. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-3084. The following is the description of the original issue: 

—

Upstream Issue: https://github.com/kubernetes/kubernetes/issues/77603



Long log lines get corrupted when using '--timestamps' by the Kubelet.



The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.



https://github.com/kubernetes/kubernetes/blob/f892ab1bd7fd97f1fcc2e296e85fdb8e3e8fb82d/pkg/kubelet/kuberuntime/logs/logs.go#L325





apiVersion: v1
kind: Pod
metadata:
  name: logs
spec:
  restartPolicy: Never
  containers:
  - name: logs
    image: fedora
    args:
    - bash
    - -c
    - 'for i in `seq 1 10000000`; do echo -n $i; done'








kubectl logs logs --timestamps




            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1659:
            
             set relatedObjects in ClusterOperator manifests
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
 As mentioned in [1], the cluster monitoring operator doesn't define the relatedObjects field in the ClusterOperator manifest which is initially deployed by CVO [2].

If the CMO pod fails to start, the must-gather might miss information from the monitoring namespace. Note that once CMO runs, it will update the initial ClusterOperator object with the proper information [3].



[1] http://mailman-int.corp.redhat.com/archives/aos-devel/2021-May/msg00139.html

[2] https://github.com/openshift/cluster-monitoring-operator/blob/master/manifests/0000_50_cluster-monitoring-operator_06-clusteroperator.yaml

[3] https://github.com/openshift/cluster-monitoring-operator/blob/a6bc9824035ceb8dbfe7c53cf0c138bfb2ec5643/pkg/client/status_reporter.go#L49-L63

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1872:
            
             Use upstream kube-thanos in cluster-monitoring-operator jsonnet
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
As per [1], the jsonnet code for managing thanos-ruler resources should reuse the upstream kube-thanos project.



[1] https://github.com/openshift/cluster-monitoring-operator/blob/399c84dbca596b611b0c30a0d2df63a5d2b0b8cc/jsonnet/components/thanos-ruler.libsonnet#L1

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1037:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-249:
            
             Incorrect NAT when using cluster networking in control-plane nodes to install a VRRP Cluster 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of 

Bug #2070318

 +++



Description of problem:

In OCP VRRP deployment (using OCP cluster networking), we have an additional data interface which is configured along with the regular management interface in each control node. In some deployments, the kubernetes address 172.30.0.1:443 is nat’ed to the data management interface instead of the mgmt interface (10.40.1.4:6443 vs 10.30.1.4:6443 as we configure the boostrap node) even though the default route is set to 10.30.1.0 network. Because of that, all requests to 172.30.0.1:443 were failed. After 10-15 minutes, OCP magically fixes it and nat’ing correctly to 10.30.1.4:6443.



Version-Release number of selected component (if applicable):




How reproducible:




Steps to Reproduce:



1.Provision OCP cluster using cluster networking for DNS & Load Balancer instead of external DNS & Load Balancer. Provision the host with 1 management interface and an additional interface for data network. Along with OCP manifest, add manifest to create a pod which will trigger communication with kube-apiserver.  



2.Start cluster installation. 



3.Check on the custom pod log in the cluster when the first 2 master nodes were installing to see GET operation to kube-apiserver timed out. Check nft table and chase the ip chains to see the that the data IP address was nat'ed to kubernetes service IP address instead of the management IP. This is not happening all the time, we have seen 50:50 chance. 



Actual results:

After 10-15 minutes OCP will correct that by itself.




Expected results:

Wrong natting should not happen. 



Additional info:

ClusterID: 24bbde0b-79b3-4ae6-afc5-cb694fa48895

ClusterVersion: Stable at "4.8.29"

ClusterOperators:

	clusteroperator/authentication is not available (OAuthServerRouteEndpointAccessibleControllerAvailable: Get "

https://oauth-openshift.apps.ocp-binhle-wqepch.contrail.juniper.net/healthz

": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) because OAuthServerRouteEndpointAccessibleControllerDegraded: Get "

https://oauth-openshift.apps.ocp-binhle-wqepch.contrail.juniper.net/healthz

": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

	clusteroperator/baremetal is degraded because metal3 deployment inaccessible

	clusteroperator/console is not available (RouteHealthAvailable: failed to GET route (

https://console-openshift-console.apps.ocp-binhle-wqepch.contrail.juniper.net/health

): Get "

https://console-openshift-console.apps.ocp-binhle-wqepch.contrail.juniper.net/health

": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) because RouteHealthDegraded: failed to GET route (

https://console-openshift-console.apps.ocp-binhle-wqepch.contrail.juniper.net/health

): Get "

https://console-openshift-console.apps.ocp-binhle-wqepch.contrail.juniper.net/health

": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

	clusteroperator/dns is progressing: DNS "default" reports Progressing=True: "Have 4 available DNS pods, want 5."

	clusteroperator/ingress is degraded because The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)

	clusteroperator/insights is degraded because Unable to report: unable to build request to connect to Insights server: Post "

https://cloud.redhat.com/api/ingress/v1/upload

": dial tcp: lookup cloud.redhat.com on 172.30.0.10:53: read udp 10.128.0.26:53697->172.30.0.10:53: i/o timeout

	clusteroperator/network is progressing: DaemonSet "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes)



— Additional comment from 

bnemec@redhat.com

 on 2022-03-30 20:00:25 UTC —



This is not managed by runtimecfg, but in order to route the bug correctly I need to know which CNI plugin you're using - OpenShiftSDN or OVNKubernetes. Thanks.



— Additional comment from 

lpbinh@gmail.com

 on 2022-03-31 08:09:11 UTC —



Hi Ben,



We were deploying Contrail CNI with OCP. However, this issue happens at very early deployment time, right after the bootstrap node is started

and there's no SDN/CNI there yet.



— Additional comment from 

bnemec@redhat.com

 on 2022-03-31 15:26:23 UTC —



Okay, I'm just going to send this to the SDN team then. They'll be able to provide more useful input than I can.



— Additional comment from 

trozet@redhat.com

 on 2022-04-04 15:22:21 UTC —



Can you please provide the iptables rules causing the DNAT as well as the routes on the host? Might be easiest to get a sosreport during initial bring up during that 10-15 min when the problem occurs.



— Additional comment from 

lpbinh@gmail.com

 on 2022-04-05 16:45:13 UTC —



All nodes have two interfaces:



eth0: 10.30.1.0/24

eth1: 10.40.1.0/24



machineNetwork is 10.30.1.0/24

default route points to 10.30.1.1



The kubeapi service ip is 172.30.0.1:443



all Kubernetes services are supposed to be reachable via machineNetwork (10.30.1.0/24)



To make the kubeapi service ip reachable in hostnetwork, something (openshift installer?) creates a set of nat rules which translates the service ip to the real ip of the nodes which have kubeapi active.



Initially kubeapi is only active on the bootstrap node so there should be a nat rule like



172.30.0.1:443 -> 10.30.1.10:6443 (assuming that 10.30.1.10 is the bootstrap nodes' ip address in the machine network)



However, what we see is

172.30.0.1:443 -> 10.40.1.10:6443 (which is the bootstrap nodes' eth1 ip address)



The rule is configured on the controller nodes and lead to asymmetrical routing as the controller sends a packet FROM machineNetwork (10.30.1.x) to 172.30.0.1 which is then translated and forwarded to 10.40.1.10 which then tries to reply back on the 10.40.1.0 network which fails as the request came from 10.30.1.0 network.



So, we want to understand why openshift installer picks the 10.40.1.x ip address rather than the 10.30.1.x ip for the nat rule. What's the mechanism for getting the ip in case the system has multiple interfaces with ips configured.



Note: after a while (10-20 minutes) the bootstrap process resets itself and then it picks the correct ip address from the machineNetwork and things start to work.



— Additional comment from 

smerrow@redhat.com

 on 2022-04-13 13:55:04 UTC —



Note from Juniper regarding requested SOS report:



In reference to 

https://bugzilla.redhat.com/show_bug.cgi?id=2070318

 that @Binh Le has been working on. The mustgather was too big to upload for this Bugzilla. Can you access this link? 

https://junipernetworks-my.sharepoint.com/:u:/g/personal/sleigon_juniper_net/ETOrHMqao1tLm10Gmq9rzikB09H5OUwQWZRAuiOvx1nZpQ


    
	
  • Making note private to hide partner link
    • 





— Additional comment from 

smerrow@redhat.com

 on 2022-04-21 12:24:33 UTC —



Can we please get an update on this BZ?



Do let us know if there is any other information needed.



— Additional comment from 

trozet@redhat.com

 on 2022-04-21 14:06:00 UTC —



Can you please provide another link to the sosreport? Looks like the link is dead.



— Additional comment from 

smerrow@redhat.com

 on 2022-04-21 19:01:39 UTC —



See mustgather here: 

https://drive.google.com/file/d/16y9IfLAs7rtO-SMphbYBPgSbR4od5hcQ

— Additional comment from 

trozet@redhat.com

 on 2022-04-21 20:57:24 UTC —



Looking at the must-gather I think your iptables rules are most likely coming from the fact that kube-proxy is installed:



[trozet@fedora must-gather.local.288458111102725709]$ omg get pods -n openshift-kube-proxy

NAME                        READY  STATUS   RESTARTS  AGE

openshift-kube-proxy-kmm2p  2/2    Running  0         19h

openshift-kube-proxy-m2dz7  2/2    Running  0         16h

openshift-kube-proxy-s9p9g  2/2    Running  1         19h

openshift-kube-proxy-skrcv  2/2    Running  0         19h

openshift-kube-proxy-z4kjj  2/2    Running  0         19h



I'm not sure why this is installed. Is it intentional? I don't see the configuration in CNO to enable kube-proxy. Anyway the node IP detection is done via:

https://github.com/kubernetes/kubernetes/blob/f173d01c011c3574dea73a6fa3e20b0ab94531bb/cmd/kube-proxy/app/server.go#L844

Which just looks at the IP of the node. During bare metal install a VIP is chosen and used with keepalived for kubelet to have kapi access. I don't think there is any NAT rule for services until CNO comes up. So I suspect what really is happening is your node IP is changing during install, and kube-proxy is getting deployed (either intentionally or unintentionally) and that is causing the behavior you see. The node IP is chosen via the node ip configuration service:

https://github.com/openshift/machine-config-operator/blob/da6494c26c643826f44fbc005f26e0dfd10513ae/templates/common/_base/units/nodeip-configuration.service.yaml

This service will determine the node ip via which interfaces have a default route and which one has the lowest metric. With your 2 interfaces, do they both have default routes? If so, are they using dhcp and perhaps its random which route gets installed with a lower metric?



— Additional comment from 

trozet@redhat.com

 on 2022-04-21 21:13:15 UTC —



Correction: looks like standalone kube-proxy is installed by default when the provider is not SDN, OVN, or kuryr so this looks like the correct default behavior for kube-proxy to be deployed.



— Additional comment from 

lpbinh@gmail.com

 on 2022-04-25 04:05:14 UTC —



Hi Tim,



You are right, kube-proxy is deployed by default and we don't change that behavior.



There is only 1 default route configured for the management interface (10.30.1.x) , we used to have a default route for the data/vrrp interface (10.40.1.x) with higher metric before. As said, we don't have the default route for the second interface any more but still encounter the issue pretty often.



— Additional comment from 

trozet@redhat.com

 on 2022-04-25 14:24:05 UTC —



Binh, can you please provide a sosreport for one of the nodes that shows this behavior? Then we can try to figure out what is going on with the interfaces and the node ip service. Thanks.



— Additional comment from 

trozet@redhat.com

 on 2022-04-25 16:12:04 UTC —



Actually Ben reminded me that the invalid endpoint is actually the boostrap node itself:

172.30.0.1:443 -> 10.30.1.10:6443 (assuming that 10.30.1.10 is the bootstrap nodes' ip address in the machine network)



vs

172.30.0.1:443 -> 10.40.1.10:6443 (which is the bootstrap nodes' eth1 ip address)



So maybe a sosreport off that node is necessary? I'm not as familiar with the bare metal install process, moving back to Ben.



— Additional comment from 

lpbinh@gmail.com

 on 2022-04-26 08:33:45 UTC —



Created attachment 1875023 [details]sosreport



— Additional comment from 

lpbinh@gmail.com

 on 2022-04-26 08:34:59 UTC —



Created attachment 1875024 [details]sosreport-part2



Hi Tim,



We observe this issue when deploying clusters using OpenStack instances as our infrastructure is based on OpenStack.



I followed the steps here to collect the sosreport: 

https://docs.openshift.com/container-platform/4.8/support/gathering-cluster-data.html

Got the sosreport which is 22MB which exceeds the size permitted (19MB), so I split it to 2 files (xaa and xab), if you can't join them then we will need to put the collected sosreport  on a share drive like we did with the must-gather data.



Here are some notes about the cluster:



First two control nodes are below, ocp-binhle-8dvald-ctrl-3 is the bootstrap node. 



[core@ocp-binhle-8dvald-ctrl-2 ~]$ oc get node

NAME                       STATUS   ROLES    AGE   VERSION

ocp-binhle-8dvald-ctrl-1   Ready    master   14m   v1.21.8+ed4d8fd

ocp-binhle-8dvald-ctrl-2   Ready    master   22m   v1.21.8+ed4d8fd




We see the behavior that wrong nat'ing was done at the beginning, then corrected later:



sh-4.4# nft list table ip nat | grep 172.30.0.1

		meta l4proto tcp ip daddr 172.30.0.1  tcp dport 443 counter packets 3 bytes 180 jump KUBE-SVC-NPX46M4PTMTKRN6Y

sh-4.4# nft list chain ip nat KUBE-SVC-NPX46M4PTMTKRN6Y

table ip nat {

	chain KUBE-SVC-NPX46M4PTMTKRN6Y 

{
		 counter packets 3 bytes 180 jump KUBE-SEP-VZ2X7DROOLWBXBJ4
	}

}

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 

{
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 3 bytes 180 dnat to 10.40.1.7:6443
	}

}

sh-4.4#

sh-4.4#

<....after a while....>

sh-4.4# nft list chain ip nat KUBE-SVC-NPX46M4PTMTKRN6Y

table ip nat {

	chain KUBE-SVC-NPX46M4PTMTKRN6Y 

{
		 counter packets 0 bytes 0 jump KUBE-SEP-X33IBTDFOZRR6ONM
	}

}

sh-4.4# nft list table ip nat | grep 172.30.0.1

		meta l4proto tcp ip daddr 172.30.0.1  tcp dport 443 counter packets 0 bytes 0 jump KUBE-SVC-NPX46M4PTMTKRN6Y

sh-4.4# nft list chain ip nat KUBE-SVC-NPX46M4PTMTKRN6Y

table ip nat {

	chain KUBE-SVC-NPX46M4PTMTKRN6Y {
		 counter packets 0 bytes 0 jump KUBE-SEP-X33IBTDFOZRR6ONM
	}

}

sh-4.4# nft list chain ip nat KUBE-SEP-X33IBTDFOZRR6ONM

table ip nat {

	chain KUBE-SEP-X33IBTDFOZRR6ONM 

{
		ip saddr 10.30.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 0 bytes 0 dnat to 10.30.1.7:6443
	}

}

sh-4.4#



— Additional comment from 

lpbinh@gmail.com

 on 2022-05-12 17:46:51 UTC —



@

trozet@redhat.com

 May we have an update on the fix, or the plan for the fix? Thank you.



— Additional comment from 

lpbinh@gmail.com

 on 2022-05-18 21:27:45 UTC —



Created support Case 03223143.



— Additional comment from 

vkochuku@redhat.com

 on 2022-05-31 16:09:47 UTC —



Hello Team,



Any update on this?



Thanks,

Vinu K



— Additional comment from 

smerrow@redhat.com

 on 2022-05-31 17:28:54 UTC —



This issue is causing delays in Juniper's CI/CD pipeline and makes for a less than ideal user experience for deployments. 



I'm getting a lot of pressure from the partner on this for an update and progress. I've had them open a case [1] to help progress.



Please let us know if there is any other data needed by Juniper or if there is anything I can do to help move this forward.



[1] 

https://access.redhat.com/support/cases/#/case/03223143

— Additional comment from 

vpickard@redhat.com

 on 2022-06-02 22:14:23 UTC —



@

bnemec@redhat.com

 Tim mentioned in 

https://bugzilla.redhat.com/show_bug.cgi?id=2070318#c14

 that this issue appears to be at BM install time. Is this something you can help with, or do we need help from the BM install team?



— Additional comment from 

bnemec@redhat.com

 on 2022-06-03 18:15:17 UTC —



Sorry, I missed that this came back to me.



(In reply to Binh Le from 

comment #16

)> We observe this issue when deploying clusters using OpenStack instances as

> our infrastructure is based on OpenStack.This does not match the configuration in the must-gathers provided so far, which are baremetal. Are we talking about the same environments?



I'm currently discussing this with some other internal teams because I'm unfamiliar with this type of bootstrap setup. I need to understand what the intended behavior is before we decide on a path forward.



— Additional comment from 

rurena@redhat.com

 on 2022-06-06 14:36:54 UTC —



(In reply to Ben Nemec from 

comment #22

)> Sorry, I missed that this came back to me.

> 

> (In reply to Binh Le from comment #16)

> > We observe this issue when deploying clusters using OpenStack instances as

> > our infrastructure is based on OpenStack.

> 

> This does not match the configuration in the must-gathers provided so far,

> which are baremetal. Are we talking about the same environments?

> 

> I'm currently discussing this with some other internal teams because I'm

> unfamiliar with this type of bootstrap setup. I need to understand what the

> intended behavior is before we decide on a path forward.I spoke to the CU they tell me that all work should be on baremetal. They were probably just testing on OSP and pointing out that they saw the same behavior.



— Additional comment from 

bnemec@redhat.com

 on 2022-06-06 16:19:37 UTC —



Okay, I see now that this is an assisted installer deployment. Can we get the cluster ID assigned by AI so we can take a look at the logs on our side? Thanks.



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-06 16:38:56 UTC —




Here is the cluster ID, copied from the bug description:

ClusterID: 24bbde0b-79b3-4ae6-afc5-cb694fa48895



In regard to your earlier question about OpenStack & baremetal (2022-06-03 18:15:17 UTC):



We had an issue with platform validation in OpenStack earlier. Host validation was failing with the error message “Platform network settings: Platform OpenStack Compute is allowed only for Single Node OpenShift or user-managed networking.”



It's found out that there is no platform type "OpenStack" available in [

https://github.com/openshift/assisted-service/blob/master/models/platform_type.go#L29

] so we set "baremetal" as the platform type on our computes. That's the reason why you are seeing baremetal as the platform type.



Thank you



— Additional comment from 

ercohen@redhat.com

 on 2022-06-08 08:00:18 UTC —



Hey, first you are currect, When you set 10.30.1.0/24 as the machine network, the bootstrap process should use the IP on that subnet in the bootstrap node.



I'm trying to understand how exactly this cluster was installed.

You are using on-prem deployment of assisted-installer (podman/ACM)?

You are trying to form a cluster from OpenStack Vms?

You set the platform to Baremetal where?

Did you set user-managed-netwroking?




Some more info, when using OpenStack platform you should install the cluster with user-managed-netwroking.

And that's what the failing validation is for.



— Additional comment from 

bnemec@redhat.com

 on 2022-06-08 14:56:53 UTC —



Moving to the assisted-installer component for further investigation.



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-09 07:37:54 UTC —



@Eran Cohen:



Please see my response inline.



You are using on-prem deployment of assisted-installer (podman/ACM)?

--> Yes, we are using on-prem deployment of assisted-installer.



You are trying to form a cluster from OpenStack Vms?

--> Yes.



You set the platform to Baremetal where?

--> It was set in the Cluster object, Platform field when we model the cluster.



Did you set user-managed-netwroking?

--> Yes, we set it to false for VRRP.



— Additional comment from 

itsoiref@redhat.com

 on 2022-06-09 08:17:23 UTC —



@

lpbinh@gmail.com

 can you please share assisted logs that you can download when cluster is failed or installed? 

Will help us to see the full picture



— Additional comment from 

ercohen@redhat.com

 on 2022-06-09 08:23:18 UTC —



OK, as noted before when using OpenStack platform you should install the cluster with user-managed-netwroking (set to true).

Can you explain how you workaround this failing validation? “Platform network settings: Platform OpenStack Compute is allowed only for Single Node OpenShift or user-managed networking.”

What does this mean exactly? 'we set "baremetal" as the platform type on our computes'



To be honest I'm surprised that the installation was completed successfully.



@

oamizur@redhat.com

 I thought installing on OpenStack VMs with baremetal platform (user-managed-networking=false) will always fail?



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-10 16:04:56 UTC —



@

itsoiref@redhat.com

 : I will reproduce and collect the logs. Is that supposed to be included in the provided must-gather?

@

ercohen@redhat.com

:


    
	
  • user-managed-networking set to true when we use external Load Balancer and DNS server. For VRRP we use OpenShift's internal LB and DNS server hence it's set to false, following the doc.
    • 
	
  • As explained OpenShift returns platform type as 'none' for OpenStack:
    
https://github.com/openshift/assisted-service/blob/master/models/platform_type.go#L29
    
, therefore we set the platformtype as 'baremetal' in the cluster object for provisioning the cluster using OpenStack VMs.
    • 





— Additional comment from 

itsoiref@redhat.com

 on 2022-06-13 13:08:17 UTC —



@

lpbinh@gmail.com

 you will have download_logs link in UI. Those logs are not part of must-gather



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-14 18:52:02 UTC —



Created attachment 1889993 [details]cluster log per need info request - Cluster ID caa475b0-df04-4c52-8ad9-abfed1509506



Attached is the cluster log per need info request.

Cluster ID: caa475b0-df04-4c52-8ad9-abfed1509506

In this reproduction, the issue is not resolved by OpenShift itself, wrong NAT still remained and cluster deployment failed eventually



sh-4.4# nft list table ip nat | grep 172.30.0.1

		meta l4proto tcp ip daddr 172.30.0.1  tcp dport 443 counter packets 2 bytes 120 jump KUBE-SVC-NPX46M4PTMTKRN6Y

sh-4.4# nft list chain ip nat KUBE-SVC-NPX46M4PTMTKRN6Y

table ip nat {

	chain KUBE-SVC-NPX46M4PTMTKRN6Y 

{
		 counter packets 2 bytes 120 jump KUBE-SEP-VZ2X7DROOLWBXBJ4
	}

}

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 {
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 2 bytes 120 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 17:40:19 UTC 2022

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 {
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 2 bytes 120 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 17:59:19 UTC 2022

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 {
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 9 bytes 540 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 18:17:38 UTC 2022

sh-4.4#

sh-4.4#

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 {
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 7 bytes 420 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 18:49:28 UTC 2022

sh-4.4#



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-14 18:56:06 UTC —



Created attachment 1889994 [details]cluster log per need info request - Cluster ID caa475b0-df04-4c52-8ad9-abfed1509506



Please find the cluster-log attached per your request. In this deployment the wrong NAT was not automatically resolved by OpenShift hence the deployment failed eventually.



sh-4.4# nft list table ip nat | grep 172.30.0.1

		meta l4proto tcp ip daddr 172.30.0.1  tcp dport 443 counter packets 2 bytes 120 jump KUBE-SVC-NPX46M4PTMTKRN6Y

sh-4.4# nft list chain ip nat KUBE-SVC-NPX46M4PTMTKRN6Y

table ip nat {

	chain KUBE-SVC-NPX46M4PTMTKRN6Y {
		 counter packets 2 bytes 120 jump KUBE-SEP-VZ2X7DROOLWBXBJ4
	}

}

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 

{
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 2 bytes 120 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 17:40:19 UTC 2022

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 

{
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 2 bytes 120 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 17:59:19 UTC 2022

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 

{
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 9 bytes 540 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 18:17:38 UTC 2022

sh-4.4#

sh-4.4#

sh-4.4# nft list chain ip nat KUBE-SEP-VZ2X7DROOLWBXBJ4; date

table ip nat {

	chain KUBE-SEP-VZ2X7DROOLWBXBJ4 

{
		ip saddr 10.40.1.7  counter packets 0 bytes 0 jump KUBE-MARK-MASQ
		meta l4proto tcp   counter packets 7 bytes 420 dnat to 10.40.1.7:6443
	}

}

Tue Jun 14 18:49:28 UTC 2022

sh-4.4#



— Additional comment from 

itsoiref@redhat.com

 on 2022-06-15 15:59:22 UTC —



@

lpbinh@gmail.com

 just for the protocol, we don't support baremetal ocp on openstack that's why validation is failing



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-15 17:47:39 UTC —



@

itsoiref@redhat.com

 as explained it's just a workaround on our side to make OCP work in our lab, and from my understanding on OCP perspective it will see that deployment is on baremetal only, not related to OpenStack (please correct me if I am wrong). 



We have been doing thousands of OCP cluster deployments in our automation so far, if it's why validation is failing then it should be failing every time. However it only occurs occasionally when nodes have 2 interfaces, using OCP internal DNS and Load balancer, and sometime resolved by itself and sometime not.



— Additional comment from 

itsoiref@redhat.com

 on 2022-06-19 17:00:01 UTC —




For now i can assume that this endpoint is causing the issue:

{

            "apiVersion": "v1",

            "kind": "Endpoints",

            "metadata": {

                "creationTimestamp": "2022-06-14T17:31:10Z",

                "labels": 

{
                    "endpointslice.kubernetes.io/skip-mirror": "true"
                }

,

                "name": "kubernetes",

                "namespace": "default",

                "resourceVersion": "265",

                "uid": "d8f558be-bb68-44ac-b7c2-85ca7a0fdab3"

            },

            "subsets": [

                {

                    "addresses": [

                        {
                            "ip": "10.40.1.7"
                        }

                    ],

                    "ports": [

                        {

                            "name": "https",

                            "port": 6443,

                            "protocol": "TCP"

                        }

                    ]

                }

            ]

        },



— Additional comment from 

itsoiref@redhat.com

 on 2022-06-21 17:03:51 UTC —



The issue is that kube-api service advertise wrong ip but it does it cause kubelet chooses the one arbitrary and we currently have no mechanism to set kubelet ip, especially in bootstrap flow.



— Additional comment from 

lpbinh@gmail.com

 on 2022-06-22 16:07:29 UTC —



@

itsoiref@redhat.com

 how do you perform OCP deployment in setups that have multiple interfaces if letting kubelet chooses an interface arbitrary instead of configuring a specific IP address for it to listen on? With what you describe above chance of deployment failure in system with multiple interfaces would be high.



— Additional comment from 

dhellard@redhat.com

 on 2022-06-24 16:32:26 UTC —



I set the Customer Escalation flag = Yes, per ACE EN-52253. 

The impact is noted by the RH Account team: "Juniper is pressing and this impacts the Unica Next Project at Telefónica Spain. Unica Next is a critical project for Red Hat. We go live the 1st of July and this issue could impact the go live dates. We need clear information about the status and its possible resolution.



— Additional comment from 

itsoiref@redhat.com

 on 2022-06-26 07:28:44 UTC —



I have sent an image with possible fix to Juniper and waiting for their feedback, once they will confirm it works for them we will proceed with the PRs.



— Additional comment from 

pratshar@redhat.com

 on 2022-06-30 13:26:26 UTC —



 === In Red Hat Customer Portal Case 03223143 ===

— Comment by Prateeksha Sharma on 6/30/2022 6:56 PM —



//EMT note//



Update from our consultant Manuel Martinez Briceno -



====

on 28th June, 2022 the last feedback from Juniper Project Manager and our Partner Manager was that they are testing the fix. They didn't give an Estimate Time to finish, but we will be tracking this closely and let us know of any news.

====



Thanks & Regards,

Prateeksha Sharma

Escalation Manager | RHCSA

Global Support Services, Red Hat

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5012:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-1656:
            
             Add a Makefile rule in CMO for verifications and checks
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Add a Makefile rule in CMO to execute all the different rule that are used for verification and validation. Currenctly, some of them might not be at the right place, for example `check-assets` which is part of `generate` despite not being responsible of any generation. https://github.com/openshift/cluster-monitoring-operator/pull/1151/files#r629371735



DoD:


    
	
  • Add a new rule in CMO to handle verification
    • 
	
  • Add a CI job for this rule
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1634:
            
             Developer catalog fails to load
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Customer is facing issue similar to https://github.com/devfile/api/issues/897




Version-Release number of selected component (if applicable):



OCP 4.10.17



How reproducible:

N/A

Steps to Reproduce:

1.

2.

3.



Actual results:




Expected results:




Additional info:



Tried working around it with ALL_PROXY but it did not help. Note because the console operator reverts changes pretty quickly testing this was a bit of a PITA

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5507:
            
             [vsphere-CSI-Driver-Operator] The storageclass "thin-csi" could not be re-created after deleting 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




The storageclass "thin-csi" is created by vsphere-CSI-Driver-Operator, after deleting it manually, it should be re-created immediately. 




Version-Release number of selected component (if applicable):




4.11.4




How reproducible:




Always




Steps to Reproduce:




1. Check storageclass in running cluster, thin-csi is present:
$ oc get sc 
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate              false                  41m
thin-csi         csi.vsphere.vmware.com         Delete          WaitForFirstConsumer   true                   38m







2. Delete thin-csi storageclass:
$ oc delete sc thin-csi
storageclass.storage.k8s.io "thin-csi" deleted







3. Check storageclass again, thin-csi is not present:
$ oc get sc
NAME             PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
thin (default)   kubernetes.io/vsphere-volume   Delete          Immediate           false                  50m







4. Check vmware-vsphere-csi-driver-operator log:
......
I0909 03:47:42.172866       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1662695014\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1662695014\" (2022-09-09 02:43:34 +0000 UTC to 2023-09-09 02:43:34 +0000 UTC (now=2022-09-09 03:47:42.172853123 +0000 UTC))"I0909 03:49:38.294962       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295468       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOFI0909 03:49:38.295765       
1 streamwatcher.go:111] Unexpected EOF during watch stream event decoding: unexpected EOF








5. Only first time creating in vmware-vsphere-csi-driver-operator log:
$ oc -n openshift-cluster-csi-drivers logs vmware-vsphere-csi-driver-operator-7cc6d44b5c-c8czw | grep -i "storageclass"I0909 03:46:31.865926   1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-csi-drivers", Name:"vmware-vsphere-csi-driver-operator", UID:"9e0c3e2d-d403-40a1-bf69-191d7aec202b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'StorageClassCreated' Created StorageClass.storage.k8s.io/thin-csi because it was missing 




Actual results:




The storageclass "thin-csi" could not be re-created after deleting




Expected results:




The storageclass "thin-csi" should be re-created after deleting




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1024:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-174:
            
             [OVN] New pods unable to establish TCP connections and get constant timeouts causing application downtime
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of Bug #2118717 +++

https://bugzilla.redhat.com/show_bug.cgi?id=2118717



Description of problem:

This BZ is a spin-off of BZ-2114945 so we can track possible issues with new TCP connections from pods failing to be created on the nodes leading to pods being unable to start or crash.



Version-Release number of selected component (if applicable):

OCP 4.10.24 with OVN-Kubernetes



How reproducible:

Periodically on the customer only so far.



— Additional comment from Andre Costa on 2022-08-10 16:30:00 UTC —



There are 3 must-gathers here that were gathered during the issues and after the restart of OVNk-masters which makes all these issues go away and pods start connections immediately.



This must-gather was taken at 11 AM today when they received a report from one of the customers:



https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/3f820e22-3d1e-4c01-b56d-e479a6cc5578



Customer reported the issue again and this time we also got sosreport and inspect from the project.

In the pod they get errors like this (the same we saw on the call last week with them where it seems no TCP connections entries are created at all. First we though it was DNS but even with IPs directly there were issues like this):

-----------------

mx-toni-dev                                          toni-dev-build                           0/1     Error       0             18m   10.195.80.253   demchdc5vvx   <none>           <none>



[z0003rbj-z07@stuart ~]$ oc logs toni-dev-build

time="2022-08-10T10:59:02Z" level=info msg="Start building app with registry type openshift"

time="2022-08-10T10:59:02Z" level=info msg="Adding ssl certificate /etc/ssl/certs/ca-bundle.crt"

time="2022-08-10T10:59:02Z" level=info msg="Certificate /etc/ssl/certs/ca-bundle.crt has been added successfully"

time="2022-08-10T10:59:02Z" level=info msg="Updating docker config with registry credentials"

time="2022-08-10T10:59:02Z" level=info msg="Docker config has been updated with registry credentials"

time="2022-08-10T10:59:02Z" level=info msg="Downloading MDA from https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/eba21059-8896-4cb7-8971-4d61f6756273/71042"

time="2022-08-10T10:59:32Z" level=error msg="Failed to build mendix app, failed to create application layer failed to download MDA from https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/eba21059-8896-4cb7-8971-4d61f6756273/71042, Get \"https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/eba21059-8896-4cb7-8971-4d61f6756273/71042\": proxyconnect tcp: dial tcp: i/o timeout: Get \"https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/eba21059-8896-4cb7-8971-4d61f6756273/71042\": proxyconnect tcp: dial tcp: i/o timeout"

-----------------------



This keeps happening if they continue to run the builds which they did and created the MG and sosreport:



https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/d10d9646-5437-4806-b34f-3acab4d83266



https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/1e9e0575-66b0-4184-a3d4-e84fa245c9a2



And like we have seen so far restarting the ovnk-master pods makes these connections work immediately again:



https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/e03ee9c6-2ce4-40d3-a484-84da61607de5



https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/652ebe47-b71f-4d63-8499-cf989b3ebfc7



— Additional comment from Andre Costa on 2022-08-10 16:30:43 UTC —





— Additional comment from Tim Rozet on 2022-08-10 22:36:06 UTC —



Thanks for the must gathers. From Flavio and I examining them, there is definitely a bug here in ovn-kube. The toni-dev-build pod is deleted/recreated multiple times, and during this time it moves to different nodes. However due to a bug in OVNK, this port is updated with the new ip address and information as if it was moving to the new node, but stays on the previous logical switch. So for example, this is what happens:



1. The pod is originally assigned to node demchdc6zax. This node's cluster subnet is 10.195.79.0/24:



2022-08-09T09:22:57.078688727+00:00 stderr F I0809 09:22:57.078632 2239319 cni.go:248] [mx-toni-dev/toni-dev-build b1a4fb0be20ff717f85fd0fffab4fb303bbcb0f8b68aced4852fb7a2465d2df1] ADD finished CNI request [mx-toni-dev/toni-dev-build b1a4fb0be20ff717f85fd0fffab4fb303bbcb0f8b68aced4852fb7a2465d2df1], result "{\"interfaces\":[

{\"name\":\"b1a4fb0be20ff71\",\"mac\":\"a6:68:38:ad:66:c8\"}

,

{\"name\":\"eth0\",\"mac\":\"0a:58:0a:c3:4f:52\",\"sandbox\":\"/var/run/netns/e354d2d5-83cb-406f-a2d9-c5f3e786bae4\"}

],\"ips\":[

{\"version\":\"4\",\"interface\":1,\"address\":\"10.195.79.82/24\",\"gateway\":\"10.195.79.1\"}

],\"dns\":{}}", err <nil



2. Over time this pod is completed, deleted, recreated many times. Until eventually it lands on demchdc5vvx the next day:

2022-08-10T08:43:58.428759111Z I0810 08:43:58.428719 1837017 cni.go:248] [mx-toni-dev/toni-dev-build 5d4f195cbda5269e5451593987be9d69ea828ee549bc447a5bbe50db847c182a] ADD finished CNI request [mx-toni-dev/toni-dev-build 5d4f195cbda5269e5451593987be9d69ea828ee549bc447a5bbe50db847c182a], result "{\"interfaces\":[

{\"name\":\"5d4f195cbda5269\",\"mac\":\"a2:22:6c:1d:43:c1\"}

,

{\"name\":\"eth0\",\"mac\":\"0a:58:0a:c3:50:1e\",\"sandbox\":\"/var/run/netns/bae8f77a-b368-4b3a-86dc-df925330fa26\"}

],\"ips\":[

{\"version\":\"4\",\"interface\":1,\"address\":\"10.195.80.30/24\",\"gateway\":\"10.195.80.1\"}

],\"dns\":{}}", err <nil>



3. Although it lands on a new node, in OVNK we update the old port (somehow the old port is not being removed) that is attached to the old switch:

[root@fedora ~]# ovn-nbctl list logical_switch_port  c345cc07-8a89-4e70-beff-d8d9f4dac46a

_uuid               : c345cc07-8a89-4e70-beff-d8d9f4dac46a

addresses           : ["0a:58:0a:c3:50:1e 10.195.80.30"]

dhcpv4_options      : []

dhcpv6_options      : []

dynamic_addresses   : []

enabled             : []

external_ids        : 

{namespace=mx-toni-dev, pod="true"}

ha_chassis_group    : []

name                : mx-toni-dev_toni-dev-build

options             : 

{iface-id-ver="655cc043-53d5-4550-9c0a-74095a0fbde4", requested-chassis=demchdc5vvx}

parent_name         : []

port_security       : ["0a:58:0a:c3:50:1e 10.195.80.30"]

tag                 : []

tag_request         : []

type                : ""

up                  : false




[root@fedora ~]# ovn-nbctl lsp-list demchdc6zax  | grep c345cc07-8a89-4e70-beff-d8d9f4dac46a

c345cc07-8a89-4e70-beff-d8d9f4dac46a (mx-toni-dev_toni-dev-build)

[root@fedora ~]# ovn-nbctl lsp-list demchdc5vvx  | grep c345cc07-8a89-4e70-beff-d8d9f4dac46a

[root@fedora ~]# 



This will cause the pod not to be able to send any traffic as its IP is in the wrong subnet for this switch.



4. Additionally the default node SNAT for this pod is in the right place:

[root@fedora ~]# ovn-nbctl lr-nat-list GR_demchdc6zax | grep 10.195.80.30

[root@fedora ~]# ovn-nbctl lr-nat-list GR_demchdc5vvx | grep 10.195.80.30

snat             139.25.144.25                       10.195.80.30



5. But there is no egress IP reroute or SNAT entry for this pod:

Egress IP:

status:

  items:


    
	
  • egressIP: 139.25.144.72
    
    node: demchdc5z6x
    
name: egress-mx-toni-dev
    • 





[root@fedora ~]# ovn-nbctl lr-nat-list GR_demchdc5z6x | grep 10.195.80.30

[root@fedora ~]# ovn-nbctl lr-nat-list GR_demchdc5z6x | grep 139.25.144.72

snat             139.25.144.72                       10.195.77.156

snat             139.25.144.72                       10.195.80.40

snat             139.25.144.72                       10.195.76.65

snat             139.25.144.72                       10.195.76.184

snat             139.25.144.72                       10.195.80.42

snat             139.25.144.72                       10.195.80.156



6. We see in the ovnkube-master logs that ovnk attempts to delete this pod, but it fails because we try to delete a logical switch port that is still bound to the wrong logical switch:

2022-08-10T09:07:32.033303027Z I0810 09:07:32.033270       1 client.go:781]  "msg"="transacting operations"  "database"="OVN_Northbound" "operations"="[{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.80.30]}}] Timeout:<nil> Where:[where column _uuid == 

{4fd75aab-222c-4f17-9407-3764f9b6760b}

] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[

{GoUUID:c345cc07-8a89-4e70-beff-d8d9f4dac46a}]}}] Timeout:<nil> Where:[where column _uuid == {f5073eaa-3f72-4ec2-94c3-3744d412864a}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {c345cc07-8a89-4e70-beff-d8d9f4dac46a}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}]"



2022-08-10T09:07:32.037857343Z I0810 09:07:32.037163       1 pods_retry.go:57] [655cc043-53d5-4550-9c0a-74095a0fbde4/mx-toni-dev/toni-dev-build] teardown retry failed; will try again later



From the engineering side we need to try to reproduce this. Flavio and I think this toni-dev-build must be a stateful set running to completion. We need to investigate further to understand how we ended up trying to add the new pod without first deleting the old. Could be a bug related to completed pods logic or retry logic that was added in 4.10.



— Additional comment from Anurag saxena on 2022-08-11 04:52:37 UTC —



Thanks @trozet@redhat.com for investigation. @huirwang@redhat.com @jechen@redhat.com Probably we can follow these steps to internally repro the issue.



— Additional comment from Andre Costa on 2022-08-11 15:09:25 UTC —



Hi all,



Many thanks for the great work here and customer is relieved that we found something  



In the meantime a new example that looks to be another caused by this bug. We have seen this live last week and today the customer saw it again on another project:



 - Similar build pod started by their operator:



1h21m       Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      deleteLogicalPort failed for pod mx-ngat-dev_ngat-dev-build: cannot delete logical switch port mx-ngat-dev_ngat-dev-build, error in transact with ops [{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.76.185]}}] Timeout:<nil> Where:[where column _uuid == {961a2f8f-14bb-44f9-a945-0dc2211324c7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[{GoUUID:d300fdb7-d337-4c64-8e31-7ff02889d9fb}]}}] Timeout:<nil> Where:[where column _uuid == {bc92e86b-a1ae-4771-8e59-0588657bf70e}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {d300fdb7-d337-4c64-8e31-7ff02889d9fb}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:1 Error: Details: UUID:

{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s) UUID:

{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s)

1h21m       Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      failed to ensurePod mx-ngat-dev/ngat-dev-build since it is not yet scheduled

Unknown     Normal    Scheduled                pod/ngat-dev-build                      Successfully assigned mx-ngat-dev/ngat-dev-build to demchdc5vux

1h21m       Normal    AddedInterface           pod/ngat-dev-build                      Add eth0 [10.195.81.116/24] from ovn-kubernetes

1h21m       Normal    Pulling                  pod/ngat-dev-build                      Pulling image "private-cloud.registry.mendix.com/image-builder:2.2.0"

1h21m       Normal    Pulled                   pod/ngat-dev-build                      Successfully pulled image "private-cloud.registry.mendix.com/image-builder:2.2.0" in 622.636779ms

1h21m       Normal    Created                  pod/ngat-dev-build                      Created container mendix-build

1h21m       Normal    Started                  pod/ngat-dev-build                      Started container mendix-build

1h20m       Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      deleteLogicalPort failed for pod mx-ngat-dev_ngat-dev-build: cannot delete logical switch port mx-ngat-dev_ngat-dev-build, error in transact with ops [{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.81.116]}}] Timeout:<nil> Where:[where column _uuid == {961a2f8f-14bb-44f9-a945-0dc2211324c7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[{GoUUID:d300fdb7-d337-4c64-8e31-7ff02889d9fb}]}}] Timeout:<nil> Where:[where column _uuid == {95072525-7d91-4f1f-9ad8-38a4171b978e}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {d300fdb7-d337-4c64-8e31-7ff02889d9fb}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:1 Error: Details: UUID:

{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s) UUID:

{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s)

1h2m        Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      failed to ensurePod mx-ngat-dev/ngat-dev-build since it is not yet scheduled

Unknown     Normal    Scheduled                pod/ngat-dev-build                      Successfully assigned mx-ngat-dev/ngat-dev-build to demchdc5vvx

1h2m        Normal    AddedInterface           pod/ngat-dev-build                      Add eth0 [10.195.80.33/24] from ovn-kubernetes

1h2m        Normal    Pulling                  pod/ngat-dev-build                      Pulling image "private-cloud.registry.mendix.com/image-builder:2.2.0"

1h2m        Normal    Pulled                   pod/ngat-dev-build                      Successfully pulled image "private-cloud.registry.mendix.com/image-builder:2.2.0" in 599.594895ms

1h2m        Normal    Created                  pod/ngat-dev-build                      Created container mendix-build

1h2m        Normal    Started                  pod/ngat-dev-build                      Started container mendix-build

Unknown     Normal    Scheduled                pod/ngat-dev-build                      Successfully assigned mx-ngat-dev/ngat-dev-build to demchdc6z2x

28m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      failed to ensurePod mx-ngat-dev/ngat-dev-build since it is not yet scheduled

27m         Normal    AddedInterface           pod/ngat-dev-build                      Add eth0 [10.195.76.114/24] from ovn-kubernetes

27m         Normal    Pulling                  pod/ngat-dev-build                      Pulling image "private-cloud.registry.mendix.com/image-builder:2.2.0"

27m         Normal    Pulled                   pod/ngat-dev-build                      Successfully pulled image "private-cloud.registry.mendix.com/image-builder:2.2.0" in 575.907145ms

27m         Normal    Created                  pod/ngat-dev-build                      Created container mendix-build

27m         Normal    Started                  pod/ngat-dev-build                      Started container mendix-build

26m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      deleteLogicalPort failed for pod mx-ngat-dev_ngat-dev-build: cannot delete logical switch port mx-ngat-dev_ngat-dev-build, error in transact with ops [{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.76.114]}}] Timeout:<nil> Where:[where column _uuid == {961a2f8f-14bb-44f9-a945-0dc2211324c7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[{GoUUID:d300fdb7-d337-4c64-8e31-7ff02889d9fb}]}}] Timeout:<nil> Where:[where column _uuid == {bc92e86b-a1ae-4771-8e59-0588657bf70e}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {d300fdb7-d337-4c64-8e31-7ff02889d9fb}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:1 Error: Details: UUID:

{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s) UUID:

{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s)

26m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      failed to ensurePod mx-ngat-dev/ngat-dev-build since it is not yet scheduled

Unknown     Normal    Scheduled                pod/ngat-dev-build                      Successfully assigned mx-ngat-dev/ngat-dev-build to demchdc6z2x

26m         Normal    AddedInterface           pod/ngat-dev-build                      Add eth0 [10.195.76.114/24] from ovn-kubernetes

26m         Normal    Pulling                  pod/ngat-dev-build                      Pulling image "private-cloud.registry.mendix.com/image-builder:2.2.0"

26m         Normal    Pulled                   pod/ngat-dev-build                      Successfully pulled image "private-cloud.registry.mendix.com/image-builder:2.2.0" in 606.11715ms

26m         Normal    Created                  pod/ngat-dev-build                      Created container mendix-build

26m         Normal    Started                  pod/ngat-dev-build                      Started container mendix-build

26m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      deleteLogicalPort failed for pod mx-ngat-dev_ngat-dev-build: cannot delete logical switch port mx-ngat-dev_ngat-dev-build, error in transact with ops [{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.76.114]}}] Timeout:<nil> Where:[where column _uuid == {961a2f8f-14bb-44f9-a945-0dc2211324c7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[{GoUUID:d300fdb7-d337-4c64-8e31-7ff02889d9fb}]}}] Timeout:<nil> Where:[where column _uuid == {bc92e86b-a1ae-4771-8e59-0588657bf70e}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {d300fdb7-d337-4c64-8e31-7ff02889d9fb}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:1 Error: Details: UUID:

{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s) UUID:

{GoUUID:} Rows:[]}] and errors []: referential integrity violation: cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s)

22m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      failed to ensurePod mx-ngat-dev/ngat-dev-build since it is not yet scheduled

Unknown     Normal    Scheduled                pod/ngat-dev-build                      Successfully assigned mx-ngat-dev/ngat-dev-build to demchdc6z2x

22m         Normal    AddedInterface           pod/ngat-dev-build                      Add eth0 [10.195.76.114/24] from ovn-kubernetes

22m         Normal    Pulling                  pod/ngat-dev-build                      Pulling image "private-cloud.registry.mendix.com/image-builder:2.2.0"

22m         Normal    Pulled                   pod/ngat-dev-build                      Successfully pulled image "private-cloud.registry.mendix.com/image-builder:2.2.0" in 534.430683ms

22m         Normal    Created                  pod/ngat-dev-build                      Created container mendix-build

22m         Normal    Started                  pod/ngat-dev-build                      Started container mendix-build

21m         Warning   ErrorAddingLogicalPort   pod/ngat-dev-build                      deleteLogicalPort failed for pod mx-ngat-dev_ngat-dev-build: cannot delete logical switch port mx-ngat-dev_ngat-dev-build, error in transact with ops [{Op:mutate Table:Address_Set Row:map[] Rows:[] Columns:[] Mutations:[{Column:addresses Mutator:delete Value:{GoSet:[10.195.76.114]}}] Timeout:<nil> Where:[where column _uuid == {961a2f8f-14bb-44f9-a945-0dc2211324c7}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:mutate Table:Logical_Switch Row:map[] Rows:[] Columns:[] Mutations:[{Column:ports Mutator:delete Value:{GoSet:[{GoUUID:d300fdb7-d337-4c64-8e31-7ff02889d9fb}]}}] Timeout:<nil> Where:[where column _uuid == {bc92e86b-a1ae-4771-8e59-0588657bf70e}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:} {Op:delete Table:Logical_Switch_Port Row:map[] Rows:[] Columns:[] Mutations:[] Timeout:<nil> Where:[where column _uuid == {d300fdb7-d337-4c64-8e31-7ff02889d9fb}] Until: Durable:<nil> Comment:<nil> Lock:<nil> UUIDName:}] results [{Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:1 Error: Details: UUID:

{GoUUID:} Rows:[]} {Count:1 Error: Details: UUID:{GoUUID}

 Rows:[]} {Count:0 Error:referential integrity violation Details:cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s) UUID:

{GoUUID:}

 Rows:[]}] and errors []: referential integrity violation: cannot delete Logical_Switch_Port row d300fdb7-d337-4c64-8e31-7ff02889d9fb because of 1 remaining reference(s)



    
	
  • Eventually the pod seems to start but then fails again with the same TCP timeout:
    • 





2022-08-11T08:26:24.264204783Z time="2022-08-11T08:26:24Z" level=error msg="Failed to build mendix app, failed to create application layer failed to download MDA from https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/ef924516-7121-4372-a306-b0055c445766/71268, Get \"https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/ef924516-7121-4372-a306-b0055c445766/71268\": proxyconnect tcp: dial tcp: i/o timeout: Get \"https://privatecloud.mendixcloud.com/rest/mdarepository/v1/download/ef924516-7121-4372-a306-b0055c445766/71268\": proxyconnect tcp: dial tcp: i/o timeout"




    
	
  • Customer took a sosreport from the node where the pod terminated in case is worth it to check:
    • 





https://attachments.access.redhat.com/hydra/rest/cases/03096770/attachments/bd3f86b2-6d2a-44f7-bb0f-eec85630586d?usePresignedUrl=true



— Additional comment from Tim Rozet on 2022-08-11 21:19:12 UTC —



We were able to reproduce the issue locally. There are two potential paths that can cause a stale logical switch port to be re-used on the wrong node:

Scenario 1: pod is created, deleted, and recreated on another node extremely quickly. This plays out like this:

Events:

1. pod toni is created on node A

2. pod toni is deleted on node A

3. pod toni is recreated on node B



What happens in ovnk:

1. pod toni is created on node B

2. pod toni is deleted on node A (fails)



This happens because we grab the latest version of the pod to add in event 1, which by the time we grab it is actually the value from event 3. Then after processing event 1, we move to event 2 and when we delete the pod, we use what was given to us in the event, which is now incorrect to remove the pod from node A, because it actually is on node B. The fix is to ignore the value in the pod spec on deletion, and use what we store internally in our internal port cache. If there is no entry in the cache, we search OVN NBDB to find the right switch (this is an expensive operation so we want to avoid it when possible).



Flavio is working on a fix for this.



Scenario 2: pod is created, runs to completion, is deleted very quickly, and then is recreated on another node:

Events:

1. pod toni is created on node A

2. pod toni runs to completion

3. pod toni immediately is deleted

4. pod toni is recreated on node B



What happens in ovnk:

1. pod toni is created on node A

2. completion causes an update event, ovnk does not delete the pod (bug)

3. deletion event is processed, but since it is a completed pod, we ignore it (since we should have already delted it in step 2)

4. pod toni is recreated on node B, ovnk sees there is already an switch port for toni on the wrong switch, and just updates that with the new information



This happens when a pod goes to completed, but is deleted very quickly. In step 2 during update event we try to grab the latest version of the pod, but it doesn't exist anymore since it was deleted. In this case we skip the update, instead of tearing down the pod. The delete code in step 3 assumes that if the pod is completed we must have already handled it in update, so the stale port stays around until an add later re-uses it. Patryk already has a fix for this:



https://github.com/ovn-org/ovn-kubernetes/pull/3071



There may still be more egress IP issues to investigate (tracked in other bugs) after fixing this, and we will look into those after fixing these fundamental pod issues.



— Additional comment from Tim Rozet on 2022-08-11 22:19:30 UTC —



Andre, re: comment 5. This looks like the same issue. If you have a must gather we can confirm. I think the sosreport from the worker node is not enough. The referential integrity violation occurs because we attempt to do these operations:

1. remove the logical switch port from the logical switch

2. delete the logical switch port



In this case:

1. we remove the logical switch port from the logical switch, but the switch port actually exists on a different switch (node) - this is a noop

2. we try to delete the logical switch port- NBDB complains this is an violation and refuses to delete it, because there is another switch that holds a reference to this object



— Additional comment from Anurag saxena on 2022-08-12 21:16:56 UTC —



@rbrattai@redhat.com Can you help verifying this and backports? feel free to re-assign to someone else if needed while I am on PTO. Thanks!


            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1815:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2070:
            
             Missing  $SEARCH domain in /etc/resolve.conf for OCP v4.9.31 cluster 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-1099. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-224. The following is the description of the original issue: 

—

Description of problem:

OCP v4.9.31 cluster didn't have the $search domain in /etc/resolv.conf, which was there in the v4.8.29 OCP cluster. This was observed in all the nodes of the v4.9.31 cluster.

~~~

OpenShift 4.9.31

sh-4.4# cat /etc/resolv.conf 


    
	
  1. Generated by KNI resolv prepender NM dispatcher script
    
nameserver 172.xx.xx.xx
    
nameserver 10.xx.xx.xx
    
nameserver 10.xx.xx.xx
    2. 
	
  2. nameserver 10.xx.xx.xx
    3. 





OpenShift 4.8.29


    
	
  1. Generated by KNI resolv prepender NM dispatcher script
    
search   sepia.lab.iad2.dc.paas.redhat.com
    
nameserver 172.xx.xx.xx
    
nameserver 10.xx.xx.xx
    
nameserver 10.xx.xx.xx
    2. 
	
  2. nameserver 10.xx.xx.xx
    
~~~
    3. 





ENV: OpenStack IAD2, IPI installation. Connected cluster.



Version-Release number of selected component (if applicable):

OCP v4.9.31



How reproducible:

Always



Steps to Reproduce:

1. Install IPI cluster on OpenStack IAD2 platform having cluster version 4.9.31

2. Debug to any of the node(master/worker)

3. Check and confirm the missing search domain on all nodes of the cluster.



Actual results:

The search domain was missing when checked in `/etc/resolv.conf` file on all nodes of the cluster causing serious issues in the cluster.



Expected results:

The installer should embed the search domain in /etc/resolv.conf file on all nodes of the cluster.



Additional info:


    
	
  • Cu was trying to deploy secure Kerberos on the CoreOS nodes and it failed when the IPA-client install command failed. This is when the customer noticed this unusual behavior. They did not manually update the resolv.conf file to include the $search domain. They instead added the script below to /etc/NetworkManager/dispatcher.d/ and restarted NetworkManager on the node to fix this issue and installation was successful.
    
~~~
    
#!/bin/bash
    • 





set -eo pipefail



DISPATCHER_FILE="/etc/NetworkManager/dispatcher.d/30-resolv-prepender"

DOMAINS="$(grep -E '\s*DOMAINS=.*iad2.dc.paas.redhat.com' $DISPATCHER_FILE \







 grep -oE '[a-z0-9]*.dev.iad2.dc.paas.redhat.com' \




 tr '\n' ' ')"









>&2 echo "IT-PaaS: overwriting search domains in /etc/resolv.conf with: $DOMAINS"



sed -e "/^search/d" \

  -e "/Generated by/c# Generated by KNI resolv prepender NM dispatcher script  \nsearch $DOMAINS" \

  /etc/resolv.conf > /etc/resolv.tmp



mv /etc/resolv.tmp /etc/resolv.conf

~~~



    
	
  • Cu confirms that the $search domain was missing since the cluster was freshly installed/ They even confirmed this with a fresh new cluster as well that it was missing.
    • 





    
	
  • The fresh cluster was initially installed at v4.9.31 but was updated afterward to v4.9.43 (the latest z-stream) to see if the updates fixed anything but it didn't make any difference. The cluster is currently running v4.9.43 and shows the $search domain missing in the /etc/resolv.conf file on all nodes.
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-403:
            
             Local Update Server link 404
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of OCPBUGSM-46761 +++



Description of problem:

When using the admin console, under "Cluster Settings" and choosing Upstream Configuration, the "window" that appears has a dead link to documentation for how to create a local (disconnected) update server. 



The link in the window is

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10/html/updating_clusters/installing-update-service



Assume the right link should be something like:

https://docs.openshift.com/container-platform/4.10/updating/updating-restricted-network-cluster.html#update-restricted-network-cluster-update-service



Version-Release number of selected component (if applicable):



4.10.*



— Additional comment from plarsen@redhat.com on 2022-06-17 16:33:53 UTC —



Created attachment 1890947

Screenshot showing the "window" with the 404 link



— Additional comment from rhamilto@redhat.com on 2022-06-28 19:56:17 UTC —



Thank you, Peter, for wonderfully documenting the bug!

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-784:
            
              [4.10] Remove `yq` curls from CI steps
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




+++ This bug was initially created as a clone of 

Bug #2118514

Various CI steps use the upi-installer container for it's access to the
aws cli tools among other things. However, most of those steps also
curl yq directly from GitHub. We can save ourselves some headaches
when GitHub is down by just embedding the binary in the image already.

Whenever GitHub has issues or throttles us, YQ hash mismatch error out. The hash mismatch error is because github is probably returning an error page, although our scripts hide it.





Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1266:
            
             Backport: https://github.com/openshift/kubernetes/pull/1295
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of Bug #2117423 +++



Description of problem:



Backport https://github.com/openshift/kubernetes/pull/1295 to 4.10




Version-Release number of selected component (if applicable):




How reproducible:




Steps to Reproduce:

1.

2.

3.



Actual results:




Expected results:




Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2884:
            
             [release-4.10] Make ccoctl set sts endpoints to regional in AWS credentials secrets
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug card represents work done in https://issues.redhat.com/browse/CCO-257 to set STS endpoints to regional in AWS credentials secrets and is created to facilitate backporting the change to previous releases as required by the backport process [1].



[1] https://docs.google.com/document/d/1qbKUl4K9PPW4tCbdDgWSXd5klosX77ifkxQLqzjztGE/edit#heading=h.mml0rz88o0nj



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3165:
            
             [RFE] Add new Azure instance types to the official "tested/supported" list
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
We have created a fix in 4.12 that fetches instance type information from Azure API instead of updating the lists. We feel that backporting that fix is too risky, but agreed to update the list in older versions.



Description of problem:



Add the following instance types to azure_instance_types list[1]:


    
	
  • Standard_D8s_v5
    • 
	
  • Standard_E8s_v5
    • 
	
  • Standard_E16s_v5
    • 





Version-Release number of selected component (if applicable):

OCP 4.8



Steps to Reproduce:

1. Migrate worker/infra nodes to above mentioned (missing) v5 instance types

2. "Failed to set autoscaling from zero annotations, instance type unknown"



Actual results:


    
	
  • "Failed to set autoscaling from zero annotations, instance type unknown"
    • 
	
  • New v5 instance types not officially tested/supported
    • 





Expected results:

The new instance types are available in the azure_instance_types list[1] and no errors/warnings are observed after migrating:


    
	
  • Standard_D8s_v5
    • 
	
  • Standard_E8s_v5
    • 
	
  • Standard_E16s_v5
    • 





Additional info:



The related v4 instance types are already available[1] - I suspect adding the mentioned v5 instance types is a minor update:


    
	
  • Standard_D8s_v4
    • 
	
  • Standard_E8s_v4
    • 
	
  • Standard_E16s_v4
    • 





1) azure_instance_types.go

https://github.com/openshift/cluster-api-provider-azure/blob/release-4.8/pkg/cloud/azure/actuators/machineset/azure_instance_types.go

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4864:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1788:
            
             oc adm inspect --rotated-pod-logs not working properly for static pods
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-613. The following is the description of the original issue: 

—


Description of problem:



The path used by --rotated-pod-logs to gather the rotated pod logs from /var/log/pods node folder via /api/v1/nodes/${NODE}/proxy/logs/${LOG_PATH} is only valid for regular pods but not for static pods.



The main problem is that, while normal pods have their rotated logs at this /var/log/pods/${POD_NAME}_${POD_UID_IN_API}/${CONTAINER_NAME}, static pods have them at /var/log/pods/${POD_NAME}_${CONFIG_HASH}/${CONTAINER_NAME} because the UID cannot be known at the time that the static pod is born (because static pods are created by kubelet before registering them in the kube-apiserver, and UID is assigned by the kube-apiserver).



The visible results of that are:


    
	
  • Spurious errors of not found resources related to the pods.
    • 
	
  • Rotated pod logs are not gathered even if present.
    • 





Version-Release number of selected component (if applicable):



4.10



How reproducible:



Always if there are static pods.



Steps to Reproduce:



1. oc adm inspect --rotated-pod-logs ns/openshift-etcd (or any other project with static pods).



Actual results:



    
	
  • Rotated pods not gathered.
    • 
	
  • Errors like these

    

    error: errors occurred while gathering data:
    one or more errors occurred while gathering pod-specific data for namespace: openshift-etcd

    [one or more errors occurred while gathering container data for pod etcd-master-0.example.net:

    the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-1.example.net:

    the server could not find the requested resource, one or more errors occurred while gathering container data for pod etcd-master-2.example.net:

    the server could not find the requested resource]

    

    • 





Expected results:



No errors like the ones above and rotated pod logs to be gathered, if present.



Additional info:



Despite being marked as experimental, this --rotated-pod-logs is used in must-gather, so this issue can be easily reproduced by just running a default must-gather. I focused on bare oc adm inspect reproducers for simplicity.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2107:
            
             unable to install IPI PRIVATE OpenShift cluster in Azure due to organization policies
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2056519](https://bugzilla.redhat.com/show_bug.cgi?id=2056519). The following is the description of the original bug:

—

Version: 4.9



Platform:

azure



Please specify:


    
	
  • IPI
    • 





What happened?

Issue: Customer reports unable to install IPI PRIVATE OpenShift cluster in Azure. They have orgnization policy which do not allow them to create storage account with public access. It should be disallowed. 



What did you expect to happen?

Installer completes successfully.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2894:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2940:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-400:
            
             Disable xmlstarlet code to fix current CVEs [release-4.10]
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
+++ This bug was initially created as a clone of Bug #2117811 +++



Description of problem:

We are currently unable to merge any pull requests to fix CVEs because of the use of the xmlstarlet command line utility which is not currently packaged for and available for RHEL.



Version-Release number of selected component (if applicable):




How reproducible:

ALways



Steps to Reproduce:

1.

2.

3.



Actual results:




Expected results:




Additional info:

Another bug will be opened to reenable this code once the xmlstarlet package is available in RHEL or we find an alternative fix.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2602:
            
             Intermittent DNS Resolution Errors After Upgrading to 4.10.25
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
When using cluster scaling and querying a URL in a pod on the side. All the while running some custom watches on endpoints and nodes.



When the nodes scale down, it seems a few seconds before an event marks the node as Not Ready and before the dns-default endpoint is removed from the endpoints list a DNS query can fail.



We wrote some simply watcher (see below for details) to log this and got the following events:



DNS lookup failure:






Tue Oct 18 12:33:23 UTC 2022 - Lookup success                                                                                                                                                                                                                                                                                                      
Tue Oct 18 12:33:28 UTC 2022 - DNS failure                                                                                                                                                                                                                                                                                                         
Tue Oct 18 12:33:41 UTC 2022 - Lookup success 






The node was not yet Not Ready and the endpoint was still in the list of endpoints at that time (ntrdy indicates a NotReadyEndpoint):






2022-10-18 12:33:21.712180649 +0000 UTC m=+1047.610174444 - ip-10-0-137-100.ec2.internal -  MemoryPressure - False, DiskPressure - False, PIDPressure - False, Ready - True,
2022-10-18 12:33:39.11806612 +0000 UTC m=+1065.016059955 - ip-10-0-129-193.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:39.525574893 +0000 UTC m=+1065.423568712 - dns-default rdy: 10.128.0.2 rdy: 10.128.10.4 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.130.8.4 rdy: 10.131.0.3 ntrdy: 10.131.8.4
2022-10-18 12:33:39.526424974 +0000 UTC m=+1065.424418833 - dns-default rdy: 10.128.0.2 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.130.8.4 rdy: 10.131.0.3 ntrdy: 10.128.10.4 ntrdy: 10.131.8.4
2022-10-18 12:33:39.528532869 +0000 UTC m=+1065.426526744 - ip-10-0-129-193.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:39.729859144 +0000 UTC m=+1065.627852917 - ip-10-0-150-205.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:39.936928994 +0000 UTC m=+1065.834922825 - ip-10-0-150-205.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:44.749587947 +0000 UTC m=+1070.647581767 - ip-10-0-188-175.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:44.952196646 +0000 UTC m=+1070.850190469 - dns-default rdy: 10.128.0.2 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.131.0.3 ntrdy: 10.128.10.4 ntrdy: 10.130.8.4 ntrdy: 10.131.8.4
2022-10-18 12:33:44.954865089 +0000 UTC m=+1070.852858965 - ip-10-0-188-175.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:45.159460169 +0000 UTC m=+1071.057454007 - ip-10-0-150-205.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:48.641412229 +0000 UTC m=+1074.539406059 - ip-10-0-188-175.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:48.846438064 +0000 UTC m=+1074.744431900 - ip-10-0-129-193.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:33:54.068542745 +0000 UTC m=+1079.966536563 - ip-10-0-150-205.ec2.internal -  MemoryPressure - Unknown, DiskPressure - Unknown, PIDPressure - Unknown, Ready - Unknown,
2022-10-18 12:34:31.752294563 +0000 UTC m=+1117.650288381 - ip-10-0-253-198.ec2.internal -  MemoryPressure - False, DiskPressure - False, PIDPressure - False, Ready - True,
2022-10-18 12:34:39.531848219 +0000 UTC m=+1125.429842032 - dns-default rdy: 10.128.0.2 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.131.0.3 ntrdy: 10.128.10.4 ntrdy: 10.131.8.4
2022-10-18 12:34:39.736866622 +0000 UTC m=+1125.634860439 - dns-default rdy: 10.128.0.2 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.131.0.3 ntrdy: 10.128.10.4
2022-10-18 12:34:39.941934912 +0000 UTC m=+1125.839928742 - dns-default rdy: 10.128.0.2 rdy: 10.128.2.5 rdy: 10.129.0.2 rdy: 10.130.0.16 rdy: 10.131.0.3






So we can observe that the node goes into 'Unknown' at 12:33:39, and the endpoint goes into Not Ready soon after.

Not sure if this is a logic problem of draining a node or an issue with the autoscaler at this point in time, but it fixes itself at the next lookup 5 seconds later.



—



Detailed breakdown of how this was reproduced:

1. A cluster with autoscaling enabled is required.

2. Deploy a daemonset that attempt to use DNS / HTTP in a loop, e.g. the following Daemonset was used to test this:






apiVersion: apps/v1
kind: DaemonSet
metadata: 
  name: dns-tester
  labels: 
    app: dns-tester
spec: 
  selector: 
    matchLabels: 
      app: dns-tester
  template: 
    metadata: 
      labels: 
        app: dns-tester
    spec: 
      containers: 
      - name: dns-tester
        image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72f2f7e906c321da6d6a00ce610780e8766e8432f7c553c5d03492f65fe5416c
        command: ["/bin/sh", "-c"]
        args: ['while true; do CURL=$(curl redhat.com 2>&1); if [[ "$CURL" == *"not resolve"* ]]; then echo `date` - "DNS failure"; else echo `date` - "Lookup success"; fi;  sleep 5; done']
        resources: 
          limits: 
            cpu: 100m
            memory: 200Mi 






3. Run the following go program against the same cluster (this is what watches the node and endpoint events for the dns-default endpoints):






package main

import (
	"context"
	"fmt"
	"path/filepath"
	"sync"
	"time"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	_ "k8s.io/client-go/plugin/pkg/client/auth"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
)

const dnsNamespace = "openshift-dns"
const dnsEndpoint = "dns-default"

func nodeWatch(clientset *kubernetes.Clientset, waitGroup sync.WaitGroup) {
	ctx := context.Background()
	defer waitGroup.Done()

	var nodes = clientset.CoreV1().Nodes()
	watcher, err := nodes.Watch(ctx, metav1.ListOptions{})
	if err != nil {
		panic(err.Error())
	}
	ch := watcher.ResultChan()

	for {
		event := <-ch
		node, ok := event.Object.(*corev1.Node)
		if !ok {
			fmt.Printf("%v", event)
			panic("Could not cast to nodes")
		}
		fmt.Printf("%v - %s - ", time.Now(), node.Name)
		for _, condition := range node.Status.Conditions {
			fmt.Printf(" %v - %v,", condition.Type, condition.Status)
		}
		fmt.Println()
	}
}

func dnsWatch(clientset *kubernetes.Clientset, waitGroup sync.WaitGroup) {
	ctx := context.Background()
	defer waitGroup.Done()
	var api = clientset.CoreV1().Endpoints(dnsNamespace)
	watcher, err := api.Watch(ctx, metav1.ListOptions{})
	if err != nil {
		panic(err.Error())
	}
	ch := watcher.ResultChan()
	for {
		event := <-ch
		endpoints, ok := event.Object.(*corev1.Endpoints)
		if !ok {
			fmt.Printf("%v", event)
			panic("Could not cast to Endpoint")
		}
		fmt.Printf("%v - %v", time.Now(), endpoints.ObjectMeta.Name)
		for _, endpoint := range endpoints.Subsets {
			for _, address := range endpoint.Addresses {
				fmt.Printf(" rdy: %v", address.IP)
			}
			for _, address := range endpoint.NotReadyAddresses {
				fmt.Printf(" ntrdy: %v", address.IP)
			}
		}
		fmt.Println()
	}
}

func main() {
	// AUTHENTICATE
	var home = homedir.HomeDir()
	var kubeconfig = filepath.Join(home, ".kube", "config")
	config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
	if err != nil {
		panic(err.Error())
	}
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		panic(err.Error())
	}
	wg := sync.WaitGroup{}
	wg.Add(2)
	go dnsWatch(clientset, wg)
	go nodeWatch(clientset, wg)
	wg.Wait()
}






4. Create simulated pressure on the nodes to force a scaleup - e.g. use the following deployment:






apiVersion: apps/v1
kind: Deployment
metadata: 
  name: resource-eater
spec: 
  replicas: 4
  selector: 
    matchLabels: 
      app: resource-eater
  template: 
    metadata: 
      labels: 
        app: resource-eater
    spec: 
      containers: 
      - name: resource-eater
        image: busybox:latest
        command: ["/bin/sh", "-c"]
        args: ["sleep 3600"]
        resources: 
          requests: 
            memory: "8Gi"
            cpu: "1000m"
      affinity: 
        podAntiAffinity: 
          requiredDuringSchedulingIgnoredDuringExecution: 
          - labelSelector: 
              matchExpressions: 
              - key: app
                operator: In
                values: 
                - store
            topologyKey: "kubernetes.io/hostname" 






5. Wait for the scale up to happen.

6. Delete the deployment that created the node pressure, so the scale down can happen (this can easily take 15 minutes).

7. Observe the events in the watcher program and the logs for the daemonset - this should show the same behavior as detailed above.




we found a few logged bugs that seemed related to this issue affecting clusters on 4.8 through 4.10.  Those bugs are as follows:






https://issues.redhat.com/browse/OCPBUGS-647
https://issues.redhat.com/browse/OCPBUGS-488
https://bugzilla.redhat.com/show_bug.cgi?id=2061244







Using the boave mentioned steps, we have been able to reliably reproduce the issue of DNS failures during autoscale-down in 4.10 clusters.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4598:
            
             [4.10z] Spike in pod-latency graph observed due to ovnkube-master restarts
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



When running node-density (245 pods/node) on a 120 node cluster, we see that there is a huge spike (~22s) in Avg pod-latency. When the spike occurs we see all the ovnkube-master pods go through a restart. 



The restart happens because of (ovnkube-master pods)




2022-08-10T04:04:44.494945179Z panic: reflect: call of reflect.Value.Len on ptr Value




Version-Release number of selected component (if applicable):



4.12.0-0.nightly-2022-08-09-114621



How reproducible:



Steps to Reproduce:

1. Run node-density on a 120 node cluster



Actual results:



Spike observed in pod-latency graph ~22s



Expected results:



Steady pod-latency graph ~4s



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2795:
            
             Fail to deprovision cluster on 4.10 and below when swift speaks HTTP2
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



We're seeing CI fail to deprovision the cluster on OpenStack.





Deprovisioning cluster ...
/tmp/secret/metadata.json
Copying the installation artifacts to the Installer's asset directory...
Running the Installer's 'destroy cluster' command...
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
level=error msg=Cannot extract names from response with content-type: []
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:164","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 1h0m0s timeout","severity":"error","time":"2022-10-23T16:40:20Z"}
Copying the Installer logs to the artifacts directory...
{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:254","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Process did not exit before 10m0s grace period","severity":"error","time":"2022-10-23T16:50:20Z"}
{"component":"entrypoint","error":"os: process already finished","file":"k8s.io/test-infra/prow/entrypoint/run.go:256","func":"k8s.io/test-infra/prow/entrypoint.gracefullyTerminate","level":"error","msg":"Could not kill process after grace period","severity":"error","time":"2022-10-23T16:50:20Z"}
 





A quick search shows this only affects OpenStack clusters, for OCP versions <= 4.10:

https://search.ci.openshift.org/?search=Cannot+extract+names+from+response+with+content-type&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job



Version-Release number of selected component (if applicable):




4.7 to 4.10





How reproducible:




Always? To be confirmed. It seems to have started happening sometime in Aug.
 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5788:
            
             ClusterResourceQuota values are not reflecting.
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-723. The following is the description of the original issue: 

—

Description of problem:

I have a customer who created clusterquota for one of the namespace, it got created but the values were not reflecting under limits or not displaying namespace details.

~~~

$ oc describe AppliedClusterResourceQuota

Name:           test-clusterquota

Created:        19 minutes ago

Labels:         size=custom

Annotations:    <none>

Namespace Selector: []

Label Selector:

AnnotationSelector: map[openshift.io/requester:system:serviceaccount:application-service-accounts:test-sa]

Scopes:         NotTerminating

Resource        Used    Hard

--------        ----    ----

~~~



WORKAROUND: They recreated the clusterquota object (cache it off, delete it, create new) after which it displayed values as expected.



In the past, they saw similar behavior on their test cluster, there it was heavily utilized the etcd DB was much larger in size (>2.5Gi), and had many more objects (at that time, helm secrets were being cached for all deployments, and keeping a history of 10, so etcd was being bombarded).



This cluster the same "symptom" was noticed however etcd was nowhere near that in size nor the amount of etcd objects and/or helm cached secrets.



Version-Release number of selected component (if applicable): OCP 4.9



How reproducible: Occurred only twice(once in test and in current cluster)



Steps to Reproduce:

1. Create ClusterQuota

2. Check AppliedClusterResourceQuota

3. The values and namespace is empty



Actual results: ClusterQuota should display the values



Expected results: ClusterQuota not displaying values

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5425:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2743:
            
             Fatal error while configuring sts endpoint in install-config
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Install failed if specify sts endpoint in the install-config.yaml file:






platform:
  aws:
    region: us-east-2
    serviceEndpoints:
    - name: sts
      url: https://sts.us-east-2.amazonaws.com





Errors:




level=error msg=Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: SignatureDoesNotMatch: Credential should be scoped to a valid region. 
level=error msg=  status code: 403, request id: f4e877fe-9e90-4cba-a455-2538d489a8d0
level=error
level=error msg=  on ../../../../../../../../tmp/openshift-install-cluster-1552617948/main.tf line 11, in provider "aws":
level=error msg=  11: provider "aws" {
level=error
level=error
level=error msg=Failed to read tfstate: open /tmp/openshift-install-cluster-1552617948/terraform.cluster.tfstate: no such file or directory





Version-Release number of selected component (if applicable):




4.10.z




How reproducible:




* Always




Steps to Reproduce:




1. Create an install-config.yaml, and specify sts endpoint 
2. Create an STS cluster 




Actual results:




Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: SignatureDoesNotMatch: Credential should be scoped to a valid region. 




Expected results:




No errors, create cluster successfully




Additional info:




No such issues on 4.8, 4.9, 4.11, 4.12



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5929:
            
             [4.10] nodeport not reachable port connection timeout
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



NodePort port not accessible 



Version-Release number of selected component (if applicable):



OCP 4.8.20




How reproducible:



 $oc -n ui-nprd get services -o wide

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                                           PORT(S)          AGE     SELECTOR

docker-registry      ClusterIP      10.201.219.240   <none>                                                                                5000/TCP         24d     app=registry

docker-registry-lb   LoadBalancer   10.201.252.253   internal-xxxxxx.xx-xxxx-1.elb.amazonaws.com   5000:30779/TCP   3d22h   app=registry

docker-registry-np   NodePort       10.201.216.26    <none>                                                                                5000:32428/TCP   3d16h   app=registry



$oc debug node/ip-xxx.ca-central-1.compute.internal

Starting pod/ip-xxx.ca-central-1computeinternal-debug ...

To use host binaries, run `chroot /host`

Pod IP: 10.81.23.96

If you don't see a command prompt, try pressing enter.

sh-4.2# chroot /host

sh-4.4# nc -vz 10.81.23.96 32428

Ncat: Version 7.70 ( https://nmap.org/ncat )

Ncat: Connection timed out.




In a new-created namespaces the same deployment works:



[RHEL7:> oc project

Using project "test-c1" on server "https://api.xx.xx.xxxx.xx.xx:6443".

[RHEL7:- ~/tmp]> oc port-forward service/docker-registry-np 5000:5000

Forwarding from 127.0.0.1:5000 -> 5000



[1]+  Stopped                 oc4 port-forward service/docker-registry-np 5000:5000

[RHEL7:  ~/tmp]> bg %1

[1]+ oc4 port-forward service/docker-registry-np 5000:5000 &

[RHEL7:  ~/tmp]> nc -v localhost 5000

Ncat: Version 7.50 ( https://nmap.org/ncat )

Ncat: Connected to 127.0.0.1:5000.

Handling connection for 5000



[RHEL7:  ~/tmp]> kill %1

[RHEL7: ~/tmp]>

[1]+  Terminated              oc4 port-forward service/docker-registry-np 5000:5000

[RHEL7:  ~/tmp]> oc get services

NAME                 TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE

docker-registry-np   NodePort   10.201.224.174   <none>        5000:31793/TCP   68s



[RHEL7:  ~/tmp]> oc get pods -o wide

NAME                        READY   STATUS    RESTARTS   AGE    IP            NODE                                           NOMINATED NODE   READINESS GATES

registry-75b7c7fd94-rx29j   1/1     Running   0          7m5s   10.201.1.29   ip-xxx.ca-central-1.compute.internal   <none>           <none>

[RHEL7:  ~/tmp]> oc debug node/ip-xxx.ca-central-1.compute.internal

Starting pod/ip-xxxca-central-1computeinternal-debug ...

To use host binaries, run `chroot /host`

Pod IP: 10.81.23.87

If you don't see a command prompt, try pressing enter.

sh-4.2# chroot /host

sh-4.4# nc -v 10.81.23.87 31793

Ncat: Version 7.70 ( https://nmap.org/ncat )

Ncat: Connected to 10.81.23.87:31793.



Actual results:



    
	
  • Working on new created namespace
    • 
	
  • Not working on already created namespace
    • 





Expected results:



    
	
  • Suppose to work on all namespaces.
    • 





Additional info:



    
	
  • This cluster get upgrade from 4.7.x to 4.8 and then they manually enable OVN.
    • 
	
  • The issue was happening on all namespaces but after restarting the ovnkube-master-xxxx pods only the newly created namespaces work.
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2943:
            
             Fix sshd embedded version
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-2250. The following is the description of the original issue: 

—

This is a clone of issue OCPBUGS-2099. The following is the description of the original issue: 

—

Description of problem:




 




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3697:
            
             [release-4.10] Update OWNERS file to reflect current team members
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
The OWNERS file for multiple branches in the openshift/jenkins repository need to be updated to reflect current team members for approvals.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-376:
            
             storageclass should not be created for unsupported vsphere version
          

          
          
          

            
            
            
            
            
            
            
          

          
          
          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5415:
            
             [OCP 4.10] ironic container images have old packages
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
we need to make sure that the ironic containers use the latest available bugfix versions

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-579:
            
             Setting a telemeter proxy in the cluster-monitoring-config config map does not work as expected
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-516. The following is the description of the original issue: 

—

Description of problem:



Setting a telemeter proxy in the cluster-monitoring-config config map does not work as expected



Version-Release number of selected component (if applicable):




How reproducible:




Steps to Reproduce:

the following KCS details steps to add a proxy. 

The steps have been verified at 4.7 but do not work at 4.8, 4.9 or 4.10



https://access.redhat.com/solutions/6172402



When testing at 4.8, 4.9 and 4.10 the proxy setting where also nested under `telemeterClient` 



which triggered a telemeter restart but the proxy setting do not get set in the deployment as they do in 4.7  



Actual results:



4.8, 4.9 and 4.10 without the nested `telemeterClient` 

does not trigger a restart of the telemeter pod 



Expected results:



I think the proxy setting should be nested under telemeterClient

but should set the environment variables in the deployment 



Additional info:



This is a backport of https://bugzilla.redhat.com/show_bug.cgi?id=2116382 from 4.12 to 4.11.z. Creating manually because as seen in https://github.com/openshift/cluster-monitoring-operator/pull/1743 `/cherry-pick` doesn't work for bugs originally created in bugzilla

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-691:
            
             [2112237] [ Cluster storage Operator 4.x(10/11) ] DefaultStorageClassController report fake message "No default StorageClass for this platform" on Alicloud, IBM, Nutanix
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Backport to 4.10.z

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-713:
            
             Add user coreydaley to list of approvers for openshift/jenkins
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-709. The following is the description of the original issue: 

—

Description of problem:








Version-Release number of selected component (if applicable):








How reproducible:








Steps to Reproduce:




1.
2.
3.





Actual results:








Expected results:








Additional info:







            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1087:
            
             OLM Reports ResolutionFailed when there are multiple upgrade paths between channel entries
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Catalog affected : icr.io/cpopen/datapower-operator-catalog:1.6.2





opm render icr.io/cpopen/datapower-operator-catalog:1.6.2 -o yaml > catalog.yaml

 yq 'select(.schema == "olm.channel") | select(.name=="v1.6")' catalog.yaml
entries:
  - name: datapower-operator.v1.6.0
    skipRange: '>=1.0.0 <1.6.0'
  - name: datapower-operator.v1.6.1
    replaces: datapower-operator.v1.6.0
    skipRange: '>=1.0.0 <1.6.1'
  - name: datapower-operator.v1.6.2
    replaces: datapower-operator.v1.6.1
    skipRange: '>=1.0.0 <1.6.2'
name: v1.6
package: datapower-operator
schema: olm.channel







This have worked fine untill 4.10 resolver changes. Also using both the replaces and skiprange seems to be okay the way it was explained here

https://v0-18-z.olm.operatorframework.io/docs/concepts/olm-architecture/operator-catalog/creating-an-update-graph/#skiprange





How reproducible:

use following subscription, install 1.6.0 and then upgrade to 1.6.2






apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    operators.coreos.com/datapower-operator.openshift-operators: ""
  name: datapower-operator
  namespace: openshift-operators
spec:
  channel: v1.6
  installPlanApproval: Manual
  name: datapower-operator
  source: datapower
  sourceNamespace: openshift-marketplace
  startingCSV: datapower-operator.v1.6.0








Error n subscription yaml : 





  conditions:
  - lastTransitionTime: "2022-09-09T13:42:08Z"
    message: all available catalogsources are healthy
    reason: AllCatalogSourcesHealthy
    status: "False"
    type: CatalogSourcesUnhealthy
  - message: 'a unique replacement chain within a channel is required to determine
      the relative order between channel entries, but 2 replacement chains were found
      in channel "v1.6" of package "datapower-operator": datapower-operator.v1.6.2...datapower-operator.v1.6.0,
      datapower-operator.v1.6.1...datapower-operator.v1.6.0'
    reason: ErrorPreventedResolution
    status: "True"
    type: ResolutionFailed







Logs 




I0909 13:43:51.492784       1 event.go:282] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"openshift-operators", UID:"aacda32d-748f-4408-88df-f895e74a23fe", APIVersion:"v1", ResourceVersion:"1260", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' a unique replacement chain within a channel is required to determine the relative order between channel entries, but 2 replacement chains were found in channel "v1.6" of package "datapower-operator": datapower-operator.v1.6.2...datapower-operator.v1.6.0, datapower-operator.v1.6.1...datapower-operator.v1.6.0
E0909 13:43:52.095288       1 queueinformer_operator.go:290] sync "openshift-operators" failed: a unique replacement chain within a channel is required to determine the relative order between channel entries, but 2 replacement chains were found in channel "v1.6" of package "datapower-operator": datapower-operator.v1.6.2...datapower-operator.v1.6.0, datapower-operator.v1.6.1...datapower-operator.v1.6.0









Expected results:



if this upgrade strategy which has worked fine before is still okay, this error should not be there. As per affected catalog maintainer, this seems to affecting reconciling of other resources.


            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2920:
            
             Update Prometheus Alerts
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Our Prometheus alerts are inconsistent with both upstream and sometimes our own vendor folder. Let's do a clean update run before the next release is branched off.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5299:
            
             [4.10] [OVN]Sometimes after reboot egress node, egress IP cannot be applied anymore.
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




[OVN][OSP] After reboot egress node, egress IP cannot be applied anymore.






Version-Release number of selected component (if applicable):




4.12.0-0.nightly-2022-11-07-181244






How reproducible:




Frequently happened in automation. But didn't reproduce it in manual.






Steps to Reproduce:




1. Label one node as egress node

2.
Config one egressIP object
STEP: Check  one EgressIP assigned in the object.

Nov  8 15:28:23.591: INFO: egressIPStatus: [{"egressIP":"192.168.54.72","node":"huirwang-1108c-pg2mt-worker-0-2fn6q"}]

3.
Reboot the node, wait for the node ready.







Actual results:




EgressIP cannot be applied anymore. Waited more than 1 hour.
 oc get egressip
NAME             EGRESSIPS       ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip-47031   192.168.54.72    





Expected results:




The egressIP should be applied correctly.





Additional info:





Some logs
E1108 07:29:41.849149       1 egressip.go:1635] No assignable nodes found for EgressIP: egressip-47031 and requested IPs: [192.168.54.72]
I1108 07:29:41.849288       1 event.go:285] Event(v1.ObjectReference{Kind:"EgressIP", Namespace:"", Name:"egressip-47031", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'NoMatchingNodeFound' no assignable nodes for EgressIP: egressip-47031, please tag at least one node with label: k8s.ovn.org/egress-assignable


W1108 07:33:37.401149       1 egressip_healthcheck.go:162] Could not connect to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107): context deadline exceeded
I1108 07:33:37.401348       1 master.go:1364] Adding or Updating Node "huirwang-1108c-pg2mt-worker-0-2fn6q"
I1108 07:33:37.437465       1 egressip_healthcheck.go:168] Connected to huirwang-1108c-pg2mt-worker-0-2fn6q (10.131.0.2:9107)






After this log, seems like no logs related to "192.168.54.72" happened.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Story 
            
            
            MON-1679:
            
             use static authorizer feature of kube-rbac-proxy 
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
The static authorizer feature has landed in upstream kube-rbac-proxy. Lets use it by configuring a static authorizer for all requests that hit a /metrics endpoint.



DoD:


    
	
  • Downstream kube-rbac-proxy is synced.
    • 
	
  • All CMO operands are configured with static authorization.
    • 
	
  • Bugzillas created for all non-monitoring components using kube-rbac-proxy for metrics authn/authz.
    • 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5864:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5077:
            
             Service spec value `externalTrafficPolicy` does not trigger rules update in ovnkube-node pod handlers on edit
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




[ovn] [ocp 4.10.z] Service `spec.externalTrafficPolicy` does not trigger rules update in ovnkube-node pod handlers on edit, even though it does successfully update the rules if deployed explicitly with that spec value set, or if you delete the handler pods for ovn (forces a refresh).




Version-Release number of selected component (if applicable):




observed in 4.10.32 and 4.10.40, tested on azure platform.




How reproducible:




every time




Steps to Reproduce:




1. Deploy a test pod with curlable resource in a test namespace

2. create a service from yaml exposing pod at internal clusterIP (example yaml provided by customer below)
~~~ 
apiVersion: v1
kind: Service
metadata:
  labels:
    run: test
  name: test
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: paas1
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster ##MODIFY THIS SPEC VALUE AND OBSERVE FAIL CONDITION
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    run: test
  sessionAffinity: None
  type: LoadBalancer
~~~

3. curl against service succeeds
4. edit service to change `spec.externalTrafficPolicy: local`
5. observe externalIP does not change, but healthz port updates
6. curl against same externalIP:port time out indefinitely, no response.

//workaround: 

delete service and redeploy with spec line set already to `local`, or delete ovnkube-node pod serving pod(s) to force refresh the local ruleset and allow traffic (curls subsequently will succeed).




Actual results:




spec change appears to update properly in the database but does not send a notification to update the ovnkube-node pod handlers (or similar) to allow traffic through once the externalTrafficPolicy spec value is changed.




Expected results:




spec change to service yaml should be immediately updated in DB AND update ovnkube-node handlers for same.




Additional info:




Attachments available and case number with specifics in next internal comment. 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1568:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4494:
            
             [release-4.10] openshift-ingress-operator with mTLS does not download CRL
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a clone of issue OCPBUGS-3049. The following is the description of the original issue: 

—

This is a copy of Bugzilla bug 2117524 for backport to 4.11.z



Original Text:



Description of problem:

On routers configured with mTLS and CRL defined in the CA with a CDP ; new CRL is downloaded only when restarting the ingress-operator.




2022-07-20T23:36:26.943Z	INFO	operator.clientca_configmap_controller	controller/controller.go:298	reconciling	{"request": "openshift-ingress-operator/service-bdrc"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:69	certificate revocation list has expired	{"subject key identifier": "6aa909992e9890457b2a8de5659a44cab8e867a8"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:69	retrieving certificate revocation list	{"subject key identifier": "6aa909992e9890457b2a8de5659a44cab8e867a8"}
2022-07-20T23:36:26.943Z	INFO	operator.crl	crl/crl_configmap.go:169	retrieving CRL distribution point	{"distribution point": "http://crl.domain.com/der/CN=XXXX,OU=XXX,O=XXX,C=XXX"}





Version-Release number of selected component (if applicable):

4.9.33



How reproducible:

Enable mTLS with a CRL



Actual results:

CRL is not download when expired 

Clients get "SSL client certificate not trusted" errors while accessing resources



Expected results:

ingress-operator triggers CRL download when approaching expiration date so that the configmap is updated without manual action

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5040:
            
             [release-4.10] unit-tests flaking on 4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Unit-tests flaking on 4.10 PRs 




Version-Release number of selected component (if applicable):




 




How reproducible:




Sometimes




Steps to Reproduce:




1.
2.
3.





Actual results:




Unit-test job fails with the following error: 

[Fail] OVN Pod Operations during execution [It] should not deallocate in-use and previously freed completed pods IP 
/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/pods_test.go:560
 Ran 153 of 153 Specs in 215.549 seconds
 FAIL! -- 152 Passed | 1 Failed | 0 Pending | 0 Skipped
 --- FAIL: TestClusterNode (216.05s)
 FAIL	github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn	228.843s
  {"component":"entrypoint","error":"wrapped process failed: exit status 2","file":"k8s.io/test-infra/prow/entrypoint/run.go:79","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2022-12-17T0




Expected results:




All tests pass




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-6736:
            
             [2072601] ovn-nbctl.log is never rotated
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Description of problem:
The startup script for ovnkube-master container starts ovn-nbctl daemon that logs to ovn-nbctl.log file[1]. However, apart from tailing it, nothing is done with that file. Concretely, it is never rotated.
Recently, one 4.8 cluster has hit the situation where this file grew up to ~24GB size. It was necessary to manually remove that file and restart ovnkube-master.
Version-Release number of selected component (if applicable):
4.8. It seems that 4.10 can hit this problem as well[2] but latest master branch has changed so much that I am not sure if it will be possible for this to happen in 4.11.
How reproducible:
Always (but may take very long and requires no ovnkube-master container restart)
Steps to Reproduce:
1. Let /run/ovn/ovn-nbctl.log grow for long enough
Actual results:
File growing without control
Expected results:
Log file to be eventually rotated.
References:
[1] - https://github.com/openshift/cluster-network-operator/blob/release-4.8/bindata/network/ovn-kubernetes/ovnkube-master.yaml#L813
[2] - https://github.com/openshift/cluster-network-operator/blob/release-4.10/bindata/network/ovn-kubernetes/ovnkube-master.yaml#L750




Version-Release number of selected component (if applicable):




 




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




 



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1239:
            
             [4.10] Disconnected IPI OCP cluster install on baremetal fails when hostname of master nodes does not include the text "master
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"    



Version-Release number of selected component (if applicable): 4.10.22



How reproducible:  Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"



Steps to Reproduce:



Perform disconnected IPI install of OCP 4.10.22 on bare metal with master nodes that do not contain the text "master"



Actual results: master nodes do come up.



Expected results: master nodes should come up despite that the text "master" is not in their hostname.



Additional info:



Disconnected IPI OCP 4.10.22 cluster install on baremetal fails when hostname of master nodes does not include "master"    




The code for the cluster-baremetal-operator at the following link: 



https://github.com/openshift/cluster-baremetal-operator/blob/49d7b249c5dcef8228f206eff4530a25f03b201f/controllers/provisioning_controller.go#L441



The following condition is concerning:



if strings.Contains(bmh.Name, "master") && len(bmh.Spec.BootMACAddress) > 0



The packages reveal that bmh.Name references the name inside the metadata of the BMH object. 



Should a customer have masters with names that do not include the text "master", the above condition can never become true, and so, the following slice is never created :



macs = append(macs, bmh.Spec.BootMACAddress)



 



 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1659:
            
             Whereabouts should allow non default interfaces to Pod IP list [backport 4.10]
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




Whereabouts doesn't allow the use of network interface names that are not preceded by the prefix "net", see https://github.com/k8snetworkplumbingwg/whereabouts/issues/130.




Version-Release number of selected component (if applicable):




 




How reproducible:




Always




Steps to Reproduce:



1. Define two Pods, one with the interface name 'port1' and the other with 'net-port1':




test-ip-removal-port1:	
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "port1",
                    "namespace": "default"
                  }
                ]

test-ip-removal-net-port1:
              k8s.v1.cni.cncf.io/networks:
                [
                  {
                    "name": "test-sriovnd",
                    "interface": "net-port1",
                    "namespace": "default"
                  }
                ]





2. IP allocated in the IPPool:




kind: IPPool
...
spec:
  allocations:
    "16":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-port1
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1





3. When the ip-reconciler job is run, the allocation for the port with the interface name 'port1' is removed:




[13:29][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        14m             11d

[13:29][]$ oc get ippools.whereabouts.cni.cncf.io -n openshift-multus   2001-1b70-820d-2610---64 -o yaml
apiVersion: whereabouts.cni.cncf.io/v1alpha1
kind: IPPool
metadata:
...
spec:
  allocations:
    "17":
      id: ...
      podref: test-ecoloma-1/test-ip-removal-net-port1
  range: 2001:1b70:820d:2610::/64

[13:30][]$ oc get cronjob -n openshift-multus
NAME            SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
ip-reconciler   */15 * * * *   False     0        9s              11d





 



Actual results:




The network interface with a name that doesn't have a 'net' prefix is removed from the ip-reconciler cronjob.




Expected results:




The network interface must not be removed, regardless of the name.




Additional info:




Upstream PR @ https://github.com/k8snetworkplumbingwg/whereabouts/pull/147 master PR @ https://github.com/openshift/whereabouts-cni/pull/94



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2196:
            
             Symptom Detection.Undiagnosed panic detected in pod
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This bug is a backport clone of [Bugzilla Bug 2075091](https://bugzilla.redhat.com/show_bug.cgi?id=2075091). The following is the description of the original bug:

—

Symptom Detection.Undiagnosed panic detected in pod



is failing frequently in CI, see:

https://sippy.ci.openshift.org/sippy-ng/tests/4.11/analysis?test=Symptom%20Detection.Undiagnosed%20panic%20detected%20in%20pod



This problem seemed existing before. But number of cases surged and caused two nightly payloads to be rejected:




https://amd64.ocp.releases.ci.openshift.org/releasestream/4.11.0-0.nightly/release/4.11.0-0.nightly-2022-04-12-150057

https://amd64.ocp.releases.ci.openshift.org/releasestream/4.11.0-0.nightly/release/4.11.0-0.nightly-2022-04-12-185124



After that, it mysteriously disappeared. 



Here is a specific case:



https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-e2e-azure-ovn-upgrade/1513895844351315968



Message from the test case:



{  pods/openshift-monitoring_kube-state-metrics-67c5b7c7c6-88vxn_kube-state-metrics_previous.log.gz:E0412 15:52:33.358619       1 runtime.go:78] Observed a panic: runtime.boundsError

{x:4, y:4, signed:true, code:0x0}

 (runtime error: index out of range [4] with length 4)}




Panic trace from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-e2e-azure-ovn-upgrade/1513895844351315968/artifacts/e2e-azure-ovn-upgrade/gather-extra/artifacts/pods/openshift-monitoring_kube-state-metrics-67c5b7c7c6-88vxn_kube-state-metrics_previous.log:



E0412 15:52:33.358619       1 runtime.go:78] Observed a panic: runtime.boundsError

{x:4, y:4, signed:true, code:0x0}

 (runtime error: index out of range [4] with length 4)

goroutine 77 [running]:

k8s.io/apimachinery/pkg/util/runtime.logPanic(

{0x1741840, 0xc000b635f0})

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d

k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000ac9740})

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75

panic({0x1741840, 0xc000b635f0}

)

	/usr/lib/golang/src/runtime/panic.go:1038 +0x215

k8s.io/kube-state-metrics/v2/internal/store.createPodContainerInfoFamilyGenerator.func1(0xc003422c00)

	/go/src/k8s.io/kube-state-metrics/internal/store/pod.go:134 +0x375

k8s.io/kube-state-metrics/v2/internal/store.wrapPodFunc.func1(

{0x1804880, 0xc003422c00})

	/go/src/k8s.io/kube-state-metrics/internal/store/pod.go:1386 +0x5a

k8s.io/kube-state-metrics/v2/pkg/metric_generator.(*FamilyGenerator).Generate(...)

	/go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:67

k8s.io/kube-state-metrics/v2/pkg/metric_generator.ComposeMetricGenFuncs.func1({0x1804880, 0xc003422c00}

)

	/go/src/k8s.io/kube-state-metrics/pkg/metric_generator/generator.go:107 +0xd8

k8s.io/kube-state-metrics/v2/pkg/metrics_store.(*MetricsStore).Add(0xc0000c13c0, 

{0x1804880, 0xc003422c00})

	/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:72 +0xd4

k8s.io/kube-state-metrics/v2/pkg/metrics_store.(*MetricsStore).Update(0xc003422c00, {0x1804880, 0xc003422c00}

)

	/go/src/k8s.io/kube-state-metrics/pkg/metrics_store/metrics_store.go:87 +0x25

k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc000192fc0, 

{0x0, 0x0, 0x26cdee0}

, 

{0x1a373f8, 0xc0011c24c0}

, 0xc000623d60, 0xc0005ff380, 0xc0002cc480)

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/client-go/tools/cache/reflector.go:506 +0xa55

k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000192fc0, 0xc0002cc480)

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/client-go/tools/cache/reflector.go:429 +0x696

k8s.io/client-go/tools/cache.(*Reflector).Run.func1()

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/client-go/tools/cache/reflector.go:221 +0x26

k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f02ffada1d0)

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67

k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00036a2c0, 

{0x1a1daa0, 0xc000386e60}

, 0x1, 0xc0002cc480)

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6

k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000192fc0, 0xc0002cc480)

	/go/src/k8s.io/kube-state-metrics/vendor/k8s.io/client-go/tools/cache/reflector.go:220 +0x1f8

created by k8s.io/kube-state-metrics/v2/internal/store.(*Builder).startReflector

	/go/src/k8s.io/kube-state-metrics/internal/store/builder.go:508 +0x2c8

panic: runtime error: index out of range [4] with length 4 [recovered]

	panic: runtime error: index out of range [4] with length 4




It points to https://github.com/openshift/kube-state-metrics/blob/6efa87f858ee53028fd2de40941b61c09e9ee049/internal/store/pod.go#L134 where the len of p.Status.ContainerStatuses and p.Spec.Containers seems to diverge. 



Unfortunately the condition is ephemeral and the condition that caused the panic does not exist in the must-gather data. 



The ask is to safe guard the code to avoid the panic and log useful debugging info to track down offenders.

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-2208:
            
             [4.10] Dual stack cluster fails on installation when multi-path routing entries exist
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




We observed that a dual stack cluster deployed with AI gui only fails.
This cluster is dhcp for ipv4, RA/RS autoconfiguration for ipv6.

It fails with error in the onvkube container

```
I0906 07:45:43.044090   87450 gateway_init.go:261] Initializing Gateway Functionality
I0906 07:45:43.046398   87450 gateway_localnet.go:152] Node local addresses initialized to: map[10.131.31.214:{10.131.31.208 fffffff0} 10.255.0.2:{10.255.0.0 fffffe00} 127.0.0.1:{127.0.0.0 ff000000} 2001:1b74:480:613a:f6e9:d4ff:fef1:6f26:{2001:1b74:480:613a:: ffffffffffffffff0000000000000000} ::1:{::1 ffffffffffffffffffffffffffffffff} fd01:0:0:1::2:{fd01:0:0:1:: ffffffffffffffff0000000000000000} fe80::8ce9:b4ff:fe1a:1208:{fe80:: ffffffffffffffff0000000000000000} fe80::c8ef:ecff:fee3:64c7:{fe80:: ffffffffffffffff0000000000000000} fe80::f6e9:d4ff:fef1:6f26:{fe80:: ffffffffffffffff0000000000000000}]
I0906 07:45:43.047759   87450 helper_linux.go:71] Provided gateway interface "br-ex", found as index: 7
I0906 07:45:43.048045   87450 helper_linux.go:97] Found default gateway interface br-ex 10.131.31.209
I0906 07:45:43.048152   87450 helper_linux.go:71] Provided gateway interface "br-ex", found as index: 7
F0906 07:45:43.048318   87450 ovnkube.go:133] failed to get default gateway interface
```

on the node we observed that there is multi-path entry during

```
default proto ra metric 48 pref medium
        nexthop via fe80::e2f6:2d01:ab14:ec71 dev br-ex weight 1
        nexthop via fe80::e2f6:2d01:ab11:c271 dev br-ex weight 1
```

I manually remove one of the entries (`ip route delete`) and then delete the ovnkube-node pod. Then the installation continues, container works.

Every time there is multiple entry, if the onvkube-node starts, it fails.







Version-Release number of selected component (if applicable):




4.10.30




How reproducible:




 




Steps to Reproduce:




1.
2.
3.





Actual results:




 




Expected results:




 




Additional info:




There might a side issue: the interface of the node upon boot takes time to get the ipv6 autoconfiguration, no RS packets seemed to be sent out (observed zero on all routers).



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-3078:
            
             Backkport OCPBUGS-1718 to OCP 4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
This is a request to back port the fix in OCPBUGS-1718 to Openshift 4.10.



Description cloned from that bug:

Description of problem:




prometheus-k8s-0 ends in CrashLoopBackOff with evel=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0" on SNO after hard reboot tests




Version-Release number of selected component (if applicable):




4.11.6




How reproducible:




Not always, after ~10 attempts




Steps to Reproduce:




1. Deploy SNO with Telco DU profile applied
2. Hard reboot node via out of band interface
3. oc -n openshift-monitoring get pods prometheus-k8s-0 




Actual results:




NAME               READY   STATUS             RESTARTS          AGE
prometheus-k8s-0   5/6     CrashLoopBackOff   125 (4m57s ago)   5h28m





Expected results:




Running




Additional info:




Attaching must-gather.

The pod recovers successfully after deleting/re-creating.


[kni@registry.kni-qe-0 ~]$ oc -n openshift-monitoring logs prometheus-k8s-0
ts=2022-09-26T14:54:01.919Z caller=main.go:552 level=info msg="Starting Prometheus Server" mode=server version="(version=2.36.2, branch=rhaos-4.11-rhel-8, revision=0d81ba04ce410df37ca2c0b1ec619e1bc02e19ef)"
ts=2022-09-26T14:54:01.919Z caller=main.go:557 level=info build_context="(go=go1.18.4, user=root@371541f17026, date=20220916-14:15:37)"
ts=2022-09-26T14:54:01.919Z caller=main.go:558 level=info host_details="(Linux 4.18.0-372.26.1.rt7.183.el8_6.x86_64 #1 SMP PREEMPT_RT Sat Aug 27 22:04:33 EDT 2022 x86_64 prometheus-k8s-0 (none))"
ts=2022-09-26T14:54:01.919Z caller=main.go:559 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-09-26T14:54:01.919Z caller=main.go:560 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-09-26T14:54:01.921Z caller=web.go:553 level=info component=web msg="Start listening for connections" address=127.0.0.1:9090
ts=2022-09-26T14:54:01.922Z caller=main.go:989 level=info msg="Starting TSDB ..."
ts=2022-09-26T14:54:01.924Z caller=tls_config.go:231 level=info component=web msg="TLS is disabled." http2=false
ts=2022-09-26T14:54:01.926Z caller=main.go:848 level=info msg="Stopping scrape discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:862 level=info msg="Stopping notify discovery manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:951 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-09-26T14:54:01.926Z caller=manager.go:961 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:899 level=info msg="Stopping scrape manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:858 level=info msg="Notify discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:891 level=info msg="Scrape manager stopped"
ts=2022-09-26T14:54:01.926Z caller=notifier.go:599 level=info component=notifier msg="Stopping notification manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:844 level=info msg="Scrape discovery manager stopped"
ts=2022-09-26T14:54:01.926Z caller=manager.go:937 level=info component="rule manager" msg="Starting rule manager..."
ts=2022-09-26T14:54:01.926Z caller=main.go:1120 level=info msg="Notifier manager stopped"
ts=2022-09-26T14:54:01.926Z caller=main.go:1129 level=error err="opening storage failed: /prometheus/chunks_head/000002: invalid magic number 0"



            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Task 
            
            
            MON-975:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-5575:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-647:
            
             Prefer local dns does not work expectedly on OCPv4.10
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:




When queried dns hostname from certain pod on the certain node, responded from random coredns pod, not prefer local one. Is it expected result ?

# In OCP v4.8.13 case
// Ran dig command on the certain node which is running the following test-7cc4488d48-tqc4m pod.
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
:
07:16:33 :172.217.175.238
07:16:34 :172.217.175.238 <--- Refreshed the upstream result
07:16:36 :142.250.207.46
07:16:37 :142.250.207.46

// The dig results is matched with the running node one as you can see the above one.
$ oc rsh  test-7cc4488d48-tqc4m bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
:
07:16:35 :172.217.175.238 
07:16:36 :172.217.175.238 <--- At the same time, the pod dig result is also refreshed.
07:16:37 :142.250.207.46
07:16:38 :142.250.207.46


But in v4.10 case, in contrast, the dns query result is various and responded randomly regardless local dns results on the node as follows.

# In OCP v4.10.23 case, pod's response from DNS services are not consistent.
$ oc rsh test-848fcf8ddb-zrcbx  bash -c 'while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done'
07:23:00 :142.250.199.110
07:23:01 :142.250.207.46
07:23:02 :142.250.207.46
07:23:03 :142.250.199.110
07:23:04 :142.250.199.110
07:23:05 :172.217.161.78

# Even though the node which is running the pod keep responding the same IP...
sh-4.4# while : ; do echo -n "$(date '+%H:%M:%S') :"; dig google.com +short; sleep 1; done
07:23:00 :172.217.161.78
07:23:01 :172.217.161.78
07:23:02 :172.217.161.78
07:23:03 :172.217.161.78
07:23:04 :172.217.161.78
07:23:05 :172.217.161.78






Version-Release number of selected component (if applicable):




v4.10.23 (ROSA)
SDN: OpenShiftSDN




How reproducible:




You can always reproduce this issue using "dig google.com" from both any pod and the node the pod running according to the above "Description" details.




Steps to Reproduce:




1. Run any usual pod, and check which node the pod is running on.
2. Run dig google.com on the pod and the node.
3. Check the IP is consistent with the running node each other. 




Actual results:




The response IPs are not consistent and random IP is responded.




Expected results:




The response IP is kind of consistent, and aware of prefer local dns.




Additional info:




This issue affects EgressNetworkPolicy dnsName feature.




 

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-374:
            
             openshift-tests: allow -f to match tests for any test suite
          

          
          
          

            
            
            
            
            
            
            
          

          
          
          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-957:
            
             [2087981] PowerOnVM_Task is deprecated use PowerOnMultiVM_Task for DRS ClusterRecommendation
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:



Sometimes we see VMs fail to power on when the land on a host that does not have enough resources. The current power on does not retry or leverage DRS to power on the node on a suitable host.



https://github.com/vmware/govmomi/issues/1026



Our code is still making calls to PowerOnVM_Task which, according to the vsphere docs, is deprecated and we should use PowerOnMultiVM_Task instead.



PowerOnVM_Task does not return a DRS ClusterRecommendation, no vmotion nor host power operations will be done as part of a DRS-facilitated power on. To have DRS consider such operations use PowerOnMultiVM_Task.



https://vdc-download.vmware.com/vmwb-repository/dcr-public/b50dcbbf-051d-4204-a3e7-e1b618c1e384/538cf2ec-b34f-4bae-a332-3820ef9e7773/vim.VirtualMachine.html#powerOn:

As of vSphere API 5.1, use of this method with vCenter Server is deprecated; use PowerOnMultiVM_Task instead.



Version-Release number of selected component (if applicable):

4.8.x



How reproducible:

Always



Steps to Reproduce:

1.

2.

3.



Actual results:

Sometimes powers on fails requiring manual intervention.



Expected results:

PowerOn should use DRS to ensure it's always successful.



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-660:
            
             [release-4.10] OVN master trying to deleteLogicalPort for object which is already gone
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
Description of problem:

This is a clone of https://issues.redhat.com/browse/OCPBUGS-658



 



Description of problem: Numerous erroreneous logs in OVN master



I0823 18:00:11.163491       1 obj_retry.go:1063] Retry object setup: *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k

I0823 18:00:11.163546       1 obj_retry.go:1096] Removing old object: *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k

I0823 18:00:11.163555       1 pods.go:124] Deleting pod: openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k

I0823 18:00:11.163631       1 obj_retry.go:1103] Retry delete failed for *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k, will try again later: deleteLogicalPort failed for pod openshift-operator-lifecycle-manager_collect-profiles-27687900-hlp6k: unable to locate portUUID+nodeName for pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k: error getting logical port <nil>: object not found

W0823 18:00:41.163633       1 obj_retry.go:1031] Dropping retry entry for *v1.Pod openshift-operator-lifecycle-manager/collect-profiles-27687900-hlp6k: exceeded number of failed attempts



Must-gather: http://shell.lab.bos.redhat.com/~anusaxen/must-gather.local.2234927131259452300/



 



Version-Release number of selected component (if applicable): 4.12.0-0.nightly-2022-08-23-031342



How reproducible: Always



Steps to Reproduce:

1. Bring up OVN cluster on 4.12

2.

3.



Actual results: deleteLogicalPort failed for already gone object



Expected results: deleteLogicalPort should not keep retrying post object deletion



Additional info:

            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-4260:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1804:
            
             The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              The details of this Jira Card are restricted (Red Hat Employee and Contractors only)
            

          

          
          
          
          
          
          
        

      

      

        

        

      

    

    

  
  

    

      

        

          

            
            Bug 
            
            
            OCPBUGS-1517:
            
             [OCP 4.10] Fix generate script in CBO
          

          
          
          

            
            
            
            
            
            
            
          

          
          

            

              
plus smaller fixes

this is a backport of https://github.com/openshift/cluster-baremetal-operator/pull/263