Incomplete Features
When this image was assembled, these features were not yet completed. Therefore, only the Jira Cards included here are part of this release
OLM would have to support a mechanism like podAffinity which allows multiple architecture values to be specified which enables it to pin operators to the matching architecture worker nodes
Ref: https://github.com/openshift/enhancements/pull/1014
Cut a new release of the OLM API and update OLM API dependency version (go.mod) in OLM package; then
Bring the upstream changes from OLM-2674 to the downstream olm repo.
A/C:
- New OLM API version release
- OLM API dependency updated in OLM Project
- OLM Subscription API changes downstreamed
- OLM Controller changes downstreamed
- Changes manually tested on Cluster Bot
Epic Goal
- Enabling integration of single hub cluster to install both ARM and x86 spoke clusters
- Enabling support for heterogeneous OCP clusters
- document requirements deployment flows
- support in disconnected environment
Why is this important?
- clients request
Scenarios
- Users manage both ARM and x86 machines, we should not require to have two different hub clusters
- Users manage a mixed architecture clusters without requirement of all the nodes to be of the same architecture
Acceptance Criteria
- Process is well documented
- we are able to install in a disconnected environment
We have a set of images
- quay.io/edge-infrastructure/assisted-installer-agent:latest
- quay.io/edge-infrastructure/assisted-installer-controller:latest
- quay.io/edge-infrastructure/assisted-installer:latest
that should become multiarch images. This should be done both in upstream and downstream.
As a reference, we have built internally those images as multiarch and made them available as
- registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:latest
- registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:latest
- registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:latest
They can be consumed by the Assisted Serivce pod via the following env
- name: AGENT_DOCKER_IMAGE
value: registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:latest
- name: CONTROLLER_IMAGE
value: registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:latest
- name: INSTALLER_IMAGE
value: registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:latest
As a user, I should be able to configure CSI driver to have a storage topology.
Feature Overview
We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.
Goals
- Feature enhancements (performance, scale, configuration, UX, ...)
- Modernization (incorporation and productization of new technologies)
Requirements
- Core Networking Stability
- Core Networking Performance and Scale
- Core Neworking Extensibility (Multus CNIs)
- Core Networking UX (Observability)
- Core Networking Security and Compliance
In Scope
- Network Edge (ingress, DNS, LB)
- SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
- Networking Observability
Out of Scope
There are definitely grey areas, but in general:
- CNV
- Service Mesh
- CNF
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
Goal: Provide queryable metrics and telemetry for cluster routes and sharding in an OpenShift cluster.
Problem: Today we test OpenShift performance and scale with best-guess or anecdotal evidence for the number of routes that our customers use. Best practices for a large number of routes in a cluster is to shard, however we have no visibility with regard to if and how customers are using sharding.
Why is this important? These metrics will inform our performance and scale testing, documented cluster limits, and how customers are using sharding for best practice deployments.
Dependencies (internal and external):
Prioritized epics + deliverables (in scope / not in scope):
Not in scope:
Estimate (XS, S, M, L, XL, XXL):
Previous Work:
Open questions:
Acceptance criteria:
Epic Done Checklist:
- CI - CI Job & Automated tests: <link to CI Job & automated tests>
- Release Enablement: <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR orf GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
- Notes for Done Checklist:
- Adding links to the above checklist with multiple teams contributing; select a meaningful reference for this Epic.
- Checklist added to each Epic in the description, to be filled out as phases are completed - tracking progress towards “Done” for the Epic.
Description:
As described in the Design Doc, the following information is needed to be exported from Cluster Ingress Operator:
- Number of routes/shard
Design 2 will be implemented as part of this story.
Acceptance Criteria:
- Support for exporting the above mentioned metrics by Cluster Ingress Operator
Description:
As described in the Metrics to be sent via telemetry section of the Design Doc, the following metrics is needed to be sent from OpenShift cluster to Red Hat premises:
- Minimum Routes per Shard
- Recording Rule – cluster:route_metrics_controller_routes_per_shard:min : min(route_metrics_controller_routes_per_shard)
- Gives the minimum value of Routes per Shard.
- Maximum Routes per Shard
- Recording Rule – cluster:route_metrics_controller_routes_per_shard:max : max(route_metrics_controller_routes_per_shard)
- Gives the maximum value of Routes per Shard.
- Average Routes per Shard
- Recording Rule – cluster:route_metrics_controller_routes_per_shard:avg : avg(route_metrics_controller_routes_per_shard)
- Gives the average value of Routes per Shard.
- Median Routes per Shard
- Recording Rule – cluster:route_metrics_controller_routes_per_shard:median : quantile(0.5, route_metrics_controller_routes_per_shard)
- Gives the median value of Routes per Shard.
- Number of Routes summed by TLS Termination type
- Recording Rule – cluster:openshift_route_info:tls_termination:sum : sum (openshift_route_info) by (tls_termination)
- Gives the number of Routes for each tls_termination value. The possible values for tls_termination are edge, passthrough and reencrypt.
The metrics should be allowlisted on the cluster side.
The steps described in Sending metrics via telemetry are needed to be followed. Specifically step 5.
Depends on CFE-478.
Acceptance Criteria:
- Support for sending the above mentioned metrics from OpenShift clusters to the Red Hat premises by allowlisting metrics on the cluster side
Epic Goal
- Make it possible to disable the console operator at install time, while still having a supported+upgradeable cluster.
Why is this important?
- It's possible to disable console itself using spec.managementState in the console operator config. There is no way to remove the console operator, though. For clusters where an admin wants to completely remove console, we should give the option to disable the console operator as well.
Scenarios
- I'm an administrator who wants to minimize my OpenShift cluster footprint and who does not want the console installed on my cluster
Acceptance Criteria
- It is possible at install time to opt-out of having the console operator installed. Once the cluster comes up, the console operator is not running.
Dependencies (internal and external)
- Composable cluster installation
Previous Work (Optional):
- https://docs.google.com/document/d/1srswUYYHIbKT5PAC5ZuVos9T2rBnf7k0F1WV2zKUTrA/edit#heading=h.mduog8qznwz
- https://docs.google.com/presentation/d/1U2zYAyrNGBooGBuyQME8Xn905RvOPbVv3XFw3stddZw/edit#slide=id.g10555cc0639_0_7
Open questions::
- The console operator manages the downloads deployment as well. Do we disable the downloads deployment? Long term we want to move to CLI manager: https://github.com/openshift/enhancements/blob/6ae78842d4a87593c63274e02ac7a33cc7f296c3/enhancements/oc/cli-manager.md
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
In the console-operator repo we need to add `capability.openshift.io/console` annotation to all the manifests that the operator either contains creates on the fly.
Manifests are currently present in /bindata and /manifest directories.
Here is example of the insights-operator change.
Here is the overall enhancement doc.
This is a epic bucket for all activities surrounding the creation of declarative approach to release and maintain OLM catalogs.
Epic Goal
- Allow Operator Authors to easily change the layout of the update graph in a single location so they can version/maintain/release it via git and have more approachable controls about graph vertices than today's replaces, skips and/or skipRange taxonomy
- Allow Operators authors to have control over channel and bundle channel membership
Why is this important?
- The imperative catalog maintenance approach so far with opm is being moved to a declarative format (
OLM-2127andOLM-1780) moving away from bundle-level controls but the update graph properties are still attached to a bundle - We've received feedback from the RHT internal developer community that maintaining and reasoning about the graph in the context of a single channel is still too hard, even with visualization tools
- making the update graph easily changeable is important to deliver on some of the promises of declarative index configuration
- The current interface for declarative index configuration still relies on skips, skipRange and replaces to shape the graph on a per-bundle level - this is too complex at a certain point with a lot of bundles in channels, we need to something at the package level
Scenarios
- An Operator author wants to release a new version replacing the latest version published previously
- After additional post-GA testing an Operator author wants to establish a new update path to an existing released version from an older, released version
- After finding a bug post-GA an Operator author wants to temporarily remove a known to be problematic update path
- An automated system wants to push a bundle inbetween an existing update path as a result of an Operator (base) image rebuild (Freshmaker use case)
- A user wants to take a declarative graph definition and turn it into a graphical image for visually ensuring the graph looks like they want
- An Operator author wants to promote a certain bundle to an additional / different channel to indicate progress in maturity of the operator.
Acceptance Criteria
- The declarative format has to be user readable and terse enough to make quick modifications
- The declarative format should be machine writeable (Freshmaker)
- The update graph is declared and modified in a text based format aligned with the declarative config
- it has to be possible to add / removes edges at the leave of the graph (releasing/unpublishing a new version)
- it has to be possible to add/remove new vertices between existing edges (releasing/retracting a new update path)
- it has to be possible to add/remove new edges in between existing vertices (releasing/unpublishing a version inbetween, freshmaker user case)
- it has to be possible to change the channel member ship of a bundle after it's published (channel promotion)
- CI - MUST be running successfully with tests automated
- it has to be possible to add additional metadata later to implement
OLM-2087andOLM-259if required
Dependencies (internal and external)
- Declarative Index Config (
OLM-2127)
Previous Work:
- Declarative Index Config (
OLM-1780)
Related work
- https://docs.google.com/document/d/1RR8EQ-7VEi0ItlnKsbHBlnSP8M1MaI0J5gSbqc9JM2w/edit#heading=h.gscpgotgkcn3\
OLM-2087
Open questions:
- What other manipulation scenarios are required?
When working on this Epic, it's important to keep in mind this other potentially related Epic: https://issues.redhat.com/browse/OLM-2276
enhance the veneer rendering to be able to read the input veneer data from stdin, via a pipe, in a manner similar to https://dev.to/napicella/linux-pipes-in-golang-2e8j
then the command could be used in a manner similar to many k8s examples like
```shell
opm alpha render-veneer semver -o yaml < infile > outfile
```
Upstream issue link: https://github.com/operator-framework/operator-registry/issues/1011
Jira Description
As an OPM maintainer, I want to downstream the PR for (OCP 4.12 ) and backport it to OCP 4.11 so that IIB will NOT be impacted by the changes when it upgrades the OPM version to use the next/future opm upstream release (v1.25.0).
Summary / Background
IIB(the downstream service that manages the indexes) uses the upstream version and if they bump the OPM version to the next/future (v1.25.0) release with this change before having the downstream images updated then: the process to manage the indexes downstream will face issues and it will impact the distributions.
Acceptance Criteria
- The changes in the PR are available for the releases which uses FBC -> OCP 4.11, 4.12
Definition of Ready
- PRs merged into downstream OCP repos branches 4.11/4.12
Definition of Done
- We checked that the downstream images are with the changes applied (i.e.: we can try to verify in the same way that we checked if the changes were in the downstream for the fix
OLM-2639)
Feature Overview
Provide CSI drivers to replace all the intree cloud provider drivers we currently have. These drivers will probably be released as tech preview versions first before being promoted to GA.
Goals
- Framework for rapid creation of CSI drivers for our cloud providers
- CSI driver for AWS EBS
- CSI driver for AWS EFS
- CSI driver for GCP
- CSI driver for Azure
- CSI driver for VMware vSphere
- CSI Driver for Azure Stack
- CSI Driver for Alicloud
- CSI Driver for IBM Cloud
Requirements
| Requirement | Notes | isMvp? |
|---|---|---|
| Framework for CSI driver | TBD | Yes |
| Drivers should be available to install both in disconnected and connected mode | Yes | |
| Drivers should upgrade from release to release without any impact | Yes | |
| Drivers should be installable via CVO (when in-tree plugin exists) |
Out of Scope
This work will only cover the drivers themselves, it will not include
- enhancements to the CSI API framework
- the migration to said drivers from the the intree drivers
- work for non-cloud provider storage drivers (FC-SAN, iSCSI) being converted to CSI drivers
Background, and strategic fit
In a future Kubernetes release (currently 1.21) intree cloud provider drivers will be deprecated and replaced with CSI equivalents, we need the drivers created so that we continue to support the ecosystems in an appropriate way.
Assumptions
- Storage SIG won't move out the changeover to a later Kubernetes release
Customer Considerations
Customers will need to be able to use the storage they want.
Documentation Considerations
- Target audience: cluster admins
- Updated content: update storage docs to show how to use these drivers (also better expose the capabilities)
This Epic is to track the GA of this feature
Goal
- Make available the Google Cloud File Service via a CSI driver, it is desirable that this implementation has dynamic provisioning
- Without GCP filestore support, we are limited to block / RWO only (GCP PD 4.8 GA)
- Align with what we support on other major public cloud providers.
Why is this important?
- There is a know storage gap with google cloud where only block is supported
- More customers deploying on GCE and asking for file / RWX storage.
Scenarios
- Install the CSI driver
- Remove the CSI Driver
- Dynamically provision a CSI Google File PV*
- Utilise a Google File PV
- Assess optional features such as resize & snapshot
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Customers::
- Telefonica Spain
- Deutsche Bank
Done Checklist
- CI - CI is running, tests are automated and merged.
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
As an OCP user, I want images for GCP Filestore CSI Driver and Operator, so that I can install them on my cluster and utilize GCP Filestore shares.
We need to continue to maintain specific areas within storage, this is to capture that effort and track it across releases.
Goals
- To allow OCP users and cluster admins to detect problems early and with as little interaction with Red Hat as possible.
- When Red Hat is involved, make sure we have all the information we need from the customer, i.e. in metrics / telemetry / must-gather.
- Reduce storage test flakiness so we can spot real bugs in our CI.
Requirements
| Requirement | Notes | isMvp? |
|---|---|---|
| Telemetry | No | |
| Certification | No | |
| API metrics | No | |
Out of Scope
n/a
Background, and strategic fit
With the expected scale of our customer base, we want to keep load of customer tickets / BZs low
Assumptions
Customer Considerations
Documentation Considerations
- Target audience: internal
- Updated content: none at this time.
Notes
In progress:
- CSI certification flakes a lot. We should fix it before we start testing migration.
- In progress (API server restarts...) https://bugzilla.redhat.com/show_bug.cgi?id=1865857
- Get local-storage-operator and AWS EBS CSI driver operator logs in must-gather (OLM-managed operators are not included there)
- In progress for LSO (must-gather script being included in image) https://bugzilla.redhat.com/show_bug.cgi?id=1756096
- CI flakes:
- Configurable timeouts for e2e tests
- Azure is slow and times out often
- Cinder times out formatting volumes
- AWS resize test times out
- Configurable timeouts for e2e tests
High prio:
- Env. check tool for VMware - users often mis-configure permissions there and blame OpenShift. If we had a tool they could run, it might report better errors.
- Should it be part of the installer?
- Spike exists
- Add / use cloud API call metrics
-
- Helps customers to understand why things are slow
- Helps build cop to understand a flake
- With a post-install step that filters data from Prometheus that’s still running in the CI job.
- Ideas:
- Cloud is throttling X% of API calls longer than Y seconds
- Attach / detach / provisioning / deletion / mount / unmount / resize takes longer than X seconds?
- Capture metrics of operations that are stuck and won’t finish.
- Sweep operation map from executioner???
- Report operation metric into the highest bucket after the bucket threshold (i.e. if 10minutes is the last bucket, report an operation into this bucket after 10 minutes and don’t wait for its completion)?
- Ask the monitoring team?
- Include in CSI drivers too.
- With alerts too
- Report events for cloud issues
- E.g. cloud API reports weird attach/provision error (e.g. due to outage)
- What volume plugins actually users use the most? https://issues.redhat.com/browse/STOR-324
Unsorted
- As the number of storage operators grows, it would be grafana board for storage operators
- CSI driver metrics (from CSI sidecars + the driver itself + its operator?)
- CSI migration?
- Get aggregated logs in cluster
- They're rotated too soon
- No logs from dead / restarted pods
- No tools to combine logs from multiple pods (e.g. 3 controller managers)
- What storage issues customers have? it was 22% of all issues.
- Insufficient docs?
- Probably garbage
- Document basic storage troubleshooting for our supports
- What logs are useful when, what log level to use
- This has been discussed during the GSS weekly team meeting; however, it would be beneficial to have this documented.
- Common vSphere errors, their debugging and fixing.
- Document sig-storage flake handling - not all failed [sig-storage] tests are ours
Epic Goal
- Update all images that we ship with OpenShift to the latest upstream releases and libraries.
- Exact content of what needs to be updated will be determined as new images are released upstream, which is not known at the beginning of OCP development work. We don't know what new features will be included and should be tested and documented. Especially new CSI drivers releases may bring new, currently unknown features. We expect that the amount of work will be roughly the same as in the previous releases. Of course, QE or docs can reject an update if it's too close to deadline and/or looks too big.
Traditionally we did these updates as bugfixes, because we did them after the feature freeze (FF). Trying no-feature-freeze in 4.12. We will try to do as much as we can before FF, but we're quite sure something will slip past FF as usual.
Why is this important?
- We want to ship the latest software that contains new features and bugfixes.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
This includes ibm-vpc-node-label-updater!
(Using separate cards for each driver because these updates can be more complicated)
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update all OCP and kubernetes libraries in storage operators to the appropriate version for OCP release.
This includes (but is not limited to):
- Kubernetes:
- client-go
- controller-runtime
- OCP:
- library-go
- openshift/api
- openshift/client-go
- operator-sdk
Operators:
- aws-ebs-csi-driver-operator
- aws-efs-csi-driver-operator
- azure-disk-csi-driver-operator
- azure-file-csi-driver-operator
- openstack-cinder-csi-driver-operator
- gcp-pd-csi-driver-operator
- gcp-filestore-csi-driver-operator
- manila-csi-driver-operator
- ovirt-csi-driver-operator
- vmware-vsphere-csi-driver-operator
- alibaba-disk-csi-driver-operator
- ibm-vpc-block-csi-driver-operator
- csi-driver-shared-resource-operator
- cluster-storage-operator
- csi-snapshot-controller-operator
- local-storage-operator
- vsphere-problem-detector
- https://github.com/openshift/alibaba-disk-csi-driver-operator/pull/36
- https://github.com/openshift/aws-ebs-csi-driver-operator/pull/161
- https://github.com/openshift/azure-disk-csi-driver-operator/pull/53
- https://github.com/openshift/azure-file-csi-driver-operator/pull/37
- https://github.com/openshift/cluster-csi-snapshot-controller-operator/pull/128
- https://github.com/openshift/cluster-storage-operator/pull/317
- https://github.com/openshift/csi-driver-manila-operator/pull/155
- https://github.com/openshift/csi-driver-shared-resource-operator/pull/57
- https://github.com/openshift/gcp-pd-csi-driver-operator/pull/52
- https://github.com/openshift/ibm-vpc-block-csi-driver-operator/pull/42
- https://github.com/openshift/openstack-cinder-csi-driver-operator/pull/92
- https://github.com/openshift/ovirt-csi-driver-operator/pull/107
- https://github.com/openshift/vmware-vsphere-csi-driver-operator/pull/109
- https://github.com/openshift/vsphere-problem-detector/pull/88
There is a new driver release 5.0.0 since the last rebase that includes snapshot support:
https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/releases/tag/v5.0.0
Rebase the driver on v5.0.0 and update the deployments in ibm-vpc-block-csi-driver-operator.
There are no corresponding changes in ibm-vpc-node-label-updater since the last rebase.
Update the driver to the latest upstream release. Notify QE and docs with any new features and important bugfixes that need testing or documentation.
(Using separate cards for each driver because these updates can be more complicated)
Update all CSI sidecars to the latest upstream release.
- external-attacher
- external-provisioner
- external-resizer
- external-snapshotter
- node-driver-registrar
- livenessprobe
This includes update of VolumeSnapshot CRDs in https://github.com/openshift/cluster-csi-snapshot-controller-operator/tree/master/assets
The End of General support for vSphere 6.7 will be on October 15, 2022. So, vSphere 6.7 will be deprecated for 4.11.
We want to encourage vSphere customers to upgrade to vSphere 7 in OCP 4.11 since VMware is EOLing (general support) for vSphere 6.7 in Oct 2022.
We want the cluster Upgradeable=false + have a strong alert pointing to our docs / requirements.
related slack: https://coreos.slack.com/archives/CH06KMDRV/p1647541493096729
Epic Goal
- Enable the migration from a storage intree driver to a CSI based driver with minimal impact to the end user, applications and cluster
- These migrations would include, but are not limited to:
- CSI driver for AWS EBS
- CSI driver for GCP
- CSI driver for Azure (file and disk)
- CSI driver for VMware vSphere
Why is this important?
- OpenShift needs to maintain it's ability to enable PVCs and PVs of the main storage types
- CSI Migration is getting close to GA, we need to have the feature fully tested and enabled in OpenShift
- Upstream intree drivers are being deprecated to make way for the CSI drivers prior to intree driver removal
Scenarios
- User initiated move to from intree to CSI driver
- Upgrade initiated move from intree to CSI driver
- Upgrade from EUS to EUS
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
On new installations, we should make the StorageClass created by the CSI operator the default one.
However, we shouldn't do that on an upgrade scenario. The main reason is that users might have set a different quota on the CSI driver Storage Class.
Exit criteria:
- New clusters get the CSI Storage Class as the default one.
- Existing clusters don't get their default Storage Classes changed.
This Epic tracks the GA of this feature
Epic Goal
- Provide the ability to move, as seamlessly as possible from the intree GCE driver to it's CSI equivalent
- https://github.com/kubernetes/enhancements/issues/1488
Why is this important?
- OpenShift needs to maintain it's ability to enable PVCs and PVs of the main storage types
- CSI Migration is getting close to GA, we need to have the feature fully tested and enabled in OpenShift
- Upstream intree drivers are being deprecated to make way for the CSI drivers prior to intree driver removal
Scenarios
- User initiated move to from intree to CSI driver
- Upgrade initiated move from intree to CSI driver
- Upgrade from EUS to EUS
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
- ...
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
On new installations, we should make the StorageClass created by the CSI operator the default one.
However, we shouldn't do that on an upgrade scenario. The main reason is that users might have set a different quota on the CSI driver Storage Class.
Exit criteria:
- New clusters get the CSI Storage Class as the default one.
- Existing clusters don't get their default Storage Classes changed.
tldr: three basic claims, the rest is explanation and one example
- We cannot improve long term maintainability solely by fixing bugs.
- Teams should be asked to produce designs for improving maintainability/debugability.
- Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.
While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.
One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.
I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.
We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.
Relevant links:
- Documentation:
- Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
- Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
- Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
- RFC: A proper feedback loop on Alerts.
- OpenShift Router Reload Technical Overview on Access.
- Performance Scaling HAProxy with OpenShift on Access.
- How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
- OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
- Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
- Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
- Timing web requests with cURL and Chrome from the Cloudflare blog.
- tcpdump advanced filters, some useful tcpdump commands.
- OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
- Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
- Observability tips for HAProxy, a slide deck by Willy Tarreau.
- Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
- The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
- Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
- Troubleshooting throughput issues from the OCP 4.5 documentation.
- Troubleshooting OpenShift Clusters and Workloads.
- Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
- openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
- Diagnostic tools:
- dropwatch to watch for packet drops.
- ethtool to check NIC configuration.
- iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
- r-curler to gather timing information about HTTP/HTTPS connections.
- route-monitor, to monitor routes for reachability.
- hping(3), a programmable packet generator.
- OpenTracing / Jaeger in OpenShift.
- node-problem-detector, a possible integration point for new diagnostics.
- Using SystemTap by Brendan Gregg.
- DTrace SystemTap cheatsheet (PDF).
- Visualization and more sophisticated diagnostic tools:
- eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
- ironcladlou/ditm, Dan's "Dan in the Middle" tool.
- Skydive, network diagnostic and visualization tool.
- ali, a "load testing tool capable of performing real-time analysis" with visualization.
- Testing tools:
- Case studies:
- BZ1763206 is an example of diagnosing DNS latency/timeouts.
- BZ1829779 Investigation details the diagnosis of route latency.
- BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
- Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.
Epic Goal
- Change the default value for the spec.tuningOptions.maxConnections field in the IngressController API, which configures the HAProxy maxconn setting, to 50000 (fifty thousand).
Why is this important?
- The maxconn setting constrains the number of simultaneous connections that HAProxy accepts. Beyond this limit, the kernel queues incoming connections.
- Increasing maxconn enables HAProxy to queue incoming connections intelligently. In particular, this enables HAProxy to respond to health probes promptly while queueing other connections as needed.
- The default setting of 20000 has been in place since OpenShift 3.5 was released in April 2017 (see BZ#1405440, commit, RHBA-2017:0884).
- Hardware capabilities have increased over time, and the current default is too low for typical modern machine sizes.
- Increasing the default setting improves HAProxy's performance at an acceptable cost in the common case.
Scenarios
- As a cluster administrator who is installing OpenShift on typical hardware, I want OpenShift router to be tuned appropriately to take advantage of my hardware's capabilities.
Acceptance Criteria
- CI is passing.
- The new default setting is clearly documented.
- A release note informs cluster administrators of the change to the default setting.
Dependencies (internal and external)
- None.
Previous Work (Optional):
- The haproxy-max-connections-tuning enhancement made maxconn configurable without changing the default. The enhancement document details the tradeoffs in terms of memory for various settings of nbthreads and maxconn with various numbers of routes.
Open questions::
- ...
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
- bump current value from 20000 to 50000 in openshift/router
- update documentation to reflect new default in: https://github.com/openshift/api/blob/1265e99256880f8679d1b74561c0bc7932067c43/operator/v1/types_ingress.go#L1475-L1483
- bump cluster-ingress-operator to pickup openshift/api documentation change.
Epic Template descriptions and documentation.
Epic Goal
- Add an E2E test to avoid regressions such as https://issues.redhat.com/browse/OCPBUGS-488 in which we test DNS local endpoint preference.
Why is this important?
- This regression is a major performance and stability issue and it has happened once before.
Drawbacks
The E2E test may be complex due to trying to determine what DNS pods are responding to DNS requests.This is straightforward using the chaos plugin.
Scenarios
- CI Testing
Acceptance Criteria
- CI - MUST be running successfully with tests automated
Dependencies (internal and external)
- SDN Team
Previous Work (Optional):
- N/A
Open questions::
- Where do these E2E test go? SDN Repo? DNS Repo?
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub
Issue> - DEV - Upstream documentation merged: <link to meaningful PR or GitHub
Issue> - DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Enable the chaos plugin https://coredns.io/plugins/chaos/ in our CoreDNS configuration so that we can use a DNS query to easily identify what DNS pods are responding to our requests.
Feature Overview
- This Section:* High-Level description of the feature ie: Executive Summary
- Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.
Goals
- This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature
- …
Requirements
- This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.
| Requirement | Notes | isMvp? |
|---|
| CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
| Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
(Optional) Use Cases
This Section:
- Main success scenarios - high-level user stories
- Alternate flow/scenarios - high-level user stories
- ...
Questions to answer…
- ...
Out of Scope
- …
Background, and strategic fit
This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.
Assumptions
- ...
Customer Considerations
- ...
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
As a console user I want to have option to:
- Restart Deployment
- Retry latest DeploymentConfig if it failed
For Deployments we will add the 'Restart rollout' action button. This action will PATCH the Deployment object's 'spec.template.metadata.annotations' block, by adding 'openshift.io/restartedAt: <actual-timestamp>' annotation. This will restart the deployment, by creating a new ReplicaSet.
- action is disabled if:
- Deployment is paused
For DeploymentConfig we will add 'Retry rollout' action button. This action will PATCH the latest revision of ReplicationController object's 'metadata.annotations' block by setting 'openshift.io/deployment/phase: "New"' and removing openshift.io/deployment.cancelled and openshift.io/deployment.status-reason.
- action is enabled if:
- latest revision of the ReplicationController resource is in Failed phase
- action is disabled if:
- latest revision of the ReplicationController resource is in Complete phase
- DeploymentConfig does not have any rollouts
- DeploymentConfigs is paused
Acceptance Criteria:
- Add the 'Restart rollout' action button for the Deployment resource to both action menu and kebab menu
- Add the 'Retry rollout' action button for the DeploymentConfig resource to both action menu and kebab menu
BACKGROUND:
OpenShift console will be updated to allow rollout restart deployment from the console itself.
Currently, from the OpenShift console, for the resource “deploymentconfigs” we can only start and pause the rollout, and for the resource “deployment” we can only resume the rollout. None of the resources (deployment & deployment config) has this option to restart the rollout. So, that is the reason why the customer wants this functionality to perform the same action from the CLI as well as the OpenShift console.
The customer wants developers who are not fluent with the oc tool and terminal utilities, can use the console instead of the terminal to restart deployment, just like we use to do it through CLI using the command “oc rollout restart deploy/<deployment-name>“.
Usually when developers change the config map that deployment uses they have to restart pods. Currently, the developers have to use the oc rollout restart deployment command. The customer wants the functionality to get this button/menu to perform the same action from the console as well.
Design
Doc: https://docs.google.com/document/d/1i-jGtQGaA0OI4CYh8DH5BBIVbocIu_dxNt3vwWmPZdw/edit
As a developer, I want to make status.HostIP for Pods visible in the Pod details page of the OCP Web Console. Currently there is no way to view the node IP for a Pod in the OpenShift Web Console. When viewing a Pod in the console, the field status.HostIP is not visible.
Acceptance criteria:
- Make pod's HostIP field visible in the pod details page, similarly to PodIP field
When OCP is performing cluster upgrade user should be notified about this fact.
There are two possibilities how to surface the cluster upgrade to the users:
- Display a console notification throughout OCP web UI saying that the cluster is currently under upgrade.
- Global notification throughout OCP web UI saying that the cluster is currently under upgrade.
- Have an alert firing for all the users of OCP stating the cluster is undergoing an upgrade.
AC:
- Console-operator will create a ConsoleNotification CR when the cluster is being upgraded. Once the upgrade is done console-operator will remote that CR. These are the three statuses based on which we are determining if the cluster is being upgraded.
- Add unit tests
Note: We need to decide if we want to distinguish this particular notification by a different color? ccing Ali Mobrem
Created from: https://issues.redhat.com/browse/RFE-3024
Feature Overview
- As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.
- As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.
Goals
- Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings
Requirements
4.11 MVP Requirements
- Customers and partners needs to be able to download the installer
- Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
- Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
- Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
- In the MVP automation has higher priority than interactive, user-driven deployments.
- For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
- Installer should prioritize support for platforms None, baremetal, and VMware.
- The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
- The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.
Use Cases
- As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
- As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
- As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
- As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
- As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.
Questions to answer…
Out of Scope
Out of scope use cases (that are part of the Kubeframe/factory project):
- As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
- As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.
Background, and strategic fit
- This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.
Assumptions
- The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
- The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
- We cannot assume that we will have access to a computer to run an installer or installer helper software.
Customer Considerations
- ...
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
References
Epic Goal
As a OpenShift infrastructure owner, I want to deploy OpenShift clusters with dual-stack IPv4/IPv6
As a OpenShift infrastructure owner, I want to deploy OpenShift clusters with single-stack IPv6
Why is this important?
IPv6 and dual-stack clusters are requested often by customers, especially from Telco customers. Working with dual-stack clusters is a requirement for many but also a transition into a single-stack IPv6 clusters, which for some of our users is the final destination.
Acceptance Criteria
- Agent-based installer can deploy IPv6 clusters
- Agent-based installer can deploy dual-stack clusters
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
Previous Work
Karim's work proving how agent-based can deploy IPv6: IPv6 deploy with agent based installer]
Done Checklist * CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>|
For dual-stack installations the agent-cluster-install.yaml must have both an IPv4 and IPv6 subnet in the networkking.MachineNetwork or assisted-service will throw an error. This field is in InstallConfig but it must be added to agent-cluster-install in its Generate().
For IPv4 and IPv6 installs, setting up the MachineNetwork is not needed but it also does not cause problems if its set, so it should be fine to set it all times.
Epic Goal
As an OpenShift infrastructure owner, I want to deploy a cluster zero with RHACM or MCE and have the required components installed when the installation is completed
Why is this important?
BILLI makes it easier to deploy a cluster zero. BILLI users know at installation time what the purpose of their cluster is when they plan the installation. Day-2 steps are necessary to install operators and users, especially when automating installations, want to finish the installation flow when their required components are installed.
Acceptance Criteria
- A user can provide MCE manifests and have it installed without additional manual steps after the installation is completed
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
User Story:
As a customer, I want to be able to:
- Install MCE with the agent-installer
so that I can achieve
- create an MCE hub with my openshift install
Acceptance Criteria:
Description of criteria:
- Upstream documentation including examples of the extra manifests needed
- Unit tests that include MCE extra manifests
- Ability to install MCE using agent-installer is tested
- Point 3
(optional) Out of Scope:
We are only allowing the user to provide extra manifests to install MCE at this time. We are not adding an option to "install mce" on the command line (or UI)
Engineering Details:
- (optional) https://github/com/link.to.enhancement/
- (optional) https://issues.redhat.com/link.to.spike
- Engineering detail 1
- Engineering detail 2
This requires/does not require a design proposal.
This requires/does not require a feature gate.
User Story:
As a customer, I want to be able to:
- Install MCE with the agent-installer
so that I can achieve
- create an MCE hub with my openshift install
Acceptance Criteria:
Description of criteria:
- Upstream documentation including examples of the extra manifests needed
- Unit tests that include MCE extra manifests
- Ability to install MCE using agent-installer is tested
- Point 3
(optional) Out of Scope:
We are only allowing the user to provide extra manifests to install MCE at this time. We are not adding an option to "install mce" on the command line (or UI)
Engineering Details:
- (optional) https://github/com/link.to.enhancement/
- (optional) https://issues.redhat.com/link.to.spike
- Engineering detail 1
- Engineering detail 2
This requires/does not require a design proposal.
This requires/does not require a feature gate.
Set the ClusterDeployment CRD to deploy OpenShift in FIPS mode and make sure that after deployment the cluster is set in that mode
In order to install FIPS compliant clusters, we need to make sure that installconfig + agentoconfig based deployments take into account the FIPS config in installconfig.
This task is about passing the config to agentclusterinstall so it makes it into the iso. Once there, AGENT-374 will give it to assisted service
Feature Overview
Add GA support for deploying OpenShift to IBM Public Cloud
Goals
Complete the existing gaps to make OpenShift on IBM Cloud VPC (Next Gen2) General Available
Requirements
- OpenShift can be deployed into an existing network provided by the user (https://issues.redhat.com/browse/CORS-2263)
- OpenShift can be deployed with both Public and Private publish options (https://issues.redhat.com/browse/SPLAT-731)
Optional requirements
- OpenShift can be deployed using Mint mode and STS for cloud provider credentials (future release, tbd)
- OpenShift can be deployed in disconnected mode https://issues.redhat.com/browse/SPLAT-737)
- OpenShift on IBM Cloud supports User Provisioned Infrastructure (UPI) deployment method (future release, 4.14?)
This epic tracks the changes needed to the ingress operator to support IBM DNS Services for private clusters.
Epic Goal
- Enable installation of private clusters on IBM Cloud. This epic will track associated work.
Why is this important?
- This is required MVP functionality to achieve GA.
Scenarios
- Install a private cluster on IBM Cloud.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
- Release Technical Enablement - Provide necessary release enablement details and documents.
Dependencies (internal and external)
- ...
Previous Work (Optional):
- …
Open questions::
- …
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
Background and Goal
Currently in OpenShift we do not support distributing hotfix packages to cluster nodes. In time-sensitive situations, a RHEL hotfix package can be the quickest route to resolving an issue.
Acceptance Criteria
- Under guidance from Red Hat CEE, customers can deploy RHEL hotfix packages to MachineConfigPools.
- Customers can easily remove the hotfix when the underlying RHCOS image incorporates the fix.
Before we ship OCP CoreOS layering in https://issues.redhat.com/browse/MCO-165 we need to switch the format of what is currently `machine-os-content` to be the new base image.
The overall plan is:
- Publish the new base image as `rhel-coreos-8` in the release image
- Also publish the new extensions container (https://github.com/openshift/os/pull/763) as `rhel-coreos-8-extensions`
- Teach the MCO to use this without also involving layering/build controller
- Delete old `machine-os-content`
After https://github.com/openshift/os/pull/763 is in the release image, teach the MCO how to use it. This is basically:
- Schedule the extensions container as a kubernetes service (just serves a yum repo via http)
- Change the MCD to write a file into `/etc/yum.repos.d/machine-config-extensions.repo` that consumes it instead of what it does now in pulling RPMs from the mounted container filesystem
As a OCP CoreOS layering developer, having telemetry data about number of cluster using osImageURL will help understand how broadly this feature is getting used and improve accordingly.
Acceptance Criteria:
- Cluster using Custom osImageURL is available via telemetry
Why?
- Decouple control and data plane.
- Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
- Improve security
- Shift credentials out of cluster that support the operation of core platform vs workload
- Improve cost
- Allow a user to toggle what they don’t need.
- Ensure a smooth path to scale to 0 workers and upgrade with 0 workers.
Assumption
- A customer will be able to associate a cluster as “Infrastructure only”
- E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
- OR the entire cluster is labeled infrastructure , and node roles are ignored.
- Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.
Doc: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit
Overview
Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
Assumption
- A customer will be able to associate a cluster as “Infrastructure only”
- E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
- OR the entire cluster is labeled infrastructure, and node roles are ignored.
- Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.
DoD
Run cluster-storage-operator (CSO) + AWS EBS CSI driver operator + AWS EBS CSI driver control-plane Pods in the management cluster, run the driver DaemonSet in the hosted cluster.
More information here: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit
As OCP support engineer I want the same guest cluster storage-related objects in output of "hypershift dump cluster --dump-guest-cluster" as in "oc adm must-gather ", so I can debug storage issues easily.
must-gather collects: storageclasses persistentvolumes volumeattachments csidrivers csinodes volumesnapshotclasses volumesnapshotcontents
hypershift collects none of this, the relevant code is here: https://github.com/openshift/hypershift/blob/bcfade6676f3c344b48144de9e7a36f9b40d3330/cmd/cluster/core/dump.go#L276
Exit criteria:
- verify that hypershift dump cluster --dump-guest-cluster has storage objects from the guest cluster.
As HyperShift Cluster Instance Admin, I want to run AWS EBS CSI driver operator + control plane of the CSI driver in the management cluster, so the guest cluster runs just my applications.
- Add a new cmdline option for the guest cluster kubeconfig file location
- Parse both kubeconfigs:
- One from projected service account, which leads to the management cluster.
- Second from the new cmdline option introduced above. This one leads to the guest cluster.
- Only on HyperShift:
-
- When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
-
- Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
-
- Pass only the guest kubeconfig to the operand (control-plane Deployment of the CSI driver).
Exit criteria:
- Control plane Deployment of AWS EBS CSI driver runs in the management cluster in HyperShift.
- Storage works in the guest cluster.
- No regressions in standalone OCP.
As HyperShift Cluster Instance Admin, I want to run cluster-storage-operator (CSO) in the management cluster, so the guest cluster runs just my applications.
- Add a new cmdline option for the guest cluster kubeconfig file location
- Parse both kubeconfigs:
- One from projected service account, which leads to the management cluster.
- Second from the new cmdline option introduced above. This one leads to the guest cluster.
- Tag manifests of objects that should not be deployed by CVO in HyperShift
- Only on HyperShift:
-
- When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
-
- Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
-
- Pass only the guest kubeconfig to the operands (AWS EBS CSI driver operator).
Exit criteria:
- CSO and AWS EBS CSI driver operator runs in the management cluster in HyperShift
- Storage works in the guest cluster.
- No regressions in standalone OCP.
Epic Goal
- To improve debug-ability of ovn-k in hypershift
- To verify the stability of of ovn-k in hypershift
- To introduce a EgressIP reach-ability check that will work in hypershift
Why is this important?
- ovn-k is supposed to be GA in 4.12. We need to make sure it is stable, we know the limitations and we are able to debug it similar to the self hosted cluster.
Acceptance Criteria
- CI - MUST be running successfully with tests automated
Dependencies (internal and external)
- This will need consultation with the people working on HyperShift
Previous Work (Optional):
Done Checklist
- CI - CI is running, tests are automated and merged.
- Release Enablement <link to Feature Enablement Presentation>
- DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
- DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
- DEV - Downstream build attached to advisory: <link to errata>
- QE - Test plans in Polarion: <link or reference to Polarion>
- QE - Automated tests merged: <link or reference to automated tests>
- DOC - Downstream documentation merged: <link to meaningful PR>
CNCC was moved to the management cluster and it should use proxy settings defined for the management cluster.
Overview
Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
Assumption
- A customer will be able to associate a cluster as “Infrastructure only”
- E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
- OR the entire cluster is labeled infrastructure, and node roles are ignored.
- Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.
DoD
cluster-snapshot-controller-operator is running on the CP.
More information here: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit
As OpenShift developer I want cluster-csi-snapshot-controller-operator to use existing controllers in library-go, so I don’t need to maintain yet another code that does the same thing as library-go.
- Check and remove manifests/03_configmap.yaml, it does not seem to be useful.
- Check and remove manifests/03_service.yaml, it does not seem to be useful (at least now).
- Use DeploymentController from library-go to sync Deployments.
- Get rid of common/ package? It does not seem to be useful.
- Use StaticResourceController for static content, including the snapshot CRDs.
Note: if this refactoring introduces any new conditions, we must make sure that 4.11 snapshot controller clears them to support downgrade! This will need 4.11 BZ + z-stream update!
Similarly, if some conditions become obsolete / not managed by any controller, they must be cleared by 4.12 operator.
Exit criteria:
- The operator code is smaller.
- No regressions in standalone OCP.
- Upgrade/downgrade from/to standalone OCP 4.11 works.
As HyperShift Cluster Instance Admin, I want to run cluster-csi-snapshot-controller-operator in the management cluster, so the guest cluster runs just my applications.
- Add a new cmdline option for the guest cluster kubeconfig file location
- Parse both kubeconfigs:
-
- One from projected service account, which leads to the management cluster.
-
- Second from the new cmdline option introduced above. This one leads to the guest cluster.
- Move creation of manifests/08_webhook_service.yaml from CVO to the operator - it needs to be created in the management cluster.
- Tag manifests of objects that should not be deployed by CVO in HyperShift by
- Only on HyperShift:
-
- When interacting with Kubernetes API, carefully choose the right kubeconfig to watch / create / update objects in the right cluster.
-
- Replace namespaces in all Deployments and other objects that are created in the management cluster. They must be created in the same namespace as the operator.
-
- Don’t create operand’s PodDisruptionBudget?
-
- Update ValidationWebhookConfiguration to point directly to URL exposed by manifests/08_webhook_service.yaml instead of a Service. The Service is not available in the guest cluster.
-
- Pass only the guest kubeconfig to the operands (both the webhook and csi-snapshot-controller).
-
- Update unit tests to handle two kube clients.
Exit criteria:
- cluster-csi-snapshot-controller-operator runs in the management cluster in HyperShift
- csi-snapshot-controller runs in the management cluster in HyperShift
- It is possible to take & restore volume snapshot in the guest cluster.
- No regressions in standalone OCP.
OC mirror is GA product as of Openshift 4.11 .
The goal of this feature is to solve any future customer request for new features or capabilities in OC mirror
Epic Goal
- Mirror to mirror operations and custom mirroring flows required by IBM CloudPak catalog management
Why is this important?
- IBM needs additional customization around the actual mirroring of images to enable CloudPaks to fully adopt OLM-style operator packaging and catalog management
- IBM CloudPaks introduce additional compute architectures, increasing the download volume by 2/3rds to day, we need the ability to effectively filter non-required image versions of OLM operator catalogs during filtering for other customers that only require a single or a subset of the available image architectures
- IBM CloudPaks regularly run on older OCP versions like 4.8 which require additional work to be able to read the mirrored catalog produced by oc mirror
Scenarios
- Customers can use the oc utility and delegate the actual image mirror step to another tool
- Customers can mirror between disconnected registries using the oc utility
- The oc utility supports filtering manifest lists in the context of multi-arch images according to the sparse manifest list proposal in the distribution spec
Acceptance Criteria
- Customers can use the oc utility to mirror between two different air-gapped environments
- Customers can specify the desired computer architectures and oc mirror will create sparse manifest lists in the target registry as a result
Dependencies (internal and external)
- Sparse Manifest List Support in Red Hat Quay (PROJQUAY-3114)
Previous Work:
Related Work:
Pre-Work Objectives
Since some of our requirements from the ACM team will not be available for the 4.12 timeframe, the team should work on anything we can get done in the scope of the console repo so that when the required items are available in 4.13, we can be more nimble in delivering GA content for the Unified Console Epic.
Overall GA Key Objective
Providing our customers with a single simplified User Experience(Hybrid Cloud Console)that is extensible, can run locally or in the cloud, and is capable of managing the fleet to deep diving into a single cluster.
Why customers want this?
- Single interface to accomplish their tasks
- Consistent UX and patterns
- Easily accessible: One URL, one set of credentials
Why we want this?
- Shared code - improve the velocity of both teams and most importantly ensure consistency of the experience at the code level
- Pre-built PF4 components
- Accessibility & i18n
- Remove barriers for enabling ACM
Phase 2 Goal: Productization of the united Console
- Enable user to quickly change context from fleet view to single cluster view
- Add Cluster selector with “All Cluster” Option. “All Cluster” = ACM
- Shared SSO across the fleet
- Hub OCP Console can connect to remote clusters API
- When ACM Installed the user starts from the fleet overview aka “All Clusters”
- Share UX between views
- ACM Search —> resource list across fleet -> resource details that are consistent with single cluster details view
- Add Cluster List to OCP —> Create Cluster
As a developer I would like to disable clusters like *KS that we can't support for multi-cluster (for instance because we can't authenticate). The ManagedCluster resource has a vendor label that we can use to know if the cluster is supported.
cc Ali Mobrem Sho Weimer Jakub Hadvig
UPDATE: 9/20/22 : we want an allow-list with OpenShift, ROSA, ARO, ROKS, and OpenShiftDedicated
Acceptance criteria:
- Investigate if console-operator should pass info about which cluster are supported and unsupported to the frontend
- Unsupported clusters should not appear in the cluster dropdown
- Unsupported clusters based off
- defined vendor label
- non 4.x ocp clusters
Feature Overview
RHEL CoreOS should be updated to RHEL 9.2 sources to take advantage of newer features, hardware support, and performance improvements.
Requirements
- RHEL 9.x sources for RHCOS builds starting with OCP 4.13 and RHEL 9.2.
| Requirement | Notes | isMvp? |
|---|---|---|
| CI - MUST be running successfully with test automation | This is a requirement for ALL features. | YES |
| Release Technical Enablement | Provide necessary release enablement details and documents. | YES |
(Optional) Use Cases
9.2 Preview via LayeringNo longer necessary assuming we stay the course of going all in on 9.2
Assumptions
- ...
Customer Considerations
- ...
Documentation Considerations
Questions to be addressed:
- What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
- Does this feature have doc impact?
- New Content, Updates to existing content, Release Note, or No Doc Impact
- If unsure and no Technical Writer is available, please contact Content Strategy.
- What concepts do customers need to understand to be successful in [action]?
- How do we expect customers will use the feature? For what purpose(s)?
- What reference material might a customer want/need to complete [action]?
- Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
- What is the doc impact (New Content, Updates to existing content, or Release Note)?
PROBLEM
We would like to improve our signal for RHEL9 readiness by increasing internal engineering engagement and external partner engagement on our community OpehShift offering, OKD.
PROPOSAL
Adding OKD to run on SCOS (a CentOS stream for CoreOS) brings the community offering closer to what a partner or an internal engineering team might expect on OCP.
ACCEPTANCE CRITERIA
Image has been switched/included:
DEPENDENCIES
The SCOS build payload.
RELATED RESOURCES
OKD+SCOS proposal: https://docs.google.com/presentation/d/1_Xa9Z4tSqB7U2No7WA0KXb3lDIngNaQpS504ZLrCmg8/edit#slide=id.p
OKD+SCOS work draft: https://docs.google.com/document/d/1cuWOXhATexNLWGKLjaOcVF4V95JJjP1E3UmQ2kDVzsA/edit
Acceptance Criteria
A stable OKD on SCOS is built and available to the community sprintly.
This comes up when installing ipi-on-aws on arm64 with the custom payload build at quay.io/aleskandrox/okd-release:4.12.0-0.okd-centos9-full-rebuild-arm64 that is using scos as machine-content-os image
```
[root@ip-10-0-135-176 core]# crictl logs c483c92e118d8
2022-08-11T12:19:39+00:00 [cnibincopy] FATAL ERROR: Unsupported OS ID=scos
```
The probable fix has to land on https://github.com/openshift/cluster-network-operator/blob/master/bindata/network/multus/multus.yaml#L41-L53
Feature Overview (aka. Goal Summary)
The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike.
Some customer cases have revealed scenarios where the MCO state reporting is misleading and therefore could be unreliable to base decisions and automation on.
In addition to correcting some incorrect states, the MCO will be enhanced for a more granular view of update rollouts across machines.
The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike.
For this epic, "state" means "what is the MCO doing?" – so the goal here is to try to make sure that it's always known what the MCO is doing.
This includes:
- Conditions
- Some Logging
- Possibly Some Events
While this probably crosses a little bit into the "status" portion of certain MCO objects, as some state is definitely recorded there, this probably shouldn't turn into a "better status reporting" epic. I'm interpreting "status" to mean "how is it going" so status is maybe a "detail attached to a state".
Exploration here: https://docs.google.com/document/d/1j6Qea98aVP12kzmPbR_3Y-3-meJQBf0_K6HxZOkzbNk/edit?usp=sharing
https://docs.google.com/document/d/17qYml7CETIaDmcEO-6OGQGNO0d7HtfyU7W4OMA6kTeM/edit?usp=sharing
The current property description is:
configuration represents the current MachineConfig object for the machine config pool.
But in a 4.12.0-ec.4 cluster, the actual semantics seem to be something closer to "the most recent rendered config that we completely leveled on". We should at least update the godocs to be more specific about the intended semantics. And perhaps consider adjusting the semantics?
Verified
Unsatisfied