Weekly gate dashboard
2026-10-19 → 2026-10-26 · master · failure statistics: gate · flaky analysis: gate & check
1. Gate job failures
Gate only. Sorted by failure count, then failure rate. Failures = FAILURE + TIMED_OUT. Runs = SUCCESS + FAILURE + TIMED_OUT. Every attempt counts.
| Job | Pipeline | Failures / runs | Failure rate | Other results |
|---|---|---|---|---|
| nova-tox-functional-py314-threading | gate | 1 / 10 | 10.0% | 4 |
| build-openstack-api-guide | gate | 0 / 1 | 0.0% | 0 |
| build-openstack-api-ref | gate | 0 / 1 | 0.0% | 0 |
| build-openstack-releasenotes | gate | 0 / 13 | 0.0% | 1 |
| grenade-skip-level-always | gate | 0 / 8 | 0.0% | 1 |
| nova-alt-configurations | gate | 0 / 8 | 0.0% | 1 |
| nova-ceph-multistore | gate | 0 / 8 | 0.0% | 1 |
| nova-grenade-multinode | gate | 0 / 8 | 0.0% | 2 |
| nova-live-migration | gate | 0 / 8 | 0.0% | 1 |
| nova-multi-cell | gate | 0 / 8 | 0.0% | 1 |
| nova-next | gate | 0 / 8 | 0.0% | 1 |
| nova-tox-functional-py311 | gate | 0 / 12 | 0.0% | 1 |
| nova-tox-functional-py314 | gate | 0 / 12 | 0.0% | 1 |
| nova-tox-py314-threading | gate | 0 / 13 | 0.0% | 0 |
| nova-tox-validate-backport | gate | 0 / 13 | 0.0% | 0 |
| openstack-tox-docs | gate | 0 / 13 | 0.0% | 0 |
| openstack-tox-pep8 | gate | 0 / 13 | 0.0% | 0 |
| openstack-tox-py311 | gate | 0 / 13 | 0.0% | 0 |
| openstack-tox-py314 | gate | 0 / 13 | 0.0% | 2 |
| openstacksdk-functional-devstack | gate | 0 / 8 | 0.0% | 2 |
| requirements-check | gate | 0 / 4 | 0.0% | 0 |
| tempest-integrated-compute | gate | 0 / 8 | 0.0% | 2 |
| tempest-ipv6-only | gate | 0 / 8 | 0.0% | 1 |
| test-release-openstack | gate | 0 / 3 | 0.0% | 0 |
2. Top 10 failing gate tests
Gate only. One occurrence per test and failed build, across all gate jobs. Error text describes the observed failure, not necessarily its root cause.
| Test | Occurrences | Failed builds | Jobs | Observed errors | Evidence |
|---|---|---|---|---|---|
| nova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestVmStatus.test_queued_live_migration_abort_vm_status | 1 | 1 | nova-tox-functional-py314-threading | RuntimeError: the executors still have tasks that did not finish in 10s1 observation · View 1 error detail
| build 6dff84ad |
Top 3 investigations — gate
Only 1 ranked test(s) available; no additional incidents are invented.
1. Live-migration worker is still waiting for a scheduler RPC reply during test teardownnova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestVmStatus.test_queued_live_migration_abort_vm_statusHigh confidence in the blocking location; teardown race remains a hypothesis
Observed in this build: The gate build failed during fixture cleanup, not at the VM-state assertions. At 16:01:34 UTC, cleanup reported 10.02 seconds elapsed, a live_migration executor with _do_live_migration still queued, and worker Thread-3970 still running. Its captured stack follows _post_live_migration → _update_scheduler_instance_info → scheduler RPC → oslo.messaging fake transport → reply_q.get()/threading.Condition.wait(). The same test explicitly passed after retry on patchset 7.
What the source explains: The test deliberately holds one migration and queues a second, cancels the queued migration, releases the lock, and waits for both VMs to become ACTIVE. That state check is not an explicit join of every background migration callback. The fixture added by queued dependency 1006351/8 waits at most 10 seconds for unfinished executor work and raises the exact RuntimeError seen here. The stack identifies a pending post-migration scheduler update, rather than proving the migration-state assertion is wrong.
Working hypothesis: A teardown ordering race lets service/RPC cleanup overlap a migration callback that has not finished its scheduler update. The test can observe ACTIVE while background work is still in progress. A blocked fake-RPC reply can then outlive the fixture deadline. This is more specific than attributing the failure to generic CI slowness.
What remains unproven: The captured stack proves where the worker was blocked, but does not by itself prove why the scheduler reply was absent. The test was not reproduced locally, and the complete speculative merge tree was not reconstructed. A successful retry establishes observed intermittency, not a production live-migration defect or a verified fix.
Source revision: Test source was inspected at target patchset 7 (a099e5e99faf71552016036e3a3d54b20bbf7f88). Cleanup source was inspected at queued dependency patchset 8 (7f9dbeb5e408489daaf7205a58c09ee9d82d85bb), explicitly listed in workspace-repos.json. These identify relevant inputs, not the full speculative merge commit.
Next checks
- Check scheduler/service shutdown timestamps against the blocked RPC and inspect whether the scheduler consumer was already stopped.
- Reproduce under functional-py314-threading and explicitly wait for migration completion/background callbacks before service fixtures shut down; retain worker and queued-task dumps.
- Prefer fixing synchronization or cleanup order over merely increasing the 10-second timeout.
Evidence and source: Failing gate build · Passing retry · Worker stack and cleanup log · Test source · Executor cleanup source
3. Flaky jobs and tests
Failure(s) followed by success on the same change, patchset, job and pipeline, with Gerrit retry evidence. A test is confirmed only when its individual result is explicitly successful in the passing run.
Gate flaky jobs
| Job | Resolved sequences | Failed attempts |
|---|---|---|
| nova-tox-functional-py314-threading | 1 | 1 |
Top 10 confirmed flaky tests — gate
| Test | Sequences | Failed builds | Jobs | Observed errors | Failure / success evidence |
|---|---|---|---|---|---|
| nova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestVmStatus.test_queued_live_migration_abort_vm_status | 1 | 1 | nova-tox-functional-py314-threading | RuntimeError: the executors still have tasks that did not finish in 10s1 observation · View 1 error detail
| build 57ef4479 · build 6dff84ad |
Top 3 investigations — gate
Only 1 ranked test(s) available; no additional incidents are invented.
1. Live-migration worker is still waiting for a scheduler RPC reply during test teardownnova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestVmStatus.test_queued_live_migration_abort_vm_statusHigh confidence in the blocking location; teardown race remains a hypothesis
Observed in this build: The gate build failed during fixture cleanup, not at the VM-state assertions. At 16:01:34 UTC, cleanup reported 10.02 seconds elapsed, a live_migration executor with _do_live_migration still queued, and worker Thread-3970 still running. Its captured stack follows _post_live_migration → _update_scheduler_instance_info → scheduler RPC → oslo.messaging fake transport → reply_q.get()/threading.Condition.wait(). The same test explicitly passed after retry on patchset 7.
What the source explains: The test deliberately holds one migration and queues a second, cancels the queued migration, releases the lock, and waits for both VMs to become ACTIVE. That state check is not an explicit join of every background migration callback. The fixture added by queued dependency 1006351/8 waits at most 10 seconds for unfinished executor work and raises the exact RuntimeError seen here. The stack identifies a pending post-migration scheduler update, rather than proving the migration-state assertion is wrong.
Working hypothesis: A teardown ordering race lets service/RPC cleanup overlap a migration callback that has not finished its scheduler update. The test can observe ACTIVE while background work is still in progress. A blocked fake-RPC reply can then outlive the fixture deadline. This is more specific than attributing the failure to generic CI slowness.
What remains unproven: The captured stack proves where the worker was blocked, but does not by itself prove why the scheduler reply was absent. The test was not reproduced locally, and the complete speculative merge tree was not reconstructed. A successful retry establishes observed intermittency, not a production live-migration defect or a verified fix.
Source revision: Test source was inspected at target patchset 7 (a099e5e99faf71552016036e3a3d54b20bbf7f88). Cleanup source was inspected at queued dependency patchset 8 (7f9dbeb5e408489daaf7205a58c09ee9d82d85bb), explicitly listed in workspace-repos.json. These identify relevant inputs, not the full speculative merge commit.
Next checks
- Check scheduler/service shutdown timestamps against the blocked RPC and inspect whether the scheduler consumer was already stopped.
- Reproduce under functional-py314-threading and explicitly wait for migration completion/background callbacks before service fixtures shut down; retain worker and queued-task dumps.
- Prefer fixing synchronization or cleanup order over merely increasing the 10-second timeout.
Evidence and source: Failing gate build · Passing retry · Worker stack and cleanup log · Test source · Executor cleanup source
Check flaky jobs
| Job | Resolved sequences | Failed attempts |
|---|---|---|
| nova-multi-cell | 7 | 7 |
| tempest-integrated-compute | 5 | 5 |
| nova-tox-functional-py314-threading | 2 | 2 |
| nova-next | 1 | 1 |
| openstack-tox-cover | 1 | 1 |
Top 10 confirmed flaky tests — check
| Test | Sequences | Failed builds | Jobs | Observed errors | Failure / success evidence |
|---|---|---|---|---|---|
| tempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_server_with_scheduler_hint_group[id-ed20d3fb-9d1f-4329-b160-543fbd5d9811] | 5 | 5 | nova-multi-cell tempest-integrated-compute | Server 0fa3bd31-8939-40c7-a181-0043184f40d9 failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…5 observations · View 5 error details
| build 21f5468d · build 2769c79d · build 2cab2ec5 · build 34ba517f · build 3d84557a · build 423a660d · build 4945530e · build 5d2164b3 · build d81e862e · build d9db2cb2 |
| tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV346.test_instance_boot_after_snapshot_deletion[compute,id-77889046-1a75-4f14-9b3a-fbbfdd8e5093,image,slow,volume] | 5 | 5 | nova-multi-cell | Server e39e303e-aed2-4347-ba61-3e430ec173aa failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…5 observations · View 5 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build 8acbd820 · build 9dcfadde · build b441bce5 · build d81e862e |
| tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port[id-73fe8f02-590d-4bf1-b184-e9ca81065051,network] | 4 | 4 | nova-multi-cell tempest-integrated-compute | Details: Interface 6a8f9390-35c8-41f4-8372-5841c5a803e5 failed to reach ACTIVE status (current DOWN) within the required time (196 s).4 observations · View 4 error details
| build 21f5468d · build 2cab2ec5 · build 34ba517f · build 423a660d · build 4945530e · build 5d2164b3 · build d81e862e · build d9db2cb2 |
| tempest.api.compute.servers.test_servers.ServersTestJSON.test_create_with_existing_server_name[id-8fea6be7-065e-47cf-89b8-496e6f96c699] | 4 | 4 | nova-multi-cell tempest-integrated-compute | Server 3ebc72ed-8d3b-4e9f-9e7e-ce53d39dd9c8 failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 4945530e · build 5d2164b3 · build d81e862e · build d9db2cb2 |
| tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_resize_revert[compute,id-d653af24-fa18-470d-b9bf-af287cefeba9,image,slow,volume] | 4 | 4 | nova-multi-cell | Server 9193f4b2-3f3c-46c9-acbb-cdc1e0b3778a failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build b441bce5 · build d81e862e |
| tempest.scenario.test_instances_with_cinder_volumes.TestInstancesWithCinderVolumes.test_instances_with_cinder_volumes_on_all_compute_nodes[compute,id-d0e3c1a3-4b0a-4b0e-8b0a-4b0e8b0a4b0e,image,multinode,network,slow,volume] | 4 | 4 | nova-multi-cell | Details: Fault: {'code': 500, 'created': '2026-10-23T13:28:23Z', 'message': 'Build of instance 669c8370-3c06-460b-b693-fef478dc0fcc aborted: Failed to alloca…4 observations · View 4 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build b441bce5 · build d81e862e |
| tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_port_security_disable_security_group[compute,id-7c811dcc-263b-49a3-92d2-1b4d8405f50c,multinode,network,slow] | 4 | 4 | nova-multi-cell | Server 9cf95978-6d83-4230-b032-8acd06d662da failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build b441bce5 · build d81e862e |
| tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern[compute,id-608e604b-1d63-4a82-8e3e-91bc665c90b4,image,network,slow] | 4 | 4 | nova-multi-cell | Server 110d5854-f06c-47d1-a36b-5f0d93d49c96 failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build b441bce5 · build d81e862e |
| tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_boot_server_from_encrypted_volume_luks[compute,id-cb78919a-e553-4bab-b73b-10cf4d2eb125,volume] | 4 | 4 | nova-multi-cell | Server afd226ef-3bb7-4aa9-b47f-2621efa35a38 failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 14a8987c · build 21f5468d · build 2769c79d · build 34ba517f · build 3d84557a · build 5d2164b3 · build b441bce5 · build d81e862e |
| tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_image_defined_boot_from_volume[compute,id-36c34c67-7b54-4b59-b188-02a2f458a63b,image,volume] | 4 | 4 | nova-multi-cell | Server 588e2941-7817-42f0-8f71-f21327cb4c1a failed to reach ACTIVE status and task state "None" within the required time (196 s). Request ID of server operat…4 observations · View 4 error details
| build 14a8987c · build 2769c79d · build 3d84557a · build 5d2164b3 · build 73ac9727 · build b441bce5 · build d81e862e · build e34a4580 |
Top 3 investigations — check
1. Server-group test is blocked by a missing network VIF-plug event during spawntempest.api.compute.servers.test_server_group.ServerGroupTestJSON.test_create_server_with_scheduler_hint_group[id-ed20d3fb-9d1f-4329-b160-543fbd5d9811]High confidence for the representative build; upstream event-loss cause unproven
Observed in this build: Five confirmed flaky sequences have a server stuck in BUILD/spawning at the 196-second Tempest deadline. In representative build 2cab2ec5, instance 0fa3bd31-8939-40c7-a181-0043184f40d9 starts spawning at 03:21:28 UTC. Nova prepares to wait for network-vif-plugged-257a66a8-2319-4246-b83d-d59bd7ea1a3c at 03:21:30. At 03:26:31 it logs a 300-second InstanceEventTimeout, followed by VirtualInterfaceCreateException. Detailed service-log correlation was performed for this representative occurrence; all five ranked failure messages were compared.
What the source explains: The Tempest test creates a server with a group scheduling hint and waits for ACTIVE before checking group membership. The observed failure therefore precedes the membership assertion. Nova libvirt creates the guest paused while waiting for required network-vif-plugged events; when a fatal VIF timeout expires, it cleans up and raises VirtualInterfaceCreateException. Tempest gives up at 196 seconds while Nova is still inside the longer 300-second event wait.
Working hypothesis: Investigate the Neutron/OVN port-status notification path to Nova. The confirmed immediate blocker is the missing VIF-plug event for this instance. Neither affinity scheduling nor slow guest application boot is established as the cause by these traces.
What remains unproven: The inspected Nova logs locate the timeout but do not determine whether the event was never emitted, delayed, routed incorrectly or lost. One service-level occurrence does not prove that all five sequences share the same lower-level fault. The raw excerpts are bounded searches; they are not represented as exhaustive copies of all service logs.
Source revision: Tempest b098f924927a1a08d32dde6044858957ef7154d5 and Nova baseline 0e701bfd411ec8d4c15419710c4968a582abedc7 were recorded in the representative build workspace metadata. Nova also includes the proposed patch; baseline links are not a claim that the entire speculative tree was inspected.
Next checks
- Trace port 257a66a8-2319-4246-b83d-d59bd7ea1a3c through OVN binding, Neutron port-status updates, Nova external-event API delivery and compute event consumption.
- Compare the same path in a passing retry, including vif_plugging_timeout and Tempest build_timeout values.
- Cluster other occurrences by port/event timeout signature before assuming a single server-group regression.
Evidence and source: Representative failed build · Nova compute event timeout · Tempest group test · Nova guest/event handshake · Nova fatal VIF-timeout handling · Example passing retry
2. Volume scenario contains two signatures: missing VIF-plug event and SSH readiness timeouttempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV346.test_instance_boot_after_snapshot_deletion[compute,id-77889046-1a75-4f14-9b3a-fbbfdd8e5093,image,slow,volume]High confidence in the two observed signatures; no single root cause assigned
Observed in this build: Among the five confirmed flaky sequences, four failure messages show BUILD/spawning after 196 seconds; one (9dcfadde) fails later in wait_for_ssh with an empty Details field. In representative multi-cell build 2769c79d, instance e39e303e-aed2-4347-ba61-3e430ec173aa is paused during spawn and waits for network-vif-plugged-9b288dae-14d3-49d2-93b9-088ad011b0e6. At 13:46:25 UTC Nova records the 300-second event timeout and then VirtualInterfaceCreateException. This representative service trace was correlated; the SSH case was distinguished using its full archived exception tail.
What the source explains: The scenario boots instances from volumes at several stages around snapshot creation/deletion. A failure during create_test_server or SSH readiness does not itself prove snapshot deletion corrupted the volume. For the representative BUILD/spawning occurrence, the Nova libvirt network-event handshake explains the delay and eventual spawn error. The separate SSH trace represents a different phase and must not be collapsed into the same diagnosis.
Working hypothesis: The four BUILD/spawning symptoms may share the VIF-notification problem observed in the server-group test. Treat that as a candidate common failure family, not as a proven common root cause. Investigate the SSH occurrence separately for guest readiness, network reachability, security groups and metadata/key setup.
What remains unproven: The exact scenario stage of every occurrence was not reconstructed, and the Neutron-side origin of the missing notification remains unknown. A Cinder/snapshot-specific defect is not demonstrated. The SSH case has not been correlated with guest console or packet-level evidence.
Source revision: The source explanation uses pinned Tempest b098f924927a1a08d32dde6044858957ef7154d5 and Nova 0e701bfd411ec8d4c15419710c4968a582abedc7, verified for the other representative build. The exact source revision of the earlier 2769c79d multi-cell build was not established here; this limits source-to-build attribution, while its quoted timeout events come from that build itself.
Next checks
- Separate BUILD/spawning and SSH timeout occurrences in follow-up triage; retain the scenario stage and server UUID for each.
- For e39e303e and port 9b288dae, correlate OVN/Neutron notification delivery with Nova event registration and the passing run.
- For 9dcfadde, inspect the guest console, floating-IP path, security groups and SSH initialization before changing volume/snapshot logic.
Evidence and source: Representative multi-cell failure · Nova compute log · Separate SSH failure · Tempest scenario stages · Nova VIF event wait · Example passing retry
3. OVN reports the hotplugged port up while the API still exposes DOWNtempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port[id-73fe8f02-590d-4bf1-b184-e9ca81065051,network]High confidence in the observed status mismatch; propagation failure is a hypothesis
Observed in this build: Four confirmed flaky sequences time out after 196 seconds waiting for an attached interface to become ACTIVE. In representative build 2cab2ec5, Nova creates port 6a8f9390-35c8-41f4-8372-5841c5a803e5 at 03:00:45 UTC. OVN claims it at 03:00:47.067 and sets it up in Southbound at 03:00:47.085; DHCPACK is logged at 03:00:49.645. Tempest nevertheless reports that same interface as DOWN. Nova later receives its deletion event during cleanup. Detailed OVN/Nova correlation was performed for this one representative occurrence, rather than all four.
What the source explains: The Tempest test attaches an interface by network ID, then polls its API port_state before testing interface-list/show/delete consistency. Nova constructs port_state from the Neutron port status. OVN Southbound up is therefore a different observation from the API status consumed by the waiter; successful local binding alone does not guarantee the Neutron status update and notification have propagated.
Working hypothesis: A status propagation or notification race between OVN and Neutron is a stronger lead here than a port that never physically plugged. The missing API transition may share infrastructure with the VIF-event timeouts in the other two check investigations, but the cross-incident causal connection is not yet proven.
What remains unproven: OVN up and DHCPACK do not prove complete end-to-end connectivity. The Neutron API/database state and northbound notification processing were not traced directly. Do not interpret the absence of a matching plugged event in bounded excerpts as an exhaustive proof of its absence everywhere.
Source revision: Nova baseline 0e701bfd411ec8d4c15419710c4968a582abedc7 and Tempest b098f924927a1a08d32dde6044858957ef7154d5 come from the representative workspace metadata. Proposed-patch changes in the speculative tree were not reconstructed.
Next checks
- Correlate port 6a8f9390 through OVN Southbound, Neutron ML2/OVN status handling, the Neutron API response and Nova external-event delivery.
- Compare timestamps and port bindings against a passing retry; inspect whether status changes after the test deadline or only during cleanup.
- Keep this failure family distinct from affinity assertions and snapshot semantics unless additional evidence connects them.
Evidence and source: Representative failed build · OVN port-up and DHCP timeline · Nova compute port timeline · Tempest interface waiter call · Nova port_state mapping · Example passing retry
1 candidate sequences lack Gerrit retry evidence. 65 failed builds have no qualifying later success in the collected timeline.
Sequence evidence
| Job / pipeline | Change / patchset | Failures → success | Gerrit evidence | Confirmed tests |
|---|---|---|---|---|
| tempest-integrated-compute / check | 1012185 / 1 | 2cab2ec5 → 423a660d | Patch Set 1: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 1: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | tempest.api.compute.admin.test_server_diagnostics.ServerDiagnosticsTest.test_get_server_diagnostics[id-31ff3486-b8a0-4f56-a6c0-aab460531db3]44 confirmed tests · View list
|
| tempest-integrated-compute / check | 1011959 / 2 | d0941e83 → 4804bd67 | Patch Set 2: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 2: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_building_state[id-9e6e0c87-3352-42f7-9faf-5d6210dbd159]1 confirmed test · View list
|
| nova-multi-cell / check | 1009444 / 4 | 2769c79d → 3d84557a | Patch Set 4: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 4: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/5c58136b3fbb40d | tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server[id-aaa6cdf3-55a7-461a-add9-1c8596b9a07c]37 confirmed tests · View list
|
| tempest-integrated-compute / check | 1009444 / 4 | cd086d54 → 758bc4b8 | Patch Set 4: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 4: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/5c58136b3fbb40d | No individual pass proof |
| openstack-tox-cover / check | 1011991 / 2 | 995b3129 → cc789a24 | Patch Set 2: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 2: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/fd1db3cc6d074b6 | nova.tests.unit.db.api.test_migrations.TestMigrationsWalkPostgreSQL.test_single_head_revision1 confirmed test · View list
|
| nova-tox-functional-py314-threading / check | 982871 / 12 | 8b8a9ea8 · eedd1086 → d32a21e1 | Unverified | No individual pass proof |
| tempest-integrated-compute / check | 982871 / 12 | d9db2cb2 → 4945530e | Patch Set 12: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.or · Patch Set 12: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/i | tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port[id-73fe8f02-590d-4bf1-b18…38 confirmed tests · View list
|
| nova-multi-cell / check | 1011606 / 2 | 5d2164b3 → d81e862e | Patch Set 2: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 2: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/0cc78b14a66546f | tempest.api.compute.images.test_images_oneserver_negative.ImagesOneServerNegativeTestJSON.test_delete_image_that_is_not_yet_active[id-0894954d-2db2-4195-a45b…39 confirmed tests · View list
|
| nova-multi-cell / check | 1011450 / 3 | e34a4580 → 73ac9727 | Patch Set 3: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 3: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/cb26bc461b4648c | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_image_defined_boot_from_volume[compute,id-36c34c67-7b54-4b59-b188-02a2f458a63b,image,vol…1 confirmed test · View list
|
| nova-multi-cell / check | 1011470 / 4 | 9dcfadde → 8acbd820 | Patch Set 4: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV346.test_instance_boot_after_snapshot_deletion[compute,id-77889046-1a75-4f14-9b3a-fbbfdd8e509…1 confirmed test · View list
|
| nova-multi-cell / check | 1011470 / 4 | 99b121bf → af53158d | Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in · Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | No individual pass proof |
| nova-tox-functional-py314-threading / check | 1011470 / 4 | 30d93bf2 → 41fafe89 | Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in · Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | nova.tests.functional.test_policy.HostStatusPolicyTestCase.test_post_server_rebuild_host_status_unknown_only1 confirmed test · View list
|
| nova-next / check | 1011470 / 4 | e5a9bda3 → c466e331 | Patch Set 4: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in · Patch Set 4: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/bf1f21f2f207428 | tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic[compute,id-e79f879e-debb-440c-a7e4-efeda05b6848,multinod…1 confirmed test · View list
|
| tempest-integrated-compute / check | 998557 / 7 | 439bf959 → 18cdd256 | Patch Set 7: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 7: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | No individual pass proof |
| nova-multi-cell / check | 998557 / 7 | 14a8987c → b441bce5 | Patch Set 7: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in · Patch Set 7: Verified+1 Build succeeded (check pipeline). https://zuul.opendev.org/t/openstack/buildset/91d658f239b7418 | tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_shelve_shelved_server[id-443e4f9b-e6bf-4389-b601-3a710f15fddd,negative]21 confirmed tests · View list
|
| nova-multi-cell / check | 1011366 / 1 | 34ba517f → 21f5468d | Patch Set 1: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 1: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_fixed_ip[id-d290c06c-f5b3-11e7-8ec8-00…48 confirmed tests · View list
|
| nova-tox-functional-py314-threading / check | 1011450 / 1 | 01652b0f → b98e8bf9 | Patch Set 1: Verified-1 Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org · Patch Set 1: Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/in | nova.tests.functional.test_servers_resource_request.NonAdminServerMoveWithMultiGroupResReqTests.test_live_migrate_with_qos_port1 confirmed test · View list
|
| nova-tox-functional-py314-threading / gate | 998557 / 7 | 6dff84ad → 57ef4479 | Patch Set 7: Verified-2 Build failed (gate pipeline). For information on how to proceed, see https://docs.opendev.org/ · Patch Set 7: Verified+2 Build succeeded (gate pipeline). https://zuul.opendev.org/t/openstack/buildset/710e50a4f45a482e | nova.tests.functional.libvirt.test_live_migration.LiveMigrationQueuedAbortTestVmStatus.test_queued_live_migration_abort_vm_status1 confirmed test · View list
|
Coverage and methodology
Time zone: UTC. Weekly window: [2026-10-18 22:00:00 UTC, 2026-10-25 23:00:00 UTC). Flaky qualification cutoff: 2026-10-26 08:00:00 UTC.
Build inventory: complete. Fully analysed failed gate builds: 1 / 1. Fully analysed failures for flaky analysis (gate + check): 62 / 84. Complete Gerrit histories: 28 / 28. Passing builds with incomplete test evidence: 3. Missing build identity: 0.
- Build inventory was collected without time filters: check offsets 800–3699 and gate offsets 0–499, with 3,400 unique build details verified. The inventory includes newer builds beyond the Monday cutoff and older builds back to 14 October, providing a margin around the reporting week.
- All 28 Gerrit change histories associated with weekly failures were retrieved. Seventeen retry sequences are corroborated by exact build/buildset publications; one candidate lacks qualifying Gerrit evidence before the cutoff.
- 22 failed builds have no Subunit or JUnit test artifacts in their manifests. They remain visible below; missing test results are not interpreted as successful tests.
- Flaky statistics indicate observed instability on an unchanged patchset. Dependencies, gate queue context and infrastructure can still differ between runs.
Gate results excluded from the failure ratio
| Result | Builds |
|---|---|
| ABORTED | 15 |
| RETRY | 7 |
Unique top-ranked tests investigated: 4 / 4. The same gate test appearing in both rankings shares one investigation.
Failed builds with incomplete test evidence (22)
| Job | Pipeline | Test coverage | Details | Observed job failure | Build |
|---|---|---|---|---|---|
| nova-graceful-shutdown | check | unavailable | No Subunit or test XML in manifest | Task 'Dest cold migration: Verify compute service is stopped after graceful shutdown' failed: | 6c50f65d |
| tempest-integrated-compute | check | unavailable | No Subunit or test XML in manifest | Job timed out with no task failures (killed mid-execution) | 8c6d44d8 |
| tempest-integrated-compute | check | unavailable | No Subunit or test XML in manifest | Job timed out with no task failures (killed mid-execution) | 6433c594 |
| tempest-integrated-compute | check | unavailable | No Subunit or test XML in manifest | Job timed out with no task failures (killed mid-execution) | cd086d54 |
| openstack-tox-pep8 | check | unavailable | No Subunit or test XML in manifest | Task 'Run tox' failed: non-zero return code | d5adb7bc |
| openstack-tox-pep8 | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 528c1c16 |
| nova-grenade-multinode | check | unavailable | No Subunit or test XML in manifest | Task 'Run grenade' failed: non-zero return code | 61feb6e5 |
| nova-ceph-multistore | check | unavailable | No Subunit or test XML in manifest | Task 'Run devstack' failed: non-zero return code | 4dd4036d |
| tempest-integrated-compute | check | unavailable | No Subunit or test XML in manifest | Task 'Run devstack' failed: non-zero return code | f3b5ad33 |
| tempest-ipv6-only | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 4afc2f75 |
| nova-graceful-shutdown | check | unavailable | No Subunit or test XML in manifest | Task 'Run devstack' failed: non-zero return code | 0c8daeb8 |
| grenade-skip-level-always | check | unavailable | No Subunit or test XML in manifest | Task 'Run grenade' failed: non-zero return code | 08e60044 |
| nova-multi-cell | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 078d923f |
| barbican-tempest-plugin-simple-crypto | check | unavailable | No Subunit or test XML in manifest | Task 'Run devstack' failed: non-zero return code | 010e308a |
| nova-whitebox | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 3e6752e4 |
| cyborg-tempest-py3 | check | unavailable | No Subunit or test XML in manifest | Diagnosis unavailable | 9ca58b81 |
| nova-multi-cell | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 99b121bf |
| openstack-tox-docs | check | unavailable | No Subunit or test XML in manifest | Task 'Run tox' failed: non-zero return code | 927d0c01 |
| openstack-tox-docs | check | unavailable | No Subunit or test XML in manifest | No failed tasks found and no error patterns matched | 0fc876ce |
| openstack-tox-pep8 | check | unavailable | No Subunit or test XML in manifest | Task 'Run tox' failed: non-zero return code | cd43f348 |
| openstack-tox-pep8 | check | unavailable | No Subunit or test XML in manifest | Task 'Run tox' failed: non-zero return code | d484d0bd |
| tempest-integrated-compute | check | unavailable | No Subunit or test XML in manifest | Job timed out with no task failures (killed mid-execution) | 439bf959 |