[KVM] Native support for QEMU disk error policy (rerror/werror) via disk/compute offering #14366

The required feature described as a wish

Summary

KVM-hosted VM disks in CloudStack have no way to configure libvirt/QEMU's error_policy (rerror/werror in domain XML). Every disk defaults to QEMU's report policy, which means any I/O error from the backing storage — even a brief, otherwise-recoverable one — is passed straight through to the guest as a real disk error. This proposes adding first-class error_policy support to disk/compute offerings, modeled on the existing cacheMode offering attribute.

Problem / Motivation

Tested against a KVM cluster backed by NFS primary storage during a planned, successful storage-side HA failover (floating IP moved between NFS nodes in ~6s, well within normal NFS client timeouts): a VM with a continuous write workload hit a hard blk_update_request: I/O error on its data disk within 3-4 seconds of the failover starting — long before any NFS-level recovery (RPC retry, lock reclaim, grace period) had a chance to run. The guest's ext4 filesystem aborted its journal and remounted read-only (JBD2: Error -5, EXT4-fs: Remounting filesystem read-only).

Root cause: the disk's libvirt XML is <driver name='qemu' type='qcow2' cache='none'/> with no rerror/werror attribute, so QEMU's default report error policy applies — any hiccup on the backing file (NFS, iSCSI, etc.) is surfaced to the guest immediately and unconditionally. This is independent of how well the storage/HA layer itself is tuned: the guest reacts and can corrupt its own filesystem state before storage-level recovery even starts.

We confirmed (manually patching a live domain's persisted XML with error_policy='stop' and re-running the same failover) that QEMU's stop policy — which pauses the VM on I/O error and auto-resumes once the backing storage responds, instead of surfacing the error to the guest — eliminates the problem entirely: identical failover, zero guest-visible impact, domain never even needed to pause.

This affects any KVM deployment where primary storage can have a brief interruption that the storage layer itself would otherwise recover from transparently: HA/failover storage (our case), network blips to NFS/iSCSI targets, storage maintenance, etc. Today the only way to get error_policy='stop' is manually patching each VM's persisted domain XML outside of CloudStack's orchestration — which gets silently overwritten the next time CloudStack redefines the domain.

Current state (verified against main)

LibvirtVMDef.DiskDef (plugins/hypervisors/kvm/.../LibvirtVMDef.java) has no error_policy/rerror/werror field at all today — only cache, discard, io, iothread, block-size, etc. are supported on the <driver> element.

Proposed solution

Add error_policy as a first-class disk/compute offering attribute, following the exact existing plumbing for cacheMode (DiskOffering.DiskCacheMode) end to end:

1. Schema: new disk_offering.error_policy column (VARCHAR(16), default 'report'), added to disk_offering_view, service_offering_view, volume_view — same pattern as the cache_mode column.
2. Domain model: DiskOffering.DiskErrorPolicy enum (REPORT, STOP, IGNORE, ENOSPACE — QEMU's valid error_policy values), getErrorPolicy()/setErrorPolicy() on DiskOffering/DiskOfferingVO.
3. API: new errorpolicy param on createDiskOffering/updateDiskOffering (and compute-offering equivalents), surfaced on DiskOfferingResponse/ServiceOfferingResponse, validated in ConfigurationManagerImpl the same way cacheMode is today.
4. Orchestration: carry the value DiskOfferingVO → VolumeObject/VolumeObjectTO → hypervisor command, mirroring getCacheMode()'s path through VolumeObjectTO.
5. KVM agent: LibvirtVMDef.DiskDef gets an _errorPolicy field + setter, rendered as error_policy='<value>' on the <driver> element in toString(), set from the volume's offering in LibvirtComputingResource/KVMStorageProcessor next to the existing setCacheMode(...) calls.
6. Other hypervisors: this is a libvirt/QEMU-specific concept; XenServer/VMware plugins would simply ignore the new offering field, consistent with other KVM-only offering attributes.

Open questions for discussion

• Should this live on disk offering only, or also on compute offering (root disk)? cacheMode supports both — proposing the same here.
• Default value: report (current implicit behavior, fully backward compatible) vs. defaulting new offerings to stop for safety? Leaning toward report as default to avoid surprising existing deployments with VMs that can now pause.
• Operational concern: a VM with stop policy can sit paused indefinitely if the underlying storage problem never clears. Should this feature ship with (or require) companion alerting on paused-domain state, or is that left to operators?

Happy to put together a PR for this if the general direction sounds right — wanted to float the approach first since it touches the offering schema/API surface.