Intermittent Template Download Failure (NOT_DOWNLOADED) in CloudStack 4.20.1.0 with Linstor Primary Storage

[gw769]

Hi all,

I’m encountering an intermittent issue when uploading templates to CloudStack 4.20.1.0 where Linstor is used as Primary Storage. Here’s the detailed breakdown:

CloudStack Version: 4.20.1.0
Primary Storage:
Linstor-controller 1.32.3-1ppa1-noble1
Linstor-satellite 1.32.3-1ppa1-noble1
DRBD_KERNEL_VERSION=9.2.14
System: Ubuntu 24.04.3 LTS
Hypervisor: KVM

When uploading a template (e.g., an Ubuntu 22.04 qcow2 image), the process intermittently fails with NOT_DOWNLOADED in Primary Storage. In my tests, 2 out of 3 upload attempts fail, while 1 succeeds randomly.

Two critical symptoms:

1. qemu-img Binary Not Found: The CloudStack agent not execute qemu-img convert (verified via strace ):
   ps -ef | grep cloudstack-agent
   strace -f -s 256 -p 1318808 -o /tmp/agent_trace.log

can't find execute qemu-img convert like this:

execve("/usr/local/sbin/qemu-img", ["qemu-img", "convert", "-n", "--target-is-zero", "-W", "-S", "1M", "-O", "raw", "-t", "none", "-U", "--image-opts", "driver=qcow2,file.filename=/mnt/.../xxx.qcow2", "/dev/drbd1105"], ...) = -1 ENOENT (No such file or directory)

DRBD Devices Stay “Unused”: Linstor resource listings show DRBD devices as Unused (instead of one of device InUse):

root@NODE76:~# linstor v l |grep  80bc7788-638e-4211-9b31-d76d037b50b3
| cs-80bc7788-638e-4211-9b31-d76d037b50b3    | NODE158 | DfltDisklessStorPool |     0 |    1105 | /dev/drbd1105 |            | Unused |   Diskless | Established(2) |
| cs-80bc7788-638e-4211-9b31-d76d037b50b3    | NODE76  | x86_pool_zfs_ssd     |     0 |    1105 | /dev/drbd1105 |  20.00 GiB | Unused |   UpToDate | Established(2) |
| cs-80bc7788-638e-4211-9b31-d76d037b50b3    | NODE83  | x86_pool_zfs_ssd     |     0 |    1105 | /dev/drbd1105 |  20.00 GiB | Unused |   UpToDate | Established(2) |

Steps to Reproduce:

1. Upload a qcow2 template (e.g., ubuntu-22.04.4-amd64-base-UEFI.qcow2) to CloudStack.
2. Monitor the Primary Storage (Linstor) download status.
3. Repeat the upload 3+ times and observe intermittent NOT_DOWNLOADED failures.

Expected Behavior:
The template should download to Primary Storage successfully every time, with qemu-img convert executing properly and DRBD devices showing InUse.

Actual Behavior:

• Intermittent failures (2 out of 3 attempts fail).
• DRBD resources remain Unused during failed attempts.

Questions:

• What configuration or troubleshooting steps should I take to resolve this intermittent behavior?

Any insights or guidance would be greatly appreciated!

Thanks,

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[DaanHoogland]

@rp- , can you assess this?

[rp-]

I'll check it this week.

[rp-]

@gw769 How exactly do you upload the template?
Are you using the Upload tempalte from local? (but this would only upload to secondary storage).

Do you register template from URL with direct download?

[gw769]

> @gw769 How exactly do you upload the template?

Are you using the Upload tempalte from local? (but this would only upload to secondary storage).

Do you register template from URL with direct download?

Hi, I registered template like this
, use URL but not direct download.

[rp-]

and then you just create an instance? from it?

[gw769]

and then you just create an instance? from it?

yes

[rp-]

and then you just create an instance? from it?

yes

uff, I was hoping you did a bit more fancy variant.
I never heard of/had any problems with the standard copy template image.

Is this always failing on the same host? Could you attach a full debug log of the failed copycommand from the host?

[gw769]

and then you just create an instance? from it?

yes

uff, I was hoping you did a bit more fancy variant. I never heard of/had any problems with the standard copy template image.

Is this always failing on the same host? Could you attach a full debug log of the failed copycommand from the host?

linstor-check.log
management.log
NODE74-agent_trace.log
NODE76-agent_trace.log

Thanks for following up. I've attached the full debug logs as requested: linstor-check.log, management.log, NODE74-agent_trace.log, and NODE76-agent_trace.log.
The template image in question has the ID: afeb2d54-5993-47d9-8704-e9f9c2b9f09c, and the instance ID is i-2-647-VM.
Let me know if you need any additional details or logs to help track down why the copy command is failing.

[rp-]

Could I have just the normal cloudstack agent logs, also the node76 log can't be downloaded.
But I already have something interesting:

2025-10-31 09:50:02,034 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 9-31243722414883234: Processing:  { Ans: , MgmtId: 200229881867143, via: 9, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "com.cloud.hypervisor.kvm.storage.KVMStoragePool.getUuid()" because "pool" is null
	at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.connectPhysicalDisk(LinstorStorageAdaptor.java:311)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.connectPhysicalDisk(KVMStoragePoolManager.java:164)
	at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyTemplateToPrimaryStorage(KVMStorageProcessor.java:285)
	at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:103)
	at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1958)
	at com.cloud.agent.Agent.processRequest(Agent.java:779)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
","wait":"0","bypassHostMaintenance":"false"}}] }