[Solved] Mikrok8s storage cluster with Microceph, rook-ceph, PVC pending, waiting for a volume to be created by external provisioner

@brandon-lee 

In meantime i've created 3 vm instances in google cloud and exactly the same problem. So i can continue in LAN.

operator logs:

@mscrocdile wow interesting. Hey which Ubuntu version are you using by the way?

@brandon-lee 

on both lan and google cloud its the same:

and 2GB of memory. Could that be problem?

@mscrocdile hmmm that might be something to try. I know my lab VMs have 4 GB. Also, can you follow my steps here and see what you come up with? I can't remember if I diverged from the steps in the official guide. This is what I used yesterday: Kubernetes Persistent Volume Setup with Microk8s Rook and Ceph - Virtualization Howto

@brandon-lee 

only difference i can see in:

snap install microk8s --channel=1.28/stable --classic but in screenshot there is "candidate". Is it something that could be important?

and also here

sudo snap install microceph --channel=latest/edge

i did not use channel parameter at all

I'm increasing that memory...

4GB of memory did not help

(true is i did not installed whole machine or microk8s, just disabled/enabled rook-ceph addon + external)

Another difference i can see is that i use everywhere sudo.

If i add my user into group:

sudo usermod -a -G microk8s spravce

and reboot then if i run microk8s enable rook-ceph (without sudo) it results to error:

PermissionError: [Errno 13] Permission denied: '/var/snap/microk8s/common/plugins/connect-external-ceph'

@mscrocdile I think when I first tested rook-ceph with Ubuntu 22.04, I tested with the release candidate. I don't think that matters, since I used the 1.28 and 1.29 stable release without issues after they were released.

Also, what do you see if you describe the rook-ceph pods that are not spun up? I don't think you will see storage provisioned if those pods are not healthy.

@brandon-lee 

Problem fixed!

In that pending csi pod there was message "Insufficient cpu."

I used only 1 cpu all the time for each node.

Not exactly, i increased it while testing but it was not enough. This time i installed all nodes with 2 cpu and 4GB mem from the beginning. I also think i described those pods before and there was nothing about cpu.

But i was so tired from that error that i don't know what is true and what is not.

Thank you very much! I will talk about you everywhere now.

@mscrocdile That is awesome! Great troubleshooting...you were very methodical and tried what I would have tried as well. Good job! Please do talk up the VHT forums and point others here for help. Don't hesitate to post on the forums and help us grow the community here Thanks again.