[Solved] Mikrok8s storage cluster with Microceph, rook-ceph, PVC pending, waiting for a volume to be created by external provisioner

@mscrocdile I'm going to setup a lab really quick to reproduce and follow steps, should be able to see if I run into the same issue with the steps. Also, your cluster is healthy and nodes ready?

microk8s kubectl get nodes

@brandon-lee 

@mscrocdile got it...looks like everything should be good...I'm going to go through setting up a little cluster in the lab and let you know what I find

@mscrocdile I just built a quick little cluster in the lab and was able to get it to work. I am using the latest microk8s 1.29. I am adding (1) local disk to each VM in ubuntu as Ceph storage. What type of storage are you using for Ceph?

@mscrocdile also in my testing since it is the only storage provisioner on my cluster, and looks like yours as well, I didn't have to set as default so that was a red herring.

@brandon-lee 

i use MicroK8s v1.28.3. Could that be the difference?

I was also adding 1 local drive into each hyperv Ubuntu VM.

@mscrocdile Well, i would say that isn't in play since when i wrote my blog post, I was using 1.28. However, you might try, if you have quick and easy snapshots to roll back to, with 1.29 and see what you get. Also, I am guessing the disks are showing up correctly in Ubuntu for use with Ceph....I would think so since your Ceph status shows Healthy.

@brandon-lee 

Everything is ok but it doesn't work

Well, i need to read fairytales for children now.. i will continue tomorrow morning.

I will upgrade to 1.29 if possible and tell you the result.

Thank you for your help.

@mscrocdile Curious if you have had a chance to try the cluster with v1.29? Let me know what you find on your ceph configuration. 👍

@brandon-lee 

i ran this on each node to update

sudo snap refresh microk8s --classic --channel=1.29/stable

I'm not sure i updated correctly and how it is then in production. Maybe rook-ceph plugin had to be disabled first and then enabled (including microk8s connect-external-ceph) - i  don't know.

However this did not help. PVC is still pending.

sudo ceph fs ls
No filesystems enabled

Is this correct there is no filesystem?

I'm also adding microceph status:

Just for sure for comparison - this is my microk8s status:

I wonder if these pendings are correct..

I've installed VM's again and started 

sudo snap install microk8s --classic --channel=1.29/stable

but even if i was waiting more then 20 minutes microk8s status -w is not finished and nodes are not ready

I will return to 1.28 ...

I thought it will help me to run sudo microceph init at each node. 

I noticed there is also IO section which i never noticed before so i though it will be better now

However that PVC is still pending. No change.

There are two pools now and IO disappeared after enabling rook-ceph :

@mscrocdile I think you are onto something when you posted the status of the rook-ceph pods. Here is what I see:

 Also, to answer your question about the file system, this is what I see when running the command (also see the no filesystems enabled):

sudo ceph fs ls

See what you see when you run this command:

microk8s kubectl -n rook-ceph logs -l app=rook-ceph-operator

Also, check out their troubleshooting guide, steps through the commands to run:

Ceph Common Issues - Rook Ceph Documentation