Windows share from a Proxmox HA Ceph Cluster

@wikemanuel welcome to the forums! Ok, so it looks like using CephFS you can share out a directory to clients.

Take note of these official forum responses from Proxmox:

This first post makes it sound like Proxmox recommends simply creating a VM on top of your Ceph storage as normal and create your SMB/Samba share from the VM. This route would probably eliminate some complexities of exposing it directly from Ceph:

https://forum.proxmox.com/threads/cephfs-usage-as-samba-or-something.81373/

This looks to be a walkthrough of creating Ceph storage to expose the storage directly:
https://forum.proxmox.com/threads/shared-storage-for-vms-on-cephfs.109494/

I would probably venture to say that it might be best to take the VM route as you could then just backup your VM with your attached storage using Proxmox Backup server to make sure everything is safe. Otherwise, you could probably rsync the directory from CephFS to a storage location outside of Proxmox.

Let me know if this makes sense?

@brandon-lee thank you for the two links. The first option of creating a VM on top of Ceph and subsequent pool from inside the VM seems easiest. The storage pool I am trying to create spans 3 servers and a total of 168TB of disk storage. I may be out of my knowledge realm when saying this, but, it is my understanding that there is a "drive size limitation" in the VM environment when carving out storage (something like 2TB per volume). The goal is to have a single mountpoint for all 168TB and not several drives. Is there a particular type of VM/share combo I should look at using?

Ok so there are several layers to think about here. First is the layer assigned to Proxmox. It looks like Proxmox has no issues with very large volumes:

According to this post you can assign these very large volumes to a VMs:

https://forum.proxmox.com/threads/maximun-size-of-volume-or-disk-that-can-be-assign-to-proxmox.7300/

For 32-bit CPUs on 2.6 kernels, the maximum LV size is 16TB.
For 64-bit CPUs on 2.6 kernels, the maximum LV size is 8EB. (Yes, that is a very large number.)
For 2.4 based kernels, the maximum LV size is 2TB.

Also, on the operating system side, you would need to consider the configuration there as well. It looks like in Windows 2019, NTFS can support an 8 PB volume with 2048 KB cluster size. Note the documentation here:

https://learn.microsoft.com/en-us/windows-server/storage/file-server/ntfs-overview

Also, here is a pretty cool discussion from 45 drives on really large Ceph storage:

@brandon-lee this is all really great stuff. I also ran into this video that appears to show that a ceph fs can be accessed directly from Windows via the use of Ceph Dokan drivers:

@wikemanuel Nice! that is super cool....will have to try this out myself 👍

@brandon-lee it would be cool for you to do a similar video.
They say there is no such thing as a stupid question but let me try ...
In your video "Proxmox 8 Cluster with Ceph Storage configuration" you create a 3 node pmox cluster, with a 53.69GB disc for each node. I suspect the total useable pool size is only 53.69GB (a 3-way mirror if you will)... what happens if you add another node? 4-way mirror or something else? What happens if you add an additional disc to a node but not all nodes (does the pool grow)? All these questions are leading up to understanding how ceph grows not only in performance (additional nodes) but also in useable pool space. 

@wikemanuel I am setting up my lab to confirm my thoughts on your questions and I will post back soon. stay tuned.

@brandon-lee not all hero's wear capes! Thank you for doing this... my continued research points to storage efficiency settings 3x replica (33% efficiency) vs. erasure coding (66% for 3 node cluster with 1 failure tolerance) or as high as 71.5% efficiency in larger 14 node clusters with 4 failure tolerance. In summary native ceph 33% efficiency does not excite me for my home lab storage pool solution... but 66%+ with erasure coding might be the cat's meow 

@wikemanuel No worries, ok so I will post answers as I work through them. The first is simply adding an additional disk to one of the hosts. 

I grabbed some screenshots of a 3 node cluster that I built, each with a single disk (much like my blog post). The pool was around 150 GB. I added one disk on one of the nodes, and as you can see at the end, the pool size is near 200 GB, so it does add space to the pool.

@wikemanuel This is after adding a 4th host to the cluster. You can see the Pool size is growing as expected. This is immediate....pretty cool.

This is all super useful! If I am reading this all correctly; that 249.98GB on the end is 4 nodes each with a ~50GB disc and 1 of the nodes using 2 of the 50GB discs... so I am assuming it is showing you total available (similar to a stripe) but once you implement some sort of redundancy the total useable will be lower no? It would be mine boggling to think it adds drives up like a stripe and manages to have redundancy. I would assume 33% or 66% useable of the 249.98GB depending on if you do or do not use erasure coding.

@wikemanuel Actually I should have clarified that screenshot above...this is the RAW storage that is displayed by Ceph on the Ceph dashboard. It does increase your total capacity to add disks and nodes. However, when I started with the 3 node configuration the Ceph configuration was set as the 3/2 configuration:

This results in only seeing 33% of the space available from the host perspective (doing the math 249.98 GB/ 3 = 83 GB roughly which you see on the hosts side.

@wikemanuel CephFS is awesome from Proxmox. Been playing around with this on top of the Ceph storage and was able to successfully mount my Ceph storage running on Proxmox on both Linux and Windows. Blog post will be forthcoming: