Looking for massive file system sector size support



  • So i've been in the plotting mode for awhile and i see these different file systems and their cluster sizes. Just wondering if anyone has stumbled on anything with larger than microsoft clusters. (or allocation blocks)
    32mb is the largest for Microsoft that's with the extended file allocation tale. I was hoping for 64mb or something for when I reach beyond the 256tb limit.



  • ZFS can technically be unlimited but your still limited to the 4K sector size on disk



  • @ZapbuzZ Im not sure what you mean? The largest storagecluster from microsoft today is 1PB using ReFS.



  • ok lets say I have 100tb. inside that 100tb is clusters or allocation blocks. The maximum sector size I can ever get with Microsoft is 32mb. so, I want to build a disk set of 1 partition of with say 64mb sectors within it to store 1 file on it. Being one massive file it cannot waste. This would allow faster reading for this size of data.note with disks sector size is also related to cluster or block sizes these terms are confusing because servers are called clusters etc.



  • @ZapbuzZ said in Looking for massive file system cluster support:

    32mb

    Still not with you. Maybe im slow.

    Refs limits:
    Max. volume size 1 yobibyte
    Max. file size 16 exbibytes



  • yes that's capacity i'm talking disk allocation clusters amount of data in each cluster in the sectors because 1 file over a massive stripe of partition will seek faster.



  • @ZapbuzZ said in Looking for massive file system cluster support:

    yes that's capacity i'm talking disk allocation clusters amount of data in each cluster in the sectors because 1 file over a massive stripe of partition isn't gonna waste space.

    Again 4kb is the max regardless of filesystem. Your limited my hardware. you don't seem to understand why sector and block sizes are used.



  • @ZapbuzZ said in Looking for massive file system cluster support:

    ster size I meant disk cluster

    Ahh.. we are talking about improving read speed. I first thought you had a limit in filesize for 256TB



  • yes now you see what I'm trying to say. I appreciate the education about refs btw.



  • oh I only have 20tb at the moment it feels like 10mb Winchester days I wanna get up to 4000tb one day.



  • Well. As @manfromafar pointed out there are hardware limits to consider. i'm not an expert on the subject but i find this information pretty good even if they talk about backups. https://www.virtualtothecore.com/en/refs-cluster-size-with-veeam-backup-replication-64kb-or-4kb/



  • @manfromafar said in Looking for massive file system cluster support:

    Again 4kb is the max regardless of filesystem. Your limited my hardware. you don't seem to understand why sector and block sizes are used.

    not correct, not for zfs nor the underlying devices or drivers.
    I have a number of SSDs with 8k formatted sectors; NETAPP X446_1625, STEC Z16IZF2E, hardware Raids usually 16k+ to match large pages.
    ..and internally zfs has a default recordsize of 128k, which is dynamically adjusted (down to 1 sector) to the actual write load.

    But this is all mute as @ZapbuzZ is asking the wrong question - you need to find out

    • what is your desired capacity/scan time
    • how many read threads are needed to keep the (correctly chosen) CPU/GPU busy

    and THEN you have a target requirement for your storage.
    For 20,40,80,?? threads this won't be a single filesystem with an antiquated file allocation table design, whatever the allocation size ("cluster size" for NTFS).

    Also, keep in mind that when scaling to dozens or hundreds of disks, you will lose some. And this should not take down your whole array - but you don't want to spend too much on redundancy either.

    What a funky homework assignment. 8))



  • @ZapbuzZ said in Looking for massive file system cluster support:

    ent it feels like 10mb Winchester days I wanna get u

    dude, get real 🙂



  • I remember an old 10mb hard disk when I was little. I got fun out of it. then they grew with faster processors. Megabytes then gigabytes and now terabytes. Those whom didn't have a megabyte and take a gigabyte for granted wouldn't appreciate my approach to quotation. By the time I die 1 terabyte will be the new floppy? lol



  • I actually found and purchased a 1tb usb1 flash stick its so impressively slow but it works perhaps a modern replacement for tape drive? very cheap to buy.



  • No such thing as a cheap 1TB flash drive, at least not the small thumb drives we know and love. If you bought it on Fleabay or Amazon and it's less than $500 it's a likely scam and it's just an SD card with a 32GB "If you're lucky" or so of storage space on it, but they hack the firmware to report more. Try filling it up and see how much it holds before it gives errors.

    There are a very few legitimate USB drives out there or in development that have that capacity, but they're certainly not cheap or small. Currently only one made by Kingston is on the market that I can see and it's $734 for 1TB and it looks like a small brick.

    Also the legitimate USB 1TB drives are astoundingly fast and should be USB 3.0, if it's USB 1.0 or 2.0 I can almost guarantee it's a fake sight unseen.

    If you have the chance post the model # and manufacturer, and do your homework testing it out!

    I keep a baseline of $/TB and anything that's below them is suspicious. Current baselines are around here from what I can find.

    Regular HD ~$15+TB
    SSD ~$200+TB
    USB STICK ~$400+TB



  • @captinkid I will try fill it up see what happens 44gb at the moment been plotting to it for 10 hours lol not good for plotting



  • unfortunately I have misplaced it somewhere in my home so stuff it anyhow its too slow