Looking for massive file system sector size support

  • yes now you see what I'm trying to say. I appreciate the education about refs btw.

  • oh I only have 20tb at the moment it feels like 10mb Winchester days I wanna get up to 4000tb one day.

  • Well. As @manfromafar pointed out there are hardware limits to consider. i'm not an expert on the subject but i find this information pretty good even if they talk about backups. https://www.virtualtothecore.com/en/refs-cluster-size-with-veeam-backup-replication-64kb-or-4kb/

  • @manfromafar said in Looking for massive file system cluster support:

    Again 4kb is the max regardless of filesystem. Your limited my hardware. you don't seem to understand why sector and block sizes are used.

    not correct, not for zfs nor the underlying devices or drivers.
    I have a number of SSDs with 8k formatted sectors; NETAPP X446_1625, STEC Z16IZF2E, hardware Raids usually 16k+ to match large pages.
    ..and internally zfs has a default recordsize of 128k, which is dynamically adjusted (down to 1 sector) to the actual write load.

    But this is all mute as @ZapbuzZ is asking the wrong question - you need to find out

    • what is your desired capacity/scan time
    • how many read threads are needed to keep the (correctly chosen) CPU/GPU busy

    and THEN you have a target requirement for your storage.
    For 20,40,80,?? threads this won't be a single filesystem with an antiquated file allocation table design, whatever the allocation size ("cluster size" for NTFS).

    Also, keep in mind that when scaling to dozens or hundreds of disks, you will lose some. And this should not take down your whole array - but you don't want to spend too much on redundancy either.

    What a funky homework assignment. 8))

  • @ZapbuzZ said in Looking for massive file system cluster support:

    ent it feels like 10mb Winchester days I wanna get u

    dude, get real 🙂

  • I remember an old 10mb hard disk when I was little. I got fun out of it. then they grew with faster processors. Megabytes then gigabytes and now terabytes. Those whom didn't have a megabyte and take a gigabyte for granted wouldn't appreciate my approach to quotation. By the time I die 1 terabyte will be the new floppy? lol

  • I actually found and purchased a 1tb usb1 flash stick its so impressively slow but it works perhaps a modern replacement for tape drive? very cheap to buy.

  • No such thing as a cheap 1TB flash drive, at least not the small thumb drives we know and love. If you bought it on Fleabay or Amazon and it's less than $500 it's a likely scam and it's just an SD card with a 32GB "If you're lucky" or so of storage space on it, but they hack the firmware to report more. Try filling it up and see how much it holds before it gives errors.

    There are a very few legitimate USB drives out there or in development that have that capacity, but they're certainly not cheap or small. Currently only one made by Kingston is on the market that I can see and it's $734 for 1TB and it looks like a small brick.

    Also the legitimate USB 1TB drives are astoundingly fast and should be USB 3.0, if it's USB 1.0 or 2.0 I can almost guarantee it's a fake sight unseen.

    If you have the chance post the model # and manufacturer, and do your homework testing it out!

    I keep a baseline of $/TB and anything that's below them is suspicious. Current baselines are around here from what I can find.

    Regular HD ~$15+TB
    SSD ~$200+TB
    USB STICK ~$400+TB

  • @captinkid I will try fill it up see what happens 44gb at the moment been plotting to it for 10 hours lol not good for plotting

  • unfortunately I have misplaced it somewhere in my home so stuff it anyhow its too slow