Help increasing mining read time?

  • Hey all,

    I could use some assistance with trying to shave some time off of my read times for mining if at all possible. Here is my current setup:

    (Links are only for reference)
    OS: Windows 7 Pro
    Miner: jminer 0.4.11
    CPU: Dual Xeon X5670
    Memory: 48GB RAM
    Video: Gigabyte AMD Radeon R9 280 OC 3GB
    USB: Rosewill 4-port USB 3.0 PCI-e card

    2x 8TB Seagate Backup Plus Hub USB 3.0 drives (5900 RPM)
    6x 8TB WD Easystore USB 3.0 drives (5400 RPM)
    2x 4TB misc drives via Network share (not sure of the brand off hand) (5900 RPM)

    So, 72TB total

    Here is a sample of my output from jminer (v0.4.11):

    04:05:26.003    -       1% done (0TB 1GB), avg.'1 MB/s'
    04:05:34.953    -       12% done (8TB 487GB), avg.'223 MB/s', eff.'231 MB/s'
    04:05:42.660    -       24% done (16TB 825GB), avg.'241 MB/s', eff.'264 MB/s'
    04:05:50.181    -       34% done (24TB 39GB), avg.'239 MB/s', eff.'234 MB/s'
    04:05:50.870    -       read 'E:/Burst/plots' (3TB 900GB) in '25s 189ms'
    04:05:53.191    -       read 'P:/burst/plots' (4TB 0GB) in '27s 525ms'
    04:05:58.907    -       45% done (32TB 26GB), avg.'235 MB/s', eff.'223 MB/s'
    04:06:09.633    -       57% done (40TB 446GB), avg.'224 MB/s', eff.'191 MB/s'
    04:06:20.740    -       67% done (47TB 934GB), avg.'212 MB/s', eff.'164 MB/s'
    04:06:25.300    -       read 'J:/burst/plots' (8TB 1GB) in '59s 619ms'
    04:06:28.591    -       read 'I:/burst/plots' (8TB 1GB) in '62s 912ms'
    04:06:29.181    -       79% done (56TB 688GB), avg.'217 MB/s', eff.'253 MB/s'
    04:06:29.184    -       read 'H:/burst/plots' (8TB 1GB) in '63s 504ms'
    04:06:33.365    -       read 'L:/burst/plots' (8TB 0GB) in '67s 686ms'
    04:06:33.710    -       read 'M:/burst/plots' (8TB 0GB) in '68s 42ms'
    04:06:34.959    -       read 'K:/burst/plots' (8TB 0GB) in '69s 270ms'
    04:06:40.075    -       89% done (63TB 984GB), avg.'209 MB/s', eff.'163 MB/s'
    04:06:53.201    -       read 'F:/burst/plots' (7TB 997GB) in '87s 534ms'
    04:06:53.301    -       100% done (71TB 899GB), avg.'200 MB/s', eff.'146 MB/s'
    04:06:53.307    -       read 'G:/burst/plots' (7TB 997GB) in '87s 627ms'
    04:06:53.551    -       FINISH block '381277', best deadline '16951', round time '87s 639ms'

    E:\ - 4TB Network Share
    F:\ - 8TB Seagate
    G:\ - 8TB Seagate
    H:\ - 8TB WD
    I:\ - 8TB WD
    J:\ - 8TB WD
    K:\ - 8TB WD
    L:\ - 8TB WD
    M:\ - 8TB WD
    P:\ - 4TB Network Share
    (All of the plot files for each of the drives are already optimized)

    Since the PCI-e USB 3.0 card is only 4 ports, 2x of the seagate drives and 2x of the WD drives connected direct to the USB card. Then I have 2x of the WD drives daisy-chained off of each of the Seagate drive's USB 3.0 hub like this:

    • Port1
      • Seagate
        • WD
        • WD
    • Port2
      • Seagate
        • WD
        • WD
    • Port3
      • WD
    • Port4
      • WD

    Watching some YouTube videos, I see some people with 100TB claiming mining speeds of around ~15s.... and I'm sitting at ~90s with only 3/4 of the capacity....and I'm seeing several blocks recently that I'm getting a "N/A" deadline due to the longer read times and not being able to read through all of my drives in time.

    Benchmarks of the drives put them as being capable of ~140MBps. So, I would expect that I would be able to see ~1100MBps (8.96Gbps)...which is almost double the bandwidth that USB 3.0 is rated for. So, I'm guessing that USB is bottle-necking me. However, the miner is only reporting read speeds of what I would expect of 1-2 drives... not any where the ~1100MBps that should theoretically be possible with 10 disks .

    Any idea what is happening? Is there a better USB card I should be using? Should I shuck these drives from their USB enclosures and throw them on a SATA III card or something (The WD drives are WD Reds inside)? any recommendation on SATA controller cards?

    Any thoughts and/or help on what can be done to shaving off some read time would be greatly appreciated.

  • @twig123 Are your plots optimized ? That is the case when last and second to last fields in the filename are identical;

    id_start_length_stagger where length==stagger.

    If not, you need to optimize them (reshuffle internal organization, so no excessive seeks are performed).

  • @vaxman Yeah, they are all optimized already. So, it's a matter of tweaking other things for speed now.

  • Sigh...
    Who's bright idea was it to mix Gen2 and Gen1 PCI Express slots on a board?!

    My mining computer:

    PCI Express performances:

    I'm pretty sure that my USB 3.0 Card may just be plugged into the PCI Express Gen1 slot.... which is likely causing the heartburn that I'm having of speed bottlenecks.

    I really hope it's as simple as just moving the USB 3.0 card to a different slot..... I'm going to see if I can test this later tonight and see if read speeds improve or not.

  • you may find this helpfull

  • @Jaxblack Yeah! I was literally just looking at these, after I realized these points:

    1. That my current USB 3.0 PCI-e 1x card is likely in a PCI-e Gen1 slot (250MB/s max throughput)
    2. With the USB 3.0 drives @ 140MB/s each, I could only attach 4 drives to my current card PCI-e Gen2 1x slot before the cards throughput would be saturated.
    3. That even if I swap my current card into a PCI-e Gen2 slot, that I would still be maxing out the upward limit of PCI-e Gen2 1x (500MB/s)... which can be fully consumed by a mere 4 drives. So, I would need at least 2 cards to handle the bandwidth.

    I literally didn't know USB 3.0 PCI-e 4x cards exist until just a few minutes ago. lol

    I found these 2 cards, both are PCI-e Gen2 4x cards.. which means I could attach 16 drives (4 to each port) before I reach the bandwidth limit of the card:
    $80.99 - StarTech PCIe SuperSpeed USB 3.0 Card
    $108.03 - Sonnet Allegro Pro USB 3.0 PCIe card

    I'm going to test swapping the card into another PCI-e slot first, and if I see speeds closer to the PCI-e Gen2 1x limits... I'll likely be purchasing one of the above cards, so that I can use a PCI-e Gen2 4x bandwidth instead of 1x bandwidth.

    I'll update my findings in an hour or two when I get a chance to swap slots

  • Sounds like the card might be your issue. If you get it sorted out, you might gain a bit more by playing with some settings in your file.

    Turning of the display of progress (readProgressPerRound=0) gained me a bit.

    I shaved a few seconds off by increasing my chunkPartNonces value. Default is 320,000. I went up in increments of 160000 till I saw no further gains. Gains depend on your system...RAM versus CPU. For some, going smaller may be better.

    I'm currently reading 111TB/12 drives on two USB controllers in 23-26 seconds.

  • Moved my USB 3.0 card to another slot.... instant upgrade!!!

    00:26:19.110    -       1% done (0TB 1GB), avg.'1 MB/s'
    00:26:23.888    -       12% done (8TB 32GB), avg.'387 MB/s', eff.'410 MB/s'
    00:26:28.070    -       23% done (16TB 254GB), avg.'429 MB/s', eff.'479 MB/s'
    00:26:32.186    -       34% done (23TB 985GB), avg.'438 MB/s', eff.'458 MB/s'
    00:26:36.311    -       45% done (31TB 984GB), avg.'446 MB/s', eff.'473 MB/s'
    00:26:37.602    -       read 'E:/Burst/plots' (3TB 900GB) in '18s 737ms'
    00:26:37.867    -       read 'P:/burst/plots' (4TB 0GB) in '19s 43ms'
    00:26:42.212    -       56% done (39TB 954GB), avg.'417 MB/s', eff.'329 MB/s'
    00:26:47.865    -       68% done (48TB 375GB), avg.'406 MB/s', eff.'363 MB/s'
    00:26:52.099    -       read 'J:/burst/plots' (8TB 1GB) in '32s 973ms'
    00:26:52.335    -       78% done (55TB 999GB), avg.'407 MB/s', eff.'416 MB/s'
    00:26:52.335    -       read 'H:/burst/plots' (8TB 1GB) in '33s 501ms'
    00:26:52.443    -       read 'I:/burst/plots' (8TB 1GB) in '33s 71ms'
    00:26:54.436    -       read 'M:/burst/plots' (8TB 0GB) in '35s 582ms'
    00:26:54.935    -       read 'K:/burst/plots' (8TB 0GB) in '36s 111ms'
    00:26:55.018    -       read 'L:/burst/plots' (8TB 0GB) in '36s 184ms'
    00:26:58.573    -       89% done (63TB 984GB), avg.'392 MB/s', eff.'312 MB/s'
    00:27:07.154    -       read 'G:/burst/plots' (7TB 997GB) in '48s 330ms'
    00:27:08.457    -       100% done (71TB 899GB), avg.'353 MB/s', eff.'195 MB/s'
    00:27:08.458    -       read 'F:/burst/plots' (7TB 997GB) in '49s 634ms'
    00:27:09.057    -       FINISH block '382297', best deadline 'N/A', round time '50s 125ms'

    (a pic of the board I have)

    Like for real... why would the manufacturer put a single PCI-e Gen1 slot on the board?!

    Now that I've confirmed I'm able to get faster read times by moving my USB 3.0 PCI-e 1x card into a new slot.... now the limitation is the actual USB card itself as the 1x card/slot is only capable of (500 MB/s). So, I'll be grabbing one of the above mentioned USB 3.0 PCI-e 4x cards that is capable of (1800MB/s) to further increase me read speeds.

  • @twig123 perhaps I can help as to why the manufacturer would put an icky yucky mean old slot on a motherboard... because you're using what appears to be an old HP workstation, don't worry, I'm an HP420 guy myself, still rocking it, but you do have to be aware of your limitations on these older systems... sometimes I tell people to move their two slot GPU to the lower PCIE x16 just so they can use the slot covered up by the GPU and wasted, but it would seem in your case, that IS the icky mean yucky one so its okay if its covered up out of sight out of mind...

    just put a "Mr Yuck" sticker on it, so anyone in the future knows its poisonous...

    just remember, you don't get the 2GB/s speeds simply because the card is PCIE 2.0 x4... you get that because the card has 4 separate USB controllers also, each controller is giving you a maximum of 625MB/s for a total of 2.5GB/s but since its only PCIE 2.0 x4 you top out at 2GB/s... if it didn't have the 4 separate controllers you'd still be buzzing along with a maximum of 625MB/s instead... the 4 controllers is the key to those cards, hence why they SHOULD be PCIE 2.0 x4 minimum...

  • @Darkbane Indeed, it's a HP Z600 board that I have buzzing with Dual Xeon X5670's, and 48GB RAM.

    Yeah, the PCI-e 4x would still need to be limited to 3-4 drives per port to prevent contention. I found the mention of a PCI-e 8x USB 3 card, but it was an overseas company and I couldnt find a source for ordering one.

  • @twig123 said in Help increasing mining read time?:

    @Darkbane Indeed, it's a HP Z600 board that I have buzzing with Dual Xeon X5670's, and 48GB RAM.

    Yeah, the PCI-e 4x would still need to be limited to 3-4 drives per port to prevent contention. I found the mention of a PCI-e 8x USB 3 card, but it was an overseas company and I couldnt find a source for ordering one.

    be careful when ordering that stuff, I've seen some single controller cards use x8 slots, which is just beyond me why they would spend the extra cost making a card that can't utilize the x8 bandwidth with a single controller... you can put x1 cards in x8 slots so when purchasing items like that, make sure it has more controllers and channels so you get what you're expecting...

    thats why those 4-chip cards deliver the goods, they have the power to utilize the x4 slot, but you can just as easily put them in x8 and x16 slots as well, it won't increase the performance obviously but it allows you to expand... I just wish the folks making those x8 and a few x16 cards I have seen would acknowledge they are single chip cards and are literally tricking people into thinking they are getting more from them...

    it reminds me of all the hubs/switches that used to advertise their extreme speeds, only for you to find out the backplane supporting all those ports, could only offer full speed to a couple ports at a time, so plugging in 24 machines didn't improve your overall network, it was still bottlenecked inside the device, and many of them never published the backplane speeds, because they knew anyone paying attention would realize they were being sold a bunch of magic tonic water that didn't cure anything...

    the startech quad chip controller is a good card, works well, will get you a lot more bandwidth to work with, but as we noted, you'll bottleneck that at 2GB/s which in theory is likely what you'll get anyhow, once you daisychain things or use other USB hubs, you lose a little performance anyhow, most devices weren't designed to give you full bandwidth, because they never figured we would utilize all that, and they think we simply just wanted another plug to use...

  • @Darkbane Nah, it was an 8x with 8 independent host controllers. So, 40Gbps capable. Couldn't find anywhere to buy it though:

    However, I think I'm going to just go with the Sonnet Allegro Pro USB 3.0 PCIe 4x card that have the 4 host controllers, and then if I end up expanding, I'll just buy a 2nd card.

  • @twig123 said in Help increasing mining read time?:

    @Darkbane Nah, it was an 8x with 8 independent host controllers. So, 40Gbps capable. Couldn't find anywhere to buy it though:

    However, I think I'm going to just go with the Sonnet Allegro Pro USB 3.0 PCIe 4x card that have the 4 host controllers, and then if I end up expanding, I'll just buy a 2nd card.

    I just looked at that link, very interesting card, although I am sitting here looking at the claims, and I'm wondering how they actually achieve them... and if I am reading their data correctly, you basically have 8 "independent controllers" that are bottlenecking down to the bandwidth of 4... which degeats the whole purpose of 8...

    either way they can't actually deliver 40Gb/s because PCIE 2.0 x8 only supports 4GB/s which amounts to 32Gb/S... I hate how GB and Gb mean very different things when you capitalize letters differently, it leads to a lot of confusion, really wish folks would just pick bytes or bits and extend that through to everything so people don't have to wonder if websites simply meant an upper or lower case letter...

    I guess I would want to know more about their "proprietary dual lane internal PCIE port cables" they have, and what chips are actually making this possible... I just don't even see all the necessary chips on the board to begin with, little alone extra chips to perform this task... maybe there is a reason why you can't find this card heh... (I guess this is why I never finished my degree in mechanics)

    but if this does exist, I think this is gold for the larger miners with tons of externals...

  • The Sonnet Allegro Pro USB 3.0 PCIe 4x card arrived today.
    After a struggled to find drivers that actually work for Windows 7, the latest drivers resulted in a "code 37" and Windows couldn't start the driver, things are humming along VERY nicely compared to my previous read times:

    18:14:00.120    -       1% done (0TB 1GB), avg.'1 MB/s'
    18:14:02.921    -       13% done (8TB 759GB), avg.'653 MB/s', eff.'763 MB/s'
    18:14:05.239    -       23% done (16TB 7GB), avg.'699 MB/s', eff.'763 MB/s'
    18:14:07.314    -       35% done (24TB 455GB), avg.'778 MB/s', eff.'994 MB/s'
    18:14:09.108    -       45% done (32TB 244GB), avg.'832 MB/s', eff.'1059 MB/s'
    18:14:11.271    -       56% done (39TB 998GB), avg.'840 MB/s', eff.'875 MB/s'
    18:14:13.595    -       67% done (47TB 978GB), avg.'839 MB/s', eff.'838 MB/s'
    18:14:14.265    -       read 'H:/burst/plots' (8TB 1GB) in '14s 612ms'
    18:14:14.327    -       read 'I:/burst/plots' (8TB 1GB) in '14s 674ms'
    18:14:15.902    -       78% done (55TB 928GB), avg.'840 MB/s', eff.'841 MB/s'
    18:14:17.467    -       read 'E:/Burst/plots' (3TB 900GB) in '17s 814ms'
    18:14:18.944    -       read 'P:/burst/plots' (4TB 0GB) in '19s 291ms'
    18:14:19.002    -       89% done (63TB 913GB), avg.'806 MB/s', eff.'629 MB/s'
    18:14:19.650    -       read 'G:/burst/plots' (7TB 997GB) in '19s 999ms'
    18:14:20.716    -       read 'J:/burst/plots' (8TB 1GB) in '21s 64ms'
    18:14:20.810    -       read 'F:/burst/plots' (7TB 997GB) in '21s 156ms'
    18:14:22.728    -       read 'M:/burst/plots' (8TB 0GB) in '23s 76ms'
    18:14:22.741    -       read 'L:/burst/plots' (8TB 0GB) in '23s 87ms'
    18:14:22.812    -       100% done (71TB 899GB), avg.'757 MB/s', eff.'511 MB/s'
    18:14:22.818    -       read 'K:/burst/plots' (8TB 0GB) in '23s 167ms'
    18:14:23.062    -       FINISH block '383828', best deadline '1230', round time '23s 170ms'

    ~87s when using the USB3 1x card in a PCIe Gen1 slot
    ~50s when using the USB3 1x card in a PCIe Gen2 slot
    ~23s when using the USB3 4x card in a PCIe Gen2 slot (This is the new Sonnet Allegro Pro card)

    I'm amazed at how big of an improvement the PCIe 4x card was. Even though the card was kinda pricey, I'm very happy with the purchase!

    If anyone else stumbles upon this, and grabs one of these Sonnet cards to be used on Win7, these are the drivers that actually worked for me:
    Fresco Logic USB 3.0 Driver 64-bit