Plotting and mining in the cloud
DCSSquared last edited by DCSSquared
I've been using: https://github.com/Mirkic7/mdcct to plot and optimize plots on a dropbox business connection using rclone mount to mount the filesystem in Linux: https://rclone.org/commands/rclone_mount/
I have 4 machines running, one is doing 1gb plots on a 50mbps upload and taking a long time doing it, the other three are using the three dropbox user's individually and plotting and 'rclone move'ing 4gb plots at a time in ~18 minutes on a 336mbps connection to a subdirectory.
First off, it looks as though my connection is rate limited. At first I thought it was my ISP until I spun up the google cloud instances, so it must definitely be Dropbox rate limiting me - I'll be running at 336mbps upload and then it'll drop to 0, then back up, then back down I've emailed Dropbox about any throttles and we'll see what they say.
My miner is Uray's miner: https://github.com/uraymeiviar/burst-miner/releases which I'm running on the local machine, and getting tons of errors like:
<head><title>504 Gateway Time-out</title></head>
<center><h1>504 Gateway Time-out</h1></center>
and just appearing on the screen (but not staying in logs) stuff like:
72017/06/25 15:30:55 ERROR : 7876566878528197112_200001_4000_4000: ReadFileHandle.Read error: low level retry 1/10: Get https://api-content.dropbox.com/1/files/auto/7876566878528197112_200001_4000_4000: dial tcp: lookup api-content.dropbox.com on 127.0.1.1:53: read udp 127.0.0.1:53814->127.0.1.1:53: i/o timeout
2017/06/25 15:30:55 ERROR : google/7876566878528197112_245761_16384_16384: ReadFileHandle.Read error: low level retry 1/10: Get https://api-content.dropbox.com/1/files/auto/google%2F7876566878528197112_245761_16384_16384: dial tcp: lookup api-content.dropbox.com on 127.0.1.1:53: read udp 127.0.0.1:53814->127.0.1.1:53: i/o timeout
2017/06/25 15:31:47 ERROR : 7876566878528197112_72001_4000_4000: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
EDIT: a new one:
2017/06/25 15:42:34 ERROR : google/7876566878528197112_65537_16384_16384: ReadFileHandle.Read error: low level retry 1/10: read tcp 192.168.2.18:52250->220.127.116.11:443: read: connection reset by peer
anyone have any ideas how I can better fuse mount a dropbox folder so that it behaves better for burst mining?
https://github.com/joe42/CloudFusion this solution doesn't work unless someone can tell me how to fix it
neither does this: https://github.com/arekzb/dropfuse
neither does this: https://github.com/realriot/ff4d (API issues)
Dropbox has an official sync client but the problem is it actually syncs. With my three users @1pb each I can't be syncing the plots back to my local Ubuntu machines. If anyone knows a way to configure Dropbox's official client to only allow writing and reading to a mounted drive that'd work, where it doesn't sync back to the local drive?
Or if someone knows anything about the above errors and can help me with them, that'd be great.
DCSSquared last edited by
I'm going to retry with rclone mount --allow-other --no-modtime --read-only