Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

So is there a server program to partner this with? Something that acts as a torrent file builder, tracker and simple file server for the torrent? I can imagine in a large org you could store a gigantic quantity of public data on a server that creates a torrent whenever the data changes, serves the.torrent file over http and also acts as a tracker. You could wrap the FUSE client in some basic logic that detects newer torrents on the server and reloads/remounts.

Many moons ago I created a Linux distribution for a bank. It was based on Ubuntu NetBoot with minimal packages for their branch desktop. As the branches were serverless, the distro was self-seeding. You could walk into a building with one of them and use it to build hundreds of clones in a pretty short time. All you needed was wake-on-lan and PXE configured on the switches. The new clones could also act as seeds. Under the hood it served a custom Ubuntu repo on nginx and ran tftp/inetd and wackamole (which used libspread, neither have been maintained for years). Once a machine got built, it pulled a torrent off the "server" and added it to transmission. Once that was completed the machine could also act as a seed, so it would start up wackamole, inetd, nginx, tracker etc. At first you seed 10 machines reliably, but once they were all up, you could wake machines in greater numbers. Across hundreds of bank branches I deployed the image onto 8000 machines in a few weeks (most of the delays due to change control and staged rollout plan). Actually the hardest part was getting the first seed downloaded to the branches via their old Linux build, and using one of them to act as a seed machine. That was done in 350+ branches, over inadequate network connections (some were 256kbps ISDN)



This may interest you, although as far as I know only AWS's S3 implements it: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec...

I actually have never used it in order to know if AWS puts their own trackers in the resulting .torrent or what


hmmm, pretty cool. Of course you have to pay the egress tax, but still...


> You could wrap the FUSE client in some basic logic that detects newer torrents on the server and reloads/remounts.

Wouldn't creating new torrents for each update to a dataset cause clients to retransfer data that hasn't changed?


From memory, with a standard torrent, when it loads up a new torrent file it verifies hashes on existing data, and if it matches up it leaves it. That's how it worked on my old distribution. Not sure how this would work on BTFS, but if it's lazy-loading data and if it handled existing data in the same way, it shouldn't be an issue. But I think this would suit infrequently changing data anyways.


I suppose blocks with the same hash as existing won't get re-requested.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: