downloading only 1 torrent

[tudor-pop]

Related to:
webtorrent/bittorrent-tracker#199
webtorrent/webtorrent#1349

Latest version ^4.0.1

OS: macOS Catalina 10.15.4, nodejs 12.16.3, npm 6.14.5

I'm trying to do browser -> server upload but after the first webtorrent.add(magnetUri, t=>{..})
it just halts.
This weekend I was trying to find out what could be the issue and here are my findings
Setup:

• 2 browser windows(not tabs) seeding 2 different files to simulate 2 peers
• localhost tracker, not modified, the standard one from github(also not working with the default ones but not sure if it works over DHT)
• express js app listening on port 8103 for POST /. The req body has a magnet uri that gets added to webtorrent. First call goes fine and uploads the file. Second POST calls hangs
• to eliminate possible problem with expressjs I tried webtorrent-hybrid/examples/node-download.js. Basically create an array of the magnet URIs that are seeding in the 2 browser windows, iterate over the array and wait for download. IF the array has only 1 magnet link, the download happens, regardless of which magnet link I use. If I use 2 links, it hangs...I tried promises because I was thinking maybe .add blocks somehow but nothing changed.

webtorrent-hybrid/examples/node-download.js

const WebTorrent = require('../index')
const client = new WebTorrent({
	dht: false,
	tracker: {
		announce: ["ws://localhost:8765","http://localhost:8765/announce","udp://localhost:8765"]
	}
})
const uris = [
	'magnet:?xt=urn:btih:106e92bba52ebe993717971c02fd251681f641e4&dn=website_ad.json&tr=ws%3A%2F%2Flocalhost%3A8765&tr=http%3A%2F%2Flocalhost%3A8765%2Fannounce&tr=udp%3A%2F%2Flocalhost%3A8765'
	  ,
	'magnet:?xt=urn:btih:89ce5fa31606e833ab3606c5a79de914588335cd&dn=maymounkov2002kademlia.pdf&tr=ws%3A%2F%2Flocalhost%3A8765&tr=http%3A%2F%2Flocalhost%3A8765%2Fannounce&tr=udp%3A%2F%2Flocalhost%3A8765'
]

console.log('uris:\t', uris)
function add(uri) {
	client.add(uri, function (torrent) {
		const files = torrent.files
		files.forEach(file => {
				console.log('file:\t\t', file.name)
		})
	})
}
for (let it of uris)
	add(it)

JS code run in browser:

        const trackers = process.env.REACT_APP_WS.split(',');
	window.client = new WebTorrent({
		dht: false,
		tracker: {
			announce: trackers
		}
	});
        let opts = {
		name: this.generateTorrentName(files),
		// store: ChunkStore,
		announce: trackers,
		// private: true
	};
	console.log("hashing files", files);
	this.client.seed(files, opts, (torrent) => {
		console.log('Client is seeding ', torrent);
		let fileHashes = [];
		for (let file of torrent.files) {
			fileHashes.push({
				name: file.name,
				length: file.length,
				hash: torrent.magnetURI
			});
		}
		this.torrent = torrent;
		resolve({fileHashes});
		
	});

• Using the TCP or UDP trackers does not seem to work at all(expected because TCP/UDP is only for bittorrent) only the WS one works with 1 torrent
• If I copy webtorrent-hybrid/examples/node-download.js and put it into another window, everything works fine, I can download from an array of magnet links
• I followed the issue down to webtorrent-tracker.js where I tried to remove the socket pool reusability thinking something might be wrong but that is fine & the issue still here so now I'm debugging this function but at this point I'm thinking that I'm just noob with Nodejs since the same code works fine in the browser.
  Can the issue be with node12? should I try node 10? since package.json says
  "engines": { "node": ">=10" },
  It could also be wrtc module but in that case I'm just unlucky....that module is being correctly set webtorrent-hybrid since I followed it down to the last assignment.
  I'm out of ideas...I will try node 10 but I doubt that is the issue....am I using node wrong? can't I call .add multiple times? seems like I should be able to...
  Thank you!

Update 1
it seems that when using an array of torrents the function onConnect is not getting called inside peer.js(which should mean that the signaling data is not getting through somehow or is not being correctly set...I also removed the parallel package to run it sync just to be sure and once it worked fine! <3)

Update 2
It seems that on my machine the problem is with the number of generated offers. By default there are 10 but it hangs when this line is reached. If I change that to 1..8 it seems to work fine & it downloads all my browsers seeded torrents. If I set that to 15 it just crashes with exit code 139 (interrupted by signal 11: SIGSEGV). There is a comment that says the offer generation can be slow but in my case it just hangs. I will try to find out why... such a strange issue...

Update 3
Issue is creating too many peers const peer = new Peer(opts) from simple-peer module. Anything over 15 peers will cause a nodejs sigfault. Tried it on Node 10/12/14. 10 peers will hang, 9 will sometimes work and anything below always works

Update 4
So I the limitation is with Bitorrent-tracker....I would need your advice @feross since you put this in place in 2015. Don't know if you remember but if you do, can you clarify why are we creating 10 peers to connect to 1 single peer when going through WSTracker? from these lines
Math.min(50,10) and 10 or 50 peers. Having 10-50 offerId -> Peer() seems wasteful because 1 nodejs host tries to start 10 peer connections to a browser to download the same content... only 1 peer with 1 offerId seems more logical. The same is happening reverse, 1 browser tries to start 10 connections to the other peer. What do you think?
Another issue is that if I call .add(torrentURI) only once and set that number(DEFAULT_ANNOUNCE_PEERS) to 50 (default value) then again node will crash with sigfault. This also means that it can't scale beyond 50 .add calls on a backend within the same process :(

[KilianKilmister]

Oof. this is above my expertise. It's to much work to set up an environment for checking it out myself, but i dug through all the relevant code and i can't i can't really add anything to this conversation. All i can do is suggest trying nodes child_module.fork() to easily create a subprocess. This should help with the limit-issue.

You could also try to iterate over the array asyncronously. this might do something:

async function add(...args) { // dont forget the spread operator on calling this `add(...uris)`
  for await (const uri of args) {
    client.add(uri, function (torrent) {
      const files = torrent.files
      files.forEach(file => {
	console.log('file:\t\t', file.name)
      })
    })
  }
}

Just a shot n the dark, but worth a try.

But again, this is to high for me. I can just about use a unix-socket. I've done very little with web-technology but i'll keep an eye on this as i planned to use it in the future

[tudor-pop]

nope that doesn't work. A child process might solve the issue but it won't scale unfortunately

[sergethompson]

I am definitely running into the same problem. I think this technology is so cool. It would be very cool if this could be fixed!

[github-actions]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?