Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

P2P loader. #305

Merged
merged 116 commits into from
Nov 2, 2023
Merged
Show file tree
Hide file tree
Changes from 106 commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
c78c60c
Fix issue with wrong segment endTime.
i-zolotarenko Aug 21, 2023
eb21613
Fix issue with hls wrong segment local id.
i-zolotarenko Aug 21, 2023
99c234d
Add linked-map, load queue and http-loader.
i-zolotarenko Aug 23, 2023
40881d6
Add load queue class.
i-zolotarenko Aug 27, 2023
02a07fe
Add segments storage.
i-zolotarenko Aug 28, 2023
30ed915
Add segments load logic.
i-zolotarenko Aug 30, 2023
2dc9029
Add plugin requests promises map.
i-zolotarenko Aug 31, 2023
3ea3362
Rewrite load queue.
i-zolotarenko Aug 31, 2023
2ff4aa6
Move segment queue status determination to queue class.
i-zolotarenko Aug 31, 2023
77f6804
Separate main and secondary stream to different contexts.
i-zolotarenko Sep 1, 2023
d933e7a
Remove unnecessary loader layer.
i-zolotarenko Sep 1, 2023
72d33eb
Fix issue with not loading after moving playhead to random position.
i-zolotarenko Sep 4, 2023
f6713b7
Rename hybrid loader file.
i-zolotarenko Sep 4, 2023
ff79e37
Initialize playback on first segment request by plugin.
i-zolotarenko Sep 4, 2023
fe3ea0e
Make storage async. Add core destroy logic.
i-zolotarenko Sep 4, 2023
e13d358
Use StreamWithReadonlySegments type in plugins.
i-zolotarenko Sep 5, 2023
7c4197d
Add bandwidth approximator.
i-zolotarenko Sep 5, 2023
72e426a
Refactor load queue code.
i-zolotarenko Sep 5, 2023
03a388b
Load controlling: pure functions style. (#302)
i-zolotarenko Sep 7, 2023
9ac874f
Install bittorrent-tracker and ripemd160 to core.
i-zolotarenko Sep 12, 2023
81e8521
Create type declaration for bittorrent tracker.
i-zolotarenko Sep 12, 2023
5f899c7
Create P2PLoader and Peer classes.
i-zolotarenko Sep 12, 2023
9d7d6f3
Move enums to separate file.
i-zolotarenko Sep 12, 2023
08f4c23
Remove unused position core property.
i-zolotarenko Sep 13, 2023
23e9abc
Create secondary hybrid loader only if necessary.
i-zolotarenko Sep 13, 2023
fb5b0ea
Add force parameter to process queue hybrid loader method.
i-zolotarenko Sep 13, 2023
ffab215
Merge remote-tracking branch 'origin/load-controlling' into p2p-manager
i-zolotarenko Sep 13, 2023
d0da2ae
Add sendCommand method to Peer.
i-zolotarenko Sep 13, 2023
e294cad
Add request type.
i-zolotarenko Sep 13, 2023
d67e4c8
Create and apply RequestContainer class.
i-zolotarenko Sep 14, 2023
7b61d44
Rename settings time window properties.
i-zolotarenko Sep 15, 2023
90f5d1d
Add AbortError.
i-zolotarenko Sep 15, 2023
db389b0
Merge with v1
i-zolotarenko Sep 15, 2023
3e05601
Merge with v1
i-zolotarenko Sep 15, 2023
1cb7353
Add streams map to hybrid loader.
i-zolotarenko Sep 15, 2023
a18826e
Add peer self segments map generation logic.
i-zolotarenko Sep 18, 2023
70bff75
Notify peers on segment loadings update.
i-zolotarenko Sep 18, 2023
f61b852
Fix types issues.
i-zolotarenko Sep 18, 2023
3305c19
Add peer events.
i-zolotarenko Sep 18, 2023
676ddae
Merge with requests-container.
i-zolotarenko Sep 18, 2023
a2cee74
Add downloadSegment method to p2p-loader. Use p2p request type in Pee…
i-zolotarenko Sep 18, 2023
b93677a
Use request container in p2p-loader.
i-zolotarenko Sep 19, 2023
098b10a
Add receiving segment chunks logic.
i-zolotarenko Sep 19, 2023
7cf3524
Reject promise with errors in peer class.
i-zolotarenko Sep 19, 2023
bb22472
Rewrite fetch segment with async await.
i-zolotarenko Sep 19, 2023
486058f
Use object with booleans instead of set for segment queue statuses.
i-zolotarenko Sep 19, 2023
8ef7dae
Rename fetch segment function.
i-zolotarenko Sep 19, 2023
920db7d
Rename hybrid loader public method.
i-zolotarenko Sep 19, 2023
7783df1
Use engine callbacks instead of promise creation in core.
i-zolotarenko Sep 20, 2023
34e6881
Merge pull request #306 from Novage/requests-container-p2p-integration
i-zolotarenko Sep 20, 2023
4ad6e5e
Remove unnecessary getControlledPromise function from engines.
i-zolotarenko Sep 20, 2023
aa369ab
Merge branch 'requests-container-p2p-integration' into p2p-manager
i-zolotarenko Sep 20, 2023
2bb356e
Create hybrid loaders when necessary data is already set.
i-zolotarenko Sep 20, 2023
695deca
Add progress to http loading.
i-zolotarenko Sep 20, 2023
ddf2265
Add progress functionality to Peer class.
i-zolotarenko Sep 20, 2023
7bd2267
Fix type issues.
i-zolotarenko Sep 20, 2023
6fb3fbe
Use only callbacks instead of promise creation in hls engine.
i-zolotarenko Sep 21, 2023
ffc4424
Add segment storage initialization.
i-zolotarenko Sep 21, 2023
513b361
Add peer settings options.
i-zolotarenko Sep 21, 2023
6e6e108
Move storage cleanup logic into storage class.
i-zolotarenko Sep 21, 2023
541d78b
Add p2p loaders container.
i-zolotarenko Sep 22, 2023
7893230
Add p2p loaders destroy logic.
i-zolotarenko Sep 22, 2023
83d067f
Remove unnecessary toString on string variable.
i-zolotarenko Sep 22, 2023
c6419ae
Don't use response.arrayBuffer() if getting data from stream.
i-zolotarenko Sep 25, 2023
b443b03
Change peer announcement type and corresponding logic.
i-zolotarenko Sep 25, 2023
0c8af4f
Use single PeerRequestError with different types instead of lot of Er…
i-zolotarenko Sep 25, 2023
1938c55
Change peer announcement type and corresponding logic.
i-zolotarenko Sep 25, 2023
f966ca4
Remove memory request container counters.
i-zolotarenko Sep 26, 2023
a0864cf
Remove unnecessary segmentsToDelete from memory segment storage.
i-zolotarenko Sep 26, 2023
093f87b
Fix issue with typo in "chunk"
i-zolotarenko Sep 27, 2023
7363354
Add ability to subscribe to segments store update.
i-zolotarenko Sep 27, 2023
e5e24c9
Merge with origin/p2p-manager.
i-zolotarenko Sep 28, 2023
ff29655
Add stale p2p loader destroy timeout.
i-zolotarenko Sep 28, 2023
fe58ea0
Add request container http requests subscriptions.
i-zolotarenko Sep 28, 2023
978d95d
Add vite-plugin-node-polyfills to demo.
i-zolotarenko Sep 28, 2023
9671b0c
Move utils to separate folder.
i-zolotarenko Sep 28, 2023
083d593
Fix issues with abort looping, not handling requests in container.
i-zolotarenko Sep 28, 2023
420ffcd
Force queue process if playback position was significantly changed.
i-zolotarenko Sep 28, 2023
7622bdf
Add stream property to segment type.
i-zolotarenko Sep 29, 2023
44ed56b
Fix issue with fetch result data wrong promise error handling.
i-zolotarenko Oct 1, 2023
2b59ee2
Fix issue with not clearing aborted engine request.
i-zolotarenko Oct 2, 2023
15cfdda
Add random http loading.
i-zolotarenko Oct 2, 2023
6a607a3
Change JsonSegmentAnnouncement type.
i-zolotarenko Oct 2, 2023
7b6330a
Add loading through p2p to process queue.
i-zolotarenko Oct 2, 2023
7599820
Change load method parameter to queue item.
i-zolotarenko Oct 3, 2023
3cc30fe
Add loggers to hybrid loader, p2p-manager.
i-zolotarenko Oct 5, 2023
e564a76
Add load probability to http random load.
i-zolotarenko Oct 5, 2023
67886a9
Show stat in Demo. Destroy p2p manager if there is no uploadable data.
i-zolotarenko Oct 5, 2023
7f11315
Refactor code related to stat components.
i-zolotarenko Oct 6, 2023
52d88d0
Add loggers select to demo.
i-zolotarenko Oct 6, 2023
220b440
And onPeerClose event to Peer class.
i-zolotarenko Oct 8, 2023
a480959
Fix issue with wrong array buffer slicing. Modify choosing peer conne…
i-zolotarenko Oct 10, 2023
08203df
Add peer logger. Use utf-8 string for peer ids instead of hashes.
i-zolotarenko Oct 10, 2023
321cc6b
Remove ripemd160 package.
i-zolotarenko Oct 10, 2023
9a14429
Install node types. Patch bittorrent-tracker. Remove modules ignoring…
i-zolotarenko Oct 11, 2023
548320d
Sent ArrayBuffer to peer instead of Buffer. Use custom function to ch…
i-zolotarenko Oct 11, 2023
0bd2822
Add remain time predicting logic.
i-zolotarenko Oct 11, 2023
4c01697
Create universal bandwidth-approximator.
i-zolotarenko Oct 15, 2023
1315281
Add bitrate adaptation logic.
i-zolotarenko Oct 16, 2023
8bc001f
Rename p2p-loader logger.
i-zolotarenko Oct 17, 2023
11cbfa3
Use simple number variable instead of object creation.
i-zolotarenko Oct 17, 2023
62e312a
Modify uploading cancellation logic.
i-zolotarenko Oct 18, 2023
ebb843b
Broadcast announcement not more than 1 time for macrotask.
i-zolotarenko Oct 18, 2023
a033a7f
Generate user-friendly stream hash.
i-zolotarenko Oct 24, 2023
1f3b760
Configure tsconfig module, target options.
i-zolotarenko Oct 25, 2023
f3bf527
Change event handler name to onSegmentLoaded.
i-zolotarenko Oct 25, 2023
1297e87
Add core event handlers to shaka.
i-zolotarenko Oct 30, 2023
2e78864
Fix issues with shaka streams playback.
i-zolotarenko Oct 30, 2023
0932063
Revise and refactor code.
i-zolotarenko Oct 31, 2023
4162ab8
Move p2p files to separate p2p directory.
i-zolotarenko Oct 31, 2023
19ec6af
Rename request container file and p2p loader and container files.
i-zolotarenko Oct 31, 2023
bcec5be
Add request-container, storage loggers. Use queueMicrotask instead of…
i-zolotarenko Oct 31, 2023
589eda6
Fix issue with skipping first segment in queue generation function.
i-zolotarenko Oct 31, 2023
0389b57
Add binary serialization functions.
i-zolotarenko Nov 1, 2023
a2f44b8
Remove bits file.
i-zolotarenko Nov 2, 2023
8af127c
Merge branch 'v1' into p2p-manager
mrlika Nov 2, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion p2p-media-loader-demo/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
"@types/react-dom": "^18.2.4",
"@vitejs/plugin-react": "^4.0.0",
"eslint-plugin-react-hooks": "^4.6.0",
"eslint-plugin-react-refresh": "^0.4.1"
"eslint-plugin-react-refresh": "^0.4.1",
"vite-plugin-node-polyfills": "^0.14.1"
}
}
268 changes: 218 additions & 50 deletions p2p-media-loader-demo/src/App.tsx
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import { useEffect, useRef, useState } from "react";
import React, { useCallback, useEffect, useRef, useState } from "react";
import { Engine as HlsJsEngine } from "p2p-media-loader-hlsjs";
import { Engine as ShakaEngine } from "p2p-media-loader-shaka";
import Hls from "hls.js";
import DPlayer from "dplayer";
import shakaLib from "shaka-player";
import muxjs from "mux.js";
import debug from "debug";

window.muxjs = muxjs;

Expand Down Expand Up @@ -49,12 +50,41 @@ function App() {
localStorage.player
);
const [url, setUrl] = useState<string>(localStorage.streamUrl);
const shakaEngine = useRef<ShakaEngine>(new ShakaEngine(shakaLib));
const hlsEngine = useRef<HlsJsEngine>(new HlsJsEngine());
const shakaInstance = useRef<shaka.Player>();
const hlsInstance = useRef<Hls>();
const containerRef = useRef<HTMLDivElement>(null);
const videoRef = useRef<HTMLVideoElement>(null);
const shakaEngine = useRef<ShakaEngine>(new ShakaEngine(shakaLib));

const [httpLoaded, setHttpLoaded] = useState<number>(0);
const [p2pLoaded, setP2PLoaded] = useState<number>(0);
const [httpLoadedGlob, setHttpLoadedGlob] = useLocalStorageItem<number>(
"httpLoaded",
0,
(v) => v.toString(),
(v) => (v !== null ? +v : 0)
);
const [p2pLoadedGlob, setP2PLoadedGlob] = useLocalStorageItem<number>(
"p2pLoaded",
0,
(v) => v.toString(),
(v) => (v !== null ? +v : 0)
);

const hlsEngine = useRef<HlsJsEngine>(
new HlsJsEngine({
onSegmentLoaded: (byteLength, type) => {
const MBytes = getMBFromBytes(byteLength);
if (type === "http") {
setHttpLoaded((prev) => round(prev + MBytes));
setHttpLoadedGlob((prev) => round(prev + MBytes));
} else if (type === "p2p") {
setP2PLoaded((prev) => round(prev + MBytes));
setP2PLoadedGlob((prev) => round(prev + MBytes));
}
},
})
);

useEffect(() => {
if (
Expand Down Expand Up @@ -150,6 +180,7 @@ function App() {
customHls: (video: HTMLVideoElement) => {
const hls = new Hls({
...engine.getConfig(),
liveSyncDurationCount: 7,
});
engine.initHlsJsEvents(hls);
hls.loadSource(video.src);
Expand All @@ -159,6 +190,7 @@ function App() {
},
},
});
player.play();
setPlayerToWindow(player);
};

Expand All @@ -180,10 +212,8 @@ function App() {
};

const createNewPlayer = () => {
if (!localStorage.videoUrl) {
localStorage.streamUrl = streamUrl.live2;
setUrl(streamUrl.live2);
}
setHttpLoadedGlob(0);
setP2PLoadedGlob(0);
switch (playerType) {
case "hls-dplayer":
initHlsDplayer(url);
Expand Down Expand Up @@ -214,54 +244,192 @@ function App() {
};

return (
<div style={{ textAlign: "center", width: 1000, margin: "auto" }}>
<div style={{ marginBottom: 20 }}>
<h1>This is Demo</h1>
<div style={{ textAlign: "start" }}>
<select
value={playerType}
onChange={(event) =>
onPlayerTypeChange(event.target.value as Player)
}
>
{players.map((player) => {
return (
<option key={player} value={player}>
{player}
</option>
);
})}
</select>
<select
value={url}
onChange={(event) => onVideoUrlChange(event.target.value)}
>
{Object.entries(streamUrl).map(([name, url]) => {
return (
<option key={name} value={url}>
{name}
</option>
);
})}
</select>
<button onClick={createNewPlayer}>Create new player</button>
<button onClick={loadStreamWithExistingInstance}>
Load stream with existing hls/shaka instance
</button>
<div style={{ width: 1000, margin: "auto" }}>
<div style={{ textAlign: "center" }}>
<div style={{ marginBottom: 20 }}>
<h1>This is Demo</h1>
<div style={{ textAlign: "start" }}>
<select
value={playerType}
onChange={(event) =>
onPlayerTypeChange(event.target.value as Player)
}
>
{players.map((player) => {
return (
<option key={player} value={player}>
{player}
</option>
);
})}
</select>
<select
value={url}
onChange={(event) => onVideoUrlChange(event.target.value)}
>
{Object.entries(streamUrl).map(([name, url]) => {
return (
<option key={name} value={url}>
{name}
</option>
);
})}
</select>
<button onClick={createNewPlayer}>Create new player</button>
<button onClick={loadStreamWithExistingInstance}>
Load stream with existing hls/shaka instance
</button>
</div>
</div>
<div style={{ display: "flex", justifyContent: "center" }}>
<div
ref={containerRef}
id="player-container"
style={{ width: 1000 }}
></div>
</div>
{!!playerType && ["hlsjs", "shaka-player"].includes(playerType) && (
<video ref={videoRef} controls muted style={{ width: 800 }} />
)}
</div>
<div style={{ display: "flex", justifyContent: "center" }}>
<div
ref={containerRef}
id="player-container"
style={{ width: 1000 }}
></div>
<div style={{ textAlign: "left" }}>
<LoadStat title="Local stat" http={httpLoaded} p2p={p2pLoaded} />
<LoadStat
title="Global stat"
http={httpLoadedGlob}
p2p={p2pLoadedGlob}
/>
<LoggersSelect />
</div>
{!!playerType && ["hlsjs", "shaka-player"].includes(playerType) && (
<video ref={videoRef} controls muted style={{ width: 800 }} />
)}
</div>
);
}

export default App;

function LoadStat({
http,
p2p,
title,
}: {
http: number;
p2p: number;
title: string;
}) {
const sum = http + p2p;
return (
<div style={{ textAlign: "left" }}>
<h4 style={{ marginBottom: 10 }}>{title}</h4>
<div>
Http loaded: {http.toFixed(2)} MB; {getPercent(http, sum)}%
</div>
<div>
P2P loaded: {p2p.toFixed(2)} MB; {getPercent(p2p, sum)}%
</div>
</div>
);
}

function LoggersSelect() {
const [activeLoggers, setActiveLoggers] = useLocalStorageItem<string[]>(
"debug",
[],
(list) => {
setTimeout(() => debug.enable(localStorage.debug), 0);
if (list.length === 0) return null;
return list.join(",");
},
(storageItem) => {
setTimeout(() => debug.enable(localStorage.debug), 0);
if (!storageItem) return [];
return storageItem.split(",");
}
);

const onChange = (event: React.ChangeEvent<HTMLSelectElement>) => {
setActiveLoggers(
Array.from(event.target.selectedOptions, (option) => option.value)
);
};

return (
<div>
<h4 style={{ marginBottom: 10 }}>Loggers: </h4>
<select
value={activeLoggers}
onChange={onChange}
multiple
style={{ width: 300, height: 150 }}
>
{loggers.map((logger) => (
<option key={logger} value={logger}>
{logger}
</option>
))}
</select>
</div>
);
}

function getPercent(a: number, b: number) {
if (a === 0 && b === 0) return "0";
if (b === 0) return "100";
return ((a / b) * 100).toFixed(2);
}

function round(value: number, digitsAfterComma = 2) {
return Math.round(value * Math.pow(10, digitsAfterComma)) / 100;
}

function getMBFromBytes(bytes: number) {
return round(bytes / Math.pow(1024, 2));
}

function useLocalStorageItem<T>(
prop: string,
initValue: T,
valueToStorageItem: (value: T) => string | null,
storageItemToValue: (storageItem: string | null) => T
): [T, React.Dispatch<React.SetStateAction<T>>] {
const [value, setValue] = useState<T>(
storageItemToValue(localStorage[prop]) ?? initValue
);
const setValueExternal = useCallback((value: T | ((prev: T) => T)) => {
setValue(value);
if (typeof value === "function") {
const prev = storageItemToValue(localStorage.getItem(prop));
const next = (value as (prev: T) => T)(prev);
const result = valueToStorageItem(next);
if (result !== null) localStorage.setItem(prop, result);
else localStorage.removeItem(prop);
} else {
const result = valueToStorageItem(value);
if (result !== null) localStorage.setItem(prop, result);
else localStorage.removeItem(prop);
}
}, []);
const eventHandler = useCallback((event: StorageEvent) => {
i-zolotarenko marked this conversation as resolved.
Show resolved Hide resolved
if (event.key !== prop) return;
const value = event.newValue;
setValue(storageItemToValue(value));
}, []);

useEffect(() => {
window.addEventListener("storage", eventHandler);
return () => {
window.removeEventListener("storage", eventHandler);
};
}, []);

return [value, setValueExternal];
}

const loggers = [
"core:hybrid-loader-main",
"core:hybrid-loader-main-engine",
"core:hybrid-loader-secondary",
"core:hybrid-loader-secondary-engine",
"core:p2p-loader",
"core:peer",
"core:p2p-loaders-container",
] as const;
1 change: 1 addition & 0 deletions p2p-media-loader-demo/src/vite-env.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/// <reference types="vite/client" />
9 changes: 5 additions & 4 deletions p2p-media-loader-demo/tsconfig.json
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
{
"compilerOptions": {
"target": "ESNext",
"lib": ["DOM", "DOM.Iterable", "ESNext"],
"target": "ES2020",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "node",
"allowImportingTsExtensions": false,
"moduleResolution": "Bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
Expand All @@ -15,6 +15,7 @@
"noUnusedParameters": false,
"noFallthroughCasesInSwitch": true,
"allowSyntheticDefaultImports": true,
"useDefineForClassFields": true
},
"include": ["src"],
"references": [{ "path": "./tsconfig.node.json" }]
Expand Down
2 changes: 1 addition & 1 deletion p2p-media-loader-demo/tsconfig.node.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"composite": true,
"skipLibCheck": true,
"module": "ESNext",
"moduleResolution": "node",
"moduleResolution": "Bundler",
"allowSyntheticDefaultImports": true
},
"include": ["vite.config.ts"]
Expand Down
3 changes: 2 additions & 1 deletion p2p-media-loader-demo/vite.config.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import { nodePolyfills } from "vite-plugin-node-polyfills";

export default defineConfig({
plugins: [react()],
plugins: [nodePolyfills(), react()],
});
5 changes: 5 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,10 @@
},
"dependencies": {
"debug": "^4.3.4"
},
"pnpm": {
"patchedDependencies": {
"[email protected]": "patches/[email protected]"
}
}
}
Loading