Latest watchmarker version broke YouTube watched badge detection

[lbmaian]

Version:
4.5.1 Chrome extension

Steps to reproduce:

1. Either stay logged out.
2. In Watchmarker settings, uncheck all conditions EXCEPT leaving "Mark a video as watched once it has the official Youtube watched badge." as checked.
3. Go to Youtube homepage or a channel page then add at least two unwatched videos to the queue to create a queue playlist.
4. Let Youtube finish playing some videos in the queue.

Update: Seems to be broken even when logged in.

Expected result:
The videos except maybe the last all become marked as watched by Watchmarker. The last may not be marked since it doesn't show the Youtube watched badge yet (until adding another video to the queue).

Actual result:
None of the videos are registered as marked by you

There are other ways to reproduce, since YT recommendations and playlist items with YT watched badge also no longer register as watched in Watchmarker.

I believe the culprit is something has changed that removed or broke the usage of youtubeEnsure messages from YT-page-injected code.

[sniklaus]

Thanks for bringing this up! I opened a fresh instance of Firefox, installed the Watchmarker, and configured it as follows:

I then added some videos from a channel to the queue and let them play. After they were done playing I refreshed the page and none of them were marked as watched. However, they also don't have a red progress bar underneath them so the "Mark a video as watched once it has the official Youtube watched badge." doesn't apply. So this seems like expected behavior, no?

[lbmaian]

That's strange. Just installed the latest watchmarker on Firefox, copied those settings, remained logged out, added some music vids to the queue, and they definitely do show YT watched progressbars

They just don't register as marked in watchmarker.

In fact, I just tried enabling all the conditions and nothing is getting marked. I think something's gone wrong with the "client-side" video watched detection. If you're logged in, YT watch history condition still seems to work, but nothing when you're logged out.

[lbmaian]

Actually scratch that last part. I was doing that in a personal container and watchmarker apparently isn't compatible with multi-account containers? edit: Also means these conditions probably won't work in private windows as well in Firefox.

Doing it without any containers, the "opened in the browser" and "appears in browser history" conditions do work, but "official Youtube watched badge" still doesn't despite showing YT progressbars.

[sniklaus]

I just logged in, then it showed the progress bar underneath videos I watched when logged in, and it successfully added them to the Watchmarker's watch history. I then logged out and all the progress bars disappeared (the Watchmarker correctly kept marking videos from it's history database as watched). I am under the impression that Youtube doesn't show the progress indicator unless you are logged in, so I am not sure what is going on. 🤔

What has changed in the recent update is the mechanism of how Youtube's progress indicator is being used. That is, in the old version of the Watchmarker it was being inferred from the DOM but there were false positives due to race conditions (Polymer will reuse a DOM elements, update a video title to that of a new video but not remove the progress indicator yet, then the Watchmarker may discover the updated DOM element before Polymer completed the update and removed the progress indicator). The new version monitors the network traffic instead to fix this (percentDurationWatched attribute in certain API calls).

Considering your excellent previous contributions to the Watchmarker, you are probably skilled enough to debug what is going on. I would do it myself but I haven't been able to replicate this behavior on my end yet. The relevant code is this and this. Maybe adding a debugger; in there and stepping through what is going on might reveal some useful information.

[lbmaian]

Hmm, I'll try to look into it but it'll have to be later.

The easiest thing to do would be a "revert" of sorts, where the old method is still available perhaps under an option.

Glancing into the linked new code, the usage of chrome.tabs.executeScript to inject a script is iffy since I was under the impression that MV3 is going to restrict this, and I believe the MV3 migration is going to be required this year.

[sniklaus]

The easiest thing to do would be a "revert" of sorts, where the old method is still available perhaps under an option.

I am not sure why the old approach would work for your case, something else seems off. If they have a progress indicator then that needs to be in the traffic somewhere and it should be picket up by the new mechanism. 🤔

Glancing into the linked new code, the usage of chrome.tabs.executeScript to inject a script is iffy since I was under the impression that MV3 is going to restrict this

Thanks for the heads-up! We can add that script to this one instead. The only reason I am using executeScript because I have access to the settings in background.js but not easily in a client script (so it kept the code easier).

I believe the MV3 migration is going to be required this year.

I am still sitting this one out for as long as possible (hopefully forever). Forcing all developers to sink an exceptional amount of time into the migration just to crack down on adblocking isn't a game that I want to participate in.

[lbmaian]

I am not sure why the old approach would work for your case, something else seems off. If they have a progress indicator then that needs to be in the traffic somewhere and it should be picket up by the new mechanism. 🤔

Well I noticed this issue because it DID work before. At least on Chrome, and given my image above, should've also worked on Firefox.

If the YT watch progressbars aren't appearing for you when logged out, I suspect it's one of those Youtube A/B testing experiments that are specific to your "account" (even when logged out, YT assigns some sort of persistent session via cookies for you)

[sniklaus]

Good guess regarding the A/B testing. 👍

To achieve the behavior that you expect, we could add another mechanism to detect whether a video is being watched: intercept the network traffic for video playback. If we notice that a video is being watched we mark it as such. This could replace the current mechanism for "Mark a video as watched once it has been opened in the browser." altogether.

[lbmaian]

I thought about something like that. But I'm worried about the cases where the video doesn't autoplay (disabled by an extension, or in an embed with autoplay disabled) and YT prefetches video data but doesn't actually play it yet. YouTube itself doesn't consider the video watched yet, so it shouldn't be marked as watched in this case.

There must be some other mechanism in play that YouTube uses to recognize actual playback rather than just fetching video data.

[lbmaian]

I think the YT watched badge detection is broken even when logged in. I thought it was working before but that was actually just the YT watch history detection eventually marking the videos as watched.

[GaabluhnMowd]

Can confirm broken even when logged in. Have to perform a Sync from options page for it to detect watched status.
Version 4.4.2 still working as expected.

[sniklaus]

Not sure why this is happening for you folks, works just fine for me. Below is Firefox on the left and Chromium on the right, both having the Watchmarker installed. When I watch a video on the left and refresh the page on the right it gets marked and vice versa. That can only happen through the badge detection. Not sure how I can reproduce the behavior you are experiencing to debug this. 🤔

2024-01-06.14-37-48.mp4

[lbmaian]

The badge detection may work on channel pages (and maybe home and subscription pages), but it's not working for playlist/queue entries for me. And I think in earlier testing it also didn't work on recommendation entries.

[sniklaus]

I just made some modifications, does the attached version improve things? Make sure to backup your database before you try.

• chrome.zip
• firefox (rename the extension to xpi).zip

[lbmaian]

Preliminary testing on firefox:

YouTube watched badge detection now works sometimes for playlists. I can't reproduce it consistently but it sometimes somehow gets into a state where the detection breaks on the page and no longer works even after queuing more videos or playing different videos within the playlist. And then it sometimes it resumes working for a video. I can't make rhyme or reason of it.

The logging isn't helpful either. There's a lack of "ensure video <video obj>" in the logs, both on the YT page and in the extension's background page, only a bunch of "lookup video <video obj>". Oh, there's some "ensure video" but no indication of what video it was.

I wonder if the problem is that the script isn't being injected correctly. I see that it's trying to override XMLHttpRequest.prototype.open or fetch. Is there a particular reason the extension is doing this client-side rather than via chrome.webRequest? edit: Oh wait, maybe reading response body isn't available in that api?

[sniklaus]

The new version looks for the player telling Youtube how much of the video has been watched. If the video is not muted and more than one minute has passed then the extension considers the video as watched. Maybe the "1 minute" condition didn't hold true in your test? And yeah, I should improve logging.

Oh wait, maybe reading response body isn't available in that api?

That's exactly it and it is very frustrating not to be able to intercept the body. After all, I can do it myself in a janky way, so why not provide the feature out of the box in the browser extension interface. 🤷

[lbmaian]

No it's way past 1 minute.

Another example where the badge detector simply isn't working despite going through that whole queue:

I'm not sure if your http interceptors are actually firing - maybe add some logging there to verify.

Hmm, this is interesting. When I look in the debugger, the injected code shows as "sandbox eval code":

I do see the added script element, but I'm not sure if it's actually executed due to the lack of a nonce:

Can you try moving all that functionality into the injected youtube.js instead to avoid the possibility of script non-execution?

[sniklaus]

Can you try moving all that functionality into the injected youtube.js instead to avoid the possibility of script non-execution?

Unfortunately not, the XMLHttpRequest inside of youtube.js is in a different scope from the first-party Youtube code. So trying to apply the hook in there is a futile effort. 🙃

[sniklaus]

By the way, hooking into XMLHttpRequest and fetch is for figuring out which videos have a progress bar such that they can be marked. The progress for intercepting the watchtime reporting is here:

youtube-watchmarker/background.js

Lines 1905 to 1949 in 81ddf26

	chrome.webRequest.onSendHeaders.addListener(function(objData) {
	if (objData.url.indexOf('muted=1') !== -1) {
	return;
	}
	for (let strElapsed of objData.url.split('&et=')[1].split('&')[0].split(',')) {
	if (parseFloat(strElapsed) < 60.0) {
	continue;
	}
	let strIdent = objData.url.split('&docid=')[1].split('&')[0];
	let strTitle = strTitlecache.hasOwnProperty(strIdent) === true ? strTitlecache[strIdent] : '';
	if (strIdent.length !== 11) {
	continue;
	} else if (strTitle === '') {
	continue;
	}
	Youtube.ensure({
	'strIdent': strIdent,
	'strTitle': strTitle
	}, function(objResponse) {
	console.debug('ensure video');
	});
	chrome.tabs.query({
	'url': '*://www.youtube.com/*'
	}, function(objTabs) {
	for (let objTab of objTabs) {
	funcSendmessage(objTab.id, {
	'strMessage': 'youtubeMark',
	'strIdent': strIdent,
	'intTimestamp': 0,
	'strTitle': strTitle,
	'intCount': 0
	});
	}
	});
	}
	}, {
	'urls': ['https://www.youtube.com/api/stats/watchtime*']
	});

[lbmaian]

Oh, the 1 min thing was for the new watchtime detector. That does work in my testing.

Though the 1 min requirement shouldn't be there. In my experience, the moment there's any actual playback (presumably the first watchtime request), it counts as watched and the watched badge appears for it. There are also many videos that are shorter than 1 minute, notably all the #shorts crap.

Unfortunately not, the XMLHttpRequest inside of youtube.js is in a different scope from the first-party Youtube code. So trying to apply the hook in there is a futile effort. 🙃

Well that sucks. I wonder how Tampermonkey and other userscript managers do this, since I've intercepted XHR just fine in YT userscripts before.

[sniklaus]

I reduced the 1 minute requirement to 3 seconds, feel free to give the attached versions a try.

• chrome
• firefox (rename to end in .zip)

Well that sucks. I wonder how Tampermonkey and other userscript managers do this, since I've intercepted XHR just fine in YT userscripts before.

Good question. I also wonder whether other extensions are preventing the Watchmarker's hooks (since they work fine for me).

[lbmaian]

The YT watchtime detection does work, but 3 seconds is still insufficient for extremely short videos. For example, top result for "1 second video" for me: https://www.youtube.com/watch?v=MvsAesQ-4zA

YT badge detection still doesn't work for me. I'm going to try this later on chrome when I have time, since firefox devtools isn't showing me the injected code so I can't even try figuring out if the injected code is actually working.

[GaabluhnMowd]

Was not able to test the xpi versions in this thread, but latest update 4.6.1 seems to be working as expected upon initial testing.