I'm wondering if any of the geniuses here on the BBS might be able to help with this one.

I've posted this issue in the YouTube community forum already, and I'm hoping someone there can help me, but I'm guessing that the answer is going to come back "that's just the way it works" and I'm going to have to invent my own workaround. So I'm looking to you guys for help with the workaround.

I have a YouTube live stream where we feed birds on the back porch. My girlfriend loves the local crows and feeds them at home and at her day job. They're highly intelligent, very playful, and recognize her by her face. She asked me to set this camera up for her. The camera stream automatically begins every morning at about sunrise and then we put out food for the birds. Then the stream ends every evening around sunset. So the stream lasts for the daylight hours of a single day, each day. I'm currently using my Synology NAS as the source for the stream, using its security camera feature to manage the camera and the live stream. I use a Bash script to control when the camera turns on and off based on the time of year and the position of the sun, and even to restart the stream after a network interruption, if one occurs (works around an existing bug in the Synology).

I'm at the point where I'm debating whether to continue to improve this thing (better camera, better mounting, better stream quality, better system for feeding the birds so the seagulls can't get any of the crows' food, more improvements to the Bash script, etc.), or to give up entirely and take it down, because it's not working the way we want it to work.

Here's the problem description:

Normally when viewing a YouTube live stream, you can scroll backwards in the timeline by approximately four hours and see things that have happened during the day.

However there is a problem: If no one else is watching the channel, this "scroll backwards in the timeline" does not work for the times when no one is watching. There is a gap in the timeline whenever no one is watching the stream. I can tell this is happening because the webcam has a timestamp on the screen, so I can clearly see the gaps in the timeline. Here is a specific example:

- Stream starts at 7am.
- Put out food for the birds around 8-ish AM and leave the house for our day jobs.
- At noon, during the day at my desk at my day job, I open up the live stream and look at it.
- If no one else has been watching the live stream, it will show the following in its scrollback history: Approximately one hour of visible history will exist from about 7am to about 8am. Then there is a "jump" in the history, and the timestamp on the webcam video feed skips from about 7am to about noon, and the visible history resumes at noon, which is the time when I started watching the stream from my desk at my day job.
- This problem is only visible in the live stream during that day. If you later go back and look at the video archive of the prior day's stream, the whole thing is there. So I know that the problem is only with the "Live DVR" type functionality, it is not with the upstream video feed (since the whole thing appears in the archive).

In other words, the YouTube scrollback DVR-history of the live stream only activates when people are watching the stream. It shuts down after a while if no one is watching it.

I want myself, and, anyone else tuning into the stream, to be able to scroll back in the stream and see what has happened during the time period from 8am to Noon, but that is not always possible. The reason this is important is that most of the actual activity happens in that range of 8am to Noon, and that's the part that's usually getting dropped. For example, today the dishes got put out around 8am and all the food was gone by Noon, and the stream literally jumps over the entire time that the dishes were out and had food in them. The stream shows an empty table from 7am (approx sunrise) to 8am (before the food gets put out), then jumps to Noon where empty dishes magically appear because the food has already been eaten.

Okay, so that's the problem statement. Assuming that I don't get any solution from YouTube about this, I'm looking for a good work-around that I can implement.

Some things I have tried so far:
- Changing all the stream quality and latency settings, including those on the camera, those in the Synology, and those on the YouTube channel. No alteration of the problem with any of these changes.
- Deliberately bouncing the stream in the middle of the day at solar Noon. This is actually worse since it splits the stream and there's no scrollback at all after the split point. If you tune into the the live stream prior to the split point, the same problem occurs anyway.
- The most ambitious thing I have tried so far: I have tried using the Bash script on my Synology to pretend that it's a viewer of the stream; the Bash script performs a "Wget" on the streaming page at one-minute-intervals. This does not fix the problem, most likely because Wget only loads the page's HTML and doesn't actually engage the video stream at all.

I'm wondering if any of you know a secret trick where I could use Wget to figure out how to engage the feature that makes YouTube think it's really watching the stream itself as opposed to just loading the parent web page. Or something similar. Or perhaps some other idea I haven't thought of.

Any ideas?

To create a dummy viewer, can you just start a browser in a VM/docker to load the stream's page in a browser via a cron job and similarly shut it down at the end of the day? i.e. basically what your wget is attempting.

Not sure how that might go with some ads where you have to click to get to the stream, but I reckon some AutoHotKey magic could do the clicking for you even.

Alternatively, why YouTube?

The SecCam features of the NAS probably include a web-viewing feature that you could export over VPN or SSH. Allowing direct web access to it (probably the default setup) would be exceptionally unwise though, which I suppose is a reason to rely on YouTube instead?

You do run the risk with this of the possibility of running foul of YouTube's fake ad click detections systems. So be careful, if having your YouTube channel deleted would be a problem.

You all bring up excellent points, thank you very much. Some specific responses to questions that were posed:


Good point. Indeed such a scheme would likely work to solve the problem. However I don't think I can spend that level of complexity and CPU usage on my little Synology NAS. It frequently gets asked to do more intensive things like transcode video streams on the fly, and I don't want to make that job harder by filling its little head with a massive VM/browser implementation just for this.

I'm hoping to find a command-line equivalent that will work in a bash script. I believe that Wget isn't functioning here because it merely downloads the page's initial HTML and doesn't execute any javascript or load any media files. If I could figure out which bit of the YouTube HTML code points to the "thing" that I need to load to get this thing to "trigger" so that YouTube does the right thing, I could do that. I just don't know what that "thing" is.



Excellent question. It's the most popular video platform, and I want the video to be easy for us, and anyone else, to find. My long term goal is to improve the camera stuff and improve the feeding stuff so that the stream is actually interesting for more hours during the day and maybe some folks besides us would like to watch it.

I'm also using it as a testbed for learning how to properly manage video streams, trying to learn the pitfalls of YouTube itself specifically, so that I could do live concerts from our band which are reliable. (Many amateur bands I've seen that try to stream their shows have problems getting the stream started and keeping it reliable.)



Whoa, you have a really good point there. That might put the kibosh on this whole idea.

I'm not familiar with the NAS, but if you can run wget, perhaps you can try youtube-dl, instead? Otherwise, I'd pursue what Mark suggests -- why involve YouTube at all?

I checked out the stream. I could rewind to 8:04am, where there was a gluttonous seagull pacing back and forth on the railing. It ate the majority of what food was there. A crow got one piece of food, a squirrel got two pieces of food, and then another seagull came and ate the rest, and tried to dump out the water to spite anyone else. The food was gone by 8:40am. If this is normal, I don't think you can say you're feeding the crows... you're feeding the seagulls.

Indeed, one of the problems I need to solve is how to feed the crows without also feeding the seagulls.

I'll look into youtube-dl, thanks!

I poked at youtube-dl a bit and it is very promising. I'll poke at it some more.

It's my hope that if I only run it briefly every half hour or so, and randomize its runtime slightly each time, then hopefully it won't trigger bot detection.

Thank you!

I have a script working with youtube-dl that (cautiously optimistic) appears to solve the problem in initial tests. I need it to bake for a few days to be sure, but things seem to be working correct-ish now. Thank you so much, canuckInOR!

There was an interesting technical snag with it that I'll recount here for fun:

Youtube-dl needs ffmpeg. It calls out to ffmpeg to get some of its work done. What's really interesting is that youtube-dl doesn't actually download the data itself and then feed it to ffmpeg, no, it actually shells out to ffmpeg to get the downloading done too, once it's worked out the actual final URL with all the doodads and geegaws added in.

That's fine though, because the Synology distro already has ffmpeg installed for its video transcoding functions.

But there's a problem. The version of ffmpeg that's installed/running in the Synology distro doesn't work properly with youtube-dl. The youtube-dl program seems to start up OK and do its initial work, but then it tries to launch ffmpeg, which then displays an error message saying that it can't download from HTTPS sites and that I have to recompile it if I want that. (Of course, since the Synology uses ffmpeg for its transcoding I can't be recompiling it willy nilly.)

I can't just change the URL to HTTP instead of HTTPS, because I tried that, and after the youtube-dl processing where the URL doodads and geegaws get put in there, it changes it to HTTPS anyway.

Fine, I say, I'll just use the "--hls-prefer-native" or the "--external-downloader wget" parameters to youtube-dl. Either one of those should have solved the problem. NOPE! Those parameters actually don't work at all and have no effect in the current version of youtube-dl. I put the parameters in there, but ffmpeg still tries to download the video and still gives me the error message saying it can't download HTTPS. I will someday file a bug against youtube-dl once I can gather all the necessary logs and stuff, but they're like 2200+ bugs deep at the moment, and I don't expect they'll ever get round to fixing it at this rate.

Last ditch effort: Use this youtube-dl parameter: "-g, --get-url Simulate, quiet but print URL". This works to get me the final URL with all the doodads and geegaws added. Which I then farm out to wget myself.

This seemed to work, but looking closely at what got downloaded, I see that it doesn't actually start downloading real video. It just downloads a text file which catalogs additional URLs to sections of the video stream.

Before I went any further, I just thought, you know what? Let's just see if that alone, is enough to keep the stream alive. And left it that way.

So far? Seems to work. Crossing my fingers.

Thanks again!

Clarification: The "--hls-prefer-native" and "--external-downloader wget" parameters do indeed get processed, but they don't prevent the error message in ffmpeg.

Update:

- All my code for this little project is now at This GitHub Page, including full documentation.

- Code-wise, it's working really well, thanks to the help from everyone on this BBS.

- I am still in the process of trying to solve the physical problem of the seagulls eating the food before the crows get to it. I have modified a "squirrel proof" bird feeder so that it will accept the weight of crows and squirrels, but a seagull is too heavy. It works perfectly, but the crows are skittish about the feeder and are not yet using it regularly. My hope is that, long-term, the crows will get used to the feeder and be able to use it unhindered all day long, allowing for more crow activity on the camera during the day.

Examples of the feeder at work:
Seagull thwarted by feeder: https://youtu.be/3M-P3IpzxM8?t=8791
Crows tentatively using the feeder: https://youtu.be/EFMM2s0IonI?t=10355

In those examples, we've put some additional food out on the platform to attract the crows, but long-term, we expect to put the food only in the hopper, once the crows learn to use the feeder.

Thanks again for all your help!

Crows are pretty smart birds, they also hold grudges.

Indeed, seeing that very research, by John Marzluff, is specifically what inspired Vixy to start feeding them in the first place. Here's a TED talk that he did on the topic.