How To Find Embedded YouTube Videos That Are Unavailable or Marked Private Using Screaming Frog

Estimated reading time: 9 minutes

• About The Author

There’s nothing more frustrating than coming across articles containing YouTube videos that have been removed for some reason. It’s extremely frustrating for users and can easily elude site owners (especially on large-scale sites). And from an SEO perspective, addressing unavailable videos can avoid UX barriers and unhappy users. So join me as we go hunting for pages containing YouTube videos that never play.

When covering major algorithm updates, and especially broad core updates, I have explained many times the importance of meeting or exceeding user expectations. I’ve also explained that Google has an uncanny ability to understand when users are not happy, which over time could lead to impact from major algorithm updates (like broad core updates)… And now that we know about Navboost, and how Google leverages user interaction signals to impact rankings, we understand how Google can learn if content is meeting user expectations.

So when auditing sites, I’m on the lookout for situations where users can get frustrated, not get what they need based on query, etc. As an example, there’s nothing worse than visiting a page to learn about a topic, trying to watch video supporting the content, and the video has been removed or marked private. That’s a huge UX barrier in my opinion and something I have found often during audits when helping companies impacted by major algorithm updates (and especially broad core updates). If the video was extremely important for the page, then how could that content meet or exceed user expectations? And what kind of impression are you leaving when there are broken media elements on your pages?

Maybe the site owner or author once had a great video to share that supported the content, but the YouTube video was removed for some reason. For example, maybe the person publishing that video violated a YouTube policy or they could have marked it as private. We have all seen the dreaded “This video is unavailable” message when trying to watch an embedded YouTube video and it’s never a good feeling…

Hunting Broken YouTube Videos:
OK, so we know there’s a major user experience issue when people come across embedded YouTube videos that have been removed for some reason, but how do we go about finding those videos? Well, that’s the problem. There hasn’t been a great way to surface those videos historically. And that’s especially the case on large-scale sites with tens of thousands, hundreds of thousands, or even millions of urls. You really can’t go url by url trying to find those issues…

So again, there was never a great solution for surfacing those YouTube videos that were removed. Until now. 🙂

Sending A ‘Frog Signal’ For Help!
As I was auditing yet another large-scale site with this problem, I decided to call in the big guns. I shot an email to Dan Sharp from Screaming Frog. It was like sending a bat signal, but green and with a frog. 🙂 It didn’t take long for Dan to get back to me explaining he was looking into it. And then not long after, boom, he sent an email explaining how to tackle the situation.

Dan explained that you could crawl a site adding YouTube as a CDN (so it’s treated internally), use custom search looking for error messages that get displayed when a video has been removed, make sure JavaScript rendering is on and that .js files can be crawled, and then you could surface those videos that were removed.

I was excited to try it out… so I fired up Screaming Frog, used the configuration that Dan explained, and crawled a small site that had YouTube videos that had been removed.

And I saw this:

Amazing! The video problem plaguing Gotham would soon be resolved! OK, enough with the Batman references.

Now, this was great, but the approach required you crawl an entire site. What if I wanted to crawl a set of urls looking for this problem? That would be incredible since you could export top urls from GSC and then crawl them (since if you read one of my recent posts, you know that highly visible AND lower-quality is a most dangerous SEO combination). Maybe you just want to crawl the most visible urls that dropped during a recent algorithm update

I emailed Dan about this, and he once again came up with a great solution. For the updated configuration, you need to add the urls you want to crawl to the Include tab, including youtube.com and .js files.

Then you need to place Screaming Frog in List Mode and crawl away. After the crawl is completed, you can check the Custom Search tab to view the urls being returned that meet the criteria. For example, YouTube videos that were either marked as private or are unavailable.

Let’s run through a quick example for checking a specific set of urls. Maybe these were the top urls that dropped during a major algorithm update (hint-hint).  

Step-by-step: How To Surface YouTube Videos That Are Unavailable Or Marked As Private Based On A Specific List Of URLs

1. Fire up Screaming Frog and enter List Mode.

2. Add YouTube as a CDN.
Click the configuration menu and then CDNs. Add https://www.youtube.com/embed/ in the list. Those urls will now be considered internal urls during the crawl.

3. Make sure JavaScript rendering is on.
Click the Configuration menu, Spider, and then Rendering. Use the dropdown and select JavaScript.

4. Choose Googlebot Smartphone as the user-agent.
Click the Configuration menu and then User-agent. Select Googlebot smartphone as the user-agent.

5. Click the Configuration menu and then Include. We want to add the urls that should be checked. For a test, maybe just enter ten of those urls. Tip, you can mix in urls that don’t have video problems just to make sure all is working ok.

6. Make sure youtube.com and .js are in the Include list! We want to crawl YouTube.com like it was a CDN (internal) and then we want to crawl JavaScript files.

7. Add your Custom Search filters.
Click Configuration, Custom, and then Custom Search. Click the +Add button in the bottom right corner. Enter a label for the first filter (like “Video unavailable”). And then Contains, Text, and enter “Video unavailable”, and select Page Text from the final drop down. Complete this for “The video is private” as well. You will end up with two custom search filters. Click OK to finish.

8. Upload Your List of URLs.
This is the same list you added to the Include tab. You do not need to include youtube.com and .js in this list. It should just be the list of urls you want to check from the site you are auditing. Click Upload in the top menu, and then select the way you want to enter the urls (paste, manually, via a file, etc.)

9. Crawl Away!
Click “Start” and run your crawl. Grab your beverage of choice as you wait for Screaming Frog to do the heavy lifting.

10. Check the Custom Search results.
Click the Custom Search tab and then use the dropdown menu to select one of the reasons the videos didn’t resolve. If you click one of the YouTube videos in the list and check the Inlinks tab, you can see the url where it was embedded.

11. Export by YouTube issue.
You can also export all of the urls surfaced by clicking the Bulk Export menu, Custom Search, and then by custom search filter.

That’s it! Now you have a solution for surfacing those dreaded YouTube embeds that are unavailable or marked as private.

How Google Could Help: Provide Better Video Indexing Reporting in GSC.
Some of you might be wondering why GSC doesn’t surface those urls with videos that have been removed. It does have an error category labeled, “Video not found on host service”. That would be a solid report for surfacing all of those videos that won’t resolve… but it doesn’t. If Google could simply list all videos that aren’t available on the source service (even beyond YouTube), that would be incredible. I’ll ping Google about this to see if something like that is possible.

Final tips and recommendations for site owners looking to surface YouTube videos that are unavailable or marked as private:

You can crawl an entire site hunting YouTube videos that have been removed, but crawling a set of urls might be a better solution for larger-scale sites. For example, exporting urls via the GSC API based on some criteria and then running those through the process I documented in this post.

Don’t forget to add youtube.com and .js to the Include tab. That’s necessary in order for this to work properly.

Make sure JavaScript rendering is on and make sure you are using a mobile user-agent. I recommend Googlebot smartphone.

You can add other YouTube errors to catch via Custom Search if you want. If you notice other messages on your site beyond what I listed in this post, just add another custom search filter.

Watch the number of urls you are crawling per second. I would start lower and watch for any issues. Then adjust as necessary.

Remember that you can export the results by custom search filter, which can be extremely handy for sending to clients or other members of your team.

Improving the user experience by hunting down YouTube videos that never play:
I hope you can utilize this solution to help your company or clients surface YouTube videos that are unavailable or marked as private. Again, there’s nothing more annoying or frustrating than coming across articles with embedded YouTube videos that don’t play. And since we know Google is leveraging user interaction signals for rankings, I would always make sure you can meet or exceed user expectations based on query. Finding those videos is definitely one step in the right direction.

Now go hunt some unavailable YouTube videos. 🙂

GG

The post How To Find Embedded YouTube Videos That Are Unavailable or Marked Private Using Screaming Frog first appeared on The Internet Marketing Driver.

About The Author

artificialrace.com

See author's posts