Sample Size Bias

The sample size bias is one of the biggest pitfalls I've witnessed with YouTubers (particularly NewTubers) that I speak to. While the term isn't technically "real" in a sense (as trolls love to tell me), the issue is still prevalent.

It's pretty self-explanatory (which is the exact reason I chose that name for the concept).

It's a statistical bias, that forms due to something to do with a dataset's sample size. Shocking, I know.

Snarkiness aside, a sample size bias (in the context of content creation) occurs when a creator starts looking at their calculated statistics such as Click-Through Rate (CTR), Average View Duration (AVD), Viewed Vs Swiped Away (VVSA), etc. and begins making major decisions or judgements about their video which has low views.

The issue here is that the "low views" in question are not representative of a substantial or even relevant audience, making any "objective statistics" to be misleading at best.

For example, if a thousand random people off the street probably won't want to watch a Minecraft video, so if you gave them the option, maybe only tens of them (let's pretend, 50 people) would watch.

This translates to a `50/1000*100` = 5% CTR. Which hey, is pretty average!

Now let's say that of these 50 people, 35 of them clicked off within the first 30 seconds (which is VERY realistic!). That 70% drop off is going to absolutely tank the video's AVD.

So we now have this video with an alright CTR, and a terrible AVD. So this means that the packaging (title and thumbnail) is good, but the video is pretty bad, right?

Wrong, or well, not necessarily.

To avoid more napkin math, just consider the fact that this hypothetical creator has decided that the video is "bad" due to the judgement of just 50 people. In a world where there are millions of views to be had, does that sound reasonable?

Don't let a miniscule of your potential audience sway how you fundamentally make videos. Those hypothetical 50 people might only like Minecraft PvP videos, while the video itself is a building guide.

Would MrBeast let 50 random Grandmas dictate his editing style? Are you going to let some snotty-nosed child with no attention span tell you that your video is boring?

Wait until you have some real numbers before looking at those stats.

Okay, shut up.

...Oh you're still here? Great. Scroll down to this section.

Well, while the phrase's existence is arguably real, the raw concept itself is perfectly valid.

"Sample size bias" is sort of a summarised phrase I use that encapsulates the concept that undersized datasets are not usable to draw valid conclusions.

For example, 200 viewers is not a decent representation of "the masses" on YouTube. 200 random adults on the street probably won't watch a Minecraft video even if you paid them to, but there are millions upon millions of people who would.

If 1 person can affect your metrics by >1%, then your metrics don't count.

There's no hard rule, but a good rule of thumb: wait until you have enough views that a single viewer moving the needle by more than ~1% is unlikely. For most, that's in the thousands, not hundreds.

Ask real people — ideally from NewTubers, or other creator communities — what they actually thought. Qualitative feedback beats a 50-person sample every time.

Let videos breathe. YouTube's algorithm needs days, not hours, to find an audience. Judging a video at 200 views is judging it before it's been tried.

Separate "the stats say it's bad" from "I can see a real flaw in the editing/pacing/title". The latter is useful; the former on a tiny sample is noise.

The only reliable way to get a sample size worth analysing is to keep putting work out. Volume is what makes the stats meaningful.

Small samples lie. Don't let a handful of viewers dictate how you make videos. Get feedback, be patient, stay critical, and keep uploading until the numbers actually mean something.