> users consider each title for a whopping 1.8 seconds
Users are most likely not “considering” them at all. We’re not machines going over one tile at a time and generating a score. Most of the time you’re looking for something specific and just trying to find it.
This type of metric is the worst. I really hope this data-driven fad dies down and we start designing with human factors in mind again.
I'm not certain what you mean by human factors, so correct me if I've misinterpreted.
If by human factors you mean basic heuristics based on human intuition, that fails too. It might satisfy 50% of viewers but there's too much diversity in taste for this to be a good model.
I'm not defending the metric as described, but there's obviously good data driven approaches and bad ones. The trick is to find a solid proxy for the business problem and to try to weed out any dark side effects by applying it.
"Human factors" could include doing something as radical as observing how someone uses your product (no, not with a computer - just watch) and then - and this is the shocking part - speak to them. Ask them why they did that or what they liked and didn't like.
What fantasy world are you living in where you think Netflix and Facebook don't already do this? They constantly run paid interviews with people to source this information.
The issue is many-fold: 1) formalising multiple half hour interviews into something you can A/B test is non-trivial, 2) given the logistics of these interviews the sample you end up with is small and therefore bias.
I worked for the company running the most A/B tests in the industry for most of my career.
Having a user testing session is relatively rare, saved for special one-off products/features. Actually having feedback from those sessions distilled into product feature proposals, and getting them into the product roadmap is even rarer.
In the meantime, hundreds of A/B tests based on metrics like these would have been ran and resulted in a permanent change to the product. It’s just so much easier for everyone involved to trust “the data” vs something that requires nuanced interpretation.
Users are most likely not “considering” them at all. We’re not machines going over one tile at a time and generating a score. Most of the time you’re looking for something specific and just trying to find it.
This type of metric is the worst. I really hope this data-driven fad dies down and we start designing with human factors in mind again.