Why we have a leaderboard

And why it’s a good thing

Apr 5, 2019 25 17

Last week, we launched publisher leaderboards on substack.com to help publishers find new subscribers and subscribers find great publishers. We’ve received a lot of good feedback since the launch, but a few people worried that the leaderboards could turn Substack into a popularity contest. That’s understandable. At first glance, they look a bit like the algorithmic feeds on other social networks that cause people to compete for attention.

But the problem with algorithmic feeds is not that they use an algorithm. The problem comes when the algorithm rewards bad behavior. When you optimize for superficial engagement, you get clickbait, outrage, and addiction. For some companies, that superficial engagement is where the money comes from.

At Substack, because we don’t serve ads, we don’t make any money from superficial engagement. Instead, we make money when readers decide to pay publishers, by taking a cut of the revenue. We like this model because it means we only do well when publishers do well. Consequently, we want to do our best to help publishers find more (paying) subscribers.

The leaderboards are one step towards that, but we are being careful not to mess things up. We want to make sure that the rules that drive the algorithmic feed are based not on superficial engagement but on value. The beauty of a publishing ecosystem based on payments is that you can get a sense of value by looking at what subscribers like.

We want to design our leaderboards – and every feature we build – based on good rules, where attention is valued instead of exploited. We want to amplify the work that readers are happily paying for. This is obviously easier said than done, and we know the first versions of these discovery tools won’t be perfect. But we are taking steps that we think will help.

Subscribers matter most

For a start, we don’t look at what drives clicks. Instead, we look at signals that indicate reader satisfaction. For the free publications, that means we rank according to active readership. For the top paid publications list, we focus on revenue, which serves two important purposes: 1) It shows what readers deeply value; and 2) it gives other publishers a clear idea of what’s possible on Substack and how to get there.

For the top posts list, we focus on how many times a post has been "liked." But there are no like buttons on the leaderboard page for a reason: these aren’t simple drive-by upvotes. The only people who can like posts are a publication’s free or paid subscribers. We attribute greater significance to the likes from paying subscribers because we think the satisfaction of your most invested readers is the best measure of success.

As a matter of principle we believe that publishers and their subscribers should get to determine the terms of their relationships. That’s why publishers on Substack set their own prices, and why they can leave and take their mailing list, content, and payment relationships with them any time they want. And that’s why publishers can opt out from not only the leaderboards but also the community features (likes and comments) generally.

Our goal is to power rewarding relationships between publishers and subscribers. We want to build a place where publishers get paid to do their best work and subscribers get signal instead of the noise they get everywhere else. As we said when we started the company a year and a half ago, we are striving to build a system where we can all "be selective with our media choices, honing in on the interests, writers, and localities that we find most meaningful."

When you buy a subscription to a publication, you are making a considered choice. Rather than refreshing a feed in search of a dopamine hit, you are deciding that you want to bring the publisher’s voice into your mind. This dynamic is the antidote to the attention economy because it puts your best self back in charge.

The leaderboards are just one initiative in a larger project of helping independent publishers succeed, and although we hope we’re headed in the right decision it is likely we’ll make some mistakes along the way. We are grateful to the people who have given us feedback on this, both positive and negative, and will continue to listen. Please feel free to leave a comment or email us at [email protected].

25 17
Discussion
Danica SwansonApr 5, 2019Liked by Hamish McKenzie

While I'm fine with Substack having a leaderboard and understand why you want one - the "big names" attract more interest and new customers, for one thing - I also think you should seriously consider some of the criticisms from Jesse in the threads here. While I don't agree with all of what Jesse wrote, he makes some of the same points I came here to make.

First of all, I greatly miss the "new public posts made in real time" discovery tool. I used it regularly, and found some very interesting writers that way. I was hoping you'd develop a robust search tool that would enable me to search all Substack newsletters by subject matter and/or keyword, but I figured the real-time posts discovery page was a step in that direction. I didn't worry about it too much because, well, you're in beta. Eventually you'd have a keyword search tool of some sort, right?

But this leaderboard that displays only the currently popular writers - combined with the loss of the real-time new posts feed - is NOT a good sign. If you kept that real-time discovery tool AND the leaderboard, I wouldn't be worried.

I have niche interests. I'm not interested in any of the subjects currently being covered by writers on the leaderboard. I want a tool to find great writers who cover obscure topics and maybe don't have the level of PR skill or name recognition that your top writers have, so they'll never show up on the leaderboard. Many of those writers are the ones who are most in need of financial support and assistance in reaching potential readers!

Are you going to help us find THOSE writers, Substack? Please say yes. I've been here a year already, I left Patreon for Substack, and I still love your platform. I really, really want to keep on loving it. I hope you'll eventually have a keyword search tool.

Hi Danica, thanks for the thoughtful comments. The short answer is yes! We do plan to add new tools like the ones you describe. These leaderboards are just a first step. There will be many more ways to find publishers — big, small, new, old, popular, esoteric — in the future. One step at a time...

Danica SwansonApr 5, 2019Liked by Hamish McKenzie

I'm glad to hear you've got plans to add new discovery tools that offer more ways to find publishers, even the small and obscure ones. That's very important to me, both as a reader and a publisher of a well-loved newsletter that covers a small but growing musical genre (dark ambient). I don't have much time to do publicity work, because when I'm not at my day job, I'm usually working on time-consuming interviews with musicians.

If I could attract more paying readers, I could cut down the hours at my day job and spend more time working on the newsletter. My interview schedule for the newsletter is booked solid for months. My readers are leaving many lovely comments and telling me they're happy they paid for subscriptions because the interviews are excellent, so the future looks bright. (Although it's a major challenge to get readers to leave those lovely comments on the newsletters themselves. Most of them only comment about the newsletters on Facebook, even when I let them know they'll be helping my SEO rankings by commenting/liking directly, instead of on social media. Old habits die hard for this crowd. Sigh.)

Anyway...even if there were some kind of sorting feature on Substack to filter for only newsletters about music, that would be a step in the right direction.

I hope your plans don't get derailed or scrapped. I was so disappointed that Patreon never offered a search tool like I describe; their tool only displayed the top creators in each category. I trust that Substack can do better, which is one of the many reasons I'm here. So thanks for the reassurance.

Admittedly, I have no concept of the technical requirements involved in developing and implementing tools like this; I imagine it's very time-consuming. I'll try to be patient!

Thank you for your patience, and for believing in us. We will do our best to use that trust wisely.

Hello, Hamish!

I'm a few months late to this thread, so I want to ask––what is the current status of helping new publishers, those of us firmly in the subscription-building phase, connect to new eyes? It seems impossible to make it to the top posts page as a new writer on Substack, which is too bad. Also, it's unclear to me whether the free publications ranking is based purely on active readership, or if bigger subscription lists are weighted there, too. Thanks so much for your thoughtfulness and transparency.

Hi Lauren – we have some tools on this front (such as the featured slot on substack.com and the "Three to Read" recommendation posts that go out each week), but we're still very much in the phase where the majority of the audience-building responsibility falls on the writer. Those "top posts" lists can drive some traffic but are not intended to be a huge audience-builder. Still, you can get featured on those lists if enough of your readers click the "like" button on your posts. In the meantime, this post might be helpful: https://on.substack.com/p/a-growth-masterclass-with-judd-legum

Brendan SchlagelApr 5, 2019Liked by Hamish McKenzie

Very interesting, I like that you're trying new things and explaining what you're aiming for here.

With there being two goals at once — "to help publishers find new subscribers and subscribers find great publishers" — it strikes me that it may be really hard to achieve both with a single discovery mechanism, but seems like a good thing to experiment with.

One thing I think is worth keeping in mind is that, even if the algorithm doesn't reward bad behavior, it may still have biases. One example I'm thinking of might be bias against new creators. Ranking based on revenue or subscriber counts seem like it could bias toward early adopters / those who have most subscribers already.

Some alternatives could be using other factors like growth trajectory, conversion rate, or e.g. the ratio of likes to total subscribers rather than absolute numbers. Basically, ways to recognize quality as separate from metrics tied to quantity.

Look forward to seeing more experiments / new features in the future!

JonathanApr 12, 2019Liked by Hamish McKenzie

While I appreciate the focus on surfacing, I’m curious about longer term implications, if the goal is to help readers find the content they want. E.g. I’d find it much more interesting if I could search across terms to find content I’m interested in, that surfaced in a way that was a leaderboard in the sense that it matched by interest. If I wanted to see work focused on commerce, hip - hop, and community, a leaderboard isn’t going to show me that, unless it expands in scope. I appreciate the need to balance product choices with what people want, but right now it seems that those things might be in a conflict of sorts. You have one of the best opportunities in the market right now, and I’m looking forward to seeing how you prioritize both makers and readers.

Thanks for the thoughtful feedback. A search tool is in the works! One step at a time...

Well said, Jonathan. I agree: Substack does have one of the best market opportunities right now. And I'm in agreement about the need for a tool that can be used to search for newsletters by interest as well. If Substack can pull that off as effectively as they've done everything else I've seen so far, I think they can look forward to long-term success.

BusterApr 5, 2019Liked by Hamish McKenzie

Thanks for the transparency here. I think you make a good point about the difference between algorithms that reward engagement vs value. How far are you willing to go with this algorithm? One metric you can include because this is a subscription model is a measure of high-quality subscribers (how long they’ve subscribed + how often they open/read/like/comment). Newsletters that have a more of these kind of readers and posts that are opened/read/liked/commented on by these readers are probably an extra high-signal proxy for value. And really tough to game.

Nathan BaschezApr 5, 2019Liked by Hamish McKenzie

I like that idea! There's something powerful in the simplicity of "revenue" but it does tend to over-weight newsletters that people can pay for on their corporate expense account, and under-weight newsletters that individuals love and pay for on their own dime. I think it'll also help to break rankings out into categories.

Chris BestApr 5, 2019Liked by Hamish McKenzie

Yes this is exactly the idea. We want readers to find things that they will love so much they want to pay for, and so "how much is this beloved by the people already paying for it" is the right idea. How much $ it's making is a good starting point, but not necessarily the end.

We also don't want to be too prescriptive about how people find value. Maybe some people love reading but never commenting, etc. so we have to be a bit thoughtful about how to structure.

Problem is that you actually have done VERY VERY little to help readers find things. The old "these are the new posts made in real time" page, which you killed in favor of the leaderboard, wasn't perfect but at least let you find people/newsletters as they posted and let you see new content/writers as they uploaded content to the website. Substack still is lacking a search engine or any sort of index where you can find newsletters based upon subject, content, political preference, etc. The algorithm nature of the leaderboard leaves a heck of a lot to be desired quite frankly, especially since A. you don't spell out what the criteria is to even place on the leaderboard and B. shoves a hell of a lot of writers/columns you may not like down your throat as there is zero way to filter/curate the leaderboard to get rid of people who's work you have absolutely ZERO interest in, let alone paying money for.

Unless you have a direct link to someone's newsletter, you'll NEVER find it. There is ZERO way to explore substack to find new people to subscribe and the leaderboard, no matter how hard you deny it, DOES rig the game to only allow big names to be seen by people. Especially since (again) you don't spell out how it is you even qualify for the leader board.

It also means, garbage articles stay on the leaderboard even though there is ZERO way for people to let their displeasure be known if an article and what it contains is 100% garbage. The way you have to subscribe to comment (and can't comment unless you pay money for a newsletter subscription) might keep out trolls, but it does ZERO to provide any legitimate filtering of actual freaking quality on a system like the leaderboard, which operates on an algorithm that doesn't allow for any true quality control or filtering, to allow people to not be forced to deal with bad writers/bad columns stinking up room that could go for other writers/articles that people might actually care about reading/subscribing/paying for.

With only a leaderboard you are giving a huge advantage to people with established audiences already. People who have brought a portion of their audience from another platform are naturally going to have a competitive advantage in engagement statistics to start. Substack needs an effective discovery tool to allow readers to find new content and writers, otherwise it really isn't any different than any other "influencer" model out there.

We will for sure be adding more tools to help other publications be discovered.

The move to the leaderboard function is a horrible idea, quite frankly.

It's bad enough you guys don't have a passable search engine to find newsletters or even a sorting section by theme/subject. But eliminating the ability to see new posts/newsletters sent out in real time, further creates a huge gap in being able to find and explore content, and rewards the worst excesses of tepid hive mind banality. Garbage rises to the top, in part because the people are "Name" people backed by larger publication get preferential treatment and worse, paid ones get even higher priority in the algorhythm.

Worse, is that the idea of "interaction" is a shrill, shrieking joke. You can't even freaking comment on a post without subscribing. How can you interact if you force us to subscribe to leave feedback to a piece? This is especially galling, given the priority of paid newsletters over free ones and the hoops you have to jump through to leave feedback.

Not to mention, that the metrics for how you even end up on the leaderboard have NEVER been explained by you guys. Is it based upon website hits? Number of subscribers? Number of subscribers who click on the email links to read it on the website? Amount of cash each newsletter makes per month?

As it stands, the feces is what rises to the top at this time. There is zero option to actually explore, to discover newly created newsletters or even seek out newsletters by themes one might be interested in. The same 4-5 newsletters/posts clog up the leaderboard and you can't even leave feedback telling them what you think (good OR bad) without being forced to subscribe to them or paying the writer money. Hell, you can't even email a newsletter writer to give them feedback! It makes the entire experience a chore, especially since older posts linger on the leaderboard for DAYS with no way to even vote down the stories, if you don't like them or what they say.....