Skip to content

Social implications of crawler neutrality

Noted by on his . Last updated . Changelog


You said something I’d like to draw attention to:

I have been banned from LinkedIn temporarily myself 4 times for “looking at too many profiles”. It is in their Terms and Conditions that they can ban you for that.

LinkedIn is aggressively anti-scraping for all the wrong reasons. Bing (obviously), Google, Yandex, and some others get preferential treatment; Microsoft is incentivised to keep the number of non-Bing web indexes low. Companies that are both search engines and platforms for user-generated content are incentivized to monopolize access to said content.

Unfortunately, its anti-scraping tactics blow back on users whose behavior is not sufficiently different from scrapers. Users with unreliable connections, who open many pages in advance while good connection lasts. Spoonies who take the “open now, review later” route. Privacy-conscious users on anonymized connections. The list goes on. It’s funny how hoarding privilege tends to blow back on those without.

A diverse group of users will browse in diverse ways. Normalizing that behavior has consequences that show how Microsoft’s pro-diversity messaging is, unsurprisingly, only as shallow as its bottom line.