Miri
Guest
offline
Cloudflare claims that Perplexity has been bypassing robots.txt directives using undeclared crawlers with rotating IPs and user-agents to avoid being blocked.
blog.cloudflare.com
Perplexity responded by saying the traffic likely came from a third-party partner (Browserbase), and emphasized that their system only accesses websites in response to direct user queries — not for autonomous...
Read more
Continue reading...
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.
Perplexity responded by saying the traffic likely came from a third-party partner (Browserbase), and emphasized that their system only accesses websites in response to direct user queries — not for autonomous...
Read more
Continue reading...