Hacker Newsnew | past | comments | ask | show | jobs | submit | from login

Analyzing 5,818 Publishers' robots.txt Files (newoldweb.com) 4 points by donohoe 6 months ago | past Most publishers allow AI bots to crawl their sites (newoldweb.com) 4 points by netaustin 6 months ago | past

Consider applying for YC's Summer 2026 batch! Applications are open till May 4 Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact Search: