Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
tompec
on July 31, 2024
|
parent
|
context
|
favorite
| on:
Show HN: Turn any website into a knowledge base fo...
It does respect robots.txt when crawling. I'll add more details about this in the docs.
pryelluw
on July 31, 2024
[–]
I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.
Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
