Out of curiosity, a question to other website administrators:'nUnder what URL does the LLM crawler for Elon Musk's #Grok show up in the website logs?
-
Out of curiosity, a question to other website administrators:
Under what URL does the LLM crawler for Elon Musk's #Grok show up in the website logs? I haven't noticed it in my own website logs so far.
-
Out of curiosity, a question to other website administrators:
Under what URL does the LLM crawler for Elon Musk's #Grok show up in the website logs? I haven't noticed it in my own website logs so far.
@juergen_hubert I don't know about Grok specifically, but I do know from my attempts to set up a no-AI website the crawlers are pretty notorious for getting around anything you may filter for--so probably there isn't any single thing that would identify it.
-
@juergen_hubert I don't know about Grok specifically, but I do know from my attempts to set up a no-AI website the crawlers are pretty notorious for getting around anything you may filter for--so probably there isn't any single thing that would identify it.
I mean, sure, the crawlers all ignore robots.txt. But most of them also seem to be fairly blatant about what they are. I have no problem identifying these crawlers:
Amazonbot
OAI-SearchBot
meta-externalagentand a bunch of others.
