This is useful for When you've got located your First listing of url prospective clients with Google or A further Software that lacks Website positioning metrics and facts.txt to disallow every little thing or unique endpoints (hackers can nevertheless research robots.txt for endpoints) which prevents Google bots from crawling sensitive endpoints