Task 4 - Beepboop - Robots.txt
Task 4 - Beepboop - Robots.txt
Where would "robots.txt" be located on the domain "ablog.com"
If a website was to have a sitemap, where would that be located?
How would we only allow "Bingbot" to index the website?
How would we prevent a "Crawler" from indexing the directory "/dont-index-me/"?
What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers"?
Last updated