Task 4 - Beepboop - Robots.txt
Task 4 - Beepboop - Robots.txt
Where would "robots.txt" be located on the domain "ablog.com"
HINT: full path!
If a website was to have a sitemap, where would that be located?
How would we only allow "Bingbot" to index the website?
How would we prevent a "Crawler" from indexing the directory "/dont-index-me/"?
What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers"?
HINT: system files are usually 3/4 characters!
Last updated