> For the complete documentation index, see [llms.txt](https://thmflags.gitbook.io/thm-walkthroughs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://thmflags.gitbook.io/thm-walkthroughs/difficulty-easy/google-dorking/task-4-beepboop-robots.txt.md).

# Task 4 - Beepboop - Robots.txt

## Task 4 - Beepboop - Robots.txt

### Where would "robots.txt" be located on the domain "**ablog.com**" 

{% hint style="warning" %}
**HINT:** full path!
{% endhint %}

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`ablog.com/robots.txt`

</details>

### If a website was to have a sitemap, where would that be located?

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`/sitemap.xml`

</details>

### How would we only allow "Bingbot" to index the website?

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`User-agent: Bingbot`

</details>

### How would we prevent a "Crawler" from indexing the directory "/dont-index-me/"?

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`Disallow: /dont-index-me/`

</details>

### What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers"?

{% hint style="warning" %}
**HINT:** system files are usually 3/4 characters!
{% endhint %}

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`.conf`

</details>