> For the complete documentation index, see [llms.txt](https://thmflags.gitbook.io/thm-walkthroughs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://thmflags.gitbook.io/thm-walkthroughs/difficulty-easy/google-dorking/task-2-lets-learn-about-crawlers.md).

# Task 2 - Let's Learn About Crawlers

## Task 2 - Let's Learn About Crawlers

### Name the key term of what a "Crawler" is used to do

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`index`

</details>

### What is the name of the technique that "Search Engines" use to retrieve this information about websites?

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`crawling`

</details>

### What is an example of the type of contents that could be gathered from a website?

<details>

<summary>Reveal Flag <span data-gb-custom-inline data-tag="emoji" data-code="1f6a9">🚩</span></summary>

:triangular\_flag\_on\_post:`keywords`

</details>