TryHackMe Google Dorking Walkthrough

Google is a very powerful search engine. Use this room to learn how to harness the power of google. TryHackMe Google Dorking Walkthrough

[Task 1] Ye Ol' Search Engine

Google is arguably the most famous example of “Search Engines”, I mean who remembers Ask Jeeves? shudders


Now it might be rather patronising explaining how these “Search Engines” work, but there’s a lot more going on behind the scenes then what we see. More importantly, we can leverage this to our advantage to find all sorts of things that a wordlist wouldn’t. Researching as a whole - especially in the context of Cybersecurity encapsulates almost everything you do as a pentester. MuirlandOracle has created a fantastic room on learning the attitudes towards how to research, and what information you can gain from it exactly.


"Search Engines" such as Google are huge indexers – specifically, indexers of content spread across the World Wide Web.


These essentials in surfing the internet use “Crawlers” or “Spiders” to search for this content across the World Wide Web, which I will discuss in the next task.

[Task 2] Let's Learn About Crawlers

What are Crawlers and how do They Work?


These crawlers discover content through various means. One being by pure discovery, where a URL is visited by the crawler and information regarding the content type of the website is returned to the search engine. In fact, there are lots of information modern crawlers scrape – but we will discuss how this is used later. Another method crawlers use to discover content is by following any and all URLs found from previously crawled websites. Much like a virus in the sense that it will want to traverse/spread to everything it can.


Let's Visualise Some Things...


The diagram below is a high-level abstraction of how these web crawlers work. Once a web crawler discovers a domain such as mywebsite.com, it will index the entire contents of the domain, looking for keywords and other miscellaneous information - but I will discuss this miscellaneous information later.


















In the diagram above, "mywebsite.com" has been scraped as having the keywords as “Apple” “Banana" and “Pear”. These keywords are stored in a dictionary by the crawler, who then returns these to the search engine i.e. Google. Because of this persistence, Google now knows that the domain “mywebsite.com” has the keywords “Apple", “Banana” and “Pear”. As only one website has been crawled, if a user was to search for “Apple”...“mywebsite.com” would appear. This would result in the same behaviour if the user was to search for “Banana”. As the indexed contents from the crawler report the domain as having “Banana”, it will be displayed to the user.











As illustrated below, a user submits a query to the search engine of “Pears".
Because the search engine only has the contents of one website that has been crawled with the keyword of “Pears” it will be the only domain that is presented to the user.




However, as we previously mentioned, crawlers attempt to traverse, termed as crawling, every URL and file that they can find! Say if “mywebsite.com” had the same keywords as before (“Apple", “Banana” and “Pear”), but also had a URL to another website “anotherwebsite.com”, the crawler will then attempt to traverse everything on that URL (anotherwebsite.com) and retrieve the contents of everything within that domain respectively.



This is illustrated in the diagram below. The crawler initially finds “mywebsite.com”, where it crawls the contents of the website - finding the same keywords (“Apple", “Banana” and “Pear”) as before, but it has additionally found an external URL. Once the crawler is complete on “mywebsite.com”, it'll proceed to crawl the contents of the website “anotherwebsite.com”, where the keywords ("Tomatoes", “Strawberries” and “Pineapples”) are found on it. The crawler's dictionary now contains the contents of both “mywebsite.com” and “anotherwebsite.com”, which is then stored and saved within the search engine.














Recapping


So to recap, the search engine now has knowledge of two domains that have been crawled:


1. mywebsite.com
2. anotherwebsite.com



Although note that “anotherwebsite.com” was only crawled because it was referenced by the first domain “mywebsite.com”. Because of this reference, the search engine knows the following about the two domains:

• Domain Name           Keyword
• mywebsite.com          Apples
• mywebsite.com        Bananas
• mywebsite.com        Pears
• anotherwebsite.com    Tomatoes
• anotherwebsite.com    Strawberries
• anotherwebsite.com    Pineapples

Or as illustrated below:










Now that the search engine has some knowledge about keywords, say if a user was to search for “Pears” the domain “mywebsite.com” will be displayed - as it is the only crawled domain containing "Pears":











Likewise, say in this case the user now searches for "Strawberries". The domain "anotherwebsite.com" will be displayed, as it is the only domain that has been crawled by the search engine that contains the keyword "Strawberries":









This is great...But imagine if a website had multiple external URL's (as they often do!) That'll require a lot of crawling to take place. There's always the chance that another website might have similar information as of that another website crawled - right? So how does the "Search Engine" decide on the hierarchy of the domains that are displayed to the user?






In the diagram below in this instance, if the user was to search for a keyword such as "Tomatoes" (which websites 1-3 contain) who decides what website gets displayed in what order?








A logical presumption would be that website 1 -> 3 would be displayed...But that's not how real-world domains work and/or are named.


So, who (or what) decides the hierarchy? Well...



#1 Name the key term of what a "Crawler" is used to do







Ans :- Index

 

#2 What is the name of the technique that "Search Engines" use to retrieve this information about websites?



Ans :- Crawling



#3 What is an example of the type of contents that could be gathered from a website?


Ans :- Keywords

[Task 3] Enter: Search Engine Optimisation

Search Engine Optimisation


Search Engine Optimisation or SEO is a prevalent and lucrative topic in modern-day search engines. In fact, so much so, that entire businesses capitalise on improving a domains SEO “ranking”. At an abstract view, search engines will “prioritise” those domains that are easier to index. There are many factors in how “optimal” a domain is - resulting in something similar to a point-scoring system.


To highlight a few influences on how these points are scored, factors such as:


• How responsive your website is to the different browser types I.e. Google Chrome, Firefox and Internet Explorer - this includes Mobile phones!


• How easy it is to crawl your website (or if crawling is even allowed ...but we'll come to this later) through the use of "Sitemaps"


• What kind of keywords your website has (i.e. In our examples, if the user was to search for a query like “Colours” no domain will be returned - as the search engine has not (yet) crawled a domain that has any keywords to do with “Colours”


There is a lot of complexity in how the various search engines individually "point-score" or rank these domains - including vast algorithms. Naturally, the companies running these search engines such as Google don't share exactly how the hierarchic view of domains ultimately ends up. Although, as these are businesses at the end of the day, you can pay to advertise/boost the order of which your domain is displayed.



- Find a good example of how websites pay to boost their domains in the search listings -



There are various online tools - sometimes provided by the search engine providers themselves that will show you just how optimised your domain is. For example, let's use SEO Site Checkup to check the rating of TryHackMe:


According to this tool, TryHackMe is rated as 62/100 (as of 31/03/2020). That's not too bad and it'll show the justifications as to how this score was calculated below on the page.


But...Who or What Regulates these "Crawlers"?



Aside from the search engines who provide these "Crawlers", website/web-server owners themselves ultimately stipulate what content "Crawlers" can scrape. Search engines will want to retrieve everything from a website - but there are a few cases where we wouldn't want all of the contents of our website to be indexed! Can you think of any...? How about a secret administrator login page? We don't want everyone to be able to find that directory - especially through a google search.




Introducing Robots.txt...



 
#1 Using the SEO Site Checkup tool on "tryhackme.com", does TryHackMe pass the “Meta Title Test”? (Yea / Nay)


Ans :- Yea
 


#2 Does "tryhackme.com" pass the “Keywords Usage Test?” (Yea / Nay)



Ans :- Nay

 


#3 Use https://neilpatel.com/seo-analyzer/ to analyse http://googledorking.cmnatic.co.uk:

What "Page Score" does the Domain receive out of 100?





Ans :-  85/100


#4 With the same tool and domain in Question #3 (previous):

How many pages use “flash”



Ans :-  0


 

#5 From a "rating score" perspective alone, what website would list first?

tryhackme.com or googledorking.cmnatic.co.uk

Use tryhackme.com's score of 62/100 as of 31/03/2020 for this question.



Ans :- googledorking.cmnatic.co.uk

[Task 4] Beepboop - Robots.txt

Robots.txt



Similar to "Sitemaps" which we will later discuss, this file is the first thing indexed by "Crawlers" when visiting a website.


But what is it?



This file must be served at the root directory - specified by the webserver itself. Looking at this files extension of .txt, its fairly safe to assume that it is a text file.



The text file defines the permissions the "Crawler" has to the website. For example, what type of "Crawler" is allowed (I.e. You only want Google's "Crawler" to index your site and not MSN's). Moreover, Robots.txt can specify what files and directories that we do or don't want to be indexed by the "Crawler".

A very basic markup of a Robots.txt is like the following:









Here we have a few keywords...

• Keyword    Function
• User-agent    Specify the type of "Crawler" that can index your site (the asterisk being a wildcard, allowing all "User-agents"
• Allow    Specify the directories or file(s) that the "Crawler" can index
• Disallow    Specify the directories or file(s) that the "Crawler" cannot index
• Sitemap    Provide a reference to where the sitemap is located (improves SEO as previously discussed, we'll come to sitemaps in the next task)

In this case:


1. Any "Crawler" can index the site


2. The "Crawler" is allowed to index the entire contents of the site


3. The "Sitemap" is located at http://mywebsite.com/sitemap.xml



Say we wanted to hide directories or files from a "Crawler"? Robots.txt works on a "blacklisting" basis. Essentially, unless told otherwise, the Crawler will index whatever it can find.








In this case:

1. Any "Crawler" can index the site



2. The "Crawler" can index every other content that isn't contained within "/super-secret-directory/".


Crawlers also know the differences between sub-directories, directories and files. Such as in the case of the second "Disallow:" ("/not-a-secret/but-this-is/")


The "Crawler" will index all the contents within "/not-a-secret/", but will not index anything contained within the sub-directory "/but-this-is/".


3. The "Sitemap" is located at http://mywebsite.com/sitemap.xml


What if we Only Wanted Certain "Crawlers" to Index our Site?


We can stipulate so, such as in the picture below:




In this case:



1. The "Crawler" "Googlebot" is allowed to index the entire site ("Allow: /")

2. The "Crawler" "msnbot" is not allowed to index the site (Disallow: /")



How about Preventing Files From Being Indexed?


Whilst you can make manual entries for every file extension that you don't want to be indexed, you will have to provide the directory it is within, as well as the full filename. Imagine if you had a huge site! What a pain...Here's where we can use a bit of regexing.







In this case:


1. Any "Crawler" can index the site


2. However, the "Crawler" cannot index any file that has the extension of .ini within any directory/sub-directory using ("$") of the site.


3. The "Sitemap" is located at http://mywebsite.com/sitemap.xml


Why would you want to hide a .ini file for example? Well, files like this contain sensitive configuration details. Can you think of any other file formats that might contain sensitive information?




#1 Where would "robots.txt" be located on the domain "ablog.com"


Ans :-  ablog.com/robots.txt



#2 If a website was to have a sitemap, where would that be located?


Ans :-  /sitemap.xml



#3 How would we only allow "Bingbot" to index the website?


Ans :- user-agent: Bingbot




#4  How would we prevent a "Crawler" from indexing the directory "/dont-index-me/"?
 


Ans :- Disallow: /dont-index-me/



#5 What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers"?


Ans :- .conf

[Task 5] Sitemaps

Sitemaps


Comparable to geographical maps in real life, “Sitemaps” are just that - but for websites!

“Sitemaps” are indicative resources that are helpful for crawlers, as they specify the necessary routes to find content on the domain. The below illustration is a good example of the structure of a website, and how it may look on a "Sitemap":









The blue rectangles represent the route to nested-content, similar to a directory I.e. “Products” for a store. Whereas, the green rounded-rectangles represent an actual page. However, this is for illustration purposes only - “Sitemaps” don't look like this in the real world. They look something much more similar to this:









“Sitemaps” are XML formatted. I won't explain the structure of this file-formatting as the room XXE created by falconfeast does a mighty fine job of this.

The presence of "Sitemaps" holds a fair amount of weight in influencing the "optimisation" and favorability of a website. As we discussed in the "Search Engine Optimisation" task, these maps make the traversal of content much easier for the crawler!


Why are "Sitemaps" so Favourable for Search Engines?


Search engines are lazy! Well, better yet - search engines have a lot of data to process. The efficiency of how this data is collected is paramount. Resources like "Sitemaps" are extremely helpful for "Crawlers" as the necessary routes to content are already provided! All the crawler has to do is scrape this content - rather than going through the process of manually finding and scraping. Think of it as using a wordlist to find files instead of randomly guessing their names!



The easier a website is to "Crawl", the more optimised it is for the "Search Engine"




#1 What is the typical file structure of a "Sitemap"?


Ans :- XML



#2 What real life example can "Sitemaps" be compared to?



Ans :- Map



#3 Name the keyword for the path taken for content on a website


Ans :- Route

[Task 6] What is Google Dorking?

Using Google for Advanced Searching

As we have previously discussed, Google has a lot of websites crawled and indexed. Your average Joe uses Google to look up Cat pictures (I'm more of a Dog person myself...). Whilst Google will have many Cat pictures indexed ready to serve to Joe, this is a rather trivial use of the search engine in comparison to what it can be used for.
For example, we can add operators such as that from programming languages to either increase or decrease our search results - or perform actions such as arithmetic!









Say if we wanted to narrow down our search query, we can use quotation marks. Google will interpret everything in between these quotation marks as exact and only return the results of the exact phrase provided...Rather useful to filter through the rubbish that we don't need as we have done so below:








Refining our Queries


We can use terms such as “site” (such as bbc.co.uk) and a query (such as "gchq news") to search the specified site for the keyword we have provided to filter out content that may be harder to find otherwise. For example, using the “site” and "query" of "bbc" and "gchq", we have modified the order of which Google returns the results.

In the screenshot below, searching for “gchq news” returns approximately 1,060,000 results from Google. The website that we want is ranked behind GCHQ's actual website:










But we don't want that...We wanted “bbc.co.uk” first, so let's refine our search using the “site” term. Notice how in the screenshot below, Google returns with much fewer results? Additionally, the page that we didn't want has disappeared, leaving the site that we did actually want!







Of course, in this case, GCHQ is quite a topic of discussion - so there'll be a load of results regardless.



So What Makes "Google Dorking" so Appealing?


First of all - and the important part - it's legal! It's all indexed, publicly available information. However, what you do with this is where the question of legality comes in to play...


A few common terms we can search and combine include:

• Term    Action
• filetype:
•     Search for a file by its extension (e.g. PDF)
• cache:    View Google's Cached version of a specified URL
• intitle:    The specified phrase MUST appear in the title of the page

 

 

Disclaimer

This was written for educational purpose and pentest only.
The author will not be responsible for any damage ..!
The author of this tool is not responsible for any misuse of the information.
You will not misuse the information to gain unauthorized access.
This information shall only be used to expand knowledge and not for causing  malicious or damaging attacks. Performing any hacks without written permission is illegal ..!


All video’s and tutorials are for informational and educational purposes only. We believe that ethical hacking, information security and cyber security should be familiar subjects to anyone using digital information and computers. We believe that it is impossible to defend yourself from hackers without knowing how hacking is done. The tutorials and videos provided on www.hackingtruth.in is only for those who are interested to learn about Ethical Hacking, Security, Penetration Testing and malware analysis. Hacking tutorials is against misuse of the information and we strongly suggest against it. Please regard the word hacking as ethical hacking or penetration testing every time this word is used.


All tutorials and videos have been made using our own routers, servers, websites and other resources, they do not contain any illegal activity. We do not promote, encourage, support or excite any illegal activity or hacking without written permission in general. We want to raise security awareness and inform our readers on how to prevent themselves from being a victim of hackers. If you plan to use the information for illegal purposes, please leave this website now. We cannot be held responsible for any misuse of the given information.






I hope you liked this post, then you should not forget to share this post at all.
Thank you so much :-)