Open Source Intelligence

Open Source tools and techniques for information gathering

• Background

• Techniques

  • Search Engines

  • Social Media

  • Public Databases

  • Forums

  • Document Sharing Websites

  • Darkweb

  • Email Address Enumeartion

• Tools

  • Google Dorks

  • Maltego

  • OSINT Framework

  • TheHarvester

  • SimplyEmail

  • Recon-ng

  • OWASP Amass

  • SpiderFoot

  • Shodan

  • Whois Lookup Tools

  • WhoisXML API

  • URL Scanners

  • GitLeaks

  • Metagoofil

  • FOCA

• Resources

Background

Open Source Intelligence (OSINT) is the process of gathering information about a target from publicly available sources, such as websites, social media, government databases, and other sources, in order to aid with penetration testing.

Techniques

There are many techniques for gathering OSINT that include utilizing search engines, social media, public databases, forums, doument sharing websites, and the darkweb.

Search Engines

• Google, Bing, Yahoo, and other search engines are some of the most commonly used tools for OSINT. They allow you to search for information on a particular topic or individual.

  • See Google Dorks for information on using Google for OSINT

• Search Engine News aggregators: News aggregators such as Google News, Yahoo News, and the BBC News website can be used to search for news articles on a particular topic or individual.

• Reverse image search engines: Reverse image search engines such as Google Images, TinEye, and Karma Decay allow you to search for other instances of a particular image on the Internet.

• Online maps: Online maps such as Google Maps and Bing Maps can be used to gather information about a particular location or to visualize geographical data.

Social Media

• Social media platforms such as Twitter, Facebook, Instagram, and LinkedIn provide a wealth of information about individuals and organizations.

• Tools such as Social Mention, Brand24, and Hootsuite can be used to monitor social media for specific keywords and mentions.

Public Databases

• Publicly available databases such as the Securities and Exchange Commission (SEC) EDGAR database, the United States Patent and Trademark Office (USPTO) database, and the General Services Administration (GSA) database can provide information about companies and individuals.

Forums

• Online forums and discussion boards such as Reddit, Quora, and Yahoo Answers can provide valuable insights into a particular topic or individual.

Document Sharing Websites

• *Websites such as Scribd, Slideshare, and Docstoc can be used to find publicly available documents such as presentations, reports, and white papers. GitHub can also be a source of valuable OSINT.

Darkweb

• The dark web, also known as the deep web, is a portion of the internet that is not indexed by search engines and can only be accessed using special software such as Tor. While it is often associated with illegal activities, the dark web can also be a valuable source of information for OSINT.

Email Address Enumeration

site:organisation.com intext:@organisation.com

https://github.com/sherlock-project/sherlock

https://haveibeenpwned.com/

Tools

The following tools can be used to gather OSINT, including Google Dorks, Malteo, OSINT Framework, TheHarvester, Recon-ng, OWASP Amass, SpiderFoot, Shodan, Whois Lookup Tools, WhoisXML API, and URL Scanners.

Google Dorks

• Google Dorks are advanced search queries that can be used to find specific types of information on the Internet.

• Here is a simple list of Google Search Operators:

• Resources

  • Exploit-DB Google Hacking Database: This is a comprehensive database of Google Dorks, maintained by Offensive Security.

  • Google Dorking Tool: A free online tool that generates Google Dorks based on keywords and search parameters.

  • Google Hacking for Penetration Testers PDF

  • SANS Google Cheat Sheet

  • Google Dork List

  • Advanced Google Search Operators

Maltego

• Maltego is an OSINT tool that allows you to visualize and analyze relationships between people, companies, domains, IP addresses, and other entities.

• The community edition of Maltego can be used for free by following this Maltego guide

• Resources

  • Maltego Site

  • Maltego Downloads

  • Online Documentation

OSINT Framework

• OSINT Framework is a comprehensive list of OSINT resources, including websites, databases, and tools, organized by category.

TheHarvester

• TheHarvester is an OSINT tool included in Kali that allows you to gather information about a target from various public sources such as search engines, social media, and other websites. The tool gathers emails, names, subdomains, IPs and URLs using multiple public data sources that include:

• Here is an example of TheHarvester usage where -d kali.org specifies the domain to search, -l 500 limits the search to 500 results, & -b google specifies the data source to use:

theharvester -d kali.org -l 500 -b google

SimplyEmail

SimplyEmail can be used to enumerate email addreses. SimplyEmail can then verify the discovered email addresss after gathering. The tool works better when using proxies or long throttle times to avoid Captchas.

curl -s https://raw.githubusercontent.com/killswitch-GUI/SimplyEmail/master/setup/oneline-setup.sh | bash
cd SimplyEmail
./SimplyEmail.py

Recon-ng

• Recon-ng is a web reconnaissance framework included in Kali that allows you to gather information about a target from various sources such as search engines, social media, and other websites. The Recon-ng wiki is found here: https://github.com/lanmaster53/recon-ng/wiki

OWASP Amass

• OWASP Amass is an OSINT tool that allows you to gather information about a target's network, including subdomains, IP addresses, and other assets.

• Resources

  • Installation Guide: https://github.com/OWASP/Amass/blob/master/doc/install.md

SpiderFoot

• SpiderFoot is an OSINT tool that allows you to gather information about a target from various sources such as search engines, social media, and other websites.

• Resources

  • SpiderFoot GitHub Repo: https://github.com/smicallef/spiderfoot

  • SpiderFoot Guide: https://nixintel.info/osint-tools/getting-started-with-spiderfoot/

Shodan

Shodan is a search engine for Internet-connected devices and can be used to gather information about a target's network, including IP addresses, open ports, and running services.

Shodan General Filters

Shodan HTTP Filters

Shodan NTP Filters

Shodan SSL Filters

Shodan Telnet Filters

Shodan Resources

• Top Shodan Dorks

• Awesome Shodan Search Queries

• Shodan Tutorial and Primer

• Shodan Website

• Shodan Filters

• Shodan Filter Reference

Whois Lookup Tools

• Whois lookup tools allow you to gather information about the owner of a domain name or IP address.

WhoisXML API

• WhoisXML API is a database of information about domains, IP addresses, and other network assets.

URL Scanners

URL scanners such as VirusTotal and URLScan allow you to scan a website for potential security threats and to gather information about the website's content and structure.

GitLeaks

Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

Installation:

git clone https://github.com/gitleaks/gitleaks.git
cd gitleaks
make build

• For usage see GitLeak on GitHub

Metagoofil

Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target.

Metagoofil is included in Kali Linux. The help menu can be launched running:

metagoofil -h

• Metagoofil Kali Tools Page

FOCA

FOCA (Fingerprinting Organizations with Collected Archives) is an application used mainly to find metadata and hidden information in the documents it scans. These documents may be on web pages, and can be downloaded and analysed with FOCA.

• FOCA is available here.

Resources

• OSINT Toolkit

• Awesome OSINT