common-crawl-subdomains is a list of subdomains. It is intended to be used
for subdomain enumeration. The list is a compilation of all subdomains I could
find in the Common Crawl data, sorted by popularity (most common first).
www is the most common subdomain at 10.863.742 occurences. Of the total 17.525.849 subdomains found, this means that www accounts for 61.99% of all subdomains.
The popularity of the subdomains decline quickly.
- The 10th most common subdomain appeared 7482 times.
- The 100th most common subdomain appeared 977 times.
- The 1000th most common subdomain appeared 119 times.
- The 10000th most common subdomain appeared 17 times.
- The 100000th most common subdomain appeared 3 times.
As the popularity of the subdomains quickly decline, I advice you to use a much
smaller list than common-crawl-subdomains. Something like 10000 seems
reasonable. You can use head to create a smaller list:
head -n 10000 common-crawl-subdomains > common-crawl-subdomains-10000
The list with 10000 is in this repository (common-crawl-subdomains-10000).
In my experience, it yields many results.
subbrute is a (very) fast subdomain spider. It is simple to use
common-crawl-subdomains with subbrute:
./subbrute --subs common-crawl-subdomains-10000 example.com