Tool: Gospider

1.1 Description
A fast web-spider which is able to perform web crawling, brute-forcing, parsing of default files like robots.txt or sitemap.xml, link-extraction from JavaScript files and gathering of old URLs based on archives like Wayback Machine or Common Crawl.

1.2 Installation

go get -u

1.3 Example

gospider -s "" -o output -c 10 -d 1