Back

Tool: Gitscraper


 https://github.com/adamtlangley/gitscraper

1.1 Description
Scrapes public Github repositories for common variables, folders and files. Requires a Github API key (Personal Key). Scraped results will be downloaded to a local ./raw directory.


1.2 Installation
Requires a PHP processor to be installed.

git clone https://github.com/adamtlangley/gitscraper.git

1.3 Post Usage
The following command can be used for cleaning an output file.

sort raw/{filename}.txt | uniq -c -d | sort -n -r | sed '/^[[:alnum:]/-._ ]*$/!d' | cut -c 9- | sed '/^$/d' > cleaned/{filename}.txt