The repository contains a crawler, s3 scanner and the website. I started his is a project because I looked for a reason to work with redis and Flask
The scanner is based on three parts,
- Crawler that scan the web and report to the redis db on any suspicious bucket
- S3Scanner that looks for interesting files in the buckets
- Flask website to expose the files list to the world :-)
I'm using Ubuntu 17.04, so the installation is for Ubuntu 17.04
- Download, Install and Run redis server
tar -zxvf redis-4.0.2.tar.gz
cd redis-4.0.2/
make
sudo make install
redis-server &
- Download and Install Flask
sudo pip3 install flask
Edit the setup.py file with starting point
python3 setup.py
Each process is in different shell...
- Run the crawler in endless loop The request module crash from time to time due to ssl3 issues etc... I decided to just write a script to restart the python it's easier for this example
./endlessrun.sh
- Run the s3Scanner - Because the scanner is much faster than the crawler I use cron to run it every hour or so ...
./cron.job
- Run the Flask server -
FLASK_APP=server.py flask run
The s3Scanner extension is licensed under the terms of the MIT license and is available for free.
The crawler i based on dmahugh crawlerino The s3 scanner is based on jordanpotti AWSBucketDump