Collect the amount of search results for a list of search terms on RedFin with 2 filters turned on (property type = land, include=sold-3mo & ,include=sold-1mo or none).
A CSV with two columns. First column is County. Second Column is State. Combine these two in the following format for the search input into RedFin: "{COUNTY}, {STATE}" Example: Hampshire County, NC.
A new CSV with the listing, count for For Sale, Sold in 1 Month, and Sold in 3 Month.
- Read CSV file
- For each row in CSV file, determine correct search URL with the following search parameters: {COUNTY}, {STATE}
- Collect search result count for "For Sale" listings with Home Type filter of "Land"
- Add value to Available column in new CSV for row
- Collect search result count for "Sold --> Last 1 Month" listings with Home Type filter of Land
- Collect search result count for "Sold --> Last 3 Months" listings with Home Type filter of Land
- Repeat for next row
- Save CSV with the following format: redfin_counties_velocity_%Y-%m-%d %H-%M-%S.
Python Scrapy framework is entirely used in this project.
- The program first take location from list of csv.
- Find out its latitude and longitude coordinates by calling a third party API.
- Sending request to another URL combination of Google map and redfin website, to get route search URL for location.
- Then sending request to route URL with different filters and saving data into csv file.
- Install required dependecies by ( pip install -r requirements.txt) in cmd (at project directory).
- command: scrapy crawl final