Less than 1 month –
30+ hrs/week –
I am looking to pull data from sights where it is allowable to pull data from.
I need help stubbing out Scrapy to work within my parameters:
-Crawl a forum listing of new posts and go to the new posts and pull the data that meets certain requirements (I have the algorithm -mainly regex formula).
-Re-crawal the sites every X seconds
-Easily able to extend capability by adding new forums that require logins and have redirects
-It needs to scale ...