Est. Budget: $120.00
Create a basic crawler for indexing all pages of a website and then creating a list of root domain names that the sites pages link out.
Must be able to handle large sites in excess of 500 pages.
Should be able to rotate and use provided proxy lists.
C# strongly preferred but not required