Web Programmer-Database Expert Needed
Web Programmer-Database Expert Needed
Online system that automatically keeps track of expired domains, plus various information on each (eg. DA / PA, Alexa, etc.). Basically a clone of http://www.expireddomains.net/, but with a more polished and professional look / feel.
• Import all existing domains and zone files that you've obtained (about 150-200 million domains)
• Every 24 hours, scrape various sites for newly expired domains and/or domains coming up for expiration. These sites include:
o Private URL
• Full account registration, including password protected member's area, from where registered members can browse / search all expired domains, and domains coming up for expiration.
• The following basic details need to be gathered for each domain. Users would be able to browse, search, and export domain details based on the following metrics:
o Date Registered
o Date Expired / Expiring
o Available or not?
o Page Rank
o Fake PR Check (via info:// via proxies)
o Google Index (yes/no and # of pages indexed)
o Index Count
o DMOZ Listing
o Yahoo Directory Listing
o Wikipedia Listing
o # of characters in length
o Domain Age (need free check source that can be scraped initially..domaintools.com ??)
o Web Archive (display # of entries-not sure how many domains can be checked month..possible scrape job)
o Date site was last checked
o Registrar Listing (godaddy,dynadot, etc.)
o Auction (public, buy, prerelease, etc. all categories)
o Date Auction updated
o Bid Price
o Buy Now Price
o Auction Expire Date
o # of Bidders
o Alexa Rank
o Google mentions (Google Search TLD in registercompass backend)- displays # of good listings for domain in search engine
• Statistics -- On top of the metrics listed above, must scrape the following statistics for each domain. Due to API limitations and other things, must have full support for proxies, multiple API keys, and throttling limits (eg. X domains per API key/hour / day / week / month).
o Moz Stats -- DA, PA, # of backlinks, # of linking root domains, and Moz Trust score.
o SEM Rush Stats -- Rank, traffic, monthly search volume of domain keywords, semrush traffic price
o Majestic Stats -- Citation Flow, Trust Flow, # of backlinks, # of linking domains, # of referring IPs, # of pages indexed by Google.
o Spyfu Stats -- Exact local monthly search volume, cost per-click, SEO difficulty, value per-month.
o Grab top 3 anchor text keywords for domains, checking for botw dir link? niche based on kw in domain? checking links for PR? maybe for upgrade?
• Domains that are within the grace period provided by registrars, the software must check them every day for availability to register.
I will also need a front end system, so your ability to either design a front end would be nice or ability to work with others.
I have gotten a lot of responses so far and I am commited to finding a quality candidate, so I would like to make a few things clear.
1) Tell me how many hours you estimate the project will take. I have heard anywhere from 20 hours to 4 months at a full time rate. I WANT this project completed within one month and I know that's more than doable. IF YOU CANNOT PROVIDE AN ESTIMATE PLEASE DO NOT RESPOND.
2) I prefer to work with people on an ongoing basis, so keep that in mind as I'm sure I will request additional support/changes in the future.
3) Seeing completed websites is fine and all, but that is not what I care about. My primary goal is whether or not I think the candidate has the necessary skills to get the job done. I know EXACTLY what is needed from a technical perspective and I have MANY api's at my disposal. I just need somebody who can get the job done right.
For example, I will need a proxy script to go grab proxies, test them to make sure they are usable and blacklist proxies that are banned. This will need to done hourly or when all proxies are banned. There are many scripts available that do this already. They just need to be coded into the backend system. I can already process about 5K domains an hour now. I WANT to ramp that up and I have dedicated servers at my disposal.
4) Show me that you have already built a backend system before to manage the website or can explain to me how you will get the job done. Only one person so far has shown me that and he hasn't responded yet, so the job is still open.