I need to write a web crawler for specific user agent

Question

I need to write a web crawler, and want to be able to crawl using a known user agent. For example, I want my crawler to act as an iphone to crawl the mobile site of a website, then crawl again using Mozilla PC agent, etc.

That way, Ill be able to crawl every "type" of site (mobile & PC). However, I also want to be able to set my crawler's user agent, so webmasters also see in their stats that it's a crawler that visited their whole website, not real users.

So my question is, do you guys know how to set a mobile agent + a crawler agent at the same time, in PHP? Is it even possible?

Answer 1

Please refer to RFC1945 for how a User Agent should be formed:

10.15 User-Agent

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. Although it is not required, user agents should include this field with requests. The field can contain multiple product tokens (Section 3.7) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application.

 User-Agent     = "User-Agent" ":" 1*( product | comment )

Example:

  User-Agent: CERN-LineMode/2.15 libwww/2.17b3

So what you put there is more or less up to you. You could pose to be a GoogleBot-Mobile:

• https://www.google.com/support/webmasters/bin/answer.py?answer=1061943

or pose as an iPhone and add your own stuff

Mozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; MyBot/1.0; +http://about.my/bot")