This document lists patterns of HTTP user-agents used by robots/crawlers/spiders. I maintain this list regularly based on my own logs.
The list is now on Github [[https://github.com/monperrus/crawler-user-agents]]. Pull requests welcome!See also: [[http://www.robotstxt.org/db.html]] (the last addition is from 2007), [[http://www.user-agents.org/]]