You are not logged in.
- Topics: Active | Unanswered
#26 2008-07-31 22:32:00
- Reines
- Lead developer
- From: Scotland
- Registered: 2008-05-11
- Posts: 3,140
- Website
Re: Web crawlers
You think they use magic to get the content of your site?
Stop giving out trade secrets.
Offline
#27 2008-07-31 23:19:44
- Gotipe
- Member
- Registered: 2008-05-10
- Posts: 181
Re: Web crawlers
You think they use magic to get the content of your site?
I've not thought about it. It is not like I go about wondering how it comes Google gets stuff from my site when I eat lunch or take a walk in the forest.
Anyhow, for me it is presumably not anything to worry about. In the pace Google updates for my part, it must take one part per million of my total bandwith or something..
Last edited by Gotipe (2008-07-31 23:19:59)
Offline
#28 2008-08-01 08:11:34
- SuperMAG
- Member
- Registered: 2008-05-10
- Posts: 700
Re: Web crawlers
i just thought they will come like a normal visitors, and save every page in their db. even if that happens, no way i my site bandwidth gonno reach 52 mb, may be 10 mb maximum
Sports2All: Watch Online all the Sports (Wrestling, Football, Cricket & All other Sports)
Offline
#29 2010-12-08 12:03:04
- scalemaster
- Member
- From: Russia, Balakovo
- Registered: 2010-01-18
- Posts: 5
Re: Web crawlers
www.majestic12.co.uk is crazy bot too.
I blocked it's IP in firewall.
Offline