Web crawlers

hcgtv · 2008-07-30 17:19:57

Seems like every day a new search engine is on the scene.

Just saw ScoutJet hitting my sites and wondered what it was, then realized it's just another crawler eating up my bandwidth. The 3 main engines, Google, Yahoo! and MSN eat up enough of my server's time, don't see the need for any others.

Maybe it's time to start blocking spiders?

Aabaz · 2008-07-30 17:25:48

Just another search engine : cuil

hcgtv · 2008-07-30 17:32:55

Cuil started hitting me a few months ago, I mean hitting me really hard, I though it was DOS attack.

Reines · 2008-07-30 17:39:57

I've never really had an issue with spiders using bandwidth or CPU, but I do tend to use a dedicated server so maybe just don't notice it.

hcgtv · 2008-07-30 17:55:52

Reines, sign on to your server and run iftop.

This will give you a real time idea of what is eating up your bandwidth, spiders never stop crawling.

Reines · 2008-07-30 18:31:45

hcgtv wrote:

Reines, sign on to your server and run iftop.
This will give you a real time idea of what is eating up your bandwidth, spiders never stop crawling.

Maybe true, but I wouldn't really say the amount of bandwidth they use up is an issue.

SuperMAG · 2008-07-30 18:42:26

how much bandwidth they can eat any way, its not that much of a problem. may be 1, 2, or may up to 10 mb

you get visitors from that search engine later.

Smartys · 2008-07-30 20:12:33

SuperMAG wrote:

how much bandwidth they can eat any way, its not that much of a problem. may be 1, 2, or may up to 10 mb

Bill Gates wrote:

640K ought to be enough for anybody.

The two statements are equally as true.

SuperMAG · 2008-07-30 20:32:48

640KB, mmm why you would worry on this much bandwidth

Smartys · 2008-07-30 20:35:02

yemgi · 2008-07-30 22:05:46

ROFL I could not have expressed it better.

SuperMAG wrote:

how much bandwidth they can eat any way, its not that much of a problem. may be 1, 2, or may up to 10 mb

Meow wrote:

They can eat the bandwidth if you have fluxBB on your server because FluxBB is a fork

Sorry, I could not resist

Last edited by yemgi (2008-07-30 22:07:14)

elbekko · 2008-07-31 10:02:02

Ahh, the eternal wisdom of Picard.

SuperMAG · 2008-07-31 10:26:08

what you guys are brambling about

Smartys · 2008-07-31 11:33:26

I quoted your post and what Bill Gates said and pointed out that they were equally true (in other words, you were both way off). Not that you were talking about the same thing.

SuperMAG · 2008-07-31 12:02:07

Smartys wrote:

I quoted your post and what Bill Gates said and pointed out that they were equally true (in other words, you were both way off). Not that you were talking about the same thing.

so that much bandwidth is truly eaten by the crawlers.

Smartys · 2008-07-31 12:16:30

According to the Google Webmaster Tools, Google (on average) crawls 52 MB of data per day from FluxBB.org. That is 1.5 GB of bandwidth per month.

damaxxed · 2008-07-31 12:50:07

If you really intent to block some spiders, you only strengthen the 3 (soon to be 4?) large search engines.

Connor · 2008-07-31 14:27:44

Smartys wrote:

According to the Google Webmaster Tools, Google (on average) crawls 52 MB of data per day from FluxBB.org. That is 1.5 GB of bandwidth per month.

According to our logs, 1.68 GB last month

hcgtv · 2008-07-31 14:48:27

The last time I made a major change to the cross references at PHPXref was November of last year:

Googlebot    1269448+76    4.91 GB    30 Nov 2007 - 21:17

Keep in mind that the Xrefs generate over 2 million crawlable pages.

SuperMAG · 2008-07-31 19:47:36

wow, that some bandwidth

its like the bigger your site the bigger bandwidth.

elbekko · 2008-07-31 19:58:21

No shit.

yemgi · 2008-07-31 20:35:51

I have this for one of mines in Google Webmaster tools Crawl Stats:
Number of kilobytes downloaded per day
Maximum 400944
Average 169390
Minimum 69074

hcgtv · 2008-07-31 20:55:29

elbekko wrote:

No shit.

If you really want to get the spiders crawling, just add Google Site Search to your website.

Gotipe · 2008-07-31 21:23:23

Did not know search engines affected your bandwith.
EDIT: Check out the post time, 23:23:23,

Last edited by Gotipe (2008-07-31 21:24:07)

elbekko · 2008-07-31 22:13:13

You think they use magic to get the content of your site?

Forums

#1 2008-07-30 17:19:57

Web crawlers

#2 2008-07-30 17:25:48

Re: Web crawlers

#3 2008-07-30 17:32:55

Re: Web crawlers

#4 2008-07-30 17:39:57

Re: Web crawlers

#5 2008-07-30 17:55:52

Re: Web crawlers

#6 2008-07-30 18:31:45

Re: Web crawlers

#7 2008-07-30 18:42:26

Re: Web crawlers

#8 2008-07-30 20:12:33

Re: Web crawlers

#9 2008-07-30 20:32:48

Re: Web crawlers

#10 2008-07-30 20:35:02

Re: Web crawlers

#11 2008-07-30 22:05:46

Re: Web crawlers

#12 2008-07-31 10:02:02

Re: Web crawlers

#13 2008-07-31 10:26:08

Re: Web crawlers

#14 2008-07-31 11:33:26

Re: Web crawlers

#15 2008-07-31 12:02:07

Re: Web crawlers

#16 2008-07-31 12:16:30

Re: Web crawlers

#17 2008-07-31 12:50:07

Re: Web crawlers

#18 2008-07-31 14:27:44

Re: Web crawlers

#19 2008-07-31 14:48:27

Re: Web crawlers

#20 2008-07-31 19:47:36

Re: Web crawlers

#21 2008-07-31 19:58:21

Re: Web crawlers

#22 2008-07-31 20:35:51

Re: Web crawlers

#23 2008-07-31 20:55:29

Re: Web crawlers

#24 2008-07-31 21:23:23

Re: Web crawlers

#25 2008-07-31 22:13:13

Re: Web crawlers

Board footer