Webspiders are working night shifts
As a file hosting company with quite a large traffic amount and hits everyday, we must make sure, that server load and traffic stay under control and do not kill the platform by any means, like it happened twice in the life of tradebit already.
Now we monitor and control the load of the whole cluster around the clock with tools like Webmon.
While doing so, we figured, that Google and Yahoo send out their spiders mainly during early morning hours (US time). The spiders are basically there all day, but the main “reading process” seems to be done in the early morning hours.
So, if your server reacts slow at a time, you should be in bed already, it could be that the robots are taking computing power away :-).
