Strict Standards: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EDT/-4.0/DST' instead in /homepages/20/d268022878/htdocs/forum/viewtopic.php on line 988

Strict Standards: getdate(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EDT/-4.0/DST' instead in /homepages/20/d268022878/htdocs/forum/viewtopic.php on line 988
[phpBB Debug] PHP Notice: in file /includes/functions.php on line 4284: Cannot modify header information - headers already sent by (output started at /viewtopic.php:988)
[phpBB Debug] PHP Notice: in file /includes/functions.php on line 4286: Cannot modify header information - headers already sent by (output started at /viewtopic.php:988)
[phpBB Debug] PHP Notice: in file /includes/functions.php on line 4287: Cannot modify header information - headers already sent by (output started at /viewtopic.php:988)
[phpBB Debug] PHP Notice: in file /includes/functions.php on line 4288: Cannot modify header information - headers already sent by (output started at /viewtopic.php:988)
FAROO Forum • View topic - Crawler queue not updating in DB

Crawler queue not updating in DB

Bug reports

Crawler queue not updating in DB

Postby TomHH » Sat Feb 05, 2011 5:32 pm

It seems that the crawler queue is not regularly updated in the database.
If the peer has hung up and needed to be killed in the TaskManager or crashes for some other reason, the re-started peer starts with exactly the same crawler queue as before. Any crawled pages or pages added to the queue before the crash happened are lost. This problem is reproducible - I re-started the peer on one of ma clients several times, always with the same pages at the top of the queue. As a work-around I am re-starting the peer manually to prevent it from crawling always the same pages again and again. :?

Could you please implement an update of the crawler queue within the database in regular intervals? :?:
TomHH
 
Posts: 40
Joined: Sat Dec 18, 2010 5:37 am

Re: Crawler queue not updating in DB

Postby Wolf » Thu Feb 10, 2011 12:16 pm

TomHH wrote:Could you please implement an update of the crawler queue within the database in regular intervals? :?:

Yes, of course. We will implement it.
Wolf
Site Admin
 
Posts: 130
Joined: Wed Dec 17, 2008 12:28 pm

Re: Crawler queue not updating in DB

Postby Wolf » Sat Feb 12, 2011 1:41 pm

Wolf wrote:
TomHH wrote:Could you please implement an update of the crawler queue within the database in regular intervals? :?:

Yes, of course. We will implement it.

Crawler queue and caches are now stored every 10 minutes:
http://www.faroo.com/lab/dev.html
Wolf
Site Admin
 
Posts: 130
Joined: Wed Dec 17, 2008 12:28 pm


Return to Bugs

Who is online

Users browsing this forum: No registered users and 1 guest

cron