Yii Framework Forum: crawl links - Yii Framework Forum

Jump to content

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

crawl links Rate Topic: -----

#1 User is offline   masih fathi 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 7
  • Joined: 01-April 17

Posted 24 February 2018 - 01:22 PM

hi,
I have a table of links, which I must check links content (a table content has many rows and growing faster each day), I need a consultant, I should crawl links (e.g every 12 hours), what should I do in this problem?
I think about run a cronjob every 12 hours and get links which I should check at this time for crawling and adding to queue to crawl links each one or multi at the same time in each queued job, is it the good idea?
and one another question, I want to retry link in a queue if it is not done at a specific situation which for example my content not available on that link, I want to retry not done jobs e.g 4 times and between each try wait 10 minutes for retrying. is it possible doing this with the yii2 queue and can anyone give me help about ttr and what is the meaning of maximum time, does it means what I need? :mellow:
0

#2 User is offline   Menapro 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 63
  • Joined: 27-October 16
  • Location:Burgos, Spain

Posted 24 February 2018 - 07:29 PM

Yii2 queue is what you need. Better than fixed cronjobs.
0

#3 User is offline   Menapro 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 63
  • Joined: 27-October 16
  • Location:Burgos, Spain

Posted 24 February 2018 - 07:31 PM

I have seen you posted this on Yii1.1. Are you using this version? If so my solution is not valid for you.

View PostMenapro, on 24 February 2018 - 07:29 PM, said:

Yii2 queue is what you need. Better than fixed cronjobs.

0

#4 User is offline   masih fathi 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 7
  • Joined: 01-April 17

Posted 25 February 2018 - 02:33 AM

exeuse me, it was mistake, im using yii2, can you plz describe a little how to do this with retry option on yii2 queue, how to set time between each try on a queue? some one tell me about new task if failed occure in crawl.
0

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users