Yii Framework Forum: CPagination oddity - Yii Framework Forum

Jump to content

Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

CPagination oddity Creating a SphinxDataProvider < CDataProvider Rate Topic: ***** 1 Votes

#1 User is offline   François Gannaz 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 87
  • Joined: 24-November 09

Posted 25 June 2010 - 05:01 AM

We wanted to use Sphinx Search instead of the (creepy) MyISAM full text search. So I wrote a SphinxDataProvider that extends CDataProvider. Then I can use it with the zii widget CGridView. It works all right, except for pagination. And I believe the problem lies more in CPagination than in my code.

CPagination::getCurrentPage() is the method that reads the page number in GET.
        $this->_currentPage=(int)$_GET[$this->pageVar]-1;
        $pageCount=$this->getPageCount();
        if($this->_currentPage>=$pageCount)
            $this->_currentPage=$pageCount-1;

CPagination::getPageCount() uses CPagination::$itemCount, the total number of results.

So CPagination needs to know the number of results of your search before reading the current page number. This is all right with a SQL search (first send a COUNT(*) query to get the page count), but it's quite strange for a generic pagination system. Even with MySQL, you can't use SQL_CALC_ROWS with CPagination. In my case, SphinxDataProvider has to cheat:
    $pagination = $this->getPagination(); // CDataProvider::getPagination()
    $pagination->setItemCount(1000000); // The right value will be set later

I suggest this behaviour of CPagination should be at least documented, and hopefully fixed (the itemCount should be optional).

(Shameless rant) BTW, I find this kind of code ugly and hard to read. On my first glance, I missed the value setting.
        if(($this->_itemCount=$value)<0)
            $this->_itemCount=0;

0

#2 User is offline   zaccaria 

  • Elite Member
  • PipPipPipPipPip
  • Yii
  • Group: Members
  • Posts: 2,232
  • Joined: 04-October 09
  • Location:Moscow

Posted 25 June 2010 - 05:59 AM

I think that CPagination needs the total itemCount before reading the current page.

That's because if, for example, you are in the 10th page and you filter your result so that the number of page will become 6, CPagination will place you at the 6th page, because the 10th is empty.

Maybe you can fix your problem by implementing a setItemCount in your DataProvider. I guess that if you use pagination you have anyway to run 2 queries, one for know the totalItemCount (SELECT count(*)) and another for retrive your items (SELECT * ... LIMIT), isn't it?
0

#3 User is offline   François Gannaz 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 87
  • Joined: 24-November 09

Posted 25 June 2010 - 09:00 AM

Sorry, I wasn't clear enough.

 zaccaria, on 25 June 2010 - 05:59 AM, said:

I think that CPagination needs the total itemCount before reading the current page.

That's exactly what I was complaining about. I even wrote it in bold font.

Quote

That's because if, for example, you are in the 10th page and you filter your result so that the number of page will become 6, CPagination will place you at the 6th page, because the 10th is empty.

If I filter my results, a new search is launched, and the current page is reset to 1. That's the current CGridView behaviour, at least.
I guess you mean that if someone modifies the parameters of the search in the URL, CPagination has to auto-fix the current page. That doesn't seem that useful to me.

Quote

Maybe you can fix your problem by implementing a setItemCount in your DataProvider. I guess that if you use pagination you have anyway to run 2 queries, one for know the totalItemCount (SELECT count(*)) and another for retrive your items (SELECT * ... LIMIT), isn't it?

I did fix it. See the above code line where I set the item count to 1000000. This is an ugly way to bypass the need of an item count.

But no, I can't make a "SELECT count(*)" because I'm not using SQL for this search, I'm using Sphinx Search. And I don't want to send the same query twice (once without the offset, once with it). I guess other NoSQL databases will suffer from the same problem.

So my main point is that CPagination::getCurrentPage() should use $this->_itemCount only if it's defined (this attribute should be null by default).
0

#4 User is offline   zaccaria 

  • Elite Member
  • PipPipPipPipPip
  • Yii
  • Group: Members
  • Posts: 2,232
  • Joined: 04-October 09
  • Location:Moscow

Posted 28 June 2010 - 03:30 AM

Quote

If I filter my results, a new search is launched, and the current page is reset to 1. That's the current CGridView behaviour, at least.


I don't know what is the behaviour of CGridView, but the CPagination in this case is setting the last one available page:

This is how CPagination is working rigth now:
	public function getCurrentPage($recalculate=true)
	{
		if($this->_currentPage===null || $recalculate)
		{
			if(isset($_GET[$this->pageVar]))
			{
				$this->_currentPage=(int)$_GET[$this->pageVar]-1;
				$pageCount=$this->getPageCount();
				if($this->_currentPage>=$pageCount)
					$this->_currentPage=$pageCount-1;
				if($this->_currentPage<0)
					$this->_currentPage=0;
			}
			else
				$this->_currentPage=0;
		}
		return $this->_currentPage;
	}


If you need the raw value, you can use:

(int)$_GET[$pagination->pageVar]-1;


That is not really stylish, but at least it works.
A better solution is to extend CPagination and implement a method like that:

public function getRowPageNumber()
{
	if(isset($_GET[$this->pageVar]))
		return (int)$_GET[$this->pageVar]-1;
	else
		return 0;
}


And, if you think that your situation is quite common, you can ask in "feature request" to add this function in the core framework. Staff usually answer to this kind if request.
0

#5 User is offline   François Gannaz 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 87
  • Joined: 24-November 09

Posted 29 June 2010 - 01:18 PM

 zaccaria, on 28 June 2010 - 03:30 AM, said:

IThis is how CPagination is working rigth now:
[...]


Please don't reply if you haven't read carefully my posts. My first post contained the same part of the CPagination code you inserted. So you're pointing me to a code I've obviously read.

I didn't ask for bad workarounds, I already had one. Reading the raw value means writing a new pagination system. Extending CPagination means that classes that use CPagination also have to be rewritten, and that's a bad idea. Setting a fake item count (my solution from the first post) was far better than theses two solutions.

I thought I'd open a feature request, but before that I just wanted to make sure I didn't misunderstand my problem. Yii recommends posting to the forum before going to the bug tracker.
0

#6 User is offline   samdark 

  • Having fun
  • Yii
  • Group: Yii Dev Team
  • Posts: 3,788
  • Joined: 17-January 09
  • Location:Russia

Posted 29 June 2010 - 02:54 PM

François Gannaz
You understand this problem in a right way. Feel free to open new issue. I had nearly the same issues trying to implement pagination for MongoDB.
Yii 1.1 Application Development Cookbook

Enjoying Yii? Star us at github: 1.1 and 2.0.
0

#7 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 13 August 2012 - 12:41 PM

Is there any news on this topic? :)
0

#8 User is offline   ixolit 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 7
  • Joined: 06-June 12

Posted 27 August 2012 - 07:22 AM

Can someone provide SphinxDataProvider ?
0

#9 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 27 August 2012 - 07:37 AM

My version (with counter hack, like described above):


class SphinxDataProvider extends CDataProvider
{
	/**
	 * @var string the primary ActiveRecord class name. The {@link getData()} method
	 * will return a list of objects of this class.
	 */
	public $modelClass;
	/**
	 * @var CActiveRecord the AR finder instance (eg <code>Post::model()</code>).
	 * This property can be set by passing the finder instance as the first parameter
	 * to the constructor. For example, <code>Post::model()->published()</code>.
	 * @since 1.1.3
	 */
	public $model;
	/**
	 * @var string the name of key attribute for {@link modelClass}. If not set,
	 * it means the primary key of the corresponding database table will be used.
	 */
	public $keyAttribute;
	
	private $_sort;
	private $_criteria; //criteria for select ActiveRecord object
	private $_sphinxCriteria; //sphinxsearch criteria for search
	private $_totalItemsCount; //total items count


	/**
	 * Constructor.
	 * @param mixed $modelClass the model class (e.g. 'Post') or the model finder instance
	 * (e.g. <code>Post::model()</code>, <code>Post::model()->published()</code>).
	 * @param array $config configuration (name=>value) to be applied as the initial property values of this class.
	 */
	public function __construct($modelClass,$config=array())
	{
		if(is_string($modelClass))
		{
			$this->modelClass=$modelClass;
			$this->model=CActiveRecord::model($this->modelClass);
		}
		else if($modelClass instanceof CActiveRecord)
		{
			$this->modelClass=get_class($modelClass);
			$this->model=$modelClass;
		}
		
		$this->setId($this->modelClass);
		foreach($config as $key=>$value)
			$this->$key=$value;
	}

	/**
	 * Returns the query criteria.
	 * @return CDbCriteria the query criteria
	 */
	public function getCriteria()
	{
		return $this->_criteria;
	}
	/**
	 * Sets the query criteria.
	 * @param mixed $value the query criteria. This can be either a CDbCriteria object or an array
	 * representing the query criteria.
	 */
	public function setCriteria($value)
	{
		$this->_criteria=$value;
	}
	/**
	 * Returns the sphinx query criteria.
	 * @return stdClass the sphinx query criteria
	 */
	public function getSphinxCriteria()
	{
		return $this->_sphinxCriteria;
	}

	/**
	 * Sets the sphinx criteria.
	 * @param mixed $value the sphinx criteria. This can be either a stdClass object
	 * representing the sphinx criteria.
	 */
	public function setSphinxCriteria($value)
	{
		$this->_sphinxCriteria=$value;
	}

	/**
	 * Returns the sorting object.
	 * @return CSort the sorting object. If this is false, it means the sorting is disabled.
	 */
	public function getSphinxSort()
	{
		if(($sort=parent::getSort())!==false)
			$sort->modelClass=$this->modelClass;
		return $sort;
	}

	/**
	 * Fetches the data from the persistent data storage.
	 * @return array list of data items
	 */
	protected function fetchData()
	{
		$criteria = clone $this->getCriteria();
		$sphinxCriteria = clone $this->getSphinxCriteria();
		
		if(($pagination=$this->getPagination())!==false)
		{
			$pagination->setItemCount(10000000);
			$pagination->applyLimit($sphinxCriteria);
			$sphinxCriteria->paginator = $pagination;
		}

		if(($sort=$this->getSort())!==false)
			$sort->applyOrder($sphinxCriteria);
		
		//var_dump($sphinxCriteria); exit;
		
		$sphinx = Yii::App()->search;
		$sphinx->setMatchMode(SPH_MATCH_EXTENDED2);
		$resArray = $sphinx->searchRaw($sphinxCriteria);
		
		$this->_totalItemsCount = isset($resArray['total_found']) ? $resArray['total_found'] : 0;
		$pagination->setItemCount($this->_totalItemsCount);
		//var_dump($resArray); exit();
		
		$values = array(0);
		if(!empty($resArray['matches']))
		{
			foreach($resArray['matches'] as $k => $v)
				array_push($values, $k);
		}
		//var_dump($values); 
		if(!empty($values)) {
			$resCriteria = new CDbCriteria();
			$resCriteria->addInCondition('t.'.$this->model->getMetaData()->tableSchema->primaryKey, $values);
			$criteria->mergeWith($resCriteria);
		}
		$criteria->order = 'FIELD(t.id,'.implode(',',$values).')';
		//var_dump($criteria); exit();
		$data=$this->model->findAll($criteria);
		return $data;
	}
	/**
	 * Returns the sort object.
	 * @return CSort the sorting object. If this is false, it means the sorting is disabled.
	 */
	public function getSort()
	{
		if($this->_sort===null)
		{
			$this->_sort=new SphinxSort;
			if(($id=$this->getId())!='')
				$this->_sort->sortVar=$id.'_sort';
		}
		return $this->_sort;
	}

	/**
	 * Fetches the data item keys from the persistent data storage.
	 * @return array list of data item keys.
	 */
	protected function fetchKeys()
	{
		$keys=array();
		foreach($this->getData() as $i=>$data)
		{
			$key=$this->keyAttribute===null ? $data->getPrimaryKey() : $data->{$this->keyAttribute};
			$keys[$i]=is_array($key) ? implode(',',$key) : $key;
		}
		return $keys;
	}

	/**
	 * Calculates the total number of data items.
	 * @return integer the total number of data items.
	 */
	protected function calculateTotalItemCount()
	{
		return $this->_totalItemsCount;
	}
}
?>


SphinxSort

class SphinxSort extends CSort
{
	public $multiSort=false;
	/**
	 * @var string the name of the model class whose attributes can be sorted.
	 * The model class must be a child class of {@link CActiveRecord}.
	 */
	public $modelClass;
	/**
	 * @var array list of attributes that are allowed to be sorted.
	 * For example, array('user_id','create_time') would specify that only 'user_id'
	 * and 'create_time' of the model {@link modelClass} can be sorted.
	 * By default, this property is an empty array, which means all attributes in
	 * {@link modelClass} are allowed to be sorted.
	 *
	 * This property can also be used to specify complex sorting. To do so,
	 * a virtual attribute can be declared in terms of a key-value pair in the array.
	 * The key refers to the name of the virtual attribute that may appear in the sort request,
	 * while the value specifies the definition of the virtual attribute.
	 *
	 * In the simple case, a key-value pair can be like <code>'user'=>'user_id'</code>
	 * where 'user' is the name of the virtual attribute while 'user_id' means the virtual
	 * attribute is the 'user_id' attribute in the {@link modelClass}.
	 *
	 * A more flexible way is to specify the key-value pair as
	 * <pre>
	 * 'user'=>array(
	 *     'asc'=>'first_name, last_name',
	 *     'desc'=>'first_name DESC, last_name DESC',
	 *     'label'=>'Name'
	 * )
	 * </pre>
	 * where 'user' is the name of the virtual attribute that specifies the full name of user
	 * (a compound attribute consisting of first name and last name of user). In this case,
	 * we have to use an array to define the virtual attribute with three elements: 'asc',
	 * 'desc' and 'label'.
	 *
	 * The above approach can also be used to declare virtual attributes that consist of relational
	 * attributes. For example,
	 * <pre>
	 * 'price'=>array(
	 *     'asc'=>'item.price',
	 *     'desc'=>'item.price DESC',
	 *     'label'=>'Item Price'
	 * )
	 * </pre>
	 *
	 * Note, the attribute name should not contain '-' or '.' characters because
	 * they are used as {@link separators}.
	 *
	 * Starting from version 1.1.3, an additional option named 'default' can be used in the virtual attribute
	 * declaration. This option specifies whether an attribute should be sorted in ascending or descending
	 * order upon user clicking the corresponding sort hyperlink if it is not currently sorted. The valid
	 * option values include 'asc' (default) and 'desc'. For example,
	 * <pre>
	 * 'price'=>array(
	 *     'asc'=>'item.price',
	 *     'desc'=>'item.price DESC',
	 *     'label'=>'Item Price',
	 *     'default'=>'desc',
	 * )
	 * </pre>
	 *
	 * Also starting from version 1.1.3, you can include a star ('*') element in this property so that
	 * all model attributes are available for sorting, in addition to those virtual attributes. For example,
	 * <pre>
	 * 'attributes'=>array(
	 *     'price'=>array(
	 *         'asc'=>'item.price',
	 *         'desc'=>'item.price DESC',
	 *         'label'=>'Item Price',
	 *         'default'=>'desc',
	 *     ),
	 *     '*',
	 * )
	 * </pre>
	 * Note that when a name appears as both a model attribute and a virtual attribute, the position of
	 * the star element in the array determines which one takes precedence. In particular, if the star
	 * element is the first element in the array, the model attribute takes precedence; and if the star
	 * element is the last one, the virtual attribute takes precedence.
	 */
	public $attributes=array();
	/**
	 * @var string the name of the GET parameter that specifies which attributes to be sorted
	 * in which direction. Defaults to 'sort'.
	 */
	public $sortVar='sort';
	/**
	 * @var string the tag appeared in the GET parameter that indicates the attribute should be sorted
	 * in descending order. Defaults to 'desc'.
	 */
	public $descTag='desc';
	/**
	 * @var mixed the default order that should be applied to the query criteria when
	 * the current request does not specify any sort. For example, 'name, create_time DESC' or
	 * 'UPPER(name)'.
	 *
	 * Starting from version 1.1.3, you can also specify the default order using an array.
	 * The array keys could be attribute names or virtual attribute names as declared in {@link attributes},
	 * and the array values indicate whether the sorting of the corresponding attributes should
	 * be in descending order. For example,
	 * <pre>
	 * 'defaultOrder'=>array(
	 *     'price'=>true,
	 * )
	 * </pre>
	 *
	 * Please note when using array to specify the default order, the corresponding attributes
	 * will be put into {@link directions} and thus affect how the sort links are rendered
	 * (e.g. an arrow may be displayed next to the currently active sort link).
	 */
	public $defaultOrder;
	/**
	 * @var string the route (controller ID and action ID) for generating the sorted contents.
	 * Defaults to empty string, meaning using the currently requested route.
	 */
	public $route='';
	/**
	 * @var array separators used in the generated URL. This must be an array consisting of
	 * two elements. The first element specifies the character separating different
	 * attributes, while the second element specifies the character separating attribute name
	 * and the corresponding sort direction. Defaults to array('-','.').
	 */
	public $separators=array('-','.');
	/**
	 * @var array the additional GET parameters (name=>value) that should be used when generating sort URLs.
	 * Defaults to null, meaning using the currently available GET parameters.
	 * @since 1.0.9
	 */
	public $params;

	private $_directions;

	/**
	 * Constructor.
	 * @param string $modelClass the class name of data models that need to be sorted.
	 * This should be a child class of {@link CActiveRecord}.
	 */
	public function __construct($modelClass=null)
	{
		
	}

	/**
	 * Modifies the query criteria by changing its {@link CDbCriteria::order} property.
	 * This method will use {@link directions} to determine which columns need to be sorted.
	 * They will be put in the ORDER BY clause. If the criteria already has non-empty {@link CDbCriteria::order} value,
	 * the new value will be appended to it.
	 * @param CDbCriteria $criteria the query criteria
	 */
	public function applyOrder($criteria)
	{
		$order=$this->getOrderBy();
		if(!empty($order))
		{
			if(!empty($criteria->orders))
				$criteria->orders.=', ';
			$criteria->orders.=$order;
		}
	}

	/**
	 * @return array the orderby represented by this sort object.
	 */
	public function getOrderBy()
	{
		$directions=$this->getDirections();
		if(empty($directions))
			return is_array($this->defaultOrder) ? $this->defaultOrder : array();
		else
		{
			$orders=array();
			foreach($directions as $attribute=>$descending)
			{
				if($descending)
					$orders[]=isset($definition['desc']) ? $definition['desc'] : $attribute.' DESC';
				else
					$orders[]=isset($definition['asc']) ? $definition['asc'] : $attribute.' ASC';
			}
			return implode(', ',$orders);
		}
	}

	/**
	 * Returns the currently requested sort information.
	 * @return array sort directions indexed by attribute names.
	 * The sort direction is true if the corresponding attribute should be
	 * sorted in descending order.
	 */
	public function getDirections()
	{
		if($this->_directions===null)
		{
			$this->_directions=array();
			if(isset($_GET[$this->sortVar]))
			{
				$attributes=explode($this->separators[0],$_GET[$this->sortVar]);
				foreach($attributes as $attribute)
				{
					if(($pos=strrpos($attribute,$this->separators[1]))!==false)
					{
						$descending=substr($attribute,$pos+1)===$this->descTag;
						if($descending)
							$attribute=substr($attribute,0,$pos);
					}
					else
						$descending=false;

					if(($this->resolveAttribute($attribute))!==false)
					{
						$this->_directions[$attribute]=$descending;
						if(!$this->multiSort)
							return $this->_directions;
					}
				}
			}
			if($this->_directions===array() && is_array($this->defaultOrder))
				$this->_directions=$this->defaultOrder;
		}
		return $this->_directions;
	}

	/**
	 * Returns the sort direction of the specified attribute in the current request.
	 * @param string $attribute the attribute name
	 * @return mixed the sort direction of the attribut. True if the attribute should be sorted in descending order,
	 * false if in ascending order, and null if the attribute doesn't need to be sorted.
	 */
	public function getDirection($attribute)
	{
		$this->getDirections();
		return isset($this->_directions[$attribute]) ? $this->_directions[$attribute] : null;
	}

	/**
	 * Creates a URL that can lead to generating sorted data.
	 * @param CController $controller the controller that will be used to create the URL.
	 * @param array $directions the sort directions indexed by attribute names.
	 * The sort direction is true if the corresponding attribute should be
	 * sorted in descending order.
	 * @return string the URL for sorting
	 */
	public function createUrl($controller,$directions)
	{
		$sorts=array();
		foreach($directions as $attribute=>$descending)
			$sorts[]=$descending ? $attribute.$this->separators[1].$this->descTag : $attribute;
		$params=$this->params===null ? $_GET : $this->params;
		$params[$this->sortVar]=implode($this->separators[0],$sorts);
		return $controller->createUrl($this->route,$params);
	}

	/**
	 * Returns the real definition of an attribute given its name.
	 *
	 * The resolution is based on {@link attributes} and {@link CActiveRecord::attributeNames}.
	 * <ul>
	 * <li>When {@link attributes} is an empty array, if the name refers to an attribute of {@link modelClass},
	 * then the name is returned back.</li>
	 * <li>When {@link attributes} is not empty, if the name refers to an attribute declared in {@link attributes},
	 * then the corresponding virtual attribute definition is returned. Starting from version 1.1.3, if {@link attributes}
	 * contains a star ('*') element, the name will also be used to match against all model attributes.</li>
	 * <li>In all other cases, false is returned, meaning the name does not refer to a valid attribute.</li>
	 * </ul>
	 * @param string $attribute the attribute name that the user requests to sort on
	 * @return mixed the attribute name or the virtual attribute definition. False if the attribute cannot be sorted.
	 */
	public function resolveAttribute($attribute)
	{
		if($this->attributes!==array())
			$attributes=$this->attributes;
		else if($this->modelClass!==null)
			$attributes=CActiveRecord::model($this->modelClass)->attributeNames();
		else
			return false;
		foreach($attributes as $name=>$definition)
		{
			if(is_string($name))
			{
				if($name===$attribute)
					return $definition;
			}
			else if($definition==='*')
			{
				if($this->modelClass!==null && CActiveRecord::model($this->modelClass)->hasAttribute($attribute))
					return $attribute;
			}
			else if($definition===$attribute)
				return $attribute;
		}
		return false;
	}

	/**
	 * Creates a hyperlink based on the given label and URL.
	 * You may override this method to customize the link generation.
	 * @param string $attribute the name of the attribute that this link is for
	 * @param string $label the label of the hyperlink
	 * @param string $url the URL
	 * @param array $htmlOptions additional HTML options
	 * @return string the generated hyperlink
	 */
	protected function createLink($attribute,$label,$url,$htmlOptions)
	{
		return CHtml::link($label,$url,$htmlOptions);
	}
}



How to use (live example):

	//SphinxSearch criteria
	$searchCriteria = new stdClass();
	$searchCriteria->select = '*';
	$searchCriteria->query = ''.$search_string.'';
	$searchCriteria->from = 'index_name';
	//...
	$filters['brand_id'] = $brands_array; //filter by brand ids, if needed
	if(!empty($filters)) $searchCriteria->filters = $filters;
	//...
	//items criteria
	$criteria = new CDbCriteria;
	$criteria->with = array('brand','category');
	//...
	$catalog = new SphinxDataProvider('CatItem',
		array(
			'criteria' => $criteria, //criteria for AR model
			'sphinxCriteria' => $searchCriteria, //SphinxSearch critria
			'pagination'=>array(
				'pageSize' => $num,
				'pageVar' => 'p',
			),
			'sort' => array(
				'attributes'=>array(
					'price'=>array(
						'asc' => 'price ASC',
						'desc' => 'price DESC',
					),
					'title'=>array(
						'asc' => 'title ASC',
						'desc' => 'title DESC',
					),
				),
			),
	));


And don't forget DGSphinxSearch fix: http://www.yiiframew...inxsearch#c9471
2

#10 User is offline   ixolit 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 7
  • Joined: 06-June 12

Posted 27 August 2012 - 08:53 AM

Dude, you rock!

This should be included into dgsphinxsearch extension.

The only thing that doesn't work for me is line in method calculateTotalItemCount()

 $baseCriteria=$this->model->getDbCriteria(false);


I got $baseCriteria as null. And following query fetches all records from db table, not only those that sphinx found. Please suggest
0

#11 User is offline   rorfun 

  • Newbie
  • Yii
  • Group: Members
  • Posts: 1
  • Joined: 29-March 12

Posted 15 November 2012 - 08:36 PM

 Yuri!, on 27 August 2012 - 07:37 AM, said:

My version (with counter hack, like described above):


class SphinxDataProvider extends CDataProvider
{
	/**
	 * @var string the primary ActiveRecord class name. The {@link getData()} method
	 * will return a list of objects of this class.
	 */
	public $modelClass;
	/**
	 * @var CActiveRecord the AR finder instance (eg <code>Post::model()</code>).
	 * This property can be set by passing the finder instance as the first parameter
	 * to the constructor. For example, <code>Post::model()->published()</code>.
	 * @since 1.1.3
	 */
	public $model;
	/**
	 * @var string the name of key attribute for {@link modelClass}. If not set,
	 * it means the primary key of the corresponding database table will be used.
	 */
	public $keyAttribute;
	
	private $_sort;
	private $_criteria; //criteria for select ActiveRecord object
	private $_sphinxCriteria; //sphinxsearch criteria for search

	/**
	 * Constructor.
	 * @param mixed $modelClass the model class (e.g. 'Post') or the model finder instance
	 * (e.g. <code>Post::model()</code>, <code>Post::model()->published()</code>).
	 * @param array $config configuration (name=>value) to be applied as the initial property values of this class.
	 */
	public function __construct($modelClass,$config=array())
	{
		if(is_string($modelClass))
		{
			$this->modelClass=$modelClass;
			$this->model=CActiveRecord::model($this->modelClass);
		}
		else if($modelClass instanceof CActiveRecord)
		{
			$this->modelClass=get_class($modelClass);
			$this->model=$modelClass;
		}
		
		$this->setId($this->modelClass);
		foreach($config as $key=>$value)
			$this->$key=$value;
	}

	/**
	 * Returns the query criteria.
	 * @return CDbCriteria the query criteria
	 */
	public function getCriteria()
	{
		return $this->_criteria;
	}
	/**
	 * Sets the query criteria.
	 * @param mixed $value the query criteria. This can be either a CDbCriteria object or an array
	 * representing the query criteria.
	 */
	public function setCriteria($value)
	{
		$this->_criteria=$value;
	}
	/**
	 * Returns the sphinx query criteria.
	 * @return stdClass the sphinx query criteria
	 */
	public function getSphinxCriteria()
	{
		return $this->_sphinxCriteria;
	}

	/**
	 * Sets the sphinx criteria.
	 * @param mixed $value the sphinx criteria. This can be either a stdClass object
	 * representing the sphinx criteria.
	 */
	public function setSphinxCriteria($value)
	{
		$this->_sphinxCriteria=$value;
	}

	/**
	 * Returns the sorting object.
	 * @return CSort the sorting object. If this is false, it means the sorting is disabled.
	 */
	public function getSphinxSort()
	{
		if(($sort=parent::getSort())!==false)
			$sort->modelClass=$this->modelClass;
		return $sort;
	}

	/**
	 * Fetches the data from the persistent data storage.
	 * @return array list of data items
	 */
	protected function fetchData()
	{
		$criteria = clone $this->getCriteria();
		$sphinxCriteria = clone $this->getSphinxCriteria();
		
		if(($pagination=$this->getPagination())!==false)
		{
			$pagination->setItemCount(10000000);
			$pagination->applyLimit($sphinxCriteria);
			$sphinxCriteria->paginator = $pagination;
		}

		if(($sort=$this->getSort())!==false)
			$sort->applyOrder($sphinxCriteria);
		
		//var_dump($sphinxCriteria); exit;
		
		$sphinx = Yii::App()->search;
		$sphinx->setMatchMode(SPH_MATCH_EXTENDED2);
		$resArray = $sphinx->searchRaw($sphinxCriteria);
		
		$total_found = isset($resArray['total_found']) ? $resArray['total_found'] : 0;
		$pagination->setItemCount($total_found);
		//var_dump($resArray); exit();
		
		$values = array(0);
		if(!empty($resArray['matches']))
		{
			foreach($resArray['matches'] as $k => $v)
				array_push($values, $k);
		}
		//var_dump($values); 
		if(!empty($values)) {
			$resCriteria = new CDbCriteria();
			$resCriteria->addInCondition('t.'.$this->model->getMetaData()->tableSchema->primaryKey, $values);
			$criteria->mergeWith($resCriteria);
		}
		$criteria->order = 'FIELD(t.id,'.implode(',',$values).')';
		//var_dump($criteria); exit();
		$data=$this->model->findAll($criteria);
		return $data;
	}
	/**
	 * Returns the sort object.
	 * @return CSort the sorting object. If this is false, it means the sorting is disabled.
	 */
	public function getSort()
	{
		if($this->_sort===null)
		{
			$this->_sort=new SphinxSort;
			if(($id=$this->getId())!='')
				$this->_sort->sortVar=$id.'_sort';
		}
		return $this->_sort;
	}

	/**
	 * Fetches the data item keys from the persistent data storage.
	 * @return array list of data item keys.
	 */
	protected function fetchKeys()
	{
		$keys=array();
		foreach($this->getData() as $i=>$data)
		{
			$key=$this->keyAttribute===null ? $data->getPrimaryKey() : $data->{$this->keyAttribute};
			$keys[$i]=is_array($key) ? implode(',',$key) : $key;
		}
		return $keys;
	}

	/**
	 * Calculates the total number of data items.
	 * @return integer the total number of data items.
	 */
	protected function calculateTotalItemCount()
	{
		$baseCriteria=$this->model->getDbCriteria(false);
		if($baseCriteria!==null)
			$baseCriteria=clone $baseCriteria;
		$count=$this->model->count($this->getCriteria());
		$this->model->setDbCriteria($baseCriteria);
		return $count;
	}
}
?>


SphinxSort

class SphinxSort extends CSort
{
	public $multiSort=false;
	/**
	 * @var string the name of the model class whose attributes can be sorted.
	 * The model class must be a child class of {@link CActiveRecord}.
	 */
	public $modelClass;
	/**
	 * @var array list of attributes that are allowed to be sorted.
	 * For example, array('user_id','create_time') would specify that only 'user_id'
	 * and 'create_time' of the model {@link modelClass} can be sorted.
	 * By default, this property is an empty array, which means all attributes in
	 * {@link modelClass} are allowed to be sorted.
	 *
	 * This property can also be used to specify complex sorting. To do so,
	 * a virtual attribute can be declared in terms of a key-value pair in the array.
	 * The key refers to the name of the virtual attribute that may appear in the sort request,
	 * while the value specifies the definition of the virtual attribute.
	 *
	 * In the simple case, a key-value pair can be like <code>'user'=>'user_id'</code>
	 * where 'user' is the name of the virtual attribute while 'user_id' means the virtual
	 * attribute is the 'user_id' attribute in the {@link modelClass}.
	 *
	 * A more flexible way is to specify the key-value pair as
	 * <pre>
	 * 'user'=>array(
	 *     'asc'=>'first_name, last_name',
	 *     'desc'=>'first_name DESC, last_name DESC',
	 *     'label'=>'Name'
	 * )
	 * </pre>
	 * where 'user' is the name of the virtual attribute that specifies the full name of user
	 * (a compound attribute consisting of first name and last name of user). In this case,
	 * we have to use an array to define the virtual attribute with three elements: 'asc',
	 * 'desc' and 'label'.
	 *
	 * The above approach can also be used to declare virtual attributes that consist of relational
	 * attributes. For example,
	 * <pre>
	 * 'price'=>array(
	 *     'asc'=>'item.price',
	 *     'desc'=>'item.price DESC',
	 *     'label'=>'Item Price'
	 * )
	 * </pre>
	 *
	 * Note, the attribute name should not contain '-' or '.' characters because
	 * they are used as {@link separators}.
	 *
	 * Starting from version 1.1.3, an additional option named 'default' can be used in the virtual attribute
	 * declaration. This option specifies whether an attribute should be sorted in ascending or descending
	 * order upon user clicking the corresponding sort hyperlink if it is not currently sorted. The valid
	 * option values include 'asc' (default) and 'desc'. For example,
	 * <pre>
	 * 'price'=>array(
	 *     'asc'=>'item.price',
	 *     'desc'=>'item.price DESC',
	 *     'label'=>'Item Price',
	 *     'default'=>'desc',
	 * )
	 * </pre>
	 *
	 * Also starting from version 1.1.3, you can include a star ('*') element in this property so that
	 * all model attributes are available for sorting, in addition to those virtual attributes. For example,
	 * <pre>
	 * 'attributes'=>array(
	 *     'price'=>array(
	 *         'asc'=>'item.price',
	 *         'desc'=>'item.price DESC',
	 *         'label'=>'Item Price',
	 *         'default'=>'desc',
	 *     ),
	 *     '*',
	 * )
	 * </pre>
	 * Note that when a name appears as both a model attribute and a virtual attribute, the position of
	 * the star element in the array determines which one takes precedence. In particular, if the star
	 * element is the first element in the array, the model attribute takes precedence; and if the star
	 * element is the last one, the virtual attribute takes precedence.
	 */
	public $attributes=array();
	/**
	 * @var string the name of the GET parameter that specifies which attributes to be sorted
	 * in which direction. Defaults to 'sort'.
	 */
	public $sortVar='sort';
	/**
	 * @var string the tag appeared in the GET parameter that indicates the attribute should be sorted
	 * in descending order. Defaults to 'desc'.
	 */
	public $descTag='desc';
	/**
	 * @var mixed the default order that should be applied to the query criteria when
	 * the current request does not specify any sort. For example, 'name, create_time DESC' or
	 * 'UPPER(name)'.
	 *
	 * Starting from version 1.1.3, you can also specify the default order using an array.
	 * The array keys could be attribute names or virtual attribute names as declared in {@link attributes},
	 * and the array values indicate whether the sorting of the corresponding attributes should
	 * be in descending order. For example,
	 * <pre>
	 * 'defaultOrder'=>array(
	 *     'price'=>true,
	 * )
	 * </pre>
	 *
	 * Please note when using array to specify the default order, the corresponding attributes
	 * will be put into {@link directions} and thus affect how the sort links are rendered
	 * (e.g. an arrow may be displayed next to the currently active sort link).
	 */
	public $defaultOrder;
	/**
	 * @var string the route (controller ID and action ID) for generating the sorted contents.
	 * Defaults to empty string, meaning using the currently requested route.
	 */
	public $route='';
	/**
	 * @var array separators used in the generated URL. This must be an array consisting of
	 * two elements. The first element specifies the character separating different
	 * attributes, while the second element specifies the character separating attribute name
	 * and the corresponding sort direction. Defaults to array('-','.').
	 */
	public $separators=array('-','.');
	/**
	 * @var array the additional GET parameters (name=>value) that should be used when generating sort URLs.
	 * Defaults to null, meaning using the currently available GET parameters.
	 * @since 1.0.9
	 */
	public $params;

	private $_directions;

	/**
	 * Constructor.
	 * @param string $modelClass the class name of data models that need to be sorted.
	 * This should be a child class of {@link CActiveRecord}.
	 */
	public function __construct($modelClass=null)
	{
		
	}

	/**
	 * Modifies the query criteria by changing its {@link CDbCriteria::order} property.
	 * This method will use {@link directions} to determine which columns need to be sorted.
	 * They will be put in the ORDER BY clause. If the criteria already has non-empty {@link CDbCriteria::order} value,
	 * the new value will be appended to it.
	 * @param CDbCriteria $criteria the query criteria
	 */
	public function applyOrder($criteria)
	{
		$order=$this->getOrderBy();
		if(!empty($order))
		{
			if(!empty($criteria->orders))
				$criteria->orders.=', ';
			$criteria->orders.=$order;
		}
	}

	/**
	 * @return array the orderby represented by this sort object.
	 */
	public function getOrderBy()
	{
		$directions=$this->getDirections();
		if(empty($directions))
			return is_array($this->defaultOrder) ? $this->defaultOrder : array();
		else
		{
			$orders=array();
			foreach($directions as $attribute=>$descending)
			{
				if($descending)
					$orders[]=isset($definition['desc']) ? $definition['desc'] : $attribute.' DESC';
				else
					$orders[]=isset($definition['asc']) ? $definition['asc'] : $attribute.' ASC';
			}
			return implode(', ',$orders);
		}
	}

	/**
	 * Returns the currently requested sort information.
	 * @return array sort directions indexed by attribute names.
	 * The sort direction is true if the corresponding attribute should be
	 * sorted in descending order.
	 */
	public function getDirections()
	{
		if($this->_directions===null)
		{
			$this->_directions=array();
			if(isset($_GET[$this->sortVar]))
			{
				$attributes=explode($this->separators[0],$_GET[$this->sortVar]);
				foreach($attributes as $attribute)
				{
					if(($pos=strrpos($attribute,$this->separators[1]))!==false)
					{
						$descending=substr($attribute,$pos+1)===$this->descTag;
						if($descending)
							$attribute=substr($attribute,0,$pos);
					}
					else
						$descending=false;

					if(($this->resolveAttribute($attribute))!==false)
					{
						$this->_directions[$attribute]=$descending;
						if(!$this->multiSort)
							return $this->_directions;
					}
				}
			}
			if($this->_directions===array() && is_array($this->defaultOrder))
				$this->_directions=$this->defaultOrder;
		}
		return $this->_directions;
	}

	/**
	 * Returns the sort direction of the specified attribute in the current request.
	 * @param string $attribute the attribute name
	 * @return mixed the sort direction of the attribut. True if the attribute should be sorted in descending order,
	 * false if in ascending order, and null if the attribute doesn't need to be sorted.
	 */
	public function getDirection($attribute)
	{
		$this->getDirections();
		return isset($this->_directions[$attribute]) ? $this->_directions[$attribute] : null;
	}

	/**
	 * Creates a URL that can lead to generating sorted data.
	 * @param CController $controller the controller that will be used to create the URL.
	 * @param array $directions the sort directions indexed by attribute names.
	 * The sort direction is true if the corresponding attribute should be
	 * sorted in descending order.
	 * @return string the URL for sorting
	 */
	public function createUrl($controller,$directions)
	{
		$sorts=array();
		foreach($directions as $attribute=>$descending)
			$sorts[]=$descending ? $attribute.$this->separators[1].$this->descTag : $attribute;
		$params=$this->params===null ? $_GET : $this->params;
		$params[$this->sortVar]=implode($this->separators[0],$sorts);
		return $controller->createUrl($this->route,$params);
	}

	/**
	 * Returns the real definition of an attribute given its name.
	 *
	 * The resolution is based on {@link attributes} and {@link CActiveRecord::attributeNames}.
	 * <ul>
	 * <li>When {@link attributes} is an empty array, if the name refers to an attribute of {@link modelClass},
	 * then the name is returned back.</li>
	 * <li>When {@link attributes} is not empty, if the name refers to an attribute declared in {@link attributes},
	 * then the corresponding virtual attribute definition is returned. Starting from version 1.1.3, if {@link attributes}
	 * contains a star ('*') element, the name will also be used to match against all model attributes.</li>
	 * <li>In all other cases, false is returned, meaning the name does not refer to a valid attribute.</li>
	 * </ul>
	 * @param string $attribute the attribute name that the user requests to sort on
	 * @return mixed the attribute name or the virtual attribute definition. False if the attribute cannot be sorted.
	 */
	public function resolveAttribute($attribute)
	{
		if($this->attributes!==array())
			$attributes=$this->attributes;
		else if($this->modelClass!==null)
			$attributes=CActiveRecord::model($this->modelClass)->attributeNames();
		else
			return false;
		foreach($attributes as $name=>$definition)
		{
			if(is_string($name))
			{
				if($name===$attribute)
					return $definition;
			}
			else if($definition==='*')
			{
				if($this->modelClass!==null && CActiveRecord::model($this->modelClass)->hasAttribute($attribute))
					return $attribute;
			}
			else if($definition===$attribute)
				return $attribute;
		}
		return false;
	}

	/**
	 * Creates a hyperlink based on the given label and URL.
	 * You may override this method to customize the link generation.
	 * @param string $attribute the name of the attribute that this link is for
	 * @param string $label the label of the hyperlink
	 * @param string $url the URL
	 * @param array $htmlOptions additional HTML options
	 * @return string the generated hyperlink
	 */
	protected function createLink($attribute,$label,$url,$htmlOptions)
	{
		return CHtml::link($label,$url,$htmlOptions);
	}
}



How to use (live example):

	//SphinxSearch criteria
	$searchCriteria = new stdClass();
	$searchCriteria->select = '*';
	$searchCriteria->query = ''.$search_string.'';
	$searchCriteria->from = 'index_name';
	//...
	$filters['brand_id'] = $brands_array; //filter by brand ids, if needed
	if(!empty($filters)) $searchCriteria->filters = $filters;
	//...
	//items criteria
	$criteria = new CDbCriteria;
	$criteria->with = array('brand','category');
	//...
	$catalog = new SphinxDataProvider('CatItem',
		array(
			'criteria' => $criteria, //criteria for AR model
			'sphinxCriteria' => $searchCriteria, //SphinxSearch critria
			'pagination'=>array(
				'pageSize' => $num,
				'pageVar' => 'p',
			),
			'sort' => array(
				'attributes'=>array(
					'price'=>array(
						'asc' => 'price ASC',
						'desc' => 'price DESC',
					),
					'title'=>array(
						'asc' => 'title ASC',
						'desc' => 'title DESC',
					),
				),
			),
	));


And don't forget DGSphinxSearch fix: http://www.yiiframew...inxsearch#c9471


How to sort by the weight of sphinx searching results instead of database fields?
0

#12 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 16 November 2012 - 02:12 AM

Quote

How to sort by the weight of sphinx searching results instead of database fields?


This is by default, so just don't sort (remove sort array from data provider).
0

#13 User is offline   yiiqs2 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 33
  • Joined: 21-August 13

Posted 21 August 2013 - 04:16 AM

Hi...
where location path to save SphinxDataProvider class above ?

thanks
0

#14 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 21 August 2013 - 04:31 AM

It depends on 'import' section in the config.

'import'=>array(
		'application.models.*',
		'application.components.*',
...


So put files in the protected/components/ folder
0

#15 User is offline   yiiqs2 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 33
  • Joined: 21-August 13

Posted 22 August 2013 - 03:15 AM

Hi all...

how to display result data from sphinx to datagrid ? using your code above.


Thanks.
0

#16 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 22 August 2013 - 03:48 AM

SphinxDataProvider - this is the data provider class for CListView (http://www.yiiframew...Provider-detail).

http://www.yiiframew...ataprovider#hh5
0

#17 User is offline   yiiqs2 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 33
  • Joined: 21-August 13

Posted 23 August 2013 - 03:34 AM

Thanks yuri..
nice to meet you.


i have create SphinxDataProvider class inside "protected/components" folder....
i have sphinx on my windows using 9321 port, my sphinx can work fine.

i implement your guidance :
    //SphinxSearch criteria
        $searchCriteria = new stdClass();
        $searchCriteria->select = '*';
        $searchCriteria->query = ''.$search_string.'';
        $searchCriteria->from = 'index_name';
        //...
        $filters['brand_id'] = $brands_array; //filter by brand ids, if needed
        if(!empty($filters)) $searchCriteria->filters = $filters;
        //...
        //items criteria
        $criteria = new CDbCriteria;
        $criteria->with = array('brand','category');
        //...
      $catalog = new SphinxDataProvider('CatItem',
                array(
                        'criteria' => $criteria, //criteria for AR model
                        'sphinxCriteria' => $searchCriteria, //SphinxSearch critria
                        'pagination'=>array(
                                'pageSize' => $num,
                                'pageVar' => 'p',
                        ),
                        'sort' => array(
                                'attributes'=>array(
                                        'price'=>array(
                                                'asc' => 'price ASC',
                                                'desc' => 'price DESC',
                                        ),
                                        'title'=>array(
                                                'asc' => 'title ASC',
                                                'desc' => 'title DESC',
                                        ),
                                ),
                        ),
        ));


i get error messages :

"$catalog = new SphinxDataProvider('CatItem',"

include(CatItem.php): failed to open stream: No such file or directory..

i don't know mean of this messages :
"include(CatItem.php): failed to open stream: No such file or directory.."

please tell me...

i confused. .

thank so much.
0

#18 User is offline   rei 

  • Advanced Member
  • PipPipPip
  • Yii
  • Group: Members
  • Posts: 332
  • Joined: 10-November 10

Posted 19 November 2013 - 01:37 AM

Yuri, first of all I would like to thank you for sharing your code with us! I believe this needs to be included in the actual extension (DGSphinxSearch).

 yiiqs2, on 23 August 2013 - 03:34 AM, said:

i don't know mean of this messages :
"include(CatItem.php): failed to open stream: No such file or directory.."


I think you need to replace 'CatItem' with your model name.


 ixolit, on 27 August 2012 - 08:53 AM, said:

The only thing that doesn't work for me is line in method calculateTotalItemCount()


I got the same problem. Total item count returns the number of all records (not filtered by Sphinx search). So when displaying CListView, it shows something like 'Displaying 5-5 of 9 results' (there are only 5 records in search result, but it shows 9 results as total).

Is there any workaround for this?
Fipick - Find and pick recommendations
0

#19 User is offline   Yuri! 

  • Junior Member
  • Pip
  • Yii
  • Group: Members
  • Posts: 43
  • Joined: 30-December 10

Posted 22 November 2013 - 09:59 AM

Quote

I think you need to replace 'CatItem' with your model name.


You right


Quote

I got the same problem. Total item count returns the number of all records (not filtered by Sphinx search). So when displaying CListView, it shows something like 'Displaying 5-5 of 9 results' (there are only 5 records in search result, but it shows 9 results as total).

Is there any workaround for this?


Fixed.

class SphinxDataProvider extends CDataProvider
{
  ...
  private $_totalItemsCount;
  ...
  $this->_totalItemsCount = isset($resArray['total_found']) ? $resArray['total_found'] : 0;
  ...
  $pagination->setItemCount($this->_totalItemsCount);
  ...
  protected function calculateTotalItemCount()
  {
	return $this->_totalItemsCount;
  }
}


Changed in the my original post also.

Please check.
1

#20 User is offline   rei 

  • Advanced Member
  • PipPipPip
  • Yii
  • Group: Members
  • Posts: 332
  • Joined: 10-November 10

Posted 27 November 2013 - 02:38 AM

 Yuri!, on 22 November 2013 - 09:59 AM, said:

Fixed.
...


Changed in the my original post also.
Please check.


+1. Thank you very much, Yuri!
Fipick - Find and pick recommendations
0

Share this topic:


Page 1 of 1
  • You cannot start a new topic
  • You cannot reply to this topic

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users