unicode (urdu in yii)

For almost a month I am trying to display urdu, arabic and punjabi in yiiframework, but failed.

The words to display in these languages are coming from mysql database.

In previous code, using the same database, everything is working fine. That means, my browser and server is able to display unicode (UTF-, and mysql table is also set to have UTF-8.

Here is a working example of what I have

901

However using yii, I cannot have the desired output.

I have also changed the CStringValidator encoding="UTF-8" from "false", but to no avail.

Here is what I am getting in yii

902

Please note that, even at the time of capturing the images, both old and yii based codes are using the same database, and not to mention the server and operating system.

Kindly help.

Have you set the database charset to UTF-8?

Example

        'db' => array(

            'connectionString' => 'mysql:host=localhost;dbname=yiitest',

            'emulatePrepare' => true,

            'username' => 'root',

            'password' => 'root',

            'charset' => 'utf8',

        ),

Sorry I forget to mention that I have followed the guide at

http://www.yiiframework.com/wiki/16/how-to-set-up-unicode/

And set up everything excluding the Apache because Apache is already showing the desired results from my old script.

I have

1. Php files saved with UTF-8 without BOM

2. Database tables set to UTF-8

3. The database connection set to utf-8

4. The carset=utf-8 in the content tag in header of output html.

5. Set the CStringValidator’s encoding to UTF-8.

that’s the place where I am stuck.

Can’t think of anything other…

Can you test if the output from Yii php scripts works - just put echo "your characters" somewhere in the view… just to see if the problem is only with the data read from the database…

Thanks for the line, I think now we are getting somewhere

The simple echo works fine, and I tried to store the same words in a variable and echoed it also, works too.

The reduces the problem to either Mysql Database output, or the yii’s stringvalidator class.

To Cut it short, my phpmyadmin (browser based) and mysql query browser (separate GUI), both are unable to show the characters and behaves much like yii. However, what confuses me, if both of them ( and including yii make them 3) cannot display the characters, how can my old script is still doing it from the same database at the same operating system and using the same web server?

p.s.: I am using windows 7 (IIS 7.5) and ubuntu 10.10 (apache2) dual boot, using single location for database and web root for both of them, i.e. I don’t have to copy files to other location to access the website in either os.

At least we are one step ahead…

to check if it’s the stringvalidator or any other thing… try to remove it…

so that you just read the data from the database and output it without any processing…

Ok, thanks for this advice also, I have tried that, and used a simple mysql_query function placed inside the SiteController’s actionIndex and passed the result to index.php in views->site and iterated them, working fine and displaying the results accurately in different unicode languages.

Whats next?

I think the problem lies either in mysql object of yii, or the CStringValidator.

The CStringValidator just checks the length of the data it does not change the data… must be something else…

You should try a simple query with the Yii model and output the value without any processing… just to so see if there you get the same output…

ok, thanks for the correction.

print_r($language) gives the same unwanted output.

I am using the line

$languages = Language::model()->findAll();

As a newbie, I cannot understand how to simplify it.

Would you please help me in that also?

You example is OK… just a read from the database and output…

You can try the ‘initSQLs’ option like @Zaccaria suggested here - http://www.yiiframework.com/forum/index.php?/topic/12821-mysql-and-its-funny-charsets/page__view__findpost__p__62733

I remember these kind of problems and the reason for me always was a mismatch in DB encoding. For example it could be that when you initially wrote your data to DB, the connection charset didn’t match the table charset. Then lot of things can happen. For me i wasn’t able to get anything useful back, except using the same wrong connection encoding when reading the data. If i remember well, my connection used ISO-8859-1 even though my DB table was UTF-8. So i always had to use ISO-8859-1 for the connection (and only there!) to get something useful back.

The only way out: If you somehow manage to get the data out correctly (which you say you can with your old script), try what i suggested in step 2. in the cookbook you mentioned: Try to export your data into a text file and carefully check that file’s encoding (using a good editor, maybe try different ones). Then double check again that all charsets are UTF-8 (File encoding, CREATE statements and DB Connection) and import the data back.

Thanks Mike, You are so right, the problem was not in fetching the data, it was in storing, simply removing the line ‘charset’=>‘utf8’ solved the problem and the data is now displayed very accurately.

The reason is the same you mentioned, before yii, I never checked what is the connection type of mysql.

Well, Changing the whole data from latin-1 to utf8 is quite a lot of work, and in future I am planning to do so.

Thanks for your valuable help, finally I can now use yii for multiple purposes.

I am planning to convert oscommerce to yii, does anyone has any idea is this work has started already?

Glad i could help.

I don’t think anyone is working on a port of oscommerce, that’s a very ambitious plan. Even though it would surely be very cool to have such a full featured shop system written in Yii.

The following project uses some code of yii in admin section

http://www.cartzlink.com/

Maybe the demo does not include yii part, but that’s quite a huge project and I have been a part of it.

The OsCommerce lacks security and is easily hacked.

Hi Rehan and Mike,

I am having the same issue with Arabic from my database. If I comment out the ‘charset’=>‘utf8’ line in the config then things are fine! Is it OK to leave it like this? What are the potential problems?

I know, I know. I’m being a totally lazy developer! But as the saying goes, if it aint broken, don’t fix it!

Thanks

Ziad

If i consider the amount of information that’s already contained in this thread, I think you have a point here …

Question: where the problem lies

Answer: In Mysql, storing UTF-8 string in a non-utf-8 table (usually latin or swedish).

      In my case, the charset of mysql table was set to utf-8 (very correct), but the charset of the actual column was WESTERN-ISO-8859 (used for English, German, French, Swedish and so on).

Solution: Check the mysql table and column, store them properly.

However I am still having the problem in getting my data out of mysql using phpmyadmin/mysql gui. because mysql gui is showing ?? and phpmyadmin some crappy characters.

The only way I have, which is a tedious one, is to write my own script using yii to generate sql statements, save the script in utf-8 and re-upload it.

So are you suggesting that the database, the table and the columns have to be specified to be utf8???

My database is quite small and it’s not live yet so I think I will just dump the database and rebuild it correctly again.

Thanks

Ziad

So sounds like the perfect thing to do now. Think about it: If you do it now, your lazyness will not be disturbed later on

Sounds like a very good idea.

Otherwise you’ll probably end up with double-encoded data:

http://forums.mysql.com/read.php?103,212447,212447