How to set up Unicode

19 0

Viewed: 151 930 times

Version: Unknown (update)

Category: Tutorials

Written by:

Mike

Updated by:

Roman Solomatin

Created: Feb 21, 2009

Updated: 13 years ago

You are viewing revision #11 of this wiki article.
This version may not be up to date with the latest version.
You may want to view the differences to the latest version.
next (#13) »

1. PHP script files
2. Database tables
3. Database connection
4. Webserver/HTTP-Header
5. PHP string functions

To fix issues with display of special language characters once and for all there's a solution: Use Unicode (UTF-8) everywhere. If everything's set up to use Unicode, you can use mostly every language in your application.

There are several places that all may need some configuration tuning to use Unicode:

1. PHP script files ¶

Make sure that you use an editor which is capable of using UTF-8 and save all your files UTF-8 encoded without BOM. If you have some older non-unicode files in your project open them with your editor and save them again UTF-8 encoded. On Linux you can also use command line tools like recode or iconv to convert a whole bunch of files.

2. Database tables ¶

Every table in your database needs to use UTF-8 charset for its content. The configuration for that might differ between database systems.

MySQL ¶

To find out if a table uses utf8 charset you have to look at the CREATE statement for that table. You can use phpMyAdmin's export feature and look at the CREATE statement.

>Info: Don't confuse the encoding of characters in a table with its collation. The latter is used for sorting in queries and can be changed easily with e.g. phpMyAdmin or even for a single query.

You could also issue this SQL statement:

[sql]
SHOW CREATE TABLE your_tablename;

You'll see a CREATE statement with the CHARSET information at the end. It should like this:

[sql]
CREATE TABLE IF NOT EXISTS `your_tablename` (
  .... your field definitions ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

If your table doesn't use UTF-8 charset yet the easiest way to change this is to export your table, adapt the CREATE statement's CHARSET parameter and re-import your table again into the database.

Be very careful when doing this conversion and make sure you save the file with the changed SQL statement in UTF-8 and convert it if neccessary. If not performed carefully you can easily end up with messed up encodings, e.g. having ISO-8859-1 encoded characters in a table with utf8 CHARSET.

>Tip: To have MySQL create all of your tables with utf8 >CHARSET by default, you can add this to your MySQL >configuration (e.g. my.cnf file): > >~~~ >[mysqld] >character-set-server = utf8 ># for older versions: >default-character-set = utf8 >~~~

3. Database connection ¶

When connecting to a database a client like PHP has to use a specific charset encoding. To specify the charset to use for a connection in Yii, configure it like this:

return array(
    ......
    'components'=>array(
        ......
        'db'=>array(
            'connectionString'=>'sqlite:protected/data/source.db',
            'charset'=>'utf8',
        ),
    ),
    ......
);

4. Webserver/HTTP-Header ¶

We also need to let the browser know, that we use UTF-8 with our pages. The best place to do this is in the header of an HTTP response. Configuring this varies between different server software.

>Tip: If you use this approach, there's no need to add additional header information about encoding to your pages. Using the HTTP header is enough.

Apache ¶

You can configure UTF-8 charset either in a VirtualHost section of your server configuration or by adding this line into a .htaccess file in your DocumentRoot:

AddDefaultCharset UTF-8

5. PHP string functions ¶

PHP needs to use UTF-8 internally in order for e.g. string length validation to work correctly. Full Unicode support will be available in PHP 6 and is still work in progress. For the time being the alternative is to use mbstring functions instead of the non-multibyte aware counterparts. Since mbstring is a non-default extension it might not be available on every host. That's one of the reasons why Yii uses the non-multibyte functions like strlen() instead of mb_strlen().

One workaround is to use mbstring's function overloading feature. This will override then non-multibyte aware functions with their mbstring counterpart. To set this up add this in your php.ini:

mbstring.func_overload "7"
mbstring.internal_encoding "UTF-8"

As an alternative you can also enable it for a single VirtualHost in Apache in the according configuration section:

php_admin_value mbstring.func_overload "7"
php_admin_value mbstring.internal_encoding "UTF-8"

>Note: Unfortunately it's not recommended to set this an an .htaccess file as this may lead to undefined behavior.

Links ¶

Chinese version

#151

1 0

Shell script

Here's the shell script i use in cygwin to remove the ... BOM

#!/bin/bash
for i in $(grep -rli $'\xEF\xBB\xBF' --include=*.php /cygdrive/c/PHP-projects/toto); do
	echo Processing $i;
	cp $i $i.bak
	cat $i | perl -pe 's/\xEF\xBB\xBF//i' > $i.new;
	mv $i.new $i;
done

cma at Aug 26, 2010, 7:48:38 AM

#595

1 0

Individual fields of the table

I was struggling to get utf8 work, my problem was that even though the DEFAULT CHARSET=utf8 was set to all of the tables, individual fields were having latin COLLATION and who knows what CHARACTER SET...

I had to do smth like this with all of the individual fields in the tables:
ALTER TABLE tbl_example DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

I hope this helps someone.

Karolis at Apr 14, 2010, 4:18:16 PM

#708

1 0

php.ini

If your php.ini has a default_charset set then everything might just get ignored (like in my case).

Just put this at the begining of the entry script (index.php)
ini_set('default_charset','utf-8');

Troto at Mar 15, 2010, 12:45:09 AM

#1023

1 0

SETTING THE CHARSET - CORRECT

Just add on your 'rootApp/protected/config/main.php' the correct charset of your app on the root of the return array, like that:

'charset'=>'iso-8859-1',

So change your main layout to use the charset off your app, changing the meta tag of header section:

< meta http-equiv="Content-Type" content="text/html; charset=<?= Yii::app()->charset ?>" />

And your problems are solved, without need to do conversions and etc.

Bye!

Solleon at Jan 7, 2010, 1:34:10 PM

#1043

1 0

Remove UTF-8 BOM from ouput

You can remove the UTF-8 BOM from the output using the ob_start function. This way you can leave the UTF-8 BOM in your source files so your editor understands it is really UTF-8.

In the /protected/config/main.php you have to add before returning the config array:

ob_start('My_OB');
function My_OB($str, $flags)
{
    //remove UTF-8 BOM
    $str = preg_replace("/\xef\xbb\xbf/","",$str);
    return $str;
}
return array( ... yii config array ...);

P.S. You don't have to call ob_end_flush(), php will do this automatically at the end of the script.

Mirco at Dec 28, 2009, 11:31:19 AM

#1946

3 0

set names

In my installation I have to do even ini set, in order to have database and application with the same data:

return array(
    ......
    'components'=>array(
        ......
        'db'=>array(
            'connectionString'=>'sqlite:protected/data/source.db',
            'charset'=>'utf8',
            'initSQLs'=>array('set names utf8'),
       ),
    ),
    ......
);

zaccaria at Oct 18, 2010, 2:18:47 PM

#2963

1 0

Really helpful

It's important to remember to set encoding parameter when using CStringValidator with not latin charactors

Leric at Mar 3, 2011, 5:12:28 AM

#5383

1 0

Great article

Very useful stuff. Some of this knowledge I learned the hard way... over the years.

Roman Solomatin at Oct 8, 2011, 7:14:30 PM

#12768

0 0

Cleanup

To the other editors: I've cleaned up and reorganized the article. I think, some content was not really part of the HOWTO (e.g. the section about DB indexes). If you still think, that's useful information please add it as a comment here.

Mike at Apr 11, 2013, 7:23:56 AM

#16006

0 0

Unicode routing

How to setup unicode routing? Is there any option in hosting or is it in the urlManager in main.php in config dir?

Firebreaker at Jan 8, 2014, 3:19:21 PM

Please only use comments to help explain the above article.
If you have any questions, please ask in the forum instead.

Join the conversation to share a note.