Bearbeiten von „TYPO3 auf utf-8 umstellen“

iQRPLW  <a href="http://fedhujaiscrp.com/">fedhujaiscrp</a>, [url=http://dfxrbfjpcwky.com/]dfxrbfjpcwky[/url], [link=http://pcjcxxhhfgwy.com/]pcjcxxhhfgwy[/link], http://gwtlherjdgxn.com/

== Überblick / Overview ==
Verarbeitungskette prüfen vhost.conf - php.ini - my.cnf

Wichtige Einstellung die leicht übersehen wird:

Install Tool: Unter „All Configuration“ muss nun „[forceCharset]“ auf „utf-8“ und „[setDBinit]“ den Eintrag „SET NAMES utf8;“ erhalten. Der zusätzliche Eintrag „SET CHARACTER_SET utf8;“, der dem Feld „[setDBinit]“ hinzugefügt werden kann, ist oft nicht notwendig.

=== Verarbeitungskette ===
Zuerst mal testen ob in der Verarbeitungskette überall utf-8 für die Übertragung von Daten verwendet wird:


==== Apache: vhost.conf ====

AddDefaultCharset UTF-8

Dies ist für TYPO3 Versionen spätestens ab 4.3 nicht notwendig. Wenn man hier im Install Tool ''[BE][forceCharset]'' gesetzt hat

Kann einfach getestet werden indem man sich die Header Daten einer Seite mit Firebug oder LiveHTTPTracker anschaut. Darin sollte eine Zeile in dieser Art enthalten sein:

 Content-Type: text/html; charset=utf-8

==== PHP: php.ini ====
 default_charset = "utf-8"

Mit dieser Einstellung sollten auch standalone Skripte mit dem richtigen Zeichensatz arbeiten. Man kann in den Skripten aber auch 

Außerdem sollte mit der Erweiterung mbstring oder iconf gearbeitet werden. Deren Performance ist wesentlich besser als die Umwandlung über den PHP-Code von TYPO3 (s.u.)

==== MySQL: my.cnf ====
Die folgende Einstellung setzt alle System Variablen für Zeichensätze und Kollationen für den MySQL SERVER. Dies betrifft also auch existierende Datenbanken. Deshalb sollte diese Einstellung nur gesetzt werden wenn ausschließlich mit utf-8 Datenbanken gearbeitet wird.

Man benötigt die Einstellung nicht wenn im Install Tool ''['SYS']['setDBinit']''  gesetzt ist.

=== TYPO3 Einstellungen ===

==== localconf.php ====

<pre>
// For backend charset
$TYPO3_CONF_VARS['BE']['forceCharset'] = 'utf-8';

$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;'; 

// For GIFBUILDER support
// Set it to 'iconv' or 'mbstring'
$TYPO3_CONF_VARS['SYS']['t3lib_cs_convMethod'] = 'mbstring';
// For 'iconv' support you need at least PHP 5!
$TYPO3_CONF_VARS['SYS']['t3lib_cs_utils'] = 'mbstring';
</pre>

Hinweise:
* Wenn die Datenbank auf  UTF-8 gesetzt ist, sollte man '''nicht die Einstellung $TYPO3_CONF_VARS['SYS']['multiplyDBfieldSize'] setzen'''. Diese wird nur benötigt, wenn die Datenbank mit latin 1 arbeitet aber der Inhalt  utf-8 ist. Außerdem führt die Einstellung des öfteren zu Problemen.
*  $TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;'; Setzt folgende 3 Einstellungen:
<pre>
 SET character_set_client = utf8; 
 SET character_set_results = utf8; 
 SET character_set_connection = utf8; 
</pre>

* Wenn '''in phpMyAdmin Sonderzeichen als 2 seltsame Zeichen''' angezeigt werden, reicht diese Einstellung nicht. Zuerst müssen die Inhaltstabellen konvertiert werden (oder löschen, umstellen, neu eingeben.

* In '''manchen Konfigurationen''' muß man auch noch eine '''Einstellung für die Sessions''' machen:

 SET NAMES utf8;
 SET SESSION character_set_server=utf8;

* '''Benutze niemals SET CHARACTER SET utf8;''' Dies kann große Probleme im TYPO3 Umfeld erzeugen. Mit der Einstellung SET NAMES hat sich das ohnehin erledigt, weil hier eigentlich die wichtigen Einstellungen gemacht werden

* Wenn [BE][forceCharset] auf utf-8 gesetzt ist (siehe oben), dann wird auch ''config.renderCharset'' und ''config.metaCharset'' per default mit utf-8 arbeiten. Deshalb müssen hier keine Einstellungen mehr gemacht werden.

=== Extensions ===

==== RealURL ====

One problem is that RealURL might not be able to understand a page title if it is in unusual (i.e. not Roman) characters. For example, with a page title in Japanese, I found that the title was not interpreted and the page was rendered as jp.html. Using the Navigation title solves this problem (to follow on the example, setting "home" as the Navigation title, my page was then rendered as jp/home.html).

==== TemplaVoila ====

Templates müssen im UTF-8 Format gespeichert werden. Möglicherweise muß neu gemappt werden.


=== Weitere Hinweise ===

==== Extensions sollten nicht strlen() benutzen====
Stattdessen sollte die Funktion t3lib_cs benutzt werden.

strlen() doesn't care for UTF-8. UTF-8 uses 1 to 3 Bytes for one char, so using strlen() will likely lead to wrong results. Extensions should instead use the functions provided in t3lib_cs.

==== Sortierung ====
It is highly recommended (although not strictly necessary) to use UTF-8 in the database. Otherwise database sorting functions will not work correctly.
[edit] Problem with indeces

==== Fehlermeldung Specified key was too long bei indizes====

SQL=Specified key was too long; max key length is 1000 bytes:

This particular problem might occur when you are using UTF-8 encoding. UTF-8 uses up to 3 bytes per character, and the maximum index length is 1000 bytes, so the effective maximum index is 1000/3 = 333 characters. Some tables are longer than this, hence the error (many other packages are being bitten by this issue too).

To solve this, simply remove the index from that field and reload.

Note: Using indeces that big anyway is not recommended and shows bad DB design.
[edit] Convert an already existing database to UTF-8


=== Konvertierung einer bestehenden Datenbank ===
Some links to the conversion topic:

    * http://dev.mysql.com/doc/refman/4.1/en/charset-convert.html (MySQL based conversion)
    * http://www.typo3-media.com/blog/article/utf8-and-typo3-updated.html
    * http://m.tacker.org/blog/64.script-to-convert-wordpress-content-encoding.html (useful PHP script to convert charsets) 


[edit] Possibility 1
Note 	This has been tested and works!


Jigal van Hemert wrote a script, which can convert your database to utf-8. He offers it for download here: [[http://www.xs4all.nl/~dcbjht/typo3/db_utf8_fix.zip]] . This script converts all columns, tables and finally the setting for the whole database to utf-8.

Jigal writes:

Read the following very carefully, because you have to make a few adjustments depending on the situation!

    * Make a backup of your database before you do such a conversion. 

    * The script was meant for the situation where utf-8 encoded data was stored in latin-1 (or other charsets) tables. This was very common in 2008. You can recognize this by looking in PHPmyAdmin and you find that characters with accents (diacriticals) are shown as weird double-character combinations:
      Instead of "Ali Gökgöz and Gültekin Tarcan" it shows as "Ali GÃ¶kgÃ¶z and GÃ¼ltekin Tarcan".
      If this is *not* the case in your situation, turn lines 97 - 107 into comments by putting // in front of them. 

    * In line 19 of the script it says define("SIMULATE", TRUE); This makes sure that the tables are not really converted, it's just a practice run; a simulation. After you executed the script at least once and there are no errors you can turn this into define("SIMULATE", FALSE); to do the conversion for real. 

    * Put the script in a subdirectory of the TYPO3 installation, for example inside 'fileadmin'. It is designed to run from a subdirectory so it can pick up the database connection data from localconf.php. 

    * Run the script from your browser: http://domain.ext/fileadmin/db_utf8_fix.php It shows each table it found and after the name of a table a dot for each column it has converted. 

    * It really doesn't matter if there are columns/tables already in utf-8 format. 

 Settings in TYPO3 

Furthermore, if you use the following settings in the Install Tool you should have a UTF-8 installation:

$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;';
$TYPO3_CONF_VARS['BE']['forceCharset'] = 'utf-8';

 Possibility 2

Dump your database, modifiy the dumped file and import it again.
Note 	If you do it that way, setting ['BE']['forceCharset'] might cause broken special chars inside TYPO3. See below for more information.


Requirements:

    * Shell access to your unix based server
    * "Sed" package installed on the server 

For this example we assume:

    * hostname: domain.com
    * database: typo3 


This example is for linux users. If your PC uses Windows, you can do the same using putty.exe. Enter the hostname in the field "Host Name (or IP adress)" and click on "Open". Then enter your ssh username, press enter and enter the password (which will not be displayed) and press enter. Then you are connected to the server.

Linux users connect to the server via ssh typing

ssh -l (user) domain.com

Create a backup of the database (for security reasons if things go wrong...)

mysqldump -u (user) -p(pass) --max_allowed_packet=10000000 typo3 > typo3_backup.sql

Dump database (without table typo3.sys_refindex. This prevents the following error: SQL=Specified key was too long; max key length is 1000 bytes. You have to rebuild the reference index afterwards!)

mysqldump -u (user) -p(pass) --max_allowed_packet=10000000 --ignore-table=typo3.sys_refindex  typo3  > typo3_utf8.sql

Now modifiy the dump: Newer versions of MySQL (at least 5.0) also save the collation for each column seperately. You have to convert all of them:

First convert all occurences of "DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci" (use the character set which you have written in your file) in typo3_utf8.sql to "DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci":

sed  -e 's/DEFAULT CHARSET=latin1 COLLATE=latin1_german1_ci/DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci/g' -i typo3_utf8.sql

Then convert all occurences of "COLLATE latin1_german1_ci" (use the charset you have written in your file) to "COLLATE utf8_general_ci".

sed  -e 's/COLLATE latin1_german1_ci/COLLATE utf8_general_ci/g' -i typo3_utf8.sql

Import database

mysql -u (user) -p(pass) --default-character-set=utf8  typo3 < typo3_utf8.sql

Alter character set and collation for the whole database

mysql -u (user) -p(pass) -e "ALTER DATABASE typo3 DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci"

[edit] Settings in TYPO3

Furthermore, if you use the following settings in the Install Tool you should have a UTF-8 installation:

$TYPO3_CONF_VARS['SYS']['setDBinit'] = 'SET NAMES utf8;';
$TYPO3_CONF_VARS['BE']['forceCharset'] = 'utf-8';

[edit] Broken special chars?

If the result of the above mentioned is that special chars are displayed incorrectly in TYPO3 (a small black box with a question mark in it instead of the special char), the following might help:

Create a new database. Make sure that it uses utf8 as default charset and utf8_general_ci as collation:

mysql -user (user) -password(password) -e "ALTER DATABASE (new-db) DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;" 

Then import the dump into that database without using sed to replace the occurences of latin1 (or what you have) with utf8.

The result will be that the tables and columns in your database still use latin1 (or what you had before).

This might be a problem, e.g. when you now add new tables to this database, they will use utf8 as charset, because the database is set to utf8. This will lead to a mix of both charsets in your DB.
[edit] Possibility 3

This might be the way to go for german speaking users with a Unix server:

A way similar to possibility 2 is recommended by t3n (german).

Basically they make the dump and replace the charset and collation statements.

Then they use iconv on the dumped file to convert the signs inside:

iconv -f iso-8859-1 -t utf8 dump.sql > dump-iconv.sql

After that they import the file using the switch --default-character-set=utf8:

mysql -u USER -p PASSWORT -h HOST --default-character-set=utf8 DB < dump-iconv.sql


Note 	If you did that and get umlauts displayed correctly, but ß and € displayed wrong inside TYPO3, you should use CP1252 in the iconv command as origin charset like that:

iconv -f CP1252 -t utf8 dump.sql > dump-iconv.sql

=== Filesystem ===

When setting [BE][forceCharset] all files in the TYPO3 root folder and below are handled as UTF-8 files by TYPO3; so you should make sure that they really are. You should e.g. check your HTML-Templates and CSS files for special chars like umlauts. If they are displayed incorrectly, you should fix that by saving the file in UTF-8 format.

When editing such files only use editors which can save files in UTF-8 format. That is any editor on Linux and e.g. Notepad++ if you use Windows.

Attention: Do not save the files in UTF-8 format with Byte Order Mark (BOM). Saving them as UTF-8 with BOM can cause problems, e.g. thumbnails in the BE will no longer be shown. Save the files in ANSI format (as UTF-8 without BOM) instead.
[edit] t3lib_cs

Developers: Use these functions e.g. to get the length of a string. strlen doesn't get the correct string-length, because the chars of UTF-8 can have 1...3 Bytes.

In PHP 5.3 PECL/intl will be available, so maybe the TYPO3 Core-Developers switch to this.
[edit] Use unicode fonts

If you use Gifbuilder to create some lines of text (e.g. in a menu) make sure that the font file you use is unicode. More info here.

If there still are problems with broken special chars in these images, you should make sure that the configuration for mb_string or iconv (the one which you set in the Install Tool) is set to UTF-8. You can check that in phpinfo() and correct the settings in php.ini or .htaccess, if needed.
[edit] HTML Tidy

If you are having problems with html entities like &nbsp; shown as ? in the browser, add the -utf8 option to the HTML tidy_path variable in the install tool, e.g.

 $TYPO3_CONF_VARS['FE']['tidy_path'] = 'tidy -i --quiet true --tidy-mark true -wrap 0 -raw --output-xhtml true -utf8'

==== External links ====

    * http://dev.mysql.com/doc/refman/5.0/en/charset.html
    * http://en.opensuse.org/SDB%3AConverting_Files_or_File_Names_to_UTF-8_Encoding
    * How to change the encoding like iso-8859-1, iso-8859-15, utf-8 of files: http://linuxwiki.de/tcs (just in german in the moment)