Php перекодировка windows 1251 в utf 8

Example:
<?php
setlocale(LC_CTYPE, 'POSIX');
echo iconv('UTF-8', 'ASCII//TRANSLIT', "Žluťoučký kůň\n");
// ?lu?ou?k? k??setlocale(LC_CTYPE, 'cs_CZ');
echo iconv('UTF-8', 'ASCII//TRANSLIT', "Žluťoučký kůň\n");
// Zlutoucky kun
?>

That means you can't currently use this function to filter invalid characters. Instead it silently fails and returns an empty string (or you'll get a notice but only if you have E_NOTICE enabled).

This has been a known bug with a known solution for at least since 2009 years but no one seems to be willing to fix it (PHP must pass the -c option to iconv). It's still broken as of the latest release 5.4.3.

https://bugs.php.net/bug.php?id=48147

https://bugs.php.net/bug.php?id=52211

https://bugs.php.net/bug.php?id=61484

[UPDATE 15-JUN-2012]

Here's a workaround...

ini_set('mbstring.substitute_character', "none");

$text= mb_convert_encoding($text, 'UTF-8', 'UTF-8');

That will strip invalid characters from UTF-8 strings (so that you can insert it into a database, etc.). Instead of "none" you can also use the value 32 if you want it to insert spaces in place of the invalid characters.

<?php
//some German
$utf8_sentence = 'Weiß, Goldmann, Göbel, Weiss, Göthe, Goethe und Götz';//UK
setlocale(LC_ALL, 'en_GB');//transliterate
$trans_sentence = iconv('UTF-8', 'ASCII//TRANSLIT', $utf8_sentence);//gives [Weiss, Goldmann, Gobel, Weiss, Gothe, Goethe und Gotz]
//which is our original string flattened into 7-bit ASCII as
//an English speaker would do it (ie. simply remove the umlauts)
echo $trans_sentence . PHP_EOL;//Germany
setlocale(LC_ALL, 'de_DE');$trans_sentence = iconv('UTF-8', 'ASCII//TRANSLIT', $utf8_sentence);//gives [Weiss, Goldmann, Goebel, Weiss, Goethe, Goethe und Goetz]
//which is exactly how a German would transliterate those
//umlauted characters if forced to use 7-bit ASCII!
//(because really ä = ae, ö = oe and ü = ue)
echo $trans_sentence . PHP_EOL;?>

<?php

$tab = array("UTF-8", "ASCII", "Windows-1252", "ISO-8859-15", "ISO-8859-1", "ISO-8859-6", "CP1256");

$chain = "";

foreach ($tab as $i)

{

foreach ($tab as $j)

{

$chain .= " $i$j ".iconv($i, $j, "$my_string");

}

}

echo

$chain;

?>



then after displaying, you use the $i$j that shows good displaying.

NB: you can add other charsets to $tab to test other cases.

The main problem here is that when your string contains illegal UTF-8 characters, there is no really straight forward way to handle those. iconv() simply (and silently!) terminates the string when encountering the problematic characters (also if using //IGNORE), returning a clipped string. The

<?php

$newstring

= html_entity_decode(htmlentities($oldstring, ENT_QUOTES, 'UTF-8'), ENT_QUOTES , 'ISO-8859-15');?>

workaround suggested here and elsewhere will also break when encountering illegal characters, at least dropping a useful note ("htmlentities(): Invalid multibyte sequence in argument in...")

I have found a lot of hints, suggestions and alternative methods (it's scary and in my opinion no good sign how many ways PHP natively provides to convert the encoding of strings), but none of them really worked, except for this one:

<?php

$newstring

= mb_convert_encoding($oldstring, 'ISO-8859-15', 'UTF-8');?>

Some PHP CMS's will default setlocale to 'C', this can be a problem.

use the "locale" command to find out a list..

$ locale -a
C
en_AU.utf8
POSIX

<?php
setlocale(LC_CTYPE, 'en_AU.utf8');
$str = iconv('UTF-8', 'ASCII//TRANSLIT', "Côte d'Ivoire");
?>

<?php

function ucs2toutf8($str)

{

for ($i=0;$i<strlen($str);$i+=4)

{

$substring1 = $str[$i].$str[$i+1];

$substring2 = $str[$i+2].$str[$i+3];

if (

$substring1 == "00")

{

$byte1 = "";

$byte2 = $substring2;

}

else

{

$substring = $substring1.$substring2;

$byte1 = dechex(192+(hexdec($substring)/64));

$byte2 = dechex(128+(hexdec($substring)%64));

}

$utf8 .= $byte1.$byte2;

}

return $utf8;

}

echo

strtoupper(ucs2toutf8("06450631062D0020"));
?>



Input:

06450631062D

Output:

D985D8B1D8AD

regards,

Ziyad

Also, you may notice that recent versions of linux (debian, ubuntu, centos, etc) the //TRANSLIT option doesn't work. since most distros doesn't include the intl packages (example: php5-intl and icuxx (where xx is a number) in debian) by default. And this because the intl package conflicts with another package needed for international DNS resolution.

Problem is that configuration is dependent of the sysadmin of the machine where you're hosted, so iconv is pretty much useless by default, depending on what configuration is used by your distro or the machine's admin.

If a character is missing from the $in_charset encoding (eg byte \x81 from CP1252 encoding), then iconv will return an error, whether with //IGNORE or not.

I use this function that does'nt need any extension :

function convert_utf8( $string ) {
if ( strlen(utf8_decode($string)) == strlen($string) ) {
// $string is not UTF-8
return iconv("ISO-8859-1", "UTF-8", $string);
} else {
// already UTF-8
return $string;
}
}

I have not tested it extensively, hope it may help.

i.e.

<?php
iconv('CP1252', 'UTF-16', $text); // with BOM
iconv('CP1252', 'UTF-16BE', $text); // without BOM

When I asked my linux for locale (by locale command) it returns "cs_CZ.UTF-8", so there is maybe correlation between it.

iconv (GNU libc) 2.6.1
glibc 2.3.6

iconv('', 'UTF-8', $str)

test it to convert from windows-1251 (stored in DB) to UTF-8 (which i use for web pages).
BTW i convert each array i fetch from DB with array_walk_recursive...

The problem is not in iconv. But 'Р' in cp1251 is chr(208) and 'Р' in UTF-8 is chr(208).chr(106). chr(106) is one of the space symbol which match '\s' in regex. So, it can be taken by a greedy '+' or '*' operator. In that case, you loose 'Р' in your string.

For example, 'ГР ' (Russian, UTF-8). Function preg_match. Regex is '(.+?)[\s]*'. Then '(.+?)' matches 'Г'.chr(208) and '[\s]*' matches chr(106).' '.

Although, it is not a bug of iconv, but it looks like it very much. That's why I put this comment here.

Here is a repost of my earlier code that now works on more systems. It converts as much as possible and replaces the rest with question marks:

<?php
if (!function_exists('utf8_to_ascii')) {
setlocale(LC_CTYPE, 'en_AU.utf8');
if (@iconv("UTF-8", "ASCII//IGNORE//TRANSLIT", 'é') === false) {
// PHP is probably using the glibc library (*NIX)
function utf8_to_ascii($text) {
return iconv("UTF-8", "ASCII//TRANSLIT", $text);
}
}
else {
// PHP is probably using the libiconv library (Windows)
function utf8_to_ascii($text) {
if (is_string($text)) {
// Includes combinations of characters that present as a single glyph
$text = preg_replace_callback('/\X/u', __FUNCTION__, $text);
}
elseif (is_array($text) && count($text) == 1 && is_string($text[0])) {
// IGNORE characters that can't be TRANSLITerated to ASCII
$text = iconv("UTF-8", "ASCII//IGNORE//TRANSLIT", $text[0]);
// The documentation says that iconv() returns false on failure but it returns ''
if ($text === '' || !is_string($text)) {
$text = '?';
}
elseif (preg_match('/\w/', $text)) { // If the text contains any letters...
$text = preg_replace('/\W+/', '', $text); // ...then remove all non-letters
}
}
else { // $text was not a string
$text = '';
}
return $text;
}
}
}

<?phpfunction utf8toucs2($str)
{
for ($i=0;$i<strlen($str);$i+=2)
{
$substring1 = $str[$i].$str[$i+1];
$substring2 = $str[$i+2].$str[$i+3];

if (

hexdec($substring1) < 127)
$results = "00".$str[$i].$str[$i+1];
else
{
$results = dechex((hexdec($substring1)-192)*64 + (hexdec($substring2)-128));
if ($results < 1000) $results = "0".$results;
$i+=2;
}
$ucs2 .= $results;
}
return $ucs2;
}

echo

strtoupper(utf8toucs2("D985D8B1D8AD"))."\n";
echo strtoupper(utf8toucs2("456725"))."\n";?>

Input:
D985D8B1D8AD
Output:
06450631062D

Input:
456725
Output:
004500670025

<?php
//Logic from http://twiki.org/cgi-bin/view/Codev/InternationalisationUTF8
$str_iso8859_1 = 'foo in ISO 8859-1';
//ISO 8859-1 to UTF-8
$str_utf8 = preg_replace("/([\x80-\xFF])/e",
"chr(0xC0|ord('\\1')>>6).chr(0x80|ord('\\1')&0x3F)",
$str_iso8859_1);
//UTF-8 to ISO 8859-1
$str_iso8859_1 = preg_replace("/([\xC2\xC3])([\x80-\xBF])/e",
"chr(ord('\\1')<<6&0xC0|ord('\\2')&0x3F)",
$str_utf8);
?>

HTH,
R. Rajesh Jeba Anbiah

function cp1251_utf8( $sInput )
{
$sOutput = "";

for ( $i = 0; $i < strlen( $sInput ); $i++ )
{
$iAscii = ord( $sInput[$i] );

if ( $iAscii >= 192 && $iAscii <= 255 )
$sOutput .= "&#".( 1040 + ( $iAscii - 192 ) ).";";
else if ( $iAscii == 168 )
$sOutput .= "&#".( 1025 ).";";
else if ( $iAscii == 184 )
$sOutput .= "&#".( 1105 ).";";
else
$sOutput .= $sInput[$i];
}

return $sOutput;
}

function encoding($string){
if (function_exists('iconv')) {
if (@!iconv('utf-8', 'cp1251', $string)) {
$string = iconv('cp1251', 'utf-8', $string);
}
return $string;
} else {
if (detectUTF8($string)) {
return $string;
} else {
return cp1251_utf8($string);
}
}
}
echo encoding($string);

<?php
function cp1251_utf8( $sInput )
{
$sOutput = "";

for (

$i = 0; $i < strlen( $sInput ); $i++ )
{
$iAscii = ord( $sInput[$i] );

if (

$iAscii >= 192 && $iAscii <= 255 )
$sOutput .= "&#".( 1040 + ( $iAscii - 192 ) ).";";
else if ( $iAscii == 168 )
$sOutput .= "&#".( 1025 ).";";
else if ( $iAscii == 184 )
$sOutput .= "&#".( 1105 ).";";
else
$sOutput .= $sInput[$i];
}

return

$sOutput;
}
?>

<?php
$string = "Winkel γ=200 für 1€"; //"γ"=HTML:&gamma;
$result = iconv('UTF-8', 'CP1252//IGNORE', $string);
echo $result;
?>

Note1
<?php
$string = "Winkel γ=200 für 1€";
$result = iconv('UTF-8', 'CP1252', $string);
echo $result; //"conv(): Detected an illegal character in input string"
?>

Note2 (ANSI is better than decode in ISO 8859-1 (ISO-8859-1==Latin-1)
<?php
$string = "Winkel γ=200 für 1€";
$result = utf8_decode($string);
echo $result; //"Winkel ?=200 für 1?"
?>

Note3 of used languages on Websites:
93.0% = UTF-8;
3.5% = Latin-1;
0.6% = ANSI <----- you shoud use (or utf-8 if your page is in Chinese or has Maths)

In other words, iconv() appears to be intended for use when converting the contents of files - whereas mb_convert_encoding() is intended for use when juggling strings internally, e.g. strings that aren't being read/written to/from files, but exchanged with some other media.

<?php
$value = iconv('Windows-1252', 'UTF-8//TRANSLIT', $value);
?>

CP1250 is Eastern European (not ISO-8859-2)
CP1251 is Cyrillic (not ISO-8859-5)
CP1252 is Western European (not ISO-8859-1)
CP1253 is Greek (not ISO-8859-7)
CP1254 is Turkish (not ISO-8859-9)
CP1255 is Hebrew (not ISO-8859-8)
CP1256 is Arabic (not ISO-8859-6)
CP1257 is Baltic (not ISO-8859-4)

If you know you're getting input from a Windows machine with those encodings, use one of these as a parameter to iconv.

<?phpfunction ucs2html($str) {
$str=trim($str); // if you are reading from file
$len=strlen($str);
$html='';
for($i=0;$i<$len;$i+=2)
$html.='&#'.hexdec(dechex(ord($str[$i+1])).
sprintf("%02s",dechex(ord($str[$i])))).';';
return($html);
}
?>

//IGNORE//TRANSLIT

strictly in that order, but NOT by appending //TRANSLIT//IGNORE, which would lead to //IGNORE being ignored ( :) ).

Anyway, it's hard to understand how one could devise a system of passing options that does not allow to couple both options in a neat manner, and also to understand why the default behaviour should be the less useful and most dangerous one (throwing away most of your data at the first unexpected character). Software design FAIL :-/

To transform "rené" into "rene" we could use the following code snippet:

<?php

setlocale

(LC_CTYPE, 'nl_BE.utf8');$string = 'rené';
$string = iconv('UTF-8', 'ASCII//TRANSLIT', $string);

echo

$string; // outputs rene?>

<?php
setlocale(LC_ALL, 'en_US.UTF8');

function

clearUTF($s)
{
$r = '';
$s1 = iconv('UTF-8', 'ASCII//TRANSLIT', $s);
for ($i = 0; $i < strlen($s1); $i++)
{
$ch1 = $s1[$i];
$ch2 = mb_substr($s, $i, 1);$r .= $ch1=='?'?$ch2:$ch1;
}
return $r;
}

echo

clearUTF('Šíleně žluťoučký Vašek úpěl olol! This will remain untranslated: ᾡᾧῘઍિ૮');
//outputs Silene zlutoucky Vasek upel olol! This will remain untranslated: ᾡᾧῘઍિ૮
?>

Just remember you HAVE TO set locale to some unicode encoding to make iconv handle //TRANSLIT correctly!