[TIMOB-1719] iOS: non-UTF-8 data from external source cannot be saved and reopened accurately

GitHub Issue	n/a
Type	Bug
Priority	High
Status	Closed
Resolution	Won't Fix
Resolution Date	2017-07-26T03:23:04.000+0000
Affected Version/s	Release 3.0.0
Fix Version/s	n/a
Components	iOS
Labels	core
Reporter	Thomas Huelbert
Assignee	Ingo Muschenetz
Created	2011-04-15T03:00:26.000+0000
Updated	2017-07-26T03:23:04.000+0000

Description

This was addressed on Android, over to iOS for parity. Below are the notes Bill wrote in the original bug. From #1690

Use case (helpdesk 38891): app fetches data via XHR. Data is XML encoded in ISO-8859-1. Data is saved to file, to be opened later. When re-opened, Titanium forces it to UTF-8, which screws up the encoding of some of the special characters such as those with umlaute (ö, ä etc) or accents (é etc).

Fail case:

script to repro:

Titanium.UI.setBackgroundColor('#000');
var win = Titanium.UI.createWindow({

title:'Feed encoding test',
backgroundColor:'#fff',
fullscreen: true,
exitOnClose: true,
url: 'main.js'

}); win.open();
main.js:

var win = Ti.UI.currentWindow;
win.title = "Feed encoding via file"
var url = [SEE HELPDESK 38891]

var dir = Titanium.Filesystem.applicationDataDirectory;
var filename = 'feed.xml';
var file = Ti.Filesystem.getFile(dir, filename);

function saveFeed(data) {

if (file.exists()){
    file.deleteFile();
}
file.write(data);

}

function buildTable() {

var blob = file.read();
var string = blob.text;
var xml = Ti.XML.parseString(string);
var rows = [];
try {
    if (xml) {
        var session = xml.getElementsByTagName('session');
        if (session) {
            session = session.item(0);
        } else { 
            alert('session list not fetched');
            return;
        }
        if (session) {
            if (!session.hasChildNodes()) {
                alert('No child nodes');
                return;
            }
            var length = session.childNodes.length;
            for (var i = 0; i < length; i++) {
                var child = session.childNodes.item(i);
                if (child.nodeType == child.ELEMENT_NODE) {
                    rows.push(Ti.UI.createTableViewRow({color: 'black', title: child.getAttribute("driver")}));
                }
            }
            win.add(Ti.UI.createTableView({data: rows}));
        } else {
            alert('"session" not found');
        }
    } else {
        alert('XML did not load');
    }
} catch(ex) {
    alert(ex);
}

}

var xhr = Ti.Network.createHTTPClient();
xhr.onload = function(e) {

try {
    var data = xhr.responseData;
    saveFeed(data);
    buildTable();

} catch(ex) {
    alert(ex);
}

}; xhr.open('GET', url);
xhr.send();

Comments

Thomas Huelbert 2011-04-15

more of Bills notes:
We want to keep things internally in UTF-8, so instead of trying to track the other encoding throughout its lifetime in Titanium code (i.e., in TiBlob, TiFile, etc), we now give the developer a way to transcode the text to utf-8 and then save that:
var utf8Text = Ti.Utils.transcodeString(origText, 'ISO-8859-1', 'UTF-8');
So Ti.Utils.transcodeString(origText, origEncoding, desiredEncoding) is new.
To fix the fail case from the description of this item, in the function saveFeed change file.write(data) to file.write(Ti.Utils.transcodeString(data, 'ISO-8859-1', 'UTF-8')).
Then, because the feed contains <?xml version="1.0" encoding="ISO-8859-1"?>, you need to change the encoding= to read encoding='UTF-8 before giving it to the xml parser. So in buildTable() change ...
var xml = Ti.XML.parseString(string);
... to ... var xml = Ti.XML.parseString(string.replace(/iso-8859-1/i, 'UTF-8'));
Finally, when the data comes back from the provider, get it with responseText instead of responseData
. So replace this... var data = xhr.responseData;
with this ...

var data = xhr.responseText;

also

xhr.responseData with xhr.responseText
Thomas Huelbert 2011-04-15

the current behavior is the test generates an alert on iOS. I posted the working test case (resource files) to Appcelerator/Users/Thomas/bugs/1719 if that helpful

Bill Dawson 2011-04-15

We added Ti.Utils.transcodeString, so Ti iOS should get one too for parity. Here's the java, fyi:

   public String transcodeString(String orig, String inEncoding, String outEncoding)
       {
           try {
               
               Charset charsetOut = Charset.forName(outEncoding);
               Charset charsetIn = Charset.forName(inEncoding);
   
               ByteBuffer bufferIn = ByteBuffer.wrap(orig.getBytes(charsetIn.name()) );
               CharBuffer dataIn = charsetIn.decode(bufferIn);
               bufferIn.clear();
               bufferIn = null;
   
               ByteBuffer bufferOut = charsetOut.encode(dataIn);
               dataIn.clear();
               dataIn = null;
               byte[] dataOut = bufferOut.array();
               bufferOut.clear();
               bufferOut = null;
               
               return new String(dataOut, charsetOut.name());
               
           } catch (UnsupportedEncodingException e) {
               Log.e(LCAT, "Unsupported encoding: " + e.getMessage(), e);
           }
           return null;
       }

Blain Hamon 2011-04-15

We could leave this as an entrance test for new hires. Very clean and clearcut function.
Stephen Tramer 2012-07-25

Despite the lack of a working existing test case (no URL provided in test) it is clear that this bug is still valid in SDK 2.2.0.014b86f because we have NO UTF-16 support. Options are:
Provide an alternative test from original customer

Find a UTF-16 datastream to test against
Tagging for core as it is part of a broader UTF-16 support push.
Lee Morris 2017-07-26

Closing due to inactivity. If this issue still exists, please raise a new ticket.

JSON Source

[TIMOB-1719] iOS: non-UTF-8 data from external source cannot be saved and reopened accurately

Description

Comments

Provide an alternative test from original customer

Find a UTF-16 datastream to test against