[TIMOB-1690] Android: non-UTF-8 data from external source cannot be saved and reopened accurately

GitHub Issue	n/a
Type	Bug
Priority	Trivial
Status	Closed
Resolution	Fixed
Resolution Date	2011-04-17T01:56:53.000+0000
Affected Version/s	n/a
Fix Version/s	Release 1.5.0
Components	Android
Labels	android, defect
Reporter	Bill Dawson
Assignee	Bill Dawson
Created	2011-04-15T02:59:27.000+0000
Updated	2011-04-17T01:56:53.000+0000

Description

Use case (helpdesk 38891): app fetches data via XHR. Data is XML encoded in ISO-8859-1. Data is saved to file, to be opened later. When re-opened, Titanium forces it to UTF-8, which screws up the encoding of some of the special characters such as those with umlaute (ö, ä etc) or accents (é etc).

Fail case:

app.js


Titanium.UI.setBackgroundColor('#000');
var win = Titanium.UI.createWindow({  
    title:'Feed encoding test',
    backgroundColor:'#fff',
    fullscreen: true,
    exitOnClose: true,
    url: 'main.js'
});
win.open();

main.js:

var win = Ti.UI.currentWindow;
win.title = "Feed encoding via file"
var url = [SEE HELPDESK 38891]

var dir = Titanium.Filesystem.applicationDataDirectory;
var filename = 'feed.xml';
var file = Ti.Filesystem.getFile(dir, filename);


function saveFeed(data) {
    if (file.exists()){
        file.deleteFile();
    }
    file.write(data);
}

function buildTable() {
    var blob = file.read();
    var string = blob.text;
    var xml = Ti.XML.parseString(string);
    var rows = [];
    try {
        if (xml) {
            var session = xml.getElementsByTagName('session');
            if (session) {
                session = session.item(0);
            } else { 
                alert('session list not fetched');
                return;
            }
            if (session) {
                if (!session.hasChildNodes()) {
                    alert('No child nodes');
                    return;
                }
                var length = session.childNodes.length;
                for (var i = 0; i < length; i++) {
                    var child = session.childNodes.item(i);
                    if (child.nodeType == child.ELEMENT_NODE) {
                        rows.push(Ti.UI.createTableViewRow({color: 'black', title: child.getAttribute("driver")}));
                    }
                }
                win.add(Ti.UI.createTableView({data: rows}));
            } else {
                alert('"session" not found');
            }
        } else {
            alert('XML did not load');
        }
    } catch(ex) {
        alert(ex);
    }
    
}

var xhr = Ti.Network.createHTTPClient();
xhr.onload = function(e) {
    try {
        var data = xhr.responseData;
        saveFeed(data);
        buildTable();
        
    } catch(ex) {
        alert(ex);
    }
};
xhr.open('GET', url);
xhr.send();

In the tableview that results from running that code, you will notice some of the names have characters that are messed up.

Comments

Jeff Haynie 2011-04-15

(from [32cf760dca3156a8478fa47e519070c900a32fab]) [#1690] Put a transcodeString() utility function into UtilsModule. http://github.com/appcelerator/titanium_mobile/commit/32cf760dca3156a8478fa47e519070c900a32fab"> http://github.com/appcelerator/titanium_mobile/commit/32cf760dca315...
Bill Dawson 2011-04-15
We want to keep things internally in UTF-8, so instead of trying to track the other encoding throughout its lifetime in Titanium code (i.e., in TiBlob, TiFile, etc), we now give the developer a way to transcode the text to utf-8 and then save that:
```
   var utf8Text = Ti.Utils.transcodeString(origText, 'ISO-8859-1', 'UTF-8');
   
```
So Ti.Utils.transcodeString(origText, origEncoding, desiredEncoding) is new.

To fix the fail case from the description of this item, in the function saveFeed change file.write(data) to file.write(Ti.Utils.transcodeString(data, 'ISO-8859-1', 'UTF-8')).

Then, because the feed contains <?xml version="1.0" encoding="ISO-8859-1"?>, you need to change the encoding= to read encoding='UTF-8 before giving it to the xml parser. So in buildTable() change ...
```
   var xml = Ti.XML.parseString(string);
   
```
... to ...
```
   var xml = Ti.XML.parseString(string.replace(/iso-8859-1/i, 'UTF-8'));
   
```
Finally, when the data comes back from the provider, get it with responseText instead of responseData
. So replace this...
```
   var data = xhr.responseData;
   
```
with this ...
```
   
   var data = xhr.responseText;
   
```
Bill Dawson 2011-04-15

I screwed up in the previous comment. I forgot to mention you need to also replace xhr.responseData with xhr.responseText. I edited the comment above to reflect this now.
Thomas Huelbert 2011-04-15

w00t - that did it, thanks Much Bill. Confirmed on nexus 1 running 2.2 and simulator.

JSON Source