[TIMOB-25921] Ti.Codec.encodeString does not handle curly quotes

GitHub Issue	n/a
Type	Bug
Priority	n/a
Status	Open
Resolution	Unresolved
Affected Version/s	n/a
Fix Version/s	n/a
Components	n/a
Labels	n/a
Reporter	Daniel Ethier
Assignee	Eric Merriman
Created	2018-03-20T00:13:18.000+0000
Updated	2018-03-31T23:35:19.000+0000

Description

Ti.Codec.encodeString with Ti.Codec.CHARSET_ISO_LATIN_1 does not handle text containing single or double curly quotes. When a string containing curly quotes is encoded, then decoded, an string containing zero bytes is returned. This happens in the most recent 7.1 version of Titanium SDK. I believe it happens in earlier versions as well. The issue only happens on iOS. The same code works fine on Android.

Attachments

File	Date	Size
encode bug.txt	2018-03-22T15:25:20.000+0000	2514

Comments

Mostafizur Rahman 2018-03-20

Hello [~dethier1958], Thanks for sharing with us. Please provide a full sample test code that regenerates the issue. Better to provide a sample app as an attachment here. We will test the issue in our environment. Also, provide the environment details you are testing on. Thanks.

Daniel Ethier 2018-03-20

Upon taking a closer look, I noticed that this is with smart quotes, which iOS substitutes for normal quotes in edit fields. For my app, I will probably just do some sort of replacement to work around this issue. And perhaps this is an Apple issue. Here is some sample code that reproduces the problem:

   //	When a string contains smart quotes (single or double), then encode
   //	and decode will not work. The string pulled out of the buffer by decode
   //	will be blank.
   
   
   var buffer = Ti.createBuffer({ length: 46800 });
   var text   = 'This “has” Quotes�\0';
   
   var str;
   
   
   Ti.API.info('text = ' + text);
   
   
   Ti.Codec.encodeString({
   	charset:		Ti.Codec.CHARSET_ISO_LATIN_1,
   	source:		text,
   	dest:		buffer,
   	destPosition:	15600
   	});
   
   str = Ti.Codec.decodeString({
   	charset:		Ti.Codec.CHARSET_ISO_LATIN_1,
   	source:		buffer,
   	position:		15600,
   	length:		48
   	});
   
   
   Ti.API.info('str = ' + str);

Just create a new mobile project for iPhone and iPad and replace the code in app.js with this. The output on the console will look like this:

   [INFO] :   Launching iOS Simulator
   [INFO] :   Codec Bug/1.0 (7.1.0.df92fbf)
   [INFO] :   text = This “has” Quotes��
   [INFO] :   str = ������������������������������������������������

This happens for smart quotes, not regular quotes, which is interesting. It only seems to happen for the CHARSET_ISO_LATIN_1 encoding.

Sharif AbuDarda 2018-03-21

Hello, I wasn't able to reproduce the issue. I don't see any quotes in the "str =". See my log. Thanks. [INFO] Launching iOS Simulator -- Start simulator log ------------------------------------------------------- [INFO] sampro/1.0 (7.1.0.df92fbf) [INFO] text = This “has” Quotes� [INFO] str =
Daniel Ethier 2018-03-21

That's the bug. The variable text is set to a string containing smart quotes. That string is encoded into a buffer, then that same position in the buffer is read out. The variable str is what comes out of decode. It should be identical to text. But it isn't. It comes out blank if there are smart quotes (single or double) in text. Remove the smart quotes around has in text and it comes out the same.
Daniel Ethier 2018-03-22

I attached a better example that demonstrates what is happening. There are two versions of the example. One encodes a string that has no curly quotes. The bytes are then pulled from those positions of the buffer into an array. Then the buffer is decoded into a string. This version works fine. The string is copied into the buffer. When you pull individual bytes from the buffer, you get the bytes for the characters in the string. The second version of the example has a string containing curly quotes around the work has. After encoding this string into the buffer, the bytes pulled from the buffer are all zeroes. And the string decoded from the buffer is a string of zero bytes. The issue seems to be that encodeString does not copy the string into the buffer when the string contains curly quotes. I hope this helps clarify the issue.
Sharif AbuDarda 2018-03-28

Hello [~dethier1958], Can you try a different charset like CHARSET_UTF8 ? We think the "special" quotes aren't part of the LATIN charset. Let us know the result. Thanks.
Daniel Ethier 2018-03-28

I had tried UTF-8 and that worked. I tried ASCII and that worked, but that causes other problems with various accented characters. I believe you are correct. I did discover that Apple recently made a change in the default text field behavior to automatically convert quotes to curly quotes. I changed my edit fields to use ASCII keyboard and that prevents that, which allows me to avoid the issue. Perhaps you could add more encoding options (Apple provides more constants for various encodings). That would provide other options to get around the issue in the future. See https://developer.apple.com/documentation/foundation/string.encoding
Sharif AbuDarda 2018-03-29

Hello [~dethier1958], Thanks for your suggestion. Our engineers will look into that.

JSON Source