VARCHAR ARRAY of UTF8 Charset fails [CORE3765]

[firebird-automations]

Submitted by: @jasonwharton

When I connect via UTF8 Charset and open a query and fetch a record and get the array data I get an error:
(From the IBO SQL trace monitor)

/*---
GET SLICE
ARRAY ID (129, 501)
ARRAY DTYPE 37
ARRAY SCALE 0
ARRAY LENGTH 15
ARRAY DIMENSIONS 1
ARRAY RELATION 'JOB'
ARRAY FIELD 'LANGUAGE_REQ'
SLICE DATA LENGTH 310
SLICE DATA:
1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
5: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
6: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
19: 0 0 0 0 ~~~~

ERRCODE = 335544321
----*/
/*---
INTERPRET BUFFER =

ERRCODE = 60
----*/
/*---
INTERPRET BUFFER = arithmetic exception, numeric overflow, or string truncation

ERRCODE = 23
----*/
/*---
INTERPRET BUFFER = string right truncation

ERRCODE = -1
----*/

This is against the Employee.gdb database that comes standard with earlier versions of InterBase. If I change to Charset ASCII then I have no problem getting the data. Somehow something inside the engine is tripping up when it comes to handling UTF8 (MBCS) in array columns.

[firebird-automations]

Modified by: @jasonwharton

description: When I connect via UTF8 Charset I get an error:
(From the IBO SQL trace monitor)

/*---
GET SLICE
ARRAY ID (129, 501)
ARRAY DTYPE 37
ARRAY SCALE 0
ARRAY LENGTH 15
ARRAY DIMENSIONS 1
ARRAY RELATION 'JOB'
ARRAY FIELD 'LANGUAGE_REQ'
SLICE DATA LENGTH 310
SLICE DATA:
1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
5: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
6: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
19: 0 0 0 0 ~~~~

ERRCODE = 335544321
----*/
/*---
INTERPRET BUFFER =

ERRCODE = 60
----*/
/*---
INTERPRET BUFFER = arithmetic exception, numeric overflow, or string truncation

ERRCODE = 23
----*/
/*---
INTERPRET BUFFER = string right truncation

ERRCODE = -1
----*/

This is against the Employee.gdb database that comes standard with earlier versions of InterBase. If I change to Charset ASCII then I have no problem getting the data. Somehow something inside the engine is tripping up when it comes to handling UTF8 (MBCS) in array columns.

=>

When I connect via UTF8 Charset and open a query and fetch a record and get the array data I get an error:
(From the IBO SQL trace monitor)

/*---
GET SLICE
ARRAY ID (129, 501)
ARRAY DTYPE 37
ARRAY SCALE 0
ARRAY LENGTH 15
ARRAY DIMENSIONS 1
ARRAY RELATION 'JOB'
ARRAY FIELD 'LANGUAGE_REQ'
SLICE DATA LENGTH 310
SLICE DATA:
1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
5: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
6: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
19: 0 0 0 0 ~~~~

ERRCODE = 335544321
----*/
/*---
INTERPRET BUFFER =

ERRCODE = 60
----*/
/*---
INTERPRET BUFFER = arithmetic exception, numeric overflow, or string truncation

ERRCODE = 23
----*/
/*---
INTERPRET BUFFER = string right truncation

ERRCODE = -1
----*/

This is against the Employee.gdb database that comes standard with earlier versions of InterBase. If I change to Charset ASCII then I have no problem getting the data. Somehow something inside the engine is tripping up when it comes to handling UTF8 (MBCS) in array columns.

[firebird-automations]

Commented by: @ibprovider

Really critical?

Very strange :)

Hint: forget about official API for work with array and use only isc_put_slice/isc_get_slice.

Good luck.

[firebird-automations]

Commented by: @jasonwharton

Ok, I'm going to try and be polite here.

YES IT IS CRITICAL!

I am using isc_put_slice() and isc_get_slice() and I'm not looking for luck, I'm looking for results.

It is currently not possible, as best as I can tell, to make use of array VARCHAR data with UTF8 Charset. This means all of my customers who are trying to migrate apps to Unicode are stuck with no path forward other than to totally rework their apps to not use array columns anymore. This is not acceptable to them as this is a feature/capability that should work.

[firebird-automations]

Commented by: @asfernandes

My personal opinion is (serious) that we should remove arrays from Firebird sooner rather than later.

[firebird-automations]

Commented by: @AlexPeshkoff

Not good idea to remove a feature that people are actively using.
For example, I've used to consult our sponsors from Luxemburg not long ago how to access them from UDF.

[firebird-automations]

Commented by: Antonio Fortuny (a.fortuny)

Hi all.

Alex is right. I'm the guy who asked him some help to deal with arrays and UDFs To be clear, I'm Antonio from Sita Software of Luxemburg. Thanks to Alexander's answers to my questions I get rid of that problem: It works. I'll be now on the way to deal with UTF8, and arrays will appear again for the varchar. And that's probably why Jason comes in because we make an intensive use of IBOs objects as well.
I'm aware that arrays should be avoided as much as possible from any DB but it's history (10 years) from our point of view and it must work.

Antonio.

[firebird-automations]

Commented by: @ibprovider

> "YES IT IS CRITICAL! "

Hm.

Very strange.

IBProvider reads (now and in 2012 year also) this array-field in connection with UTF8-charset without any problem.

I offer to close this ticket :)

[firebird-automations]

Commented by: @jasonwharton

I protest.

Note: Yes, I am irritated by how I have been treated so dismissively in this issue.

[firebird-automations]

Commented by: @ibprovider

Why "protest" ?

[IBProvider 3.27]

SQL: select LANGUAGE_REQ from JOB where JOB_CODE='SRep' and JOB_GRADE=4 and JOB_COUNTRY='France'

connection charset: UTF8 (irrelevant, because LANGUAGE_REQ has NONE charset)

[Connection through fbclient.dll]

Data from isc_get_slice (75 bytes):
0x04995F64 45 6e 67 6c 69 73 68 0a 20 20 20 20 20 20 20 46 72 65 6e 63 68 0a 20 20 20 English. French.
0x04995F7D 20 20 20 20 20 53 70 61 6e 69 73 68 0a 20 20 20 20 20 20 20 0a 20 20 20 20 Spanish. .
0x04995F96 20 20 20 20 20 20 20 20 20 20 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .

[Direct connection to Firebird through INET-protocol (TCP/IP)]

Data from op_get_slice (75 bytes):
0x02F37E6C 45 6e 67 6c 69 73 68 0a 20 20 20 20 20 20 20 46 72 65 6e 63 68 0a 20 20 20 English. French.
0x02F37E85 20 20 20 20 20 53 70 61 6e 69 73 68 0a 20 20 20 20 20 20 20 0a 20 20 20 20 Spanish. .
0x02F37E9E 20 20 20 20 20 20 20 20 20 20 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .

NOTE: My currect code obtains the VARCHAR-array as CHAR-array (for avoid the CORE1588).

My previous versions, which received the VARCHAR-array as VARCHAR-array (by fact, server returns the CSTRING-array, see CORE1588), also worked without problems.

----
So I propose to close the ticket.

[firebird-automations]

Commented by: @jasonwharton

Simple answer: I protest because the problem has not been fixed.
I do not accept the work around to treat VARCHAR as CHAR.

[firebird-automations]

Commented by: @ibprovider

Read array's blob directly and you get the VARCHAR-array without any problem.

It is not difficult.

[firebird-automations]

Commented by: @jasonwharton

Of course it isn't difficult. I'm not here groveling for tech support.
I am using the API call that does that and encountering a problem.
Have you repeated the problem or do I need to spoon feed you?

[firebird-automations]

Commented by: @ibprovider

Look to CORE2500

Use "isc_blr_text2" and "isc_blr_varying2" (with explicit definition of text charset) instead "isc_blr_text" and "isc_blr_varying".

Good luck.