Unicodes issue in DI

dataservices job design
1

Hi,
I am facing a problem to load unicode data into my dimension table.

Iam getting a PART_DESCRIPTION like ‘198G MARVEL MILK ¶1.39 2000’ But when I laod it into DIMENSION its loading like
'198G MARVEL MILK ?1.39 2000 '. The unicode symbol is getting changed. I changed the DSConfig file to AL_Engine=_.UTF-8 and changed the target data store code page to utf-8 in the Target data store options. Still not getting that. :hb:

Can any one help me in that?

Thanks in Advance,
Serin.

Serin (BOB member since 2009-04-30)

2

check if your column is capable ofr string multi-byte string ? what is the column datatype VARCHAR or NVARCHAR ?

you need to first find out where the conversion is going wrong in DI or in the database client while inserting the data to the table, what is the database type of the target datastore ?

manoj_d (BOB member since 2009-01-02)

3

The database driver is not set to UTF-8.

https://wiki.sdn.sap.com:443/wiki/display/BOBJ/Multiple+Codepages

Werner Daehn :de: (BOB member since 2004-12-17)

4

Also, if you have any Data Transfer steps, and they’re set to let DI create the table for you, DI may not be making those columns unicode. (I believe that’s changed in the latest version?)

dnewton :us: (BOB member since 2004-01-30)

5

Hi All,
Thanks for the reply. The column length is of VARCHAR(30). Actually my data base is SQL SERVER 2005. Can u tell me how to set the database driver setting in sql server 2005 to UTF-8?

Thanks & Regards,
Serin.

Serin (BOB member since 2009-04-30)

6

I am not using any data transfers. I am using DI 11.7. If I load directly with a qry, Its loading perfectly. But if I load with DI job its not working.

Serin (BOB member since 2009-04-30)

7

The driver issue doesn’t apply for SQL Server. Your column’s datatype in the database table needs to be NVARCHAR(30), not VARCHAR(30), to store Unicode data.

dnewton :us: (BOB member since 2004-01-30)

8

Hi,
Changed it to navrchar(30). Still not working.

Serin (BOB member since 2009-04-30)

9

OK, just to re-cap:

  • You’re sure the DI job engine is set to UTF8 mode.
  • The source Datastore connection is set to UTF8 (need this too, assuming the source contains unicode and the source is nvarchar)
  • The Target datastore connection is set to UTF8
  • The target table column definitions are NVARCHAR vs. VARCHAR
  • You reimported the tables after changing all of the above
  • You then re-loaded the data (changing the datatype doesn’t magically fix the data that’s already loaded)

I don’t know what else it would be…

dnewton :us: (BOB member since 2004-01-30)