#10816: Complex types shown wrong on ADS tree

slavakiev reported 2013-11-18T21:52:46Z · last modified 2017-05-18T17:11:56Z

Complex types shown wrong on ADS tree

Dev	tomconrad
QA	tariqrahiman

Priority	Low
Complexity	Unknown

Component	Hive
Version	19.0

Tested in Aqua Data Studio 14.0.3-14 Build #: 34647 on Ubuntu 12.04( Hive 0.12.0( localhost,embeded mode), Hadoop 0.23.9)

1. If create complexstruct table and expand it in ADS tree then for column arrstr substructure b will not be shown atall and for column strarr column a goes in wrong level

CREATE TABLE complexstruct (

arrstr struct<a:array<array<double>>,b:struct<c:int,d:string>> ,

strarr struct<b:struct<c:int,d:string>,a:array<array<double>>> )

2. If create table complexarray then column v the same as a in table complexstruct above shown different in ADS tree( there is no $elem$ nodes)

CREATE TABLE complexarray (

v array<array<double>> )

See attached screenshot

12 attachments

hive_complextypes.JPG

2013-11-18T21:52:46Z

25 KB
ApacheHive013-ADS15-dev269.png

2014-05-14T06:23:50Z

353 KB
HortonworksHDP211-Hive0.13.png

2014-05-14T06:23:52Z

332 KB
ClouderaCDH5Hive012-ADS15-dev-281.png

2014-05-15T11:21:54Z

256 KB
Complexdata.png

2017-04-25T12:54:35Z

92 KB
ComplexData.png

2017-04-25T21:02:02Z

108 KB
complex3.png

2017-04-27T08:28:55Z

230 KB
complex2.png

2017-04-27T08:28:55Z

125 KB
complex1.png

2017-04-27T08:28:55Z

153 KB
ComplexTypes.png

2017-05-03T15:32:02Z

152 KB
10816_AmazonEMR.png

2017-05-04T09:30:20Z

113 KB
10816_cloudera58.png

2017-05-04T09:30:20Z

113 KB

All Comments (10) Change History

tomconrad 2014-04-30T22:39:44Z

Hi Slava,

Would you retest this using the new Hive methodology in Version 15.. I did try this and found a bug with Hortonworks when displaying the complex columns. I think it is fixed now.. Remember to use the correct JDBC drivers that come with the server for each distribution.

Thanks,

Tom

anilkalshetti 2014-05-14T06:31:45Z

Aqua Data Studio 15.0.0-dev-279

This issue is still observed on below mentioned distributions,
a] [Apache Hadoop2.4, Hive 0.13, Distribution: Apache, Connection Type: Hiveserver1]

b] Hortonworks HDP2.1 Quickstart VM [Hive 0.13] , Distribution: Hortonworks, Connection Type: Hiveserver2]

Please refer screenshots:
ApacheHive013-ADS15....png

HortonworksHDP211-Hive013..png

Compare this screenshots with screenshot "hive_complextypes.JPG"[screenshot by slavakiev].

SachinPrakash 2014-06-22T23:23:47Z

Lets fix this for HDP on HiveServer2

tomconrad 2017-04-17T18:24:37Z

No code changes.

Hi QA Team, Please retest this. I believe this bug was fixed with the changes for #15025. It looks like the schema tree displays correctly for the test case. Thanks, Tom

dutondeumesh 2017-04-25T12:57:14Z

Retested in ADS 19.0.0-dev-27 and this issue is still observed

Cloudera 5.8 - Hive[default ADS drivers 1.1] - screenshot
Observed on other hive distributions also [Tested for Hortonworks and Amazon EMR ]

tomconrad 2017-04-25T21:02:42Z · (edited)

svn# 54063 - Found another issue with the parser when there are multiple arrays in a column.

Hi Umesh,

I fixed another issue with the complex data type column parser. This looks like it fixed the original issue. See attached. Please test this with different combinations of complex data types. Also, test this using Hive, Spark and Impala since their describe information varies.

Thank you, Tom

dutondeumesh 2017-04-27T08:46:11Z · (edited)

Tested different combinations of complex data types in ADS 19.0.0-dev-30

Query 1 : please refer screenshot

CREATE TABLE complexstruct1 (

arrstr struct<a:array<array<double>>,b:struct<c:struct<e:int,f:double>,d:string>> ,

strarr struct<b:struct<c:int,d:struct<g:int,h:array<smallint>>>,a:array<array<double>>> )

Query 2 : please refer screenshot

CREATE TABLE complexstruct2 (

sub array<string>,

dud map<string,int>,

addr struct<a:map<int,string>,b:map<smallint,double>>)

Query 3 : Reopening for this query. Not working as expected. This query works fine in v18 [18.0.9-5]
Please refer screenshot

CREATE TABLE complexstruct3 (

addr struct<a:map<int,string>,b:map<smallint,double>>,

arrstr struct<a:struct<a1:array<int>,a2:array<double>>,b:struct<b1:array<string>,b2:map<int,string>>>)

tomconrad 2017-05-03T15:37:26Z

svn# 54203 - Tweaked the parser some more.

Hi Umesh,

I had to tweak the parser some more to get it to handle your last test case. It looks like it is working now - See attached eye test - Ha. Please give it a try.

Thanks, Tom

dutondeumesh 2017-05-04T09:31:06Z

Query 3 works fine. Verified in ADS 19.0.0-dev-39
Cloudera58-Hive[default ads drivers 1.1] - screenshot
AmazonEMR54-Hive[default ads drivers 1.1] - screenshot

tariqrahiman 2017-05-18T17:11:56Z

Verified in 19.0.0-alpha-8. Complex types are now displayed correctly in Amazon EMR, Cloudera etc

Search Tips

Aqua Data Studio / nhilam

Complex types shown wrong on ADS tree

12 attachments

Issue #10816

Completion

1 issue link

Issue #15025

Search Tips

Aqua Data Studio / nhilam

Title

Complex types shown wrong on ADS tree

12 attachments

Issue #10816

Completion

1 issue link

Issue #15025