#13451: Fix usage of DataCache to use final value instead of intermediate value

SachinPrakash reported 2015-07-09T19:36:26Z · last modified 2015-12-03T20:07:21Z

Fix usage of DataCache to use final value instead of intermediate value

Dev	kin-hong
QA	Shavi57

Priority	Major
Complexity	Unknown

Component	App - General
Version	17.0

Related to investigation of ads#13044

2 attachments

13451_observations.xlsx

2015-09-22T13:08:15Z

6 KB
observations.xlsx

2015-09-23T09:00:08Z

9 KB

All Comments (7) Change History

kin-hong 2015-07-10T17:49:41Z · (edited)

The DataCache was first introduced in SVN r9834 as the fix to ads#2971. The following two lines of code:

value = dataCache.add(value);

objectValue = objectValue(theColumn, value, _settings, conn,

maxBlobBlocks, maxClobLines);

were added. Notice that when execution past the 2nd line and if there is data type transformation by the objectValue() method (i.e. objectValue().getValue() != value, ), the value inside the objectValue is never cached.

We noticed this in ads#13044 where the query result contains data from BIGINT columns. Although the values are almost all zeros, they are not being cached. The reason is that intermediate BIGINT values are transformed to java.math.BigDecimal, which miss the data cache pass.

The proposed fix is to replace the above two lines with the following:

objectValue = objectValue(theColumn, value, _settings, conn,

maxBlobBlocks, maxClobLines);

objectValue.setValue(dataCache.add(objectValue.getValue()));

This ensures that the final, including transformed, values use the data cache.

In the DataCache.add() method we also do not cache Strings with length > 512 to limit CPU usage.

kin-hong 2015-07-10T14:23:41Z

@QA, to verify this fix, construct a MySQL database table with 10 BIGINT columns and 10000 rows all filled with zeros (or you can use this database and query). Observe the memory counter before and after the fix.

Shavi57 2015-09-22T13:08:00Z

I have tested this issue using this database and query. I have attached the excel file which contains memory counter before and after the fix. Please let us know whether the performed tests are correct and please analyze the observations and let us know whether the readings are as per expected and should we mark this issue as verified?

kin-hong 2015-09-22T22:15:00Z

@Shavi57 - I looked through the attached observations spreadsheet, but did not spot the memory usage improvements I was expecting. Here are a few notes on what I think is unexpected:

The fix should be in the ADS >= 16.0.5-18 and >= 17.0.0-dev-75 builds. However, I don't see a significant used memory difference between the data for 16.0.5-17 and 16.0.6-1. For your reference, for 100k rows of data, on my development machine, the used memory was about 420+MB before the fix and 160+MB after the fix
The used memory for 50k rows of data and 100k rows of data are not significantly different. In most cases, the used memory for 50k rows is even more than that for 100k rows

Could you please run the test again, verify that:

ADS < 16.0.5-18 builds is used
the memory allocated to ADS is under 1 GB (I use -Xmx678m)
the database/data and query is correct, or "construct a MySQL database table with 10 BIGINT columns and 10000 rows all filled with zeros"
ADS is restarted after each test
the Trash Can icon is clicked a few times before reporting the memory data

Note, you do not need to perform multiple query executions, just the Initial Usage should reveal the facts.

Shavi57 2015-09-23T08:59:50Z

I have attached the excel file(observations.xlsx) which contains memory counter before and after the fix.Please let us know whether the performed tests are correct and please analyze the observations and let us know whether the readings are as per expected and should we mark this issue as verified?

Prerequisite :

- Assigned ADS memory: -Xmx756M

- Dataset used: this database and query.

Steps performed for 50K rows :

1. ADS is restarted.

2. Wrote query "SELECT * FROM events_waits_summary_by_account_by_event_name E, events_statements_summary_by_host_by_event_name F limit 50000" in Query Analyzer

3. Clicked on Garbage collector icon for several times

4. Note the readings for "Before query execution"

5. Executed the query

6. Clicked on Garbage collector icon for several times

7. Note the readings for "After query execution"

Steps performed for 100K rows :

1. ADS is restarted.

2. Wrote query "SELECT * FROM events_waits_summary_by_account_by_event_name E, events_statements_summary_by_host_by_event_name F limit 100000" in Query Analyzer

3. Clicked on Garbage collector icon for several times

4. Note the readings for "Before query execution"

5. Executed the query

6. Clicked on Garbage collector icon for several times

7. Note the readings for "After query execution"

kin-hong 2015-09-23T17:01:46Z

@Shavi57 - The latest test data looks good. I have extracted the pertinent info below:

Stages of Query Execution	ADS builds
	16.0.5-17 (before fix)		16.0.6-3 (after fix)		17.0.0-beta-22 (after fix)
	50k Rows	100k Rows	50k Rows	100k Rows	50k Rows	100k Rows
	Used Memory	Used Memory	Used Memory	Used Memory	Used Memory	Used Memory
Before query execution	28	29	65	65	70	70
After query execution	293	469	209	342	215	348
Est. Memory Used by ResultSet	265	440	144	277	145	278

It looks like after the fix, the memory requirements for this type of data set reduced by about 45% (for 50k rows) and 37% (for 100k rows).

We can mark this issue as verified.

Shavi57 2015-09-24T05:07:52Z

As per comment done by @kin-hong isssue have been marked as Verified in ADS 17.0.0-beta-22.

Search Tips

Aqua Data Studio / nhilam

Fix usage of DataCache to use final value instead of intermediate value

2 attachments

Issue #13451

Completion

1 issue link

Issue #13044

Search Tips

Aqua Data Studio / nhilam

Title

Fix usage of DataCache to use final value instead of intermediate value

2 attachments

Issue #13451

Completion

1 issue link

Issue #13044