o How did you insert the string? Table Data Editor or ...?

o How much memory was the app using prior to the large string insertion? In the bottom right of ADS, where we show memory consumption, you can double click on that area and a memory monitor window will pop-up. You can  then use that to monitor memory before and after your string insertion. Please attach a screenshot of this as well.

I did a normal insert from query analyzer.I have attached screen shots of memory consumption before and after insertion operation.

Hi,

I'd like to see the following screenshots:

1) A screenshot of query analyzer window just prior to pasting the large string into query analyzer window, including the memory monitor 

2) A screenshot of query analyzer window just after pasting the large string into query analyzer window, including the memory monitor

3) A screenshot of query analyzer window just after executing the insert statement, including the memory monitor.

It's important to be able to pinpoint exactly when the memory spike occurs and by how much after each of these steps.

Hi

I have attached screenshots as requested by you .I have also attached additional screenshot which is taken after around 30 seconds of query execution(Memory-AfterRecordInsert1.png).

Jonathan - pls see if you can reproduce.

Hello,

Can you paste or attach the first few lines of the INSERT statement you are using, perhaps a version that is only a few hundred K in size? 

Also, what database vendor and version is the insertion being run on?

We are using  Server version: 5.5.28 MySQL Community Server (GPL)

i have also attached insert script for you.

We made some memory fixes in RC3. Please test in RC3 and let us know whether this problem still occurs.

Can you please provide us URL from where we need to download this version.

ADS Version : ads-windows-x64-13.0-rc-4

While testing Version for the first time ADS throws Low memory error,then for another round of testing record was inserted and memory was released as well but then it again starts increasing continuously and then no GC took place for a some time.

Hi Fung,

Lets do the following (in trunk)

• disable UndoManager in SQLHistory, Table Data Editor (Preview results) and CommonSchemaDialog
• If a sql statement is > 1 MB in size, do not store it in SQL History
• Investigate why toggling between Query Analyzer & Messages tab spikes memory dramatically

Some interesting facts about SQL History: when I start ADS there are just two 14MB sql statements on the SQL History. Below you can see how much memory is allocated for these two statements:

• at startup, I wait for a few seconds than launch GC several times -> reported used memory is still aprox. 100 MB ; heap dumps report that about 90 MB are allocated for the two 14mb statements loaded by SQL History from disk;
• if I open the SQL History Window, there is a big memory spike (about 360 MB used memory); after launching GC the amount of used memory decreases to 253 MB;
• inside the SQL History window, if i toggle between the SQL History and the SQL Archive tabs, there is another memory spike (to about 403 MB) when the SQL History tab becomes visible again (probably caused by the rendition of the SQL Statement column cells); after GC the used memory decreases to 314 MB.

Repeating the last step several times causes the used memory to increase with every step, and some of the newly allocated memory cannot be released. I can easily go out of free memory by repeating this step. Heap dumps indicates that the UndoManager is responsible for this leakage, too.

Also, note how much memory is allocated for the two 14 MB statements. Maybe just a truncated version of the SQL Statement should be displayed on the SQL Statement cell of the SQL History Window. Just the 1MB storage threshold itself, though useful, might not help as expected (due to the expansion explained above).

I've attached the 14 MB insert statement (obtained by duplicating the content from the longtext.txt file) used in the above scenario, to make testing easier.

I attempted this on our MySQL 5.5 database... received the error below

Packet for query is too large (14406739 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable. 

I did notice that after attempting to run this query twice, when I clicked on SQL history, the memory usage maxed out.

I just attached an additional screenshot of a single execution attempt with a following view of SQL history

I attempted this on our MySQL 5.5 database... received the error below

Packet for query is too large (14406739 > 1048576). You can change this value on the server by setting the max_allowed_packet' variable

My MySQL Server instance is configured (probably by default, I don't remember if I had changed this setting) with max_allowed_packet = 16M , that's why I don't get that error. You can change this setting by altering the /etc/mysql/my.cnf , but is not relevant for this issue if the statement is successfully run on the MySQL server or not. It is rather a problem of the ADS GUI components (Query Analyzer and SQL History windows etc).

SVN trunk/r31259
(1) SQL History Dialog: if a sql statement is > 1 MB in size, do not store it in SQL History
(2) SQL History Dialog: Preivew Pane: disabled Undo function.
(3) Table Data Editor: Preview SQL tab: Selected Cell Content Pane: disabled Undo function.
(4) SchemaCommonDialog: Preview tab: Disabled Undo function.
(5) SchemaCommonTab: Preview tab: Disabled Undo function.

(1) and (2) should help reducing memory consumption quite a bit because SQL History Dialog is created as a static instance; not sure about (3)-(5) would help or not, maybe.


> Investigate why toggling between Query Analyzer & Messages tab spikes memory dramatically

Focus event triggers some action. I did a heap dump before and after the event is triggered, and one additional object is reported after the event is triggered, see screenshot FY/MAT-Switch-Focus for details. Since this object will be garbage-collected, I am not going to worry it for the time being.

Here is another test case (Test-Case-2) that would produce a memory leak:
(1) Launch ADS with no tabs opened; if there are restored tabs show up, close all of them and restart ADS.
(2) Open a Query Analyzer from the database where the N-MB insert statement can be executed and click the GC icon.
(3) Copy-and-paste the N-MB insert statement to the newly opened Query Analyzer tab and click the GC icon.
(4) Execute the N-MB insert statement and click the GC icon.
    In my test case, only the Grid tab and the Messages tab are enabled in the Query Analyzer.
(5) Close the Query Analyzer tab, wait for a few seconds and click the GC icon.
(6) Compare the difference of value of "Used Memory:" between step (2) and step (5).
(7) Repeat step (2) through step (6) a few times.

After step (6), you will find the "Used memory:" goes up ~ X-MB each time.

In my test case, the size of the row generated by the insert statement is 10-MB, and at step (6), the "Used memory:" approximately goes up 60-MB each time, and these memory is never freed up.  Memory Analyzer Tool shows these leaks have something to do with UndoManager.

SVN r31289 - Query Analyzer panel: clear editor's UndoManager upon closing tab.
For the 10-MB-row test case described above, this will free up 20MB memory each run.

The remaining 40-MB that cannot be freed up seems to be org.netbeans.modules.versioning.diff.DiffSidebar related, see attached FY/MAT-DiffSidebar screenshot for more info. Can you please take a look?

The tree nodes' arrow icon direction indicatees that the DiffSidebar keeps reference to the undo object (the 20 MB heap space), but as you said, I guess this is fixed in SVN r31289. Something else keeps the reference to the 40 MB heap space listed on the bottom of the tree on your screenshot, but I'm unable to figure it out yet.

Please send an updated build ( SVN r31289 included ) and I'll investigate it further.

SVN r31311
This check-in should free up the remaining 40 MB heap space described in the test case Test-Case-2.  The allocated memory won't be freed up right away after clicking the GC icon, it takes a few minutes to be garbage-collected.

The root cause of this problem is: In Test-Case-2, two editor panes are created, one for the Grid tab and one for the Messages tab. The editor pane associated with the Messages tab is not closed, i.e. not closed by PaneFactory.closedPane(), and holds a bunch of references to the allocated memory through its event listeners. r31311 fixes the problem by passing that editor pane to PaneFactory.closedPane() upon closing the query analyzer tab.  The Grid tab is empty so it won't cause problem in Test-Case-2.

Quite a few editor panes are created inside the Query Analyzer tab, none of them are closed by PaneFactory.closePane() and will cause memory leak as well. For example, if the Pivot Grid is turned on, execute a SELECT statement against the table where longtext is stored likely will produce a memory leak.  I need to turn on all of tabs inside the Query Analyzer panel and execute both INSERT and SELECT statements to make sure memory is freed up before I can close this issue.

And, other ADS tools likely will have this memory-leak problem, too.

Test-Case-3: Inside Test-Case-2, if you jump after step (2) directly to step (5), there seems to be another memory leak (quite bigger than the one from Test-Case-2). Tested with the ADS-14.0.0-dev10 build, which doesn't include the SVN revision r31311, but as the Grid and Messages tabs are not instantiated on Test-Case-3, I guess this memory leak is still there (after the r31311 fix).

The allocated memory won't be freed up right away after clicking the GC icon, it takes a few minutes to be garbage-collected.

This delay seems weird and similar to the one from #8212. Either there are some soft-references (unlike weak references, the soft ones are kept for caching purposes and are released only if there is no free heap space left), either there are some timers (on ADS classes etc) that determine these references to be eventually released (when these timers expires). Maybe this strange behaviour (i.e. the delayed release of the last references to particular objects, thus allowing the GC to free the related heap space) worth further investigation to identify what causes it.

SVN r31336 - fix for most of reproducible memory leaks.
Tested r31336 by executing 'INSERT a 10 MB row' or 'SELECT a 10 MB' with the following toolbar buttons checked:
   Show Text, Show Text History, Show DBMS_OUTPUT, Show Grid, Show Pivot Grid, Show Form, Show Execution Plan, Show Client Statistics
Most of leaks are fixed; there are still a few cases that do not happen all the time and I have not figured them out yet.

In most of cases, memory won't be freed up immediately; sometimes takes a few minutes to be garbage-collected, sometimes takes 10+ minutes. I have attached a screenshot (FY/MAT-Soft-Reference) of 'Path to GC Roots' generated from MAT for a typical soft reference. It seems to have something to do with image cache.

> Test-Case-3: Inside Test-Case-2, if you jump after step (2) directly to step (5), there seems to be another memory leak (quite bigger than the one from Test-Case-2).

My test usually shows the difference between (2) and (5) is ~7 MB; per MAT, it seems to be java2d related, I have attached a screenshot (FY/MAT-Java2D-Init) FYI, the screenshot contains 2 MAT reports, one for step (2) and one for step (5).

SVN r31397 - Query Panel: calls ResultsetTableView.clearAndCloseView() to free up memory used by instances of ResultsetTableView.