Easily find issues by searching: #<Issue ID>
Example: #1832
Easily find members by searching in: <username>, <first name> and <last name>.
Example: Search smith, will return results smith and adamsmith
| Dev |
|
| QA |
|
| Priority | Blocker |
| Complexity | Unknown |
| Component | Open API - IO |
| Version | 18.0 |
In my real project, I have 133 Excel files, each of which is about 3MB. In my actual project, by the time I open and read the 20th or so file, the script hangs and then will typically give a Java out of heap space message or it gives a GC error message. Since the contents of the actual Excel files are proprietary, I created a project to attempt to replicate the problem. In this replicated version, it makes multiple passes through the files before the issue is triggered because the Excel files are smaller.
Especially in the project to replicate the problem, I played with forcing garbage collection, but this does not seem to help. If you comment out the line in the script (see script comment) that does a read row, then the script executes quickly to completion. So, I imagine that even that first row/line read causes all/much of the file to be pulled into memory. But the memory does not seem to be released.
In my reproduction, I had 25 Excel files in the input, but because of upload limitations on the support ticket, I could not include as many. You likely will need to duplicate some of the demo excel files to help reproduce the issue.
|
1 KB
|
|
39 KB
|
|
754 KB
|
|
758 KB
@kin-hong: we've been able to reproduce the issue. Pls work on a fix.
There are two issues in this test scenario:
1. The demo_memory.xjs aquascript is using:
reader.close;
this will not invoke the CoreExcelReader#close method. Use the following instead:
reader.close();
2. The CoreExcelReader#close does not release all the resources, specifically, the row iterator is still kept in memory. Note that CoreExcelReader instances are kept in memory during the entire runtime session via the AquaTools#registerForCleanup call - I don't know the reason for needing to do our own "garbage collection", but mostly like as a precautionary measure to perform final cleanup in case scripts miss calling the close methods.
There are two issues in this test scenario:
1. The demo_memory.xjs aquascript is using:
reader.close;
this will not invoke the CoreExcelReader#close method. Use the following instead:
reader.close();
2. The CoreExcelReader#close does not release all the resources, specifically, the row iterator is still kept in memory. Note that CoreExcelReader instances are kept in memory during the entire runtime session via the AquaTools#registerForCleanup call - I don't know the reason for needing to do our own "garbage collection", but mostly like as a precautionary measure to perform final cleanup in case scripts miss calling the close methods.
Please reopen this issue - not resolved. I changed the script to ensure reader.close() is a function call. I see absolutely NO CHANGE in behavior. There is still an out-of-memory condition at the same time in the flow of processing.
Does this syntactical correction only work in a certain patched version?
I retried with reader.close() on my MAC (version is as attached to ticket) and with my PC, which frankly seemed to respond even less gracefully (unable to cancel script execution and it just hung without throwing an error). My PC is patched and is version:
Please reopen this issue - not resolved. I changed the script to ensure reader.close() is a function call. I see absolutely NO CHANGE in behavior. There is still an out-of-memory condition at the same time in the flow of processing.
Does this syntactical correction only work in a certain patched version?
I retried with reader.close() on my MAC (version is as attached to ticket) and with my PC, which frankly seemed to respond even less gracefully (unable to cancel script execution and it just hung without throwing an error). My PC is patched and is version:
See my comment. Not resolved. Absolutely no behavior difference when using reader.close()
See my comment. Not resolved. Absolutely no behavior difference when using reader.close()
Hi Bob,
We're in the process of releasing 18.0.17-6 patch which solves the issue. This comment was meant to clarify that there were 2 problems encountered:
#1 -- that reader.close() needs to be called. This is a code change you would make.
#2 -- is a bug in ADS that calling reader.close() doesn't release all the associated memory. We've fixed this in 18.0.17-6 patch. The patch will be ready shortly, & I will add a new comment with a link to download the patch.
Hi Bob,
We're in the process of releasing 18.0.17-6 patch which solves the issue. This comment was meant to clarify that there were 2 problems encountered:
#1 -- that reader.close() needs to be called. This is a code change you would make.
#2 -- is a bug in ADS that calling reader.close() doesn't release all the associated memory. We've fixed this in 18.0.17-6 patch. The patch will be ready shortly, & I will add a new comment with a link to download the patch.
The below patch addresses this issue:
The below patch addresses this issue:
I patched both PC and MAC versions. Also made the change to reader.close() in my script. The bug demonstration project now runs correctly to completion. Also, my actual business project successfully handled all 135 Excel files. So this is working for me on both MAC and PC. Thanks for the outstanding responsiveness as always!!
I patched both PC and MAC versions. Also made the change to reader.close() in my script. The bug demonstration project now runs correctly to completion. Also, my actual business project successfully handled all 135 Excel files. So this is working for me on both MAC and PC. Thanks for the outstanding responsiveness as always!!
Issue #15372 |
| Closed |
| Fixed |
| Resolved |
Completion |
| No due date |
| Fixed Build 18.0.17-6, 19.0.0-beta-6 |
| No time estimate |
@kin-hong: we've been able to reproduce the issue. Pls work on a fix.