Easily find issues by searching: #<Issue ID>
Example: #1832
Easily find members by searching in: <username>, <first name> and <last name>.
Example: Search smith, will return results smith and adamsmith
| Dev |
|
| QA |
|
| Priority | Low |
| Complexity | Unknown |
| Component | App - FluidShell - Command |
| Version | 12.0 |
Below is the list of known problems on \csv2excel command. CSV test files are attached FYI.
(1) Maximum number of rows per page:
> cat csv100K.txt | csv2excel > tmp.xls
> cat csv100K.txt | csv2excel -t 2007 > tmp.xlsx
Excel 2003 has a 65536 rows/page limitation; in the first example above, the input file contains 100K+1 rows, the output file will only have 65536 rows. Excel 2007 does not have this limitation as shown in the second example.
(2) Out of memory:
Aspose APIs does everything in memory which will cause java run out of heap space if the input file is too large. The ADS running in my environment is configured as:
-Xmx256M -XX:MaxPermSize=128M
and, the command below (input file has 1 million rows)
> csv2excel < csv1M.txt > tmp.xlsx
generates this error
•csv2excel: •'standard input': •conversion failed: Java heap space
(3) NPE on writing Excel output to fluid editor:
The command below usually will generate an NPE:
> cat csv100K.txt | csv2excel
Note that it works if output is not sent to fluid editor as shown below:
> cat csv100K.txt | csv2excel | cat | tee > tmp-pipe.xls
> cat csv100K.txt | csv2excel > tmp-pipe.xls
Below is the stack trace of NPE, since it is meaningless to write output to editor, I am not going to worry about this at the moment:
<2012-08-28 15:07:04,828> ERROR [AWT-EventQueue-0] <sun.reflect.GeneratedMethodAccessor47>
java.lang.NullPointerException
at org.netbeans.lib.terminalemulator.InterpDumb.processChar(InterpDumb.java:248)
at com.common.terminal.impl.InterpVt100.processChar(InterpVt100.java:439)
at com.common.terminal.impl.InterpXterm.processChar(InterpXterm.java:266)
at org.netbeans.lib.terminalemulator.Term.putc_work(Term.java:5057)
at org.netbeans.lib.terminalemulator.Term.access$5(Term.java:5056)
at org.netbeans.lib.terminalemulator.Term$BaseTermStream.putChars(Term.java:423)
at com.aquafold.datastudio.tools.shell.FluidTerminalLineDiscipline.putChars(FluidTerminalLineDiscipline.java:30)
at org.netbeans.lib.terminalemulator.Term$12.run(Term.java:3493)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:646)
at java.awt.EventQueue.access$000(EventQueue.java:84)
at java.awt.EventQueue$1.run(EventQueue.java:607)
at java.awt.EventQueue$1.run(EventQueue.java:605)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:616)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
|
13 KB
|
|
127 KB
I was not able to attach all of CSV data files I used above, got a message similar to the one shown below:
The file you attempted to attach, csv500K.txt, has a file size of 6.199 MB and exceeds the maximum file size upload limit of 1 MB.
If anyone needs any of the following files for testing, please let me know:
-rw-r--r-- 1 fyang None 1300034 2012-08-27 21:24 csv100K.txt
-rw-r--r-- 1 fyang None 130033 2012-08-27 20:58 csv10K.txt
-rw-r--r-- 1 fyang None 13024 2012-08-27 20:52 csv1K.txt
-rw-r--r-- 1 fyang None 13000035 2012-08-27 21:25 csv1M.txt
-rw-r--r-- 1 fyang None 6500034 2012-08-27 21:50 csv500K.txt
My 2c:
(1) is normal behavior but in Unix-land that command would exit with a non-zero exit code since it wasn't able to process all input.
(2) Not sure... are Aspose API that bad? Generating an Excel file incrementally shouldn't need to have it in RAM. I think for this command Apache POI might do a better job.
(3) Since commands are allowed to generate proper escape codes (eg. we can play with 'echo') it's normal that dumping a binary would mess things up. But I don't think we should throw an exception like that in VT100, perhaps we should just swallow it and reset the terminal state...
My 2c:
(1) is normal behavior but in Unix-land that command would exit with a non-zero exit code since it wasn't able to process all input.
(2) Not sure... are Aspose API that bad? Generating an Excel file incrementally shouldn't need to have it in RAM. I think for this command Apache POI might do a better job.
(3) Since commands are allowed to generate proper escape codes (eg. we can play with 'echo') it's normal that dumping a binary would mess things up. But I don't think we should throw an exception like that in VT100, perhaps we should just swallow it and reset the terminal state...
> (3) Since commands are allowed to generate proper escape codes (eg. we can play with 'echo') it's normal that dumping a binary would mess things up. But I don't think we should throw an exception like that in VT100, perhaps we should just swallow it and reset the terminal state...
I did not make it very clear in the original description. \csv2excel creates a custom OutputStream object if the result is written to fluid terminal. This custom OutputStream will filter out C0/C1 control characters except NL. I ran into problem (3) only when the CSV file exceeds certain size: for examples, command below always works for me in my environment:
prompt> cat csv10K.txt | csv2excel -t 2003
and, this command
prompt> cat csv100K.txt | csv2excel -t 2003
always throws NPE after running for a while.
> (3) Since commands are allowed to generate proper escape codes (eg. we can play with 'echo') it's normal that dumping a binary would mess things up. But I don't think we should throw an exception like that in VT100, perhaps we should just swallow it and reset the terminal state...
I did not make it very clear in the original description. \csv2excel creates a custom OutputStream object if the result is written to fluid terminal. This custom OutputStream will filter out C0/C1 control characters except NL. I ran into problem (3) only when the CSV file exceeds certain size: for examples, command below always works for me in my environment:
prompt> cat csv10K.txt | csv2excel -t 2003
and, this command
prompt> cat csv100K.txt | csv2excel -t 2003
always throws NPE after running for a while.
If you can duplicate this send me the sample file if possible and we'll try to fix it our side.
If you can duplicate this send me the sample file if possible and we'll try to fix it our side.
> If you can duplicate this send me the sample file if possible and we'll try to fix it our side.
I will spend some more time on problem (3) to better understand its behavior before sending you the sample file. I will look into this after I fixed other issues with higher priority.
> If you can duplicate this send me the sample file if possible and we'll try to fix it our side.
I will spend some more time on problem (3) to better understand its behavior before sending you the sample file. I will look into this after I fixed other issues with higher priority.
For the 3 problems described above:
>> (1) Maximum number of rows per page:
This is a known limitation of Excel 2003. Enhancement likely can be made so that multiple spread sheet pages can be generated if the input CSV file contains more than 65536 rows. The Aspose API we currently use does not support this sort of configuration.
>> (2) Out of memory:
A separate issue, #7136, has been created for this.
>> (3) NPE on writing Excel output to fluid editor:
I turns out that the data sent out on the sending end is not equivalent to the data received on the receiving end, and produces some undesired control codes on the receiving end which caused NPE in the Term related class. The root cause of this problem has something to do with writing out negative integer via OutputStream.write(int) on the sending end. Fixed in SVN r29474.
For the 3 problems described above:
>> (1) Maximum number of rows per page:
This is a known limitation of Excel 2003. Enhancement likely can be made so that multiple spread sheet pages can be generated if the input CSV file contains more than 65536 rows. The Aspose API we currently use does not support this sort of configuration.
>> (2) Out of memory:
A separate issue, #7136, has been created for this.
>> (3) NPE on writing Excel output to fluid editor:
I turns out that the data sent out on the sending end is not equivalent to the data received on the receiving end, and produces some undesired control codes on the receiving end which caused NPE in the Term related class. The root cause of this problem has something to do with writing out negative integer via OutputStream.write(int) on the sending end. Fixed in SVN r29474.
Verified using ADStudio 12 RC 23.
(1) Maximum number of rows per page
The limitation for Excel 2003 added as a separate feature request: #7812 - \csv2excel -t 2003 on converting more than 65536 rows/page.
(3) NPE on writing Excel output to fluid editor
Not occurring anymore.
Closed.
Verified using ADStudio 12 RC 23.
(1) Maximum number of rows per page
The limitation for Excel 2003 added as a separate feature request: #7812 - \csv2excel -t 2003 on converting more than 65536 rows/page.
(3) NPE on writing Excel output to fluid editor
Not occurring anymore.
Closed.
Issue #7486 |
| Closed |
| Fixed |
| Resolved |
Completion |
| No due date |
| Fixed Build trunk/29474 |
| No time estimate |
1 issue link |
relates to #7812
Issue #7812\csv2excel -t 2003 on converting more than 65536 rows/page |
I was not able to attach all of CSV data files I used above, got a message similar to the one shown below:
The file you attempted to attach, csv500K.txt, has a file size of 6.199 MB and exceeds the maximum file size upload limit of 1 MB.
If anyone needs any of the following files for testing, please let me know:
-rw-r--r-- 1 fyang None 1300034 2012-08-27 21:24 csv100K.txt
-rw-r--r-- 1 fyang None 130033 2012-08-27 20:58 csv10K.txt
-rw-r--r-- 1 fyang None 13024 2012-08-27 20:52 csv1K.txt
-rw-r--r-- 1 fyang None 13000035 2012-08-27 21:25 csv1M.txt
-rw-r--r-- 1 fyang None 6500034 2012-08-27 21:50 csv500K.txt