Broken Build:
17.0.0-rc-49, before 49, only last result was returned.
Observations:
Number of records are around 1550.
This issue is occuring because ADS is trying to render all records, before rc 49, only last result was returned.
@ranarohitp: the behavior that MongoJS is printing each iteration of cursor.hasNext() is acceptable. If you execute cursor.hasNext() outside of a while loop in MongoDB's own shell, you'll see that it automatically prints out the results as well.
@kin-hong: I've done further analysis on this issue & the 'hanging' behavior thrashing by the JVM once its reached it max memory available but still needs more memory to display the results. By turning off form view & other views, more memory is available to the system. I've also verified that once the operation is finished & I then execute a different query, the memory is released.
We could add an AFMemoryException test.
Hi Jenny,
I tried to debug this issue.
My observation is :
1]
MongoRuntime.call(new MongoInterpretedScriptAction(scope, SOURCENAME,
script))
Above call returns only one result which is expected as final output.
But JDBC code uses MongoScope.getPoppedResults() method to get the results.
2] In MongoStatement.executeEvalJSCommand, MongoScope.getPoppedResults() returns an ArrayList having 1509 results. This ArrayList contains the first element which is instance of DBQuery and remaining all are instance of the NativeObject.
convertJSScriptableToJDBCResultSet method returns the Resultset (Instance of JSONResultSet) which contains empty List ( iterableList of JSONResultSet). Take a look at following method.
public JSONResultSet(ScriptableObject iterableNativeObject,
SelectCmd.FlatteningTypes flatteningType,
boolean predictDataType, boolean bigEndianUUID) {
this.flatteningType = flatteningType;
this.predictDataType = predictDataType;
iterableList = new ArrayList<>();
--- Since the following condition returns false, Because of this ArrayList remains empty -----------
while (!Boolean.FALSE.equals(ScriptableObject.callMethod(iterableNativeObject, "hasNext", Context.emptyArgs))) {
Object result = ScriptableObject.callMethod(iterableNativeObject, "next", Context.emptyArgs);
if (result instanceof NativeObject) {
iterableList.add(MongoUtils.convertJSToDBObject((NativeObject) result, bigEndianUUID));
}
}
}
I tried to execute the code by removing the while loop in above code. I got following error: java.sql.SQLException: Error: error hasNext: false (mongodb32/query.js#283
3] Now, after iterating all the results MongoStatement has 1509 resultsets.
Since variable error is null in MongoStatement.execute method, resultSetIterator remains null, this causing getMoreResults method to return the false.
I have added the condition to initailize resultSetIterator, But now it will return all the results (in my case 1059) . I have checked in code to the branch. Please take a look and guide me.
I am not sure about the above solution. Please correct me if its wrong.
Thanks and regards
Vasudev.
Hi Jenny,
I tried to debug this issue.
My observation is :
1]
MongoRuntime.call(new MongoInterpretedScriptAction(scope, SOURCENAME,
script))
Above call returns only one result which is expected as final output.
But JDBC code uses MongoScope.getPoppedResults() method to get the results.
2] In MongoStatement.executeEvalJSCommand, MongoScope.getPoppedResults() returns an ArrayList having 1509 results. This ArrayList contains the first element which is instance of DBQuery and remaining all are instance of the NativeObject.
convertJSScriptableToJDBCResultSet method returns the Resultset (Instance of JSONResultSet) which contains empty List ( iterableList of JSONResultSet). Take a look at following method.
public JSONResultSet(ScriptableObject iterableNativeObject,
SelectCmd.FlatteningTypes flatteningType,
boolean predictDataType, boolean bigEndianUUID) {
this.flatteningType = flatteningType;
this.predictDataType = predictDataType;
iterableList = new ArrayList<>();
--- Since the following condition returns false, Because of this ArrayList remains empty -----------
while (!Boolean.FALSE.equals(ScriptableObject.callMethod(iterableNativeObject, "hasNext", Context.emptyArgs))) {
Object result = ScriptableObject.callMethod(iterableNativeObject, "next", Context.emptyArgs);
if (result instanceof NativeObject) {
iterableList.add(MongoUtils.convertJSToDBObject((NativeObject) result, bigEndianUUID));
}
}
}
I tried to execute the code by removing the while loop in above code. I got following error: java.sql.SQLException: Error: error hasNext: false (mongodb32/query.js#283
3] Now, after iterating all the results MongoStatement has 1509 resultsets.
Since variable error is null in MongoStatement.execute method, resultSetIterator remains null, this causing getMoreResults method to return the false.
I have added the condition to initailize resultSetIterator, But now it will return all the results (in my case 1059) . I have checked in code to the branch. Please take a look and guide me.
I am not sure about the above solution. Please correct me if its wrong.
Thanks and regards
Vasudev.
I have checked in code to the branch.
I can't find your check-in in SVN. What is the SVN revision number?
I have checked in code to the branch.
I can't find your check-in in SVN. What is the SVN revision number?
I forgot to commit the changes.
Now, I have checked in code to branch SVN#6733.
I forgot to commit the changes.
Now, I have checked in code to branch SVN#6733.
Code review comments for SVN #6733:
The code changes are unnecessary. Take a look at the MongoStatement.getResultSet method; it sets resultSetIterator. So, setting it in MongoStatement.execute is unnecessary. Please revert the code changes.
Code review comments for SVN #6733:
The code changes are unnecessary. Take a look at the MongoStatement.getResultSet method; it sets resultSetIterator. So, setting it in MongoStatement.execute is unnecessary. Please revert the code changes.
It is correct that 1,509 result sets are returned. It takes time to process all 1,509 result sets and display them in different views. It appears like a hang, but eventually the results are displayed. When I try this scenario, it takes about 2 minutes 30 seconds before 1,509 result sets are displayed in the Text, Tree, and Grid views.
When executing MongoJS commands, JSONResultSet objects are returned. JSONResultSet objects are used when displaying Text, Text History, and Tree views. If any of the other views are needed, the JSONResultSet objects are converted to MongoResultSet objects. This conversion is the bottleneck. If I turn off Grid view, the result sets are displayed in the Text and Tree views within a few seconds.
We are not going to tackle this issue for now as it is an edge case scenario.
Vasudev, after you revert your code changes, re-assign this issue to me.
It is correct that 1,509 result sets are returned. It takes time to process all 1,509 result sets and display them in different views. It appears like a hang, but eventually the results are displayed. When I try this scenario, it takes about 2 minutes 30 seconds before 1,509 result sets are displayed in the Text, Tree, and Grid views.
When executing MongoJS commands, JSONResultSet objects are returned. JSONResultSet objects are used when displaying Text, Text History, and Tree views. If any of the other views are needed, the JSONResultSet objects are converted to MongoResultSet objects. This conversion is the bottleneck. If I turn off Grid view, the result sets are displayed in the Text and Tree views within a few seconds.
We are not going to tackle this issue for now as it is an edge case scenario.
Vasudev, after you revert your code changes, re-assign this issue to me.
Issue #13936 |
Closed |
Completion |
No due date |
No fixed build |
No time estimate |
@ranarohitp: the behavior that MongoJS is printing each iteration of cursor.hasNext() is acceptable. If you execute cursor.hasNext() outside of a while loop in MongoDB's own shell, you'll see that it automatically prints out the results as well.
@kin-hong: I've done further analysis on this issue & the 'hanging' behavior thrashing by the JVM once its reached it max memory available but still needs more memory to display the results. By turning off form view & other views, more memory is available to the system. I've also verified that once the operation is finished & I then execute a different query, the memory is released.
We could add an AFMemoryException test.