× Heads up!

Aqua Data Studio / nhilam

Follow
IDE for Relational Databases
×
JasonK reported 2015-06-16T14:10:23Z  · nhilam last modified 2018-03-17T05:54:42Z

Support huge data sets in Visual Analytics


customer request
Dev
QA
Priority Low
Complexity Unknown
Component Visual Analytics
Version Future

I know this is likely a very large change but just wanted to raise it up as a future request...

I'm trying to do some analytics on some larger data sets-- 2-5 million rows of data, 10-15 columns and continue to run into issues. I've currently bumped Data Studio up to 10GB of memory and am wondering if there are things that can be done to decrease memory usage. I opened two other issues, https://www.aquaclusters.com/app/home/project/public/aquadatastudio/issue/13368 and https://www.aquaclusters.com/app/home/project/public/aquadatastudio/issue/13369 that may help but am wondering if these are "bigger" improvements that can be done.

Obviously having zero detail on how it's implemented currently but just throwing out ideas that may or may not help ;)

  • Disk backed memory cache for the workbook data
  • Database backed memory cache (possibility with the ability to define the database setup such as a local or even remote MySql database)

Note, the database backed memory cache would be very useful in other regards too-- such as being able to create a remote MySql database with a VA data extract in it that could then be shared by others. In that case it would likely make the most sense to have the remote database backed by a local disk cache to deal with high latency connections). To go pie-in-the-sky, you could even extend that to storing the workbooks themselves in a database to share :)

Issue #13370

New
 
 
Completion
No due date
No fixed build
No time estimate

About AquaClusters Privacy Policy Support Version - 19.0.2-4 AquaFold, Inc Copyright © 2007-2017