Hi,
I have an Immunohistochemistry library I created for one of the MDs here a few years ago. I built an application for him to load his library of images into but now I'd like to move the whole library to a different database. (The idea is he'll be making this available to people across the country and not just within Stanford). I just have no experience in trying to move image/blob data from one database vendor to another. I've tried using the ADS export tool but with no success and I guess that isn't meant to handle that type of data. Is there an easy way to handle this with ADS?
Thanks for any help you can give me.
John
![]() |
3 KB
This gets tricky, depending on your image sizes. ADS doesn't have a GUI to export all the data, but you could write an AquaScript to export it to a file, and then a script to import.
Can you tell me the largest size of one of your images?
If you just have one table with images and the table has a primary key, you could dump the images to a directory with the filename as the primary key value. Then import them using the primary key.
This is adhoc, but there isn't an easy way of doing this.
Huh. So it is kind of tricky at least. For a while there I was feeling that I was missing something obvious. The images are pretty high resolution but not huge. I think maybe the biggest is ~ 20/30 MBs. It is just one table with the image and a smaller thumbnail version in it. It does have a primary key and the whole database is perhaps 2 GB. Not sure what you mean by 'dumping' them to a directory. Using the export tool? Exporting as what? Does the checkbox 'Convert binary to hex' come into play or is that just for viewing within ADS?
Huh. So it is kind of tricky at least. For a while there I was feeling that I was missing something obvious. The images are pretty high resolution but not huge. I think maybe the biggest is ~ 20/30 MBs. It is just one table with the image and a smaller thumbnail version in it. It does have a primary key and the whole database is perhaps 2 GB. Not sure what you mean by 'dumping' them to a directory. Using the export tool? Exporting as what? Does the checkbox 'Convert binary to hex' come into play or is that just for viewing within ADS?
Because the files get up to 30MB, you need to stream them disk in chunks so you don't run into any memory problems. To do this you can write a script that gets a JDBC connection from a AQConnection [ AQServerConnection.getJdbcConnection() ], then execute a SELECT query to the table and then iterate through the resultset to read the BLOB in chunks and stream to disk. We should probably write a utility script for users so they can use it under these circumstances.
But there isn't any easy way of doing this from one database to another.
Because the files get up to 30MB, you need to stream them disk in chunks so you don't run into any memory problems. To do this you can write a script that gets a JDBC connection from a AQConnection [ AQServerConnection.getJdbcConnection() ], then execute a SELECT query to the table and then iterate through the resultset to read the BLOB in chunks and stream to disk. We should probably write a utility script for users so they can use it under these circumstances.
But there isn't any easy way of doing this from one database to another.
In looking a little deeper I think actually the vast majority of these images are JPGs and are under 1 MB. Some are probably TIFFs but even those don't seem to be more than 8 MBs in looking at some of the files I used when developing this. The function datalength(immuno_image) shows over 3000 of the 3092 images between 200000 and 700000. There are about 12 over 1000000 and one over 3000000. On the other hand I also copied the directory path to one column and the file name to another column when I set this up. I've emailed off to the MD I worked with to see if he has kept that structure more or less intact. I presume he has as the directory path also more or less describes the contents. Here's an example where the part beginning with '04-' is the name of the actual file:
path_to_case image_file_name
In looking a little deeper I think actually the vast majority of these images are JPGs and are under 1 MB. Some are probably TIFFs but even those don't seem to be more than 8 MBs in looking at some of the files I used when developing this. The function datalength(immuno_image) shows over 3000 of the 3092 images between 200000 and 700000. There are about 12 over 1000000 and one over 3000000. On the other hand I also copied the directory path to one column and the file name to another column when I set this up. I've emailed off to the MD I worked with to see if he has kept that structure more or less intact. I presume he has as the directory path also more or less describes the contents. Here's an example where the part beginning with '04-' is the name of the actual file:
path_to_case image_file_name
I created a sample AquaScript that will query and dump a BLOB column to disk. The script is attached (ExportBlob.xjs) and it requires the patch below.
Patch: http://dd1.aquafold.com/download/v10.0.0/ads-10.0.2_04-patch.zip
Update Instructions: http://www.aquafold.com/support-update.html
This example works on the SQL Server 2005 sever with the AdventureWorks database. But should work on any database with BLOB columns.
I created a sample AquaScript that will query and dump a BLOB column to disk. The script is attached (ExportBlob.xjs) and it requires the patch below.
Patch: http://dd1.aquafold.com/download/v10.0.0/ads-10.0.2_04-patch.zip
Update Instructions: http://www.aquafold.com/support-update.html
This example works on the SQL Server 2005 sever with the AdventureWorks database. But should work on any database with BLOB columns.
Importing the BLOB is more complex. You can't insert BLOB data into a database with an SQL statement. You need to use a programming API to import the data. You can only import the data into a column of a row that is already in the database. So, you need to INSERT the rows, then import the BLOB into each column.
Importing the BLOB is more complex. You can't insert BLOB data into a database with an SQL statement. You need to use a programming API to import the data. You can only import the data into a column of a row that is already in the database. So, you need to INSERT the rows, then import the BLOB into each column.
Issue #6172 |
Closed |
Completion |
No due date |
No fixed build |
No time estimate |
This gets tricky, depending on your image sizes. ADS doesn't have a GUI to export all the data, but you could write an AquaScript to export it to a file, and then a script to import.
Can you tell me the largest size of one of your images?
If you just have one table with images and the table has a primary key, you could dump the images to a directory with the filename as the primary key value. Then import them using the primary key.
This is adhoc, but there isn't an easy way of doing this.