0 votes
asked in MicroStream for Java by (1k points)
edited by

Hi.

I tried the following tutorial:

https://manual.docs.microstream.one/data-store/import-export

 

My code so far generates a csv file (but it's empty). And due to a lack of documentation I had to do some try and error, because there were several things unclear to me. In your documentation example, there is a variable called "typeFile" with a value of "type1.bin". But as far as I tried this is NOT the datastorage, but the EXPORT target. So this file will be written. When I am setting this to the microstream storage path, it destroys the datastorage, which can't be loaded anymore. So it has to be the output file. In this case it should be called .csv, instead of .bin. But then again, there must be another variable to point to the data to export from. And "csvDir" would be a bad name and also it doesnt work for me. As I said, my (tiny example) datastore with some persisted date objetcts and string objects won't get exported to csv, the file is empty. Do you have an idea why?

Thanks in advance

Regards, Frederik

 

public void exportToCSV(Path exportPath, Path filePath) {

    Path targetPath = Path.of("/home/fred/microstream/microstream-export/export.csv");
    Path exportPathTMP = Path.of("/home/fred/microstream/microstream-datastore/channel_1/channel_1_1.dat");

    logger.info("Starting csv export to path " + (exportPath != null ? exportPath.toString() :
        "(null)") + " and filename " + (filePath != null ? filePath.toString() : "(null)"));

    StorageDataConverterTypeBinaryToCsv converter = new StorageDataConverterTypeBinaryToCsv.UTF8(
        StorageDataConverterCsvConfiguration.defaultConfiguration(),
        new StorageEntityTypeConversionFileProvider.Default(exportPathTMP, "bin"), storageManager.typeDictionary(),
        null, 4096, 4096);

    StorageLockedFile storageFile = StorageLockedFile.openLockedFile(targetPath);

    try {
        converter.convertDataFile(storageFile);
    } finally {
        storageFile.close();
    }

    logger.info("CSV export succeded to " + exportPath.toString() + filePath.toString());

}

 

1 Answer

+1 vote
answered by (3.5k points)
selected by
 
Best answer
Hello,

The export to CSV is a two-stage process.
In the first step the content of the storage must be exported in binary format using “StorageConnection.exportTypes(StorageEntityTypeExportFileProvider)”. In the second step those exported binary files can be converted to CSV file using the “StorageDataConverterTypeBinaryToCsv” converter.

An export function may look like:

public void exportToCSV(Path exportPathBin, Path targetPath)
{       
    //Export storage data to binary data files (one file per data type)
    //into directory "exportPathTemp"
    StorageEntityTypeExportStatistics exportResult = storage.exportTypes(
            new StorageEntityTypeExportFileProvider.Default(exportPathBin, "bin"));
           
    //get all exported binary files
    List<String> binaryFiles = new ArrayList<>();
    exportResult.typeStatistics().values().forEach( v -> binaryFiles.add(v.file().identifier()));
           
    //create the CSV converter
    StorageDataConverterTypeBinaryToCsv converter =
            new StorageDataConverterTypeBinaryToCsv.UTF8(
                StorageDataConverterCsvConfiguration.defaultConfiguration(),
                new StorageEntityTypeConversionFileProvider.Default(targetPath, "csv"),
                storage.typeDictionary(),
                null, // no type name mapping
                4096, // read buffer size
                4096  // write buffer size
        );
   
    //convert all binary files to CSV files
    for (String file : binaryFiles)
    {
        final StorageLockedFile storageFile = StorageLockedFile.openLockedFile(Paths.get(file));
        converter.convertDataFile(storageFile);   
    }               
}
commented by (1k points)
Thanks for your hint. I really didn't know its a two-stage process. Solved it now.
Notes: Every question must be a separate forum post. Headline: Formulate your question shortly and precisely. Thank you!
Powered by Question2Answer
...