Recently we have encountered this issue about exporting reports to excel file.
Our report table contains more or less 1.26 million records as of now. We are trying to create an excel(2007) file for this, so approximately this would be a 2 worksheet file because only about 1million rows is supported by excel 2007 per sheet.
We have no problem during the creation of worksheets and during the filling of data into the worksheet objects, memory is manageable at this point.
Our problem is when writing(Workbook.Save()) the excel data into the stream(file, memory). The memory consumption is just too high and when it reaches 1GB or higher it will no longer continue.
P.S. This is a summary report, the upper management wants this in a single excel file.
Is there any way we can at least lessen the memory usage? Or is there any alternative to generate excel files with this amount of data.
Below is a simple test, we pulled out from the actual code that replicates this issue:
static void Main(string[] args) { var xls = Workbook.MaxExcelRowCount; var xlsx =Workbook.MaxExcel2007RowCount; var wb = new Workbook(WorkbookFormat.Excel2007, WorkbookPaletteMode.StandardPalette); var ws = wb.Worksheets.Add("Sheet1"); var sheetCtr = 1; var limit = 1256898;//the number of rows to process var rr = 0; for (var row = 0; row < limit; row++) { if (rr >= xlsx - 1) { sheetCtr++; ws = wb.Worksheets.Add("Sheet" + sheetCtr.ToString()); rr = 0; //ClearMemory(); } for (var col = 0; col < 10; col++) { ws.Rows[rr].Cells[col].Value = string.Format("data{0}-{1}", row, col); } Console.Write("Row="); Console.Write(row.ToString()); Console.WriteLine(); rr++; } Console.Write("Saving worksheet..."); //This will no longer respond when memory gets 1.gb or higher //wb.Save("wb2.xlsx"); using (var stream = new System.IO.MemoryStream()) { wb.Save(stream); stream.WriteTo(new System.IO.FileStream("test1.xlsx",System.IO.FileMode.Create)); } Console.WriteLine("Workbook saved."); Console.WriteLine("Done"); Console.Read(); }
Thank you for reposting your question from stackoverflow here. As I commented there:
"You are trying to save an extremely large amount of unique strings here: 12,568,980, each of which holds about 42 bytes in memory, which is 527,897,160 bytes, just for the strings. Then you have the memory to store the strings: roughly 300 bytes per block of 32 cells with values on a row, which is another 377,069,400 bytes for the cells. That's already 900MB, and you still have to add in the memory for rows and other excel objects and some memory to do the actual save."
So it may not be possible to get this under the 1GB limit, but it is worth looking into to see what memory improvements can be made. I have forwarded this post to the Developer Support Manager and a DS engineer will be contacting you about this issue.
Thank you very much for the quick reply, we are looking forward for any workarounds.
Hello jmaglasang,
Development had made some changes to improve performance of in our Excel assemblies. This changes are made in build Web.Nightly_11.1.20111.2135 and available for you to download in build later than this release (Web.Nightly_11.1.20111.2135). As such, I would recommend you to download latest service release from our website and upgrade your project to that service release.
I hope this helps.
Hello Bhadresh,
Based on our test on the service release, the processing now is better, but most of the time were are getting OutOfMemoryExceptions on Workbook.Save(). We have also tried 50% amount of data being processed but we are still getting OOM exceptions.
Here is the specs of the machine where we conduct our test:
Regards,
jmaglasang
Please see attached file for the test project, I have pulled out that code from the actual source. I have also tested using Excel Automation the result seems to better but sometimes I encounter OOM, the memory consumption is lesser also compared to IG.Excel.
NOTE: We need to save the excel(binary data) to the database, so instead of writing it to the file, we wrote it into a MemoryStream.
In our setup, there are 2 ways that the excel report will be generated
Thanks,
As what you have suggested, I upgraded the project to Infragistics2.Documents.Excel.v11.1.dll, tested Infragistics2.Documents.Excel.v11.2.dll also but i still encounter OOM exception.
I have tested the temporary file approach and it seems to be promising. We will try to apply this approach in our project.
Thank you very much. Appreciate your help!