The import process for the initial checklist does not start with big files

15202
Reporter: atalavan
Assignee: bko
Type: Bug
Summary: The import process for the initial checklist does not start with big files
Environment: Virtual machine with Ubuntu 13.04.3
Priority: Blocker
Resolution: WontFix
Status: Closed
Created: 2014-02-28 15:32:41.712
Updated: 2015-12-14 19:00:05.186
Resolved: 2015-12-14 19:00:05.129
        
Description: When trying to import the initial checklist file for Spain provided by BK, I can select the file and click on 'Import' but the process does not start. The system does not produce any error message. It just reloads the import page without taking any action.

I tried several options in the 'Maximum number of rows' field but it didn't work either.

The input file (https://drive.google.com/file/d/0B6HcodJ_En4haEl2dC1kVVIzWWs/edit?usp=sharing) had originally 65536 lines and it didn't work. It seemed to be a limitation at 65535 lines so I deleted a few but it didn't work either.]]>
    

Author: atalavan
Created: 2014-03-06 15:23:09.454
Updated: 2014-03-06 15:23:09.454
        
I have my suspicions that the file generated for Spain was not complete. It was just on the limit of 65K lines that old Excel format had.

I suggest that someone confirms this fact too.
    

Author: bko@gbif.org
Comment: I realised. Yes, the limit is 65k.
Created: 2014-03-07 09:31:15.273
Updated: 2014-03-07 09:31:15.273

Author: trobertson@gbif.org
Comment: Sorry, I don't understand this issue.  Is the issue that the file only had 65k lines, or that it does not work with checklists with 65k rows please?
Created: 2014-03-18 15:16:27.014
Updated: 2014-03-18 15:16:27.014

Author: bko@gbif.org
Created: 2014-03-18 15:31:49.45
Updated: 2014-03-18 15:31:49.45
        
It does not work with files with 65k rows, which is the limit of the old .xls format. The new format can [contain more|http://office.microsoft.com/en-001/excel-help/excel-specifications-and-limits-HP010342495.aspx], but then it's the [system allocated memory|http://stackoverflow.com/questions/4895230/why-phpexcel-does-not-allow-to-write-more-than-5000-rows] that decides how many can be processed.


    

Author: bko@gbif.org
Comment: For targeted audience countries, this might not be a limiting factor.
Created: 2014-03-18 15:32:41.215
Updated: 2014-03-18 15:32:41.215

Author: atalavan
Comment: I didn't do a thorough test on which is the maximum number of rows that the program admits at this stage. The tests I did were with the original file provided by Burke (65536 lines) and with the same file after deleting some (65530 lines), just in case I was hitting the precise limit. None of the tests succeeded.
Created: 2014-03-18 15:41:38.279
Updated: 2014-03-18 15:41:58.141

Author: bko@gbif.org
Comment: Yes I think when the xls file is generated the max number is already decided by the format. I should test until the defined limit works though. 
Created: 2014-03-18 15:59:07.801
Updated: 2014-03-18 15:59:07.801

Author: bko@gbif.org
Comment: Hi [~atalavan] could you please attach the xls file here so we have the same exact file for testing?
Created: 2014-04-07 11:16:06.497
Updated: 2014-04-07 11:16:06.497

Author: trobertson@gbif.org
Comment: We halted all development on that product.  Closing old issues
Created: 2015-12-14 19:00:05.17
Updated: 2015-12-14 19:00:05.17