Lazarus

Programming => Packages and Libraries => FPSpreadsheet => Topic started by: Soner on August 08, 2018, 08:21:47 pm

Title: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 08, 2018, 08:21:47 pm
Is the ODS-format(LibreOffice) not supported by fpspreadsheet?

I can not open my ods-file with the example application spready.
I have 2 filetypes in attachment. One file is very simple and created with libreoffice 6.  Spready can not  open this file when it contains the formula "WENN".
And the second file has 4 different file formats, spready can open only XLSX-File that created with Excel 2011.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 09, 2018, 11:33:50 am
Yes it  has bug and fpSpreadsheet can not handle "wenn" formula from ODS-documents. ("wenn" is "if"-formula in german.)
The problem is LibreOffice(German Version) does not accept "if" as formula name.
fpSpreadsheet can only handle xlsx-files with this formula.
for xls-Files(office 2002) in some cells accepted.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 09, 2018, 03:08:01 pm
I tried to implent it for excel8-format, it's half solution.
I created in LibreOffice 6 excel97-2003-file with one formula in cell a1:
if(1>2;1;2)

It reads the formula "=if(1>2;1;2)" as "=IF(IF(1>2);1;2) "
Here is my changes in xlscommon.pas
Code: Pascal  [Select][+][-]
  1. function TsSpreadBIFFReader.ReadRPNTokenArray(..)
  2. //...
  3.  
  4.     case token of
  5.       INT_EXCEL_TOKEN_TATTR:
  6.         begin
  7.           b := AStream.ReadByte;
  8.           supported := ReadRPNAttr(AStream, b);
  9.           if supported then begin
  10.             case b of
  11.               $10:
  12.                 begin  // one-parameter sum
  13.                   rpnItem := RPNFunc('SUM', 1, rpnItem)
  14.                 end;
  15.               $2: rpnItem := RPNFunc('IF', 1, rpnItem); //soner add:
  16.               //soner: hier case fehlt option für $8. Es wird zweimal aufgerufen, für linke(1) und rechte(2) seite von if: if(4>5;1;2)
  17.               //trotzdem hat es geklappt es wird, "if"-Funktion wird gelesen,aber nicht ganz richtig.
  18.               //die funktion: =if(1>2;1;2) wird als =IF(IF(1>2);1;2) gelesen.
  19.             end;
  20.           end;
  21.         end;
  22. //...
  23. end;
  24.  
  25. // And here:
  26. function TsSpreadBIFFReader.ReadRPNAttr(AStream: TStream; AIdentifier: Byte): Boolean;
  27. begin
  28.   Result := false;
  29.   case AIdentifier of
  30.     $01: AStream.ReadWord;     // tAttrVolatile token, data not used
  31.     $02: AStream.ReadWord;     // soner add: Excelformat.pdf->page 42 (This is a tAttrIf token (IF function control))
  32.     $08: AStream.ReadWord;     // soner add: Excelformat.pdf->page 42 (This is a tAttrSkip token (skip part of token array))
  33.     $10: AStream.ReadWord;     // tAttrSum token, data not used
  34.     else exit;                 // others not supported by fps --> Result = false
  35.   end;
  36.   Result := true;
  37. end;
  38.  
  39.  

Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 09, 2018, 03:48:13 pm
Yes it  has bug and fpSpreadsheet can not handle "wenn" formula from ODS-documents. ("wenn" is "if"-formula in german.)
The problem is LibreOffice(German Version) does not accept "if" as formula name.
fpSpreadsheet can only handle xlsx-files with this formula.
for xls-Files(office 2002) in some cells accepted.
No. Fpspreadsheet can handle IF formulas; the language does not matter: in xls the function is stored in the file as a number (Func ID on page 74 ff of "excelfileformat.pdf" which is in folder "reference" of the fpspreadsheet installation; in xlsx and ods the formula is always (well - I've never seen anything else) using its English name: unzip the ods file (after renaming to .zip) and scan the file contents.xlm for the <table:table-cell> nodes. This is from your "created-with-libreoffice6.ods":
Code: XML  [Select][+][-]
  1.  <table:table-cell table:formula="of:=IF([.A1]="";[.A1];"leer")" office:value-type="string" office:string-value="leer" calcext:value-type="string">
  2.    <text:p>leer</text:p>
  3. </table:table-cell>

The problem you are seeing is a left-over of bugs due to the last major rework of the formula engine which was required to introduce 3d references in formulas. Precisely, the trouble is caused by the semicolon used by LibreOffice to separate formula arguments - fpspreadsheet and Excel use a comma. I did not investigate the xlsx issue, yet. All this had been working in the past, and I already fixed the ods semilcolon, but came across other issues. All I can do is to ask you for some patience.

These bugs are very elemental and I wonder why they could slip through the fpspreadsheet unit tests...
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 09, 2018, 05:32:54 pm
You have right. I have here old version, spready is compiled in october 2017 and it can open the example files from first posting.

LibreOffice/OpenOffice is worsest office that ever created. It can not save simple table  in xlsx format without format errors. And they supported from many companies and programmers.
Even the spready app or fpspreadsheet can do this better and you are working mostly alone on this library.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 09, 2018, 06:04:13 pm
The current version in trunk reads your ods files of the first post without crashing.

With "Arbeitszeiten-vorlage-libreoffice6.ods" there are several issues, though:
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 09, 2018, 08:13:38 pm
...
With "Arbeitszeiten-vorlage-libreoffice6.ods" there are several issues, though:
  • The IF formulas are always refering to row 1 or row 2, instead of to the same row containing the formula. I'll still have to check this. How did you create these formulas? By typing the first formula into the top-most cell and then dragging it down? In Excel this results in a "shared formula" which fpspreadsheet would not read for ods because I've never seen it there - but maybe they changed this in v6?
Are you saying cell I1 to i41?
Yes I created it so, writing in the top most cell then dragging it with mouse to bottom or copy/paste.
The Arbeitszeiten-vorlage-libreoffice6 from first post and also yours have no problem with rows when I open it with spready*. It show (cell I10..I41) correct formulas and when i change values then spready* calculates correct.

..
  • The work times in columns E to H seem to be missing. But if you look at the cell editor you see the values. If you change the cell color they come up as well. The problem here is that these cells have a very unusual time formatting: LibreOffice displays the number format as "H:MM;@". Normally this means than positive values (part before the semicolon) will be formatted as "H:MM", and negative values (after the semicolon) will be formatted as text. But what does a "negative time" mean? How does a clock display a negative time? I think this is just nonsense. If you remove the ";@" and apply only the format "H:MM" then spready does display the work times correctly - see attached modified file. (Note that in spite of its extension this is an ods file. After saving, just remove the .zip extension by renaming).
I selected cell formats in Excel as time, 13:30. (Kategorie:'uhrzeit', Type:13:30)
In Spready* color changing has no effect. It is visible when i select a format other then custom format. (i:e. Menu>Cell>Number format>short time)

Are you meaning with "negative time" the column "I" ("Stunden/Netto")? This column is formatted as decimal.
I selected red color for negative number to show false enterings.
When you enter for in-time 8:00 and out-time 7:00 then you have -1 Hour, because: 7-8=-1.
(It is not time as hours and minutes. i.e. 7,5 is 7 Hours and 30 Minutes.)

I think we should complete the Excel8-format. Because this format is completed and MS does not change it more. Plus all office applications can read this and write this format. Better one format completed as to many half implemented formats.
ODS or XLSX-Format are changing with new office versions, you will get often this problems.

*When I say here spready then I mean compiled in octobre 5, 2017.

Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 09, 2018, 09:22:47 pm
Oh I forget it, the Excel8-Format ist bad because it only supports up to 256 Columns and one year has 365 days. Last month I made with softmaker office holiday calender, columns for days and rows for employees.  After saving it as xls-format the file was corrupt. Softmaker Office did not warn that there is more columns as the file format supports . :D

Microsoft can nothing do good, spreadsheet format with only 256 columns  :D
First Excel 2007 can create more then 256 columns, exactly max. 16,384 columns.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 09, 2018, 10:36:44 pm
Are you meaning with "negative time" the column "I" ("Stunden/Netto")? This column is formatted as decimal.
I selected red color for negative number to show false enterings.
When you enter for in-time 8:00 and out-time 7:00 then you have -1 Hour, because: 7-8=-1.
(It is not time as hours and minutes. i.e. 7,5 is 7 Hours and 30 Minutes.)
This column is no problem. It has a number format "0,0;[RED ]0,0" which formats positive and negative values as a number with one decimal place, but negative values additionally are colored in red (the sign is dropped - is that intentional?)

No, the problem is in columns E to H which have the format "H:MM;@" where the second part is meaningless in my eyes because it refers to negative times but a watch can only display positive times. And this causes trouble for fpspreadsheet because the procedure for reading the number format expects only a single part, e.g. "H.MM". I don't know why you see these cells in the old spready, and I don't - they did not get a format, I should see them at least in the default format.

In the cell format dialog of LibreOffice Calc I selected the very first item of the time formats, and this one works. If you want a one-digit hour value you can edit the format code in the bottom of the dialog to "H:MM" - see screenshot.

ODS or XLSX-Format are changing with new office versions.
I don't think so - at least not at the level of the incomplete fpspreadsheet implementation. Of course, you must not use the newest super-duper features.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 12:05:45 am
The IF formulas are always refering to row 1 or row 2, instead of to the same row containing the formula. I'll still have to check this. How did you create these formulas? By typing the first formula into the top-most cell and then dragging it down? In Excel this results in a "shared formula" which fpspreadsheet would not read for ods because I've never seen it there - but maybe they changed this in v6?
They did not add shared formulas to Calc; it was just an ordinary bug in the fpspreadsheet formula parser (the scanner of the ODS cell address considered only the highest digit of the row index,  :-[). The IF formulas should now work in the "Arbeitszeiten-vorlage-libreoffice6.ods".
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 12:21:53 am
...
This column is no problem. It has a number format "0,0;[RED ]0,0" which formats positive and negative values as a number with one decimal place, but negative values additionally are colored in red (the sign is dropped - is that intentional?)
..
I don't know, but I saw it today. I think this is the invention from stupid Microsoft programmer. I can't believe how stupid they are. How can user change positive from negative number without '-' sign. I must change the format for those cells.

I don't know about @ part in  "H:MM;@"  . I only used ExcelXp GUI for cell format.  I created the file first with Excel Xp then opened it with LibreOffice and saved it as ODS-Format.
I created the format in the same way like you with your screenshot, only I did it only with ExcelXp. I can make again tworrow and post. They are short time without negative part. I had same dialogue in Excel Xp except the edit field with  "H:MM;@" . Tomorrow I can post my old fpspreadsheet with old spreadsheet. Now I am not at computer.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 12:47:32 am
I tried to implent it for excel8-format, it's half solution.
I created in LibreOffice 6 excel97-2003-file with one formula in cell a1:
if(1>2;1;2)

It reads the formula "=if(1>2;1;2)" as "=IF(IF(1>2);1;2) "
Here is my changes in xlscommon.pas
Code: Pascal  [Select][+][-]
  1. function TsSpreadBIFFReader.ReadRPNTokenArray(..)
  2. //...
  3.  
  4.     case token of
  5.       INT_EXCEL_TOKEN_TATTR:
  6.         begin
  7.           b := AStream.ReadByte;
  8.           supported := ReadRPNAttr(AStream, b);
  9.           if supported then begin
  10.             case b of
  11.               $10:
  12.                 begin  // one-parameter sum
  13.                   rpnItem := RPNFunc('SUM', 1, rpnItem)
  14.                 end;
  15.               $2: rpnItem := RPNFunc('IF', 1, rpnItem); //soner add:
  16.               //soner: hier case fehlt option für $8. Es wird zweimal aufgerufen, für linke(1) und rechte(2) seite von if: if(4>5;1;2)
  17.               //trotzdem hat es geklappt es wird, "if"-Funktion wird gelesen,aber nicht ganz richtig.
  18.               //die funktion: =if(1>2;1;2) wird als =IF(IF(1>2);1;2) gelesen.
  19.             end;
  20.           end;
  21.         end;
  22. //...
  23. end;
  24.  
[ ...]
Now I looked at this patch in more detail - yes it is working. To avoid the double IF you only must not do anything after the $2 tag. Obviously all the reading of the bytes following has been done correctly by the already implemented code, the only problem was that the tag $2 was not handled. Thank you for helping!

Code: Pascal  [Select][+][-]
  1.     case token of
  2.       INT_EXCEL_TOKEN_TATTR:
  3.         begin
  4.           b := AStream.ReadByte;
  5.           supported := ReadRPNAttr(AStream, b);
  6.           if supported then begin
  7.             case b of
  8.               $02: ;
  9.               $10: rpnItem := RPNFunc('SUM', 1, rpnItem);  
  10.             end;
  11.           end;
  12.         end;
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 09:44:45 am
Thank you wp, this is greatest news in this week.

The IF formulas are always refering to row 1 or row 2, instead of to the same row containing the formula. I'll still have to check this. How did you create these formulas? By typing the first formula into the top-most cell and then dragging it down? In Excel this results in a "shared formula" which fpspreadsheet would not read for ods because I've never seen it there - but maybe they changed this in v6?
They did not add shared formulas to Calc; it was just an ordinary bug in the fpspreadsheet formula parser (the scanner of the ODS cell address considered only the highest digit of the row index,  :-[). The IF formulas should now work in the "Arbeitszeiten-vorlage-libreoffice6.ods".

Maybe you have right with this or the parser for xl-files has same  bug like ods parser.
I have here one file that show this error.
The Zip-File contains 2 files:
1) Arbeitszeiten-vorlage.xls is the original file created with Excel Xp.
fpspreadsheet does not show formulas from row I13 to I40. I think i dragged this cell with mouse.
2) Arbeitszeiten-vorlage_if-formulas-dont-visible-in-fpsprsh-2.xls is cutted copy from Arbeitszeiten-vorlage.xls
3) Arbeitszeiten-vorlage-3.xls is copy from Arbeitszeiten-vorlage.xls
Then i copied the cell i10 to i11,i12 and i13 (copy & paste).
As next i selected cell i13 and dragged with mouse until i40.
(Mouse click on + sign on the right bottom corner of selected cell and drag)
Now when you open the file in tsspreadsheetgrid then you will see the cell from i15 to i40 have no formula only result.
I think excel copies in the first row formula and in the next rows reference to it.


Ignore the cell d41. fpspreadsheet does not unterstand linebreak and it is not possible to implent it because this signs are used for tokens tLE and tGT (Page 40). And maybe I am the only one in whole world that created formel in notepad++ and pasted it in Excel  :)
Softmaker and Libreoffice removes it and in Softmaker office, Libreoffice and Excel it is not possible to enter line break.



Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Thaddy on August 10, 2018, 10:26:32 am
Yes. I believe linebreaks should be considered whitespace for formulae (just like in pascal). Possibly soft linebreaks should be allowed for display purposes..
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 10:50:03 am
@Thaddy you have right with soft line breaks, but hard line breaks(char #13#10)   in formula is not possible. User cannot enter it by typing in libreoffice, softmaker office or excel.  I made it only with copy/paste from notepad++. In future we could implent this too(removing break instead of saying formula error), but now it is very difficult and other things have a higher priority.

@wp
Forget the file form post #12 with empty formulas.
Now, I created blank workbooks with Excel 2011 and Excel Xp and created  other cells with mouse drag, same way as in post #12, now there is no error.
I think the file Arbeitszeiten-vorlage-.... has some error. It is very old, I used, changed it too often.
Now, I can do my real work and look clooser to this component at weekend.

Thanks for your erffort and your and felipes great work in this components.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 10:57:40 am
fpspreadsheet does not unterstand linebreak
The linebreak inside a formula has some special attribute within the biff structure (described in section 3.10.5 of "excelfileformat.pdf"). fpspreadsheet does not handle these attrbutes correctly. If it finds an unknown attribute then it stops working. This is also the reason why the IF functions did not work - because it has that attribute token at the beginning of the token sequence (which turns out to be unnecessary - but if Excel writes it we must consider it!).

These crashes must not happen - if fpspreadsheet does not support an attribute it must at least jump over them to stay in sync with the data structure. I am working at this issue at the momenet.

[EDIT]
Done in ccr-r6581. "Arbeitszeiten-Vorlage.xls" is now read correctly without error message; the shared formula issue, however, is not yet resolved.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 12:14:24 pm
There is one other issue with xlsx-Files. (office 2007 + files)
I put  it here if want look it.
When you save xlsx-Files with LibreOffice6 or Planmaker in same format as xlsx then Spready has display errors (draws border lines where there is no border lines or wrong cell widths).
And when you open same files with Excel 2011, LibreOffice6 or Planmaker 2018 then they show them without error.
I put all the files as zip-File here.
Look at file names, .._pm2018 for planmaker2018, .._lo6 for Libreoffice and with no endig excel 2011 file.

Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 12:50:37 pm
You are real fast, latest svn-version understand line breaks.

I see you changed fpsTypes.pas and removed the variable :
UTF8FormatSettings: TFormatSettings;

Now spready can not compiled either you must change spready or add in  fpsTypes the variable again.
I created it in spreadyfiles sCSVParfamsform.pas and smain.pas as global variables made this:
Code: Pascal  [Select][+][-]
  1. //in both units
  2. implementation
  3. var
  4.   UTF8FormatSettings: TFormatSettings;
  5.  
  6. //at end of both units
  7. initialization
  8.   InitUTF8FormatSettings(UTF8FormatSettings);
  9. end.
  10.  
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 01:33:08 pm
Ah, I forgot to commit the updated spready (ccr-r6582). I removed the UTF8FormatSettings because using the procedure, it is not needed as a global record within the library any more. And there are already too many FormatSettings around.

In r.6583, you find drag and drop of files from the Explorer to Spready.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 01:54:32 pm
Now when you open the file in tsspreadsheetgrid then you will see the cell from i15 to i40 have no formula only result.
This is a shared formula issue. Excel writes repeated formulas only once and tags the following formulas as "shared formulas" which means that the formula basically is the same but the cell references have to be adapted according to the new position of the formula.

This feature was broken by the new formula engine needed for the 3D cell references: the shared formulas were transfered to the worksheet correctly, but were erased again, when the formula result was read in the next FORMULA record. Another stupid mistake... (I guess there will be more of them).

Fixed in r6584.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 02:53:45 pm
Woow what a speed, thanks.
I tried new version they are working.

One thing is missing,  but it is not important.
When you save xls-workbook with "shared formula" from spready then you get "range  check error" but you can save it as xlsx-format.
It is the file "Arbeitszeiten-vorlage.xls" from reply #12.


Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 03:04:03 pm
Cannot reproduce. You are loading "Arbeitszeiten-vorlage.xls" of reply #12 into Spready, save it as .xls again, ang get a "Range check error"?
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 10, 2018, 05:56:29 pm
When you save xlsx-Files with LibreOffice6 or Planmaker in same format as xlsx then Spready has display errors (draws border lines where there is no border lines or wrong cell widths).
Should be fixed now (ccr-r6588).
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 10, 2018, 11:24:17 pm
woow you are really fast  :o I thought some things are not solveable.

Most problems are solved, I saw at testing testing two issues:
1) When you save in xls-Format from Spready then the comments are lost.
2) Some cells xlsx-Files from LibreOffice6 are false drawn. Look at screenshot.
I think more people will ask here about this problem, because LibreOffice is used
often for saving as xlsx-format.

Cannot reproduce. You are loading "Arbeitszeiten-vorlage.xls" of reply #12 into Spready, save it as .xls again, ang get a "Range check error"?
Yes, but now I see this is not the file with "range check eror". I can't find it on this computer. When I find it then i post it here.

We started here with ODS-format and now fpSpreadsheet and TsWorksheetGrid are supporting all formats from 3 office software big thanks.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 11, 2018, 02:47:39 pm
1) When you save in xls-Format from Spready then the comments are lost.
Yes, unfortunately (http://wiki.lazarus.freepascal.org/FPSpreadsheet#Progress_by_supported_cell_content) But I gave up to implement writing of comments to BIFF8 files which turned out to be too complicated for me. I was sure that everything is correct, but fpspreadsheet writes only unreadable files. Therefore, I added an "exit" immediately after entering TsSpreadBiff8Writer.WriteComments (in unit xlsBiff8). So, the old code is still there but currently not used. Maybe somebody would like to dive into it and submit a patch. But be warned: the exelfileformat.pdf is of little help, and you need a lot of reverse engineering using originally written xls files and BIFFExplorer.

2) Some cells xlsx-Files from LibreOffice6 are false drawn. Look at screenshot.
You mean the row heights and the truncated cells, like "Arbeitgeber" in F2 etc? The latter is due the wordwrap which was falsely detected for the way LibreOffice writes the xlsx files - should be fixed in ccr-r6589. As for the row height I tried to measure the row heights, it seems that they are correct. But the positioning of the text might be off by 1 or 2 pixels. Normally this does not matter, but since the row heights (and column widths as well) in this file are very tight, every pixel counts. Instead of trying to solve this for this file (and still seeing issues in other files) I activated the WorksheetGrid's RowHeight calculation routine in Spready. In the new version, check "Update row heights after loading" in the Settings menu, and the internal row height calculations will be used when a new file is loaded. The row heights may turn out to be a bit different, but the text positioning will probably be much better. One warning, though: This option should not be used when loading large files with inidividual character formatting within many cells: Because the height of every cell must be calculated here, the loading process can be slowed down considerably.
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: wp on August 11, 2018, 10:59:16 pm
Finally fixed also the detection of the time format 'H:MM;@' in ODS files (ccr-r6597) making also the ODS file of the very first post readable. Some of what I had written above (reply #5, #8) was nonsense...
Title: Re: Is the ODS-format(LibreOffice) not supported by fpspreadsheet?
Post by: Soner on August 12, 2018, 05:05:24 pm
Good news, thanks.
TinyPortal © 2005-2018