Recent

Author Topic: Copy a Linux file in use?  (Read 4911 times)

KarenT

  • Full Member
  • ***
  • Posts: 120
Copy a Linux file in use?
« on: November 14, 2018, 06:10:23 pm »
UPDATE: I have managed to step through FileUtil.inc to this line and this is where it hangs. It is supposed to have 3-tries and then fail, but it goes wandering off when I step into line #600 in my copy of FileUtil.inc. It never returns to increment the retry count.

SrcHandle := LazFileUtils.FileOpenUTF8(SrcFilename, fmOpenRead or fmShareDenyWrite);

Hi, I need to copy some files as a backup and I am hitting a wall with CopyFile, Laz 1.8.4 and Ubuntu 16.04. This backup is just for my own use, so I tried changing file permissions in the offending folder but still it hits the wall.

Code: [Select]
  CopyGood:=CopyFile(Filefrom,FileTo,[cffOverwriteFile,cffCreateDestDirectory,cffPreserveTime]);

The backup is working well until it hits the Veracrypt folder that **appears** to have a file in use or locked. I say "appears," as I am not sure of the cause of the problem, but it just stops before a file.

Quote
.config/VeraCrypt/Configuration.xml
.config/VeraCrypt/Favorite Volumes.xml
// I think the following file is the problem
.config/VeraCrypt/.show-request-queue

If I shut down Veracrypt, the ".config/VeraCrypt/.show-request-queue" is no longer there and the backup works fine. But then I have nowhere safe to store the backups so I need Veracrypt running. :)

I tried using a "try-except" but it still hangs there. How can I get it ignore this and carry on?

Thanks

« Last Edit: November 14, 2018, 08:40:47 pm by KarenT »

soerensen3

  • Full Member
  • ***
  • Posts: 213
Re: Copy a Linux file in use?
« Reply #1 on: November 14, 2018, 09:55:05 pm »
Probably it's something in VeraCrypt that locks the file for writing, the same way you try to do it (fmShareDenyWrite). Maybe there is a way to tell if a file is already locked before you try to open it, maybe you can open it for reading only then. What happens if you try to copy it from a file manager?

I don't have any idea about the function crashing in this strange way though.
Lazarus 1.9 with FPC 3.0.4
Target: Manjaro Linux 64 Bit (4.9.68-1-MANJARO)

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Copy a Linux file in use?
« Reply #2 on: November 14, 2018, 10:19:15 pm »
UPDATE: I have managed to step through FileUtil.inc to this line and this is where it hangs. It is supposed to have 3-tries and then fail, but it goes wandering off when I step into line #600 in my copy of FileUtil.inc. It never returns to increment the retry count.

SrcHandle := LazFileUtils.FileOpenUTF8(SrcFilename, fmOpenRead or fmShareDenyWrite);

It shouldn't "go wandering off": it goes only from there through LazFileUtils to end up in sysutils and back again. What is really happening is that, by default,  ExceptionOnError, the last parameter of CopyFile(), is set to False, so the call fails silently.

Try to see what happens if you use:

CopyGood:=CopyFile(Filefrom, FileTo, [cffOverwriteFile,cffCreateDestDirectory,cffPreserveTime], True);

You can catch that exception and do whatever it's you want done, in this case you can probably ignore the error and go on to the next file, since the cause is almost certainly what soerensen3 said.

HTH!

ETA: On second thought, even without that parameter it should just return False which you could catch with an old-fashioned: if not CopyGood . . . If it's not doing that then something else is (very?) wrong.
« Last Edit: November 14, 2018, 10:32:47 pm by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: Copy a Linux file in use?
« Reply #3 on: November 15, 2018, 08:01:31 am »
From the manual:
"Note that sharing is advisory on Unix-like platforms." so your mileage may vary anyway.
Indeed, checking the function result is a better idea.

https://www.freepascal.org/docs-html/rtl/classes/tfilestream.create.html

Specialize a type, not a var.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #4 on: November 15, 2018, 02:54:36 pm »
If it's not doing that then something else is (very?) wrong.

Thanks, I had already tried all of those things and it was still doing the wandering off. The program would go dark and stay that way.

I was able to trace another level deeper and it is getting to the SystUtils OpenFile call and not coming back but have been unable to get any deeper into it than that.

I also tried changing the FileUtil.inc Permissions to allow me to change the "OpenFile" flags to "DenyNone" but still it crashes. I am with you, something is VERY wrong.

For now I will hard program out the Veracrypt folder. But, methinks a bug is wallowing in there somewhere.


Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: Copy a Linux file in use?
« Reply #5 on: November 15, 2018, 03:00:46 pm »
If it's not doing that then something else is (very?) wrong.

Thanks, I had already tried all of those things and it was still doing the wandering off. The program would go dark and stay that way.

I was able to trace another level deeper and it is getting to the SystUtils OpenFile call and not coming back but have been unable to get any deeper into it than that.

I also tried changing the FileUtil.inc Permissions to allow me to change the "OpenFile" flags to "DenyNone" but still it crashes. I am with you, something is VERY wrong.

For now I will hard program out the Veracrypt folder. But, methinks a bug is wallowing in there somewhere.
No.
methinks it is OK. Did you try my suggestion - from the other thread - to use a file stream? That gives you more control.
Specialize a type, not a var.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #6 on: November 15, 2018, 04:08:40 pm »
Thanks Thaddy, but let's not be hasty. :)

As mentioned, I tried changing the calls within FileUtil.inc and still the same. That call allows setting the "share" flags etc. And, I had tried all "or" combinations of **all* available Flags. Always the same crash.

Code: [Select]
    SrcHandle := LazFileUtils.FileOpenUTF8(SrcFilename, fmOpenRead or fmShareDenyWrite);

Also I copied the program to an i5 Surface Pro running 16.04 and same thing with two other files. I am not able to step through iton the Surface Pro as Lazarus is not installed.


lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Copy a Linux file in use?
« Reply #7 on: November 15, 2018, 04:38:14 pm »
As mentioned, I tried changing the calls within FileUtil.inc and still the same. That call allows setting the "share" flags etc. And, I had tried all "or" combinations of **all* available Flags. Always the same crash.

Code: [Select]
    SrcHandle := LazFileUtils.FileOpenUTF8(SrcFilename, fmOpenRead or fmShareDenyWrite);

Also I copied the program to an i5 Surface Pro running 16.04 and same thing with two other files. I am not able to step through iton the Surface Pro as Lazarus is not installed.

Take into account that if other processes have completely locked those files your program won't be able to open them no matter what flags you use, which is as it should. The problem, really, seems to be that in those conditions the (ultimate) call to fpOpen() locks your application, as if it were waiting for the file to become unlocked. I haven't been able to dive deeper (lack of time) but something about this should be mentioned in either the kernel or the C library documentation.

All in all, Thaddy's sugestion is sound: try to implement it yourself, which will make it more easy to debug.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #8 on: November 15, 2018, 05:00:55 pm »
All in all, Thaddy's sugestion is sound: try to implement it yourself, which will make it more easy to debug.

Thanks, the "waiting to open" makes sense as it just hangs. When debugging I get to the SysUtils.OpenFile line and the visual line selection goes away.

But, writing my own copy-file function is way out of my league, especially with Linux.

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Copy a Linux file in use?
« Reply #9 on: November 15, 2018, 05:47:59 pm »
But, writing my own copy-file function is way out of my league, especially with Linux.

There's very little "magic" in a CopyFile function; a very basic algorithm is something like:
Code: [Select]
if not OpenFile(Source) then
  return False or raise an exception
else begin
  if not OpenFile(Destination) then
    return False or raise an exception
  else
    while not Eof(Source) do begin
      Read(Source, Buffer)
      Write(Destination, Buffer)
    end
  CloseFile(Source)
  CloseFile(Destination)
end
return True

Add in some error detection and recovery--and  a few bells&whistles--and you're done :)

It can be made shorter using streams, though then the "complexities" go into the streams implementation.

ETA I may be wrong but I think the original problem is due to the implementation of sysutils.FileOpen. At its heart there is this:

Code: Pascal  [Select][+][-]
  1.   repeat
  2.     FileOpen:=fpOpen (pointer(SystemFileName),LinuxFlags);
  3.   until (FileOpen<>-1) or (fpgeterrno<>ESysEINTR);
  4.  

Under certain conditions that loop will run forever. First, because the call is blocking (no O_NonBlock in LinuxFlags) and then because it doesn't allow for fpOpen to fail (which would return -1). My main doubt is about when the error will (or won't) be ESysEINTR.
« Last Edit: November 15, 2018, 06:22:59 pm by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #10 on: November 15, 2018, 08:24:39 pm »
Thanks, but a long time back I had actually done file copy that way but after moving to Ubuntu I found it abysmally slow and did not handle the Permissions and Flags etc, at all well.

Plus files copied that way generally could not be loaded back into another Ubuntu PC unless the user name was the same. Linux, I love it -- I hate it. :)

I am going about it another way and will only select those Folders to backup that I know will not crash. Not perfect but doable for now until I can find a working FileCopy.

That endless loop potential in SysUtils does seem like a bug to me. ;)

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Copy a Linux file in use?
« Reply #11 on: November 16, 2018, 12:56:51 am »
Linux, I love it -- I hate it. :)

Ah, yes ... I pass about 5% of my day using man or info to ascertain what exactly this or that tool does and what its syntax is :D
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #12 on: November 16, 2018, 04:19:52 pm »
Ah, yes ... I pass about 5% of my day using man or info to ascertain what exactly this or that tool does and what its syntax is :D

And, just when you think you know enough, you don't. :)

OK, back to the thread. I have it solved in a brute force kinda way. I did some searching after each wandering-off and all of the problem files are less than 10-Bytes long. In fact, most are zero Bytes, just being used as a Flag I guess, so I have added a line

Code: [Select]
  if FileSize(FileFrom) > 20 then
    CopyGood:=Copy...

A "lost" 20 Byte file should not be a great issue.

Thanks to all.


lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Copy a Linux file in use?
« Reply #13 on: November 16, 2018, 05:49:16 pm »
This may be more elegant than just ignoring files based on size:

Code: Pascal  [Select][+][-]
  1. {uses BaseUnix}
  2. function CanbeOpened(const Filename: String): boolean;
  3. var
  4.   fdesc: cint;
  5.   tries: Integer;
  6. begin
  7.   Result := False; {let's be pesimistic}
  8.  
  9.   tries := 0
  10.  
  11.   repeat
  12.     inc(tries);
  13.     fdesc := fpOpen(FileName, O_NonBlock, O_RdWr);
  14.   until (fdesc >= 0) or (tries = 3);
  15.  
  16.   if fdesc >= 0 then begin
  17.     fpClose(fdesc); {We assume this never fails, but remember Murphy!}
  18.     Result := True;
  19.   end;
  20. end;
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

KarenT

  • Full Member
  • ***
  • Posts: 120
Re: Copy a Linux file in use?
« Reply #14 on: November 27, 2018, 06:25:36 pm »
This may be more elegant than just ignoring files based on size:

Thanks, sorry for the delay but did not think to check back. You are correct, a much softer way to handle it. I'll add that into my standard "tools" unit.

 

TinyPortal © 2005-2018