Lazarus

Announcements => Lazarus => Topic started by: Martin_fr on October 14, 2018, 09:05:00 pm

Title: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on October 14, 2018, 09:05:00 pm
With the Lazarus Release Candidate 1 for 2.0 a new debugger for Mac users has been shipped.
It is based on LLDB, which is provided by apple and is ready to use. So there should no longer be a need to build and codesign gdb.

As the LLDB integration is all new, it needs a lot of testing. So this is a ...

call to all Mac users:
Please test the new LLDB based Lazarus debugger.

To test the new debugger, please download the Lazarus 2.0RC1.

- Open the IDE and install the package "LazDebuggerFpLLdb".
  It should be in the list of available packages, but if not it is in components/lazdebuggers/lazdebuggerfplldb
  Make sure to use the one with "Fp" in the name. (There also is LazDebuggerLLdb, but it is not as good.)
  Restart the IDE.


- Go to Tools > Options > Debugger
  In the "debugger type" dropdown, you should find and select "LLDB debugger (with fpdebug)"
  The edit below this (where you normally have the path to gdb), should be changed to the path of lldb.
  (lldb comes with the tools from Apple; and as such it is already codesigned by Apple).


To report any problems:
Start Lazarus from a shell with the following command line (replace pathes according to your installation):

You may need to use startlazarus, or specify the re-build lazarus which may be in your home folder / pcp.

Code: Pascal  [Select][+][-]
  1. /path/to/lazarus/lazarus.app/Contents/MacOS/lazarus --debug-log=/path/to/yourfiles/laz.log --debug-enable=DBG_CMD_ECHO,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_DEBUG,FPDBG_DWARF_ERRORS,FPDBG_DWARF_WARNINGS,FPDBG_DWARF_VERBOSE_LOAD,FPDBG_DWARF_DATA_WARNINGS,DBG_VERBOSE,DBG_WARNINGS,DBG_STATE,DBG_EVENTS,DBG_THREAD_AND_FRAME,DBG_ERRORS
  2.  

EDIT ON 16 DEC 2020:
Please also add to the list of keywords for logging
  ,FPDBG_COMMANDS,FPDBG_THREADS,FPDBG_QUEUE


Attach the log file after reproducing the error.

--------------------
EDIT:
If you have a lot of units with debug info (eg all of LCL with debug info) you may have to increase ulimit.
https://bugs.freepascal.org/view.php?id=34467
(should be fixed in RC3)

Known Issues in the 2.0 release that are fixed for 2.0.2:
Svn users can update in fixes branch. None svn users can download replacement files via http as directed below.

Known Issues that will be in 2.0 release:

Known Issues in RC3 fixed for 2.0:


For the testcase please see: http://forum.lazarus-ide.org/index.php/topic,42869.msg307134.html#msg307134

More on "debug info type" (dwarf): http://forum.lazarus-ide.org/index.php/topic,42869.msg311363.html#msg311363
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: dbannon on October 15, 2018, 03:10:30 am
Martin, forgive my ignorance, I am unsure of how gdb / LLdb interact with heaptrc.

Specifically about memory leaks, will using LLdb give me the same sort of info on the Mac I am used to getting on Linux ?

Even if not, this is an astounding step forward. We loose so many potential Mac user who just read through those code signing instructions and shake their heads sadly...

Davo
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Cyrax on October 15, 2018, 12:19:58 pm
GDB/lldb have nothing to do with heaptrc functionality. Heaptrc uses compiler generated debug info to display the lines and their numbers where the leak happened.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on October 15, 2018, 01:08:09 pm
Heaptrc, works without the debugger.

GDB/LLDB is for setting breakpoints, single stepping, inspecting variables.

The LLDB based does not yet have all the features. E.G. "step to cursor" does not yet work. And the disassembler may not always work.

As for properties (with getter function): They can not be watched, same as under gdb.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Josh on October 15, 2018, 02:30:36 pm
Hi

Had a quick look and did not see a folder for mac osx x86_64 to test? Just OSXi386

Am I missing something?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on October 15, 2018, 03:25:16 pm
Had a quick look and did not see a folder for mac osx x86_64 to test? Just OSXi386

Am I missing something?

Not really related to the debugger, you may want to follow this up on the RC announcment. http://forum.lazarus-ide.org/index.php/topic,42868.0.html

Afaik currently only carbon is pre-build. cocoa (64bit) is still alpha/beta.
But I believe (not sure) that the fpc package installs both, 64 and 32 bit. Then you can rebuild the IDE for 64 bit.

The debugger should work for both 64 and 32 bit. (As long as Apple does not remove 32bit support from lldb)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: jwdietrich on October 18, 2018, 08:35:44 pm
Sorry, but I can't find the package, see screenshot in the attachment. Did I overlook anything?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Cyrax on October 18, 2018, 08:58:02 pm
Sorry, but I can't find the package, see screenshot in the attachment. Did I overlook anything?

Are you using the Lazarus Release Candidate 1 for 2.0? I don't think that OPM have it. You need to install it standard/old way (Package->Install/Uninstall Packages), not via OPM.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on October 18, 2018, 10:16:14 pm
It is not it OPM.

The installer for 2.0RC1 contains the files, and they will be on your disk.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: jwdietrich on October 19, 2018, 10:23:48 pm
Thanks, it woks now.

In Lazarus 2.0RC1 the package is in /Developer/lazarus/components/lazdebuggers/lazdebuggerfplldb/. The path to the debugger can easily be found by typing which lldb in the terminal.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: jwdietrich on October 20, 2018, 03:38:30 pm
My first impressions are:
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: ChrisR on October 22, 2018, 08:23:50 pm

I realize that the debugger and heaptrc are separate, but having the combination of a simple-to-install debugger and a heaptrc that provides line numbers would be a terrific advance for Lazarus. Here is a simple project that shows that heaptrc is able to report leaks, but on the Mac you do not get the line numbers seen when run on Linux or Windows:
http://forum.lazarus.freepascal.org/index.php/topic,42755.0.html
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on October 28, 2018, 01:06:05 pm
About HeapTrc
but on the Mac you do not get the line numbers seen when run on Linux or Windows:
If that is the case, and if the settings are correct (e.g the debugger gets line info, but the heaptrc does not), then this is an issue in fpc (and may need reporting there)

HeapTrc is entirely part of fpc.

There are several possible issues though:
1) It may affect dwarf only (or be limited to certain versions of it), or stabs only.
2) It may be broken only for external / none-external debug info.

Additionally the stacktrace dumping code in fpc has a safety feature, that can have "side effects".
If an address in the trace can not be resolved, the code will disable address resolving for all further addresses. This is, in case there was an error, which could lead to the error dumping another trace, ending in endless recursion. But this also happens, if there just is no data for that one address. (At least that is what it used to be, may have changed).

So if your top address, is not resolvable....
You can try (not tested myself on Mac), to load/paste the trace into the IDE's "Leaks and traces", then click resolve. In the following open dialog, select the application executable.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: dbannon on October 28, 2018, 11:28:30 pm
Said Martin_Fr :
Quote
About HeapTrc
Quote from: ChrisR on October 23, 2018, 04:23:50 am

    but on the Mac you do not get the line numbers seen when run on Linux or Windows:

If that is the case, and if the settings are correct (e.g the debugger gets line info, but the heaptrc does not), then this is an issue in fpc (and may need reporting there)

Has been reported -
https://bugs.freepascal.org/view.php?id=32775

The mytest.pas demo pasted there is just fpc, no lazarus component. On Linux shows a line number, actually the number of the line where  BadProcedure() is called from rather than the new() statement but thats still enough to track leaks down. On mac, no line number or unit name.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: ChrisR on October 29, 2018, 03:18:53 am
@dbannon Thanks for issue 32775, a very elegant bug report, as you note, shows this is a fpc issue, not Lazarus. Thanks.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: baldzhang on November 02, 2018, 02:40:00 pm
testing passed.

macos: 10.13.6
lazarus: trunk
fpc: 3.0.4
widgets: cocoa

will this merge to 2.0 ?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on November 02, 2018, 03:34:19 pm
This is already present in the 2.0 branch. So yes, it will be part of 2.0
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: RayoGlauco on November 03, 2018, 12:26:01 pm
I am using High Sierra on VirtualBox and Windows 10.
LLDB works well, and setup is easy in Lazarus.
Maybe a bit slow, but it's VirtualBox and my computer is not quite fast.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Akira1364 on November 03, 2018, 07:30:36 pm
I've mentioned this before, but it's worth noting that this debugger does also work fine (and quite well) on Windows (and should technically work on any platform that can run LLDB, which is all desktop platforms for the most part.)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on November 03, 2018, 09:19:35 pm
On Windows you should get a much better experience with the gdb based debugger. Or if you want: fpdebug (which is used if the FpLldb debugger). If you want fpdebug (on win/linux) you can choose between pure fpdebug, or gdb+fpdebug.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Akira1364 on November 05, 2018, 12:54:45 am
On Windows you should get a much better experience with the gdb based debugger. Or if you want: fpdebug (which is used if the FpLldb debugger). If you want fpdebug (on win/linux) you can choose between pure fpdebug, or gdb+fpdebug.

I'm aware the GDB debugger is currently more feature-complete in Lazarus. I just meant that LLDB is not in any way only a "Mac" thing, and that generally speaking it's a better debugger than GDB (as all LLVM/Clang tools are compared to their GNU equivalents.)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on November 05, 2018, 10:27:54 am
I'm aware the GDB debugger is currently more feature-complete in Lazarus. I just meant that LLDB is not in any way only a "Mac" thing, and that generally speaking it's a better debugger than GDB (as all LLVM/Clang tools are compared to their GNU equivalents.)
That comparison is currently hard to make...
First of all, lldb (6.0.1 on win) still prints a lot of warnings about dwarf tags that it does not know (If you watch variables without fpdebug). That may depend on the exact version of lldb, and for different OS, it is not always easy to get the latest version. (On fedora the only version I could get (6.0.1) crashed in my tests)

Also (and that may just be me), I could not get the "MI" interface to run on windows. That made lldb much harder to use, because there is no common way to detect when the output of a command finished. (The prompt is not printed in time, if using a pipe).

Further LLDB (unlike GDB) only understands c-based syntax. GDB has a (very) limited pascal understanding.
This means to get a pointer deref in lldb you have to write "*foo".
The reason you can do Pascal style in the IDE is that there is "lldb (with fpdebug)": fpdebug is a separate debugger, that handles the pascal syntax. It has its own dwarf reader, and uses lldb, to dump raw memory (watch the console, or "ide internals > debug output)

For many other things I dont have a comparison. And also I did not spent that much time on lldb, to really judge it. So I cant fully compare the two.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Igor Kokarev on November 13, 2018, 10:47:20 am
Thanks for the new lldb debugger! It works fine in macOS High Sierra.

I only noticed that starting of an app with LLDB debugger takes more time (4-6 seconds longer).

Also lldb can't show values for WideString variables.

One problem. I can't install lazdebuggerfplldb on macOS Mojave 10.14. Please include pre-built lldb debuger for Lazarus 2.0 RC2.

Because I can't compile any carbon or 32-bit app on macOS Mojave 10.14 in Lazarus, I get linking error. Only 64-bit / cocoa apps can be compiled and works fine.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on November 13, 2018, 12:55:00 pm
I only noticed that starting of an app with LLDB debugger takes more time (4-6 seconds longer).
Sorry about the slow down There is probably not much that can be done right now.
You can try the package LazDebuggerLLdb (without the FP). But that is pure lldb, no support for pascal style. All watches need to be specified in c syntax (*form1 or form1->FFooBar), and everything displays in c syntax. If that is slow too, then it is lldb itself (and nothing can be done at all, but hoping that lldb improves).
Also this may not support all datatypes. Or it may support more. Its not tested for that. (hence Alpha / also its more a proof of concept, so it will not get much fixes in that area)

Quote
Also lldb can't show values for WideString variables.
For LazDebuggerFpLLdb, (the "with fpdebug")  version, I added a workaround for widestring. They *should* work in the next version. WideChar is still broken (including SomeWideString[n]). Will have to see if it can be fixed in time.

Quote
One problem. I can't install lazdebuggerfplldb on macOS Mojave 10.14. Please include pre-built lldb debuger for Lazarus 2.0 RC2.
Because I can't compile any carbon or 32-bit app on macOS Mojave 10.14 in Lazarus, I get linking error. Only 64-bit / cocoa apps can be compiled and works fine.

Already committed for RC3, the IDE will have lazdebuggerfplldb pre-installed. (Only lazdebuggerfplldb, only the one with fpdebug).

EDIT:
widechar should also work

EDIT:
Quote
Also lldb can't show values for WideString variables.
Just for info/background: That is actually fpdebug. (Not tested if lldb itself can). The IDE bypasses lldb for all work on pascal related data. lldb just provides the memory dump (raw bytes)
But that is internal info...
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Igor Kokarev on November 13, 2018, 02:41:21 pm
Martin,

Many thanks for your quick reply and for all your efforts to improve LLDB debugger in Lazarus! I will try RC3 version once it ready.
Title: Re: New debugger for Mac based on lldb - "ignore exceptions" ineffective
Post by: kcandrews on November 27, 2018, 06:39:59 pm
Quick test with High Sierra OSX 10.13.6, Lazarus 2.0.0RC2, FPC 3.04, i386-darwin-carbon:  I don't yet have time to come up with test projects to prove this (in a simple way), but my big project consistently had troubles when debugging with lldb (fplldb) -- always stopping on every raised exception -- even when the exceptions were set to be ignored.  This made debugging totally impractical (and requires sticking with gdb for now).  I also had regular crashes when viewing objects/variables during debug breakpoints.  Do know that really I greatly appreciate the effort being made to move away from gdb on the mac since it is so messy to fool with codesigning, crsutil non-standard settings, etc.
Title: Re: New debugger for Mac based on lldb - "ignore exceptions" ineffective
Post by: Martin_fr on November 27, 2018, 07:49:37 pm
always stopping on every raised exception -- even when the exceptions were set to be ignored.

Thanks for pointing out. That was a simple oversight when putting the new debugger together. The code for this is identical with what all other debuggers do.
I fixed this, and it will be in RC3.

Also fixed incorrect classnames for exceptions (the 2 digits at the start).
And fixed a mem leak.

Quote
I also had regular crashes when viewing objects/variables during debug breakpoints. 
Crashes in the IDE, or crashes of lldb itself (is lldb still in the task list?)

If it is not to much trouble, run the logfile
Code: Pascal  [Select][+][-]
  1.     /path/to/lazarus/lazarus.app/Contents/MacOS/lazarus --debug-log=/path/to/yourfiles/laz.log --debug-enable=DBG_CMD_ECHO,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_DEBUG,FPDBG_DWARF_ERRORS,FPDBG_DWARF_WARNINGS,FPDBG_DWARF_VERBOSE_LOAD,FPDBG_DWARF_DATA_WARNINGS,DBG_VERBOSE,DBG_WARNINGS,DBG_STATE,DBG_EVENTS,DBG_THREAD_AND_FRAME

You can do that always, but then I suggest to start the IDE from a script, that will first delete the old log. Or it will grow really big).
Then when the crash happens, attach or send the file.
(If you did inspect watches with sensitive data, or do not whish to reveal library names, or procedure names, that may be part of debugging, you can edit the file. You can also sent it in private)

There also already are various fixes (including some crashes) that will be in RC3. (fixes made after RC2)

Quote
Do know that really I greatly appreciate the effort being made to move away from gdb on the mac since it is so messy to fool with codesigning, crsutil non-standard settings, etc.
Thanks.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on November 28, 2018, 02:44:12 am
Martin,
Thanks so much for your response!  Great to know these things are already addressed.
I'll plan to do this again with logfile output sometime in the next few days and send it along!  I know how important getting the specifics on bugs can be (long time software architect/consultant/programmer, now retired mostly).
Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on November 28, 2018, 03:50:34 pm
Martin,

Thanks, this is fantastic news! I just set up lldb with the latest Cocoa trunk. Installation went smoothly following your directions. So far so good, I'll try to run it through all of my projects and report back any issues.

Cheers,
VTwin
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Trenatos on November 28, 2018, 05:10:10 pm
Fantastic news, much appreciated!

Will give lldb a go very soon, would be nice to have a debugger working on my Mac!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: MISV on November 30, 2018, 03:08:57 pm
Problem (lazarus trunk + Mojave):


Whenever I run my app with the new lldb debugger beta I get this error:

Unable to  to open file
Quote
"/users/example/lazarus64bit/config_lazarus/onlinepackagemanager/packages/Indy10/lib/x86-64-darwin/idServerIOHandlerStack.o"

I have of course:

Not quite sure what is to blame here or what is going on
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on November 30, 2018, 05:36:40 pm
Have you tried to increase "ulimit"? (allowed amount of open files / filehandles)

Unfortunately the debugger attempts to open a lot of files and runs into the limit.
It's a known issue, but may be a while until fixed.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: MISV on November 30, 2018, 10:27:22 pm
Had to Google it

If I in terminal write
Code: Pascal  [Select][+][-]
  1. launchctl limit maxfiles
I get
Quote
max files 256 unlimited


If I in terminal write
Code: Pascal  [Select][+][-]
  1. launchctl limit maxfiles 10240 unlimited
I get
Quote
Could not set resource limits: 1: Operation not permitted


If I in terminal write
Code: Pascal  [Select][+][-]
  1. sudo launchctl limit maxfiles 10240 unlimited
it appears I set maxfiles correctly :)

I will now test....

And it seems to work

Yay! :)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 01, 2018, 03:19:31 am
For app I'm currently developing, using lldb(alpha) it worked, kind of (buggy but at least runs). With lldb(with fpdebug), it crashes immediately:

Unable to open file "/Developer/lazarus/components/tachart/lib/x86_64-darwin/qt5/tadatapointseditor.o".

Laz 2.0RCL2/64bit/Cocoa. Compling to 64bit/Qt5.

bigDan
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 01, 2018, 10:40:41 am
For app I'm currently developing, using lldb(alpha) it worked, kind of (buggy but at least runs). With lldb(with fpdebug), it crashes immediately:

Unable to open file "/Developer/lazarus/components/tachart/lib/x86_64-darwin/qt5/tadatapointseditor.o".

Laz 2.0RCL2/64bit/Cocoa. Compling to 64bit/Qt5.
See above
Code: Pascal  [Select][+][-]
  1. sudo launchctl limit maxfiles 10240 unlimited
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 01, 2018, 06:24:21 pm
That did the trick!

Looks like step-over still does a step-into, though.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 01, 2018, 09:16:49 pm
Looks like step-over still does a step-into, though.
32 and 64 bit?

Can you send a log when that happens?
(I was not aware that happens on the Mac at all)

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 01, 2018, 10:57:33 pm
I can only test for 64bit. Qt5 won't link for i386 for me.

I tried compiling 32bit/cocoa, but that results in the message

Code: Pascal  [Select][+][-]
  1. Lazarus IDE v2.0.0RC2 rexported - sjsconfig
  2. Access Violation
  3.  

I don't know how to provide the log you desire. If you send me a link to the procedure, I'll give it a shot.

Enclosed are 2 pics of the failure. In debug1.png, I am at breakpoint on line 596. When I press the step over button, instead of going to line 597, it goes into GridVisible as seen in debug2.png.

bigDan
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 02, 2018, 01:14:29 am
Instructions for the log file are in the first message of this grid.

I currently do not have a Mac, so I can not reproduce issues myself (many problems I can test with lldb on windows).


Did you get this stepping issue with "lldb (alpha)" too?
Probably, yes, because stepping is done the same in both of the lldb based debuggers.  The difference with the fpdebug version, is only about displaying watches and locals.

There are 2 possibilities here (and the log will tell which one)
1) The IDE sends the correct command, but lldb has a problem. Then there is little that can be done.
   (There is a very similar issue on win64 with gdb. It is not yet known what causes this. )
2) The IDE sends the wrong command. Or there was an ignored exception, and the IDE did not manage to recover that.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 02, 2018, 06:25:04 pm
Crud. Unable to get log working. I get the same 'unable to open file /Developer/....../qt5/taenumerators.o' even though I have done the 'launch limit maxfiles 10240 unlimited'.

Yes, this same issue occurs with the lldb(alpha) as well.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 02, 2018, 07:02:50 pm
In this case, open the IDE normally, and before starting to debug go to menu:
   View > Ide Internals > Debug Output

Once you run your project, the window will fill with the IDE <> lldb communication.

When you press F8, you can see what the IDE issues:
thread step-in
thread step-over

Thought there will be plenty of more lines, as stack and watches gets done.

You can copy the content, and attach it.
Take the copy immediately after step-over went wrong, so I know the last step command should have been step over.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 02, 2018, 08:19:18 pm
OK, finally got it!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 02, 2018, 09:18:15 pm
Martin,

I managed to get lldb working on macOS cocoa lazarus 2.0 revision 59715, the latest fixes, after a great deal of fiddling. lldb seems to be working on very simple projects, but I tried to run one of my large projects with it, and get an access violation. The project runs fine without lldb.

I have attached the log in case it may be helpful.

VTwin
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 02, 2018, 09:24:19 pm
@bigDan

Ok, I had a look. LLDB actually believes that it stopped at the end of the "step over". Or in other words, lldb seems to think that the (first (few) ?) line(s) of the subroutine are part of the caller.

I have seen the same issue with gdb on win (but only 64 bit apps).

Given that 2 different external debuggers give that issue, I would currently guess that this is some problem with the debug info written by fpc.

I dont know what the equivalent of objcopy is on Mac, on Windows it seems the following may help: http://forum.lazarus-ide.org/index.php/topic,42183.msg294247.html#msg294247


Ultimately what is needed is a small project with as little dependencies as possible that allows to reproduce this, and then report it to the fpc bugtracker.
But sofar it has only occurred in  complex projects and was not reproduced: https://bugs.freepascal.org/view.php?id=34159
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 02, 2018, 09:34:30 pm
I managed to get lldb working on macOS cocoa lazarus 2.0 revision 59715, the latest fixes, after a great deal of fiddling. lldb seems to be working on very simple projects, but I tried to run one of my large projects with it, and get an access violation. The project runs fine without lldb.

I have attached the log in case it may be helpful.

Is that RC2 ?

There is one known crash (inspecting watches/locals/hint...) that is fixed for RC3 already.

If you want to apply the changes:
https://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=59443


Unfortunately the log has no line numbers on the traces, there is a bug in FPC that prevents them from being printed.

If you run the IDE itself under lldb (probably easiest to use lldb from commandline for that / but you can debug the IDE inside the IDE too).
"access violation" should get caught, and lldb should be able to do a proper stack trace with lines.

---
EDIT:
just seen you are on svn.

If you could try to get the stacktrace please.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 02, 2018, 10:18:55 pm
I have not done a stack trace before, so am trying using instructions here:

http://wiki.lazarus.freepascal.org/Logging_exceptions

to override TApplication.OnException:

Code: Pascal  [Select][+][-]
  1. Application.OnException := @CustomExceptionHandler;
  2.  

Unfortunately, it is not triggered. Any suggestions?

EDIT: I'm now trying to override System.ExceptProc
EDIT: Can't seem to figure out how to do this

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 12:48:58 am
lldb is working in another large project. One thing I notice is that "Step Over" does not seem to be working, it seems to "Step Into". I have to use "Step Out" to get back.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 03, 2018, 01:28:37 am
lldb is working in another large project. One thing I notice is that "Step Over" does not seem to be working, it seems to "Step Into". I have to use "Step Out" to get back.
The step problem appears to be related to some debug info that fpc writes for the exe. Since both gdb and lldb react the same way to it, I think it is a problem in fpc.

Quote
to override TApplication.OnException:
Tracking the exception from inside the running IDE may not work anyway. DumpStack and similar commands will produce the same address-only trace on Mac. Fpc does not have the line info reader for dwarf on Mac https://bugs.freepascal.org/view.php?id=32775

If you can start the IDE from inside lldb (on the command line), then lldb should stop it, once you get the access violation (for other exception that are "raise"d, you need a breakpoint at FPC_RAISEEXCEPTION). You can run the debugger in the IDE, even while the IDE is debugged in lldb. When the command line lldb stops the IDE at the error, you can get a trace (with filenames and lines).

If you want comfort (and have a big monitor):
Open the IDE, (with high file limit). Open the project ide/lazarus.lpi (in the lazarus install dir). Run F9. A 2nd IDE will be started, this is running in the debugger of the first. Get the error in the 2nd IDE, and then get the stack from the first.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 03, 2018, 03:46:11 am
I've created a tiny application that will duplicate the DEBUG STEP OVER bug for me.

Instructions are at the top of the form unit.  It fails consistently on DARWIN. I don't have 2RCL2 installed on anything else to try at the moment.

bigDan
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 03, 2018, 12:04:11 pm
Instructions are at the top of the form unit.  It fails consistently on DARWIN. I don't have 2RCL2 installed on anything else to try at the moment.
It all comes down to if it will also fail for someone in the fpc team. See the bug report.

This is not going to change with further RC, or 2.0. The issue is in fpc. Last time I checked it was also present in fpc trunk. So it likely will be in 3.2 too.


--
For reference: http://lists.freepascal.org/pipermail/fpc-devel/2018-December/039813.html
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: bigDan on December 03, 2018, 04:48:30 pm
Thanks for looking into this, I appreciate it. Overall, the new lldb code is definitely much better!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 07:10:24 pm
Many thanks. Perhaps bigDan's bug report will help fix the step bug.

Thank you for the suggestions on stacktrace, I will keep working at it.

The exception is puzzling, as it occurs before even Application.Initialize. Perhaps I have an incorrect setting in the project options, possibly between setting up lldb and cocoa I made an error. 
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 03, 2018, 07:21:38 pm
The exception may be while fpdebug is getting ready. Maybe while it gathers dwarf info from the files on disk.

The debugger is kind of split into 2 parts.

1) LLDB: all interactions with the running exe. Used for breakpoints, stepping, running, pause, ....
  used to read memory dumps from the running app

  In the NONE fpdebug version lldb also does watches, locals ....  (no line info though, therefore the none-fpdebug version does not have the blue dots in the editors gutter)

2) Because lldb only understands C style input for watches (object->field), and displays all results in c style too, the IDE uses fpdebug for watches and locals...
Fpdebug is a debugger written in Pascal, for Pascal.
In the lldb(with fpdebug) debugger, Fpdebug reads the apps memory by getting raw memory dumps from lldb. Fpdebug loads its own copy of the dwarf info directly from the files on disk (bundle).

---------------
That means, if the IDE crashes, it could be Fpdebug. You can use the pure lldb. But getting watches will be all c style work... But better to get Fpdebug fixed.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 08:07:07 pm
Thanks! One puzzle is why it works on one of my projects, and not the other. They are configured the same (as far as I can figure out yet). This time, I at least got an error message:

Unable to open file "/pathtoproject/lib/x86_64-darwin/vsizedlg.o"

The file is, however, there. It is for a dialog box.

I had some initialization code in that unit (vsizedlg.pas/vsizedlg.lfm) that created a class (not a form), that the form later used. I rewrote the code to remove initialization (and finalization) code. Now it no longer crashes!  %)

Could that be "while fpdebug is getting ready"?



Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 09:37:42 pm
Still not solved. Now it says it can not open:

"/Developer/lazarus_2_0/components/lazutils/lib/x86_64-darwin/laz2_xmlwrite.o"

which exists. :(

EDIT: I removed the initialization (and finalization) code from 3 other similar dialog units. For now lldb seems to be working again. Fingers crossed!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 03, 2018, 09:44:27 pm
As for the "unable to open file"

Could you open and edit components\fpdebug\fpimgreaderbase.pas
line 166 in
constructor TDbgFileLoader.Create(AFileName: String);
Code: Pascal  [Select][+][-]
  1.   FStream := TFileStreamUTF8.Create(AFileName, fmOpenRead or fmShareDenyNone);
  2.  

into
Code: Pascal  [Select][+][-]
  1.   try
  2.   FStream := TFileStreamUTF8.Create(AFileName, fmOpenRead or fmShareDenyNone);
  3.   except
  4.     DebugLN(['Error opening file ', AFileName, ' error: ', GetLastOSError]);
  5.   end;
  6.  
and add "LazLogger" to the uses clause.

No need to reraise the exception. Lets see what happens, if it is ignored.

--------------------------------------------
About the crash:

If I read your log correctly, then you restarted the IDE
Code: Pascal  [Select][+][-]
  1. LAZARUS END - cleaning up ...
  2. [FORMS.PP] ExceptionOccurred
  3.   Sender=EAccessViolation
  4.   Exception=Access violation
  5.   Stack trace:
  6.   $000000010022F261
  7.   $00000001000825DB
  8.   $0000000100078F93
  9.  
and there where plenty of errors then. But they will be from something that happened before.

The next crash is shortly after (but not during)   
TFpLldbDebugger.LoadDwarf

It is right before the debugger should sent the "process launch" command. That is the one we are trying to catch.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 11:31:42 pm
Ok, I made the edit (was in line 165). Deleted lazarus_2_0 units in case that helps. Then 'Clean up and Build'.

I'm getting the same unable to open warning dialog for "laz2_xmlutils.o" again.

BTW, I'm using "Dwarf with sets", that was default.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: VTwin on December 03, 2018, 11:36:52 pm
Code: Pascal  [Select][+][-]
  1.   //////////////////
  2.   ////FStream := TFileStreamUTF8.Create(AFileName, fmOpenRead or fmShareDenyNone);
  3.   try
  4.     FStream := TFileStreamUTF8.Create(AFileName, fmOpenRead or fmShareDenyNone);
  5.   except
  6.     DebugLN(['Error opening file ', AFileName, ' error: ', GetLastOSError]);
  7.   end;  
  8.   //////////////////
  9.  

Plus uses LazLogger.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 05, 2018, 02:57:40 am
Ok, I may have made progress on the too many file handles issue.  I found and tested where the linux loader opens its file. The Mac loader should do the same. The file(s) are no longer kept open.

Currently committed in trunk r59729.

If you could please test, and confirm if the problem with "unable to open file" is resolved?

For testing, set your file limit lower (not sure what else the IDE needs (project, units, lfm, lps, codetools may scan units that are not open...), but maybe try 30 to 50). Also do some debugging with this setting, to see if there are any issues while you inspect data (watches, locals, hints, opening new editor files (blue dots), ...)

Thanks for testing.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Arvur on December 06, 2018, 02:44:30 pm
It seems that unchecking "Use Application Bundle" in Project Options breaks debugger. Is this a known bug? Or should I check it deeper?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 06, 2018, 06:22:47 pm
It seems that unchecking "Use Application Bundle" in Project Options breaks debugger. Is this a known bug? Or should I check it deeper?
No, not known. Please provide a log.

How exactly does the "break" manifest? What error is displayed, or what feature stops working.

Does it work or break with a simple program? (Just one pas file)

Are you testing with:
RC2
fixes2_0 branch svn
trunk svn
?

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: carl_caulkett on December 07, 2018, 09:52:34 pm
I've installed Lazarus v2.0.0RC2 and have got the LLDB debugger working there. I've also installed separately v2.1.0 r59743 from SVN and built using the Cocoa widget set. I cannot find the option for LLDB in the options there. Is this expected behaviour?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: carl_caulkett on December 07, 2018, 10:00:33 pm
Sorry! My mistake! In my eagerness I forgot what I did last night and forgot to install the LazDebuggerFpLLdb package. Seems to be working fine now.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Arvur on December 10, 2018, 02:49:27 pm
It seems that unchecking "Use Application Bundle" in Project Options breaks debugger. Is this a known bug? Or should I check it deeper?
No, not known. Please provide a log.

I didn't manage to reproduce it on a simple project.

Lazarus RC2, Mac OS Mojave (10.14.1), dwarf2 debug info, cleanup and build before run.
No error when I turn off bundle option, it just shows me Assembler window. So I supposed debug info cannot be found.
But... If I press Run to continue, Assembler window stucks to the same address point (938B7284 : 8b 50 04 movl 0x4(%eax), %edx).
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 10, 2018, 03:23:59 pm
Try to get a log file (see 1st message of this thread)

Stepping, Running, Stopping, are done by lldb. If lldb can not handle that situation, then there is probably nothing that can be done to fix it.

If you do change between app-bundle / no app-bundle, then make sure the old app bundle is no longer there. (The same, if you had compiled with external debug info, and change to none external, ensure the old external info is deleted).
If you have such remains, the debugger (both lldb, which for that part is outside our control / and fpdebug in the IDE) may find the old left overs. And that may cause hick ups.
(lldb may (?) give warnings, and they may show in the logfile).

Please also note, that once compiled, you must not touch any of the .o files that are in the project's and packages' lib directories. The debugger (lldb and fpdebug) will read those. They must match what is in the exe.

You may be able to notice this, if you try to debug the IDE itself within the IDE.
If you build the IDE with extra options (Tools menu), and then open the ide/lazarus.lpi and select Run. The project may get compiled again, but without the final linking (because the IDE project is setup specially). So the .o files will then be newer than the exe. (But the debug info might still match, IF the opt level, or other code generating related flags (asserts, range checks ...) have NOT changed.
If you run the IDE from console (or tail the logfile) you should see warnings in this case.
But then this case should not happen with normal projects.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 11, 2018, 01:22:24 pm
You can also try to compile with -godwarfcpp
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 12, 2018, 11:44:47 pm
Could those on trunk please update to revision 59812

rev 59811: This enables capturing debug history.
 if you think it affects the speed of continuous stepping (repeated F7/F8), compile with -dLLDB_SKIP_IDLE
 if you have issues stepping (F8) over (ignored) exceptions, or stepping from raise to except/finally-block then try -dLLDB_SKIP_SNAP

Note: independent of this rev, stepping from raise to except/finally-block will always visually stop at the last statement before except. even though it is on except. (or finally)


rev 59812: some startup processing moved to thread. On very large apps this may lead to slightly faster start up.
  no side effects expected

Please let me know if those work well for you.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Arvur on December 18, 2018, 10:44:45 am
Try to get a log file (see 1st message of this thread)
Removed bundle, cleanup&build, run -> laz1.log
Added -godwarfcpp, cleanup&build, run -> laz2.log
In both cases breakpoints have ticks but Lazarus falls to Assembler window after run.

I can also provide successfull log with bundle option turned on. But it is 55Mb.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 18, 2018, 01:15:22 pm
I had a look at the log.

Your app stops with
   stop reason = EXC_BAD_ACCESS (code=1, address=0x4)
at  CoreFoundation  CFGetTypeID

called from
/Users/Administrator/Projects/Tokens/src/OCX.SYS/BSSHost/../../PACKAGES/BSSTOOLS/BSFiles.pas
BSFiles.pas
LINE: 1100
MOD: bssPluginHostTest
FUNC: GETINFOPLISTSTRIN

That would be a good starting point, to see if you can find if any address/object is nil (and should not be).


I don't know if lldb does something that triggers this, or if there is a bug in your app.

Shortly before the crash, the LCL (I assume) logged the following: "(V)TCarbonWSCustomForm.ShowHide Error: GetWindowClass  failed with result -5600"
I do not know if that is related (I don't know the cocoa, nor carbon workings; just the debugger)


---
The debugger should of course show the stop-reason. That needs to be fixed
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: fasdfasdfasdfasdfasdf on December 18, 2018, 01:29:55 pm
What is the status for LLDB on Windowze?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Arvur on December 18, 2018, 02:31:15 pm
FUNC: GETINFOPLISTSTRIN

That would be a good starting point, to see if you can find if any address/object is nil (and should not be).
You are right! It was error in sample from wiki (http://wiki.lazarus.freepascal.org/Mac_Show_Application_Title,_Version,_and_Company). CFBundleGetMainBundle is not good for checking bundle existance. I've described it in wiki.

It seems there are some other errors in project that prevent working without bundle... But I can trace this GetInfoPlistString routine successfully and no assembler window by now. Thanks a lot!!

Shortly before the crash, the LCL (I assume) logged the following: "(V)TCarbonWSCustomForm.ShowHide Error: GetWindowClass  failed with result -5600"
I do not know if that is related (I don't know the cocoa, nor carbon workings; just the debugger)
I suppose it is not related. The same lines are in log with bundle option turned on.

The debugger should of course show the stop-reason. That needs to be fixed
That would be great!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 18, 2018, 02:59:02 pm
The debugger should of course show the stop-reason. That needs to be fixed
That would be great!
Just committed. Should even make it into RC3
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 18, 2018, 03:09:30 pm
What is the status for LLDB on Windowze?
Same as on Mac.
Of course:
- you have to download lldb yourself.
- And you relay on lldb being stable (this may differ between OS)

On Linux, lldb keeps crashing (must be a problem with the build from the repro of my Linux (bit older). So may well work if you are all up to date, but I do not know.)
On Windows, I have tested the 32 bit version of LLDB, and it works. I have no idea about 64 bit on Win.

Mind the LLDB based debuggers in Lazarus are all more basic than the GDB based ones. So I am not sure why you want to use LLDB on Windows? What are you trying to fix?

On Windows (Linux too) you have a choice of other options.
- "GDB (with fpdebug)". Use fpdebug for data display
- Pure fpdebug. You should be at least on fixes2.0 (better trunk). On Linux it does not do well if your app uses threads. It is still beta, but "late beta". Works well in many cases.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: att2 on December 20, 2018, 02:38:31 pm
I would test LLDB on Linux if you tell me how-to.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 20, 2018, 03:35:23 pm
I would test LLDB on Linux if you tell me how-to.

Install lldb (e.g. from your distros package repository).

Then follow the instructions from the initial post of this thread (install packages, and config in Tools > Options > Debugger)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Igor Kokarev on December 24, 2018, 01:06:06 pm
I installed new Lazarus 2.0 RC3 on macOS Mojave.

New LLDB debugger is available by default. It works fine on first glance.

Also I can see values of WideString under debugger.

Thanks, Martin!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on December 24, 2018, 01:32:20 pm
I added "known issues" to my initial post on page 1.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: felipemdc on January 02, 2019, 09:48:34 am
Which command will build an IDE from zero with this lldb installed? Because I was using "make bigide" and it wasn't available, but maybe my revision was too old?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 02, 2019, 12:49:30 pm
Which command will build an IDE from zero with this lldb installed? Because I was using "make bigide" and it wasn't available, but maybe my revision was too old?

Afaik it should be part of bigide (but I have to test). But only in fixes 2.0 (probably past RC2) and in trunk (2.1).

Not a direct answer to your quesition: You can always install the package LazDebuggerFpLldb

Also once build bigide, check that the package is installed, to make sure the error (of its absence) is not something else/unexpected.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: felipemdc on January 03, 2019, 04:45:05 pm
Indeed, it seams to be available in "make bigide LCL_PLATFORM=cocoa"! I must have been using too old revisions. Thanks for the great work by the way!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 05, 2019, 06:05:52 pm
I added a testcase for "lldb with fpdebug".

Currently it only focuses on watch evaluations. Stepping, breakpoints and the like are not yet tested.

If you are
- on trunk, at/past revision 60012
- on fixes2.0, at/past revision 60013
then you can run it. (and it be nice if you do, please)

The test is in  components\lazdebuggers\lazdebuggerfplldb\test
TestFpLldb.lpi

Before you can run it you need
* CONFIG
copy and edit
  fpclist.txt.sample => fpclist.txt
  lldblist.txt.sample => lldblist.txt

Uncomment what you need. Edit the path to your fpc/lldb.
-You can ignore the version info.
-You can add/use one off (or leave away, which means both)
  bits=32
  bits=64
  bits=32,64
-For symbols=gw,gwset,gw3,gw4 you can use any flag that fpc recognizes for dwarf

-You can duplicate a section and extend it with (to pass extra param to fpc)
  opts=-O- -godwarfcpp
godwarfcpp is not fully supported

* DIRECTORIES
you may have to rename the folder testapps to lowercase
After you run tests, you should periodically delete any files in it, except for the "lib" folder, which must exist (as an empty folder)

You may create a "logs" folder.

If a file dialog opens, when you run the test, then it did not find the testapps/lib folder.

----------------------
Currently there is a known issue, with global variables from other units (other than the current paused line).

This leads to 61 fails in the "Scope" test. The test will list the number of fails at the end.

----------------------
Running the test may take some time. Between 10 and 30 minutes, depending on your system. Run it from a console, to see it is not stack (it prints dots to the console, from time to time)
Title: Re: New debugger for Mac based on lldb - "ignore exceptions" ineffective
Post by: kcandrews on January 28, 2019, 04:27:47 am
Hi Martin,

Following up from our late November conversation, I am still working on figuring out exception issues I encountered in my Mac OSX "big project" with the new LazDebuggerFpLldb 0.0 setup.  I am working with Lazarus 2.0.0RC3, FPC 3.04, X86_64-darwin-cocoa(alpha).

I note that you fixed incorrect class names which prepended: "2 digits at the start."  There is still something peculiar going on with exception class names for me -- apparently related to suffix characters appended to the end of the classnames.

Simple code from my project...

   if not assigned(Order)
   then
      raise ETradeException.Create('TTrade.GetSLPrice: No SL order!')
   else
      result := Order.Price;

In the IDE Options "Ignore these exceptions" list, I have added and checked the exception item "ETradeException".  When debugging, unexpectedly, I receive an exception dialog as a result of the "raise" in the code above.  This is true even when I uncheck "Notify on Exceptions" on the IDE Options dialog.  The interesting thing is that if I respond to the dialog saying that I want to "Ignore this exception type", that does work, but it also apparently is saving the exception with an additional suffix: "ETradeException\x10\v".  So when I now look at the revised "Ignore these exceptions" list, I see both the original "ETradeException" as well as the new one to be ignored: "ETradeException\x10\v".

Having no knowledge of the debugger internals, I don't know why the extra characters, but it appears they are causing a mismatch as the code tries to identify named exceptions.  Please let me know if I can explain better :-)

Thanks again for all your work on this!

always stopping on every raised exception -- even when the exceptions were set to be ignored.
Thanks for pointing out. That was a simple oversight when putting the new debugger together. The code for this is identical with what all other debuggers do.
I fixed this, and it will be in RC3.
Also fixed incorrect classnames for exceptions (the 2 digits at the start).
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 28, 2019, 04:58:13 am
Can you try to get a log, as described in the first messages of this thread?
That would really help. Thanks.

I may have an idea... (next 2 or 3 days)
But in case.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 28, 2019, 09:12:55 pm
Martin,

I sent you a link to the 80 Mg file (because of the 250K forum limit).

Hope this helps!
Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 29, 2019, 04:12:10 pm
Ok, I think I got it.

Please test with rev 60239 (trunk / merge to fixes is pending)
Or apply the changes from
https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/lazdebuggers/lazdebuggerlldb/lldbdebugger.pas?root=lazarus&r1=60239&r2=60238&pathrev=60239

There is probably NOT going to be another RC.  So you need to apply the changes yourself for testing
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 29, 2019, 06:22:44 pm
Martin,

Thanks so much!!  I applied your changes directly into the code in LibdbDebugger.pas, compiled and  rebuilt the package.

I know my log was probably big/complex enough to be a pain to search through, so I especially thank you for going through this.  Hopefully this will prevent similar exception name issues for others.

This seems to solve the Exception naming for me :-)

I'm doing more testing today...I am having instances of debug runs (without any breakpoints set) where things just hung and I had to actually kill and restart Lazarus.  As into, the application is a financial portfolio backtest system developed over many years (initially in Delphi and for the last 5 years in Lazarus).  So when I'm testing, I actually run tests on financial data analysis/technical indicators (stocks/futures/bonds) from 1935 to the current date.  The current test made it to 1980 but then hung.  (There are no issues like this without running under the lldb debugger and no issues when using gdb.)

I'll learn what I can from the laz.log but will probably need your help again, if you have time :-)

Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 29, 2019, 06:47:59 pm
I'm doing more testing today...I am having instances of debug runs (without any breakpoints set) where things just hung and I had to actually kill and restart Lazarus.

You could try to run the IDE in lldb. You need to build the IDE with -gw for this.
Open a terminal, and run (afaik)
Code: Pascal  [Select][+][-]
  1. lldb /path/to/lazarus/lazarus.app/Contents/MacOS/lazarus
Then enter r and return
and your IDE should run as normal. (after exit, you need to quit lldb: q)

That way you may get more info, should the IDE hang.
If lldb reports an error, then just enter: bt
and hopefully that gets a useful stack trace

If the IDE hangs, but lldb does not report, you should (IIRC) be able (in lldb / the terminal) to hit ctrl-c. You may have to select a diff thread after that. "threads list" and "t 1", "t 2", ...


If you run with --debug-log, delete the old logfile before each run. The IDE will keep appending....



I am not sure if you test RC3 or fixes branch.

There have been plenty of fixes since RC3. (including crashes in the debugger).
A list can be found here http://wiki.lazarus.freepascal.org/Lazarus_2.0_fixes_branch#Merged_revisions_for_2.0.0RC4 (note there likely will not be an RC4)


So it would be good to test on fixes branch. You can (most likely) download fixes, and replace some folders in your rc3, if you prefer. Folders needed to be copied:
components/fpdebug
components/lazdebuggers 
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 29, 2019, 07:22:09 pm
Thanks Martin.

Did as suggested compiling IDE with -gw...

So lldb said Lazarus "stopped":

Process 39997 stopped
* thread #14, stop reason = EXC_BAD_ACCESS (code=1, address=0x11d0c1000)
    frame #0: 0x000000010000cc44 Lazarus`SYSTEM_$$_DEFAULTANSI2UNICODEMOVE$PCHAR$WORD$UNICODESTRING$INT64 + 52
Lazarus`SYSTEM_$$_DEFAULTANSI2UNICODEMOVE$PCHAR$WORD$UNICODESTRING$INT64:
->  0x10000cc44 <+52>: movzbw (%rbx), %dx
    0x10000cc48 <+56>: movw   %dx, (%rcx)
    0x10000cc4b <+59>: addq   $0x1, %rbx
    0x10000cc4f <+63>: addq   $0x2, %rcx
Target 0: (Lazarus) stopped.

Here is the bt result:
(lldb) bt
* thread #14, stop reason = EXC_BAD_ACCESS (code=1, address=0x11d0c1000)
  * frame #0: 0x000000010000cc44 Lazarus`SYSTEM_$$_DEFAULTANSI2UNICODEMOVE$PCHAR$WORD$UNICODESTRING$INT64 + 52
    frame #1: 0x00000001000be445 Lazarus`CWSTRING_$$_ANSI2WIDEMOVE$PCHAR$WORD$WIDESTRING$INT64 + 197
    frame #2: 0x000000010000d05f Lazarus`fpc_ansistr_to_unicodestr + 111
    frame #3: 0x000000010000a92d Lazarus`SYSTEM_$$_ANSISTR_CONCAT_COMPLEX$RAWBYTESTRING$RAWBYTESTRING$RAWBYTESTRING$WORD + 133
    frame #4: 0x000000010000aa8c Lazarus`fpc_ansistr_concat + 220
    frame #5: 0x0000000100277ead Lazarus`DODEBUGLN(this=0x0000000103ff61c0, S="TTTTT data avail continue") at lazlogger.pas:730
    frame #6: 0x000000010013cc19 Lazarus`DEBUGLN(this=0x0000000103ff61c0, S="TTTTT data avail continue") at lazloggerbase.pas:897
    frame #7: 0x0000000100c69db8 Lazarus`EXECUTE(this=0x00000001134ec140) at debugprocess.pas:178
    frame #8: 0x00000001000d78c0 Lazarus`CLASSES_$$_THREADFUNC$POINTER$$INT64 + 272
    frame #9: 0x0000000100023f0c Lazarus`CTHREADS_$$_THREADMAIN$POINTER$$POINTER + 156
    frame #10: 0x00007fff79741661 libsystem_pthread.dylib`_pthread_body + 340
    frame #11: 0x00007fff7974150d libsystem_pthread.dylib`_pthread_start + 377
    frame #12: 0x00007fff79740bf9 libsystem_pthread.dylib`thread_start + 13
(lldb)

Not sure how to interpret this -- many years ago analyzed backtraces, but brain has lost this :-)

Thanks!

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Cyrax on January 29, 2019, 07:34:00 pm
It seems that debugprocess (or something else) is launched as separate thread, so there is need for critical section when outputting text to stdout (or for processing strings). Possible solution is to create a critical section during initialization of the lazlogger unit and using it around TLazLoggerFile.DoDebugLn method.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 29, 2019, 08:22:36 pm
LazLogger "transfers" the string to the main thread, which will write it to the logfile (TLazLoggerFileHandleMainThread).

Processing strings should be allowed within each thread.
So long as each thread uses its own strings (actually fpc has some provision for sharing strings, 2 threads can have separate references to the same string).

But there is indeed an issue.
FDebugIndent is a global var (a field of a global object). So all threads share the same reference. And that is not ok.

Please find the function:
TLazLoggerFile.CreateIndent
in
components\lazutils\lazlogger.pas
line 635

Comment out the entire function body. (or place an "exit" at the very start).
Lets see if that helps.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 30, 2019, 12:53:35 am
Martin,

Well done!!

I put the "exit" at the top of the LazLogger.CreateIndent function and the backtest is chugging away...without any issues so far (this will be a several hour test).  I did notice that setting a breakpoint "on the fly" while program was running in the lldb debugger caused an unknown class error -- but this is something I never "need" to do.

Also, KUDOS to those involved in making Cocoa work so well under Lazarus.  With the the your lldb debugger setup work and Cocoa (vs Carbon) allowing for easy 64-bit OSX applications, Lazarus will be well set to remain a great environment for all platforms :-)

Will report more later...
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 30, 2019, 01:43:45 am
I put the "exit" at the top of the LazLogger.CreateIndent function and the backtest is chugging away...without any issues so far (this will be a several hour test). 
I still need to get an official fix in. But the trace you delivered was gold. And errors like this are most often hell to find.

Quote
I did notice that setting a breakpoint "on the fly" while program was running in the lldb debugger caused an unknown class error -- but this is something I never "need" to do.
More details, please?

Error in the IDE,  or in your app?
Setting a breakpoint, normally involves pausing your app, setting the breakpoint and starting it again. But all that is done by lldb.

Quote
Also, KUDOS to those involved in making Cocoa work so well under Lazarus.
You can find them here. If you go further back there are others.
https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/lcl/interfaces/cocoa/?root=lazarus&view=log

Quote
With the the your lldb debugger setup work and Cocoa (vs Carbon) allowing for easy 64-bit OSX applications, Lazarus will be well set to remain a great environment for all platforms :-)

The lldb debugger is making big use of fpDebug. Which also (over the years) is the work of many. But yes, I did the final touch stringing it together with lldb for the Mac. ;)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 30, 2019, 01:57:05 am
Try with the changes from https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/lazutils/lazlogger.pas?root=lazarus&r1=60246&r2=60245&pathrev=60246

You can probably download the entire file (even it is from trunk), and replace it in your fixes or RC installation.


Lazlogger will need more fixes. But the way it is used in the IDE, this should be enough.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 31, 2019, 05:56:15 am
Martin,

Thanks again.  I did get the your latest file revision from the trunk and rebuilt the IDE.  (So I'm using RC3 but with the several debug related folders updated from the trunk.)  In a number of tests on my big project, due to your fixes, the debugger did not crash and works well with breakpoints I set.  As BigDan and possibly others have said (as you suggested probably needs an FPC fix), on OSX, stepping in/out after a breakpoint doesn't work and just takes me into the assembly code window (appearing to lose the source code "position"). 

But for now, it is great that the breakpoints work to successfully stop and allow me to view variable values.

Glad my backtrace was helpful -- from decades of consulting work often searching for those "hellish" bugs, I know how great it is when they reveal themselves.

All for now!  Thanks for all your work and help to so many :-)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 31, 2019, 08:36:04 pm
Hi Martin,

I do have a question re: "type of debug info" output when compiling a project.

For OSX 10.13.6 (High Sierra), Lazarus 2.0.0RC3, X86_64-darwin-cocoa (alpha) (with your current trunk debugger fixes for the LazDebuggerFPLldb 0.0 package)...

I do know that using gdb I was to set my "Project Options/Compiler Options/Debugging/Type of debug info" to "Dwarf with sets (-gw -godwarfsets).

BUT I am unclear on what "Type of debug info" is required for the new LazDebuggerFPLldb setup since I really don't know the various layers involved making the lldb debugging work (for OSX).  Earlier in this thread there is mention of  "-godwarfcpp" so I added that to my Complier Options/Custom Options -- but this feels like an overlap?

Just quite ignorant on all this...please advise when you have time :-)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 31, 2019, 09:03:09 pm
The lldb based debugger(s) need dwarf.

So basically, any dwarf info will do. So far I have not found any differences. (well a few below)

I do not know which versions of dwarf are supported by lldb. Or to which extend. On windows lldb warns, that it does not know all of the dwarf tags. But it still works. So no worry. Using "lldb + fpdebug", it does not matter which tags lldb knows....


Looking at it systematically:

Dwarf info is used for 2 purposes:
A) line info / and the ability to read the stack, and map it to functions and unit/lines)
B) symbol info, displaying watches, locals, ....

If you use lldb WITHOUT fpdebug, then LLDB must understand all of it. So the choice of which dwarf info may have effects on lldb.

The recommended "lldb with fpdebug", lldb mainly needs line-info (IIRC, lldb displays the stack, so the symbols shown as parms in the stackview require symbol info. If lldb has issues there then the various settings (below) may affect this.


fpdebug can use any of the dwarf settings.

And here are the dwarf settings, and any known effects they have on FpDebug. (I have no data, how they affect lldb (for the stacklist))

* Dwarf -gw  (this is actually dwarf2)
- Works fine, but you can not inspect "set of (...)"
- Symbols are shown all uppercased
- issues with strings (strings == pchar)

* Dwarf with sets
Same as above, but can do sets (sets are part of dwarf3, but included here)

* Dwarf3
- Works fine. I recommend this one (unless your lldb bails out, then check out if it works with any of the above)
- Symbols are uppper/lower case as in source
- strings are fine
- also may improve len detection for dynamic array
- widestrings are still pwidechar

*Fpc trung also has dwarf4 -gw4
I am not aware of where it actually adds different info. Not known if it does anything yet

* -godwarfcpp
This is an option that can be added to any of the above.
If or What it does may depend on the version of fpc, and the dwarf version chosen from the above.
- It may impact fpdebug (eg ability to differ between string and pchar)
- From the description I found (IIRC fpc mail list) it attempts to write more c-ish info, so lldb can better function.
  I am unaware if this impacts line info in anyway.
=> IF you do not have any problems without it, then do not use it.
=> Probably it will only do good, if you use lldb without fpdebug, fpc trunk, and dwarf3 or 4



For any of those debug infos there are currently hard coded upper limits for the size of string/array. See the initial post of this thread. This has other reasons.

All else is trial and error.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 31, 2019, 09:11:05 pm
Martin,

Great and thorough explanation on the compiler settings re: dwarf.  Great to have all that in one place.  Thank you.

And since I have your attention again, just curious that in my OSX setup, I can successfully use "Step Over" and "Step Into" IF I use gdb.  But stepping does not work with the new fp/lldb setup (for me).  I'm trying to wrap my head around why this could be an "FPC" problem.  Again, probably just ignorance and thank you for tolerating it :-)

FWIW, I attached a screen shot of the assembler window after a breakpoint successfully stopped.  But when I hit F7 (step) the program goes into some other code entirely (not showing any related source code nearby).  [see the assembler window after F7 in my next message]
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 31, 2019, 09:34:31 pm
About the stepping...
It is not certain where the problem is.

But it is either in fpc, or in gdb/lldb.

It has been seen on different platforms. On some with gdb and on others with lldb. Or maybe on some with both. 
Most of the info relies on reports from others. I have seen it myself with gdb on windows 64 bit (but all ok with 32 bit).

Pure fpdebug (Win/Linux) does not have the problem (as far as I can tell).

The IDE only send a command to gdb/lldb. And gdb/lldb then reports back where it ended the step.

It would require to go through the implementation of gdb or lldb, and see how they determine where the step ends.
This is currently not on by todo list.


Pure fpdebug for Mac is an option for the future.
Current state is: (That is if it works at all, there is code there, but it hadn't been tested in a long time)
1) stepping / breakpoint / pausing / control flow: Only support one thread. Behaviour in multi thread targets is unpredictable.
2) the entire IDE needs to be codesigned

As for 2: There is fpdebugserver, which comes with a small app, that acts as external debugger. Then only that app needs to be codesigned. But that has not been tested in at least as long.


The decision was for lldb now, because lldb is code signed by apple. So no steps to create your own cert and all.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on January 31, 2019, 10:18:12 pm
Martin,

Understand..thank you.  Since OSX has been and is becoming gdb unfriendly and lldb works there, the decision makes sense.

I'll try my same tests after shifting all my stuff & IDE back to 32 bit (Carbon) to see what stepping does there.

FWIW, I've attached a screen shot follow-up -- where my code goes (assembler-window-wise) after hitting F7 after the previous screen shot.

Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on January 31, 2019, 10:24:27 pm
I'll try my same tests after shifting all my stuff & IDE back to 32 bit (Carbon) to see what stepping does there.

You can build and debug 32 bit apps from a 64 bit IDE (and vice versa).

Though for debugging cross-bitness you do need svn fixes (or trunk). RC3 has still bugs preventing this.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 01, 2019, 04:49:11 pm
Martin,

If I wanted to take a look at the sending and receiving directives (for stepping) to lldb & gdb, what units are involved?

Thanks!
Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 01, 2019, 05:26:47 pm
The easiest way to see the communication is to open: menu View> Ide Internals > Debugger Output
On mac, you may need to resize the window, if the memo scrambles the text....

Compile the IDE with -dDBG_WITH_DEBUGGER_DEBUG and the window allows sending commands to lldb. However sending step commands can hang the debugger, as internal state is not kept for those commands.
This function is only tested with gdb, and was designed for data inspection. But you can try....


The 2nd way to watch, is to start the IDE from console and use the --debug-enable (from the start of this thread) => the output goes to console.
Or use --debug-log=file and tail the file
Actually you only need  --debug-enable=DBG_CMD_ECHO  the other flags show other info. (like state changes of the internal debugger object).


The code for dealing with stepping is in the package LazDebuggerLldb (it is the same with/without fpdebug)

unit LldbInstructions, contains mini-objects for each instruction and receiving the response.

unit LldbDebugger is in control of triggering them, and dealing with the result.

The IDE calls  TLldbDebugger.RequestCommand
which then calls TLldbDebugger.LldbStep
which creates a TLldbDebuggerCommandRunStep.Create object
This is queued and when the queue get to it, it calls TLldbDebuggerCommandRun.DoExecute
DoExecute will be called twice: First goes to DoInitialExecute, and then goes to TLldbDebuggerCommandRun.ResumeWithNextStepCommand
This creates TLldbInstructionProcessStep.Create and sets the finished handler to RunInstructionSucceeded
However the actual code flow resumes in TLldbDebuggerCommandRun.DoLineDataReceived


A lot of the code deals with what happens if you step over an exception.
If you do (eg step over a procedure that raises an exception) then the breakpoint at fpc_raise_exception is hit, the debugger will stop at that breakpoint, and the step is interrupted.
If the particular exception is ignored that is handled, and the IDE calculates where the step should have ended.


If there is no exception then it should read (in several calls to that method) the stop/stopped messages from lldb, and exit the command after setting the debuggerstate, and calling Finished.

------------------------
It may be possible to detect, if stepping ends inside a sub-function, and maybe get back to the main method in the same way this is done for exception (except you would not need to check for pop_exception, until you actually encounter an exception...



https://bugs.freepascal.org/view.php?id=34159

I don't know what tools there are on Mac for this. But if there is a possibility to strip away, or prevent certain sections of the debug info....

On windows it appears that this helps: objcopy.exe -R .debug_frame project1.exe project1n.exe

Removing a section called ".debug_frame"
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 01, 2019, 05:47:42 pm
Martin,
More than enough to go on :-)
Semi-retired, off to try out some new ski boots at Steamboat (Colorado), but then will slowly see what I can learn.
Thank you!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 01, 2019, 10:23:09 pm
Try the following:

FILE components\lazdebuggers\lazdebuggerlldb\lldbdebugger.pas
LINE 1065 (existing code)
Code: Pascal  [Select][+][-]
  1.     else
  2.     if FUnknownStopReason <> '' then begin
  3.       DoUnknownStopReason(FUnknownStopReason);
  4.     end
  5. // >>>>>>>>>>>>>>>>>>>> BEGIN NEW CODE
  6.     else
  7.     if StrContains(found[1], ' over') and (FMode = cmRun) and
  8.        (FStepAction = saOver) and (FFramePtrAtStart <> 0) and (frame < FFramePtrAtStart)
  9.     then begin
  10.       // SetDebuggerState(dsInternalPause);
  11.       ContinueRunning;
  12.     end
  13. // <<<<<<<<<<<<<<<<<<<<< END NEW CODE
  14.     else
  15.       SetDebuggerState(dsPause);
  16.  
  17.     if (FState = crRunning) then
  18.       exit;
  19.  

It may be that instead of "(frame < FFramePtrAtStart)" it needs to be "(frame > FFramePtrAtStart)"  ("<" vs ">")

I have not tested myself. Need to get out the test equipment....

--
I think/hope  "SetDebuggerState(dsInternalPause);" is not needed. But if it does not work, try with this line.




EDIT:

Just started some tests on gdb. It may not work that way, because the stepping stops before the frame(base)pointer (variable "frame") is changed.

That means the code probably must do a lot more.
It could detect if the stackpointer (rsp/esp) changed (got smaller). But that can happen without entering a procedure (if the code "push"es something onto the stack / this is rare in fpc, especially in none optimized code, but it has to be assumed possible).

Interestingly gdb actually shows a change in the framepointer..... Need to see what lldb does.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 02, 2019, 09:50:08 pm
Hi Martin,

While you've been working, me 2 a bit.  As I had suspected, based on the assembly window looking like maybe some other thread, I've been theorizing that the wrong thread (not the thread where my code is) is being selected on the step-in.  So I used your suggestion of compiling so I could intervene by sending commands to lldb directly.  It turns out if after a breakpoint, I give the lldb command "thread select 10" (my code thread), THEN when I do a step-in using the Lazarus IDE, I wind up at the next line of my source code as expected.

So I haven't looked at your latest post, but I'll do that too.  In the meantime, you may know where the "thread select" directive is either missing, or choosing the wrong thread?  From a little more step work, it appears that this "wrong" thread is regularly being selected -- since the wrong one is usually chosen when I just run to the next breakpoint, and try a step again (at which point the thread is no longer my #10).  But at that point if I give the "thread select 10" again, all works as expected. 

So all this leads me to believe that lldb itself is working correctly and that the directives to it from the IDE need to specify the correct thread. 

As info, other than to learn that gdb works for me both in generating 32 bit and 64 bit OSX applications, I've just been working with lldb since that's what I need to have working.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 02, 2019, 10:09:09 pm
That sound like the code that reads the lldb input after running, is not updating the debuggers (LazDebuggerLLdb) copy of the selected thread.

I will have a look.
Unfortunately to late for 2.0.  That is already tagged.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 02, 2019, 10:28:14 pm
Thanks...

In the meantime, I added your suggested new code to lldbdebugger but with no effect :-(

Also as info for what I am seeing.
On a line of code where I'm testing:
    FPrice := CleanPrice(GenericSymbol, sentprice);
If I breakpoint here, then set the thread via lldb command, step-in does take me to the "property" function (GetGenericSymbol) to get my Generic Symbol, and I can keep successfully stepping while I am within that function, BUT when I get to the end of that function and "step" out expecting to wind in in CleanPrice before the actual setting of FPrice, I note that the assembler code agains shows some other thread's info methinks, so from lldb's point of view, the correct thread is once again lost.

FWIW, the symptom is that quite regularly the source thread is "lost" to lldb :-)  but I guess I already said that.  I can beat dead horses without request.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 03, 2019, 05:05:08 pm
The code I posted was about "F8" not stepping over, but stepping in. But as I wrote later, it may not work. I have not yet been able to reproduce the issue with lldb, so I can not do much about it.
A logfile (with register and stack window open, so I can see values received for them in the log) might help... maybe/hopefully.

Once step-over paused incorrectly, the stack may be incorrect (hide some callers). If so, please describe the true stack. (top 3 frames)


Threads and stepping. (Still have to look into it)
A log would be welcome too...

To be sure I understand correctly.

1) You want to continue stepping, in the thread that was interrupted by the breakpoint?
But (unless you do changes in the thread window), stepping happens in a different thread? (main thread? or thread in which previous breakpoint occured?)

2) You stop at a breakpoint, change the thread to a different thread, that the one that hit the breakpoint, and then want to continue stepping in the thread that you switched too? But it steps in the thread of the breakpoint?


Also, when you reach the breakpoint, in the thread window, is the correct thread selected?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 03, 2019, 07:11:09 pm
Hi Martin,

I don't understand the internals of the Lazarus IDE or debugger code, but am trying to rebuild the Lazarus IDE in debug mode -- with the hope of doing some debugging through lazdebuggerlldb.pas -- though at the moment packages don't seem to be generating the debug output needed to stop at breakpoint source code lines.  (With your knowledge and experience, of course, your efficiency is superior but I thought I'd give this a shot since the OSX environment seems likely to be the issue for the laz lldb code?)

I will try to provide information re: the stack though I'm not sure how exactly to do that.  My last assembly language programming (mostly 8080 and 8086) was back in the 1980's and then moved onto "C/C++" and since I've mainly worked in applications-level programming since then, I rarely even think about stacks :-(

I will do what I can to put together a relevant logfile.

Quote
1) You want to continue stepping, in the thread that was interrupted by the breakpoint?
Yes
Quote
But (unless you do changes in the thread window), stepping happens in a different thread? (main thread? or thread in which previous breakpoint occured?)
Yes, using the capablilty you showed me of being able to send a command directly to lldb, after the breakpoint successfully works, I can select/set the source code thread.  Only then can I use F7 and/or F8 (from the IDE) successfully.  Not sure what you mean by the "main thread" (IDE or my source) but stepping does not occur in my source code thread -- shown by the assembler window not longer showing my source code as it did when the breakpoint occurred.  Without lldb intervention to select my thread, after doing an "IDE" step (F7 or F8), the assembler window shows some other section of code.  (I'll give you attachments to show this.)

Quote
2) You stop at a breakpoint, change the thread to a different thread, that the one that hit the breakpoint, and then want to continue stepping in the thread that you switched too? But it steps in the thread of the breakpoint?
If I don't force lldb to use the thread for my source code, then and F7 and F8 IDE commands cause stepping in what appears to be some other thread.

Quote
Also, when you reach the breakpoint, in the thread window, is the correct thread selected?
Haven't looked at the "thread window" -- but the assembler window at the breakpoint stop is correct (showing my source code lines).  However, unless I then reset the thread to the source code thread directly sending the "select thread" command to lldb, any stepping (F7 or F8) does NOT step through the source code -- and appears to step through some other thread (though if I F9) things proceed again to the next breakpoint.

I'm sure this will become clearer once I get more info/logs about this to you.  I hope to get somewhere with that today :-)

Thanks for your time and patience with me.  I have a lot of coding/architecture/consulting history but as you can tell, I don't know the Lazarus internals landscape at all.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 03, 2019, 08:53:51 pm
you missed inserting [ quote ] ... [ /quote ] tags...

I don't understand the internals of the Lazarus IDE or debugger code, but am trying to rebuild the Lazarus IDE in debug mode -- with the hope of doing some debugging through lazdebuggerlldb.pas -- though at the moment packages don't seem to be generating the debug output needed to stop at breakpoint source code lines.
In many cases it is enough, to open menu: Tools > Configure build Lazarus => There you can add options for compiling the IDE. You can use -gw3  (dwarf3). (Should that give unexpected trouble (errors produced by lldb), then use only -gw (dwarf2)).
Also add -O-  to disable optimizations, which could interfere with the debugger.
A good set of options:
Code: [Select]
-gw3 -O- -gh -gt -Criot
Not all package follow this. You can open individual packages from the menu "Package". (If you open LazDebuggerFPLldb first, then you can open LazDebuggerLldb from that package / might be quicker).
Each package has a button "Options". There you can set any options you want.
Both the LazDebugger...Lldb packages should have "$(IDEBuildOptions)" in the "costum options" section of their options. This means they follow the global options.

While usually not needed, if in doubt, in "Configure build Lazarus" choose the "clean all" option.


To debug the IDE, inside another IDE:
- Open the project (inside the lazarus install folder) ide/lazarus.lpi
- Press F9 to run
This will not correctly recompile changes that you made (even if it looks like). If you made changes always recompile from the Tools menu.

If you debug the IDE a lot, open the project (ide/lazarus.lpi), go to menu Run > Run Params, and add as "command line parameters"
  --primary-config-path=/path/to/empty/folder
This allows you to have a separate global config for the debugged IDE.
You can also add any --debug-enable, that you need for logging.


Quote
  (With your knowledge and experience, of course, your efficiency is superior but I thought I'd give this a shot since the OSX environment seems likely to be the issue for the laz lldb code?)
The LLDB debugger was written only for the OSX environment. All other OS are better of with GDB.
Yet I myself am a Windows/Linux user. So most of my development happens on those platforms. I have access to a MAC, which I use for dedicated testing.
But since I do all my normal debugging on Win/Linux, the testing of LLDB on Mac from my side is limited to a few dedicated test runs.

So for stress testing, I have to relay on feedback. And extra help is always welcome.

Quote
I will try to provide information re: the stack though I'm not sure how exactly to do that. 
If you run a logfile --debug-log as specified in the very first post of this thread, then the data will be included in the logfile. All you need to do, is to keep the IDE's stack window open, so the IDE will need the data, and get it from lldb.
The same applies to the register window (from menu View > Debug Windows). Just leave it open.

The stack window has a copy button, to copy its content, but that will not be needed.

What I meant is:
- You may be executing code in FormCreate (or any function / Since you step through it, you know the functions name)
- It makes a call like "Top := 7", you STEP-INTO it.
You are now in TControl.SetTop function. You do know it was called from FormCreate.
But if you look at the stack, it will show
  SetTop
  SomethingInTheLCL
The function FormCreate is missing from the list. Shoud that case happen, then simply say so.

Quote
My last assembly language programming (mostly 8080 and 8086) was back in the 1980's and then moved onto "C/C++" and since I've mainly worked in applications-level programming since then, I rarely even think about stacks :-(
Well all I mean is the high level representation of the Stack. Just look at the Stack window.

Quote
I will do what I can to put together a relevant logfile.

Quote
1) You want to continue stepping, in the thread that was interrupted by the breakpoint?

Yes
Quote
But (unless you do changes in the thread window), stepping happens in a different thread? (main thread? or thread in which previous breakpoint occured?)

Yes, using the capablilty you showed me of being able to send a command directly to lldb, after the breakpoint successfully works, I can select/set the source code thread.  Only then can I use F7 and/or F8 (from the IDE) successfully.  Not sure what you mean by the "main thread" (IDE or my source) but stepping does not occur in my source code thread -- shown by the assembler window not longer showing my source code as it did when the breakpoint occurred.  Without lldb intervention to select my thread, after doing an "IDE" step (F7 or F8), the assembler window shows some other section of code.  (I'll give you attachments to show this.)
So you send for example
Code: [Select]
thread select 5 to the debugger.

And then you press F7 which sends "thread step-in"

Quote

Quote
Also, when you reach the breakpoint, in the thread window, is the correct thread selected?

Haven't looked at the "thread window" -- but the assembler window at the breakpoint stop is correct (showing my source code lines).  However, unless I then reset the thread to the source code thread directly sending the "select thread" command to lldb, any stepping (F7 or F8) does NOT step through the source code -- and appears to step through some other thread (though if I F9) things proceed again to the next breakpoint.
Ok that answers the question I did ask 2 lines above....

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 03, 2019, 09:13:59 pm
Your previous answer already gave me some idea where to look.

Please try with the following patch
Code: Diff  [Select][+][-]
  1. Index: components/lazdebuggers/lazdebuggerlldb/lldbdebugger.pas
  2. ===================================================================
  3. --- components/lazdebuggers/lazdebuggerlldb/lldbdebugger.pas    (revision 60312)
  4. +++ components/lazdebuggers/lazdebuggerlldb/lldbdebugger.pas    (working copy)
  5. @@ -1175,7 +1175,7 @@
  6.    else begin
  7.      Debugger.FCatchesBreak.Disable;
  8.      Debugger.FPopExceptStack.Disable;
  9. -    Instr := TLldbInstructionProcessStep.Create(FNextStepAction);
  10. +    Instr := TLldbInstructionProcessStep.Create(FNextStepAction, Debugger.FCurrentThreadId);
  11.    end;
  12.    Instr.OnFinish := @RunInstructionSucceeded;
  13.    QueueInstruction(Instr);
  14. Index: components/lazdebuggers/lazdebuggerlldb/lldbinstructions.pas
  15. ===================================================================
  16. --- components/lazdebuggers/lazdebuggerlldb/lldbinstructions.pas        (revision 60312)
  17. +++ components/lazdebuggers/lazdebuggerlldb/lldbinstructions.pas        (working copy)
  18. @@ -133,7 +133,7 @@
  19.    protected
  20.      function ProcessInputFromDbg(const AData: String): Boolean; override;
  21.    public
  22. -    constructor Create(AStepAction: TLldbInstructionProcessStepAction);
  23. +    constructor Create(AStepAction: TLldbInstructionProcessStepAction; AThread: Integer = -1);
  24.    end;
  25.  
  26.    { TLldbInstructionProcessKill }
  27. @@ -713,15 +713,24 @@
  28.  end;
  29.  
  30.  constructor TLldbInstructionProcessStep.Create(
  31. -  AStepAction: TLldbInstructionProcessStepAction);
  32. +  AStepAction: TLldbInstructionProcessStepAction; AThread: Integer);
  33.  begin
  34.    case AStepAction of
  35.         saContinue: inherited Create('process continue');
  36. -    saOver: inherited Create('thread step-over');
  37. -       saInto: inherited Create('thread step-in');
  38. -    saOut: inherited Create('thread step-out');
  39. -    saInsIn: inherited Create ('thread step-inst');
  40. -    saInsOver: inherited Create('thread step-inst-over');
  41. +  (*
  42. +    saOver: inherited Create('thread step-over', AThread);
  43. +       saInto: inherited Create('thread step-in', AThread);
  44. +    saOut: inherited Create('thread step-out', AThread);
  45. +    saInsIn: inherited Create ('thread step-inst', AThread);
  46. +    saInsOver: inherited Create('thread step-inst-over', AThread);
  47. +  // *)
  48. +  //(*
  49. +    saOver: inherited Create(Format('thread step-over %d', [AThread]));
  50. +       saInto: inherited Create(Format('thread step-in %d', [AThread]));
  51. +    saOut: inherited Create(Format('thread step-out %d', [AThread]));
  52. +    saInsIn: inherited Create (Format('thread step-inst %d', [AThread]));
  53. +    saInsOver: inherited Create(Format('thread step-inst-over %d', [AThread]));
  54. +  // *)
  55.    end;
  56.  end;
  57.  
  58.  

In lldbinstructions.pas please also try the commented out code (and comment the active added code).


Background:
The InstructionQueue keeps track of the selected thread (and stackframe). If a command requests to run in a specific thread then it will be selected, unless already done.

Some commands do not request a specific thread. Eg listing all threads gets the same result, never mind the selected thread.
Run F9 runs all thread, so it should not care either.

But step commands need to care. Yet by accident they did not request a specific thread.

So depending on what the debugger did before, and what thread got selected, step command mis-function.




If that does not help, then the next point of interest is at line 1030 (lldebugger.pas)
Code: Pascal  [Select][+][-]
  1.     ParseNewThreadLocation(ALine, AnId, AnIsCurrent, Name, AnAddr,
  2.       Stack, Frame, AFuncName, AnArgs, AFile, AFullFile, SrcLine, AReminder);
  3.     AnArgs.Free;
  4.  
  5.     Debugger.FCurrentThreadId := AnId;
  6.  
Does the debugger get the correct ID for FCurrentThreadId

Add a
debugln(['**** Got FCurrentThreadId ', AnId]);
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 03, 2019, 10:55:30 pm
The patch worked!  Not passing the threadid to the stepping definitely was the main issue.  Stepping now works for me...I didn't need to go to the next point of interest -- line 1030.

Will report as I test...

I have two attachments as a result of running the IDE from lldb and then debugging out of breakpoint and "achieving" a crash or two.  Each has the tail end of the lldb output followed by a "bt."

1. "Step_Crash" -- may be an lldb problem...see attached file and:
https://github.com/WebFreak001/code-debug/issues/106
https://github.com/rust-lang/rust/issues/29154

(lldb) bt
error: need to add support for DW_TAG_base_type 'FormalDef' encoded with DW_ATE = 0x7, bit_size = 0

**** UPDATE: this may not be a notable issue -- running IDE not from within lldb does not show problem -- did a lot of successful stepping f7 & f8

2. "LLDB_Pause" -- got this to happen by hitting IDE "pause" while debugging beyond a breakpointed section (but running):

    >> Running Instruction:
2019-02-03 15:35:23.989430-0700 Lazarus[18768:4189792] [General] An uncaught exception was raised
2019-02-03 15:35:23.989490-0700 Lazarus[18768:4189792] [General] *** -[NSBigMutableString replaceCharactersInRange:withString:]: nil argument

** UPDATE: for me, I rarely need to interrupt a debug session with "Pause" unless I've created an inadvertent "forever loop" (rare)

Summary: Congrats, Martin.  With the exception suffix and prefix fixes and fixing the step threadID, debugging is functional on my OSX.  I've been delaying going to OSX Mojave 10.14 until laz debugging with lldb works -- and now it does -- at least under OSX 10.13, High Sierra!  Thank you.

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 05, 2019, 04:42:26 am
OSX Mojave upgrade from High Sierra 10.13.6 report with hope of using new lldb debugger setup:

[EDIT/UPDATE] 2/8/19 - SUCCESS -- not sure of cause but starting out this time with Lazarus 2.0.0 and "fixed" lldb stuff, after OSX upgrade, all is working as it was in High Sierra, but now in Mojave 10.14.3 :-)

1. Performed standard Mojave OSX upgrade from Apple App Store and updated to 10.14.3 Mojave using new System Preferences/Software Update.

2. Upgraded/installed xcode command line tools using terminal command: "xcode-select --install"  Note Xcode is now 10.1 version.

3. Installed FPC 3.04a  FWIW

4. (2/8/19) needed to manually add line to /etc/fpc.cfg to avoid IDE build problem on link due to not finding library crt10.5.1
Quote
-XR/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk.

https://forum.lazarus.freepascal.org/index.php/topic,42657.msg300589.html#msg300589

Thank you Hansaplast!

Lazarus (Cocoa/x86_64) itself worked fine, it and packages could be rebuilt with no issues, big project (also Cocoa/x86_64) compiled and ran outside of the debugger as in High Sierra.  [UPDATE 2/8/19] Lldb debugger setup works as it does in High Sierra -- totally functional for breakpoints, stepping, and previous exception issues (thanks Martin!).

Finally, attempted to debug using lldb (as worked before in High Sierra earlier today).  Was able to get first breakpoint to stop code, but any debug commands F7/F8 stepping, F9 restart-run, failed to get code to proceed (a lot of UI delays occurring with constant spinning wheel icon (processor busy).  Had to kill project executable and Lazarus IDE executable.  Noted that lldb version is lldb-1000.0.38.2 Swift-4.2 -- SAME as in High Sierra 10.13.6.

Tried tinkering with csrutil (OSX System Integrity Protection) which had under High Sierra which had been set to "csrutil enable --without debug" but to no avail.

Oh well, for now, going back to High Sierra :-(  Would love to hear if anybody has succeeded in Lazarus debugging using the new lldb setup under OSX Mojave (along with any upgrade steps I missed).
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 05, 2019, 06:07:34 am
The "pause" crash happens in the TMemo code. (between LCL and cocoa). So I guess LCL cocoa still as an issue....

Keep the "Internals" > "Debug Output" window closed, and this should go away.

--
It is also possible that this is an issue in cocoa (similar stuff can happen under linux with certain x-servers).

The text to be inserted was "\\x17EBarDataAccessException?\"
That means that lldb had just paused your app at an exception.

1) In gtk (just example), if an app is paused in an event it received from gtk (mouse/key/paint/...), then x-server can hang.
Who knows if there is something similar.

2) You said you just hit pause. Maybe an issue in lldb, (sending pause and at the same time hitting a breakpoint (raise exception is a breakpoint)).
Though that would not explain an error in cocoa memo code....


-------------------
Btw: future logs, if possible please as text file (pfd looses linefeeds).
You can zip,rar,tar.gz

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 05, 2019, 06:14:38 am
==EDIT ==
I have an idea in the meantime. So probably no need for the extra log.
<< == END EDIT ==


The step crash... That looks like an issue in the IDE debugger code... But have not yet figured it fully.

Can you check a few things.

When you start the IDE, check the commandline
Code: [Select]
--debug-enable=DBG_CMD_ECHO,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_DEBUG,FPDBG_DWARF_ERRORS,FPDBG_DWARF_WARNINGS,FPDBG_DWARF_VERBOSE_LOAD,FPDBG_DWARF_DATA_WARNINGS,DBG_VERBOSE,DBG_WARNINGS,DBG_STATE,DBG_EVENTS,DBG_THREAD_AND_FRAME
It looks like the DBG_CMD_ECHO part may be missing, or maybe have missing/xtra chars.
The relevant data is not in the log. (But other part of the debug output was)
Check that the log has lines like/containig:
   >> >> TDebugProcess.SendCmdLn


Can you compile your IDE with range checking please? (Tools > Configure build Lazarus : add to options: -Cr

I will try further to reproduce. I can see what causes it, but not yet how....

--- EDIT
can you apply this path for more debug info please

Code: Diff  [Select][+][-]
  1. Index: fpdmemorytools.pas
  2. ===================================================================
  3. --- fpdmemorytools.pas  (revision 60335)
  4. +++ fpdmemorytools.pas  (working copy)
  5. @@ -26,7 +26,7 @@
  6.  
  7.  uses
  8.    Classes, SysUtils, math, DbgIntfBaseTypes, FpErrorMessages, LazClasses,
  9. -  Laz_AVL_Tree;
  10. +  Laz_AVL_Tree, LazLoggerBase;
  11.  
  12.  type
  13.  
  14. @@ -638,6 +638,7 @@
  15.  constructor TFpDbgMemCacheSimple.Create(ACacheAddress: TDBGPtr;
  16.    ACacheSize: Cardinal);
  17.  begin
  18. +DebugLn(['!!! CreateCache ',ACacheAddress, ', ',ACacheSize]);
  19.    FCacheAddress := ACacheAddress;
  20.    FCacheSize := ACacheSize;
  21.  end;
  22. @@ -646,6 +647,7 @@
  23.    ): Boolean;
  24.  begin
  25.    Result := (AnAddress >= FCacheAddress) or (AnAddress + ASize <= FCacheAddress + FCacheSize);
  26. +debugln(['!!! HasMem ',FCacheAddress, ', ',FCacheSize, ' Mem ',AnAddress, ', ',ASize]);
  27.  end;
  28.  
  29.  function TFpDbgMemCacheSimple.ReadMemory(AnAddress: TDbgPtr; ASize: Cardinal;
  30. @@ -656,6 +658,7 @@
  31.       FFailed
  32.    then
  33.      exit;
  34. +debugln(['!!! ReadMem ',FCacheAddress, ', ',FCacheSize, ' Mem ',AnAddress, ', ',ASize, ' ', FMem=nil, ' ', Length(FMem)]);
  35.  
  36.    if FMem = nil then begin
  37.      SetLength(FMem, FCacheSize);
  38.  

For this bug, I only need a new log, no new backtrace. So you can start the IDE without surrounding lldb.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 05, 2019, 06:16:03 am
Quote
Mojave
Definetely need to see a log. Check the DBG_CMD_ECHO as it is important in this case.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 05, 2019, 06:38:21 pm
The "step crash" issue: Please test with rev 60338.

You can download a copy of the  changed file: https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/fpdebug/fpdmemorytools.pas?view=markup&root=lazarus&pathrev=60338
And replace the entire file (should work for trunk and 2.0.0)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 06, 2019, 06:53:33 pm
Hi Martin,

As info, now using laz 2.0.0 as base but replaced directories fpdebug, lazdebuggers, and lazutils with "working" ones from our recent work.

Quote
The "step crash" issue: Please test with rev 60338.
Replaced the file as suggested, and have no problems with stepping f7 & f8 -- but as noted earlier, even before your 60338 change, I actually couldn't reproduce the "step crash" unless running under the laz IDE under lldb.
Quote
Can you compile your IDE with range checking please? (Tools > Configure build Lazarus : add to options: -Cr
I know you didn't need me to do this in the end but...Interesting that in my setup, Lazarus IDE won't start up successfully with the range check option "Cr" set in the IDE build configuration options.  See the first attachment laz.log generated from this terminal command:
 
lldb /Developer/lazarus/lazarus.app/Contents/MacOS/Lazarus -- --debug-enable=DBG_CMD_ECHO,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_DEBUG,FPDBG_DWARF_ERRORS,FPDBG_DWARF_WARNINGS,FPDBG_DWARF_VERBOSE_LOAD,FPDBG_DWARF_DATA_WARNINGS,DBG_VERBOSE,DBG_WARNINGS,DBG_STATE,DBG_EVENTS,DBG_THREAD_AND_FRAME --debug-log=/Developer/lazarus/laz.log

On the IDE "pause" during debugging, I will try to generate a useful log file.  I am not having a crash, but "pause" puts the debugging into a state where I can only "stop" and "reset debugger" to regain control.  Not sure exactly what pause is doing but if I recall correctly, under gdb, "pause" allows for "show execution point" and f9 run to be available.  (Not so with lldb in my setup.)

On Mojave problems, having only one machine I am working with, I have reverted back to High Sierra, and will probably wait until someone else is able to do similar testing under that OS.  (It is fairly painful to go to a new OSX version and then have to go back to previous one -- requiring replacing the entire disk image.)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 06, 2019, 07:24:54 pm
Quote
The "step crash" issue: Please test with rev 60338.
but as noted earlier, even before your 60338 change, I actually couldn't reproduce the "step crash" unless running under the laz IDE under lldb.
This probably occured while stepping into, and therefore pausing on a "begin" line of a procedure/function.
The function needs at least 2 params/locals. One of them a local string var. And then some bad luck. And it is 64bit only

Code: Pascal  [Select][+][-]
  1. var a: integer; b: ansistring;
  2. a:= 1;
  3. pointer(b):= pointer(high(qword));
  4. a:=a;// break on this line
watch a and b (a as first entry in watches). That should likely cause the crash.


Quote
Quote
Can you compile your IDE with range checking please? (Tools > Configure build Lazarus : add to options: -Cr
I know you didn't need me to do this in the end but...Interesting that in my setup, Lazarus IDE won't start up successfully with the range check option "Cr" set in the IDE build configuration options.
So the cocoa code as a range check issue....
Unfortunately fpc for Mac does not resolve the stacktraces. So you need to run in the debugger (breakpoint on fpc_raise_exception).

But this is unrelated. Feel free to bug report it though.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 06, 2019, 09:31:40 pm
So probably of little interest but FWIW re:
Quote
This probably occured while stepping into, and therefore pausing on a "begin" line of a procedure/function.
The function needs at least 2 params/locals. One of them a local string var. And then some bad luck. And it is 64bit only

Code: Pascal  [Select]
var a: integer; b: ansistring;
a:= 1;
pointer(b):= pointer(high(qword));
a:=a;// break on this line
watch a and b (a as first entry in watches). That should likely cause the crash.
In my main form, just as a test I created a new procedure TestMe which I then called at the start of FormCreate:
Code: [Select]
procedure TfrmSystemRulesMain.TestMe;
var
   a: integer; b: ansistring;
begin
a:= 1;
pointer(b):= pointer(high(qword));
a:=a;// break on this line
end;

procedure TfrmSystemRulesMain.FormCreate(Sender: TObject);
var
   i: integer;
   o: TOrderType;
   e: TEvaluator;
   Userdir: string;
   TechRules: TTechRules;
begin
   TestMe;
   ...
I successfully stepped (F7) into and within the TestMe procedure, but on f7 stepping out of the TestMe procedure an exception is raised:
 "raised exception class: 'Debugger stopped with reason: EXC_BAD_ACCESS (code=1,address=0xffffffffef)'

Commenting out the line "pointer(b):= pointer(high(qword));" eliminates the exception.

Also possibly interesting is that doing a try/except around the line  "pointer(b):= pointer(high(qword));" fails to trap the exception.  Try/except around TestMe procedure doesn't trap it either.

Relevant? (from https://www.freepascal.org/docs-html/rtl/system/high.html):
"The return type is always the same type as the type of the argument (This can lead to some nasty surprises !)."

UNRELATED...On the IDE "pause" during debugging problem I had, I did a very simple project (not my big multi-threaded one).  Use the little project, an IDE "Pause" successfully stops the debugger as it did in gdb.  So I can have a "forever loop" in the source code, and hitting "pause" takes me to a stop in the source code within the loop as expected.  Even while in the windows event loop (not executing my source code since waiting for a button click), even though "pause" can't find a source line (as expected), I can hit run/f9 and things start up again as expected.

So I suspect my debug "pause" problem may be that my big project is multi-threaded and perhaps the lldb code doesn't know the threadid (perhaps like in the earlier "step" issue?) to successfully pause in the source code.  I believe gdb handled this -- but I will verify.
UPDATE: gdb does NOT pause successfully in my multi-threaded program either!  (Tested both 32 bit/carbon and 64 bit/cocoa)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 06, 2019, 10:55:36 pm
Quote
Commenting out the line "pointer(b):= pointer(high(qword));" eliminates the exception.
Yes this line of code intentionally and seriously screws up the string. And it is expected that the test app, does no longer work thereafter.

The point is, the debugger should not crash.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 08, 2019, 06:17:20 pm
As info, starting from 2.0.0 Lazarus base with your fixes, today I successfully upgraded to Mojave as a working environment.  Lldb debugger works as it did in High Sierra with fully functional breakpoints, exception "prefix/suffix name-mangling" fixes, and stepping thread-id fixes.  Thanks again for all your work on this!

Steps involved:

1. Performed standard Mojave OSX upgrade from Apple App Store and updated to 10.14.3 Mojave using new System Preferences/Software Update.

2. Upgraded/installed xcode command line tools using terminal command: "xcode-select --install"  Note Xcode is now 10.1 version.

3. Installed FPC 3.04a  to automatically cover needed library path for /usr/lib/crt...
https://forum.lazarus.freepascal.org/index.php/topic,42657.msg300589.html#msg300589

Keith
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 10, 2019, 09:53:47 pm
Martin,
I hadn't really checked out debug "watches" in my multi-threaded app.  Now I have and I discovered that local variables in a function, though they show the correct values when "moused over" (possibly important clue - I will look for where that's done), a watch item existing in the new thread's code, once in the new thread, reveals irrelevant data, apparently from another thread or frame.  Watches displayed under a single threaded app work as expected.  (Also, FWIW, watches work fine in gdb for multi or single-threaded apps).

Also, FWIW, the "arrowed" current thread in the debug/threads window, starts at threadID #1 before I initiate a new thread (proven by breakpoint at start of program).  After a new thread starts up and a breakpoint in that thread #12 successfully stops, the threads window still shows thread ID #1 "selected" (arrow) even though I can also see that the code is breakpointed in threadid #12.  Accordingly the local "str: string" variable which I set in the same thread #12 just before my second breakpoint displays incorrectly.

So, copying your approach with what you did in fixing the step issue, I modified the TLldbInstructionWatchSet.Create in LldbInstructions.pas -- see {kca}:
Code: [Select]
constructor TLldbInstructionWatchSet.Create(AWatch: String;
  AKind: TDBGWatchPointKind
  ; {kca} AThread: Integer = -1; AFrame: Integer = -1);
begin
  case AKind of
  wpkWrite:     inherited Create(Format('watchpoint set variable -w write %s', [AWatch]), {kca}AThread, AFrame);
    wpkRead:      inherited Create(Format('watchpoint set variable -w read %s', [AWatch]),{kca}AThread, AFrame);
    wpkReadWrite: inherited Create(Format('watchpoint set variable -w read_write %s', [AWatch]),{kca}AThread, AFrame);
  end;
end;
And in LldbDebugger.pas, I modified TLLdbBreakpoint.SetBreakPoint adding this under bpkData:
Code: [Select]
    bpkData: begin
      if not Enabled then // do not set, if not enabled
        exit;
      // TODO: scope
      // TODO: apply , Expression, not Enabled
      Instr := TLldbInstructionWatchSet.Create(WatchData, WatchKind,
      {kca} TLldbDebugger(Debugger).FCurrentThreadId, TLldbDebugger(Debugger).FCurrentStackFrame);
      if Expression <> '' then
        FNeededChanges := FNeededChanges + [ciCondition];
    end;

Actually, initially I tried just adding the threadid parameter, but since that didn't work, I added the additional stackframe parameter.  Unfortunately, this didn't solve the problem -- but then I'm pretty much winging it :-)

Any ideas?


Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 10, 2019, 11:02:55 pm
TLldbInstructionWatchSet is setting a watchpoint. And yes it is broken, but not the issue here. Since a watchpoint is based a watch, it needs thread and frame.

TLldbInstructionExpression is for eval of watches. Well in the NONE FpDebug version. And that is given the thread and frame.

In the LLDB-FpDebug code, it is TLldbDebuggerCommandRegister, and that too is given thread and frame.
It also is TLldbInstructionMemory, but that should not need thread or frame. The memory at a given address is the same for all of them. (Well I have to check if that is true for threadvar)
The address of local vars, is determined by certain register. So if they match the frame, it should be fine.

If you watch the debug output, you will see thread and frame commands issued:
Code: Pascal  [Select][+][-]
  1. >> thread select 4
  2. (lldb) thread select 4
  3. >> frame select 3
  4. * thread #4: tid=0x2e38: 0x0042b81b, 0x07A4FEE8, 0x07A4FF00 &&//FULL: \tmp\unit1.pas &&//SHORT: unit1.pas &&//LINE: 93 &&//MOD: project1.exe &&//FUNC: EXECUTE(this=0x048a6ac8) <<&&//FRAME, stop reason = breakpoint 6.1
  5. >> register read --all
They will not always be right before the register command. THe debugger knows what it last did, and if it already is on a thread the command is not needed.



But while debugging this I observed something going wrong.

I get pause at thread 3, frame 0
I switch to thread 1, frame 2
I switch back to thread 3 (which still is at frame 0)

Yet the debugger issues
thread 3
frame 2 

So it did not reset the frame when switching threads.

Please observe if that explains when things go wrong, otherwise observe what the debugger sends.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 11, 2019, 12:52:39 am
Martin,
Sorry!  Misunderstood what I was changing -- what I get for "winging it."

Anyway, I somewhat follow you but don't understand the territory very well.

I have attached the debug output from me running my program into the start of the "DoTest" procedure within the new thread.  I looked for through the debug output for "thread select" and "frame select"-- but not sure if I see any issues.  It does seem strange that the debugger goes back to "thread select 1" after "thread select 8" (the new thread).  FWIW, see the attached.  The second attachment is the "Threads window" at the time of the breakpoint stop in my DoTest procedure within the new thread 8.  Do note that the green arrow is on thread one FWIW.

If I look at local variables within DoTest in the "watches" window, they show nonsense values (like the frame or thread is wrong).  However, strangely I can "cursor over" any of these same local variables and see valid values.

Related code where I breakpointed (end of debug output) on the try statement after setting str := 'frogman'.  Again cursor hovering over str in the source code gets me "str = frogman" (strange that should work) while the watch window shows str with a nonsense value:
Code: [Select]
procedure TBacktest.DoTest(SentIndex: integer; SentSingleRun: Boolean);
var
   i, CompleteCount: integer;
   valuelo, valuehi: double;
   str: string;
begin
   str := 'frogman';
try // try finally
try // try except
with TradeSession
do begin             
...

I may need to write a simple threaded program to be more helpful.  My current program uses MTProcs (MultiThreadProcsLaz) which I use to create a pool of threads for efficient parallel financial backtesting over the number of cores my Mac has available.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 11, 2019, 06:33:37 pm
I was able to reproduce a simple illustration of the "watch" problem using the Lazarus/Examples/MultiThreadingExample1 project.

To reproduce, simply build the project as a Cocoa/64bit.  Set a breakpoint at the TMyThreadExecute line 102 "If NewStatus <> fStatusText then begin".  Set a watch for the newStatus string.  Observe that at the breakpoint, you can cursor over a newStatus reference in the source code and see
    ANSISTRING($00000xxxxx)'TMyThreadTime:2019.....'
[Personally I wish it didn't show "ANSISTRING" since that uses a lot of screen real estate when I really just want to see the string, but that's personal.]

Note that in the watch window, newStatus shows a value again showing "ANSISTRING($xxxxxxxxx)ssssssss" where "sssssss" is nonsensical hex characters, etc. picked up from an invalid memory location.

In my case, the new thread is #6 but as before the threads window at the breakpoint has the green arrow at thread #1.  Attached is the IDE Internals/debug output.  I note that only one "thread select" occurs in the debug output -- "thread select 1".

Would appreciate any OSX users confirming this issue too!  Very simple to test...Hopefully you'll see what I see!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 11, 2019, 07:35:14 pm
Personally I wish it didn't show "ANSISTRING" since that uses a lot of screen real estate when I really just want to see the string, but that's personal.
Dry dwarf3 (at least for strings)

Still looking into the rest
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 11, 2019, 07:43:56 pm
Dwarf3 does stop the "ANSISTRING" output and address info.  Thanks!  (I periodically try to see if I can get gdb to work too -- so accidentally had "dwarf with sets" selected.)  But agree 3 Dry dwarves (dwarfs?) are more pleasant than 3 wet ones :-)
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Hansaplast on February 12, 2019, 11:29:48 am

By request;


I tried Keith's example;


Dwarf2: I do get the same ANSISTRING thing ...
Dwarf3: I get an empty string in the watchlist. However when I hover over the NewStatus variable, it does show me string content without the "ANSISTRING" ...


See attached screenshots.


I'm running Lazarus 2.1.0 r60408M FPC 3.0.4 x86_64-darwin-cocoa (alpha) on Mojave 10.14.3.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 12, 2019, 05:30:51 pm
Hans,
Thanks for the confirmation!  There aren't that many of us using OSX and so many configurations are possible...so I don't trust my results as the norm.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 12, 2019, 06:22:20 pm
The "AnsiString" prefix is by design (at least for now)

With dwarf2 the compiler does not tell the debugger that this is a string. The debugger gets the definition
 type AnsiString = ^Char;

And for pointer types, the debugger includes the type name. (this may be made optional in future, but for now it is hardcoded).

In dwarf3 the debugger encodes the string as "array [0..x] of char".
But internally it currently uses different dwarf tokens to encode that type, that it does when encoding an actual array.

So if the debugger detects fpc as compiler, it looks for this internal difference. (If fpc changes that in future, then the debugger will be clueless (again))

So with current fpc, and dwarf 3 the debugger can identify the string type. And then it will not display the type name.

--
This applies to all "FpDebug" based debuggers.

LLDB has no clue of the above.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 12, 2019, 08:49:14 pm
I found one issue that matches your description. (I have not yet tested on Mac, that always takes some extra time)

https://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=60412
Just copy the one line into your install. This is the commit on trunk, so I am not sure if you can take the entire file.

There may be some other issue, that I am still looking into. But if you could test that.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 12, 2019, 10:22:55 pm
As per revision 60412, replaced line 2398 in lldbdebugger.pas
Code: [Select]
FInstr := TLldbInstructionExpression.Create(FWatchValue.Expression, FWatchValue.ThreadId, FWatchValue.StackFrame)
and rebuilt the IDE.

As before to test, using the /lazarus/examples/multithreading project, set a single breakpoint at the MyThread.Execute line 102.  Unfortunately, results were as before: hovering over newStatus shows the string value but the "watch" shows newStatus as ''.

FWIW, again note that there is only one "select thread" command in the debug output: "thread select1".  Also the Threads window shows the green arrow on thread ID 1, but thread ID 7 (showing line 102 Function: Execute) implies the watch info should be coming from thread 7.

If this example works on your machine, then it would appear it is OSX related? 

Sorry this is such a "bugger!"  Please let me know if there is anything I can do to help with the debugging.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 13, 2019, 01:28:51 am
OK, lets sync yours and my experiences...
Based on the latest changes, including the fix in rev 60413 (One more, over the previous fix)

I have now the MultiThreadingExample1 running on mac 64 bit.

Yes, when it stops, it shows a "useless" value.
The reason appears that the wrong thread is selected. Thread 1 is selected, which is somewhere in the kernel.

Thread 8 should be selected.

If I do select that thread (select in list, then press "current") then all seems fine.

-----------------
I have to look into the correct thread being selected by default.

But please confirm, if this is the same as your issue, or if there are further issues.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 13, 2019, 05:40:57 pm
Manually made changes to match rev 60413 but to no effect here :-(

I would be interested to know if the "watch" issue arises in this example (multi-threaded) for your normal working non/Mac environment or if it is specific to OSX, or maybe 64 bit environments?

I confirmed that selecting/pressing current made the "watch" work!   I agree totally that the watch isn't working correctly because the thread isn't set correctly at watch evaluation time.

Personally, I can't get gdb to work under Mojave so I can't test this but it seems if one looked at the GDB debug output for the MultiThreadingExample1, and compared it to the LLDB debug output, it could become apparent where the LLDB version is not matching the commands sent/received to/from the code.  (For me, GDB had no issues in the multi-threaded situation.)

Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 13, 2019, 06:04:11 pm
I confirmed that selecting/pressing current made the "watch" work!   I agree totally that the watch isn't working correctly because the thread isn't set correctly at watch evaluation time.
Ok, technically that means the watch works. But the thread does not. (and in more ways, than just being the wrong one selected)
I am working on this.

Quote
I would be interested to know if the "watch" issue arises in this example (multi-threaded) for your normal working non/Mac environment or if it is specific to OSX, or maybe 64 bit environments?
I have not checked 32 bit Mac, but it does not arise on windows.
I also already spotted the differences. So it should be fix-able.

Just fitting it into my busy schedule.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 14, 2019, 03:54:49 am
I committed a further part of the fix in r60419. It should select the correct thread now.

There is still something wrong.

"newStatus" is a local variable.
Yet if I select thread 1, which is not in that procedure, it still shows a value for it (random or nil). It should show "invalid" since the variable does not exist.

If thread1 was in valid pascal code, it could then not see its own variables.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 14, 2019, 04:16:29 am
Confirmed correct thread selection and proper watch evaluation w/ r60419 :-)
Understand the remaining scope issue -- let me know if there is anything I can do to help.  I really appreciate all your work on this -- knowing there are other priorities!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: dbannon on February 15, 2019, 11:00:29 am

Neither a complaint nor a bug report  :)

This thread is up to 10 pages, I'd like to update the wiki page as its probably easier for new users. Can you confirm a couple of things please, relating to Lazarus 2.0.0 -

1. Right at the start, you mentioned that LazDebuggerFpLLdb "should be in the list of available packages" but on my system, Packages->Install/UninstallPackages does not list it. It is, as you also mention, in Components->....
Should I assume thats as expected ?

2. When setting the IDE to use LLdb, we should choose "LLDB debugger (with fpdebug) (Beta)" ?

3. The IDE then requires us to choose one of several exe formats. Is there any  particular preference ?

Davo

PS I don't do much debugging om the Mac, prefer Linux but recently, when I have had occasion to need it, it works fine on my very old, very secondhand Sierra Powerbook. Thanks !
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 15, 2019, 03:57:34 pm
Can you confirm a couple of things please, relating to Lazarus 2.0.0 -

1. Right at the start, you mentioned that LazDebuggerFpLLdb "should be in the list of available packages" but on my system, Packages->Install/UninstallPackages does not list it. It is, as you also mention, in Components->....
Should I assume thats as expected ?
It should be in the left side list, as it should already be installed.

If you open Tools > Options, and go to the Debugger page, then "LLDB with fpdebug) should be in the dropdown of debugger types
Once selected, you need to give it the location of lldb (/usr/bin/lldb) in the edit field below.

Quote
2. When setting the IDE to use LLdb, we should choose "LLDB debugger (with fpdebug) (Beta)" ?
Yes.

Not tested, but afaik lldb (without fpdebug) does not understand a watch like "MyObject.FValue". Instead it takes "MyObject->FValue"
And typecasts would be c-style too "(int) FValue"

Quote
3. The IDE then requires us to choose one of several exe formats. Is there any  particular preference ?
Not sure what you refer to? I assume "debug info"?

This applies on all platforms, and for all type of debuggers. (So it is the same for gdb on win/linux). Only which one is best, is different depending on the debugger.

For "lldb + fpdebug" the best is "dwarf3".
In the unexpected case of problems, it may be worth trying "dwarf with sets".

This setting only affects the units directly in your project.
However units in packages may have debug info too.
This can be
- set per package
- for many, but not all packages in menu Tools > Configure build Lazarus
- project settings > "Additions and overrides"(search wiki)

If you change settings for package, you may as well check into which package you expect to step. Package you do not step into, do not need debug info.
If you use a type from a package (such as TForm from LCL) it is enough that your unit (in which you declare the variable / must declare and use a variable to include the type) has debug info.

Reducing the amount of packages with debug info (including those that default to have debug info), can shorten the debuggers start-up time.

Also it may be worth comparing (I have not tested) the debuggers start up time for the same settings, only changing the checkbox "use external debug info".
This needs to be only set in your project. If set in your project it will affect all packages. (If set in a package it will do nothing / at least should...)

--EDIT
I updated the initial post. Hope it is more accurate now.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 15, 2019, 11:03:20 pm
I added a further fix for the threading context issue. (See first post of topic, for info on where to get the files)

This should hopefully be it.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 17, 2019, 02:12:08 am
Just requesting a point of clarification...
From 1st post of this thread:
Quote
Known Issues in the 2.0 release that are fixed for 2.0.2:
Single stepping, may sometimes step the wrong thread (if paused in a multithreading app)
To apply the fix in Lazarus 2.0.0 go to https://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=60339 and download the 2 files. You can replace the entire files in your 2.0.0 install. Rebuild the IDE and restart.
Fix a crash for local/watches on begin/end of procedure.
To apply the fix in Lazarus 2.0.0 go to https://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=60353 and download the file. You can replace the entire files in your 2.0.0 install. Rebuild the IDE and restart.
Fix several issues when debugging multi-threaded apps:
- Eval watches,hints,locals according to selected thread (and stackframe)
- Select correct thread at breakpoint
- Select correct Stackframe for thread, when switching threads
To apply the fix in Lazarus 2.0.0 go to https://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=60436 and download the file. You can replace the entire files in your 2.0.0 install.
In order for this to work you must also install the file lldbinstructions.pas from for revision=60339 / 1st in this list. (the file lldbdebugger.pas must be taken from this changeset (revision=60436).
Rebuild the IDE and restart.

My current svn fixes checkout confirms I "Checked out revision 60436".

If one gets the current fixes through and including r60436, aren't all earlier debugger fixes covered or is there a problem with lldbintructions.pas that was introduced AFTER r60339?
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 17, 2019, 02:43:02 am
If you do an svn checkout up to and including 60436, then you have all of those fixes.

The statements are directed at people without svn, who want to download (not patch, but get entire file) the files and replace them.
Someone who only wants the fixes in 60436 (no idea why they would want that, but...) must use patch. Because if they download the file it contains the earlier revisions. And the earlier revision depends on the changes in the other file.


Made some updates. If you have a better wording...
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: kcandrews on February 17, 2019, 02:53:33 am
Understood!  Thank you :-)  You are far better at wording than I!
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: dbannon on February 17, 2019, 11:48:57 am
Can you confirm a couple of things please, relating to Lazarus 2.0.0 -
.... LazDebuggerFpLLdb "should be in the list of available packages" but ......
It should be in the left side list, as it should already be installed.
......
OK, clarified. Its there on the left side (Installed) only in the bigide model. On the Mac at least it probably needs to be in either build.

Will update that page soon....

Davo
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: felipemdc on February 25, 2019, 10:58:54 am
Just thought I'd post a problem and solution I had with lldb in Mac OS X in case anyone else has the same problem.

The reproduction steps of the problem are:
1> Install Lazarus 2.0 in a Mac with lldb and without gdb
2> Start a new project, save it, run it.

Result: The following error message appears:

Debugger
Initialization output:
error: file specified in --source (-s) options doesn't exist: 'ilent'

In the next dialog there is:

The GDB command:
-gdb-set confirm off
did not return any result

Solution:

Go to Tools->Options->Debugger

You see that the path will be right "/usr/bin/lldb", but the debugger type is wrong! It is GDB, change to "LLDB Debugger (with fpdebug) (Beta)" and it will work fine!

This can be tricky because if you just look at the path to the debugger tool, it will be right, so you might miss that the combobox default value is wrong.

Lazarus should probably auto-change the combobox depending on the detected debugger.

I actually had previously installed 1.8 so maybe this could be the cause of the problem (old lazarus config pointing to GDB?), I can't be sure.
Title: Re: New debugger for Mac based on lldb (Call for testers)
Post by: Martin_fr on February 25, 2019, 12:02:10 pm
IIRC There is no autodetect for lldb yet.

And if you have old config it should not have been changed at all. Strange.
TinyPortal © 2005-2018