Stack unwinding in x64 assembler

[CuriousKit]

So I'm writing a lot of heavily-optimised assembler code, mostly for mathematical purposes, but also as potential patch submissions for the x86-64 build of FPC and Lazarus.  One thing that Florian brought up was the necessity inserting some extra information into the assembler routines so that the stack unwinds properly if a Windows exception is raised (so it properly resets the stack and restores any non-volatile registers that have been modified).

How do I go about this, because I can't seem to find a straight answer anywhere?

(As a simple example, say I need to take note of the fact that %rdi is pushed to the stack, a non-volatile register under 64-bit Windows)

[CuriousKit]

MASM had "pushreg" hints that FPC's inline assembler doesn't appear to have (or at least not in an advertised form), and such things don't show up in the disassembly window anyway.

EDIT: There was a post previous to this one from another user that was since removed, so this message has been modified accordingly.

[marcov]

iirc FPC does have something, but only for blocks, not assembler procedures:

procedure xx;
begin
 asm

 end['xmm6','rdi']
end;

For SEH, easiest is to simply look at compiler code generated with -a.  The stack must also always be aligned.

[CuriousKit]

Thanks Marcov - I admit I forgot about outputting the generated assembler to a file, and of course the disassembly window, being interpreted from the compiled binary, has all the labels and SEH hints stripped out.

Turns out that you can't insert ".seh_proc" and ".seh_endproc" etc. into a routine, but you can insert ".seh_stackalloc", ".seh_pushreg" and ".seh_savereg" (used if you store a register at a particular point in the stack frame rather than using "push").  It seems this was the answer I was looking for.  The only drawback is that this is for AT&T syntax, and I don't think there's an equivalent yet available for Intel syntax.

ADDENDUM: If you use these, you also need to put ".seh_endprologue" at the very start of the procedure if you specify the "nostackframe" directive.

[CuriousKit]

Note to anyone who stumbles across this topic later.

As pointed out by Sergei Gorelkin, my advice about .seh_endprologue and nostackframe is very much incorrect.  You have to insert .seh_endprologue after all of the PUSH commands (and the respective .seh_pushreg hints) and your stack allocation.  Additionally, you cannot use PUSH operations after the prologue (i.e. in the middle of the procedure), as doing so will break the stack unwinding process.  If you need to save and restore a register, then you need to write and read it to the stack space that you allocated, while making a note with .seh_savereg that the register is being preserved.

(Is that about right, Sergei?)

[marcov]

Thanks. I always use the above construct (and similar but different ones in delphi using .pushnv/.savenv) because for my routines it doesn't really matter. (operating on whole images at a time, runtimes of 100us till 10-20ms)

E.g. current working on rotation of images (by 90/270 degrees only, to compensate for camera fixture angle)

[CuriousKit]

Well, in my case it was for writing an assembler-optimised version of FillWord, FillDWord and FillQWord, which need to be as safe as possible because there's no guarantee that the memory passed into the function is valid (along with the problem of buffer overruns, for example).

It's good to meet another assembly programmer!