Lazarus

Free Pascal => FPC development => Topic started by: Nitorami on June 06, 2018, 11:11:25 pm

Title: Dynamic array extensions ?
Post by: Nitorami on June 06, 2018, 11:11:25 pm
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices.

With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

What is your opinion on that  ?
Title: Re: Dynamic array extensions ?
Post by: Phil on June 07, 2018, 03:13:58 am
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

What is your opinion on that  ?

Sounds like Pascal array syntax is getting dragged into the 21st century so it's more like other languages. Take the Swift MLPClassifier example givenhere (under Interpreter):

https://github.com/tensorflow/swift/blob/master/Usage.md

That gives several examples of constant arrays being declared within the code that uses the arrays. I assume something like that will be available in Pascal. If so, that should make for cleaner code, less code.

I'm kind of indifferent to operator overloading. Since you already have to declare a function that gets called when you use the overloaded operator, maybe just make that function available in the interface section so it can be used in case "+" is lost.

The Swift example also uses operator overloading for tensor arithmetic. So something like this:

tanh(matmul(x, w1) + b1)

Could be written without operator overloading like this:

tanh(add(matmul(x, w1), b1))

That still seems pretty clear. After all, you can't get away completely from calling functions when you're doing this kind of stuff. I suppose you could overload operators for both tanh and matmul if there were obvious math symbols that made sense. Are there limitations on what characters you can overload? In Swift, I believe you can use any Unicode character as an overloaded operator. In fact, they actually overload ⊗ for matrix multiplication, so you could write this:

tanh(x ⊗ w1 + b1)

Interesting that they don't use that overloaded operator in their own example. Perhaps cooler heads prevailed.

In my Pascal interface to TensorFlow I overloaded the arithmetic operators too, per Swift, but I don't think I would miss them if they weren't available.

Perhaps just rejoice in the "lower-ceremony" syntax for arrays and make peace with the loss of an overloaded "+" in this one case.
Title: Re: Dynamic array extensions ?
Post by: taazz on June 07, 2018, 04:34:03 am
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices.

With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

What is your opinion on that  ?
on one hand it nice not to have to write those pesky init routines, on the other I have no idea why it is needed to become an intrinsic, until I get some insight on that I can't really comment. I guess it makes things both better and worst. Its better because now + becomes standard, stable, when I see it on piece of code that I debug, I know exactly what it does, it is worst well I guess you know why. Now you have to type all kinds of staff and your code moves a bit farther away from math symbolism. You could say I'm on the fence on that.
ps
Oh by the way becoming like swift is a bad thing not a good one :P (just kidding)
Title: Re: Dynamic array extensions ?
Post by: Phil on June 07, 2018, 04:53:09 am
Oh by the way becoming like swift is a bad thing not a good one :P (just kidding)

You'll probably find this interesting, on why Google selected Swift over Python (considering that Python rules in ML).

https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md

I'm looking to see whether Pascal makes sense to use with TensorFlow.
Title: Re: Dynamic array extensions ?
Post by: Thaddy on June 07, 2018, 09:19:09 am
Well the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on.
For use as vectors you can write a type helper Add(), something like:
Code: Pascal  [Select][+][-]
  1. Type
  2.  
  3.   TMyIntvector = array of array of integer;
  4.  
  5.   TMyIntVectorHelper = type helper for TMyIntVector
  6.     class function Add(const a,b:TMyIntVector):TMyIntVector;static;
  7.     function Add(const b:TMyIntVector):TMyIntVector;
  8.   end;
Still, how do you retrieve the dimensions easily? (also goes for the operators on vector cells)
These must be known and furthermore dynamic arrays can be ragged arrays, which complicates it even further.
Ragged:
Given an array of array of integer, you can do this:
  Setlength(a,3,5);
  Setlength(a[0],3);
  Setlength(a[4],2);
Title: Re: Dynamic array extensions ?
Post by: marcov on June 07, 2018, 09:28:01 am
Well the logical thing for dynamic arrays is really concat after all, because it not necessarily holds data that you can perform vector math on.

You could invert that logic, and say that contrary to strings because dynamic arrays CAN contain data you can do math on, you reserve that operator for its mathematical meaning.

And for appending you can write a type helper Append()

Then you keep the association between static and dynamic arrays instead of strings and dynamics arrays.  Also, Strings are commonly concatenated, but general arrays much less often, while running an operator on every element of an array is more common.




Title: Re: Dynamic array extensions ?
Post by: Thaddy on June 07, 2018, 09:56:00 am
That was my initial opinion. That I reversed because of the complexity. Either way is fine with me but I understand the ratio.
Title: Re: Dynamic array extensions ?
Post by: circular on June 13, 2018, 03:04:19 pm
Where can we say that we want to keep the + operator as it is?
Title: Re: Dynamic array extensions ?
Post by: marcov on June 13, 2018, 03:25:46 pm
I updated my answer
Title: Re: Dynamic array extensions ?
Post by: GAN on June 13, 2018, 09:43:50 pm
Where can we say that we want to keep the + operator as it is?
I agree.
Title: Re: Dynamic array extensions ?
Post by: hnb on June 13, 2018, 11:30:18 pm
Where can we say that we want to keep the + operator as it is?
I agree.
probably the only solution is combo of both :


while I like most of improvements, the support for "+" operator seems strange even for me (I generally like new things)... Also strange is breaking backward compatibility (which AFAIK is one of priority) with questionable gain.
Title: Re: Dynamic array extensions ?
Post by: schuler on June 14, 2018, 09:29:57 am
Hi Nitorami,
These improvements are really fantastic!

I would like to take the opportunity to share what I miss: faster math vector operations. Some vectors I use have 10000 elements that are summed/multiplied with other vectors. I ended coding it in assembler (the full code has more than 4000 lines):

TNeuralFloatArrPtr is a pointer to the first element of a single precision floating point dynamic array.
 
Code: Pascal  [Select][+][-]
  1. procedure AVXMulAdd(PtrA, PtrB: TNeuralFloatArrPtr; MulOp: TNeuralFloat; NumElements: integer);
  2. var
  3.   MulOpPtr: pointer;
  4.   localNumElements, MissedElements: integer;
  5. begin
  6.   localNumElements := (NumElements div 4) * 4;
  7.   MissedElements := NumElements - localNumElements;
  8.   if localNumElements > 0 then
  9.   begin
  10.     MulOpPtr := Addr(MulOp);
  11.   asm
  12.   mov ecx, localNumElements
  13.   mov rax, PtrB
  14.   mov rdx, MulOpPtr
  15.  
  16.   VBROADCASTSS ymm5, [rdx]
  17.   mov rdx, PtrA
  18.  
  19.   push rcx
  20.   shr ecx,5  // number of large iterations = number of elements / 32
  21.   jz @SkipLargeAddLoop
  22.  
  23. @LargeAddLoop:
  24.   {$IFDEF AVX2}
  25.   vmovups ymm0, [rdx]
  26.   vmovups ymm1, [rdx+32]
  27.   vmovups ymm2, [rdx+64]
  28.   vmovups ymm3, [rdx+96]
  29.  
  30.   vfmadd231ps ymm0, ymm5, [rax]
  31.   vfmadd231ps ymm1, ymm5, [rax+32]
  32.   vfmadd231ps ymm2, ymm5, [rax+64]
  33.   vfmadd231ps ymm3, ymm5, [rax+96]
  34.   {$ELSE}
  35.   vmulps  ymm0, ymm5, [rax]
  36.   vmulps  ymm1, ymm5, [rax+32]
  37.   vmulps  ymm2, ymm5, [rax+64]
  38.   vmulps  ymm3, ymm5, [rax+96]
  39.  
  40.   vaddps  ymm0, ymm0, [rdx]
  41.   vaddps  ymm1, ymm1, [rdx+32]
  42.   vaddps  ymm2, ymm2, [rdx+64]
  43.   vaddps  ymm3, ymm3, [rdx+96]
  44.   {$ENDIF}
  45.  
  46.   vmovups [rdx],    ymm0
  47.   vmovups [rdx+32], ymm1
  48.   vmovups [rdx+64], ymm2
  49.   vmovups [rdx+96], ymm3
  50.  
  51.   add rax, 128
  52.   add rdx, 128
  53.   dec ecx
  54.   jnz @LargeAddLoop
  55.  
  56. @SkipLargeAddLoop:
  57.   vzeroupper
  58.  
  59.   pop rcx
  60.   and ecx,$0000001F
  61.   jz @EndAdd
  62.   shr ecx, 2 // number of small iterations = (number of elements modulo 16) / 4
  63.  
  64. @SmallAddLoop:
  65.  
  66.   movups  xmm2, [rax]
  67.   movups  xmm4, [rdx]
  68.  
  69.   mulps   xmm2, xmm5
  70.   addps   xmm4, xmm2
  71.  
  72.   movups  [rdx], xmm4
  73.  
  74.   add rax, 16
  75.   add rdx, 16
  76.  
  77.   dec ecx
  78.   jnz @SmallAddLoop
  79.  
  80. @EndAdd:
  81.   end  [
  82.     'RAX', 'RCX', 'RDX',
  83.     'ymm0', 'ymm1', 'ymm2', 'ymm3', 'ymm4', 'ymm5'
  84.   ];
  85.   end; // of if
  86.  
  87.   if MissedElements>0 then
  88.   begin
  89.     PtrA^[localNumElements] += MulOp*PtrB^[localNumElements];
  90.     if MissedElements>1 then
  91.     begin
  92.       PtrA^[localNumElements+1] += MulOp*PtrB^[localNumElements+1];
  93.       if MissedElements>2 then PtrA^[localNumElements+2] += MulOp*PtrB^[localNumElements+2];
  94.     end;
  95.   end;
  96. end;

In the case that you are interested in the full source code, please let me know.

:) Wish everyone happy coding :)
Title: Re: Dynamic array extensions ?
Post by: VTwin on June 14, 2018, 12:57:47 pm
With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

Oh dear!

I consistently use operator overloading on vectors and matrixes. Seems like going backwards to rewrite everything as functions or container classes. If I add two vectors I don't want them concatenated!! :/
Title: Re: Dynamic array extensions ?
Post by: circular on June 14, 2018, 06:35:04 pm
Nitorami, could you point out the FPC e-mail that announces those changes?
http://lists.freepascal.org/pipermail/fpc-devel/
Title: Re: Dynamic array extensions ?
Post by: Nitorami on June 14, 2018, 09:15:27 pm
@circular: Of course.
http://free-pascal-general.1045716.n5.nabble.com/Feature-announcement-Dynamic-array-extensions-td5731410.html
You may have to register with your email address.

@schuler:
1. I think you did not really read the original post; I did not advertize these "improvements", which in my view are unnecessary as part of the core language, as it is trivial to implement them manually. I wanted to hear other user's opinion on the developer's plan to implement the + operator as intrinsic for concatenation of dynamic strings, making it unavailable for simple vector addition: I use vector addition a lot but never needed concatenation.

2. Thanks for the code. For my current project the normal non-vectorised math is sufficient, although optimisation is always tempting. My own knowledge of assembler is poor, and I appreciated the native support for vectorised math which had once been introduced (I think by switch $Sv). But unfortunately it has not been maintained any further and is now broken, at least under windows :-(
Title: Re: Dynamic array extensions ?
Post by: circular on June 14, 2018, 10:18:59 pm
I read the FP thread. Seems there has been a debate over there.

If I understand correctly:
Code: Delphi  [Select][+][-]
  1. var a: array of integer;
  2. begin
  3.   a := [1,2,3];
  4.   a += 1;          // will give a = [1,2,3,1] instead a = [2,3,4]
  5.   a := [1,2,3];
  6.   a += [4,5,6];  // will give a = [1,2,3,4,5,6] instead of a = [5,7,9]
  7. end;

Correct?
Title: Re: Dynamic array extensions ?
Post by: Nitorami on June 15, 2018, 10:23:37 am
Yes, that seems to be the plan - the "+" operator acting as concatenation, defined as intrinsic in the language so it can't be overloaded anymore, e.g. to do numeric addition instead.




Title: Re: Dynamic array extensions ?
Post by: schuler on June 15, 2018, 10:43:21 am
Nitorami,
Sorry. I evidently didn't understand your post first time I read.

I do have concerns with "+". I think that everyone that uses Free Pascal for math will have concerns.

In my case, it won't brake any code although I'm looking for another symbol in the keyboard that could be used for concat. What about & for concatenation?

BTW, is this concatenation or union? If it is union, I might suggest || instead.
Title: Re: Dynamic array extensions ?
Post by: Nitorami on June 15, 2018, 11:16:48 am
Code: Pascal  [Select][+][-]
  1. BTW, is this concatenation or union? If it is union, I might suggest || instead.

They meant concatenation, probably for similarity between ansistrings and dynamic arrays. Other operators have been proposed, read more in the mailing list discussion http://free-pascal-general.1045716.n5.nabble.com/Feature-announcement-Dynamic-array-extensions-td5731410.html

But I don't understand why we need an intrinsic operator at all. I would leave it to the programmer to define one, whether it is for concatenation, addition, or whatever else is required.
Title: Re: Dynamic array extensions ?
Post by: circular on June 15, 2018, 10:46:44 pm
@schuler

Agreed that "&" would be nice, like in VB. It could be also used in strings:
Code: Delphi  [Select][+][-]
  1. var s: string; i: integer;
  2. begin
  3.   i := 36;
  4.   s := 'hello number ' & i; // would yield 'hello number 36'
  5. end;
 
Title: Re: Dynamic array extensions ?
Post by: Bart on June 16, 2018, 12:59:33 pm
OMG. This is Pascal, it is strongtyped.
If you want such syntax, please use Basic.

Bart
Title: Re: Dynamic array extensions ?
Post by: circular on June 16, 2018, 03:47:07 pm
 :D
Title: Re: Dynamic array extensions ?
Post by: lainz on June 16, 2018, 04:03:33 pm
You can just add the + to convert automatically to string.

Code: Pascal  [Select][+][-]
  1. Operator + (s : string; i : integer) z : string;
  2. begin
  3.   z := s + IntToStr(i);
  4. end;
  5.  
  6. {$R *.lfm}
  7.  
  8. { TForm1 }
  9.  
  10. procedure TForm1.Button1Click(Sender: TObject);
  11. var
  12.   s: string;
  13.   i: integer;
  14. begin
  15.   s := 'Hello world ';
  16.   i := 10;
  17.   s := s + i; // 'Hello world 10'
  18.   ShowMessage(s);
  19. end;  
Title: Re: Dynamic array extensions ?
Post by: circular on June 16, 2018, 07:02:42 pm
Indeed that's cool  :)
Title: Re: Dynamic array extensions ?
Post by: Abelisto on June 17, 2018, 09:26:48 pm
@circular, @lainz

Code: Pascal  [Select][+][-]
  1. WriteStr(s, 'Hello number ', 10, '!');
Title: Re: Dynamic array extensions ?
Post by: circular on June 18, 2018, 05:10:39 pm
@abelisto
Thanks I did not know this one
Title: Re: Dynamic array extensions ?
Post by: lainz on June 18, 2018, 07:05:03 pm
@circular, @lainz

Code: Pascal  [Select][+][-]
  1. WriteStr(s, 'Hello number ', 10, '!');

Looks really good =)

I can format float as well with it.
Code: Pascal  [Select][+][-]
  1. WriteStr(s, 'hello cañón 人物 ', 10.5678:2:2, '!');
Title: Re: Dynamic array extensions ?
Post by: PascalDragon on June 20, 2018, 10:30:14 pm
I would like to draw your attention to a new feature announced on the FPC mailing list: Dynamic array extensions. In short:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

Sounds ok, however: The "+" operator is effectively an alias for concat(), and it will be an intrinsic, which means that existing "+" operator overloads for dynamic arrays will no longer compile. Intrinsic operators cannot be overloaded, hence it will no longer be possible to custom define the "+" operator to do a mathematical elementwise addition of numerical vectors/matrices.

With the "+" operator hijacked, all other obvious mathematical operators i.e. -, *, / for numeric vectors are effectively obliterated. We would either have to revert to procedural style as in turbo pascal days, such as in function add (V1, V2: TVector): TVector; or declare extra container classes for numeric vectors / matrices, to which mathematical operators can be applied.

What is your opinion on that  ?

Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

If the modeswitch is active then the "+" operator overload is not possible and the compiler will warn if such an overload is in scope. If the modeswitch is not set, then everything is as before.

The modeswitch is enabled by default in Delphi modes, because that's why the feature was added. To disable the modeswitch there use this:

Code: Pascal  [Select][+][-]
  1. {$mode Delphi}
  2. {$modeswitch ArrayOperators-} // Note the "-"
Title: Re: Dynamic array extensions ?
Post by: Thaddy on June 20, 2018, 10:32:10 pm
Neat solution!
Title: Re: Dynamic array extensions ?
Post by: Nitorami on June 21, 2018, 09:45:49 pm
Quote
Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

Thank you. Guess that is an acceptable solution.

Code: Pascal  [Select][+][-]
  1. {$modeswitch ArrayOperators-} // Note the "-"
Why is it "-" in this case rather than "OFF" ?
Title: Re: Dynamic array extensions ?
Post by: taazz on June 21, 2018, 10:00:10 pm
Quote
Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

Thank you. Guess that is an acceptable solution.

Code: Pascal  [Select][+][-]
  1. {$modeswitch ArrayOperators-} // Note the "-"
Why is it "-" in this case rather than "OFF" ?
As far as I understand, + is an alias to on,  - an alias to off, (or vice versa) they are interchangeable, he only drawn your attention to the - because it is easy to miss, if he had used off instead then the note would not have been necessary.
Title: Re: Dynamic array extensions ?
Post by: VTwin on June 22, 2018, 06:43:03 pm
Quote
Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

Thank you. Guess that is an acceptable solution.

Code: Pascal  [Select][+][-]
  1. {$modeswitch ArrayOperators-} // Note the "-"
Why is it "-" in this case rather than "OFF" ?

https://www.freepascal.org/docs-html/prog/progsu105.html

Quote
The syntax is as follows:
{$MODESWITCH XXX} 
{$MODESWITCH XXX+} 
{$MODESWITCH XXX-}

The first two will switch on feature XXX, the last one will switch it off.

A mode switch seems like a good solution. Thanks!
Title: Re: Dynamic array extensions ?
Post by: PascalDragon on June 22, 2018, 10:36:52 pm
Quote
Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

Thank you. Guess that is an acceptable solution.

Code: Pascal  [Select][+][-]
  1. {$modeswitch ArrayOperators-} // Note the "-"
Why is it "-" in this case rather than "OFF" ?

Because modeswitches currently don't support ON/OFF, only +/- (the latter can also be used with other switches, like e.g. $ScopedEnums).  :-[
Title: Re: Dynamic array extensions ?
Post by: ASBzone on June 23, 2018, 01:31:03 am
Addendum to that: the "+" operator is now connected to a "ArrayOperators" modeswitch.

If the modeswitch is active then the "+" operator overload is not possible and the compiler will warn if such an overload is in scope. If the modeswitch is not set, then everything is as before.

The modeswitch is enabled by default in Delphi modes, because that's why the feature was added. To disable the modeswitch there use this:

Code: Pascal  [Select][+][-]
  1. {$mode Delphi}
  2. {$modeswitch ArrayOperators-} // Note the "-"

Awesome.  Much appreciated.
Title: Re: Dynamic array extensions ?
Post by: PascalDragon on June 23, 2018, 01:49:56 pm
Because modeswitches currently don't support ON/OFF, only +/- (the latter can also be used with other switches, like e.g. $ScopedEnums).  :-[

And now FPC also supports ON and OFF in modeswitches :D
TinyPortal © 2005-2018