Plex/Pyacc read from source file?

[Thaddy]

Yacc parses LARL(1) in principle. What you call "recursion" is the wrong term. We also have "recursive descent parsers"..... So that confused me.  (Use e.g. coco/r for that)
If your grammar is sound it will produce correct code for the brackets by looking ahead: it will find its right place in the parse tree.
There are many such examples on the web (including the simple calculator example).
I'll see if I can come up with a more elaborate example myself tomorrow, with explanation.
Within the context of compiler generators! it is not really difficult.

Because I need it for some other project, the example will:
define a grammar for an integer function that takes an integer parameter that can be the output of the (a) function itself.
It is not written yet, but I have done this many times.

[Trenatos]

Ahk! You're right, sorry about the confusion, like I said I'm not formally trained in these things, I'm just learning as I go.

That example would be wonderfully helpful, I'd really appreciate it Thaddy.

[Thaddy]

Well, I encourage you to persist. Compiler generators can save a lot of manual labor. (Not only full programming languages, but also other logically repetitive problems (much more so). I use them a lot: 4 times a year..... in serious fashion)

BTW: did you really check there is no grammar available? Usually there is something? Taking on a full language and write a grammar for it is no mean feat.
(Usually you'll get a master's degree for that if you've done some more work, at least a bachelor)

[Trenatos]

I'm going to keep at it until I succeed. See that's the upside to tackling brand new things, everything is difficult so it doesn't matter if it takes a long time 

Would you mind sharing any of your previous work, and an example source file for the target language?  ^_^

There are several grammars for CFML (ColdFusion), more or less complete, I think they're all using ANTLR and Java.

But I'd like to learn how to write tools/languages using these tools (GoldParser, Plex/Pyacc, etc.) because it's a field I've been curious about for way too long without diving into.

There are examples out there for working with Lex/Yacc but most of what I've found has either been way too simple or way too complex.

Now if only I could get a school to hand me a bachelors when I'm done... 

[Thaddy]

Would you mind sharing any of your previous work, and an example source file for the target language?  ^_^

Yes, of course. Search the forum 
I believe I gave some grammars to go with several compiler generators, not only for Yacc (apart from the original ones) as examples.

[avra]

There are several grammars for CFML (ColdFusion), more or less complete, I think they're all using ANTLR and Java.

There is ANTLR for Delphi that you can use with those grammars. Since ANTLR uses EBNF, you can also adapt the EBNF grammar to BNF for use in GOLD Parser. From there you can use it with Lazarus GoldEngine (search the forum for more info), or use GOLD Delphi/Pascal code generators, or export grammar from GOLD to YACC - which can be used with PYACC if you prefer it. You can even import YACC into GOLD, but you will probably have to clean it first since importer is very picky.

[Trenatos]

avra, I haven't tried GoldParser, but part of this is that I want to learn how to do it, not just copy a finished grammar.

Thaddy has graciously offered to help with a basic grammar to get me going but that's a starting point rather than someone else's finished project 

[Trenatos]

Progress, I've managed to make a simple looping grammar, letting me nest function calls within function calls

Code: Pascal  [Select][+][-]

1.   first("First string");
2.   first(second("Second string"));
3.   first(second(third("Third string")));
4.  

Code: Pascal  [Select][+][-]

1. commands: function_call
2.         |
3.         commands function_call;
4.  
5. function_call:
6.         TOKSTRING LPAREN function_call RPAREN
7.         |
8.         function_with_value;
9.  
10. function_with_value:
11.         function_string
12.         |
13.         function_number;
14.  
15. function_number:
16.         TOKSTRING LPAREN TOKNUMBER RPAREN
17.         {
18.           WriteLn('Function: ' + $1);
19.           WriteLn('Value: ' + IntToStr($3));
20.         };
21.  
22. function_string:
23.         TOKSTRING LPAREN TOKSTRING RPAREN
24.         {
25.           WriteLn('Function: ' + $1);
26.           WriteLn('Value: ' + $3);
27.         };
28.  

Not sure how to return values to the calling function though.

For testing I'd just like to append "abc" or increment by 10 if it's a number, but it's still progress 

[Thaddy]

Yes. That looks good! For returning results, have a look (again) at the second corrected example in the wiki.
I am rather busy right now, but if you haven't figured it out yet, I will add the promised example.

[Trenatos]

This is exciting

I think I understand the data return, it looks like you just set $$ to the value to be returned, and it'll be used as input.

I can't test it until this evening, but I'll give it a go then.

[Thaddy]

Yep. That will work...

[Trenatos]

Ok I'm stuck and confused..

I'm trying to get a very basic test going.

Two functions, console() and double()

Console just does WriteLn() and double should take a string or the result of another function and double it before returning it.

lexer

Code: Pascal  [Select][+][-]

1. %{
2. %}
3. %%
4. \(                      return (LPAREN);
5. \)                      return (RPAREN);
6. double                  return (TOKDOUBLE);
7. console                 return (TOKCONSOLE);
8. \n                      ;
9. [a-zA-Z0-9"' ]+         begin yylval.yyString := yytext; return (TOKSTRING); end;
10.  

parser

Code: Pascal  [Select][+][-]

1. %{
2. program dostuff;
3.  
4. {$mode objfpc}
5.  
6. uses SysUtils, yacclib, lexlib;
7.  
8. var
9.   srcFile: Text;
10.  
11. procedure yyerror(s: string);
12. begin
13.         writeln(Format('Error: %s', [s]));
14. end;
15.  
16. %}
17.  
18. %start commands
19. %token LPAREN RPAREN TOKDOUBLE TOKCONSOLE
20. %token <String> TOKSTRING
21. %type <String> function_double function_call function_console function
22. %%
23.  
24. commands:
25.         function
26.         |
27.         commands function;
28.  
29. function:
30.         function_double
31.         |
32.         function_console;
33.  
34. function_console:
35.         TOKCONSOLE LPAREN TOKSTRING RPAREN
36.         |
37.         TOKCONSOLE LPAREN function RPAREN
38.         { WriteLn($3); };
39.  
40. function_double:
41.         TOKDOUBLE LPAREN TOKSTRING RPAREN
42.         { writeln('Doubling..'); $$ := $3 + $3; };
43. %%
44.  
45. {$include simplelexer.pas}
46.  
47. begin
48.   yydebug := false;
49.  
50.   WriteLn('================');
51.  
52.   Assign(srcFile, ParamStr(1));
53.   reset(srcFile);
54.   yyinput := srcFile;
55.  
56.   yyparse();
57. end.
58.  

Test target file

Code: Pascal  [Select][+][-]

1. console(first)
2. double(second)
3. console(double(third))
4.  

Here's what's throwing me for a spin..

Look at the function_console, if I use the order above for the matches, it outputs doubling doubling thirdthird but not first

If I change the order to

Code: Pascal  [Select][+][-]

1. function_console:
2.         TOKCONSOLE LPAREN function RPAREN
3.         |
4.         TOKCONSOLE LPAREN TOKSTRING RPAREN
5.         { WriteLn($3); };
6.  

It outputs first doubling doubling but not the third...

And they both fall down on console(double(double(third)))

I thought I was getting somewhere 

Thaddy, any advice?

[Thaddy]

To parse the parentheses I normally write something like this in the in your case commands block:  |'('commands ')'
That should be sufficient. A simple example yacc file  would look like this:

Code: Text  [Select][+][-]

1. %left '+' '-' '*' '/'
2. %token <real> NUMBER
3. %type  <Real> expr
4. %%
5. expr   : expr '+' expr { $$ := $1+$3; }
6.        | expr '-' expr { $$ := $1-$3; }
7.        | expr '*' expr { $$ := $1*$3; }
8.        | expr '/' expr { $$ := $1/$3; }
9.        | '(' expr ')'
10.        | NUMBER
11.        ;

This should work similarly for your commands.
Hope that helps. (As you can see I am building a new calculator example <smile>)

[Trenatos]

Thanks Thaddy, and awesome that you're writing a new calculator

So if I understand it right, I should be able to do something like

Code: Pascal  [Select][+][-]

1. func : func '(' func ')' { // Identify function, do stuff; $$ := 'some result' }
2.        | func '(' NUMBER ')' { // Identify function, do stuff; $$ := 'some result' }
3.        | func '(' STRING ')' { // Identify function, do stuff; $$ := 'some result' }
4.        | func '(' NUMBER ')' { // Identify function, do stuff; $$ := 'some result' }

[Thaddy]

Thanks Thaddy, and awesome that you're writing a new calculator

So if I understand it right, I should be able to do something like

Code: Pascal  [Select][+][-]

1. func : func '(' func ')' { // Identify function, do stuff; $$ := 'some result' }
2.        | func '(' NUMBER ')' { // Identify function, do stuff; $$ := 'some result' }
3.        | func '(' STRING ')' { // Identify function, do stuff; $$ := 'some result' }
4.        | func '(' NUMBER ')' { // Identify function, do stuff; $$ := 'some result' }

No, just once... think of expr the same as function. I will give you an example this weekend, but go back to your old code and add a similar construct. It is not that complex.