# Nessaid CLI Framework

This package implements a grammar specification and tools to compile the grammar and use it for driving CLIs. The CLI commands can be expressed in grammar with the support for custom tokens with suggestion and autocompletion.

The grammar specification is parsed with PLY python package and the CLI objects are implemented on top of readline. The command loop is in async mode.

## Requirements
<b>asyncio</b>: The commandline interpreter will be looping for console input in async mode

<b>ply</b>: The lex-yacc like implementation of python. Will be used for parsing the grammar specification and tokenizing the line input

<b>pyreadline</b>: For windows platforms. This is the readline implementation for windows platforms

## Enduser Utilities

This package provides two classes for CLI building. NessaidCmd is intended to work like the standard Cmd class for simple CLI designs. It is a stripped down sub class of the more robest NessaidCli class. For the initial release only NessaidCmd is being documented here.

<b>NessaidCmd</b>: The basic Cmd like tool for end user. The CLI command handlers are defined as python methods with chosen prefix and the grammar definitions to drive them are expressed as the method docstring.

<b>NessaidCli</b>: The generic base class for the CLI impelemntation. NessaidCmd is the stripped down version as an alternative for the default Cmd implementation

## Installation
This package can be installed by cloning this repo and running python3 setup.py install from top directoy.
It's also available in pypi. <b>pip3 install nessaid_cli</b> will install the package.

## The Grammar
The grammar used to define the CLI commands is a simple context free grammar. The grammar specification can have definitions for individual tokens and multiple named grammars composed of the tokens.

## Tokens
Tokens are terminal symbols which are passed to the parser as strings. By default the CLI framework splits the input sequence with space as delimiter. If we use quoted string, we can have spaces inbetween words of a single token.
ie.
token1, "token2", "token 3" are 3 possible inputs for tokens

The following sub sections gives more details on token types

### Basic tokens
In the grammar we can express a basic token by either quoted string which will be used as a token matching to itself.

Or we can have a token definition like the following
```
token tokenx;
```
which will be used in the CLI as a token matching the string "tokenx"

### Token classes
We can specify the tokens by the following expressions
```
token token1 TokenClass1();
token token2 TokenClass2(arg1, arg2);
```
In this case token definitions will be generated by initializing objects of the classes with optional parameters. For example,

```
token str_token StringToken();
```
will generate a token which will match any string from the input

```
token integer RangedIntToken(-100, 200, 5);
```
will match any integer between -100 and 200. The third parameter in the above example is an extra argument we pass to limit the number of suggestion. If we add the token like this and type 2 and TAB from CLI, sine the number of possible matches are more than 5, ie 2, 20-29 ,and 200, the CLI will print the suggestion as "An integer between -100 and 200" If we type 20 and TAB we have only 2 possible completions, 20 and 200 so the CLI will give suggestion 20 and 200

NOTE: If we use token classes, we should override the get_token_classes method of NessaidCmd class so as to return the token classes used for the CLI. See examples in the end.

## Basic grammar
We can define all the tokens at the start of the CLI specification. And later we can use them in grammar definitions.

### Token definitions
```
token token1;
token token2;
token str_token StringToken();
token integer RangedIntToken(-100, 200, 5);
```

### Defining custom tokens
Please refer to README.tokens.md in the doc folder

### Token sequences
From the above tokens we can build a simple grammar like below

```
test_grammar:
    "test-string-constant-token"
    token1
    token2
    ;
```

This will match the only possible input

<b>test-string-constant-token token1 token2</b>

Now if we give the following definition

```
test_grammar2:
    str_token integer
    ;
```
it will match any string followed by an integer between -100 and 200

### Optional tokens
Any token or combination of tokens can be made optional by enclosing them in braces. For example modifying the above example,
```
test_grammar:
    "test-string-constant-token"
    { token1 }
    token2
    ;
```
will match the following possible inputs
```
test-string-constant-token token1 token2
test-string-constant-token token2
```

### Parentheses
Enclosing the tokens or combinations in parentheses will make it act as a single block when combining with other grammar blocks

### Alternatives
We can specify alternatives of tokens and combinations by the | (OR) symbol eg.

```
test_grammar:
    "test-string-constant-token"
    |
    (
        token1
        token2
    )
    ;
```
Will match either of the following
```
token1 token2
test-string-constant-token
```

### Multiplier
Any grammar block can be suffixed with a multiplier to match it more than once
```
test_grammar:
    (
        token1
        token2
    ) * 2
    ;
```
will match
```
token1 token2 token1 token2
```

```
test_grammar:
    "test-string-constant-token" * (1:3)
    ;
```
will match minimum 1 and maximum 3 repetitions of test-string-constant-token

### Named grammar references
The grammars can be referenced by their name from other grammars too, like normal tokens. See the example.
```
grammar1[$var]:
    "grammar1"
    <<$var = "From grammar 1";>>
    ;

grammar2[$var]:
    "grammar2"
    <<$var = $1;>>
    ;

main_grammar[]:
    <<
        $msg = "";
    >>
    (
        grammar1[$msg]
        |
        grammar2[$msg]
    )
    <<print($msg);>>
    ;
```
If we run the grammar <b>main_grammar</b> from CLI, it will match either of the strings "grammar1" or "grammar2" and because main_grammar passes it's local variable $msg to the sub grammars, they will fill the variable and when the print statement hits, we can see different messages getting printed according to which token we input. ie either "From grammar 1" or "grammar2" will get printed.

### CLI grammar actions
We can specify the actions to be executed when we match parts or the entire grammar. The actions are statements enclosed in <br><b> << </b> and <b> >> </b>
These actions will let us bind the grammar with our CLI and do operations based on grammar match

### Grammar variables
The grammar can incorporate the following types of variables

#### Grammar parameters
Sample usage
```
test_grammar[$param1, $param2]:
    <<
    print("Incoming params:", $param1, $param2);
    >>
    (
        "test-string-constant-token" * (1:3)
    )
    <<
        $param1 = 1;
        $param2 = 2;
    >>
    ;
```
If the grammar matches, the arguments passed to the grammar will be printed from the initial action defined. print there is an internal function call supported for the grammar actions. Also later the values of the parameters will be changed to 1 and 2 respectively.

#### Local variables
Sample usage
```
test_grammar[$param1, $param2]:
    <<
    $var1 = 1;
    $var2 = "Two";
    $var3 = list();
    $var4 = dict();
    $var5 = set();
    >>
    (
        "test-string-constant-token" * (1:3)
    )
    <<
        $var5 = append($var5, 1);
        $var4 = update($var4, "key", "value");
    >>
    ;
```

#### Token variables
The input values for tokens can be extracted using token variables. A sequence of terminal tokens will be numbered 1 to n and $1 to $n will hold the matched values
For example in the following grammar
```
test_grammar[$var1]:
    str_token <<$var1 = $1;>>
    integer << $var2 = $2;>>
    {
        integer << $var3 = $1; >>
        "test-string-constant-token" <<print($2);>>
    }
    ;
```
The statement
```
str_token <<$var1 = $1;>>
```
Will work like this. The $1 expression will extract the string input (str_token is defined to match any string)
and change the value of $var1, which is a variable passed to the grammar from outer levels. This inturn will modify the value of the variable in called context

```
integer << $var2 = $2;>>
```
This will create a local variable $var2 and set the value to the matched integer

```
integer << $var3 = $1; >>
```
A new variable. As this is inside a new optional block, this is a new sequence, so $1 will be used to hold the value

```
"test-string-constant-token" <<print($2);>>
```
This will print the string "test-string-constant-token" to console

### Calling inline functions
As part of the grammar actions, we can either call functions internal to the CLI framework or designated hook functions. Here are the inline functions supported
#### print
Works similar to python's print method. Accepts a number of printable arguments and retuns None
#### list
Creates a list object with optional arguments and returns the list
#### dict
Accepts no arguments and returns an empty dict. Can be filled with the update function call
#### set
Creates a set object with optional arguments and returns the set
#### append
Accepts at least two arguments. First argument is a list or set. The subsequent arguments are added to it and returned.
#### update
Accepts 3 arguments. 1st one is a dict the second one is key and 3rd one value. The dist is updated with {key: value}
#### add
Accepts n arguments and retuns the sum
#### dec/inc
Mathematical operations. accepts one argument. Argument should be a number. returns arg + 1 or arg -1 respectively

### Calling external functions
To call external functions, we should define them in the Cli/Cmd class. Then we can use it the following way.
```
$var = call func1();
call $func2(arg1, arg2);
```
We can override the get_cli_hook method to control the functions designated for calling from grammar. The method accepts the name of the function used in grammar and returns the actual method in the object which should be called.

### Comments in grammar
Comments can be added either in grammars portions or in action blocks
Single line comments can be done using either '#' (Hash)  or "//" (Double forward slash)
Multi line comments can be done by enclosing the text between /* and */

## How to run the CLI
Here's a basic example of running a custom grammar on CLI. However the initial release intend to focus on the Cmd utility, which automates most of the steps.
Here's the basic usage

The python code test.py
```python
import sys
from nessaid_cli.compiler import compile
from nessaid_cli.cli import NessaidCli

from nessaid_cli.tokens import (
    RangedIntToken,
    RangedStringToken
)

class TestCli(NessaidCli):

    def get_token_classes(self):
        """Method to override.
        It should return the list of token classes being used"""
        return [RangedIntToken, RangedStringToken]

    def exit(self):
        """This will be called from exit command of the CLI grammar"""
        self.exit_loop()


if __name__ == '__main__':
    with open("test_input.g") as fd:
        inp_str = fd.read()
        grammar_set = compile(inp_str)

    cli = TestCli(grammar_set, prompt="# ")
    try:
        cli.loop.run_until_complete(cli.cmdloop('test_grammar', intro="Starting Nessaid CLI Demo"))
        # 'test_grammar' above is the grammar to load with the CLI, part of test_input.g
    except KeyboardInterrupt:
        sys.exit(0)
    sys.exit(1)
```

The grammar file test_input.g
```
token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10

command1[$number_var]:
	"command1"
	TEST_NUMBER
	<<
		print("Incoming number is:", $number_var); # This uses inline print function
		call print("Input number is:", $2); /* Note call prefix. So routed to the CLI class's print function */
	>>
	;

command2[$str_var]:
	"command2"
	STRING_TOKEN
	<<
		print("Incoming str is:", $str_var);
		call print("Input str is:", $2);
	>>
	;

command3:
	"command3"
	<< print("This is simply command3"); >>
	;

exit:
	("exit" | "quit")
	<<call exit();>>
	;

test_grammar:
	<<
		$number = 5;
		$msg = "This is test_grammar";
	>>
	command1[$number]
	|
	command2[$msg]
	|
	command3
	|
	exit
	;
```

The CLI session (Windows,  \<ENTER> and \<TAB> means Enter and Tab key presses)
```bash
python test.py
Starting Nessaid CMD Demo
# <ENTER>
# <TAB>
command1
command2
command3
# command1 <TAB>
An integer between 1 and 100
# command1 101 <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# command1 100  <ENTER>
Incoming number is: 5
External function: Input number is: 100
# comm <TAB>
command1
command2
command3
# command2 <TAB>
Any string of length (5-10)
# command2 asdf <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# command2 asdfg <ENTER>
Incoming str is: This is test_grammar
External function: Input str is: asdfg
# command2 asdfgsSsSSS <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# comm <TAB>
command1
command2
command3
# command3 <ENTER>
This is simply command3
#
```

## Example usage of Cmd

Some of the steps in above CLI implementation is automated in NessaidCmd class, which is stripped down to be used like pythons cmd package. The Cmd loop is in async mode. Here we dont have to define each grammar, instead we will add the global token and shared grammar definitions as part of the class docstring. We will define methods in our class and grammar for each command will be given as the docstring for the method. When the grammar in the docstring matches, the corresponding method will be invoked. The methods designated as Cmd command handlers should be named with a common prefix. <b>do_</b> is the default prefix used in the class. Now let's try to implement the above CLI ecample using NessaidCmd class. Note that here we need only the python file. The grammar will be docstrings

```python
import sys
from nessaid_cli.cmd import NessaidCmd

from nessaid_cli.tokens import (
    RangedIntToken,
    RangedStringToken
)

class TestCmd(NessaidCmd):
    r"""
    token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
    token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10
    """

    def get_token_classes(self):
        """Method to override.
        It should return the list of token classes being used"""
        return [RangedIntToken, RangedStringToken]

    def do_command1(self, number):
        r"""
        "command1"
        TEST_NUMBER
        <<
            $number = $2;
        >>
        """

        """
        The Cmd framework here does the following

        1. Generate a named grammar corresponding to the do_command1 method
            with the method's docstring as grammar body.
        2. The grammar body will have local variables generated for each parameter
            of the method. ie here a local variable $number will be available for the
            grammar so that we can assign the input number to it in the action.
            If we don't assign anything the value of the variable will be ""
        3. Generate a master grammar which is the alternatives of the method grammars
            ie. Like

            master_grammar:
                do_command1
                |
                do_command2
                |
                do_command3
                ;

        4. When the grammar of a method matches the method will be called with the
            arguments.
        """

        # Now that the grammar matched, the function will be called with the arguments
        # we prepared. Process them in the function
        print("Incoming variable is not there. The feature is not yet ready for Cmd. This is the python print method")
        print("Input number is:", number)

    def do_command2(self, string):
        r"""
        "command2"
        STRING_TOKEN
        <<
            $string = $2;
            print("Inline print: Input number:", $2);
            call print("Input number:", $2);
        >>
        """
        print("Input str is:", string);

    def do_command3(self, string, number, numbers):
        r"""
        <<
            $numbers = list();
        >>

        "command3"
        STRING_TOKEN
        <<
            $string = $2;
        >>
        TEST_NUMBER
        <<
            $number = $3;
        >>
        # Now we may input an optional list with minimum 1 and maximum 3 integers
        {
            (
                TEST_NUMBER << $numbers = append($numbers, $1);>>
            ) * (1:3)
        }
        """
        print("Input str is:", string);
        print("Input number is:", number);
        print("Input list is:", numbers);


if __name__ == '__main__':
    cmd = TestCmd(prompt="nessaid-cmd # ", show_grammar=True)
    #show_grammar will print the generated grammar specification
    try:
        cmd.loop.run_until_complete(cmd.cmdloop(intro="Starting Nessaid CMD Demo"))
    except KeyboardInterrupt:
        sys.exit(0)
    except Exception as e:
        print("Exception in cmdloop:", e)
        sys.exit(1)
    sys.exit(0)
```

Now let's see the interactive CLI. The commands 'exit' 'quit' and 'end' are handled by base class with the method do__exit method to end the CLI session. We may override the function to disable, reconfigure it.

```bash
python test.py
# Generated CLI grammar:

token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10


do__exit[$arg_1, $arg_2, $arg_3]:
  (
    "exit" | "quit" | "end"

  )
  <<call do__exit();>>
  ;

do_command1[$arg_1, $arg_2, $arg_3]:
  <<
  $number = "";
  >>
  (
    "command1"
    TEST_NUMBER
    <<
        $number = $2;
    >>

  )
  <<call do_command1($number);>>
  ;

do_command2[$arg_1, $arg_2, $arg_3]:
  <<
  $string = "";
  >>
  (
    "command2"
    STRING_TOKEN
    <<
        $string = $2;
        print("Inline print: Input number:", $2);
        call print("Input number:", $2);
    >>

  )
  <<call do_command2($string);>>
  ;

do_command3[$arg_1, $arg_2, $arg_3]:
  <<
  $string = "";
  $number = "";
  $numbers = "";
  >>
  (
    <<
        $numbers = list();
    >>

    "command3"
    STRING_TOKEN
    <<
        $string = $2;
    >>
    TEST_NUMBER
    <<
        $number = $3;
    >>
    # Now we may input an optional list with minimum 1 and maximum 3 integers
    {
        (
            TEST_NUMBER << $numbers = append($numbers, $1);>>
        ) * (1:3)
    }

  )
  <<call do_command3($string, $number, $numbers);>>
  ;

TestCmd[]:
  do__exit
  |
  do_command1
  |
  do_command2
  |
  do_command3
  ;

Starting Nessaid CMD Demo
nessaid-cmd # <TAB>
command1
command2
command3
end
exit
quit
nessaid-cmd # c <TAB>
command1
command2
command3
nessaid-cmd # command <TAB>
command1
command2
command3
nessaid-cmd # command1 <TAB>
An integer between 1 and 100
nessaid-cmd # command1 55 <TAB>
NEWLINE: Complete command
nessaid-cmd # command1 55 <ENTER>
Incoming variable is not there. The feature is not yet ready for Cmd. This is the python print method
Input number is: 55
nessaid-cmd # comm <TAB>
command1
command2
command3
nessaid-cmd # command2 <TAB>
Any string of length (5-10)
nessaid-cmd # command2 as <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # command2 asdfgh <ENTER>
Inline print: Input number: asdfgh
External function: Input number: asdfgh
Input str is: asdfgh
nessaid-cmd # command2 asdfghhhhhhh <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # com <TAB>
command1
command2
command3
nessaid-cmd # command3 <TAB>
Any string of length (5-10)
nessaid-cmd # command3 12345
An integer between 1 and 100
nessaid-cmd # command3 12345 20 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 <TAB>
10
100
nessaid-cmd # command3 12345 20 10 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 <TAB>
Input str is: 12345
Input number is: 20
Input list is: [10, 11]
nessaid-cmd # command3 12345 20 10 11 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 30 <TAB>
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 30 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: [10, 11, 30]
nessaid-cmd # command3 12345 20 10 11 30 40 <TAB>
Failed to parse: Could not match any rule for this sequence
nessaid-cmd # command3 12345 20 10 11 30 40 <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # <ENTER>
nessaid-cmd # command3 12345 20 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: []
nessaid-cmd # command3 12345 20 15 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: [15]
nessaid-cmd # e <TAB>
end
exit
nessaid-cmd # ex <ENTER>
```

Note the parameters in the generated method grammars. They are dummy as of now.

## TODO
* Add support for non interactive processing, including processing a file content
* Support for importing other grammar files. The parser supports import statements but processing is not there.
* Add some kind of privilege states for tokens so that tokens will be matched only if the privilege level of the CLI instance is higher than that of the token.
* Compiler to generate a json or similar format for compiled grammar so that it can be loaded to CLI processors. Now the python parsed tree is used. If we can convert between json and the parse tree, I think it will be good to use across different languages as implementation of the compiler might be tedious in all languages.
* A proper way to feed variables to Cmd method grammars from calling python code
* Expensive validation and checks for recursive grammars. Basic recursive grammars are tested.
* Enhancement of the parsers to track the syntax and parsing errors clearly with line and column numbers. The current implementation on the part is partial. Sometimes, tough to track the grammar mistakes.
* Support for more inline functions and types. May be a way to register inline functions.
* Enhance NessaidCli to support Cli context stack, with each CLI context having dedicated prompt and environment
* Rework completion logic for Linux/Mac platforms. Now the tab completion is not as tidy as that of windows.
