Returns the row and column for the fault token detected in the Parser module.

Specs

scs source code string with special characters.

otl output token list generated by the Parser.

etl error token list and is the second elements of the error_cause attribute produced by the Parser. This list contains all the fault tokens.

Returns an output list of tokens (OTL) by breaking the source code string (SCS).

Specs

arg is a tuple that contains the following elements:

• source_code_string: source code string.
• general_token_list: target language's valid token list. Refer to Reader.read_code_and_tokens/3 for more information.

Examples

iex> source = Reader.read_code_and_tokens("examples/test_s2.c", 
    "specification_files/c_tokens.xml")
iex> Lexer.tokenize(source)