Lexical Analyzer

Lexical Analyzer is the first phase of my compiler (Genepiler). It's planned to join my 'Oh My Compiler' project!

This phase converts the source code (based on the lexical file) into tokens, to be processed later in the second phase - parser-generator

---

See the next phases 👇

• Parser Generator
• Intermediate Code Generator

---

Tasks

• input to RD
• add concat to RD (needs improvement)[done]
• RD to prefix
• node
• build AST tree
• show tree
• get nullable
• get firstpos
• get lastpos
• get followpos
• RD to DFA
• Simulate DFA
• make unique id for leaves only (expect for epsilon)
• add + and ? to Regex
• Minimize DFA (algorithm already minimize it (lex algorithm))
• add RE nodes to RD tree
• append keywords and punctuation
• format input code
• make sub DFAs for each RD
• subs REs in RDs
• sep REs
• sep keywords (not really)
• sep punctuation (not really)
• tabulate DFA table
• test the given test cases by diffmerge tool
• consider module file to add your modules in
• replace state names with smaller unique names
• add arguments (argparse)
• consider adding directory for input and directory for output
• add DFA table to Lexical (from lexical_aux)
• add Lexical to modules and run from main
• report

New Tasks:

• Enhance the lexical file format

---

Notes (related to the internal structure):

• rd to be entered without '#'
• add '#' . directly to post-fix expression to avoid confusion
• dfa_simulate_2 is recursive it returns output in reverse order
• as you replace 'e' with its symbol, replace '(' with LBRKT and ')' RBRKT and mind that in postfix

---

[Memory] Plan (schedule)

• Today: 23 May 2020

• Actual Deadline: 6 Jun 2020

• Phase Deadline: 29 May 2020

  Schedule:

• 23/5: Getting to know what's going on

• (24-29)/5: Building the program (see todo)