ICS 142 Klefstad

Lexical Analysis Outline Sample Tasks Performed by a Lexical Analyzer Motivation for a Separate Lexical Analyzer Lexical Analysis Terms A Hand-coded Lexical Analyzer Language Theory Terms Operations on Strings Operations on Languages Regular Expressions (RE) Regular Expression Grammar Regular Definitions The Lex Compiler Limitations of Regular Languages Finite Automata Regular expressions to NFA NFA to DFA Optimization of DFA-based pattern matchers Handling Reserved Words

Sample Tasks Performed by a Lexical Analyzer

convert input characters into a sequence of tokens
strip out white space (blank, tab, newline)
strip out comments
macro expansion
keep track of input line and column numbers for error reporting
making a source listing

Motivation for a Separate Lexical Analyzer

follows good software engineering principles: EG separation of concerns
simplifies the specification of syntax
- EG describe lexical tokens via RE rather than context-free grammars
improves human efficiency
- EG automatic generators simplify scanner construction
improves machine efficiency
- EG use efficient input text buffering techniques, which is important since only the lexical analyzer looks at every input character
improves portability
- EG hides input method and character set

Lexical Analysis Terms

lexeme

a sequence of characters that is considered atomic

char *my_string = "Hello Bozo!"; 
has six lexemes:
"char" "*" "my_string" "=" ""Hello Bozo!"" ";"

token
- the abstract classification of a lexeme
- usually represented by an an enumeration type or unique integers)
- EG
```
TYPE token IS ('*', identifier, assign, string_literal, ';');
```
pattern
- a rule describing how to recognize the lexemes constituting a token class
- EG
```
IDENTIFIER --> a letter possibly followed by letters and digits
```

free format
- the positioning of language constructs within the source file is insignificant
key words
- pre-defined strings used as punctuation marks or to identify constructs
- EG
```
IF ... THEN BEGIN ... END
```
reserved words
- key words that cannot appear as user-defined identifiers
- EG in PL/1 key words are not reserved, therefore:
```
IF if THEN then; ELSE else;
is legal and semantically meaningful!
```
token attributes
- information associated with a token
- EG
  - location in the source text
  - specific name of identifiers or value of literal constants

A Hand-coded Lexical Analyzer

EG Lisp

   #include "lisp.h" /* defines s_expression, MAKE_ for various atoms,
      the variable atom_val of type s_expression, and the token
      values atom_token, '(', ')', and '.'. Topch() returns the
      next character in the input without reading it. */
  typedef int token;
   #define is_let_dig_or_opr(c) (IN(c,'0','9') || IN(c,'A','Z') IN(c,'a','z') || MEMBER(c, "+-*/"))
  char yytext[100];
  token yylex() {
    int ch, d;
    char *p;
  start_of_lexeme:
    switch (ch = getchar()) {
    case '(': case ')': case '.':
      return ch;
    case '"':
      p = yytext;
    nested_quote:
      while ((*p++ = getchar()) != '"')
        ;
      if (topch() == '"') {
        getchar();
        goto nested_quote;
      }
      *(p-1) = 0; /* remove the last quote */
      atom_val = MAKE_STRING(yytext);
      return atom_token;
    case '\'':
      atom_val = MAKE_CHARACTER(getchar());
      if (topch() == '\'') getchar();
      return atom_token;

Example scanner (continued)

    case '0': ... case '9':
      d = to_int(ch);
      while (isdigit(topch())) {
        d *= 10;
        d += to_int(getchar());
      }
      atom_val = MAKE_INTEGER(d);
      return atom_token;
    case '\n': case '\t': case ' ': case ',':
      goto start_of_lexeme;
    case '-':	/* skip comments "-- this is a comment\n"	*/
      if (topch() == '-') {
        while (getchar() != '\n')
          ;
      goto start_of_lexeme;
      }
      else if (isdigit(topch())) { /* a negative number */
        d = 0;
        do {
          d *= IBASE;
          d += to_int(getchar());
        } while (isdigit(topch()));
        atom_val = MAKE_INTEGER(-d);
        return atom_token;
      }
      /* else an operator, so drop into next case */
    default: /* a literal atom, includes operators and other names */
      p = yytext;
      do {

      *p++ = ch;
      ch = topch();

      } while (is_let_dig_or_opr(ch) && getchar());
      *p = 0;
      atom_val = intern(yytext);
      return atom_token;
    }
  }

Language Theory Terms

alphabet
- a finite set of symbols
- EG the ASCII character set, binary digits {0, 1}, greek alphabet, lower-case letters, etc.
string
- a finite sequence of symbols from an alphabet
language
- a (potentially infinite) set of strings over some fixed alphabet
- EG
```
Pascal, Ada, C, French, Chinese
```
the empty string
- a string of length 0
- usually denoted as an epsilon, EPSILON

Operations on Strings

concatenation
- the operation of appending two strings
- EG
```
x = "fu", y = "bar"
xy = "fubar"
```
exponentiation
- the operation of repeating a string or language
- defined formally as:
```
s^{0} = EPSILON, for i > 0, s^{i} = s^{i-1}s
```
prefix, suffix, substring, subsequence
- technical terms for particular portions of strings (see p. 93)

Operations on Languages

Suppose L is the set of all letters
- EG L = {a,b,...,z, A,B,...,Z}
Suppose D is the set of all digits
- EG D = {0,1,...,9}
union
- L|D - the set of letters and/or digits
concatenation
- LD - represents the set of all single letters followed by a single digit
fixed repetition
- L^{4} - is the set of four-letter strings

Kleene closure

L* - set of zero or more concatenations of L

D* - the set of all strings of digits including the empty string

positive closure
- L+ - set of one or more concatenations of L
- L+ is equivalent to LL*
- EG
```
D+ - the set of all strings of digits
```
character classes
- [xya-d] = matches any single character from the set [xyabcd]
- [^xya-d] = matches any single character except [xyabcd]
optionality
- L? - represents zero or one occurrence of a set
- L? is equivalent to L|EPSILON

Regular Expressions (RE)

a meta-language describing regular languages
an algebra defined recursively over the alphabet SIGMA:
- EPSILON is a RE denoting {EPSILON}, IE the empty string
- if a MEMBER SIGMA, a is a RE that denotes the set {a}
- if r and s are RE denoting languages L(r) and L(s):
  - r | s is a RE denoting L(r) \/ L(s)
  - (rs is a RE denoting L(r)L(s)
  - r*, r+, and r? are REs denoting L(r)*, L(r)+, and L(r)? respectively
  - (r) is a RE denoting L(r)

Regular Expression Grammar

the preceding description doesn't define operator precedence:
- closures '*' and '+' and optionality '?' have the highest precedence
- concatenation has the second highest
- alternation '|' has the lowest
context-free grammars also describe RE R --> T | R '|' T T --> F | T F F --> P | P '*' | P '+' | P '?' P --> '(' R ')' | ATOM
ATOM represents
- '.' (matches any single character)
- [chars] (character class)
- any escaped or non-escaped single character.

Regular Definitions

a sequence of definitions of the form:
```
d_{1} --> r_{1}
...
d_{n} --> r_{n}
```
where d_{i} is a distinct name and r_{i} is a regular expression over alphabet SIGMA
note that the r_{i}'s may include any of the d_{i-1} previous regular definitions

LETTER --> [A-Za-z]
DIGIT --> [0-9]
IDENTIFIER --> LETTER ( LETTER | DIGIT ) *

The Lex Compiler

Lex input file format

definitions
%%
pattern/action pairs
%%
optional auxiliary routines

DIGIT [0-9]
LETTER [a-zA-Z]
OPERATOR [-+*/]
%%
[().]  {return yytext[0];}
\'.\'  {atom_value = MAKE_CHARACTER(yytext[1]); return atom_token;}
(+|-)?{DIGIT}+  {atom_value = MAKE_INTEGER(yytext); return atom_token;}
({LETTER}|{OPERATOR})({LETTER}|{DIGIT}|{OPERATOR})*  {
  atom_value := INTERN(yytext); return atom_token;}
--.*  { /* nothing - just strip out and ignore comments */ }

Limitations of Regular Languages

regular languages only describe a limited subset of all languages
they cannot describe nesting constructs
for example, no RE can recognize the language of balanced parentheses
- IE S --> S '(' S ')' | EPSILON
a regular language can recognize either a fixed or unspecified number of repetitions

Finite Automata

A recognizer for a language is a program which decides whether or not a given string is in that language
A finite automaton (FA) is a state machine which recognizes the regular sets
A FA is defines as
- a set of states S
- a set of input symbols SIGMA (sigma)
- a transition function move that maps state-symbol pairs to sets of states
- a start state S_{0}
- a set of accepting (or final) states F

nondeterministic finite automata (NFA)
- may have multiple transitions out of a state on a given input symbol
- may have epsilon transitions, and match without reading any input!
- compact representation, linear in size of the RE
deterministic finite automata (DFA)
- no epsilon transitions
- only one transition per input symbol per state
- generally much faster recognizers than NFAs
- potentially MUCH larger; worst-case is exponential in the size of the RE
A transition graph is a state diagram representing a NFA
A transition diagram is a state diagram representing an DFA

Regular expressions to NFA

Thompson's construction
Algorithm on page 122 of the text

NFA to DFA

The subset construction
Algorithm on page 118 of the text

Optimization of DFA-based pattern matchers

Minimizing the states of a DFA
- Algorithm on page 142 of the text
Regular expressions to a DFA directly
- Algorithm on page 140 of the text
Table compression
- two dimensional array is fastest, but wastes space
- list of transitions with default, slower, but uses minimal space
- four array method is a compromise, moderate speed, moderate space
```
PROCEDURE nextstate(s: state; i: input) IS
BEGIN
  IF check(base(s)+a) THEN
    RETURN next(base(s)+a);
  ELSE
    RETURN nextstate(default(s)+a);
  END IF;
END nextstate;
```

Handling Reserved Words

reserved words usually match the pattern for identifiers
may be handled in one of two ways:
- each may be specified as a pattern and handled individually
  - causes lex table to grow quite large
- the identifier pattern can match and a table lookup will tell if it is reserved
  - keeps lex table small at cost of some run-time overhead