JavaScript 2.0 Lexer Semantics

July 2000 Draft

JavaScript 2.0

Formal Description

Lexer Semantics

Monday, December 6, 1999

The lexer semantics describe the actions the lexer takes in order to transform an input stream of Unicode characters into a stream of tokens. For convenience, the lexer grammar is repeated here. See also the description of the semantic notation.

This document is also available as a Word 98 rtf file.

The start symbols are: NextToken^unit if the previous token was a number; NextToken^re if the previous token was not a number and a / should be interpreted as a regular expression; and NextToken^div if the previous token was not a number and a / should be interpreted as a division or division-assignment operator.

Semantics

type SemanticException = oneof {syntaxError}

Unicode Character Classes

Syntax

UnicodeCharacter ⇒ Any Unicode character

UnicodeInitialAlphabetic ⇒ Any Unicode initial alphabetic character (includes ASCII A-Z and a-z)

UnicodeAlphanumeric ⇒ Any Unicode alphabetic or decimal digit character (includes ASCII 0-9, A-Z, and a-z)

WhiteSpaceCharacter ⇒

«TAB» | «VT» | «FF» | «SP» | «u00A0»

| «u2000» | «u2001» | «u2002» | «u2003» | «u2004» | «u2005» | «u2006» | «u2007»

| «u2008» | «u2009» | «u200A» | «u200B»

| «u3000»

LineTerminator ⇒ «LF» | «CR» | «u2028» | «u2029»

ASCIIDigit ⇒ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Semantics

action DecimalValue[ASCIIDigit] : Integer = digitValue(ASCIIDigit)

Comments

Syntax

LineComment ⇒ / / LineCommentCharacters

LineCommentCharacters ⇒

«empty»

| LineCommentCharacters NonTerminator

NonTerminator ⇒ UnicodeCharacter except LineTerminator

SingleLineBlockComment ⇒ / * BlockCommentCharacters * /

BlockCommentCharacters ⇒

«empty»

| BlockCommentCharacters NonTerminatorOrSlash

| PreSlashCharacters /

PreSlashCharacters ⇒

«empty»

| BlockCommentCharacters NonTerminatorOrAsteriskOrSlash

| PreSlashCharacters /

NonTerminatorOrSlash ⇒ NonTerminator except /

NonTerminatorOrAsteriskOrSlash ⇒ NonTerminator except * | /

MultiLineBlockComment ⇒ / * MultiLineBlockCommentCharacters BlockCommentCharacters * /

MultiLineBlockCommentCharacters ⇒

BlockCommentCharacters LineTerminator

| MultiLineBlockCommentCharacters BlockCommentCharacters LineTerminator

White space

Syntax

WhiteSpace ⇒

«empty»

| WhiteSpace WhiteSpaceCharacter

| WhiteSpace SingleLineBlockComment

Line breaks

Syntax

LineBreak ⇒

LineTerminator

| LineComment LineTerminator

| MultiLineBlockComment

LineBreaks ⇒

LineBreak

| LineBreaks WhiteSpace LineBreak

Tokens

Syntax

τ ∈ {re, div, unit}

NextToken^re ⇒ WhiteSpace Token^re

NextToken^div ⇒ WhiteSpace Token^div

NextToken^unit ⇒

[lookahead∉{OrdinaryContinuingIdentifierCharacter, \}] WhiteSpace Token^div

| [lookahead∉{_}] IdentifierName

| _ IdentifierName

Semantics

action Token[NextToken^τ] : Token

Token[NextToken^re ⇒ WhiteSpace Token^re] = Token[Token^re]

Token[NextToken^div ⇒ WhiteSpace Token^div] = Token[Token^div]

Token[NextToken^unit ⇒ [lookahead∉{OrdinaryContinuingIdentifierCharacter, \}] WhiteSpace Token^div]
= Token[Token^div]

Token[NextToken^unit ⇒ [lookahead∉{_}] IdentifierName] = string Name[IdentifierName]

Token[NextToken^unit ⇒ _ IdentifierName] = string Name[IdentifierName]

Syntax

Token^re ⇒

LineBreaks

| IdentifierOrReservedWord

| Punctuator

| NumericLiteral

| StringLiteral

| RegExpLiteral

| EndOfInput

Token^div ⇒

LineBreaks

| IdentifierOrReservedWord

| Punctuator

| DivisionPunctuator

| NumericLiteral

| StringLiteral

| EndOfInput

EndOfInput ⇒

End

| LineComment End

Semantics

type RegExp = tuple {reBody: String; reFlags: String}

type Quantity = tuple {amount: Double; unit: String}

type Token
  = oneof {
           lineBreak;
           identifier: String;
           keyword: String;
           punctuator: String;
           number: Double;
           string: String;
           regularExpression: RegExp;
           end}

action Token[Token^τ] : Token

Token[Token^τ ⇒ LineBreaks] = lineBreak

Token[Token^τ ⇒ IdentifierOrReservedWord] = Token[IdentifierOrReservedWord]

Token[Token^τ ⇒ Punctuator] = punctuator Punctuator[Punctuator]

Token[Token^div ⇒ DivisionPunctuator] = punctuator Punctuator[DivisionPunctuator]

Token[Token^τ ⇒ NumericLiteral] = number DoubleValue[NumericLiteral]

Token[Token^τ ⇒ StringLiteral] = string StringValue[StringLiteral]

Token[Token^re ⇒ RegExpLiteral] = regularExpression REValue[RegExpLiteral]

Token[Token^τ ⇒ EndOfInput] = end

Keywords and identifiers

Syntax

IdentifierName ⇒

InitialIdentifierCharacter

| IdentifierName ContinuingIdentifierCharacter

InitialIdentifierCharacter ⇒

OrdinaryInitialIdentifierCharacter

| \ HexEscape

OrdinaryInitialIdentifierCharacter ⇒ UnicodeInitialAlphabetic | $ | _

ContinuingIdentifierCharacter ⇒

OrdinaryContinuingIdentifierCharacter

| \ HexEscape

OrdinaryContinuingIdentifierCharacter ⇒ UnicodeAlphanumeric | $ | _

Semantics

action Name[IdentifierName] : String

Name[IdentifierName ⇒ InitialIdentifierCharacter]
= [CharacterValue[InitialIdentifierCharacter]]

Name[IdentifierName ⇒ IdentifierName₁ ContinuingIdentifierCharacter]
= Name[IdentifierName₁] ⊕ [CharacterValue[ContinuingIdentifierCharacter]]

action ContainsEscapes[IdentifierName] : Boolean

ContainsEscapes[IdentifierName ⇒ InitialIdentifierCharacter]
= ContainsEscapes[InitialIdentifierCharacter]

ContainsEscapes[IdentifierName ⇒ IdentifierName₁ ContinuingIdentifierCharacter]
= ContainsEscapes[IdentifierName₁] or ContainsEscapes[ContinuingIdentifierCharacter]

action CharacterValue[InitialIdentifierCharacter] : Character

CharacterValue[InitialIdentifierCharacter ⇒ OrdinaryInitialIdentifierCharacter]
= OrdinaryInitialIdentifierCharacter

CharacterValue[InitialIdentifierCharacter ⇒ \ HexEscape]
  = if isOrdinaryInitialIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else throw syntaxError

action ContainsEscapes[InitialIdentifierCharacter] : Boolean

ContainsEscapes[InitialIdentifierCharacter ⇒ OrdinaryInitialIdentifierCharacter] = false

ContainsEscapes[InitialIdentifierCharacter ⇒ \ HexEscape] = true

action CharacterValue[ContinuingIdentifierCharacter] : Character

CharacterValue[ContinuingIdentifierCharacter ⇒ OrdinaryContinuingIdentifierCharacter]
= OrdinaryContinuingIdentifierCharacter

CharacterValue[ContinuingIdentifierCharacter ⇒ \ HexEscape]
  = if isOrdinaryContinuingIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else throw syntaxError

action ContainsEscapes[ContinuingIdentifierCharacter] : Boolean

ContainsEscapes[ContinuingIdentifierCharacter ⇒ OrdinaryContinuingIdentifierCharacter]
= false

ContainsEscapes[ContinuingIdentifierCharacter ⇒ \ HexEscape] = true

reservedWords : String[]
  = [“abstract”,
      “break”,
      “case”,
      “catch”,
      “class”,
      “const”,
      “continue”,
      “debugger”,
      “default”,
      “delete”,
      “do”,
      “else”,
      “enum”,
      “eval”,
      “export”,
      “extends”,
      “false”,
      “final”,
      “finally”,
      “for”,
      “function”,
      “goto”,
      “if”,
      “implements”,
      “import”,
      “in”,
      “instanceof”,
      “native”,
      “new”,
      “null”,
      “package”,
      “private”,
      “protected”,
      “public”,
      “return”,
      “static”,
      “super”,
      “switch”,
      “synchronized”,
      “this”,
      “throw”,
      “throws”,
      “transient”,
      “true”,
      “try”,
      “typeof”,
      “var”,
      “volatile”,
      “while”,
      “with”]

nonReservedWords : String[]
  = [“box”,
      “constructor”,
      “field”,
      “get”,
      “language”,
      “local”,
      “method”,
      “override”,
      “set”,
      “version”]

keywords : String[] = reservedWords ⊕ nonReservedWords

member(id: String, list: String[]) : Boolean
  = if |list| = 0
     then false
     else if id = list[0]
     then true
     else member(id, list[1 ...])

Syntax

IdentifierOrReservedWord ⇒ IdentifierName

Semantics

action Token[IdentifierOrReservedWord] : Token

Token[IdentifierOrReservedWord ⇒ IdentifierName]
  = let id: String = Name[IdentifierName]
     in if member(id, keywords) and not ContainsEscapes[IdentifierName]
         then keyword id
         else identifier id

Punctuators

Syntax

Punctuator ⇒

!

| ! =

| ! = =

| #

| %

| % =

| &

| & &

| & & =

| & =

| (

| )

| *

| * =

| +

| + +

| + =

| ,

| -

| - -

| - =

| - >

| .

| . .

| . . .

| :

| : :

| ;

| <

| < <

| < < =

| < =

| =

| = =

| = = =

| >

| > =

| > >

| > > =

| > > >

| > > > =

| ?

| @

| [

| ]

| ^

| ^ =

| ^ ^

| ^ ^ =

| {

| |

| | =

| | |

| | | =

| }

| ~

DivisionPunctuator ⇒

/ [lookahead∉{/, *}]

| / =

Semantics

action Punctuator[Punctuator] : String

Punctuator[Punctuator ⇒ !] = “!”

Punctuator[Punctuator ⇒ ! =] = “!=”

Punctuator[Punctuator ⇒ ! = =] = “!==”

Punctuator[Punctuator ⇒ #] = “#”

Punctuator[Punctuator ⇒ %] = “%”

Punctuator[Punctuator ⇒ % =] = “%=”

Punctuator[Punctuator ⇒ &] = “&”

Punctuator[Punctuator ⇒ & &] = “&&”

Punctuator[Punctuator ⇒ & & =] = “&&=”

Punctuator[Punctuator ⇒ & =] = “&=”

Punctuator[Punctuator ⇒ (] = “(”

Punctuator[Punctuator ⇒ )] = “)”

Punctuator[Punctuator ⇒ *] = “*”

Punctuator[Punctuator ⇒ * =] = “*=”

Punctuator[Punctuator ⇒ +] = “+”

Punctuator[Punctuator ⇒ + +] = “++”

Punctuator[Punctuator ⇒ + =] = “+=”

Punctuator[Punctuator ⇒ ,] = “,”

Punctuator[Punctuator ⇒ -] = “-”

Punctuator[Punctuator ⇒ - -] = “--”

Punctuator[Punctuator ⇒ - =] = “-=”

Punctuator[Punctuator ⇒ - >] = “->”

Punctuator[Punctuator ⇒ .] = “.”

Punctuator[Punctuator ⇒ . .] = “..”

Punctuator[Punctuator ⇒ . . .] = “...”

Punctuator[Punctuator ⇒ :] = “:”

Punctuator[Punctuator ⇒ : :] = “::”

Punctuator[Punctuator ⇒ ;] = “;”

Punctuator[Punctuator ⇒ <] = “<”

Punctuator[Punctuator ⇒ < <] = “<<”

Punctuator[Punctuator ⇒ < < =] = “<<=”

Punctuator[Punctuator ⇒ < =] = “<=”

Punctuator[Punctuator ⇒ =] = “=”

Punctuator[Punctuator ⇒ = =] = “==”

Punctuator[Punctuator ⇒ = = =] = “===”

Punctuator[Punctuator ⇒ >] = “>”

Punctuator[Punctuator ⇒ > =] = “>=”

Punctuator[Punctuator ⇒ > >] = “>>”

Punctuator[Punctuator ⇒ > > =] = “>>=”

Punctuator[Punctuator ⇒ > > >] = “>>>”

Punctuator[Punctuator ⇒ > > > =] = “>>>=”

Punctuator[Punctuator ⇒ ?] = “?”

Punctuator[Punctuator ⇒ @] = “@”

Punctuator[Punctuator ⇒ [] = “[”

Punctuator[Punctuator ⇒ ]] = “]”

Punctuator[Punctuator ⇒ ^] = “^”

Punctuator[Punctuator ⇒ ^ =] = “^=”

Punctuator[Punctuator ⇒ ^ ^] = “^^”

Punctuator[Punctuator ⇒ ^ ^ =] = “^^=”

Punctuator[Punctuator ⇒ {] = “{”

Punctuator[Punctuator ⇒ |] = “|”

Punctuator[Punctuator ⇒ | =] = “|=”

Punctuator[Punctuator ⇒ | |] = “||”

Punctuator[Punctuator ⇒ | | =] = “||=”

Punctuator[Punctuator ⇒ }] = “}”

Punctuator[Punctuator ⇒ ~] = “~”

action Punctuator[DivisionPunctuator] : String

Punctuator[DivisionPunctuator ⇒ / [lookahead∉{/, *}]] = “/”

Punctuator[DivisionPunctuator ⇒ / =] = “/=”

Numeric literals

Syntax

NumericLiteral ⇒

DecimalLiteral

| HexIntegerLiteral [lookahead∉{HexDigit}]

Semantics

action DoubleValue[NumericLiteral] : Double

DoubleValue[NumericLiteral ⇒ DecimalLiteral]
= rationalToDouble(RationalValue[DecimalLiteral])

DoubleValue[NumericLiteral ⇒ HexIntegerLiteral [lookahead∉{HexDigit}]]
= rationalToDouble(IntegerValue[HexIntegerLiteral])

expt(base: Rational, exponent: Integer) : Rational
  = if exponent = 0
     then 1
     else if exponent < 0
     then 1/expt(base, -exponent)
     else base*expt(base, exponent - 1)

Syntax

DecimalLiteral ⇒

Mantissa

| Mantissa LetterE SignedInteger

LetterE ⇒ E | e

Mantissa ⇒

DecimalIntegerLiteral

| DecimalIntegerLiteral .

| DecimalIntegerLiteral . Fraction

| . Fraction

DecimalIntegerLiteral ⇒

0

| NonZeroDecimalDigits

NonZeroDecimalDigits ⇒

NonZeroDigit

| NonZeroDecimalDigits ASCIIDigit

NonZeroDigit ⇒ 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Fraction ⇒ DecimalDigits

Semantics

action RationalValue[DecimalLiteral] : Rational

RationalValue[DecimalLiteral ⇒ Mantissa] = RationalValue[Mantissa]

RationalValue[DecimalLiteral ⇒ Mantissa LetterE SignedInteger]
= RationalValue[Mantissa]*expt(10, IntegerValue[SignedInteger])

action RationalValue[Mantissa] : Rational

RationalValue[Mantissa ⇒ DecimalIntegerLiteral] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa ⇒ DecimalIntegerLiteral .] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa ⇒ DecimalIntegerLiteral . Fraction]
= IntegerValue[DecimalIntegerLiteral] + RationalValue[Fraction]

RationalValue[Mantissa ⇒ . Fraction] = RationalValue[Fraction]

action IntegerValue[DecimalIntegerLiteral] : Integer

IntegerValue[DecimalIntegerLiteral ⇒ 0] = 0

IntegerValue[DecimalIntegerLiteral ⇒ NonZeroDecimalDigits]
= IntegerValue[NonZeroDecimalDigits]

action IntegerValue[NonZeroDecimalDigits] : Integer

IntegerValue[NonZeroDecimalDigits ⇒ NonZeroDigit] = DecimalValue[NonZeroDigit]

IntegerValue[NonZeroDecimalDigits ⇒ NonZeroDecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[NonZeroDecimalDigits₁] + DecimalValue[ASCIIDigit]

action DecimalValue[NonZeroDigit] : Integer = digitValue(NonZeroDigit)

action RationalValue[Fraction] : Rational

RationalValue[Fraction ⇒ DecimalDigits]
= IntegerValue[DecimalDigits]/expt(10, NDigits[DecimalDigits])

Syntax

SignedInteger ⇒

DecimalDigits

| + DecimalDigits

| - DecimalDigits

Semantics

action IntegerValue[SignedInteger] : Integer

IntegerValue[SignedInteger ⇒ DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger ⇒ + DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger ⇒ - DecimalDigits] = -IntegerValue[DecimalDigits]

Syntax

DecimalDigits ⇒

ASCIIDigit

| DecimalDigits ASCIIDigit

Semantics

action IntegerValue[DecimalDigits] : Integer

IntegerValue[DecimalDigits ⇒ ASCIIDigit] = DecimalValue[ASCIIDigit]

IntegerValue[DecimalDigits ⇒ DecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[DecimalDigits₁] + DecimalValue[ASCIIDigit]

action NDigits[DecimalDigits] : Integer

NDigits[DecimalDigits ⇒ ASCIIDigit] = 1

NDigits[DecimalDigits ⇒ DecimalDigits₁ ASCIIDigit] = NDigits[DecimalDigits₁] + 1

Syntax

HexIntegerLiteral ⇒

0 LetterX HexDigit

| HexIntegerLiteral HexDigit

LetterX ⇒ X | x

HexDigit ⇒ 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | a | b | c | d | e | f

Semantics

action IntegerValue[HexIntegerLiteral] : Integer

IntegerValue[HexIntegerLiteral ⇒ 0 LetterX HexDigit] = HexValue[HexDigit]

IntegerValue[HexIntegerLiteral ⇒ HexIntegerLiteral₁ HexDigit]
= 16*IntegerValue[HexIntegerLiteral₁] + HexValue[HexDigit]

action HexValue[HexDigit] : Integer = digitValue(HexDigit)

String literals

Syntax

θ ∈ {single, double}

StringLiteral ⇒

' StringChars^single '

| " StringChars^double "

Semantics

action StringValue[StringLiteral] : String

StringValue[StringLiteral ⇒ ' StringChars^single '] = StringValue[StringChars^single]

StringValue[StringLiteral ⇒ " StringChars^double "] = StringValue[StringChars^double]

Syntax

StringChars^θ ⇒

«empty»

| StringChars^θ StringChar^θ

StringChar^θ ⇒

LiteralStringChar^θ

| \ StringEscape

LiteralStringChar^single ⇒ UnicodeCharacter except ' | \ | LineTerminator

LiteralStringChar^double ⇒ UnicodeCharacter except " | \ | LineTerminator

Semantics

action StringValue[StringChars^θ] : String

StringValue[StringChars^θ ⇒ «empty»] = “”

StringValue[StringChars^θ ⇒ StringChars^θ₁ StringChar^θ]
= StringValue[StringChars^θ₁] ⊕ [CharacterValue[StringChar^θ]]

action CharacterValue[StringChar^θ] : Character

CharacterValue[StringChar^θ ⇒ LiteralStringChar^θ] = LiteralStringChar^θ

CharacterValue[StringChar^θ ⇒ \ StringEscape] = CharacterValue[StringEscape]

Syntax

StringEscape ⇒

ControlEscape

| ZeroEscape

| HexEscape

| IdentityEscape

IdentityEscape ⇒ NonTerminator except UnicodeAlphanumeric

Semantics

action CharacterValue[StringEscape] : Character

CharacterValue[StringEscape ⇒ ControlEscape] = CharacterValue[ControlEscape]

CharacterValue[StringEscape ⇒ ZeroEscape] = CharacterValue[ZeroEscape]

CharacterValue[StringEscape ⇒ HexEscape] = CharacterValue[HexEscape]

CharacterValue[StringEscape ⇒ IdentityEscape] = IdentityEscape

Syntax

ControlEscape ⇒

b

| f

| n

| r

| t

| v

Semantics

action CharacterValue[ControlEscape] : Character

CharacterValue[ControlEscape ⇒ b] = ‘«BS»’

CharacterValue[ControlEscape ⇒ f] = ‘«FF»’

CharacterValue[ControlEscape ⇒ n] = ‘«LF»’

CharacterValue[ControlEscape ⇒ r] = ‘«CR»’

CharacterValue[ControlEscape ⇒ t] = ‘«TAB»’

CharacterValue[ControlEscape ⇒ v] = ‘«VT»’

Syntax

ZeroEscape ⇒ 0 [lookahead∉{ASCIIDigit}]

Semantics

action CharacterValue[ZeroEscape] : Character

CharacterValue[ZeroEscape ⇒ 0 [lookahead∉{ASCIIDigit}]] = ‘«NUL»’

Syntax

HexEscape ⇒

x HexDigit HexDigit

| u HexDigit HexDigit HexDigit HexDigit

Semantics

action CharacterValue[HexEscape] : Character

CharacterValue[HexEscape ⇒ x HexDigit₁ HexDigit₂]
= codeToCharacter(16*HexValue[HexDigit₁] + HexValue[HexDigit₂])

CharacterValue[HexEscape ⇒ u HexDigit₁ HexDigit₂ HexDigit₃ HexDigit₄]
  = codeToCharacter(
         4096*HexValue[HexDigit₁] + 256*HexValue[HexDigit₂] + 16*HexValue[HexDigit₃] +
         HexValue[HexDigit₄])