JavaScript 2.0 Lexer Semantics

July 2000 Draft

JavaScript 2.0

Formal Description

Lexer Semantics

Monday, December 6, 1999

The lexer semantics describe the actions the lexer takes in order to transform an input stream of Unicode characters into a stream of tokens. For convenience, the lexer grammar is repeated here. See also the description of the semantic notation.

This document is also available as a Word 98 rtf file.

The start symbols are: NextToken^unit if the previous token was a number; NextToken^re if the previous token was not a number and a / should be interpreted as a regular expression; and NextToken^div if the previous token was not a number and a / should be interpreted as a division or division-assignment operator.

Semantics

type SemanticException = oneof {syntaxError}

Unicode Character Classes

Syntax

UnicodeCharacter Any Unicode character

UnicodeInitialAlphabetic Any Unicode initial alphabetic character (includes ASCII A-Z and a-z)

UnicodeAlphanumeric Any Unicode alphabetic or decimal digit character (includes ASCII 0-9, A-Z, and a-z)

WhiteSpaceCharacter

«TAB» | «VT» | «FF» | «SP» | «u00A0»

| «u2000» | «u2001» | «u2002» | «u2003» | «u2004» | «u2005» | «u2006» | «u2007»

| «u2008» | «u2009» | «u200A» | «u200B»

| «u3000»

LineTerminator «LF» | «CR» | «u2028» | «u2029»

ASCIIDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Semantics

action DecimalValue[ASCIIDigit] : Integer = digitValue(ASCIIDigit)

Comments

Syntax

LineComment / / LineCommentCharacters

LineCommentCharacters

«empty»

| LineCommentCharacters NonTerminator

NonTerminator UnicodeCharacter except LineTerminator

SingleLineBlockComment / * BlockCommentCharacters * /

BlockCommentCharacters

«empty»

| BlockCommentCharacters NonTerminatorOrSlash

| PreSlashCharacters /

PreSlashCharacters

«empty»

| BlockCommentCharacters NonTerminatorOrAsteriskOrSlash

| PreSlashCharacters /

NonTerminatorOrSlash NonTerminator except /

NonTerminatorOrAsteriskOrSlash NonTerminator except * | /

MultiLineBlockComment / * MultiLineBlockCommentCharacters BlockCommentCharacters * /

MultiLineBlockCommentCharacters

BlockCommentCharacters LineTerminator

| MultiLineBlockCommentCharacters BlockCommentCharacters LineTerminator

White space

Syntax

WhiteSpace

«empty»

| WhiteSpace WhiteSpaceCharacter

| WhiteSpace SingleLineBlockComment

Line breaks

Syntax

LineBreak

LineTerminator

| LineComment LineTerminator

| MultiLineBlockComment

LineBreaks

LineBreak

| LineBreaks WhiteSpace LineBreak

Tokens

Syntax

{re, div, unit}

NextToken^re WhiteSpace Token^re

NextToken^div WhiteSpace Token^div

NextToken^unit

[lookahead{OrdinaryContinuingIdentifierCharacter, \}] WhiteSpace Token^div

| [lookahead{_}] IdentifierName

| _ IdentifierName

Semantics

action Token[NextToken] : Token

Token[NextToken^re WhiteSpace Token^re] = Token[Token^re]

Token[NextToken^div WhiteSpace Token^div] = Token[Token^div]

Token[NextToken^unit [lookahead{OrdinaryContinuingIdentifierCharacter, \}] WhiteSpace Token^div]
= Token[Token^div]

Token[NextToken^unit [lookahead{_}] IdentifierName] = string Name[IdentifierName]

Token[NextToken^unit _ IdentifierName] = string Name[IdentifierName]

Syntax

Token^re

LineBreaks

| IdentifierOrReservedWord

| Punctuator

| NumericLiteral

| StringLiteral

| RegExpLiteral

| EndOfInput

Token^div

LineBreaks

| IdentifierOrReservedWord

| Punctuator

| DivisionPunctuator

| NumericLiteral

| StringLiteral

| EndOfInput

EndOfInput

End

| LineComment End

Semantics

type RegExp = tuple {reBody: String; reFlags: String}

type Quantity = tuple {amount: Double; unit: String}

type Token
  = oneof {
           lineBreak;
           identifier: String;
           keyword: String;
           punctuator: String;
           number: Double;
           string: String;
           regularExpression: RegExp;
           end}

action Token[Token] : Token

Token[Token LineBreaks] = lineBreak

Token[Token IdentifierOrReservedWord] = Token[IdentifierOrReservedWord]

Token[Token Punctuator] = punctuator Punctuator[Punctuator]

Token[Token^div DivisionPunctuator] = punctuator Punctuator[DivisionPunctuator]

Token[Token NumericLiteral] = number DoubleValue[NumericLiteral]

Token[Token StringLiteral] = string StringValue[StringLiteral]

Token[Token^re RegExpLiteral] = regularExpression REValue[RegExpLiteral]

Token[Token EndOfInput] = end

Keywords and identifiers

Syntax

IdentifierName

InitialIdentifierCharacter

| IdentifierName ContinuingIdentifierCharacter

InitialIdentifierCharacter

OrdinaryInitialIdentifierCharacter

| \ HexEscape

OrdinaryInitialIdentifierCharacter UnicodeInitialAlphabetic | $ | _

ContinuingIdentifierCharacter

OrdinaryContinuingIdentifierCharacter

| \ HexEscape

OrdinaryContinuingIdentifierCharacter UnicodeAlphanumeric | $ | _

Semantics

action Name[IdentifierName] : String

Name[IdentifierName InitialIdentifierCharacter]
= [CharacterValue[InitialIdentifierCharacter]]

Name[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= Name[IdentifierName₁] [CharacterValue[ContinuingIdentifierCharacter]]

action ContainsEscapes[IdentifierName] : Boolean

ContainsEscapes[IdentifierName InitialIdentifierCharacter]
= ContainsEscapes[InitialIdentifierCharacter]

ContainsEscapes[IdentifierName IdentifierName₁ ContinuingIdentifierCharacter]
= ContainsEscapes[IdentifierName₁] or ContainsEscapes[ContinuingIdentifierCharacter]

action CharacterValue[InitialIdentifierCharacter] : Character

CharacterValue[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter]
= OrdinaryInitialIdentifierCharacter

CharacterValue[InitialIdentifierCharacter  \ HexEscape]
  = if isOrdinaryInitialIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else throw syntaxError

action ContainsEscapes[InitialIdentifierCharacter] : Boolean

ContainsEscapes[InitialIdentifierCharacter OrdinaryInitialIdentifierCharacter] = false

ContainsEscapes[InitialIdentifierCharacter \ HexEscape] = true

action CharacterValue[ContinuingIdentifierCharacter] : Character

CharacterValue[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= OrdinaryContinuingIdentifierCharacter

CharacterValue[ContinuingIdentifierCharacter  \ HexEscape]
  = if isOrdinaryContinuingIdentifierCharacter(CharacterValue[HexEscape])
     then CharacterValue[HexEscape]
     else throw syntaxError

action ContainsEscapes[ContinuingIdentifierCharacter] : Boolean

ContainsEscapes[ContinuingIdentifierCharacter OrdinaryContinuingIdentifierCharacter]
= false

ContainsEscapes[ContinuingIdentifierCharacter \ HexEscape] = true

reservedWords : String[]
  = [“abstract”,
      “break”,
      “case”,
      “catch”,
      “class”,
      “const”,
      “continue”,
      “debugger”,
      “default”,
      “delete”,
      “do”,
      “else”,
      “enum”,
      “eval”,
      “export”,
      “extends”,
      “false”,
      “final”,
      “finally”,
      “for”,
      “function”,
      “goto”,
      “if”,
      “implements”,
      “import”,
      “in”,
      “instanceof”,
      “native”,
      “new”,
      “null”,
      “package”,
      “private”,
      “protected”,
      “public”,
      “return”,
      “static”,
      “super”,
      “switch”,
      “synchronized”,
      “this”,
      “throw”,
      “throws”,
      “transient”,
      “true”,
      “try”,
      “typeof”,
      “var”,
      “volatile”,
      “while”,
      “with”]

nonReservedWords : String[]
  = [“box”,
      “constructor”,
      “field”,
      “get”,
      “language”,
      “local”,
      “method”,
      “override”,
      “set”,
      “version”]

keywords : String[] = reservedWords nonReservedWords

member(id: String, list: String[]) : Boolean
  = if |list| = 0
     then false
     else if id = list[0]
     then true
     else member(id, list[1 ...])

Syntax

IdentifierOrReservedWord IdentifierName

Semantics

action Token[IdentifierOrReservedWord] : Token

Token[IdentifierOrReservedWord  IdentifierName]
  = let id: String = Name[IdentifierName]
     in if member(id, keywords) and not ContainsEscapes[IdentifierName]
         then keyword id
         else identifier id

Punctuators

Syntax

Punctuator

!

| ! =

| ! = =

| #

| %

| % =

| &

| & &

| & & =

| & =

| (

| )

| *

| * =

| +

| + +

| + =

| ,

| -

| - -

| - =

| - >

| .

| . .

| . . .

| :

| : :

| ;

| <

| < <

| < < =

| < =

| =

| = =

| = = =

| >

| > =

| > >

| > > =

| > > >

| > > > =

| ?

| @

| [

| ]

| ^

| ^ =

| ^ ^

| ^ ^ =

| {

| |

| | =

| | |

| | | =

| }

| ~

DivisionPunctuator

/ [lookahead{/, *}]

| / =

Semantics

action Punctuator[Punctuator] : String

Punctuator[Punctuator !] = “!”

Punctuator[Punctuator ! =] = “!=”

Punctuator[Punctuator ! = =] = “!==”

Punctuator[Punctuator #] = “#”

Punctuator[Punctuator %] = “%”

Punctuator[Punctuator % =] = “%=”

Punctuator[Punctuator &] = “&”

Punctuator[Punctuator & &] = “&&”

Punctuator[Punctuator & & =] = “&&=”

Punctuator[Punctuator & =] = “&=”

Punctuator[Punctuator (] = “(”

Punctuator[Punctuator )] = “)”

Punctuator[Punctuator *] = “*”

Punctuator[Punctuator * =] = “*=”

Punctuator[Punctuator +] = “+”

Punctuator[Punctuator + +] = “++”

Punctuator[Punctuator + =] = “+=”

Punctuator[Punctuator ,] = “,”

Punctuator[Punctuator -] = “-”

Punctuator[Punctuator - -] = “--”

Punctuator[Punctuator - =] = “-=”

Punctuator[Punctuator - >] = “->”

Punctuator[Punctuator .] = “.”

Punctuator[Punctuator . .] = “..”

Punctuator[Punctuator . . .] = “...”

Punctuator[Punctuator :] = “:”

Punctuator[Punctuator : :] = “::”

Punctuator[Punctuator ;] = “;”

Punctuator[Punctuator <] = “<”

Punctuator[Punctuator < <] = “<<”

Punctuator[Punctuator < < =] = “<<=”

Punctuator[Punctuator < =] = “<=”

Punctuator[Punctuator =] = “=”

Punctuator[Punctuator = =] = “==”

Punctuator[Punctuator = = =] = “===”

Punctuator[Punctuator >] = “>”

Punctuator[Punctuator > =] = “>=”

Punctuator[Punctuator > >] = “>>”

Punctuator[Punctuator > > =] = “>>=”

Punctuator[Punctuator > > >] = “>>>”

Punctuator[Punctuator > > > =] = “>>>=”

Punctuator[Punctuator ?] = “?”

Punctuator[Punctuator @] = “@”

Punctuator[Punctuator [] = “[”

Punctuator[Punctuator ]] = “]”

Punctuator[Punctuator ^] = “^”

Punctuator[Punctuator ^ =] = “^=”

Punctuator[Punctuator ^ ^] = “^^”

Punctuator[Punctuator ^ ^ =] = “^^=”

Punctuator[Punctuator {] = “{”

Punctuator[Punctuator |] = “|”

Punctuator[Punctuator | =] = “|=”

Punctuator[Punctuator | |] = “||”

Punctuator[Punctuator | | =] = “||=”

Punctuator[Punctuator }] = “}”

Punctuator[Punctuator ~] = “~”

action Punctuator[DivisionPunctuator] : String

Punctuator[DivisionPunctuator / [lookahead{/, *}]] = “/”

Punctuator[DivisionPunctuator / =] = “/=”

Numeric literals

Syntax

NumericLiteral

DecimalLiteral

| HexIntegerLiteral [lookahead{HexDigit}]

Semantics

action DoubleValue[NumericLiteral] : Double

DoubleValue[NumericLiteral DecimalLiteral]
= rationalToDouble(RationalValue[DecimalLiteral])

DoubleValue[NumericLiteral HexIntegerLiteral [lookahead{HexDigit}]]
= rationalToDouble(IntegerValue[HexIntegerLiteral])

expt(base: Rational, exponent: Integer) : Rational
  = if exponent = 0
     then 1
     else if exponent < 0
     then 1/expt(base, -exponent)
     else base*expt(base, exponent - 1)

Syntax

DecimalLiteral

Mantissa

| Mantissa LetterE SignedInteger

LetterE E | e

Mantissa

DecimalIntegerLiteral

| DecimalIntegerLiteral .

| DecimalIntegerLiteral . Fraction

| . Fraction

DecimalIntegerLiteral

0

| NonZeroDecimalDigits

NonZeroDecimalDigits

NonZeroDigit

| NonZeroDecimalDigits ASCIIDigit

NonZeroDigit 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Fraction DecimalDigits

Semantics

action RationalValue[DecimalLiteral] : Rational

RationalValue[DecimalLiteral Mantissa] = RationalValue[Mantissa]

RationalValue[DecimalLiteral Mantissa LetterE SignedInteger]
= RationalValue[Mantissa]*expt(10, IntegerValue[SignedInteger])

action RationalValue[Mantissa] : Rational

RationalValue[Mantissa DecimalIntegerLiteral] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral .] = IntegerValue[DecimalIntegerLiteral]

RationalValue[Mantissa DecimalIntegerLiteral . Fraction]
= IntegerValue[DecimalIntegerLiteral] + RationalValue[Fraction]

RationalValue[Mantissa . Fraction] = RationalValue[Fraction]

action IntegerValue[DecimalIntegerLiteral] : Integer

IntegerValue[DecimalIntegerLiteral 0] = 0

IntegerValue[DecimalIntegerLiteral NonZeroDecimalDigits]
= IntegerValue[NonZeroDecimalDigits]

action IntegerValue[NonZeroDecimalDigits] : Integer

IntegerValue[NonZeroDecimalDigits NonZeroDigit] = DecimalValue[NonZeroDigit]

IntegerValue[NonZeroDecimalDigits NonZeroDecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[NonZeroDecimalDigits₁] + DecimalValue[ASCIIDigit]

action DecimalValue[NonZeroDigit] : Integer = digitValue(NonZeroDigit)

action RationalValue[Fraction] : Rational

RationalValue[Fraction DecimalDigits]
= IntegerValue[DecimalDigits]/expt(10, NDigits[DecimalDigits])

Syntax

SignedInteger

DecimalDigits

| + DecimalDigits

| - DecimalDigits

Semantics

action IntegerValue[SignedInteger] : Integer

IntegerValue[SignedInteger DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger + DecimalDigits] = IntegerValue[DecimalDigits]

IntegerValue[SignedInteger - DecimalDigits] = -IntegerValue[DecimalDigits]

Syntax

DecimalDigits

ASCIIDigit

| DecimalDigits ASCIIDigit

Semantics

action IntegerValue[DecimalDigits] : Integer

IntegerValue[DecimalDigits ASCIIDigit] = DecimalValue[ASCIIDigit]

IntegerValue[DecimalDigits DecimalDigits₁ ASCIIDigit]
= 10*IntegerValue[DecimalDigits₁] + DecimalValue[ASCIIDigit]

action NDigits[DecimalDigits] : Integer

NDigits[DecimalDigits ASCIIDigit] = 1

NDigits[DecimalDigits DecimalDigits₁ ASCIIDigit] = NDigits[DecimalDigits₁] + 1

Syntax

HexIntegerLiteral

0 LetterX HexDigit

| HexIntegerLiteral HexDigit

LetterX X | x

HexDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F | a | b | c | d | e | f

Semantics

action IntegerValue[HexIntegerLiteral] : Integer

IntegerValue[HexIntegerLiteral 0 LetterX HexDigit] = HexValue[HexDigit]

IntegerValue[HexIntegerLiteral HexIntegerLiteral₁ HexDigit]
= 16*IntegerValue[HexIntegerLiteral₁] + HexValue[HexDigit]

action HexValue[HexDigit] : Integer = digitValue(HexDigit)

String literals

Syntax

{single, double}

StringLiteral

' StringChars^single '

| " StringChars^double "

Semantics

action StringValue[StringLiteral] : String

StringValue[StringLiteral ' StringChars^single '] = StringValue[StringChars^single]

StringValue[StringLiteral " StringChars^double "] = StringValue[StringChars^double]

Syntax

StringChars

«empty»

| StringChars StringChar

StringChar

LiteralStringChar

| \ StringEscape

LiteralStringChar^single UnicodeCharacter except ' | \ | LineTerminator

LiteralStringChar^double UnicodeCharacter except " | \ | LineTerminator

Semantics

action StringValue[StringChars] : String

StringValue[StringChars «empty»] = “”

StringValue[StringChars StringChars₁ StringChar]
= StringValue[StringChars₁] [CharacterValue[StringChar]]

action CharacterValue[StringChar] : Character

CharacterValue[StringChar LiteralStringChar] = LiteralStringChar

CharacterValue[StringChar \ StringEscape] = CharacterValue[StringEscape]

Syntax

StringEscape

ControlEscape

| ZeroEscape

| HexEscape

| IdentityEscape

IdentityEscape NonTerminator except UnicodeAlphanumeric

Semantics

action CharacterValue[StringEscape] : Character

CharacterValue[StringEscape ControlEscape] = CharacterValue[ControlEscape]

CharacterValue[StringEscape ZeroEscape] = CharacterValue[ZeroEscape]

CharacterValue[StringEscape HexEscape] = CharacterValue[HexEscape]

CharacterValue[StringEscape IdentityEscape] = IdentityEscape

Syntax

ControlEscape

b

| f

| n

| r

| t

| v

Semantics

action CharacterValue[ControlEscape] : Character

CharacterValue[ControlEscape b] = ‘«BS»’

CharacterValue[ControlEscape f] = ‘«FF»’

CharacterValue[ControlEscape n] = ‘«LF»’

CharacterValue[ControlEscape r] = ‘«CR»’

CharacterValue[ControlEscape t] = ‘«TAB»’

CharacterValue[ControlEscape v] = ‘«VT»’

Syntax

ZeroEscape 0 [lookahead{ASCIIDigit}]

Semantics

action CharacterValue[ZeroEscape] : Character

CharacterValue[ZeroEscape 0 [lookahead{ASCIIDigit}]] = ‘«NUL»’

Syntax

HexEscape

x HexDigit HexDigit

| u HexDigit HexDigit HexDigit HexDigit

Semantics

action CharacterValue[HexEscape] : Character

CharacterValue[HexEscape x HexDigit₁ HexDigit₂]
= codeToCharacter(16*HexValue[HexDigit₁] + HexValue[HexDigit₂])

CharacterValue[HexEscape  u HexDigit₁ HexDigit₂ HexDigit₃ HexDigit₄]
  = codeToCharacter(
         4096*HexValue[HexDigit₁] + 256*HexValue[HexDigit₂] + 16*HexValue[HexDigit₃] +
         HexValue[HexDigit₄])