JavaScript 2.0 Lexical Semantics

JavaScript 2.0

Formal Description

Lexical Semantics

Monday, June 30, 2003

The lexical semantics describe the actions the lexer takes in order to transform an input stream of Unicode characters into a stream of tokens. For convenience, the lexical grammar is repeated here. See also the description of the semantic notation.

This document is also available as a Word RTF file.

The lexer’s start symbols are: NextInputElement^num if the previous input element was a number; NextInputElement^re if the previous input element was not a number and a / should be interpreted as a regular expression; and NextInputElement^div if the previous input element was not a number and a / should be interpreted as a division or division-assignment operator.

In addition to the above, the start symbol StringNumericLiteral is used by the syntactic semantics for string-to-number conversions and the start symbol StringDecimalLiteral is used by the syntactic semantics for implementing the parseFloat function.

Semantics

tag lineBreak;

tag endOfInput;

tuple Keyword

end tuple;

tuple Punctuator

end tuple;

tuple Identifier

end tuple;

tuple NumberToken

value: GeneralNumber

end tuple;

tag negatedMinLong;

tuple StringToken

value: String

end tuple;

tuple RegularExpression

body: String,

flags: String

end tuple;

Token = Keyword Punctuator Identifier NumberToken {negatedMinLong} StringToken RegularExpression;

InputElement = {lineBreak, endOfInput} Token;

tag syntaxError;

tag rangeError;

SemanticException = {syntaxError, rangeError};

Unicode Character Classes

Syntax

UnicodeCharacter Any Unicode character

UnicodeInitialAlphabetic Any character in category Lu (uppercase letter), Ll (lowercase letter), Lt (titlecase letter), Lm (modifier letter), Lo (other letter), or Nl (letter number) in the Unicode Character Database

UnicodeAlphanumeric Any character in category Lu (uppercase letter), Ll (lowercase letter), Lt (titlecase letter), Lm (modifier letter), Lo (other letter), Nd (decimal number), Nl (letter number), Mn (non-spacing mark), Mc (combining spacing mark), or Pc (connector punctuation) in the Unicode Character Database

WhiteSpaceCharacter

«TAB» | «VT» | «FF» | «SP» | «u00A0»

| «u2000» | «u2001» | «u2002» | «u2003» | «u2004» | «u2005» | «u2006» | «u2007»

| «u2008» | «u2009» | «u200A» | «u200B»

| «u3000»

LineTerminator «LF» | «CR» | «u0085» | «u2028» | «u2029»

ASCIIDigit 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Semantics

DecimalValue[ASCIIDigit]: Integer = digitValue(ASCIIDigit);

Comments

Syntax

LineComment / / LineCommentCharacters

LineCommentCharacters

«empty»

| LineCommentCharacters NonTerminator

NonTerminator UnicodeCharacter except LineTerminator

SingleLineBlockComment / * BlockCommentCharacters * /

BlockCommentCharacters

«empty»

| BlockCommentCharacters NonTerminatorOrSlash

| PreSlashCharacters /

PreSlashCharacters

«empty»

| BlockCommentCharacters NonTerminatorOrAsteriskOrSlash

| PreSlashCharacters /

NonTerminatorOrSlash NonTerminator except /

NonTerminatorOrAsteriskOrSlash NonTerminator except * | /

MultiLineBlockComment / * MultiLineBlockCommentCharacters BlockCommentCharacters * /

MultiLineBlockCommentCharacters

BlockCommentCharacters LineTerminator

| MultiLineBlockCommentCharacters BlockCommentCharacters LineTerminator

White Space

Syntax

WhiteSpace

«empty»

| WhiteSpace WhiteSpaceCharacter

| WhiteSpace SingleLineBlockComment

Line Breaks

Syntax

LineBreak

LineTerminator

| LineComment LineTerminator

| MultiLineBlockComment

LineBreaks

LineBreak

| LineBreaks WhiteSpace LineBreak

Input Elements

Syntax

{re, div, num}

NextInputElement^re WhiteSpace InputElement^re

NextInputElement^div WhiteSpace InputElement^div

NextInputElement^num [lookahead{ContinuingIdentifierCharacter, \}] WhiteSpace InputElement^div

Semantics

Lex[NextInputElement]: InputElement;

Lex[NextInputElement^re WhiteSpace InputElement^re] = Lex[InputElement^re];

Lex[NextInputElement^div WhiteSpace InputElement^div] = Lex[InputElement^div];

Lex[NextInputElement^num [lookahead{ContinuingIdentifierCharacter, \}] WhiteSpace InputElement^div] = Lex[InputElement^div];

Syntax

InputElement^re

LineBreaks

| IdentifierOrKeyword

| Punctuator

| NumericLiteral

| StringLiteral

| RegExpLiteral

| EndOfInput

InputElement^div

LineBreaks

| IdentifierOrKeyword

| Punctuator

| DivisionPunctuator

| NumericLiteral

| StringLiteral

| EndOfInput

EndOfInput

End

| LineComment End

Semantics

Lex[InputElement]: InputElement;

Lex[InputElement LineBreaks] = lineBreak;

Lex[InputElement IdentifierOrKeyword] = Lex[IdentifierOrKeyword];

Lex[InputElement Punctuator] = Lex[Punctuator];

Lex[InputElement^div DivisionPunctuator] = Lex[DivisionPunctuator];

Lex[InputElement NumericLiteral] = Lex[NumericLiteral];

Lex[InputElement StringLiteral] = Lex[StringLiteral];

Lex[InputElement^re RegExpLiteral] = Lex[RegExpLiteral];

Lex[InputElement EndOfInput] = endOfInput;

Keywords and Identifiers

Syntax

IdentifierOrKeyword IdentifierName

Semantics

Lex[IdentifierOrKeyword IdentifierName]: InputElement

begin

id: String Lex[IdentifierName];

if id {“abstract”, “as”, “break”, “case”, “catch”, “class”, “const”, “continue”, “debugger”, “default”, “delete”, “do”, “else”, “enum”, “export”, “extends”, “false”, “finally”, “for”, “function”, “get”, “goto”, “if”, “implements”, “import”, “in”, “instanceof”, “interface”, “is”, “namespace”, “native”, “new”, “null”, “package”, “private”, “protected”, “public”, “return”, “set”, “super”, “switch”, “synchronized”, “this”, “throw”, “throws”, “transient”, “true”, “try”, “typeof”, “use”, “var”, “volatile”, “while”, “with”} and not ContainsEscapes[IdentifierName] then

return Keywordname: id

else return Identifiername: id

end if

end;