std.prototype 2.1

• Index

Contents

• Objects
• Initialisation
• Metamethods
• Module Functions
• Fields
• Methods

Modules

• std.prototype
• std.prototype.container
• std.prototype.contextmanager
• std.prototype.object
• std.prototype.orderedmap
• std.prototype.scanner
• std.prototype.set
• std.prototype.strbuf
• std.prototype.trie

Module std.prototype.scanner

Scanner Prototype.

This module returns a table of scanner helpers, as well as the prototype for a Scanner object.

A Scanner reads a string of text one character at a time, for the lexer of a parser to build its tokens from. It always knows where the next character is, and peek looks at characters that are not read yet, next reads them, back unreads them, and take, skip and match read a whole run of them in one call.

Classes of character are sets, which join with +, and make a lexer read like its grammar:

local Scanner = require 'std.prototype.scanner'.prototype
local charset = require 'std.prototype.scanner'.charset

local isspace = charset ' \t\r\n'
local isdelimiter = charset '()"' + isspace

local scanner = Scanner('(add 1 2)', 'sum.lsp')
scanner:next()                      --> '('
scanner:take(function(c)            --> 'add'
   return not isdelimiter[c]
end)
scanner:skip(isspace)
scanner:match '%d+'                 --> '1'
scanner:error 'unexpected end'      --> error: sum.lsp:1: unexpected end

An index is the position of a character in the text, counting from 1. Where the cursor stands is the index of the next character to read, so pos is 1 before anything is read and one more than the length of the text at the end.

In addition to the functionality described here, Scanner objects also have all the methods and metamethods of the prototype.object.prototype (except where overridden here).

Prototype Chain

table
 `-> Container
      `-> Object
           `-> Scanner

Objects

prototype

Scanner prototype object.

Initialisation

prototype._init

Scanner object initialisation.

Metamethods

prototype:__tostring ()

Return a string representation of this scanner.

Module Functions

charset (s)

The characters of a string, as a Set.

Fields

text	The text that is scanned.
pos	Where the cursor is, the index of the next character to read.
name	What the source is called in errors, if anything.

Methods

prototype:eof ()	Whether the cursor is at the end of the text.
prototype:peek ([offset=0])	Look at a character that is not read yet.
prototype:next ([n=1])	Read characters.
prototype:back ([n=1])	Unread characters, to read them again.
prototype:lookahead (s)	Whether the next characters are the text given.
prototype:match (pattern)	Read what a Lua pattern matches, if it matches next.
prototype:take (set)	Read the run of characters that are in a class.
prototype:skip (set)	Read the run of characters that are in a class, and drop them.
prototype:line ([index])	The line number of an index in the text.
prototype:error (msg[, index])	Raise an error that says where in the text it is.

Objects

prototype

Scanner prototype object.

Fields:

• _type string object name (default 'Scanner')

See also:

prototype.object.prototype

Usage:

local Scanner = require 'std.prototype.scanner'.prototype
assert(prototype.type(Scanner) == 'Scanner')
local scanner = Scanner('text to scan', 'name of the source')

Initialisation

prototype._init

Scanner object initialisation.

Returns a Scanner for text, with the cursor at the start. A table, as a type derived from this one is described with, has any field with a leading underscore, such as _type, put in the metatable, and no other fields.

Parameters:

• new table uninitialised Scanner container object
• text string the text to scan
• name string what to call the source of text in errors (optional)

Metamethods

prototype:__tostring ()

Return a string representation of this scanner.

Returns:

string the name of the source and the line of the cursor, or only the type, for a prototype that has no text

Usage:

print(scanner) --> Scanner sum.lsp:1

Module Functions

charset (s)

The characters of a string, as a Set. Sets join with +, so classes of characters are made from others.

Parameters:

• s string a string of characters

Returns:

Set the set of the characters in s

Usage:

local isspace = charset ' \t\r\n'
local isdelimiter = charset '()' + isspace

Fields

text

The text that is scanned.

• text string

pos

Where the cursor is, the index of the next character to read. Set it with prototype:back, and not directly, so that it stays within the text.

• pos integer

name

What the source is called in errors, if anything.

• name string

Methods

prototype:eof ()

Whether the cursor is at the end of the text.

Returns:

boolean true if every character has been read

Usage:

while not scanner:eof() do ... end

prototype:peek ([offset=0])

Look at a character that is not read yet.

Parameters:

• offset integer 0 is the next character, 1 the one after, and a negative offset a character that is read already (default 0)

Returns:

string or nil the character, or nil if there is none

Usage:

if scanner:peek(1) == '-' then ... end

prototype:next ([n=1])

Read characters.

Parameters:

• n integer how many to read (default 1)

Returns:

string or nil the characters read, fewer than n at the end of the text, or nil if there were none

Usage:

local c = scanner:next()

prototype:back ([n=1])

Unread characters, to read them again. The cursor stops at the start of the text.

Parameters:

• n integer how many to unread (default 1)

Returns:

prototype scanner

Usage:

if scanner:next() ~= ',' then scanner:back() end

prototype:lookahead (s)

Whether the next characters are the text given.

Parameters:

• s string the text to look for

Returns:

boolean true if the text is next

Usage:

if scanner:lookahead '--[[' then ... end

prototype:match (pattern)

Read what a Lua pattern matches, if it matches next. The pattern is matched as if it started with ^, from the cursor.

Parameters:

• pattern string a Lua pattern

Returns:

the captures of the pattern, or what it matched if it has none, and nil if it does not match

Usage:

local digits = scanner:match '%d+'

prototype:take (set)

Read the run of characters that are in a class.

Parameters:

• set table, function or string a class of character, as a table with a true value for each, such as a Set, or a function that is true for each, or a string of them

Returns:

string the characters read, possibly none

Usage:

local name = scanner:take(isidentifier)

prototype:skip (set)

Read the run of characters that are in a class, and drop them.

Parameters:

• set table, function or string a class of character

Returns:

prototype scanner

See also:

prototype:take

Usage:

scanner:skip(isspace)

prototype:line ([index])

The line number of an index in the text. The first line is 1. Line numbers are not counted until asked for, so they cost nothing to a scanner that is never asked.

Parameters:

• index integer defaults to the cursor (optional)

Returns:

int the line that the character at index is on

Usage:

local l = scanner:line()

prototype:error (msg[, index])

Raise an error that says where in the text it is. The message is name:line: msg, or line: msg for a scanner with no name, and has no Lua position added to it.

Parameters:

• msg string what is wrong
• index integer defaults to the cursor (optional)

Usage:

scanner:error "missing ')'"