|
| |
| | Token (parser) - Wikipedia, the free encyclopedia |
 | | In computing, a token is an categorized block of text, usually consisting of indivisible characters known as lexemes. | | | But they are tokens none the less, in this example. | | | A token doesn't need to be valid, in order to be recognized as a token. |
|
http://en.wikipedia.org/wiki/Token_(parser)
(304 words)
|
|
| |
| | Glossary of Parsing Terms |
| | The inputs are the tokens of your grammar. | | | A parser is a program, or more likely a procedure within a program, which scans a sequence of input characters or input tokens and accumulates them in an input buffer or stack as determined by a set of productions which constitute a grammar. | | | Nonterminal tokens are to be distinguished from terminal tokens or input tokens. |
|
http://www.parsifalsoft.com/gloss.html
(7372 words)
|
|
| |
| | Token - Wikipedia, the free encyclopedia |
| | Security token (also known as a hardware token, authentication token or cryptographic token), a physical device that an authorized user of computer services is given to aid in authentication | | | Token ring, a local area network technology in which a virtual object known as a token is passed between devices on the network, authorizing them to communicate | | | Cellular Authentication Token, a software replacement for a security token |
|
http://en.wikipedia.org/wiki/Token
(339 words)
|
|
| |
| | ML-Yacc User's Manual Version 2.3 |
| | Parsers perform bottom-up, left-to-right evaluations of parse trees using semantic actions to compute values as they do so. | | | A parser tries to recover from a syntax error by making a single token insertion, deletion, or substitution near the point in the input stream at which the error was detected. | | | Two possibilities are turning the lexer and parser structures into closed functors, that is, functors which do not refer to types or values defined outside their body or outside their parameter structures (except for pervasive types and values), and creating a functor which encapsulates the code necessary to invoke the parser. |
|
http://www.cs.princeton.edu/~appel/modern/ml/ml-yacc/manual.html
(6254 words)
|
|
| |
| | Simple Parser Framework Library |
| | Token definitions can be divided into three classes: “inert” tokens (such as whitespace) that don't need to be passed on to the consumer; punctuation-type tokens that don't have a semantic value; and token definitions that provide a function for computing the token's semantic value based on the token text. | | | The code for a recursive-descent parser is generally not as succinct as the input to a specialized parser generator, and rewriting grammars to conform to LL(1) can be inconvenient. | | | When there is a semantic value function, the token text needs to be extracted from the input (possibly complicated by the text crossing the boundary between the saved and current input text segments) and passed to the semantic value function. |
|
http://monday.sourceforge.net/lib/language/simple-parser/simple-parser.html
(4950 words)
|
|
| |
| | Parser |
| | Return the token type of the ith token of lookahead where i=1 is the current token being examined by the parser (i.e., it has not been matched yet). | | | If the user specifies a tokens{} section with heterogeneous AST node types, then ANTLR generates code to fill this mapping. |
|
http://www.antlr.org/javadoc/antlr/Parser.html
(451 words)
|
|
| |
| | ANTLR Specification: Token Streams |
| | The output of one parser can be the input to another. | | | Traditionally, a lexer and parser are tightly coupled objects; that is, one does not imagine anything sitting between the parser and the lexer, modifying the stream of tokens. | | | This is primarily because merging the token definitions of the various slices results in an ambiguous lexical language or allows invalid tokens. |
|
http://www.antlr.org/doc/streams.html#lexerstates
(3214 words)
|
|
| |
| | [No title] |
| | Lexers that are functors parametrized by a Tokens structure matching a TOKENS signature cannot examine the structure of tokens. | | | A constructor function for a integer token might be INT: int * 'a * 'a -> 'a token. | | | The representation of token was very carefully chosen here to allow the polymorphic parser to work without knowing the types of semantic values or line numbers. |
|
http://www.cs.cmu.edu/afs/cs.cmu.edu/user/fp/courses/95-lp/code/elpsml/base.sml
(1794 words)
|
|
| |
| | [No title] |
| | For example an EOF token would be specified for a parser that parses entire files and a NEWLINE token for a parser that parses entire lines individually. | | | Doing something like (define parse (lambda (y) (parser...))) can be very useful if the parser actions need to refer to a y that might be different on each execution of the parser, but will lead to multiple executions of the table setup code. | | | error cannot be defined as a token since it has special use in the parser. |
|
http://www.bath.ac.uk/~masjap/plt/collects/parser-tools/doc.txt
(1693 words)
|
|
| |
| | A BNF Parser in Forth |
| | The parser uses the fig-Forth IN as the input pointer, and the dictionary pointer DP as the output pointer. | | | Bottom-up parsers, which move directly from state to state in the parse tree according to the input tokens, are better still. | | | Thus, as the parser works its way, top-down, through the parse tree, it is constantly producing and discarding trial output. |
|
http://www.zetetics.com/bj/papers/bnfparse.htm
(2396 words)
|
|
| |
| | Parsec, a fast combinator parser |
| | Most parsers take their input from a lexical analyzer or scanner which filters white space and comments and returns a list tokens for the parser. | | | This parser garantees that all input will be parsed; if it is left out, a parser is allowed to parse only part of the input. | | | parser is used to define a parser that scans a single token. |
|
http://www.cs.uu.nl/~daan/download/parsec/parsec.html
(7625 words)
|
|
| |
| | XPA |
| | Make XML parsers generate trees that can be parsed by ANTLR generated tree parsers: This carries on the job of the XML parser and is comparable to the special case in traditional parsing technique where the token parser does nothing, but generate a tree for a tree parser to process. | | | Now for the difference: When you write a token or tree parser using grammars you not only write the symbols of your grammar, but also some semantic action taken upon occurrence of a special token in a special context. | | | From a certain point of view building a parser is a way to access information that is stored in a special pattern and syntax. |
|
http://xpa.sourceforge.net/concept.html
(1200 words)
|
|
| |
| | [No title] |
| | Tokens -- are, furthermore, printable (instance of `Show'); the resulting string -- should correspond to the lexeme of the token and not the data -- constructor used to represent it internally. | | | The equality determines whether -- tokens "match" during parsing, ie, whether they are equal modulo their -- attributes (the position is, of course, an attribute). | | | The applied technique for -- efficiently computing the parse tables makes essential use of the -- memorization build into lazy evaluation. |
|
http://www.cse.unsw.edu.au/~dons/code/stgjvm/ctkl/Parsers.hs
(1107 words)
|
|
| |
| | [No title] |
| | Upon completion the * * token, token number and operations value is printed, if * * listtoken is true. | | | Once a token * * has been recognized, it is used by the parser. | | | Upon completion, the value is * * stored as part of the token. |
|
http://www.cs.stedwards.edu/~lbaker/cosc4342/micro/parser.p
(914 words)
|
|
| |
| | [No title] |
| | If called with parser set to NULL, the lists of characters in new_delimiters and new_nulldelimiters are stored. | | | To handle this, set maxtok only when a -ve value of n is input. | | | token; /* Loop over all tokens */ for (i=0; i |
|
http://www.ccp4.ac.uk/dist/lib/src/ccp4_parser_f.c
(295 words)
|
|
| |
| | Cyrus' Blather : Only a few hours left (part 4) |
| | Verifying then that the parser code works as it should is far too difficult, and unclear. | | | So we left off on the previous post with the question of why we were using Java to work with our new token visitors. | | | Depending on how the code is structured, and how many visitors are needed, you might end up having this logic hundreds of lines apart. |
|
http://blogs.msdn.com/cyrusn/archive/2005/09/13/464637.aspx
(470 words)
|
|
| |
| | : Index |
| | Illustrates the java.io.StreamTokenizer class and how it can be used to parse tokens for a toy example programming language (elan) For this example simple instructions are parsed, but lots of infrastructure is in place for parsing a more complete language. | | | InstructionParser parses instructions; it uses token types to call appropriate parsers, and throws ParseExceptions in cases of error. | | | The token (parser sub) package holds all token classes. |
|
http://www.duke.edu/~lap3/doc/index-all.html
(1196 words)
|
|
| |
| | Cheaper by the Dozen |
| | As ; an example of the parser construction, this file includes a ; semi-validating SXML parser. | | | Like a SAX parser, the framework scans ; the document only once and permits incremental processing. | | | In ; the latter case, the user must provide functions of specific ; signatures, which are called at predictable moments during the ; parsing: to handle character data, element data, or processing ; instructions (PI). |
|
http://okmij.org/ftp/Scheme/SSAX.scm
(8772 words)
|
|
| |
| | ./Parser.curry |
| | The returned parser --- repeats zero or more times a parser p with representation and --- returns the representation of all parsers in a list. | | | (star p) xs >>> (x:xs) where x,xs free {----------------------------------------------------------------------- As a simple example we define a parser for arithmetic expressions over natural numbers. | | | satisfy :: (token->Bool) -> ParserRep token token satisfy pred sym (token:tokens) |
|
http://www.informatik.uni-kiel.de/~pakcs/lib/CDOC/Parser_curry.html
(367 words)
|
|
| |
| | Parser class Reference |
| | Referenced by getChar, parse1, parse_math, parse_table, parse_text, tokenize, and verbatim_item. | | | // handle only this single token, leave the loop if done | | | // no option found, put back token and we are done |
|
http://www.lyx.org/sourcedoc/classParser.html
(1165 words)
|
|
| |
| | [No title] |
| | where It is easy to turn the grammar for a chunk, a function parameter list, and a block into parser definitions. | | | > module Parser (parse) where > import MicroLua > import Monad > import Char > import Utility > import Control.Monad.State A microLua parser takes as input three constructor functions, which produce string values, number values, and nil values, respectively. | | | > parse :: (String -> value) -> (Integer -> value) -> value -> > String -> [Program value] Here is a null parser that fails on any input. |
|
http://www.eecs.harvard.edu/~ccshan/cs252/MicroLua/Parser.lhs
(578 words)
|
|
| |
| | [No title] |
| | * @param parser pointer to a CCP4PARSERARRAY structure which will * be used to hold the results of processing the input line. | | | * @param maxtokens maximum number of tokens on a line * @return pointer to a new CCP4PARSERARRAY structure */ CCP4PARSERARRAY* ccp4_parse_start(const int maxtokens); /** Cleans up a CCP4PARSEARRAY after being used by ccp4_parse/ ccp4_parser functions. | | | */ int ccp4_parser(char *line, const int n, CCP4PARSERARRAY *parser, const int print); /* External utility functions */ /** Test whether two keywords are identical. |
|
http://www.ccp4.ac.uk/dist/lib/src/ccp4_parser.h
(705 words)
|
|
| |
| | [No title] |
| | /usr/bin/python """Generic predictive parser for N3-like languages This is is a generic parser for a set of N3-like languages. | | | The program checks that a predictive parser *can* be built from the CFG given. | | | The parser is N3-specific only because of the built-in tokenizer. |
|
http://www.w3.org/2000/10/swap/grammar/check-grammar.py
(147 words)
|
|
| |
| | [No title] |
| | *) (* *) (*******************************************************************) (* functions for generating tokens *) infixr 4 | | | word wth (lam (x: String): Token => Word x) end val tokenLiteral = literal tokenEq implement litWord s = tokenLiteral (Word s) return s implement litInteger n = tokenLiteral (Integer n) return n implement anyInteger = any -- (lam (t: Token): Parser (Int, Token) => case t of Integer s => succeed s | | | _ => fail) implement anyWord = any -- (lam (t: Token): Parser (String, Token) => case t of Word s => succeed s |
|
http://www.cs.bu.edu/~hwxi/ATS/EXAMPLE/ParsingCombinators/token.ats
(478 words)
|
|
| |
| | [Tutor] token parser article |
| | We can use the first token as a key and the rest of the list as the value for a dictionary. | | | Here is what is looks like with kumar's original example: >>> s='[AKTPI3K][RHOABCL:CDC42:IKK:RAC1:RAL:RALBP1]' >>> from pyparsing import * I'll build the parser from the inside out, starting with simple tokens and combining them to recognize more and more complex parts of the complete string. | | | First I create a parse token to represent the key portion of each entry. |
|
http://mail.python.org/pipermail/tutor/2004-October/032350.html
(595 words)
|
|
| |
| | [No title] |
| | Result JavaDerivs Token Parser pHexLit = do char '0' oneOf "xX" digits | | | return False pFloatLit :: JavaDerivs -> Result JavaDerivs Token Parser pFloatLit = (do i | | | return [] pInitializer :: JavaDerivs -> Result JavaDerivs Initializer Parser pInitializer = (do inits |
|
http://www.pdos.lcs.mit.edu/~baford/packrat/thesis/java/JavaMonad.hs
(1095 words)
|
|
| |
| | [No title] |
| | setToList instance (Parse a,Ord a)=>Parse (Set a) where parser p= do list | | | String quoteChar '\\'="\\\\" quoteChar '\n'="\\n" quoteChar z=[z] quote::String->String quote=concatMap quoteChar showItem::(String,String)->String showItem (name,value)= name++"="++quote(value)++"\n" showStruct::String->[(String,String)]->String showStruct blockName items= "["++blockName++"]\n\n"++concatMap showItem items++"\n" --- Parser type Dict=FiniteMap String String restOfLine::Parser String restOfLine= do expectEof return "" `mplus` do line | | | Parser [a] normListParser= do expect (TokSep "[") ret |
|
http://atrey.karlin.mff.cuni.cz/~rakdver/fall/Parser.hs
(405 words)
|
|
| |
| | Parser (JEP API) |
| | Allows renetrancy of parser so that strings like "x=1; y=2; z=3;" can be parsed. | | | Sets the initial state that the token manager is in. | | | Parsing can be resumed from the current position by using this method. |
|
http://www.singularsys.com/jep/doc/javadoc/org/nfunk/jep/Parser.html
(185 words)
|
|
| |
| | [No title] |
| | #include "parser.h" using namespace std; //introduces namespace std VTable Parser::varTable = VTable(); bool Parser::fileopenfail = true; // TO IMPLEMENT: // Parser constructor should // (1) open infile, for file "filename". | | | void Parser::process_token(Token t) { } int main(void) { Parser p("program.txt"); Token t = p.next_token(); while (t.first != END_FILE) { p.process_token(t); t = p.next_token(); } p.process_token(t); return 0; } |
|
http://zippy.sonoma.edu/~jcoelho/CS215/Projects/proj4_parser.cpp
(78 words)
|
|
| |
| | [No title] |
| | "; getline(cin, input); if (input == "quit") break; if (input == "help") { help(); continue; } if (input == "") continue; char* str = strdup(input.c_str()); Parser pobj(str); Parser::Token tok = pobj.eval(); if (tok.type != Parser::ERROR) cout << tok.e << endl; free(str); } return 0; } | | | #include "parser.h" #include "set.h" using namespace std; Element vars[26]; Parser::Token Parser::eval() { Token tok = testop(); if (curtok.type != END && curtok.type != ERROR) return error("trailing garbage"); return tok; } // test operations, lowest precedence: // element [ set-operation // element ![ set-operation // set-operation |
|
http://somewhere.fscked.org/misc/setcalc/parser.cc
(110 words)
|
|
| |
| | [No title] |
| | */ } } else { char *ptr = token; ptr++; while(*ptr) { if(!isSpace(*ptr)) goto _parseContinue; ptr++; } goto _parseSingleToken; _parseContinue: if(strchr(token, '=')) { doWithArgs(token); } else if(strCaseCmp(token, "center")) { Parser.align = (Parser.tokenClosed? | | | Aileron is an email client, not a * freakin' web browser */ /* HTMLParser.c: the HTML parser that gives those nice text formats * * This file is part of Aileron, a WINGs email client. | | | ease removal */ /* this parser is getting silly... |
|
http://ucsu.colorado.edu/~nwanua/Aileron/Source/HTMLParser.c
(493 words)
|
|
| |
| | Token class Reference |
| | Referenced by handle_tabular, operator<<, Parser::parse1, parse_math, parse_preamble, parse_table, parse_text, read_hlines, and Token. | | | Referenced by handle_colalign, handle_comment, operator<<, Parser::parse1, parse_math, parse_preamble, parse_table, parse_text, Parser::parse_verbatim_item, Parser::parse_verbatim_option, skip_braces, Parser::skip_spaces, Parser::skipSpaces, Token, Parser::verbatim_item, and Parser::verbatimOption. | | | The documentation for this class was generated from the following files: |
|
http://www.lyx.org/sourcedoc/classToken.html
(111 words)
|
|
|
