org.fife.ui.rsyntaxtextarea.modes.YamlTokenMaker类的使用及代码示例

x33g5p2x  于2022-02-05 转载在 其他  
字(6.9k)|赞(0)|评价(0)|浏览(113)

本文整理了Java中org.fife.ui.rsyntaxtextarea.modes.YamlTokenMaker类的一些代码示例，展示了YamlTokenMaker类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。YamlTokenMaker类的具体详情如下：
包路径：org.fife.ui.rsyntaxtextarea.modes.YamlTokenMaker
类名称：YamlTokenMaker

YamlTokenMaker介绍

[英]This class splits up text into tokens representing a YAML file.

This implementation was created using JFlex 1.4.1; however, the generated file was modified for performance. Memory allocation needs to be almost completely removed to be competitive with the handwritten lexers (subclasses of AbstractTokenMaker, so this class has been modified so that Strings are never allocated (via yytext()), and the scanner never has to worry about refilling its buffer (needlessly copying chars around). We can achieve this because RText always scans exactly 1 line of tokens at a time, and hands the scanner this line as an array of characters (a Segment really). Since tokens contain pointers to char arrays instead of Strings holding their contents, there is no need for allocating new memory for Strings.

The actual algorithm generated for scanning has, of course, not been modified.

If you wish to regenerate this file yourself, keep in mind the following:

  • The generated YamlTokenMaker.java file will contain two definitions of both zzRefill and yyreset. You should hand-delete the second of each definition (the ones generated by the lexer), as these generated methods modify the input buffer, which we'll never have to do.
  • You should also change the declaration/definition of zzBuffer to NOT be initialized. This is a needless memory allocation for us since we will be pointing the array somewhere else anyway.
  • You should NOT call yylex() on the generated scanner directly; rather, you should use getTokenList as you would with any other TokenMaker instance.
    [中]此类将文本拆分为表示YAML文件的标记。
    这个实现是使用JFlex1.4.1创建的；但是，为了提高性能，对生成的文件进行了修改。为了与手写lexer（AbstractTokenMaker的子类）竞争，内存分配几乎需要完全删除，所以这个类已经被修改，所以字符串永远不会被分配（通过yytext（）），扫描器也不必担心重新填充缓冲区（不必要地复制字符）。我们可以做到这一点，因为RText每次只扫描一行令牌，并将这一行作为字符数组（实际上是一个段）交给扫描仪。由于令牌包含指向字符数组的指针，而不是包含其内容的字符串，因此无需为字符串分配新内存。
    当然，为扫描生成的实际算法尚未修改。
    如果您希望自己重新生成此文件，请记住以下几点：
    *生成的YamlTokenMaker.java文件将包含zzRefill和yyreset两个定义。您应该手动删除每个定义的第二个定义（由lexer生成的定义），因为这些生成的方法修改了输入缓冲区，这是我们永远不需要做的。
    *还应将zzBuffer的声明/定义更改为未初始化。这对我们来说是一个不必要的内存分配，因为我们将把数组指向其他地方。
    *您不应直接在生成的扫描仪上调用yylex()；相反，您应该像使用任何其他TokenMaker实例一样使用getTokenList。

代码示例

代码示例来源：origin: bobbylight/RSyntaxTextArea

boolean eof = zzRefill();
{ addToken(Token.IDENTIFIER);
{ addToken(Token.LITERAL_STRING_DOUBLE_QUOTE);
{ addToken(Token.ERROR_CHAR); addNullToken(); return firstToken;
{ addToken(Token.WHITESPACE);
{ addToken(Token.ERROR_CHAR);
{ addToken(Token.COMMENT_EOL);
{ addToken(Token.OPERATOR);
{ addToken(Token.LITERAL_CHAR);
{ addToken(Token.ERROR_STRING_DOUBLE); addNullToken(); return firstToken;
{ addNullToken(); return firstToken;
{ addToken(Token.ERROR_STRING_DOUBLE);
{ addToken(Token.SEPARATOR);
 switch (zzLexicalState) {
 case YYINITIAL: {
  addNullToken(); return firstToken;

代码示例来源：origin: bobbylight/RSyntaxTextArea

/**
 * Returns the first token in the linked list of tokens generated
 * from <code>text</code>.  This method must be implemented by
 * subclasses so they can correctly implement syntax highlighting.
 *
 * @param text The text from which to get tokens.
 * @param initialTokenType The token type we should start with.
 * @param startOffset The offset into the document at which
 *        <code>text</code> starts.
 * @return The first <code>Token</code> in a linked list representing
 *         the syntax highlighted text.
 */
@Override
public Token getTokenList(Segment text, int initialTokenType, int startOffset) {
  resetTokenList();
  this.offsetShift = -text.offset + startOffset;
  // Start off in the proper state.
  int state = YYINITIAL;
  s = text;
  try {
    yyreset(zzReader);
    yybegin(state);
    return yylex();
  } catch (IOException ioe) {
    ioe.printStackTrace();
    return new TokenImpl();
  }
}

代码示例来源：origin: bobbylight/RSyntaxTextArea

/**
 * Pushes the specified amount of characters back into the input stream.
 *
 * They will be read again by then next call of the scanning method
 *
 * @param number  the number of characters to be read again.
 *                This number must not be greater than yylength()!
 */
public void yypushback(int number)  {
 if ( number > yylength() )
  zzScanError(ZZ_PUSHBACK_2BIG);
 zzMarkedPos -= number;
}

代码示例来源：origin: bobbylight/RSyntaxTextArea

/**
 * Adds the token specified to the current linked list of tokens.
 *
 * @param tokenType The token's type.
 */
private void addToken(int start, int end, int tokenType) {
  int so = start + offsetShift;
  addToken(zzBuffer, start,end, tokenType, so);
}

代码示例来源：origin: bobbylight/RSyntaxTextArea

private static int [] zzUnpackAction() {
 int [] result = new int[34];
 int offset = 0;
 offset = zzUnpackAction(ZZ_ACTION_PACKED_0, offset, result);
 return result;
}

代码示例来源：origin: bobbylight/RSyntaxTextArea

/**
 * Adds the token specified to the current linked list of tokens.
 *
 * @param tokenType The token's type.
 */
private void addToken(int tokenType) {
  addToken(zzStartRead, zzMarkedPos-1, tokenType);
}

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

private static int [] zzUnpackAction() {
 int [] result = new int[34];
 int offset = 0;
 offset = zzUnpackAction(ZZ_ACTION_PACKED_0, offset, result);
 return result;
}

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

/**
 * Returns the first token in the linked list of tokens generated
 * from <code>text</code>.  This method must be implemented by
 * subclasses so they can correctly implement syntax highlighting.
 *
 * @param text The text from which to get tokens.
 * @param initialTokenType The token type we should start with.
 * @param startOffset The offset into the document at which
 *        <code>text</code> starts.
 * @return The first <code>Token</code> in a linked list representing
 *         the syntax highlighted text.
 */
@Override
public Token getTokenList(Segment text, int initialTokenType, int startOffset) {
  resetTokenList();
  this.offsetShift = -text.offset + startOffset;
  // Start off in the proper state.
  int state = YYINITIAL;
  s = text;
  try {
    yyreset(zzReader);
    yybegin(state);
    return yylex();
  } catch (IOException ioe) {
    ioe.printStackTrace();
    return new TokenImpl();
  }
}

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

boolean eof = zzRefill();
{ addToken(Token.IDENTIFIER);
{ addToken(Token.LITERAL_STRING_DOUBLE_QUOTE);
{ addToken(Token.ERROR_CHAR); addNullToken(); return firstToken;
{ addToken(Token.WHITESPACE);
{ addToken(Token.ERROR_CHAR);
{ addToken(Token.COMMENT_EOL);
{ addToken(Token.OPERATOR);
{ addToken(Token.LITERAL_CHAR);
{ addToken(Token.ERROR_STRING_DOUBLE); addNullToken(); return firstToken;
{ addNullToken(); return firstToken;
{ addToken(Token.ERROR_STRING_DOUBLE);
{ addToken(Token.SEPARATOR);
 switch (zzLexicalState) {
 case YYINITIAL: {
  addNullToken(); return firstToken;

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

/**
 * Pushes the specified amount of characters back into the input stream.
 *
 * They will be read again by then next call of the scanning method
 *
 * @param number  the number of characters to be read again.
 *                This number must not be greater than yylength()!
 */
public void yypushback(int number)  {
 if ( number > yylength() )
  zzScanError(ZZ_PUSHBACK_2BIG);
 zzMarkedPos -= number;
}

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

/**
 * Adds the token specified to the current linked list of tokens.
 *
 * @param tokenType The token's type.
 */
private void addToken(int tokenType) {
  addToken(zzStartRead, zzMarkedPos-1, tokenType);
}

代码示例来源：origin: com.fifesoft/rsyntaxtextarea

/**
 * Adds the token specified to the current linked list of tokens.
 *
 * @param tokenType The token's type.
 */
private void addToken(int start, int end, int tokenType) {
  int so = start + offsetShift;
  addToken(zzBuffer, start,end, tokenType, so);
}

内容来源于网络，如有侵权，请联系作者删除！

YamlTokenMaker

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com