java 如何在不使用rg的情况下实现令牌?[关闭]

mlmc2os5  于 2023-11-15  发布在  Java
关注(0)|答案(1)|浏览(111)

已关闭。此问题需要details or clarity。目前不接受回答。
**要改进此问题吗?**通过editing this post添加详细信息并阐明问题。

2天前关闭。
Improve this question
我得到了一个token contents,我想完成MDTokenizer类的next()方法来执行tokenization过程。
next()方法应该标识所有token及其相关值(例如,对于没有缩进(没有前缀空格)的项目,level的相应值为1)。
每个令牌都有tokenTypecontentlevelnumber,如MDToken类中所实现的。

c3frrgcw

c3frrgcw1#

public boolean hasNext() {
    return this.buffer != null && !this.buffer.isEmpty(); // You may make changes to this line if needed
}

public MDToken next() {
    
    // Return null if there are no more tokens to process
    if (!hasNext()) {
        return null;
    }

    String line = buffer.split("\n", 2)[0]; // Get the first line from the buffer
    int newlineLength = line.length() < buffer.length() ? 1 : 0;
    buffer = buffer.substring(line.length() + newlineLength); // Remove the first line from the buffer

    int level = 1 + (line.length() - line.replaceFirst("^ *", "").length()) / 4;
    String trimmedLine = line.trim();
    if (trimmedLine.isEmpty()) {
        // Ignore empty lines and proceed to next token
        return next();
    }

    MDTokenType type;
    String text;
    int number = 0;

    // Check if the line is an ordered list item
    if (Character.isDigit(trimmedLine.charAt(0))) {
        type = MDTokenType.ORDERED_ITEM;
        int dotIndex = trimmedLine.indexOf('.');
        number = Integer.parseInt(trimmedLine.substring(0, dotIndex).trim());
        text = trimmedLine.substring(dotIndex + 1).trim();
    }
    // Check if the line is an unordered list item
    else if (trimmedLine.startsWith("-")) {
        type = MDTokenType.UNORDERED_ITEM;
        text = trimmedLine.substring(1).trim();
    }
    // If the line doesn't start with a digit or '-', it's not a valid list item
    else {
        current = null;
        return null;
    }

    // Create and return the new token
    current = new MDToken(type, text, level, number);
    return current;
}

字符串
您必须使用一些rg,例如\n

相关问题