我怎样写一个LuceneFilter标准化文本

Question

所以，我有我的基本代码

public static final Pattern DIACRITICS_AND_FRIENDS 
     = Pattern.compile("[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+"); 


private static String stripDiacritics(String str) { 
    str = Normalizer.normalize(str, Normalizer.Form.NFD); 
    str = DIACRITICS_AND_FRIENDS.matcher(str).replaceAll(""); 
    return str; 
} 

但我怎么把这个变成一个TokenFilter，我用NormalizeCharMap前但那只是修改字符串文字好，即时通讯使用Lucene 4

Answer 1

你需要重写incrementToken()方法，在其中将更新CharTermAttribute：

public final class DiacriticFilter extends TokenFilter { 
    private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); 

    @Override 
    public final boolean incrementToken() throws IOException { 
     if (input.incrementToken()) { 
      String result = stripDiacritics(new String(termAtt.buffer())); 
      char[] newBuffer = result.toCharArray(); 
      termAtt.copyBuffer(newBuffer, 0, newBuffer.length) 
      termAtt.setLength(newBuffer.length); 
      return true; 
     } else { 
      return false; 
     } 
    } 

    private static String stripDiacritics(String str) { 
     str = Normalizer.normalize(str, Normalizer.Form.NFD); 
     str = DIACRITICS_AND_FRIENDS.matcher(str).replaceAll(""); 
     return str; 
    } 
}