Tabnine Logo
Tokenizer
Code IndexAdd Tabnine to your IDE (free)

How to use
Tokenizer
in
coldash.easynlu.parse

Best Java code snippets using coldash.easynlu.parse.Tokenizer (Showing top 2 results out of 315)

origin: kolloldas/EasyNLU

public List<Derivation> parseSyntactic(String input){
  List<String> tokens = tokenizer.tokenize(input);
  List<String> tokensLower = new ArrayList<>(tokens.size());
  for(String token: tokens)
    tokensLower.add(token.toLowerCase());
  int N = tokens.size();
  Chart chart = new Chart(N+1);
  for(int e = 1; e <= N; e++) {
    for(int s = e-1; s >= 0; s--) {
      applyAnnotators(chart, tokens, s, e);
      applyLexicalRules(chart, tokensLower, s, e);
      applyBinaryRules(chart, s, e);
      applyUnaryRules(chart, s, e);
    }
  }
  List<Derivation> derivations = new LinkedList<>();
  for(Derivation d: chart.getDerivations(0, N))
    if(grammar.isRoot(d.rule))
      derivations.add(d);
  return derivations;
}
origin: kolloldas/EasyNLU

  @Test
  void tokenize() {
    Tokenizer tokenizer = new BasicTokenizer();

    String example = "$100, 10:45 1/2/3 4-5-6 1st 2nd 3RD 4th 10pm 3May";
    List<String> expected = Arrays.asList(
     "$100", "1045", "1", "2", "3", "4", "5", "6", "1", "2", "3", "4", "10", "pm", "3", "May"
    );

    assertEquals(expected, tokenizer.tokenize(example));
  }
}
coldash.easynlu.parseTokenizer

Most used methods

  • tokenize

Popular in Java

  • Reactive rest calls using spring rest template
  • scheduleAtFixedRate (ScheduledExecutorService)
  • getSystemService (Context)
  • getSharedPreferences (Context)
  • Color (java.awt)
    The Color class is used to encapsulate colors in the default sRGB color space or colors in arbitrary
  • Window (java.awt)
    A Window object is a top-level window with no borders and no menubar. The default layout for a windo
  • Selector (java.nio.channels)
    A controller for the selection of SelectableChannel objects. Selectable channels can be registered w
  • CountDownLatch (java.util.concurrent)
    A synchronization aid that allows one or more threads to wait until a set of operations being perfor
  • Response (javax.ws.rs.core)
    Defines the contract between a returned instance and the runtime when an application needs to provid
  • Reflections (org.reflections)
    Reflections one-stop-shop objectReflections scans your classpath, indexes the metadata, allows you t
  • Top Sublime Text plugins
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now