congrats Icon
New! Announcing our next generation AI code completions
Read here
Tabnine Logo
Tokenizer
Code IndexAdd Tabnine to your IDE (free)

How to use
Tokenizer
in
coldash.easynlu.parse

Best Java code snippets using coldash.easynlu.parse.Tokenizer (Showing top 2 results out of 315)

origin: kolloldas/EasyNLU

public List<Derivation> parseSyntactic(String input){
  List<String> tokens = tokenizer.tokenize(input);
  List<String> tokensLower = new ArrayList<>(tokens.size());
  for(String token: tokens)
    tokensLower.add(token.toLowerCase());
  int N = tokens.size();
  Chart chart = new Chart(N+1);
  for(int e = 1; e <= N; e++) {
    for(int s = e-1; s >= 0; s--) {
      applyAnnotators(chart, tokens, s, e);
      applyLexicalRules(chart, tokensLower, s, e);
      applyBinaryRules(chart, s, e);
      applyUnaryRules(chart, s, e);
    }
  }
  List<Derivation> derivations = new LinkedList<>();
  for(Derivation d: chart.getDerivations(0, N))
    if(grammar.isRoot(d.rule))
      derivations.add(d);
  return derivations;
}
origin: kolloldas/EasyNLU

  @Test
  void tokenize() {
    Tokenizer tokenizer = new BasicTokenizer();

    String example = "$100, 10:45 1/2/3 4-5-6 1st 2nd 3RD 4th 10pm 3May";
    List<String> expected = Arrays.asList(
     "$100", "1045", "1", "2", "3", "4", "5", "6", "1", "2", "3", "4", "10", "pm", "3", "May"
    );

    assertEquals(expected, tokenizer.tokenize(example));
  }
}
coldash.easynlu.parseTokenizer

Most used methods

  • tokenize

Popular in Java

  • Reading from database using SQL prepared statement
  • setContentView (Activity)
  • getContentResolver (Context)
  • getSharedPreferences (Context)
  • BigInteger (java.math)
    An immutable arbitrary-precision signed integer.FAST CRYPTOGRAPHY This implementation is efficient f
  • HttpURLConnection (java.net)
    An URLConnection for HTTP (RFC 2616 [http://tools.ietf.org/html/rfc2616]) used to send and receive d
  • ServerSocket (java.net)
    This class represents a server-side socket that waits for incoming client connections. A ServerSocke
  • Permission (java.security)
    Legacy security code; do not use.
  • LinkedHashMap (java.util)
    LinkedHashMap is an implementation of Map that guarantees iteration order. All optional operations a
  • Options (org.apache.commons.cli)
    Main entry-point into the library. Options represents a collection of Option objects, which describ
  • 21 Best IntelliJ Plugins
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimAtomGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyStudentsTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now