Tabnine Logo
Sentence.getTokens
Code IndexAdd Tabnine to your IDE (free)

How to use
getTokens
method
in
de.tudarmstadt.ukp.dkpro.teaching.core.Sentence

Best Java code snippets using de.tudarmstadt.ukp.dkpro.teaching.core.Sentence.getTokens (Showing top 5 results out of 315)

origin: de.tudarmstadt.ukp.dkpro.teaching/de.tudarmstadt.ukp.dkpro.teaching.core

@Override
public String toString() {
  return StringUtils.join(getTokens(), ' ');
}
 
origin: de.tudarmstadt.ukp.dkpro.teaching/de.tudarmstadt.ukp.dkpro.teaching.core

public String getFormattedString() {
  return "[" + StringUtils.join(getTokens(), ' ') + "]";
}
origin: de.tudarmstadt.ukp.dkpro.teaching/de.tudarmstadt.ukp.dkpro.teaching.corpus

@Ignore
@Test
public void tazTest() throws Exception {
  TazCorpus corpus = new TazCorpus();
  for (Sentence s : corpus.getSentences()) {
    for (int i=0; i<s.getTokens().size(); i++) {
      System.out.print(s.getTokens().get(i));
    }
  }
}

origin: de.tudarmstadt.ukp.dkpro.teaching/de.tudarmstadt.ukp.dkpro.teaching.frequency

public BrownProvider(int minN, int maxN) throws Exception {
  BrownCorpus brownCorpus = new BrownCorpus();
  
  cfd = new ConditionalFrequencyDistribution<Integer,String>();
  
  if (minN > maxN) {
    throw new IllegalArgumentException("minN > maxN");
  }
  
  for (int i=minN; i<=maxN; i++) {
    for (Sentence s : brownCorpus.getSentences()) {
      cfd.addSamples(
          i,
          new NGramIterable(s.getTokens(), i, i)
      );
    }
  }
}
 
origin: de.tudarmstadt.ukp.dkpro.teaching/de.tudarmstadt.ukp.dkpro.teaching.frequency

  public static void main(String[] args) throws Exception
  {
    Web1TProvider web1t = new Web1TProvider(new Locale("de"), 1);
    BrownCorpus brown = new BrownCorpus();
    
    StopWatch watch = new StopWatch();
    watch.start();
    watch.suspend();

    for (Text text : brown.getTexts()) {
      for (Sentence s : text.getSentences()) {
        for (String t : s.getTokens()) {
          watch.resume();
          web1t.getFrequency(t);
          watch.suspend();
        }
      }
    }
    
    double time = (double) watch.getTime() / 1000;
    System.out.println(time + "s");
  }
}
de.tudarmstadt.ukp.dkpro.teaching.coreSentencegetTokens

Popular methods of Sentence

    Popular in Java

    • Creating JSON documents from java classes using gson
    • startActivity (Activity)
    • compareTo (BigDecimal)
    • addToBackStack (FragmentTransaction)
    • SocketException (java.net)
      This SocketException may be thrown during socket creation or setting options, and is the superclass
    • Charset (java.nio.charset)
      A charset is a named mapping between Unicode characters and byte sequences. Every Charset can decode
    • SQLException (java.sql)
      An exception that indicates a failed JDBC operation. It provides the following information about pro
    • SSLHandshakeException (javax.net.ssl)
      The exception that is thrown when a handshake could not be completed successfully.
    • JCheckBox (javax.swing)
    • JComboBox (javax.swing)
    • CodeWhisperer alternatives
    Tabnine Logo
    • Products

      Search for Java codeSearch for JavaScript code
    • IDE Plugins

      IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
    • Company

      About UsContact UsCareers
    • Resources

      FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
    Get Tabnine for your IDE now