congrats Icon
New! Announcing Tabnine Chat Beta
Learn More
Tabnine Logo
TokenSequenceMatchDataAndTarget
Code IndexAdd Tabnine to your IDE (free)

How to use
TokenSequenceMatchDataAndTarget
in
cc.mallet.pipe

Best Java code snippets using cc.mallet.pipe.TokenSequenceMatchDataAndTarget (Showing top 2 results out of 315)

origin: com.github.steveash.mallet/mallet

public void testStartState() {
  Pipe p = new SerialPipes(new Pipe[] {
      new LineGroupString2TokenSequence(),
      new TokenSequenceMatchDataAndTarget(Pattern
          .compile("^(\\S+) (.*)"), 2, 1),
      new TokenSequenceParseFeatureString(false), new TokenText(),
      new TokenSequence2FeatureVectorSequence(true, false),
      new Target2LabelSequence(), new PrintInputAndTarget(), });
  InstanceList data = new InstanceList(p);
  data.addThruPipe(new LineGroupIterator(new StringReader(toy), Pattern
      .compile("\n"), true));
  CRF crf = new CRF(p, null);
  crf.print();
  crf.addStatesForLabelsConnectedAsIn(data);
  crf.addStartState();
  CRFTrainerByLabelLikelihood crft = new CRFTrainerByLabelLikelihood(crf);
  Optimizable.ByGradientValue maxable = crft.getOptimizableCRF(data);
  assertEquals(-1.3862, maxable.getValue(), 1e-4);
  crf = new CRF(p, null);
  crf
      .addOrderNStates(data, new int[] { 1 }, null, "A", null, null,
          false);
  crf.print();
  crft = new CRFTrainerByLabelLikelihood(crf);
  maxable = crft.getOptimizableCRF(data);
  assertEquals(-3.09104245335831, maxable.getValue(), 1e-4);
}
origin: cc.mallet/mallet

public void testStartState() {
  Pipe p = new SerialPipes(new Pipe[] {
      new LineGroupString2TokenSequence(),
      new TokenSequenceMatchDataAndTarget(Pattern
          .compile("^(\\S+) (.*)"), 2, 1),
      new TokenSequenceParseFeatureString(false), new TokenText(),
      new TokenSequence2FeatureVectorSequence(true, false),
      new Target2LabelSequence(), new PrintInputAndTarget(), });
  InstanceList data = new InstanceList(p);
  data.addThruPipe(new LineGroupIterator(new StringReader(toy), Pattern
      .compile("\n"), true));
  CRF crf = new CRF(p, null);
  crf.print();
  crf.addStatesForLabelsConnectedAsIn(data);
  crf.addStartState();
  CRFTrainerByLabelLikelihood crft = new CRFTrainerByLabelLikelihood(crf);
  Optimizable.ByGradientValue maxable = crft.getOptimizableCRF(data);
  assertEquals(-1.3862, maxable.getValue(), 1e-4);
  crf = new CRF(p, null);
  crf
      .addOrderNStates(data, new int[] { 1 }, null, "A", null, null,
          false);
  crf.print();
  crft = new CRFTrainerByLabelLikelihood(crf);
  maxable = crft.getOptimizableCRF(data);
  assertEquals(-3.09104245335831, maxable.getValue(), 1e-4);
}
cc.mallet.pipeTokenSequenceMatchDataAndTarget

Javadoc

Run a regular expression over the text of each token; replace the text with the substring matching one regex group; create a target TokenSequence from the text matching another regex group.

For example, if you have a data file containing one line per token, and the label also appears on that line, you can first get a TokenSequence in which the text of each line is the Token.getText() of each token, then run this pipe, and separate the target information from the data information. For example to process the following,

 
BACKGROUND Then 
PERSON Mr. 
PERSON Smith 
BACKGROUND said 
... 
use new TokenSequenceMatchDataAndTarget (Pattern.compile ("([A-Z]+) (.*)"), 2, 1).

Most used methods

  • <init>

Popular in Java

  • Running tasks concurrently on multiple threads
  • getSharedPreferences (Context)
  • getExternalFilesDir (Context)
  • runOnUiThread (Activity)
  • Menu (java.awt)
  • Rectangle (java.awt)
    A Rectangle specifies an area in a coordinate space that is enclosed by the Rectangle object's top-
  • URI (java.net)
    A Uniform Resource Identifier that identifies an abstract or physical resource, as specified by RFC
  • NoSuchElementException (java.util)
    Thrown when trying to retrieve an element past the end of an Enumeration or Iterator.
  • JTextField (javax.swing)
  • StringUtils (org.apache.commons.lang)
    Operations on java.lang.String that arenull safe. * IsEmpty/IsBlank - checks if a String contains
  • Top Vim plugins
Tabnine Logo
  • Products

    Search for Java codeSearch for JavaScript code
  • IDE Plugins

    IntelliJ IDEAWebStormVisual StudioAndroid StudioEclipseVisual Studio CodePyCharmSublime TextPhpStormVimGoLandRubyMineEmacsJupyter NotebookJupyter LabRiderDataGripAppCode
  • Company

    About UsContact UsCareers
  • Resources

    FAQBlogTabnine AcademyTerms of usePrivacy policyJava Code IndexJavascript Code Index
Get Tabnine for your IDE now