How to use
TokenSequenceMatchDataAndTarget
in
cc.mallet.pipe

Best Java code snippets using cc.mallet.pipe.TokenSequenceMatchDataAndTarget (Showing top 2 results out of 315)

public void testStartState() {
  Pipe p = new SerialPipes(new Pipe[] {
      new LineGroupString2TokenSequence(),
      new TokenSequenceMatchDataAndTarget(Pattern
          .compile("^(\\S+) (.*)"), 2, 1),
      new TokenSequenceParseFeatureString(false), new TokenText(),
      new TokenSequence2FeatureVectorSequence(true, false),
      new Target2LabelSequence(), new PrintInputAndTarget(), });
  InstanceList data = new InstanceList(p);
  data.addThruPipe(new LineGroupIterator(new StringReader(toy), Pattern
      .compile("\n"), true));
  CRF crf = new CRF(p, null);
  crf.print();
  crf.addStatesForLabelsConnectedAsIn(data);
  crf.addStartState();
  CRFTrainerByLabelLikelihood crft = new CRFTrainerByLabelLikelihood(crf);
  Optimizable.ByGradientValue maxable = crft.getOptimizableCRF(data);
  assertEquals(-1.3862, maxable.getValue(), 1e-4);
  crf = new CRF(p, null);
  crf
      .addOrderNStates(data, new int[] { 1 }, null, "A", null, null,
          false);
  crf.print();
  crft = new CRFTrainerByLabelLikelihood(crf);
  maxable = crft.getOptimizableCRF(data);
  assertEquals(-3.09104245335831, maxable.getValue(), 1e-4);
}

public void testStartState() {
  Pipe p = new SerialPipes(new Pipe[] {
      new LineGroupString2TokenSequence(),
      new TokenSequenceMatchDataAndTarget(Pattern
          .compile("^(\\S+) (.*)"), 2, 1),
      new TokenSequenceParseFeatureString(false), new TokenText(),
      new TokenSequence2FeatureVectorSequence(true, false),
      new Target2LabelSequence(), new PrintInputAndTarget(), });
  InstanceList data = new InstanceList(p);
  data.addThruPipe(new LineGroupIterator(new StringReader(toy), Pattern
      .compile("\n"), true));
  CRF crf = new CRF(p, null);
  crf.print();
  crf.addStatesForLabelsConnectedAsIn(data);
  crf.addStartState();
  CRFTrainerByLabelLikelihood crft = new CRFTrainerByLabelLikelihood(crf);
  Optimizable.ByGradientValue maxable = crft.getOptimizableCRF(data);
  assertEquals(-1.3862, maxable.getValue(), 1e-4);
  crf = new CRF(p, null);
  crf
      .addOrderNStates(data, new int[] { 1 }, null, "A", null, null,
          false);
  crf.print();
  crft = new CRFTrainerByLabelLikelihood(crf);
  maxable = crft.getOptimizableCRF(data);
  assertEquals(-3.09104245335831, maxable.getValue(), 1e-4);
}

Javadoc

Run a regular expression over the text of each token; replace the text with the substring matching one regex group; create a target TokenSequence from the text matching another regex group.

For example, if you have a data file containing one line per token, and the label also appears on that line, you can first get a TokenSequence in which the text of each line is the Token.getText() of each token, then run this pipe, and separate the target information from the data information. For example to process the following,

 
BACKGROUND Then 
PERSON Mr. 
PERSON Smith 
BACKGROUND said 
...

use new TokenSequenceMatchDataAndTarget (Pattern.compile ("([A-Z]+) (.*)"), 2, 1).

Most used methods

<init>

Popular in Java

Running tasks concurrently on multiple threads
getSharedPreferences (Context)
getExternalFilesDir (Context)
runOnUiThread (Activity)
Menu (java.awt)
Rectangle (java.awt)
A Rectangle specifies an area in a coordinate space that is enclosed by the Rectangle object's top-
URI (java.net)
A Uniform Resource Identifier that identifies an abstract or physical resource, as specified by RFC
NoSuchElementException (java.util)
Thrown when trying to retrieve an element past the end of an Enumeration or Iterator.
JTextField (javax.swing)
StringUtils (org.apache.commons.lang)
Operations on java.lang.String that arenull safe. * IsEmpty/IsBlank - checks if a String contains
Top Vim plugins

How to useTokenSequenceMatchDataAndTarget in cc.mallet.pipe

Best Java code snippets using cc.mallet.pipe.TokenSequenceMatchDataAndTarget (Showing top 2 results out of 315)

How to use
TokenSequenceMatchDataAndTarget
in
cc.mallet.pipe