Support auto-evaluation

[schmmd]

Presently if we want evaluation figures for our NLP Tools we consult papers. These papers tend to be evaluated on out-of-domain corpora (WSJ) so they are not very helpful. They tend to not have any cross-domain statistics, so a parser that performs better on WSJ might actually perform worse on 4th Grade Biology Textbooks.

The NLP Stack could provide automatic evaluation for its tools. I imagine other NLP Toolkits provide this, but I can't think of anything off the top of my head. A lack of public-domain annotated corpora might be the culprit.

@mhrmm might have built something already to evaluate his parser. If not, perhaps Dirk could build this into the NLP Stack instead. I'm not sure how parsers are evaluated exactly. We might not have a public-domain gold corpus presently, but we might have one soon from Mark H's tool.

For tools such as the POS Tagger, evaluation is quite simple. Building a public-domain annotated corpus is also straightforward. We could choose some text from Wikipedia, run our POS tagger on it, and correct errors.

@dirkgr had this idea originally.

[mhrmm]

Dependency parsers are typically evaluated using attachment score (see
metric description in the attached document). It's simple enough to
implement, and there is code to compute (unlabeled) attachment score in:

https://github.com/allenai/parsers/blob/master/poly-parser/src/main/scala/org/allenai/parsers/polyparser/ParseEvaluator.scala

As Michael mentions, one question is to determine which test sets we want
to evaluate on. Currently I'm evaluating on three: WSJ test, the Brown-CM
subset (which is a collection of sentences from science fiction books), and
the Genia test set (which is a collection of biomedical abstracts).
Baseline numbers for Stanford and MALT on these test sets can be found here:

https://docs.google.com/spreadsheets/d/1lFtLI_KSZK8ULtmBiTJszeGaPqmeg9jny8W-dAEyuzI/edit?usp=sharing

Mark

On Fri, Jul 18, 2014 at 8:19 AM, Michael Schmitz notifications@github.com
wrote:

Presently if we want evaluation figures for our NLP Tools we consult
papers. These papers tend to be evaluated on out-of-domain corpora (WSJ) so
they are not very helpful. They tend to not have any cross-domain
statistics, so a parser that performs better on WSJ might actually perform
worse on 4th Grade Biology Textbooks.

The NLP Stack could provide automatic evaluation for its tools. I imagine
other NLP Toolkits provide this, but I can't think of anything off the top
of my head. A lack of public-domain annotated corpora might be the culprit.

@mhrmm https://github.com/mhrmm might have built something already to
evaluate his parser. If not, perhaps Dirk could build this into the NLP
Stack instead. I'm not sure how parsers are evaluated exactly. We might not
have a public-domain gold corpus presently, but we might have one soon from
Mark H's tool.

For tools such as the POS Tagger, evaluation is quite simple. Building a
public-domain annotated corpus is also straightforward. We could choose
some text from Wikipedia, run our POS tagger on it, and correct errors.

@dirkgr https://github.com/dirkgr had this idea originally.

—
Reply to this email directly or view it on GitHub
#52.