IOB tagging

In natural language processing, IOB tagging (abbreviation for Inside–outside–beginning) (or BIO tagging) is a common format for tagging tokens in a chunking task.

Uses your system voice; voice and rate in settings.

Snapshot

≈ 3 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

In natural language processing, IOB tagging (abbreviation for Inside–outside–beginning) (or BIO tagging) is a common format for tagging tokens in a chunking task. IOB stands for "inside, outside, beginning". It was proposed in 1995.

Drawbacks

IOB syntax does not permit any nesting, so cannot (unless extended) also represent even very simple phenomena such as sentence boundaries (which are not trivial to locate reliably), the scope of parenthetical expressions in sentences, grammatical structures, nested Named Entities such as "University of Wisconsin Dept. of Computer Science", and so on. It also leaves no place for metadata such as an identifier for the particular sample, the confidence level of the NER assignment, and so on, which are commonplace in NLP systems.

Because of these limitations, data must often be converted out of IOB format, or projects must create custom extensions, which has led to a large number of not-quite-interoperable "IOB-like" formats. Many extended variations will also "pass" a non-extended parser, so it is easy to process incorrectly without noticing.

The space and "O" (meaning "not in any chunk") convey no information and could simply be omitted. The same is true for putting the "type" suffix on "I-" or "E-" markers as in some variants of "BIOES"; and for marking both "I" and "E" (if you have begun and not ended you are "in", and if you are "in", you have begun and not ended).

Some other formats deploy verbosity to improve readability and/or error-checking, but no such benefits appear to come to IOB in exchange for its verbosity. IOB's "one token per tag" depends on the tokenization used, even though tokenization is not standardized in NLP, and details of tokenization do not have to be entangled with the representations of NERs.

"11/31/2019" could be anywhere from one to five tokens in different systems, but the NER is the same. Some systems even permit whitespace within tokens, and space as a delimiter collides with this, narrowing the applicability of IOB and motivating more extensions.

"space" might or might not include tab, multiple spaces, hard spaces, and so on, differences which are difficult to detect when proofreading. IOB variants that allow multiple tokens per tag often use "/" or another reserved character to separate the tag from the token.

This effectively "reserves" that character, which then cannot occur in tokens, or must be escaped, introducing more incompatibilities. IOB files have no place to put commonly-needed meta-data, such as the character encoding being used, the data source, internal location-markers, and so on.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1373699743 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

3 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.