{
  "id": "enwiki-001288110012",
  "slug": "text-segmentation",
  "title": "Text segmentation",
  "qid": "",
  "lead": "Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics. The term applies both to mental processes used by humans when reading text, and to artificial processes implemented in computers, which are the subject of natural language processing. The problem is non-trivial, because while some written languages have explicit word boundary markers, such as the word spaces of written English and the distinctive initial, medial and final letter shapes of Arabic, such signals are sometimes ambiguous and not present in all written languages. Compare speech segmentation, the process of dividing speech into linguistically meaningful portions.",
  "body": "Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics. The term applies both to mental processes used by humans when reading text, and to artificial processes implemented in computers, which are the subject of natural language processing. The problem is non-trivial, because while some written languages have explicit word boundary markers, such as the word spaces of written English and the distinctive initial, medial and final letter shapes of Arabic, such signals are sometimes ambiguous and not present in all written languages. Compare speech segmentation, the process of dividing speech into linguistically meaningful portions. In English and many other languages using some form of the Latin alphabet, the space is a good approximation of a word divider (word delimiter), although this concept has limits because of the variability with which languages emically regard collocations and compounds. Many English compound nouns are variably written (for example, ice&nbsp;box = ice-box = icebox; pig&nbsp;sty = pig-sty = pigsty) with a corresponding variation in whether speakers think of them as noun phrases or single nouns; there are trends in how norms are set, such as that open compounds often tend eventually to solidify by widespread convention, but variation remains systemic. In contrast, German compound nouns show less orthographic variation, with solidification being a stronger norm. However, the equivalent to the word space character is not found in all written scripts, and without it word segmentation is a difficult problem. Languages which do not have a trivial word segmentation process include Chinese, Japanese, where sentences but not words are delimited, Thai and Lao, where phrases and sentences but not words are delimited, and Vietnamese, where syllables but not words are delimited. In some writing systems however, such as the Ge'ez script used for Amharic and Tigrinya among other languages, words are explicitly delimited (at least historically) with a non-whitespace character. The Unicode Consortium has published a Standard Annex on Text Segmentation, exploring the issues of segmentation in multiscript texts. Word splitting is the process of parsing concatenated text (i.e. text that contains no spaces or other word separators) to infer where word breaks exist. Word splitting may also refer to the process of hyphenation. Some scholars have suggested that modern Chinese should be written in word segmentation, with spaces between words like written English. Because there are ambiguous texts where only the author knows the intended meaning. For example, \"美国会不同意。\" may mean \"美国 会 不同意。\" (The US will not agree.) or \"美 国会 不同意。\" (The US Congress does not agree). For more details, see Chinese word-segmented writing.",
  "categories": [
    "106426338",
    "22922048",
    "22968701",
    "23075735",
    "23118434",
    "23603337",
    "24043274",
    "25704443",
    "30525121",
    "37160764"
  ],
  "license": "CC BY-SA 4.0",
  "source_uri": "https://en.wikipedia.org/w/index.php?oldid=1288110012",
  "content_hash": "",
  "created_at": null,
  "updated_at": 1746015574.0,
  "claims": [],
  "contradictions": [],
  "confidence": 0.6,
  "csl": {
    "id": "enwiki-001288110012",
    "type": "webpage",
    "title": "Text segmentation",
    "author": [
      {
        "family": "Anticloud FZ LLE",
        "literal": "Anticloud FZ LLE"
      }
    ],
    "container-title": "encyclopegia.org",
    "publisher": "Anticloud FZ LLE",
    "publisher-place": "Dubai, United Arab Emirates",
    "URL": "https://encyclopegia.org/a/text-segmentation",
    "issued": {
      "date-parts": [
        [
          2026,
          11,
          22
        ]
      ]
    },
    "accessed": {
      "date-parts": [
        [
          2026,
          11,
          22
        ]
      ]
    },
    "license": "CC BY-SA 4.0",
    "DOI": "eo1:enwiki-001288110012",
    "isBasedOn": "https://en.wikipedia.org/w/index.php?oldid=1288110012",
    "note": "Immutable snapshot enwiki-001288110012"
  }
}