Open Source Linguistics Software

This used to be the experimentation and production space for the Special Interest Group (SIG) of the Text Encoding Initiative (TEI) called "TEI for Linguists", LingSIG for short. Currently, this is a storage place for documents produced by the SIG. Use https://github.com/LingSIG to access the current production space.

Includes • Scripts for the program/ code developed • High level architecture diagrams • Install guides for developers • Links to end user documentation on the AusNC website Note: The BSD license applies to customised plug-ins, scripts and ingest programs developed by the AusNC project team. Additional open source, 3rd party software products used by the AusNC solution are referenced on our SF wiki space.

Entity recognition and normalization software for biomedical text

The MITRE Annotation Toolkit (MAT) is a suite of tools which can be used for automated and human tagging of annotations. Annotation is a process, used mostly by researchers in natural language processing, of enhancing documents with information about the various phrase types the documents contain. MAT supports both UI interaction and command-line interaction, and provides various levels of control over the overall annotation process. It can be customized for specific tasks (e.g., named entity identification, de-identification of medical records). The goal of MAT is not to help you configure your training engine (in the default case, the Carafe CRF system) to achieve the best possible performance on your data. MAT is for "everything else": all the tools you end up wishing you had.

Annoschemer is a little tool for easy editing of MMAX2 annotationschemes.

A simulation package for investigating the dynamics of complex controversy.

The BioLemmatizer is a domain-specific lemmatization tool for the morphological analysis of biomedical literature. It is tailored to the biological domain through integration of several published lexical resources related to molecular biology. It focuses on the inflectional morphology of English, including the plural form of nouns, the conjugations of verbs, and the comparative and superlative form of adjectives and adverbs. README: https://sourceforge.net/projects/biolemmatizer/files/ The BioLemmatizer 1.2 release adds an optional functionality to normalize British English spellings into American English spellings and then retrieve corresponding lemmas. If you use the BioLemmatizer to support academic research, please cite the following paper: Haibin Liu, Tom Christiansen, William A Baumgartner Jr, and Karin Verspoor BioLemmatizer: a lemmatization tool for morphological processing of biomedical text Journal of Biomedical Semantics 2012, 3:3.

CRFSharp(aka CRF#) is a .NET(C#) implementation of Conditional Random Fields, an machine learning algorithm for learning from labeled sequences of examples. It is widely used in Natural Language Process (NLP) tasks, for example: word breaker, postagging, named entity recognized, query chunking and so on. CRF#'s mainly algorithm is the same as CRF++ written by Taku Kudo. It encodes model parameters by L-BFGS. Moreover, it has many significant improvement than CRF++, such as totally parallel encoding, optimizing memory usage and so on. Currently, when training corpus, compared with CRF++, CRF# can make full use of multi-core CPUs and only uses very low memory, and memory grow is very smoothly and slowly while amount of training corpus, tags increase. with multi-threads process, CRF# is more suitable for large data and tags training than CRF++ now. For example, in machine with 64GB, CRF# encodes model with more than 4.5 hundred million features quickly.

Encode Arabic provides tools for encoding and decoding Arabic in Haskell, Python, Perl, or LaTeX. Interprets the ArabTeX notation to generate original orthography or phonetic transcription. Supports Buckwalter and other romanizations. Converts legacy byte encodings into Unicode. http://github.com/otakar-smrz/encode-arabic

NOTE: For latest version, please visit https://github.com/ispasic/FlexiTerm. FlexiTerm is an open-source software tool for automatic term recognition. FlexiTerm uses a range of methods to neutralise the main sources of term variation. FlexiTerm is robust enough for less formally structured texts, such as those found in patient blogs or medical notes.

KneeTex is an open–source, stand–alone application for information extraction from narrative reports that describe an MRI scan of the knee. Given an MRI report as input, the system outputs the corresponding clinical findings in the form of JavaScript Object Notation objects. The extracted information is mapped onto TRAK, an ontology that formally models knowledge relevant for the rehabilitation of knee conditions. As a result, formally structured and coded information allows for complex searches to be conducted efficiently over the original MRI reports, thereby effectively supporting epidemiologic studies of knee conditions.

A collection of Metasyntaxes like EBNF for .Net including a definition file parser and an expression tree.

MinGen is a Minimalist generator, the logical opposite to a parser. MinGen generates syntactically valid sentences by following the rules of Minimalism.

ATTENTION! Morfologik is now at GitHub: https://github.com/morfologik/

A public repository of open source scripts and small programs related to linguistics and language.

The Open Translation Engine (OTE) is a web-based translation dictionary manager. The OTE allows a community of users to create and manage one or many translation dictionaries. The OTE is written in PHP and uses a MySQL database.

This project is used to segment text into semantic parts by meaning of language model.

TIES (Text Information Extraction System) is a clinical text search engine that uses Natural Language Processing techniques to extract medical concepts from free text clinical reports. It provides secure de-identified access to this information and has in built collaboration tools and honest broker functionality. It is licensed for academic use under the BSD license. For commercial use please contact Nexi at http://nexihub.com *** NOTICE: this software and forum are no longer maintained, as of 8/15/2019. You are free to continue to use this software under the license for academic use under the BSD license. For commercial use please contact Nexi at http://nexihub.com

wordseg project is a word segment module implemented by C#. It is used to segment text into tokens and to label token's attribute according its context and semantic by front-maximum matching and CRF algorithms. The following are some sentences need to be segmented: 张晓晨和付仲恺一起坐在家（西坝河东里社区）里的沙发上看非诚勿扰。 百度公司的名字源于“众里寻他千百度”这诗句。 After above sentences be segmented by wordseg, the result as follows for each sentence: 张晓晨[PER] 和 付仲恺[PER] 一起 坐 在 家 （ 西坝河东里社区[LOC] ） 里 的 沙发[PDT] 上 看 非 诚 勿扰 。 百度公司[ORG] 的 名字 源于 “ 众 里 寻 他 千百度 ” 这 诗句 。 In above, if a token has some attributes, the attribute result will be appended into the corresponding token within "[]". Since wordseg has introduced statistics model to segment text by context, for same sub string in different context, dif

CRF decoder is the simplified version of CRF++, only for decoding the sequential data. It removes the training component and its correspondent codes from CRF++, which makes CRF decoder more reabable and understandable for freshman.

A simple java library for text and object oriented code. Among the different available packages, there are for text analysis (levenshtein and ngram fingerprinting), a grammar framework, simple object persistence (very light and dependence free), ...

A (UTF-8) Chinese word segmentation program based on ngram language model. It is written in C++, provides multi-threading and high throughput.

stocleka is a project divided into a UI and a library for cleaning user stories and converting them to arff files (used for Weka). it may be mainly used for research and scientific purposes.

This project has moved to GitHub: https://github.com/rsling/texrex https://github.com/rsling/cow