Open Source Mac Linguistics Software

CRFSharp(aka CRF#) is a .NET(C#) implementation of Conditional Random Fields, an machine learning algorithm for learning from labeled sequences of examples. It is widely used in Natural Language Process (NLP) tasks, for example: word breaker, postagging, named entity recognized, query chunking and so on. CRF#'s mainly algorithm is the same as CRF++ written by Taku Kudo. It encodes model parameters by L-BFGS. Moreover, it has many significant improvement than CRF++, such as totally parallel encoding, optimizing memory usage and so on. Currently, when training corpus, compared with CRF++, CRF# can make full use of multi-core CPUs and only uses very low memory, and memory grow is very smoothly and slowly while amount of training corpus, tags increase. with multi-threads process, CRF# is more suitable for large data and tags training than CRF++ now. For example, in machine with 64GB, CRF# encodes model with more than 4.5 hundred million features quickly.

This project is used to segment text into semantic parts by meaning of language model.

wordseg project is a word segment module implemented by C#. It is used to segment text into tokens and to label token's attribute according its context and semantic by front-maximum matching and CRF algorithms. The following are some sentences need to be segmented: 张晓晨和付仲恺一起坐在家（西坝河东里社区）里的沙发上看非诚勿扰。 百度公司的名字源于“众里寻他千百度”这诗句。 After above sentences be segmented by wordseg, the result as follows for each sentence: 张晓晨[PER] 和 付仲恺[PER] 一起 坐 在 家 （ 西坝河东里社区[LOC] ） 里 的 沙发[PDT] 上 看 非 诚 勿扰 。 百度公司[ORG] 的 名字 源于 “ 众 里 寻 他 千百度 ” 这 诗句 。 In above, if a token has some attributes, the attribute result will be appended into the corresponding token within "[]". Since wordseg has introduced statistics model to segment text by context, for same sub string in different context, dif