Processing text, or word processing?

Text processing and word processing can be confusing because text and words seem to be the same. However, there is an important distinction when they are used in reference to computer-based processing. Text normally refers to plain encoding of alphanumeric characters. Files encoded as text are also known as text files. They are the most basic and common type of computer encoding.

Word processing adds the formatting of text and other information related to the text. These files are normally encoded differently than text files. These are normally referred to as binary files. Users can access a text file directly from the operating system but binary files require a type of program to decode the data.

The astute reader can see that a document so defined may not only be of words but also of any other types of elements. This is, in fact, true. A binary file can be of anything and a program would be required to decode the document, identify its elements and process them whether for display or any other type of action. In particular, a program used to access a word processing file, requires a program known as a word processor, such as the popular Microsoft Word for Windows.

Summarizing, a binary file can encode all types of documents, including still images or audio. A word processing file is a type of binary file, which may also include format, location and other such information about the file's content. A text file is formed by the alphanumeric characters of the text. Text files are the common communication encoding that different computers can use. For this reason, text is the de-facto communication encoding in the web and computer-to-computer protocols across the Internet.

Information Retrieval Systems

Applying computer technologies to the number crunching needs of the day was one of the drivers in the implementation and maturity of computer technologies, but text processing was not far behind. After all, mathematics and text converged in areas related to encoding, cryptology and compression.

Engineers were aware that the new technologies of the 1950s might make possible multiple applications, including those to create, edit, and otherwise process text documents. By the 60s the transistor had enabled a revolution in miniaturization. At about the same time, advances in database technology were providing ideas about storage and access. By the 70s, network technology took computers to a larger scale.

During all this time, a small cadre of scientists and practitioners in the little known art of information retrieval had been researching and implementing systems to process text. This included the creation and editing, as well as other supporting tasks such as the storage and retrieval of documents, which were, in their own right, complete, elaborate and complex.

Terms like text processing, word processing and information retrieval were still confusing but were slowly starting to convey distinctive and different types of activities. The idea of a system that processes information was not far fetched anymore and the dream of Vannevar Bush was now possible. Information systems consolidated networks, databases, and all the newly implemented ideas related to the process of information. The information retrieval (IR) system would only be one component.

Search engines

The IR system was the component in the system that located a document to satisfy a given query that a user would create. The paradigm consider that the user query presented to the system was in the form of a template that started with “I want a document that contains…” and then the user would type a list of words and perhaps some parameters that provided a semantic relationship among the words. These parameters would be BOOLEAN (such as AND, OR, NOT) or distance (as how physically close two words should be), or indicate that synonyms or other semantic expansion should be used, etc. The system would then identify a document, or several documents that matched the query and possibly present them in a type of ranking order.

The only difference between an IR system and a search engine is the name, and may be also some of the document processing functionality.

Retrieving information, or documents?

The answer depends on whether one document in its totality satisfies the totality of the information need. In other words, a single document completely satisfies the user’s need. Ideally, the document will perfectly fit the information need without adding anything to it.

But information needs come in all shapes and forms whereas documents are pretty much fixed. This is true even for dynamic documents once they are created. Dynamic documents are ad hoc creatures that nevertheless encapsulate static content from that point forward.

Most documents are, in fact, compositions that authors put together in response to some urge. However, documents are also composite of integrated elements. In text, these would be sections, paragraphs, chapters, etc. Going at an even deeper level of granularity, one can count words and even letters as atomic elements. Even phonemes may enter the scene as yet another form of information. But in the end, these elements are parts of the document. Current computer-based information systems deliver documents because documents are convenient unit of analysis.

However, much activity has been taking place in areas of computer-based processing of information. Text mining is one of them, and it refers to the identification of segments in a narrative that have a particular meaning and can be followed up or build upon to construct and support bold statements.

Information representation

The search process has been greatly simplified throughout the years. A user enters one or several words. The system finds the names of files where the words appear. If the user supplies a combination of words, the system can find them individually or in some type of semantic relationship as in Boolean, or logic, operations. It is possible to ask the equivalent of “I want documents that discuss giraffes in Costa Rica”.

Indexing files form a representation system, a system representing the information in a collection of documents, a computer-based bibliographic representation. But there are other bibliographic representations. Document titles and other types of summaries are good examples of representation systems as well.

The reader will notice that each type of representation provides their own set of mutual semantic relationships, degree of specificity, and other attributes that correspond to their particular interpretations of the original information.

Indexing files

A naïve approach to document retrieval was implemented in the early days of computer-based text processing: simple word match. For this, the computer would read each word of each document, one word and one document at a time. The process included a comparison of each word with the sample word initially supplied by the user. The goal was to match sequences of bits, what the user provide to what was in the documents. The reader understands that this comparison was not at the level of meaning but at the level of symbols. Meaning is at the level of information and symbol is at the level of package. The symbol, or word, or sequences of bits and bytes, is the capsule of information.

The complexity of the operation would increase if two or more words were supplied. Moreover, the semantic relationship of the supplied words was also an issue, or rather how would the words be combined. In all, the processing of text using the sequential methodology of scanning was terribly expensive.

This obstacle required a new approach to use them taking advantage of their capabilities. This is the same approach that is at the foundation of many novel computer applications. Special intermediate files were created with a particular organization that encoded the relative position of words in the stored documents. These are the index files, or indexing. More specifically, the inverted index files. These files store a list of all the word in all the documents in a collection, including the name of the document (in terms of file name) and their position of the word in the document. This organization allows for all types of automatic, or computer-based, operations expanding the capabilities of human processors.

Organizing and representing information

There are many types of documents. Also, they come in many formats. Thy go from small leaflets to volumes of books. In general, a document is a package of information created to preserve some particular focal information and items related to that information. The components are normally organized in some order that may be sequential, hierarchical, or a combination of both.

Packages are convenient containers but also serve for storage purposes. Documents with related content, or the information they carry, are organized in collections. Semantically speaking, in an abstract space of information, related documents would be place closer to each other than to unrelated documents. The semantic distance is a measure of how similar documents are to each other.

Electronic documents have particular characteristics that make them amenable to automatic processing, such as encryption and compression. Although it is important to differentiate information from the package wherein it exists, most processing of information, particularly computer-based information processing treats the symbol or package as equal to the information it carries. In other words, the paper is only a word, it is not really paper.

Likewise, computer-based information processing is really processing of words, symbols, bits and bytes, but not meanings and concepts.