Microsoft Previews PenWindows

Not to be upstaged by an upstart, Microsoft Corp. is finally talking about PenWindows, its pen-based computing environment. Even though many details of the environment haven’t been worked out (and others aren’t being talked about), Microsoft wagers that PenWindows will be ready by the end of this year, adding that Microsoft already has five publicly committed OEMs — Momenta, Wang, NCR, Grid, and Kyocera. Microsoft is expected to announce a software developers kit (SDK) for PenWindows on February 21st of this year.

PenWindows is a set of extensions to the Windows 3.1 environment that supports interaction with stylus devices and recognizes handwritten text and gestures. By building on a foundation composed of the over 600 functions in the Windows application program interface (API), the PenWindows environment can be implemented with modest incremental effort (about 35 additional functions).

PenWindows will be an OEM product tied to Windows 3.1; it hasn’t been decided if the PenWindows SDK will be part of the Windows 3.1 SDK. Nevertheless, all Windows 3.1 applications will run under PenWindows without modification and pen enhancements will be added to existing Microsoft programs (Word, Excel, PowerPoint, etc.). In addition, PenWindows provides mechanisms for using the pen to control applications that were not written with knowledge of the PenWindows API. For example, an “X” drawn with a pen across some text in a word processor window can be recognized and translated into a double-click/delete sequence of messages, which are then passed to the application. Likewise, buttons and fields in existing dialog boxes can be carefully fed input from the pen device, through a similar process of translation.

Compatibility with the Windows environment and with existing Windows applications is both PenWindows’ greatest strength as well as its greatest weakness. On the positive side, Windows compatibility gives PenWindows an automatic and substantial user and application base. The prospect of being able to use familiar applications such as Word or Excel in a pen-based environment will be a major attraction to many users. On the negative side, it forces constraints on PenWindows that may seriously affect its performance. For example, translating an “X” to a double-click/delete sequence takes machine cycles. GO’s PenPoint doesn’t have to worry about translating pen input to mouse or keyboard equivalents.

Handwriting Recognition

As with most pen-based developers, Microsoft is backing away from “handwriting” per se, maintaining that there’s more to pen-based computing than just handwriting. To this end, Microsoft changed the official moniker from Windows/H (“H” for handwriting) to PenWindows. Nevertheless, handwriting recognition is central to the system and Microsoft is investing in its own recognition technology. This contradicts some recent press reports (see Infoworld January 7, 1991), which claim that PenWindows “does not yet support true character recognition”.

Microsoft claims its recognizer functions at speeds and accuracies equal to or better than most other recognizers on the market. At the same time, the recognition subsystem is modular, allowing developers to plug in other recognition schemes (such as the one from CIC) if desired. In addition, multiple recognition modules can co-exist on the system, and the order in which these are invoked can be controlled by the application. The recognition method, like that of other interactive character recognition systems, uses more than just the bit-mapped image of a character as its raw material. Data about stylus dynamics, such as the stroke order and the speed of pen motion, is also used.

As the user “writes” on the display and the “ink” flows, the recognition process begins almost immediately. The pen driver sidesteps both Windows and DOS to effectively function as a concurrently executing process, despite the fact that neither Windows nor DOS support true multi-tasking (more on this later).

Controlling the Recognition Process

The application has various ways by which it can control how recognition is accomplished. For example, an application can state that it is expecting purely numeric input. PenWindows uses a dictionary to disambiguate choices. In addition, user dictionaries that run concurrently with the PenWindows dictionary can be added to the system. In addition, an application can ask to receive not just the closest matching character or word but rather the entire decision tree of matches. This is useful information in cases of extremely ambiguous input. For example, if a user wrote the word c-a-t, but with a misshapen last letter that resembled an “r”, the recognizer could return the tree c-a-[t/r] (note that this representation is not the exact form that is actually used).

In addition to recognizing text, PenWindows’ recognizer identifies a set of gestures — delete, insert, copy, etc. — in the context in which they’re used. As mentioned earlier, an “X” drawn across some text in a window will be recognized as a delete-word command. Additional gestures, including user-defined ones, will be added in the future. The recognition process is controlled by means of a data structure called a Recognition Context (RC). Experienced Windows programmers will notice the similarity in name to another construct in the Windows environment, called a Device Context (DC). Just as a Device Context controls how graphics are drawn on the Windows screen, a Recognition Context’s settings control the variations in how recognition is accomplished.

Tracking the Learning Curve

Microsoft believes that familiar-sounding constructs like RC/DC will drastically ease the learning process for those developers who are already experienced with programming for Microsoft Windows. According to Microsoft, this amounts to 5-10,000 additional programmers per month, if sales of the Windows SDK are any indication.

One limitation of this evolutionary approach to the PenWindows API is that it precludes adding anything that is significantly new or radically different. An example of a approach that is not evolutionary is the move to an environment which is truly object-oriented such as GO’s PenPoint system. However, given the steep learning curve for both the Windows API and Go’s PenPoint system, this high level of developer-compatibility may be the critical factor ensuring PenWindow’s success.

Proximity Awareness

A certain class of digitizers can track pen motion even before the stylus makes contact with the surface, and can detect when the pen enters and leaves a zone of proximity to the surface. PenWindows can take advantage of this proximity information, if the appropriate digitizing hardware is present.

As in other systems, the proximity data is used to enhance user interaction — for example, by detecting when the pen enters or leaves a field in a dialog box, even when it is held above the surface. Also, the proximity data is made available to the application so that the programmer could endow it with any additional meaning. For example, a drawing program could offer a stylus that behaves more like an artist’s paintbrush, in which the stroke gets wider and denser (more ink) as the brush is pushed closer to the surface.

Another area in which proximity data can be used is in enhancing the text recognition process. In a scheme invented by one of the writers here (and likely by other researchers in the field), proximity information could be added to the database of character prototypes. A prototype would contain not just the information about actual penstrokes on the surface, but also information about any “hidden strokes” that occur when the pen travels above the surface in the middle of a glyph. Although this method requires a larger volume of data and more processing, there are situations (such as signature verification in financial and legal applications) where the increased accuracy is well worth the cost.

In practice, there is one manufacturer, PenCept, that incorporates pressure data into its database of prototypes. However, no commercial recognition system that we are aware of uses proximity data in this way, and Microsoft’s recognizer is no exception. Perhaps this is because system designers want to minimize the dichotomy between proximity-based and non-proximity-based systems.

Object Linking and Embedding

Key to application development under PenWindows is a facility already available in Windows 3.0, Microsoft’s “object linking and embedding” (OLE) protocol. Not surprisingly, GO has announced a similar, if not more powerful, embedding facility for its PenPoint environment called the Embedded Document Architecture (EDA). Both EDA and OLE can be seen as commercializations of the “compound document” idea, a concept which has been intensively studied at various university research centers over the last decade (for example, the Andrew project at Carnegie-Mellon).

Microsoft’s vision of the compound document has begun to be supported by several application developers. OLE allows one document, say a word processor, to contain within it other document components that represent entirely different kinds of functionality, such as a spreadsheet or chart from another application. This is more than cut-and-paste of data from one program to another. Extending the concept of a “hot link”, one application can effectively contain and manipulate objects belonging to another application — and if need be, invoke the other application to modify these objects. The OLE protocol is powerful enough to allow a document to contain components belonging to applications written after the original application was designed.

This feature has particular importance for notebook and palmtop computers, in which storage is highly constrained, and where maximum integration and reusability of components are critical. In theory, a chunk of spreadsheet functionality needs to exist in only one place in the system, and can then be used by all other applications. In practice, compliance with this standard depends on the goodwill and consistent behavior of application vendors. A better approach is one in which the OLE functionality “falls out” naturally due to the inherently object-oriented nature of the environment. Then, an application vendor only has to go out of their way if they desire not to support the protocol.

Data Compression and Recognition

An open question is whether notebook and palmtop computers will all use hard drives for mass storage. To be truly useful, the electronic notebook must be available for most of the working day (say 5-6 hours per day). The requirement for extended battery life works against designing in a requirement for a hard drive, no matter how small or light.

Given that Windows 3.0 is basically unusable without a hard drive, it may be that PenWindow’s authors hold the same view about their creation. Thus, one important technique for conserving storage — transparent compression and decompression of data upon access — is not used in PenWindows. However, according to Microsoft, there is nothing to preclude adding in this feature at a later time.

Data compression, however, is used by Microsoft within the recognizer in order to speed-up the recognition process. The database of prototypes against which pattern-matching is done is stored in compressed form. For example, an uncompressed representation of a character might require about 90 data points. By compressing this representation to the equivalent of 10 or fewer points, the system can match a candidate character much quicker than if character and prototypes were not compressed.

System Performance Is the Question

A big challenge for all pen-based systems is providing sufficient responsiveness to the user. As with other systems, PenWindows’s performance bears examination. A limitation specific to Microsoft’s system is that, except in virtual-86 mode of Windows’s 386 enhanced mode, Windows is a non-preemptive multitasking environment. In all other modes, Windows is event-driven and the application yields control of the processor only when it wants to.

In a non-preemptive, event-driven model of operation, an application continually cycles through an endless loop of first receiving events (such as characters entered at the keyboard, or mouse clicks from the mouse) and then dispatching them to the appropriate procedure that handles that event. Although task-switching can occur between applications at event-processing time, within an application, the order of processing is strictly sequential and synchronous. But, for optimum results, pen input must be handled differently than keyboard or mouse input. The mode of operation must be asynchronous and concurrent. The pen movements must be dynamically displayed as bit-mapped strokes on the display while, at the same time, the recognition process examines the strokes, matches these with characters and then displays the characters. Additionally, there may be other system tasks that must be processed at the same time.

Microsoft claims that the pen driver in PenWindows deals with this requirement by sidestepping both Windows and DOS, and effectively providing true, pre-emptive multitasking between the application and the driver. This may be why PenWindows will only run in the Enhanced and Standard modes of Microsoft Windows, i.e., the 386 and 286 modes, respectively. The mode for 8088 CPUs — Real Mode — will not be supported.

This isn’t to say that Windows itself does not support a limited form of preemptive multitasking. Virtual-8086 mode allows multiple virtual DOS machines (a la Windows/386) utilizing preemptive multitasking. However, this offers no benefit to PenWindows applications, only to DOS programs running under PenWindows.

Applications and the PenWindows Desktop

With the exception of stylus-based interaction, PenWindows sticks closely to the Windows metaphor. As in 3.0, sample PenWindows applications will be included, among them a notebook and personal calendar. These are not full-fledged applications, but more properly called mini-apps, or, in Microsoft lingo, “applets”. The system will contain a notebook applet that is PenWindows surprisingly similar in concept to one bundled with GO’s PenPoint system.

The notebook in the PenPoint system is much more developed than a mini-app, because, according to GO, the metaphor of a notebook with a table of contents is “the central organizing concept” of the PenPoint environment. GO calls this concept the Notebook User Interface (NUI).

It will be interesting to see if GO or Microsoft attempts to patent this metaphor, given that at least one commercial computer graphics product (the page design workstation from Camex Inc., built in 1983), was based on this notebook metaphor.

Quotes From The Field: “The promise that pen-based computing holds out is that, like the Macintosh, it represents the next quantum step in terms of bringing in a new group of users.” — Vern Raburn, Slate Corp.