PenWindows Development Using Visual Basic

Moshe Lichtman is a product manager for Microsoft and coauthor of the Complete Guide to the C Language (Hod-Ami Publishing, 1988), a textbook published in Israel. He can be contacted at One Microsoft Way, Redmond, WA 98052. This article is reprinted with permission from the May 1992 issue of Dr. Dobb’s Journal, copyright 1992. Source code referenced in this article is available from Dr. Dobb’s Journal, 411 Borel Ave., San Mateo, CA 94402.

“Windows for Pens” is the name given to a set of extensions to version 3.1 of Microsoft Windows. Windows for Pens adds about 70 functions that support pen-based applications, to the existing 600-plus functions in the Windows 3.1 API. Windows for Pens functions can be divided into six groups: Pen Interface, Pen Data, Recognizer, Pen module, Drivers, and Dictionary.

The Pen Interface module includes most of the functions required to build pencentric applications. It consists of 32 functions, the most important being ProcessWriting(), a high-level recognition function which handles the entire recognition process. Your application passes control to this function, which tracks pen motion and then returns a set of recognized characters. For flexibility, the Pen Interface module also allows low-level access to the recognition and data conversion functions. The module also includes training functions which facilitate contextual training. Lastly, this module includes the virtual event handlers that add pen compatibility to existing Windows applications.

Pen Data functions include all the logic for ink and pen data manipulation and data transfer between applications. Functions in this module provide device-specific information (e.g. stylus pressure, angle, proximity to surface, and so on), data compression, and additional data conversion functions.

Most of the functions in the remaining four modules are used by specialized applications and by Windows for Pens device drivers. The Custom Recognizer module includes functions which must be supported by any Windows for Pens recognizer. These functions include: configuration and initialization, recognition, and training. The Pen module and Drivers module include functions which pen and display drivers must support or use in addition to the functionality required from standard Windows drivers.

The Dictionary module includes a single function, DictionaryProc, which enables application developers to build their own dictionaries in addition to the standard dictionary of words which is provided with Windows for Pens. Dictionaries can significantly enhance recognition in those situations where the input to be recognized can be constrained to a set of words.

The pen extensions (excluding the recognition module) are reasonably compact in size, under 130 KBytes. Windows for Pens can run on a variety of platforms spanning a range of form factors and hardware configurations. In conjunction with the forthcoming ROMable Windows, Windows for Pens will run comfortably on diskless platforms with total memory size (both ROM and RAM) of less than 2 MBytes.

Designing a windowing environment to accommodate small form factors comfortably is not a simple task. The answer is not just shrinking everything to fit the smaller screen. The standard Windows environment does not work well in this situation. However, we feel that the solution is not to adopt a new rigid paradigm that forces a specific metaphor on application developers. Rather, the system software should be modular and open enough to allow developers to easily tailor their applications to target markets and platforms. Windows for Pens provides this flexibility. Windows for Pens also provides support for power management and Flash memory, both of which are essential to small tablet and handheld computers.

Another area where architectural flexibility is needed is handwriting recognition. The embryonic nature of handwriting recognition technology and the need to support a multitude of different languages and markets dictates a very modular design. The Windows for Pen APIs which enables ISVs and handwriting recognition vendors to build custom recognizers as well as support foreign languages. In the future, recognition of cursive writing and voice input will also be able to fit into this architecture.

Visual Basic and Pens

To create a standard Windows application using Microsoft’s Visual Basic, you interactively design its interface by dragging and placing GUI elements known as “controls” onto a layout form. You specify the behavior of your application by attaching Visual Basic code to these controls. Programmers can extend the default functionality in Visual Basic by means of custom controls. Custom controls are similar to Windows controls and are typically written in C, following the specifications in the Visual Basic Control Development Kit (CDK). Once created, custom controls are basically indistinguishable from those controls that come with the product. The pen-oriented custom controls added to Visual Basic provide a high-level interactive way of creating pencentric programs. Although some pencentric applications need to use the full Windows for Pens API (for example, a calligraphy program that relies on device-specific information such as stylus pressure and angle), the needs of the majority of pencentric applications can be satisfied with smaller number of functions. These commonly used functions are found in the Windows for Pens custom controls, and are contained in the file pencntrl.vbx. Currently, this file is only available as part of the Windows 3.1 Pen SDK, but future versions of Visual Basic will include these controls as an integral part of the development environment. The BEdit and HEdit custom controls replace the standard Visual Basic text input controls. HEdit is a pen-enhanced version of the text box control. It supports most of the regular text box properties and adds the handwriting and inking capabilities from Windows for Pens API. When running, it accepts handwriting input and displays the corresponding recognized text. Since many of the initial pen applications will be form-based, the HEdit has an additional BorderStyle setting, Underline, which applies only to single line controls.

BEdit, or “boxed edit” control, is a superset of HEdit and provides the application with comb or box style guides that accept pen input. Each segment or box accepts only a single character of input. The BEdit control is used to work around the limitations of today’s recognition algorithms. BEdit provides important baseline and segmentation information to the recognizer. By dividing up handwritten text into single characters placed one at a time into box or comb guides, users write more neatly and recognition accuracy is significantly enhanced. BEdit properties control the size of the BEdit cells, their shape (box vs. comb), and the number of rows and columns in a certain control. BEdit is for fields which accept a pre-defined text length. For example, phone numbers will typically not exceed 15 characters.

Pencentric Properties

Unlike typed text whose size is controlled by the application, handwriting size depends solely on user habits. Users tend to write larger because of the limited resolution and display quality found in today’s pen computers. Pen applications address this problem by allowing inking in areas that extend beyond the displayed input box. This area is sometimes called the “gray area.” Its size is controlled via the “Inflate” properties. There are four such properties: InflateBottom, InflateLeft, InflateRight, and InflateTop. The gray area becomes active only when the associated control receives focus (which dictates that the user starts inking inside the visible boundary).

In delayed recognition mode, the gray area is not active; ink can then be drawn only inside the visible boundary.

It’s no secret that present handwriting recognition engines, even with accuracy levels as high as 95 percent, are far from perfect. Pencentric applications should be designed to minimize text entry and to utilize contextual information during the recognition process. One popular method for enhancing recognition accuracy is to constrain the character set for a particular field. For example, there is no point in allowing alphabetic characters in a social security number field. By constraining the allowed character set in the social-security number field to numerical only, we avoid misrecognizing a “Z” for a “2” and “O” for a “0” (zero).

In Visual Basic, you can use the CharSet property, which accepts the settings: ALC_DEFAULT, ALC_LCALPHA, ALC_UCALPHA, ALC_NUMERIC, ALC_PUNC, ALC_MATH, ALC_MONETARY, ALC_OTHER, and ALC_WHITE. These settings can be ORed together and are controlled either at design time via a custom dialog, or at run time.

Ink is one of the most exciting features and unique to pen applications. The ability to capture, store, manipulate, and display ink opens up new application categories as well as adding a new dimension to existing “text-centric” applications. Visual Basic supports inking via the DelayedRecog property. When set to True, all writing in the control remains as ink until the property is set to False. When it is changed from True to False, the OnTap property is examined. If OnTap is True, recognition of the collected ink will take place when the user taps on the control with the pen. The pen custom controls incorporate two new events: the RcResult and the Update. The RcResult event occurs whenever the control receives recognition results from the recognizer. The returned result is a pointer to the RcResult structure (Recognition Context) which forms a central part of the Windows for Pens API. The RcResult structure contains a symbol graph that describes the possible results from the ink, a handle to the ink, and the best guess at the symbols. If DelayedRecog is True, the RcResult structure does not contain any information about recognized symbols. The Update event occurs before the control redraws the data. This differs from the Change event which redraws the data before it occurs. Update can be used to format data so that flashes do not appear.

Aside from allowing quick erasing of the ink via the EraseInk method, Visual Basic does not provide sophisticated ink handling methods. To add advanced functions such as ink selection, re-sizing, and editing you must access the internal ink data structures and the underlying Windows for Pens API. You can do this with Visual Basic’s facility for declaring external DLL calls and by using the handles to the ink and window structures provided by the hInk and hWnd methods. I’ll discuss these later.

An Example Application

PenCheckbook is a program that started out as a regular Visual Basic application. I converted it into a pen application by replacing the standard controls with pen custom controls. The background consists of a bitmap of a check. This bitmap was first drawn with Paintbrush, and then cut-and-pasted into a picture field in the main program form. In the non-pen version of the program, the main form includes five Edit Controls to handle user input. These controls are: PayToCtrl, AmountCtrl, AmountStrCtrl, RemarksCtrl, and SignatureCtrl. There are three other fields which are display-only. These are DateLabel, CheckNumLabel, and CurrencyLabel which display, respectively, the current date, a revolving check number, and the relevant currency. These fields are controlled by the application but do not accept user input.

The application includes a global data structure that holds information about the last check. In the conversion process, I modified this structure to hold ink information and to be able to store and retrieve this information from a file. However, I’ve not yet implemented file storage and retrieval. For now, the items in the File and Edit menus (Save As, Open, Undo, and Clear) are merely placeholders.

The Options menu includes the functions Balance, Hide, and Show Last. The Balance function loads a second form which displays the current balance and enables the user to modify it. The current balance is automatically updated when a check is signed by the user. If the user tries to write an amount that exceeds the current balance, the application warns the user and cancels the check. The menu bar can be removed via the Hide function. Tapping on the form brings back the menu bar. This can be useful in pen applications which can be controlled via gestures rather than menu functions. It supports a less cluttered user interface while retaining the option to display additional controls for training purposes or for advanced functionality. Lastly, the Show Last function displays the last check which was signed by the user.

Converting Checkbook to Use Pens

In modifying Checkbook to accept pen input, the standard edit controls were replaced with pen custom controls. Consequently, the PayToCtrl, AmountStrCtrl, and RemarksCtrl fields can then process both handwriting and gestures. The original AmountCtrl field was replaced by a boxed edit field which recognizes numbers and gestures only. The BalanceCtrl field in the Balance form will be converted to a BEdit on the fly (when the user presses the Change button). This whole process requires adding less than ten lines of code. The new SignatureCtrl field accepts the user’s signature and retains it as ink.

Replacing the standard controls with pen controls is a slightly tedious task, because there is no automated way to do it. You have to delete the old control, drag the new HEdit or BEdit control and redefine various properties such as BorderStyle, BackColor, and CtlName. During this process you may notice a number of new properties. The CharSet property enables you to define the characters that will be accepted by the input field; restricting the character-set substantially improves recognition.

When adding the SignatureCtrl field, I set the DelayedRecog property to True. This results in an ink field which captures stylus input and which can later recognize ink if the user taps on the field and the OnTap property is set to True. The program uses the signature to “lock” a certain check so that users cannot change the amount on the check once it has been signed.

Changing the AmountCtrl and BalanceCtrl fields (located on the second form) requires more changes. The single edit control is replaced by two BEdit controls, for Dollars and Cents. You will need to adjust the Cell and Comb properties to achieve the best space utilization without compromising usability. The names of the new controls now become CentCtrl, DollarBalanceCtrl, and CentBalanceCtrl. I used the CharSet property to restrict input to numerical values (and gestures).

At this stage in its evolution, the Checkbook program is ready for pen input. The user can write and sign a check on the screen similar to using a paper checkbook. The user can invoke standard gestures to edit and manipulate the various fields.

After using the application for a while, you’ll realize that certain layout changes make life easier. As mentioned before, handwriting is typically larger than printed text. The controls which were built for keyboard input may be too small and result in loss of ink when the pen exits the border of the control. By using the Inflate properties of HEdit and BEdit, you can adjust the size and location of the fields to fit the new method of input. The Inflate properties don’t change the visible borders of the controls; they only affect the “gray area” which surrounds them. They thus enable the user to write beyond the visible borders without losing ink.

Accessing the Underlying API

Visual Basic provides several hooks that enable developers to tap into the full functionality supported by the Windows for Pens API. These include the RcResult and Change events and the hInk and hWnd handles. The RcResult event occurs when a BEdit or HEdit control attempts to recognize the ink. This event can be used as a trigger to create locking form fields, in form-based applications where a signature validates the data in the form. Once the user stops inking in the signature field, your program can lock all the input fields in the form so no changes can be made. To lock a field, merely set the Enabled property to False. In PenCheckBook, once the user has signed the check, the only way to unlock the input fields is by pressing the Pay button or selecting Clear in the Edit menu. In both cases the input fields will be initialized.

Since the RcResult event occurs before the recognition results affect the display, it can be used to trap custom gestures. Listing One (available from Dr. Dobb’s Journal) contains declarations from the global declarations file of Pen CheckBook. This file is basically the Visual Basic version of PENWIN.H. These declarations are used in processing the RcResult event.

The ProcessGestures subroutine (see Listing Three in Dr. Dobb’s Journal) does the same thing as the “Hide” function in the Options menu, except that it is invoked with a gesture. If the user draws the “Circle H” gesture in the edit control, this routine will hide the menubar. Process_cGestures() is defined in the global module and is called by each edit control when the RcResult event occurs.

As previously mentioned, Checkbook lets you store and display the last check that was paid. When SignatureCtrl becomes an ink field, the CheckRec global structure needs to be modified to hold a pointer to the ink structure rather than the signature text. Listing Three (available from Dr. Dobb’s Journal) contains declarations and code samples that show how using the hInk, hWnd, and the Windows for Pens API, the ink is stored in memory and retrieved later on.

By combining Visual Basic with pen custom controls, you get a tool that lets you create simple pen applications quickly, and at the same time lets you implement truly pencentric programs via the underlying Windows for Pens API.