Speaking With VeriTest’s Nemzer & Abrams

Every now and then, we exchange an interview with our friends at Mobile Insights, Inc., publisher of the highly-respected Mobile Letter. Since they were last to borrow an interview from us, it was our turn to grab one that would interest our readers. I think you’ll find that this interview, conducted by Gerry Purdy, introduces an important and often behind-the-scenes activity that will play an increasingly important role in shaping the mobile industry.

[Editor’s note: We’ve adopted the question and answer style in which this interview was originally published in November, 1996.]

Suppose you are a manufacturer of a mobile computing hardware or software product. Before you put your product out in the market, you want to make sure that it operates according to the specifications and documentation. So you wrap up your product and send it out to VeriTest Laboratories in Santa Monica, California, whose sixty-plus engineers put your product through the paces and let you know whether you’ve passed the test. If you don’t, they tell you specifically where the problems lie or what you must do in order to comply with either the specification or certification.

Sure, you could have your own staff do the same testing and get the same performance statistics, but that wouldn’t be nearly as objective as someone who tests products for most manufacturers.

VeriTest was started by Steve Nemzer and Marc Abrams and has grown to test hardware and software products with a particular emphasis on portable computer systems.

We thought it would be interesting to understand how this important but relatively unknown company operates. After reading this interview, you’ll probably have a far greater appreciation for the performance and compatibility of your portable product.

“Our clients include most of the major vendors such as Toshiba, NEC, IBM, Sharp, etc.”

GP. Steve, why don’t you summarize for us the history of VeriTest. How did the company get started?

SN. We’re in our ninth year of operation having started back in 1987. Prior to 1987, I worked at Ingram Micro (a major distributor) testing products before they went to dealers. I also tested new products before we decided to carry them. One of the first clients of VeriTest was Intel. I should also mention that we did testing on early mobile products such as Osborne and Kaypro. Those were mobile products that we’d call transportable, they were certainly not mobile in terms of what we think of mobile today.

GP. Describe the size of the business today; how many employees do you have, and tell us a little bit about your customers.

SN. Currently, we’re at sixty-plus employees. The vast majority are engineers and technical staff. Our clients include most of the major vendors such as Toshiba, NEC, IBM, Sharp, etc. In terms of the mobile products, we’re forming categories of services. The bulk of our business is in compatibility and operability testing. We also do functionality testing and acceptance testing since a lot of software comes with these products. We do extensive testing of the power management software and other unique features of mobile products.

A big area also is our standards testing. We test compliance with industry-wide standards such as Windows 95. We’re very involved with Windows BackOffice and, most recently, Windows CE testing. We also offer some usability, user interface, and performance-based testing.

GP. Would you explain a little bit about what compatibility testing is? SN. This goes by a number of different terms: interoperability testing, compatibility testing, configuration testing. In terms of mobile products, this might involve testing a new notebook system with everything from PC Cards, including CardBus, video, infrared devices, EPP and USB devices, operating systems, applications, power management, and battery life.

We also test video drivers and new LCD screens. Notebooks in particular are introducing new video components that need to be tested. And finally, there are a lot of wireless devices and software which need testing.

GP. What’s your business model?

MA. We operate on a project fee basis, get test specifications or test objectives from a customer, and based on that produce a test plan which will include cost, schedule, equipment requirements and so forth.

GP. What is the hardware/software split?

“Our growth model is to develop more labs that specialize in a given area, such as our new Windows CE lab.”

MA. We do 70% hardware and 30% software. We are a collection of specialty labs. The labs will become increasingly independent and specialized. They share resources, but our growth model is to develop more labs that specialize in a given area, such as our new Windows CE lab.

GP. How are tests managed and how do you report these results back to the client?

MA. We have a very formal test management procedure. The client works with a project manager who writes the test plan and oversees the project throughout. We don’t have dedicated sales people.

The person with whom you talk is the person managing the project from beginning to end. That’s one of the secrets of our success. The project manager oversees three or four projects simultaneously. Reporting to the project manager is the test team leader. The test team leader is the hands-on test lead who organizes the data, scheduling the test dates, and so on.

The clients sometimes speak directly with the test team leader. The test team leader writes twice-weekly progress reports which are edited by the project manager before they go to the client. The test leader manages the test engineers who are divided between day and night shifts.

SN. Working with an external test lab such as VeriTest is a little different than working with an internal testing resource. One of the ways we compensate for being an external facility is with our formalized reporting and communications.

All the emails and faxes that go between the client and VeriTest are collected and summarized twice a week. We really have to go to the extra effort in order to make the external relationship work. In many cases, over a period of years, VeriTest has become an extension of a company’s internal test resources.

MA. Software testing has a very different reporting structure than hardware testing. Software testing often has daily bug reports that are going out, so it’s a much more rapid and intimate relationship. Hardware tests are usually reported twice a week.

GP. Do you do much benchmark testing?

MA. We do benchmark testing. However, we do less now than we used to. That’s on purpose. Because of our affiliation with the Windows 95 logo program, we don’t do benchmark testing for publications. We will do work that’s for internal use only. Hardware benchmarking remains part of our business.

For example, people are particularly interested in network server benchmarking. But, it can always be argued that no matter how carefully benchmarking is done, people will disagree with the results.

GP. What other independent labs are out there, and how do you compare yourself with the magazine-owned labs like Ziff-Davis? MA. There are few labs of our size operating as independent entities. NSTL is perhaps the best known; they’re in Philadelphia. In terms of being an independent lab, we’re really a unique lab. We’re not related with a publication. We’re unique in that we’re an independent lab that does both hardware and software testing.

GP. In the area of hardware, how would the testing methods at ZD Labs be similar or different from what you do?

MA. It’s quite different. They’re very focused on benchmarking; that’s what they do best. We are often working with prototypes which are not ready for prime time. We are the place where people send their systems before they send them to ZD Labs.

SN. One way we work with magazines and publications is to monitor what kind of tests they are doing. So, if someone submits a notebook or mobile product to VeriTest, we want to make sure that when this product does get sent in for review, it’s going to go through with flying colors. We’re watching what ZD Labs is doing, and making sure we’re doing the same kind of testing.

GP. Some of the benchmarking that you do has shown up in Mobile Office (now called Mobile Computing) magazine.

MA. We do contract testing and we only do benchmarking for them. That is very restricted to running two tests. We use the industry standard BAPCO test. We have a lot of respect for this benchmark test and we do a battery life test which is accepted as the best industry standard battery life benchmark for notebook computers. That’s all we do for the magazine, and it’s a very hands-off relationship.

SN. It’s an arm’s length relationship with the magazine because of nondisclosure agreements with many vendors.

GP. Talk a little bit about the Mobile Computing lab and the Windows CE lab.

“There’s a good chance that the Win CE logo will be extremely meaningful for end-users.”

SN. We talked a little bit about some of the unique issues that come up with mobile. There’s PC Card testing where interoperability has been a big issue and a big headache for many years.

It’s starting to resolve itself somewhat, and we like to think we’ve had a hand in that. There are other technologies that go into mobile that differentiate it from the high-end server products. So when we built our Mobile Lab, we specifically developed an inventory of products to test against.

We have racks and racks of PC Cards and devices, the latest video and hardware, the controller chips and core logic chipsets, and so forth.

We’ve been working on mobile products from the beginning and it’s something that’s a very special emphasis for us.

MA. We concentrate on mobile products since we have people who are specifically oriented towards mobile testing, who understand that it’s different, who understand the issues, know all the versions of the cards, and so on.

On the Windows CE side, this is an opportunity for a new technology to be developed along standards from the beginning. The notebook market developed on many fronts all at the same time.

For example there has never been a serious interoperability standard for PC Cards. But in the Windows CE initiative by Microsoft, there’s an opportunity for all the hardware people and all the software people to get together and agree on the standards which we will then test against.

There’s a good chance that the Win CE logo will be extremely meaningful for end-users. There really will be plug and play of the software and hardware because of the standards being enforced by Microsoft.

So, we’re very excited to work on that program, and we intend to be the place where all that equipment comes together: all the hardware and software in one place available for compatibility testing.

MA. We have been named as the only authorized software logo test center for Windows CE. It’s interesting there that there are actually two kinds of applications to test.

There are applications which will exist only on the Windows CE platform, and there are Windows CE companion applications which will operate on the desktop. And that’s actually the more important category.

Many of the most common applications will have a Windows CE interface which will go out and talk to the handheld.

“We have been named as the only authorized software logo test center for Windows CE.”

GP. Let’s talk about your battery test sublab. I noticed that it’s actually in a separate physical area in your labs in the back. What is it that you do in that lab and how do you go about testing for battery life?

MA. We’ve been involved in battery life testing of portables since the Intel 386 SL was released. When the SL was released, it became possible to do real power management at the CPU level on notebooks. The question that vendors were asking was, “what benefits do you really get from power management”?

We developed a standard, along with several industry leaders, which initially was called the Olympiad Suite, and is now called the Cross Country test. It involves a machine called “Keyboard Annie” which does live typing on notebook computers.

It was and remains the only test which uses an external typing device which, in turn, exercises all the power management features on the device. We do not disable any power management when we test battery life, hence we’re getting a much better real world test.

That test has been endorsed by a lot of different vendors including Dell Computer which uses our results in their ads. As of today, the winner in the battery race is the Acer notebook which has the best battery life (the Acer Note Nuovo).

GP. Do they get longer battery life because they have a higher watt-hour battery or because they’re really great at the way they manage power on the same watt-hour battery?

MA. They have a large battery, but it’s not beyond the usual size in terms of weight and watt-hours. They are doing something remarkable in power management, and it’s actually probably the direction many vendors will be taking regarding power management.

They use what’s called heuristic power management. It basically means that there’s a learning process going on in the computer where it’s learning the way you work and shutting down things you don’t need.

GP. And what happens in the Cross Country tests. Are various BAPCO application suites tested?

SN. What we realized over time was that it was very important to have a simple straightforward test protocol that everyone could understand. We simply load Windows 95 and Office 95, and the program runs only in Excel and Word. We’re doing typing, we’re doing spreadsheet recalculations, and so on. We’re not running any macros at all on the computer.

The Keyboard Annie unit is doing everything by manual solenoid typing. So it’s completely external and is a very simple business usage specification. There is file save going on every ten minutes because we learned that the average user will do a file save every ten minutes.

Perhaps most important in comparing that benchmark with other industry benchmarks is that we have a complete specification for how the notebook itself is set up in terms of RAM, display, and disk time-outs. Also, we have a setting for the brightness of the display.

Every other benchmark out there has no parameters for that, or they’re doing crazy things like turning the brightness all the way up on the screen. Their specification is not real world. So we have what we consider to be a real world situation. Anyone can argue with what is a typical real world situation, but we’ve drawn a line in the sand and said okay, this is the definition that everyone can live with. We’re very proud of the Cross Country standard.

GP. How do you retain confidentiality if you’re testing new systems from two different vendors who want to each have someone come in and observe their tests?

“Maintaining confidentiality is something that is core to the VeriTest structure.”

SN. Maintaining confidentiality is something that is core to the VeriTest structure and we work with a lot of companies that do compete with one another. For different vendors who are working on site, we maintain separate dedicated areas.

We’ve had engineers camped out in their offices for weeks at a time as they work to debug products. In many cases it’s more convenient to do debugging at our site because we have the infrastructure to test just about any hardware feature. For projects that are done with the client off site and not on our premises, we still maintain the same confidentiality restrictions.

Everyone is under nondisclosure down to the janitor. We’ve never had a breach of security.

GP. While a company such as Phoenix Technology helps mobile system manufacturers with the hardware design, you test the product according to how it is supposed to work and verify the performance of the specification. True?

SN. I think that’s a pretty fair analysis. Phoenix Technology develops a product that is incorporated into a notebook. We’re a service company, so once the notebook has been assembled into an alpha prototype stage (with components from Phoenix and Cirrus Logic) VeriTest can get involved to validate the system design in terms of compatibility and performance.

MA. People come to us when they have already built five or six prototypes and they’re expecting to build another five or six before final production.

GP. And will you give feedback that will help them potentially define what needs to happen in their last few tweaks, or is it just that you give them performance and compatibility data that goes along with the product?

MA. No, we’re definitely part of their final tweaking process. They are making modifications based on our data as they go down to the wire. Often we’re in a three way conversation between the notebook maker, the BIOS developer and us. We’re going over bugs and we’re getting them resolved. So we’re part of getting those last bugs out.

SN. That’s very important, because so much of development itself is outsourced. The vendor selling a notebook might be a completely different entity than the vendor actually building it. Sometimes we conduct conversations that involve five or six parties. VeriTest acts as an ombudsman.

We put all the anomalies on the same page and distribute these reports via fax or e-mail to all the interested parties. Often when they all decide to meet, they end up at the VeriTest conference room.

GP. Talk about this logo program that you’ve got going, both Windows 95 and now Windows CE. What does it mean for a product to be certified?

MA. I think it’s important for people to understand what the logos mean for Microsoft and what they don’t mean. We are testing these products against certain criteria to make sure they are good citizens in the Windows environment.

We ensure they are supporting the user interface, that they are providing an un-install capability, etc. Someone has compared it to the kosher symbol on foods. It doesn’t mean it’s good tasting, it means it’s kosher. The Windows 95 logo on a piece of software means that the product is going to be a good citizen on Windows.

It’s going to support all the things you would expect in Windows. It doesn’t mean it’s a good product, nor is it an endorsement by Microsoft.

SN. I think it’s important to point out that in the early days of Windows, there was only a self-certification process. I think Microsoft unfairly picked up bad press about the stability of Windows 3.0 because a lot of the third party applications were not good citizens of the Windows API.

So it appeared that Windows was crashing a lot. We believe that if you’re going to carry a Windows logo you need to conform to some basic Windows standards. We certify that.

MA. We test them for stability and stress background testing, as well as the logo criteria. We do four logo programs now for Microsoft:

• Designed for Windows NT and Windows 95
• Designed for Windows 95;
• Microsoft Office (so third party applications function similar to Office applications), and
• the new Windows CE logo

We’re working with Microsoft on a Visual Basic application logo program.

GP. Let’s talk for a minute about industry standards. Do you guys create the standards? Or do you guys simply enforce the standards, or is it a little bit of both?

SN. It’s fairly complex. There are certain market-driven standards that just evolved based on marketplace considerations. It all started with the IBM PC-compatible standard, driven just by the force of IBM.

Now there are standards which are introduced by consortium and associations. In general, VeriTest is very involved with all of the industry associations in terms of setting standards such a PCMCIA and so forth. But we’re not an enforcer per se. When it comes to testing for standards, we consider ourselves more like facilitators.

We do the test and we provide a lot of consultation and hand-holding to get people through the process so they are compliant.

So, rather than enforcing a set of rules—we’re not policemen—we try to get their products to comply with accepted standards.

MA. We remind our testers all the time: you are not the software police. You are working for the vendors to help them to be compliant with the standards.

It’s the vendors who are paying us and the vendors for which we’re working to help them get those logos. A big reason for our success in this program is that attitude on the part of our people.

GP. Where do you go from here?

SN. I think it is very important that we continue to be the accepted independent test lab for hardware and software vendors. A big part of our sales and marketing job was convincing people that independent testing was a good idea at all.

Many of the companies who talked to us would say, “Well, I have a bigger lab than you have, what do I need you for?” Now they realize compatibility and standards compliance is far more important coming from us than from their own labs.

GP. What about the future in the mobile area?

SN. Generically speaking, we’re very excited about the recent big leap in PDA technology. That’s the most exciting thing on the horizon. Also exciting is the whole area of wireless access and where that is going.

In the mobile area in particular, the industry is beginning to grow up in terms of quality and consistent performance.

Steve Nemzer President [email protected]

Marc Abrams Executive Vice President [email protected]

VeriTest
3420 Ocean Park Blvd.
Suite 2030
Santa Monica, CA 90405
1-310-450-0062
1-310-399-1760 (fax)

www.veritest.com