Printer and Multifunction Device (MFP) Testing Methods

CopierGuide looks at hundreds of products a year, so you may be curious as to our methodology when it comes to testing. While we write quite a bit about how printers, copiers and MFPs work and what we think about ergonomics and their usability, there is a lot of objective testing that goes on on our labs on a weekly basis. In fact, there is a fixed set of tests we conduct on each product we review that lets us set a baseline for our review ratings and our overall scoring of each product. These tests are repeatable and help us document exactly how the real-world performance of each machine stacks up against manufacturer claims.

Following is a summary explanation of the benchmark testing we perform on each applicable machine and how we determine our ratings for each. Combined with our subjective impressions, this is the “special sauce” that we use for each review—with a dash of personality, of course! The other advantage this gives us is that we’re able to keep an ongoing record of each printer and how it performs against its similarly-priced peers.

All of this is designed to help you shop well and make it easier to know which products will meet your needs and get the job done to your specifications. We all know time is money…and money is money…so we hope that making it easier to understand what you get for your hard-earned dollar will help you make better purchasing choices in the digital imaging market.

Speed Tests

Test 1: Time to First Print (Warm-up Time)

The time to first print is captured from Standby/Power Saving mode. We power cycle the printer and let it settle in until it is in Standby/Power Saving mode and then we initiate a print job using a 5-page text-only document. This gives us a fairly accurate sense of how long the printer takes to get a piece of paper out when it hasn’t been in an active state. We start the timer when the job is completely sent to the printer and we stop the timer once the first page is ejected into the output tray. This test can be performed on all printing devices, both black and white as well as color (though the test is always done in mono mode).

Test 2: Time to First Copy/Scan

Similar to the test above, we begin from the Standby/Power Saving mode and then time how long it takes to get the first page of a 5-page document into the output tray from both the glass platen and (if present) the automatic document feeder. This test is obviously only run on machines which have a copy or scanning function. Any print-only products will simply bypass this test. Scoring is based on absolute speed and compared to other products available within a similar price range.

Test 3: Text-only Print Speed

This fixed test takes a defined 20-page PDF letter-sized document and tests the printer or MFP mono print speed. This is a very basic document with no graphics and very little font formatting. It tells us how fast the printer can print a page with approximately 10% toner coverage per page. This type of printing represents the typical office document printing that takes place every day and gives us a baseline for black and white text-based speed. There are no graphics and no colors on this document. We print from a standard PDF file and use the Standard quality mode for the printer or MFP with any print content settings set to Default. In terms of process, we wake up the machine and start the clock at the point when the job is actually sent. In this way we can separate the warm-up time from the actual print speed. We also clock the time it takes for the first page to exit the printer so that we understand the relationship between the first page out and the rapid succession of printing once the printer gets up to speed. Each test is done twice and results are compared. If the results are dissimilar, the test will be repeated a third time, and we’ll evaluate to make sure there aren’t any errant processes in the testing methodology or printer functionality. If there is any sort of resolution enhancement printing technology, we’ll typically leave that on unless it dramatically affects print speed (most do not).

One last thing we’ll do is compare print times in both Fine (Highest resolution) and Draft modes in order to get a baseline for what you stand to gain or lose (in time) when using those modes.

Test 4: Mixed Text-and-Graphics Speed

This test is designed to produce time and quality results for a predefined 10-page PDF letter-sized document that incorporates both text and full color charts and graphs. The test simulates a common business document output to either a color or black and white printing device. Monochrome printers and MFPs are fed the same color source document even though they are incapable of reproducing the original color output. It lets us know how fast the printer can print a mixed color document with approximately 10-12% toner coverage per page. Since we print the document from a high-resolution PDF file, we’re able to maintain consistency throughout all products tested. We also use the Standard quality mode for the printer or MFP with any print content settings set to either Default or Graphics mode. In order to get an accurate test, we ensure the machine is out of Sleep mode and only start the clock at the point when the job is actually sent to the printer. Like our Text-only Speed test, we clock the time it takes for the first page to exit the printer as well as the entire 10-page file to document the relationship between the first page output and the printer’s “cruising speed”. We perform each test two times and compare and average the results. Any anomalies result in a third test. Again, if there is any sort of resolution enhancement printing technology, we’ll typically leave that on unless it dramatically affects print speed.

Rounding out the tests, we’ll compare print speeds using the Fine (Highest resolution) and Draft modes so that all three potential speeds are evaluated, and users can observe whether the time savings or quality gains are significant enough to warrant their use.

Test 5: Photo-Printing Speed

Since we’re primarily dealing with office printing machines, the photo testing is solely an attempt to garner the color reproduction speed of the device when set to Photo mode and when handling a photo-rich document. We have found that the best photos will come from either dye-sublimation printers or certain photo-catering inkjets with the appropriate paper. Still, getting a clear and well-represented photo output from a color laser printer is desirable, and this test will let us know exactly how quickly the printer handles color photography from a source file while also providing another evaluative source for our quality testing.

For this test, we use a 3-page PDF document with two pages having a 4×6 inset photo image surrounded by text and the last sheet having a large 6×9 image with some descriptive text above (landscape orientation). These images have exactly 600-dpi of resolution (unscaled) and are embedded within the press-ready source PDF. As this is a speed test, we have separate print documents to measure color and detail quality, but we also format these files so that they can be considered as well in our objective evaluation process. The test takes into account starting with a warmed-up printing device. We then print to 52 lb glossy laser photo paper using the “Normal” mode from the PDF file and begin the clock as soon as the file is sent. When the first sheet hits the output tray, we mark the time and also mark the time it takes to print all three pages. This gives us a good sense of how quickly a one-page document will print as well as a repetitive time for multi-page documents. At the end of the test, we re-run it using, if applicable, the “Fine”, Maximum Resolution”, or “Photo” setting for the printer to get a sense of the maximum quality potential for the machine’s color photo output.

Test 6: Scanning and/or Copying Speed from ADF

The intent of this test is to see how quickly a copier or scanner can process and either store or print/output 20 documents. To accomplish this we take a stack of 10 two-sided pages of mixed text and text/graphics documents and set them in the automatic document feeder input tray. We then make sure the machine is in full duplex mode (if it has this) and, if applicable, it utilizes its automatic reversing document feeder. If it lacks the ARDF, we utilize 20 pages of the single-sided version of the same document for single-sided scanning/copying.

For a scanner, we’re seeing how fast the 20 pages of input are read to the device or attached computer at 300 dpi in both Line Art and Photo quality modes (or similar). For a copier, we’re seeing how quickly the unit generates 20 pages of output (either duplex two-sided documents, if available, or as single-sided documents if not.) The copier or scanner is utilized for a single scan or copy to wake it up, and the timer begins exactly when the Start button is pressed to initiate the 20 copies or scans. Since there are so many different models and options, the tests we run can be broken down into the following:

• 20 page duplex copying with black & white text/graphics
• 20 page duplex copying with color text/graphics
• 20 page duplex scanning at 300 dpi with black and white text/graphics
• Single page duplex scanning at 600 dpi with color photography

The idea is to use and measure the device in its various and typical functions so that we can rate it against its peers and determine, objectively, how well it performs in the marketplace.

Quality Tests

For testing quality of various printers, multifunction devices and scanners, we have developed our own internal testing methodologies which are closely tied to principles found in the ISO/IEC 15775:1999(E) International Standard.

Test 7: Line Detail and Resolution

Checking for line detail and resolution is an important part of the printer and copier testing process. For a printer, that means outputting a high-resolution PDF that has specific testing patterns (ours are based on the ISO/IEC 15775 International Standard). We use these patterns to grade and scale each printing device against other products in the market and against our reference documents. A series of “infinite” resolution vector-based lines and line raster scales that go from 15 lpi (lines per inch) down to 240 lpi let us evaluate the practical mono print quality and resolution of the printer given files with highly detailed structure.

We have offset-quality reference files and prints for all testing documents and so, in addition to printing from high-resolution files, we can compare the printer output with these masters. We can also use our offset masters as source files for testing copier and scanner resolution.

Test 8: Color Detail and Resolution

In order to objectively test color detail and contrast, as well as any color shifts or lack of detail, we needed another reference offset print that we could use to compare and contrast the capabilities of each color printing product. Combined with our objective resolution and detail testing, we’re able to truly derive final results that are more than just subjective opinions subject to the taste and biases of the reviewer. Here is an example of just one of our color test files which is also based on the ISO/IEC 15775 International Standard:

What we’re looking for here is color accuracy, detail, contrast and resolution.

Ergonomics and Use Testing

While it may be apparent that print quality is high up on our list of desired characteristics for an effective document imaging solution, ergonomics do play a factor. These qualities are harder to measure objectively, but there are some ways in which we attempt to level the playing field across manufacturers by performing similar tasks across all products. These include, among other things (as applicable):

• Document reduction and scaling
• Single-sided and duplex printing
• Single-sided and duplex scanning
• Print to file
• Print to cloud
• Test finishing options (if applicable)
• Paper loading
• Toner replacement

These testing methodologies help us to accurately test printers, copiers, scanners and MFPs accurately and with  repeatable results. Our desire is to help businesses make informed choices and discover the underlying strengths (and weaknesses) of each product we test. In the end, the goal is to aid and serve the end-user.