Samsung’s KitKat update seems to remove benchmark-boosting “shenanigans”

Behavior in Android 4.3

AnandTech published a table demonstrating when some common phones boosted some common benchmarks, and we ran all of those known-boosting tests on both an AT&T Galaxy S4 and a Sprint Galaxy Note 3 under Android 4.3 and 4.4 to see how the CPU behaved.

Pay attention to the CPU monitor widget in the corner of each screenshot. Normal idle behavior will ramp the cores down to a low clock speed and even turn them off to save power. Consistently, though, opening up Geekbench versions 2 and 3, AnTuTu, Vellamo, Basemark X, AndEBench, or 3DMark will ramp all four cores up to the maximum 2.3GHz clock rate and keep them there until you leave the app.

The same was true of the Galaxy S4 after its update—all four cores go right to the maximum speed of 1.9GHz when a benchmark app is opened. According to AnandTech's data, most of the Android OEMs game one or two benchmarks to make their numbers look better, but none are as thorough as Samsung is in its version of Android 4.3.

Boost no more: Behavior in Android 4.4

Moving to Android 4.4, that strange CPU activity stops happening. Every single benchmark we ran prompted variable-but-normal fluctuations in CPU speed based on actual activity. Speeds would ramp up for a couple of seconds while the app launched, but once it became idle, the speeds settled back down to where they would normally be.

After applying the KitKat update, none of the apps behave any differently from any other application. Most of the time, the CPU cores are running at lower frequencies, and individual cores are often turned off. While running the tests, the CPUs approach their maximum clock speeds but are allowed to fluctuate as they would under actual use rather than staying artificially inflated.

To confirm these findings, Primate Labs' John Poole was able to pull data from the Geekbench Results browser to see how the scores varied from update to update. Poole was able to confirm both that a benchmark boost was added to many of Samsung's devices beginning with the Android 4.3 update and that the boosting does not seem to be present in the Android 4.4 update. The team has also added some detection code in a recent Geekbench update that would note when boosted scores were uploaded to that database—Samsung's Android 4.3 software sets the detector off, while 4.4 does not.

"To confirm that the boost had an impact on Geekbench scores, I applied Welch's t-test to the scores for each device that the detector highlighted," Poole told Ars. "For each device on the list, I found that scores under Android 4.3 are significantly greater than scores under Android 4.2.2 (p < 0.002).

"Two of the devices on the list (the Samsung Note 3 and the Samsung Galaxy S 4) have a significant number of Android 4.4.2 results uploaded to the Browser. The detector did not detect a boost under 4.4.2, which leads me to believe that Samsung has disabled the benchmark boost in Android 4.4.2. I was concerned that the boost became more sophisticated and was able to evade the detector, but scores under Android 4.4.2 are significantly less than scores under Android 4.3 (p < 0.002)."

Poole was able to give us the mean Geekbench 3 scores from various Galaxy S4 phones running Android 4.2.2, 4.3, and 4.4.2 to demonstrate how they jumped up in 4.3 and fell back down in 4.4.

Android version	Single-core score	Multi-core score
4.2.2	660	1812
4.3	682	2114
4.4.2	674	1913

"There's a statistically significant difference between 4.2.2 and 4.4.2, but the absolute differences are much smaller," Poole told Ars. "I'm inclined to think that the change between 4.2.2 and 4.4.2 is due to Samsung tweaking the overall power and performance settings rather than a new harder-to-detect boost."

The takeaway

Does benchmark boosting really change anything about what it's like to actually hold and use a phone? No, not really. Is it a disingenuous marketing tactic that misrepresents a phone or tablet's true performance under certain conditions? Does it mislead reviewers attempting to use common performance testing tools and confuse the spec-obsessed, performance-hungry consumers who use those numbers to decide how they'll spend their money? Yes, we'd say so. However minor an effect the boosting might have had on Samsung's sales or its reputation one way or the other, we're glad to see the company reverse course.

The real reason to install these updates will be the new features they bring, though. Neither phone gets any of the redesigned TouchWiz user interface elements shown off on the Galaxy S5 last week, but both still pick up all the under-the-hood enhancements of KitKat as well as a number of other Samsung-specific tweaks. These are just the first Samsung devices to pick up their Android 4.4 update—the company earlier this month released a full list of 14 devices that would get KitKat eventually, including older handsets like the Galaxy S III and Galaxy Note 2.

We'll keep a close eye on those updates that come out to see if Samsung's benchmark boosting rears its head again, and to see how it affects devices from the other Android OEMs (Poole suggested, for example, that some Sony Xperia phones and tablets that hadn't been boosting previously were beginning to boost in a recent update). For now Samsung seems to be backing away from the practice, and that's a good example for Google's largest Android partner to set, however belatedly.