NEW & IMPROVED! – MyGolfSpy’s “Ultimate Review System”

Time For An Upgrade

When we created the “ULTIMATE REVIEW SYSTEM!” a year and a half ago we started with a single goal in mind: create the most complete, unbiased, and most importantly, objective review system we possibly could. As you are no doubt aware, the overwhelming majority of golf equipment reviews found online and in print are almost entirely subjective, and contain little or no actual data. While we believe aspects of those types of reviews have value, we also believe the greatest value comes in being able to backup perceptions with actual data. As we’ve proven with past incarnations of our review process, perceptions (the stuff subjective reviews are made of) – especially those related to distance and accuracy – often bear little correlation to the reality proven by the numbers.

We view the original incarnation of the “ULTIMATE REVIEW SYSTEM” as a great start. Since the time that process launched we’ve learned a great deal about how golf clubs perform, how golfers perceive performance, and how to sum up the correlations and contradictions into an informative and hopefully interesting package. Almost from the beginning we’ve  made minor tweaks to the system. As we found things that didn’t work (SpecCheck for example) we took them out. When we found things we’re missing, we added them in. Over time those modifications have added complexity to our scoring equations and the process itself. We realized we’d reached a point where two of our core goals: transparency, and simplicity were falling by the wayside.

Those problems, along with our enhanced knowledge, and our strong desire to improve on the overall detail and quality of our reviews made it abundantly clear it was time to release version 2.0 of the “ULTIMATE REVIEW SYSTEM!”

Why We are Upgrading our “Ultimate Review System”

• We want more detailed testing, more data, and ultimately more information for you to consider when making your next equipment purchase.
• Over time the adjustments we made led to a system that was more complex than it needed to be.
• Mediocre (or even bad clubs) sometimes got good scores. We want to add more separation between the best and the worst.
• Very, very good clubs, because of  the subjective portion of our subjective scoring system, would sometimes get lower scores than we felt they deserved.
• We’ve learned a tremendous amount, received a ton of feedback, and generated new ideas. The updated review system incorporates all of that.

What has Changed?

Simply put, we’ve changed very little…and almost everything at the same time. We’ve completely revised our performance scoring system. We’ve modified our subjective surveys, and we’ve added additional testing points to provide an even more detailed picture of a golf clubs performance. Fundamentally, data remains at the core of how we test and what we believe in.

We’ve Added Consistency

In the past we’ve pulled from a pool of testers. This method allowed us to work around scheduling conflicts, and involve more golfers in the process. The downside is we lacked a degree of consistency from review to review. To provide a more accurate comparison of golf club performance, we’ve enlisted 6 full time testers for the duration of the 2012 equipment season. As they have in the past, our testers span a wide range of ability levels, and have wide range of swing speeds. I’m very exited to announce that for the first time (and due to popular demand) we will be including a senior (60-something) tester in every one of our club tests.

We’ve Simplified (and Improved) Our Subjective Surveys

When it comes to our subjective scoring the most dramatic changes are the elimination of “Perceived Distance” and “Perceived Accuracy” from our surveys. Too often we found that the perceptions of our testers bore very little correlation to reality. Whatever the reason for the discrepancies, we found they did more harm than good. The question we asked ourselves was, “why solicit options for things we can quantify with actual data?”. When we couldn’t come up with a good answer we decided to drop them from the surveys entirely in order to better focus on the aspects of a golf club that are more difficult, if not impossible to quantify with data.

As part of the subjective scoring update, Sound and Feel, which many perceive as closely related (or even exactly the same thing) have been grouped into a single category. We’ve also raised the relative value of “LOP” (Likelihood of Purchase) since we believe it provides the best indicator of whether or not our testers would actually consider purchasing a given club.

Here is the breakdown of the new subjective scoring model and what percentage of the total subjective score each accounts for:

• Looks:   30%
• Sound & Feel:   30%
• Forgiveness:   15%
• LOP:   25%

Subjective weighting accounts for 10% of the total score for drivers, fairway woods, hybrids, and irons. For wedges it accounts for 20% of the total score.

Significant Improvements to Performance Scoring

The most significant changes have been made to the Performance portion of our review system. The changes we’ve made are designed to simplify, add quality, and eliminate unexpected anomalies.

Driver Testing

The previous system leveraged a complex formula that related distance to swing speed. While it did a good job of showing which drivers were longer than others, mathematically speaking, things were a lot more complicated than they needed to be. We’ve felt scoring should be simple and based on the actual potential of each one of our testers. It took a while to sort out, but while watching TV one night, I figured it out.

Enter “Big Break” Scoring

If you’ve ever watched The Golf Channel’s Big Break you may have seen an elimination challenge that is basically a modified long drive competition. Essentially each competitor hits a drive and the winner is decided based on the highest point total after a simple formula of Distance minus Accuracy (D – A = Total Points) is applied. A 300 yard drive that’s 15 yards offline is worth 285 points. A 250 yard drive that’s 40 yards offline is only worth 210 points.

From a simplicity standpoint it’s an excellent system. We also think it’s a fantastic way to measure golf club performance. The one challenge was to find a simple way to translate our point values into our percentage (“out of 100) score. If in testing “Club A” I told you that one tester scored 2043 points, and another scored 2316 you’d have no actual idea of how that club performed. We needed to find a way to explain why 2043 points could translate to a score of 93 out of 100, while 2316 might only be an 87.

Enter MPV

To solve the problem we created what we’re calling MPV or “Maximum Point Value”. MPV represents the theoretical highest number of points a given tester might collect for each driver we test. While it may sound a little hokey, it’s actually very, very simple.

To determine MPV we went back and looked at every shot each of our testers hit since we started collecting data.  For each and every drive our testers have hit over the better part of the last 2 years we subtracted accuracy from distance. We then added up the best 10 drives they’ve hit for us (regardless of which club they hit them with). This new number is the MPV. Because every tester is unique, the MPV is different for all 6 of our testers. MPV values range from just over 2000 for our senior tester to just over 3000 for our lowest handicapper and biggest hitter.

The Driver Testing Process

During a testing session, each tester will hit 12 shots. The 10 best shots (based on Distance minus Accuracy) will be used to calculate individual point totals. The percentage of the MPV represented by that total will dictate each tester’s “out of 100” score.

The average “out of 100” score for the top 5 of 6 testers will be used to determine the final performance score.

No More Consistency Score

After much debate we have decided to eliminate Consistency from our performance scoring (for all club types). While it was a good effort at quantifying forgiveness (and much to my surprise lined up nicely with statistical standard deviations), it did not meet our simplicity requirement. We came to the conclusion that when you’re looking at distance and accuracy a picture of consistency and forgiveness tends to emerge on its own.

Fairway Wood Testing

Under our original review system fairway wood testing was absolutely identical to driver testing. While that allowed us to give you an idea which clubs were the longest, and straightest, we felt we could do a better job of representing how golfers actually use their fairway woods. To that end, we’ve broken Fairway Wood Performance testing into 2 separate tests.

Tee-based Testing

We know golfers will often use their fairway woods off the tee. To replicate this in our tests, golfers will now hit a series of shots with the ball teed up. Scoring for tee shots is identical to driver scoring (the goal is to hit it long and straight). MPV values for fairway woods were calculated based on previous 3-wood performance. Tee-based testing accounts for 40% of the total performance score.

For the first time we are incorporating “Radius-Based” testing (previously used for iron testing only) into our fairway wood and hybrid testing. Testers will hit a series of shots with a 5-wood to a pin set at an appropriate distance (distances will vary for each tester, but will be consistent for every fairway wood we test). Scores are calculated based on the radius or distance from the hole.

To provide better comparative numbers, low handicap golfers are expected to be closer to the pin than high or middle handicap golfers. The number of points we subtract (from 100) per foot from the hole is based on those expectations:

• High Handicap Golfer (16+): .208 points per foot
• Middle Handicap Golfer (8-15.9): .227 points per foot
• Low Handicap Golfer (0-7.9): .250 points per foot

Radius-based testing accounts for 60% of Fairway Wood Performance Scoring.

The average “out of 100” score for the top 5 of 6 testers for both tee and radius-based testing will be used to determine the actual fairway wood performance score.

Hybrid Testing

Hybrid testing and scoring is almost identical to Fairway Wood testing. MPV values are calculated based on previous hybrid performance.

Testing from the tee accounts for 30% of the total score. Point per foot values have been increased (compared to fairway woods) to reflect the greater accuracy expectations that come with swinging a shorter club.

• High Handicap Golfer (16+): .238 points per foot
• Middle Handicap Golfer (8-15.9): .260 points per foot
• Low Handicap Golfer (0-7.9): .286 points per foot

Radius-based testing accounts for 70% of Hybrid Performance Scoring.

The average “out of 100” score for the top 5 of 6 testers for both tee and radius-based testing will be used to determine the actual hybrid performance score.

Iron Testing

The most significant change to our testing protocols comes in the area of iron testing. Previously we encouraged our testers to hit different clubs from the set, however, scoring (data collection) was done at 150 yards only. With the emergence of combo sets, and hybrid-iron sets, it’s more important than ever that multiple clubs be tested in order to present the clearest picture of performance. If our testers throw darts with the short irons, but struggle more so than usual with long irons, that’s information we feel you should have.

While we will still encourage testers to hit multiple irons in the set, and even play a round or two with the clubs, we will begin collecting data (and scoring) at 3 different distances. While the actual distance will vary to account for the differing abilities of our golfers, testing and scoring will be determined through individual testing of long, middle, and short irons.

As with our other tests, golfers will hit 12 shots to each of 3 targets. The best 10 shots based on proximity to the hole (radius) will be used to determine the overall score. Just as with Fairway Woods and Hybrids, our expectation is that lower handicap golfers will achieve better results, and that is again reflected in our point per foot values:

Long Irons (25% of Performance Score)

• High Handicap Golfer (16+): .278 points per foot
• Middle Handicap Golfer 8-15.9): .303 points per foot
• Low Handicap Golfer (0-7.9): .333 points per foot

Middle Irons (35% of Performance Score)

• High Handicap Golfer (16+): .333 points per foot
• Middle Handicap Golfer (8-15.9): .364 points per foot
• Low Handicap Golfer (0-7.9): .400 points per foot

Short Irons (40% of Performance Score)

• High Handicap Golfer (16+): .417 points per foot
• Middle Handicap Golfer (8-15.9): .455 points per foot
• Low Handicap Golfer (0-7.9): .500 points per foot

The average “out of 100” score for the top 5 of 6 testers for both tee and radius-based testing will be used to determine the actual iron performance score.

Wedge Testing

Wedges have proven to be at once both the easiest and most difficult clubs to test. Arguably, with the exception of the flatstick, nowhere do intangibles (looks, feel, etc.)  impact the buying decision more. We considered having our testers hit shots from awkward distances but arrived at the conclusion that more so than with other clubs, we’d be testing the golfer not the equipment. While there are some things we’d love to do in the future (bunker and deep rough testing) without the capability to collect real data, the results would be little more than subjective, and that’s not what we’re about.

With that in mind, wedge testing remains largely unchanged. Testers will hit a series of shots to an appropriate target. For most the target distance will be 100 yards. For our big hitter, the target will be set at 125 yards. For our senior tester it will be set at 75.

75% of the wedge performance will be radius based. Point Per Foot deductions are:

• High Handicap Golfer (16+): .556 points per foot
• Middle Handicap Golfer (8-15.9): .606 points per foot
• Low Handicap Golfer (0-7.9): .667 points per foot

Previously we’ve relied on a hard number to determine wedge spin scores. Almost universally this created scores that were higher than they probably should have been, and showed very little distinction from one club to the next. To remedy this problem, scoring for spin totals will be based on MPV (as determined from our previous wedge tests). 25% of the total performance score will be determined by spin.

We Will Continue to do the Bulk of our Testing Indoors

As we have in the past, we will continue to conduct our performance tests at Tark’s Indoor Golf in Saratoga Springs, NY. While it’s in our plans to try and get the clubs we test some on course playing time (for subjective-only feedback), we will continue to collect the data we use to determine our scores at our indoor facility.  There are a number of reasons why testing indoors on aboutGolf Simulators is advantageous. These include:

• Reliable and Accurate Data – aboutGolf Simulators have proven accurate enough to be put into use by household names like Callaway, Titleist, and The Golf Channel. These big names rely on aboutGolf to provide accurate data, and we do too.
• Testing indoors guarantees consistent, controlled conditions – We never have to worry about wind, temperature or humidity giving one club an advantage over another.
• We can test year round – When the snow is falling outside, it’s sunny and 70 inside.

There are certainly some enhancements I’d love to add, but until such time as we can accurately and consistently collect data outside, these things will have to wait.

The first set or reviews performed under our new guidelines will start publishing sometime in the next few weeks. Until that time (and after), we welcome your feedback on version 2.0 of our ULTIMATE Review System.