"Kickoff the Jim Harbaugh era with an autographed Michigan Wolverines Harbaugh T-Shirt", said the pop-up ad on the eCommerce site I had just logged on to. It was a very personalized message designed to target a college football fan like me, on a site selling, among other things, college football merchandise at the beginning of the Big Ten football season.
It was perfect except for one thing – I went to The Ohio State University!
For the uninitiated, presenting a personalized message hawking Michigan Wolverines merchandise, like the one above, to an Ohio State Buckeye is like waving a red cape in front of a bull. It triggers responses that are fueled by one of the storied college football rivalries dating back over a hundred years. These are not pleasant responses and certainly not the kind that the person who designed this promotion was hoping for.
So what went wrong? It could be any one of a number of things from insufficient user profile data to incorrect or conflicting personalization rules.
One thing however is certain – the personalization implemented on the site was not adequately tested.
Personalization tools and platforms providing a myriad of ways to present relevant content and promotions to users on an eCommerce site have abounded in the past few years. They have at their disposal a variety of data – volunteered and inferred, about the user to build personalization rules and drive specific digital experiences. This same variety of data and complexity of rules makes the task of testing personalization that much more complex, and the likelihood that the testing is shortchanged that much more real.
It does not have to be so. Here are three things you can do to make sure that the personalization you build on your site works correctly:
Simulate all attributes that drive personalization in your testing: Your personalization rules depend on a lot of attributes (data) about your user. Make sure that your test data includes all combinations of these to verify the personalization rules. Pay particular attention to test data that drive negative tests – if the person testing the site mentioned above had used test data corresponding to a Buckeye in addition to a Wolverine, they would have caught the problem before it made it to the public site. Testing for large combinations of test data is easier and significantly more economical if you use test automation that supports data driven testing.
Test the integration of the personalization engine and the website:Unit testing the personalization rules are important, but do not skip the step of testing the website after the personalization has been integrated. Testing the web site from the user interface will show you exactly what the user experiences and will help you identify details that can be easily missed in unit testing.
Enable people designing personalization rules to perform the tests themselves: It is obvious that the person designing the personalization rules knows how they should work and is best equipped to come up with test scenarios to make sure that they work correctly. So why not equip them to run these tests themselves right after the personalization rules are built. They can validate the results and make sure that the personalization works just the way they intend it to. By moving this testing process closer to the experts and testing often, you can reduce the time and cost required to test your personalization.
I would like to add a final note of caution when dealing with personalization that uses inferred data like geo-location or language in addition to user profile data. Make sure that your test data includes these variations to tease out potential conflicts in rules between the two. I am unsure if the site above uses this in its personalization, but as I look forward to the football season this year, I have a note of caution to the person testing this personalization – I may live in the state of the Scarlet Knights, but I still bleed Scarlet and Gray!