 amily-practice physicians are “whole-patient” docs. While a dermatologist sticks to the skin and an orthopedist knows bones, a family practitioner will examine the whole person to find the source of an illness. So it’s fitting that when the American Academy of Family Physicians (AAFP) wanted to learn why attendance growth was stagnant at its Annual Scientific Assembly, it poked and prodded its program from tip to toe.
The annual conference — which keeps family practitioners up to date on clinical and technical advances, practice-management models, and life in a managed-care world — hits all the hot buttons facing family-practice docs today. So AAFP was bedeviled by the event’s plateaued attendance. Only about 6,000 of the organization’s more than 92,000 members (along with about 5,000 guests) attend the assembly each year, with most opting to attend once every four years. Worse, that represented a widening spread. “In years past, the average attendance spread was three years, and we wanted to find out why it had gone up,” says Kathy Reid, marketing research manager at AAFP.
The event, with its comprehensive learning opportunities, should have been a shoo-in for attendance. Many state boards require doctors to obtain between 12 and 50 Continuing Medical Education (CME) credits every year in order to keep their licenses. Although AAFP attendees earn an average of 45 credits per assembly, the event has been facing stepped-up competition from regional and state medical associations, which have added credit-earning courses to their own conference agendas.
At the most basic level, Reid knew that the AAFP congress is evaluated against the same weighty criteria busy docs use to filter any learning, social, or networking opportunity: convenience, cost, and time. To make it convenient for doctors to attend, the annual assembly migrates to a new location in a different part of the United States each year. The organization also minds the fact that many of its members are more than doctors — they’re small-business owners, too. “Part of the issue is that some of our members are in solo practice,” says Sondra Biggs, CMP, director of the meetings and conventions division for AAFP. “If they attend an event, they have to close down their practices. And when the doctor’s not in, he’s not making any money. We have to make sure that whatever is going on at the assembly is powerful enough for the doctors to want to come.”
A THOROUGH EXAM
Over time, Reid and her team have collected plenty of targeted data about the event’s programs and services via post-show surveys. These efforts were not, however, providing answers to the strategic question AAFP knew was at the heart of its challenge: the assembly’s overall perceived value and its effect on attendance. “Even though our research studies have been very valuable in the development of educational programs and activities, they weren’t telling us about the members’ overall experience with the assembly,” Reid says.
Reid knew the key to unlocking the event’s value was to get into the minds of the doctors. But AAFP’s prior assembly-evaluation efforts — the usual suspects of surveys and focus groups — weren’t paying off with the needed level of insight and, more important, actionable information. AAFP needed a new approach, stat.
Reid turned to a technique she had used in a past professional life: mystery shopping. Used as a market-research tactic since the early 1940s, the Mystery Shopping Providers Association defines the method as “the practice of using shoppers who have been specially briefed to anonymously evaluate customer service, operations, merchandising, product quality, and in special cases, employee integrity.” Historically, retailers have been the largest client base for mystery shopping. Now, AAFP wanted to bring the technique to its annual assembly.
The approach would allow a peek inside docs’ brains at every step of the event marketing, registration, and attendance experience, Reid thought, delivering the kind of specific, firsthand, individual, and nonbiased gut responses she needed to find the answers to her perplexing assembly cost-value questions.
2006: ANNUAL CHECKUP
Before the 2006 Annual Scientific Assembly, scheduled for September in Washington, D.C., AAFP recruited its first team of mystery attendees. Reid’s team sent an e-mail to all conference pre-registrants three months prior to the assembly, introducing the mystery-attendee program, asking for volunteers, describing the expected time commitment, and offering an incentive (a registration rebate). Reid’s plan called for mystery attendees to commit to two conference calls prior to the event, a group meeting the first morning of the assembly, and then a second meeting on the morning following the conference, to deliver their reports and discuss their experiences.
Of the more than 100 respondents, AAFP staff named a team of 18 mystery shoppers to represent a cross section of its membership, including group versus solo practice, age, gender, years out of residency, and Scientific Assembly attendance history. The AAFP team then sent the selected mystery attendees an article about mystery shopping, asking the volunteers to review it as background on the research technique.
Reid’s goal was “to get specific input from the minute they decided to attend, all the way through the event,” she says. To facilitate that process, Reid gave the mystery shoppers an assignment: Examine seven broad groupings of event elements, including pre-show marketing, the registration process, on-site transportation, the conference program and schedule, and special on-site elements like the opening ceremony, annual celebration, and other events such as group fitness runs and tours.
First up: pre-show marketing. “We wanted them to evaluate all communication materials — e-mails, newsletters, and registration brochures — they had received thus far,” Reid says. She also asked the group to evaluate its experience registering for the event, whether they chose to register online, over the phone, or by mail. The group then convened in July for its first pre-show team call.
That first call, while confirming that the group overall was pleased with the materials and the registration process, still didn’t deliver the detailed information Reid was after. Part of the challenge was the timing: Since AAFP could only recruit its reviewers from registered attendees, those attendees already had received and read the marketing materials and completed the registration process. So many of their observations were based on recall — not the immediate, gut response Reid knew would bring about the right event diagnoses.
In order to capture more specific and actionable information, AAFP tasked its reviewers to go back to the Web site and any other marketing materials they still had on hand, and to review them again before the second group call, to be held one month prior to the event. This time, she guided the physicians with a much more specific assignment: Tell us how well these materials market our assembly. What about them compelled you to attend? What questions do you have that our materials do not answer? What did you learn from them about special sessions and opportunities at the event? What opportunities are we missing? She then instructed her volunteer researchers to can their polite bedside manner in favor of candor and specificity.
At the close of the second call, Reid laid out the on-site process and assignments for the doctors.
The first day of the assembly, the mystery attendees convened in a group meeting and received custom event-evaluation notebooks Reid’s team had prepared. Each page held an evaluation form for an assembly event Reid wanted the doctors to review. To ensure that the mystery shoppers’ insights would, in aggregate, offer a valid assessment of the various event activities and sessions, Reid’s research team built each evaluation form on a Likert scale (a scale on which respondents indicate their level of agreement with a statement, typically ranging from “strongly agree” to “strongly disagree”). Each form also included space for open-ended notes, comments, and observations.
The notebooks covered the event from end-to-end. “We asked them to evaluate the buses, the on-site registration process, the opening ceremony, the courses, the celebration, the registration, the food, and the exhibit hall,” Reid says. “We wanted them to talk to their peers on the shuttles, in the courses, or while roaming about. If their spouses attended one of our events, we wanted their feedback, too.”
The questionnaires asked the doctors to record their impressions about things like content relevancy and depth, and how it compared to other events. But unlike a typical post-show survey, these observations were being recorded, anthropology-style, through firsthand experience and observation, as the event or experience unfolded — not after the fact. AAFP would gain important insight through attendees’ immediate gut reactions to the elements and events in real time instead of fuzzy, post-event recollections about how they may have felt two weeks (or even two months) prior.
At the end of the 2006 assembly, the team delivered their completed evaluation notebooks and weighed in one last time at an in-person group meeting. This time, the team was asked about the overall experience and, in particular, the service quality they received while on site from AAFP staff. Eight AAFP staff members were called out by name for outstanding service, and each staffer received a gift card for his or her efforts.
By now, the team had grown accustomed to each other, to the AAFP staff, and to the mandate for candor. So while the final meeting took place focus-group style, it was more focus group among fearless and comfortable friends than among polite strangers.
Once all comments, suggestions, and evaluations were compiled into one document of findings, AAFP was able to critically analyze each assembly aspect and the cumulative event experience. “We looked at the research to see what was recommended, what could be implemented at the 2007 assembly to be held in Chicago, what needed to be implemented more incrementally, and which items needed further research,” Biggs says.
During the pre-show calls, Reid and her team learned that while event attendees were generally pleased with the registration process, the AAFP Web site lacked prominently displayed materials featuring the latest assembly news and course schedules. The team also suggested adding testimonials and role-modeling campaigns with past attendees as part of the pre-event marketing efforts, downplaying (or eliminating) pharmaceutical companies’ roles in CME sessions, and expanding and better promoting the peer-to-peer networking opportunities and various learning tracks so attendees could dig deeper into specific interests or needs.
Reviewers’ on-site notebooks revealed favorable impressions of non-CME events such as tours, receptions, and fitness events, except for the opening ceremony, which more than half reported was too long. AAFP also uncovered an opportunity to improve its already well-regarded CME courses, thanks to the attendees’ aggregate assessments. “Our members want their CME courses to be more clinically based, rather than feel-good lessons,” Biggs says. “So we are incorporating more hands-on courses with new processes and procedures they can bring into their practices.”
2007: DEEPER DIAGNOSIS
Based on the feedback gained from the inaugural review team in 2006, AAFP implemented several incremental changes for the 2007 event, such as including testimonials in pre-show materials, adding several features to the online registration experience, and bringing in big-name keynoters, political operatives Mary Matalin and James Carville, to add energy and attract attendees to the opening ceremony.
To gauge the efforts’ success — and to dig deeper into larger questions first raised in 2006 — Reid’s staff recruited a new mystery-attendee team to evaluate the 2007 event, again from pre-show to post. Based on its experience in 2006, AAFP also made changes to the mystery-attendee process that would bring strong results more efficiently for both AAFP and the participating doctors. Rather than two pre-assembly team calls, AAFP scheduled one, for which attendees were better prepared with specific questions and instructions ahead of time. They also allowed participants to pick up their on-site evaluation packets individually between 7:30 a.m. and noon on the first day of the assembly, where they also received a briefing about their objectives and guidelines.
The detailed preparations and specificity paid off. The AAFP team was asked to pay particular attention to the online-registration enhancements, and give candid assessments of their value during the team’s pre-assembly meeting. A new “schedule builder” function, for example, missed some components reviewers thought would help complete the registration process seamlessly. In their evaluations, the mystery attendees suggested additional features, such as a listing of alternate course times and a drag-and-drop technology for adding courses to a schedule planner that would address several of the gaps in the new tool. Attendees also expressed interest in a “one-stop-shop” event planner on the AAFP Web site for information about local happenings, transportation options, exhibit information, and hotel availability updates.
|
As part of its on-site evaluations, the 2007 panel also commented on the change in keynote speakers. A majority suggested that while Matalin and Carville certainly drew a large crowd, the content value was questionable as the topic was not relevant to family medicine.
More important was the team’s insight into the CME sessions and the overall educational program. Historically, the assembly sessions had ranged anywhere from one hour to three hours in length, and were not structured on a synchronized schedule. The mystery attendees reported that this approach “made it very hard for them to determine which courses to attend, as their next course could potentially be scheduled to begin prior to the previous one ending,” Biggs says. It also made it nearly impossible for attendees to network because there wasn’t a common break time. Without the mystery shopper program, AAFP may not have uncovered this critical, shared opinion, as traditional seminar-evaluation approaches typically focus on content and speakers, not the bigger picture of how all those seminars are puzzled together in the first place.
2008: AGGRESSIVE TREATMENT
Looking ahead to the 2008 assembly to be held in September in San Diego, AAFP is aggressively implementing conference changes based on the 2006 and 2007 mystery-attendee teams’ input to increase the event’s learning, networking, and relationship-building value — all factors critical to growing attendance. For example, the 2008 conference will feature 60- or 90-minute courses, all starting and stopping at the same time, with a 30-minute break in between. “The 30-minute break will create a little more camaraderie and bring the attendees closer together, creating more of a buzz at the convention,” Biggs says.
The keynote misalignment will be addressed, too. While the big-name approach will still be used to draw attendance to the opening ceremony, this time AAFP will blend recognizable names with a hot topic. For 2008, Terry McAuliffe (former chairman of the Democratic National Committee)
and John Kasich (former Republican representative from Ohio and current Fox News commentator) will attend, leading a town-hall-style discussion about each party’s take on national health-care concerns. “Members will e-mail questions and we will pick five of those questions, which will be the basis of their discussion,” Biggs says. It’s a potent combination of high-
visibility personalities, a highly relevant topic, interactivity, and customer-designed content.
While helping AAFP collect valuable insight into the attendee experience, the mystery-attendee program is also creating event evangelists, one doctor at a time, among previously noncommittal AAFP members such as Dr. Wayne J. Reynolds of the Port Warwick Medical Association in Newport News, VA. Reynolds, a practicing physician for more than 15 years, had previously opted to attend state meetings to earn his CME credits rather than the annual AAFP national conference. “I volunteered to be part of this program because I felt this would be a super opportunity to give my observations of the national convention, and to know that my opinions would be valued,” he says. “The location, timing, and cost deterred my attending the national convention in the past. Now I will definitely attend again.”
Members like Reynolds are exactly the group AAFP is trying to rein back in. And while cost and convenience will always be a battle, AAFP is confident the changes it is implementing for 2008 — based on ideas first hinted at in 2006 and refined by the mystery-attendee team in 2007 — will pay off with growing attendance in 2008 and beyond. But AAFP knows that this is only the beginning of an ongoing process of event improvements.
“We are going to continue with the mystery attendees for a few more years,” Biggs says. “We’ve been making good inroads into the complete attendee experience, and that’s something we hadn’t been able to do before.” With an optimistic attitude and a plan for attendee-driven enhancements, the prognosis for AAFP and its annual assembly is definitely optimistic. e
|
Historically, retailers have been the largest client base for mystery shopping. Now, AAFP wanted to bring the technique to its annual assembly.
The mystery-attendee approach would deliver the kind of firsthand, individual, and nonbiased gut responses AAFP needed to find the answers to its perplexing assembly cost-value questions.
While helping AAFP collect valuable insight into the attendee experience, the mystery-attendee program is also creating event evangelists, one doctor at a time.