pdf

CLICK HERE FOR A THE ENTIRE FIELD TEST RESULTS

BRIEF

Based on the results of my field test I do believe a sufficiently sophisticated chatbot can provide a more beneficial experience for the consumer — cutting down on information overload by putting users in the driver’s seat to ask questions. The majority of my respondents showed enthusiasm about the future of this technology. Three out of five people concluded that they would use MARK to answer specific question if it were readily available on the market. Even reading the conversations members of the focus group had with MARK gave me a glimpse into their willingness to not only use, but troubleshoot this type of technology to get information as an alternative to existing media. But my respondents also made it clear that MARK, in its current form, is not that answer.

The AI in Chatfuel is buggy at best and provides several limitations on the design side. A simple addition to the program that would allow the bot designer to segment AI rules could make the technology more robust. If “what happened” when asked under ‘story one’ provides a different response than if “what happened” where to be asked under ‘story two’ it would allow greater specificity in creating AI rules — and subsequently a smarter chatbot. Ideally this would also make the chatbot smarter in the long run.

Additionally, a chatbot designed, programmed, and launched directly to mobile platforms would have more flexibility than the chatbots (like MARK) designed and housed on Facebook Messenger. In the very near future I can see this technology take off as AI continues to improve. As AI improves, I think we will see these news fetching capabilities take a the spotlight on virtual assistants like Amazon’s Echo, Google’s Google Home, and Apple’s HomePod. These assistants will be able to provide news and context in a conversational manner that encourages the user to engage often. Who knows, maybe MARK will be among those technologies.

 

THE PROBLEM

Today, we find ourselves at a crossroads where there’s more information readily available than ever before and now consumers are looking for better ways to navigate it. “Information Overload” has been in our vocabularies since the 1970s, but now it’s more evident than ever before. And if the notification center on your smart phones looks like the one to the left — you aren’t alone. In 2016, an estimated 20% of Americans felt ‘overloaded’ with information and nearly half — 46% — of people under the age of 50 feel as though more information makes their lives ‘more complex’. (Horrigan, 2016) While current advancements in technology (the internet, smartphones/tablets, and social media) have created two-way interactive communication between the producers and the consumers of information, there’s still more to be done.

As consumers’ lives increase in complexity and become more overloaded with information, figuring out what’s important is more difficult and managing information becomes challenging. Consumers question what’s true and what’s important. They turn to Googling for answers to their pressing questions rather than sifting through lengthy articles to find them. But Google isn’t perfect and doesn’t always understand your question. Finally, there’s a solution.

THE SOLUTION

  • Chatbots are the answer. It’s a fancy word for a computer program, designed by humans, to retrieve information desired by the user interacting with said program. In its most simplistic form, a chatbot can help a user find information and, in its most complex form, the technology can predict what the user needs to know next. (Kim, 2017)
  • Birth of chatbots. The top four chat apps have more total users than the top four social media apps. (Business Insider, 2016) Chat apps allow a consumer to send text, photo, video, and emoji messages to a person or group of people of their choosing. Consumers are looking for more personal and engaging ways to connect with people and with content. For that very reason consumers are turning to chat apps instead of social media apps to fulfill that need.
  • Chatbots, as we know them. Chatbots currently exist in one of two functions. One form fits into messaging apps like Kik, WhatsApp, and Facebook Messenger and provide instantaneous responses based on questions asked or a provided prompt. The other form of chatbots we are familiar with is the virtual assistant like Apple’s Siri or Amazon’s Alexa. These programs work as companions to help us navigate the everyday world and do things like find and remember information, make purchases, schedule reminders, and control other devices (often audibly). (Kim, 2017)
  • Now is the time. 63% of respondents in a recent survey of smartphone users ages 18-35 said they would consider messaging and online chatbot to get in touch with a business or brand. (Pounder et al., 2016)
  •  Elven letters. If Millennials are the generation killing off social media apps, they’re simultaneously rushing to chat apps. According to ComScore’s 2016 mobile app report, millennial users (18-34) make up 66% of users on Kik Messenger and 76% of users on GroupMe — both increasingly popular chat apps that chatbots are now being integrated into. (McAlone,2016) As Millennials surpass Baby Boomers as America’s largest living generation it’s evident that producers must market their products to the target demographic of the not-so-distant future. (Fry, 2016) This includes newsrooms, which should begin delivering information in ways that meet Millennials where they are: using chatbots.

THE COMPETITION

The Wall Street Journal delivers top stories and customized stock reports over its chatbot which is hosted on Facebook Messenger. WSJ was one of the first news chatbots available on Messenger. (Konig, 2017)

CNN is experimenting with chatbots across several messaging apps — Kik, LINE, and Facebook Messenger. The company has also experimented with voice-activated assistants like Amazon’s Echo. CNN measures the bots success by weighing reach engagement and monetization. (McEleney, 2017)

TechCrunch provides a more personalized take on their chatbot by arming the bot with information about the user. If there is a type of article that the user reads more frequently on their TechCrunch.com the data will then be passed on to the bot to recommend those types of stories more often. The app lives on Facebook Messenger. (Bernard, 2017)

THE IDEA

Meet MARK. You get the news you want — how you want it and when you want it. MARK is the first mobile news app powered by a chatbot. With MARK, your curiosity is the focus. Questions prompt information, not the other way around. Open the app on your smartphone or tablet, choose a headline or topic, and ask MARK a question. Get instant, relevant, information that answers your questions. Go beyond the headlines. Not sure where to start? MARK suggests new information to explore on the topic you’ve chosen. MARK knows the best answers because he has the best team behind him. No more guessing. Just ask MARK.

THE HYPOTHESIS

A chatbot specifically developed for news with responsive AI will provide a more beneficial experience for the consumer — cutting down on information overload by putting users in the driver’s seat to ask questions. Ideally, this chatbot would be created as a standalone app available in the Apple App Store and Google Play Store. But, given the confines of this field test, I used Facebook Messenger as the platform to test my hypothesis. I used Chatfuel as the client to build the chatbot that I released on messenger.

THE DESIGN

  • I selected three (3) stories from NBCnews.com to serve as the information to power MARK. These stories appear in the “HEADLINES” section after the user is presented a welcome message.
    • STORY ONE: Montana GOP Candidate Charged with Assaulting Reporter on Election Eve
    • STORY TWO: ‘Wonder Woman’ Shatters Box Office, Glass Ceilings with $100.5 Million Debut
    • STORY THREE: Retail Wreck: 1,000+ Stores Close in Single Week
  • I then broke down each story, sentence by sentence, and grouped similar topics together. This allowed me to create different replies for MARK within a single story to answer similar questions. Chatfuel refers to these responses as blocks.
  • From there I assigned a series of AI rules to the corresponding information blocks in the story. I wrote out as many corresponding phrases as I could think of (without creating overlap between stories) to predict users’ questions and also improve MARK’s AI.
  • The number of AI rules varied by story. One story had as many as 115 AI rules while another had as few as 32. This isn’t one size fits all, it’s about trying to predict user behavior and beefing up MARK’s AI by arming it with enough information about a topic so it can better interpret users’ questions.
  • Some of the AI rules are a single word, others are short phrases, and finally some of the more complex rules are complete sentences.

I crafted the dialogue for MARK to be somewhat conversational while still reminding the user ‘this is still a computer’. The choice was strategic. In researching bot best practices before designing MARK I came across a critique of the Wall Street Journal Facebook Messenger bot — widely regarded as one of the best on the market. (Konig, 2016) The critique praised the conversational nature of the bot with phrases like “let’s get started,” while drawing attention to the use of “we” in referencing the product, which could make it difficult for the users to understand the product is a bot and not a team of real people. From that feedback I decided to straddle the fence with MARK’s design — allowing users to feel as if they can casually engage with the bot while reminding the user that MARK isn’t as advanced as a Siri or an Alexa.

Additionally, I realized it would likely be difficult for users to think of questions initially as they learn which questions MARK handles best and troubleshoot their interactions. So, to hopefully ease that learning curve, I created a series of help menus to assist the user in navigating MARK. If the user typed “I’m stuck” followed by the name of the story they were learning more about, MARK would suggest a question that is linked to one of the information blocks. For example, if the user typed “I’m stuck wonder woman,” MARK might suggest, “How much money did the movie make internationally?” If the user typed that question into the search field, MARK would reply with the corresponding answer. Questions to the corresponding topic would then cycle through until the user uses all of the associated information blocks. In the event MARK didn’t understand a question it would display this default answer, which attempted to redirect the user back to the headlines carousel or to the help menu where they could eventually ask MARK to suggest questions.

Here’s an example of the set of questions MARK would suggest to the user in the event the user asks for help with queries for a particular topic. This is an example of the questions the user would cycle through if typing in “I’m stuck Gianforte” in order to get help with the story or then-Republican Congressional Candidate Paul Gianforte assaulting a reporter.

THE FOCUS GROUP

These five brave and awesome souls volunteered to be my focus group for this field test. By design, I chose five millennials for the focus group to gauge their attitudes towards chatbot technology. The research indicated messaging apps are the wave of the future as millennials sign out of social media, but is that what’s actually happening? I asked some demographic, news consumption, and chatbot-interest questions of my focus group to answer that question.

THE EXPERIMENT

All of the directives for the experiment were sent to the focus group via Survey Monkey. The first page welcomed and thanked the focus group while providing and overview of the field test. I also provided an estimated duration of 45 minutes to complete the test. The next page explained the “pain” of information overload then gave an overview of MARK and presented it as the “pain reliever”. I provided a set of tips on using MARK including how to best ask question and how to ask for help.

I started by asking the respondents demographic questions (name, age, etc.). I then asked the respondents about news consumption habits — particularly how likely the respondent was to use a particular media or medium on a daily basis to consume news. The group is then told to spend roughly 5 minutes engaging with each headline in the chatbot before moving on to the next headline, and finally returning to the survey. I then present the respondent with one last set of tips on how to use MARK and then provided a link to the chatbot housed on Facebook Messenger.

After engaging with the chatbot, the respondent returns to the survey. That’s when I gave the respondents a pop quiz on the headlines programmed into MARK. There were three multiple-choice questions for each of the three headlines. Following the quiz, there were several questions to gauge the respondents’ interactions with MARK. Questions included the following:

  • The most frustrating part about using MARK is…
    • (Dropdown menu of pre-selected answers)
  • What didn’t work as expected?
    • Open-ended short text field
  • Likert Scale on the following:
    • MARK is easy to use
    • MARK understood my questions
    • MARK is helpful
    • MARK is fun
    • MARK responds quickly
    • MARK is the future of news
    • If MARK were on the market today, I would use it daily
    • If MARK were on the market today, I would use it only to get answers to certain questions
    • If MARK were on the MARK today, I would not use it. This product is not for me.
  • I would most likely use MARK to learn about this type of news:
    • Selection choices: breaking news, politics, consumer news, environmental news, lifestyle/DIY, entertainment, crime, stocks/financial news, sports
  • There was then an open-ended comment box prompting the respondent to give as much feedback as they’d like about their experience with MARK and to be honest!
  • The final question asked the group if they would recommend MARK to a friend.

DESIGN FLAWS 😦

  • I neglected to require the respondents to answer all questions on a page before advancing to the next one. As a result, respondents occasionally skipped questions perhaps in an effort to avoid sharing negative criticism of MARK. This was a major design flaw and somewhat-limited the scope of my findings. In retrospect, I would have required all fields be completed before advancing to the next page.

THE ANALYSIS

Overall, I believe the focus group is enthusiastic about the concept of using a chatbot to engage with information and news of the day in a more intimate way. I am confident in this because, despite what by any account would be characterized as a rocky field test, three out of five or my respondents said that they would recommend MARK to a friend. Sharon, in her feedback, called MARK an “awesome concept.” Adrienne said she was “very interested in the technology.” But, both of those respondents finished their respective sentences of praise with a criticism of MARK’s AI. This was one of the biggest frustrations evident from reviewing respondents’ conversations with MARK and in the subsequent survey.  Here’s an example of a couple of Ryan’s more frustrating interactions with MARK’s sometimes seemingly absent AI. Ryan fires off four different questions — in different styles and focus — and each time MARK responds with an error message generated whenever the AI can’t interpret the question and pair it with a block of information.

But there are other examples of one of the respondents getting stuck, asking for help, MARK providing a question, and the respondent then getting an answer and moving on to the next question. This is the system working as good as can be expected to work with the limited AI capabilities of Chatfuel. Michael ultimately spoke positively of his experience. “I do like the concept, and think it could be helpful to go through stories just to get what the consumer wants to know out of it, rather than reading the full story,” Michael said.

And here are a couple of examples MARK’s AI working just as planned. The AI likely works so well here because I predicted the user would ask those exact questions and loaded that exact phrasing in the AI rules. Because of her luck in getting answers from MARK early on, Adrienne never used the help feature. Perhaps that’s why she said the most frustrating part about using MARK is “not knowing what questions to ask.” While Adrienne found issue with MARK in its current iteration, she seems optimistic about what it could become. “The technology in its current state is more cumbersome than valuable, but with further development could reinvent news as we know it, which would provide some much needed relief,” she said.

3/5 Respondents said they would recommend MARK to a friend

THE CONCLUSION

Based on the results of my field test I do believe a sufficiently sophisticated chatbot can provide a more beneficial experience for the consumer — cutting down on information overload by putting users in the driver’s seat to ask questions. The majority of my respondents showed enthusiasm about the future of this technology. Three out of five people concluded that they would use MARK to answer specific question if it were readily available on the market. Even reading the conversations members of the focus group had with MARK gave me a glimpse into their willingness to not only use, but troubleshoot this type of technology to get information as an alternative to existing media. But my respondents also made it clear that MARK, in its current form, is not that answer.

The AI in Chatfuel is buggy at best and provides several limitations on the design side. A simple addition to the program that would allow the bot designer to segment AI rules could make the technology more robust. If “what happened” when asked under ‘story one’ provides a different response than if “what happened” where to be asked under ‘story two’ it would allow greater specificity in creating AI rules — and subsequently a smarter chatbot. Ideally this would also make the chatbot smarter in the long run.

Additionally, a chatbot designed, programmed, and launched directly to mobile platforms would have more flexibility than the chatbots (like MARK) designed and housed on Facebook Messenger. In the very near future I can see this technology take off as AI continues to improve. As AI improves, I think we will see these news fetching capabilities take a the spotlight on virtual assistants like Amazon’s Echo, Google’s Google Home, and Apple’s HomePod. These assistants will be able to provide news and context in a conversational manner that encourages the user to engage often. Who knows, maybe MARK will be among those technologies.

References

Bernard, T. (2016, April 19). TechCrunch launches a personalized news recommendations bot on Facebook Messenger. Retrieved June 1, 2017, from https://techcrunch.com/2016/04/19/all-your-bots-are-belong-to-us/

Business Insider. (2016, September 20). Messaging apps are now bigger than social networks. Retrieved May 7, 2017, from http://www.businessinsider.com/the-messaging-app-report-2015-11

Fry, R. (2016, April 25). Millennials overtake Baby Boomers as America’s largest generation. Retrieved June 15, 2017, from http://www.pewresearch.org/fact-tank/2016/04/25/millennials-overtake-baby-boomers/

Glover, J. (2017, June 10). Notification Center [Photograph].

Horrigan, J. B. (2016, December 07). Information Overload. Retrieved May 17, 2017, from http://www.pewinternet.org/2016/12/07/information-overload/

Kim, L. (2017, March 22). 11 Amazing Facts You Might Not Know About Chatbots. Retrieved June 15, 2017, from https://www.inc.com/larry-kim/11-amazing-facts-you-might-not-know-about-chatbots.html

König, J. (2017, February 07). Chatbot Studies: WSJ for Facebook Messenger. Retrieved June 1, 2017, from https://www.chatbot-academy.com/chatbot-studies-wsj-facebook-messenger/

McAlone, N. (2016, September 14). 20 apps millennials like way more than other age groups do. Retrieved June 15, 2017, from http://www.businessinsider.com/20-apps-popular-millennials-2016-9/#airbnb–66-millennial-users-1

McEleney, C. (2016, November 16). What CNN has learnt after six months of chatbot experimentation. Retrieved June 1, 2017, from http://www.thedrum.com/news/2016/11/16/what-cnn-has-learnt-after-six-months-chatbot-experimentation

Pounder, J., McFaul, R., Barton, S., Brauer, D., Gerson, D., Rumlova, D., . . . Leizaola, R. (2016). Humanity in the Machine (Publication). Retrieved May 17, 2017, from Mindshare website: http://www.mindshareworld.com/sites/default/files/MINDSHARE_HUDDLE_HUMANITY_MACHINE_2016_0.pd

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s