User Evaluations and Final Concept Selection

This post outlines the process of receiving user feedback on our top 3 concepts, and how we used the evaluation criteria and the user feedback to inform our final decision.

User Evaluations

We presented our 3 concepts to 6 users:

User 1 - Male, 30, high tech use, flies approx. once a year, often checks baggage

User 2 - Male, 33, professional, frequent traveller, proficient with technology, uses smart phone to do online check in.

User 3 - Male, 31, professional, only used airport a few times before, mostly domestic, usually only takes carry-on for domestic.

User 4 - Female 58, retired, doesn’t fly often, not very confident with technology, usually gets her husband to help her use her smart phone and iPad.

User 5 -Female 26, non-native English speaker, flies more than once a year.

User 6 - Male, 60, flies sometimes within Qld for work, fairly confident using tech. Sometimes checks baggage.

We presented and explained our 3 concepts - then allowed them to interact with lo-fi prototypes:

• For the modular seating, had the user move around and arrange real chairs
• For the AirSpoon and Baggage Status Notification concepts, we used the POP app (https://popapp.in/) to create a prototype for a smartphone.

We then observed and asked for feedback about the 3 concepts. The questions asked were based on the Ben's Design Thinking lecture on user evalution, and the team's evaluation criteria :

General questions to ask users:

• What do you like/dislike about this?
• Why would you/wouldn’t you recommend this to a friend?
• What problems can you foresee with this as a system in airports?
• What about this concept could be improved? How?
• What do you think are its best features?
• where would this be most useful? what for?
• Why would you not want to use this? In what circumstances?

Questions for users to rate using evaluation criteria:

Please rate the concepts from 1–5 (1 being the lowest, 2 being the highest)

• Experiential: Do you think it provides you with a real physical experience? How social and public is this experience?
• Interactive: How much can you input feedback into the system and get feedback in return? Do you think the quality of this feedback will be high?
• Engaging: How much does it hold your attention? Do you think this concept is enjoyable ?
• Innovative: Does it incorporate new ways of approaching an existing product/system? How much can the concept transform into a useful application?
• Relevant to Airport: How useful is the concept in the context of a new Brisbane Airport, Domestic Terminal for low cost carriers - considering the possible size of the terminal, types of passengers, and passenger numbers.
• Feasible: How practical is this solution? Will it be cost-effective for the Airport Corporation to implement?
• Original: Is this a completely new concept, different and unique to what already exists?

The final resulting scores, rated out of 100:

AirSpoon: 74.6

Modular Seating: 81.7

Baggage Tracking/Notifications: 78.2

Surprisingly, a few users highest rated concepts differed to their choice when asked which they preferred the most. Baggage Tracking/Notifications was preferred by 3, Modular Seating by 2, and AirSpoon by 1.

One user commented that he found the baggage collection his highest frustration and felt this concept had the most need and practicality, and that is why he preferred that concept over his highest rated which was the Modular Seating.

Selection of the final concept / problem space 

After much discussion regarding the results of the user evaluation and our team's evaluation criteria, we proceeded with choosing the Baggage Tracking/Notification concept (renamed to "Smart Track"). Although we had some reservations about some criteria, our resolutions were as follows:

Feasibility:

If NFC or RFID tags were used to track bags - then this system would preferably need to be implemented in airports Australia wide, in order for users get the full experience of tracking bags from check-in to arrivals. Instead we decided to focus on integrating existing baggage scanning technology to provide "check points" informing users the location of their luggage. Airport baggage handling systems already implement tracking, where scanning stations providing feedback about the bag's location to handlers. When the bag tag is scanned by the automatic scanners, that is linked to it's owner's booking number and the location information sent to Smart Track. For example on arrival, the tag is scanned once it is taken off the plane, and again just as it is put onto the carousel. 

If Smart Track can integrate with other domestic airport baggage systems, then passengers can use the app at both the departure and arrival legs of their journey. As Smart track integrates with existing systems, there may be little extra cost for the airport.

Originality:

How is this different from existing baggage tracking applications? The applications that allow users to track their bags all involve additional cost to the passenger. For example purchasing an electronic tag that the passenger attaches themselves, or the purchasing special luggage which itself has an inbuilt tracker.

Our concept uses existing the baggage scanning system at no extra cost for users. All they need is a device that can connect to the internet. Most passengers smartphones are connected to the phone network or those that do not can connect to airport wifi. All they need to know is their booking number which the baggage location data will be synced to.

Social aspect:

An important factor in the brief was the social criteria, which we believed our concept was lacking. From the user feedback there was a suggestion for notifications being sent to a user's friend or family. This would be useful if a friend or family member was picking a user up form the airport, and wanted to know the best time to meet at the pick up area - they could time their pick up to save waiting time for both themselves and the passenger. This would be even more useful if the passenger didn't have access to a phone to notify the person picking them up.  We thought this was a very useful application, and felt it also made the concept more social, so we included this feature in our final prototype.