1 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS.
Slide 11 DR. VASSILIS-JAVED KHAN WORKSHOP: PUTTING MOBILE SERVICES INTO CONTEXT WARSAW JUNE 2011 EVALUATING USER EXPERIENCE OF MOBILE, CONTEXT-AWARE APPLICATIONS IN AUGMENTED VIRTUALITY Slide 2 2 Dr. Vassilis-Javed Khan Dr. Paul Ketelaar Dr. Marnix van Gisbergen Dr. Andrs Lucero Dr. Koos Nuijten Arief Ernst Hhn Nils Desle FINANCIAL SPONSOR: RAAK Pro scheme of: RESEARCH PARTNERS: Slide 3 3 ME Senior Lecturer, Academy of Digital Entertainment, NHTV Breda University of Applied Sciences, The Netherlands Philips Research, Eindhoven Vodafone R&D, Maastricht MSc., CEID, UPATRAS.GR PhD., Dept. Industrial Design, Eindhoven University of Technology, TUE.NL Slide 4 4 INDEX Position the approach we take compared to Lab & Field studies Present our setup Present 3 studies of the evaluation of a mobile, location-based app in our setup Conclusions Challenges / Future work Slide 5 5 ULTIMATE GOAL OF CONTEXT- AWARE APPS Provide added value to user How to evaluate users experience while taking context into account? Lab vs. Field test Slide 6 6 LAB Highly controllable Useful for usability-related studies Ecological valid? Context? Slide 7 7 FIELD TEST Context Difficult to conduct Logistic & technical challenges Difficult to observe Few participants Qualitative Slide 8 8 EVALUATION IN AUGMENTED VIRTUALITY Context Easy to manipulate Large numbers of participants Can include mobile phones, tablets, other interaction devices Can observe behavior Khan, V.J., Nuijten, K. & Desl, N. (2011). Pervasive Evaluation Application within Virtual Environments. Proceedings of International Conference on Embedded Computing and Communication Systems, PECCS 2011, 261-264 Slide 9 9 MIXED REALITY Real EnvironmentAugmented RealityAugmented VirtualityVirtual Environment Milgram, P. & Kishino, F. (1994) A taxonomy of mixed reality visual displays. IEICE transactions on information systems, E77-D(12) VIRTUALITY CONTINUUM Slide 10 10 OUR SETUP: CAVE 4 back-projection screens 3.5 x 2.6 m. Illusion of a continuous 360 o view Head-tracking with 4 Wii-Motes User wears a headband with infrared LED Human joystick V.E. created in Maya & rendered using the OGRE 3D engine Two V.E. Supermarket, City in an island More to come Slide 11 11 Slide 12 12 Slide 13 13 Slide 14 14 OUR SETUP The virtual supermarket is a generic supermarket, based on common denominators found in typical Dutch supermarkets Contains: isles, freezers, checkout counters, etc., all rendered with photo realistic graphics Audio: typical supermarket music, typical announcements ~20% of the shelves are filled with products Android app, Bluetooth communication with V.E. Slide 15 15 NOVELTY Immersive User Experience Nearly photo-realistic vs. low resolution graphics CAVE vs. V.E. presented on monitors Research focused on UX, going beyond effectiveness and efficiency i.e. perceived intrusiveness, shopping behavior, feeling of presence, etc. Capturing actual behavior vs. opinion Marketing studies used exclusively scenario-based methods Slide 16 16 3 STUDIES Location-based ads Study 1: 27 participants (pilot) Study 2: 70 participants Study 3: 70 participants Wanted to measure: Perceived intrusiveness of ads Has been reported: Perceived intrusiveness of ads leads to ad irritation 1 High values of ad irritation lead to ad avoidance 1 1 Li Hairong, S. & Lee, J., 2002. Measuring the intrusiveness of advertisements: Scale development and validation. Journal of Advertising, 31(2):3747 Slide 17 17 STUDY1 Ad: Chewing gum Push Controlled for: Product involvement Attitude towards advertising in general Measurement: Perceived intrusiveness Results: Location-congruent ads lead to less ad intrusiveness than location-incongruent ads (virtual location) GROUP1: SAW AD WHILE AT SOUP SHELFGROUP2: SAW AD AT COUNTER (NEXT TO CHEWING GUM) Hhn, A.E., Khan, V.J., Ketelaar, P., Nuijten, K., Gisbergen van, M., Lucero, A. (2011-to appear). The Effect of Location on Perceived Intrusiveness of Mobile Ads. Proceedings Chi-Sparks 2011 Slide 18 18 STUDY2 Differences with study 1: More participants Open task Exact same location for both groups Products on shelf changed for groups Soup ad for both groups Hhn, A.E., Khan, V.J., Nuijten, K., Gisbergen van, M., Lucero, A., Ketelaar, P. (2011-to appear). The Effect of Location on Perceived Intrusiveness of Mobile Ads. Proceedings ICORIA 2011 Same result with study 1 Location-congruent ads lead to less ad intrusiveness than location-incongruent ads (virtual location) Slide 19 19 STUDY3 Same setup with Study 2 Ad of Soup Location-Based Measurement: perceived intrusiveness Differences: Same products on shelves for both groups One group saw a social recommendation along with the location-based ad Results: ad with social recommendation is perceived as more intrusive GROUP1: Along with the ad they saw a social recommendation GROUP2: Saw just the ad Study conducted by: Stephan van de Kruis Slide 20 20 CONCLUSIONS Location does positively affect perceived intrusiveness of mobile ads Social recommendations would not help mobile, location-based ads Demonstrated a V.E. where UX research of mobile, context-aware systems can be performed Adjustments of the V.E. and context are done easier, are more controllable and faster compared to an experiment in a real life setting Reproducibility If there is an initial halo-effect it does not affect the actual perceptions Slide 21 21 CHALLENGES / FUTURE WORK Fill supermarket with a representative product set (~20K products) Head tracking with MS Kinect Have a more complete phone app Shop with mobile phone Include personalization profile Ability to interact with products Use log and video input for analyzing behavior Social context: Avatars Corroborate qualitative findings of previous research endeavors & explore new research directions Validation studies Slide 22 22 SUMMARY 3 studies of mobile, location-based ads evaluation in augmented virtual supermarket ACKNOWLEDGEMENTS Photos & Graphics: Arief Ernst Hhn, Paula Isphording CONTACT email@example.com vjkhan.com @v_j_khan Research partners: Financial sponsor: RAAK Pro scheme of: