QoMEX 2019 - Berlin, Germany
Call for paper: submission for QoMEX 2019 Special Session on Quality of Experience and User Experience Research with and for Crowdsourcing

Abstracts submission deadline: January 18, 2019
Paper submission deadline: January 22, 2019
Special Session page: https://www.qomex2019.de/special-session-quality-of-experience-and-user-experience-research-with-and-for-crowdsourcing/

Motivation and objectives
-------------------------

Crowdsourcing has become a valuable tool in Quality of Experience (QoE) and User Experience (UX) research. In contrast to lab-based studies, crowdsourcing enables researchers to easily and quickly collect a large number of subjective ratings from a diverse set of participants. This diversity further enables researching cultural effects, influence factors generated by different end-user devices, and effects of different surrounding environments. However, these opportunities come at costs. Experimenters have only limited control of the test settings and in which environmental conditions the study takes place. Additionally, the remote test procedure can be error-prone as the test participants are not under the direct supervision of the test conductor.

In this context, the aim of the special session is twofold. On the one hand, we want to foster contributions following the traditional way optimizing and designing crowdsourced subjective studies for Quality of Experience and User Experience research. This includes novel methodologies for quality assurance and replicability, new fields of application like assessing the QoE of IoT devices in a crowdsources fashion, and using new technologies like wearable to collect additional environmental and user signals. On the other hand, we also want to raise awareness and foster research in a new research direction, the Quality of Experience and User Experience of crowdsourcing workers. Crowdsourcing has become mature in the academic and business usage, and much effort is put into a (cost-) efficient and quality optimizing the design of the tasks. However, no or only very little efforts are made to improve the working experience of the workers.


Topics of interest
------------------
We invite you to submit original papers in any area related, including but not limited to:

Crowdsourcing for subjective studies
-- Novel application
-- Limitations of current crowdsourcing systems
-- Quality control mechanism / Monitoring of test and environmental conditions
-- Large scale crowdsourcing studie
Reproducibility of result
-- Cross platform studies / Repeated studies
-- Assessment of hidden influence factors / Impact of hidden influence factors
-- Bias estimation and bias reduction
-- Standardisation of crowdsourcing test methods
Usability and User Experience of crowdsourcing tasks
-- Optimization of task interfaces and task workflows
-- Relation to result quality and worker motivation
-- Enhancing workers’ UX (e.g., by means of gamification of tasks)
-- Quality of complex crowdsourcing workflows (e.g. combination of AI and Crowds)
Interconnection of crowdsourcing and lab-based tests, e.g.,
-- Studies comparing results from lab and crowdsourcing
-- Adaptations of established lab-test standards to the crowdsourcing environment


About Organizers
----------
- Babak Naderi (Technical University of Berlin)
- Matthias Hirth (University of Wuerzburg)
- Katrien De Moor (Norwegian University of Science and Technology)