The Wizard Of Oz
This variation is pretty much identical with paper-prototype testing. In paper-prototype testing, a human is supposed to play the role of a computer (aka wizard) and show the study participant different paper-printed screens in response to user actions. This type of study can help inexpensively explore various possible interface implementations and choose the one that is most usable.
The Wizard of Oz
A Wizard of Oz study could mitigate some of the cost. Instead of training the system on the existing corpus, the wizard will be able to provide comparable responses at practically zero cost and thus identify whether the corpus is adequate. (The wizard must be very familiar with the corpus of responses to be able to quickly select appropriate ones.) If it is not, the corpus could be expanded and another Wizard of Oz study run on the new dataset. Once a corpus is deemed satisfactory, it could be used to train the machine-learning models.
This variation is commonly used in the discovery phase to understand user mental models, expectations, and vocabulary. The wizard needs to have substantial knowledge of the product domain to compose responses on the fly. Since the open variation can take many paths, the analysis of the data will have to extract themes among the various user behaviors and inputs collected during the test, including topics covered, questions asked, and unexpected user expectations or paths.
One of the chief critiques of the Wizard of Oz method is that the technology is nowhere near technically feasible or is misrepresented by the wizard, so you end up with useless research data. When possible, it is therefore best to encourage designers and engineers to partner on planning the design and logistics of the simulation, so that technical feasibility and usability can be taken into account. If the technology will include existing artificial-intelligence models, consider basing system responses on these models. For example, one might align system responses with a particular structure that an existing model might produce as an output.
You will need to determine how the wizard and the user will communicate, especially if the wizard will be hidden to the participant. Which technologies will be used? What hooks need to be created for the Wizard to be able to control the prototype?
Especially for closed or hybrid studies, it may be a good idea to help the wizard by providing them with a flowchart that they could follow. If the wizard needs to juggle multiple categories of responses, make sure that you categorize them under groups so that they can be quickly found.
If the wizard will compose responses on the fly, have general guidelines on maintaining an experience that is consistent with the product brand. For example, the tone for a customer-service experience should be professional and empathetic. For reservation booking, the tone should be inviting and accommodating. There can also be guidance on the use of specific words, media types, or other elements that might come into play during the test. 041b061a72