Lesson 05

Topic

Outcomes and Findings, Evaluating with participants

Reading Material

"Baumer, E., Blythe M., and Tanenbaum, T. 2020. "Evaluating Design Fiction: The Right Tool for the Job". In Proceedings of the 2020 ACM Designing Interactive Systems Conference."

baumer_blythe_tannenbaum.pdf

1) What are the new terms in this text? How are they defined? Where do they come from?

Design Fiction: a design practice aiming at exploring and criticising possible futures by creating speculative, and often provocative, scenarios narrated through designed artifacts (wikipedia)
ACM: Association for Computing Machinery
Evaluative Frame: structured approach that guides the process of evaluation

2) Who are the authors? Where do they work? Who do they refer to?

The paper was published in 2020
Baumer and Tannenbaum are academics in the field of Computer Science/Informatics, and Blythe is in the field of Design. All three authors have focused on the broad topic of HCI before.

3) What questions come to your mind from reading this text?

How can you extract the most of qualitative feedback, without exactly knowing where the persons biases are?

4) How does it affect your design practice? What applications do you see in your practice?

The paper focuses on how design fiction should be evaluated, and how this can be both more consistant and useful.
Bringing in "outside voices" is not always helpful, which is true in the field of Interaction Design/HCI, as well as practically all other artforms like literature, film, etc.
It is important to identify the "Primary Evaluative Frame" to better understand the critique.
There should not be a definite set of criteria to review design fiction or for small qualitative studies, as different people will focus on different evaluative frames.

"Bell, Genevieve, Blythe, M. & Sengers, P. 2005. “Making by Making Strange: Defamiliarization and the Design of Domestic Technologies”. In ACM Transactions on Computer-Human Interaction. 12. 149-173."

bell_et_all.pdf

1) What are the new terms in this text? How are they defined? Where do they come from?

Defamiliarization: The act of making the familiar seem unfamiliar, challenging conventional ways of thinking and creating new perspectives. This can for example be achieved by very literally explaining something using different terms than usual. It is not a scientific method.
Naturalization: The process of cultural phenomena gradually becoming seen as "natural", until their cultural roots are obscured.
"Flow": peak experience of total absorption in an activity. Close match between skill and challenge, clear goals and constant feedback on performance, characterized by decrease in self-conciousness and time distortion.

2) Who are the authors? Where do they work? Who do they refer to?

The paper was written in 2003, published in 2004.
Blythe seems to have switched universities since this paper was written
Sengers is a professor in inforamtion science/science & technology stufies
Bell is now the vice-chancellor of the australian national university, and is credited with being an industry pioneer in the field of user experience. She is a cultural anthropologist.

3) What questions come to your mind from reading this text?

in which situations is a "defamiliarization" approach useful? Should it be used for almost any design question or should it only be reserved for when an existing design poses problems?

4) How does it affect your design practice? What applications do you see in your practice?

Exchanges with people outside of a certain "bubble" can help discover blindspots due to unfamiliarity of outsiders to an established system.
"Talking about something as if you were explaining it to someone from mars"
Defamiliarization is different from ethnography, because it does not explicitly focus on the users and their needs, but can also be incorporated in the design process itself.
A Historical and cultural analysis can help with the defamiliarization process.
i should read this text again in the future as i really enjoyed it and found it fascinating.
Defamiliarization can be especially helpful when trying to change gendered design legacies, as products such as clothing irons etc have been historically designed with a "female aesthetic", and by rethinking those design decisions, gendered design legacies (probably, maybe) can be altered.
Defamiliarization provides the "opportunity to actively reflect on, rather than passively propagate, the existing politics and culture of home life".

"Preece, J., Rogers, Y., & Sharp, H. 2002. “Introducing Evaluation”. In Interaction Design. Wiley."

preece_rogers_sharp.pdf

1) What are the new terms in this text? How are they defined? Where do they come from?

Usability testing: Evaluation that tests a users performance and evaluates their satisfaction with the system. Is done in a laboratory setting. Particularly important in later stages of design.
Field studies: Evaluation that is done in a natural environment as opposed to a controlled setting.
Analytical evaluation: Evaluations that dont involve end-users. Is done in a laboratory setting, often involving heuristics.

2) Who are the authors? Where do they work? Who do they refer to?

Book published in 2013
Preece is a professor at the University of Maryland. Notably, her research focuses on online communities and is known for her work on what makes online communities successful, which i find really interesting.
Rogers is a british psychologist and computer scientist. She has a PhD in HCI, And she was/is professor at multiple prestigious english universities.
Sharp is a Mathematician, Computer Science postgraduate and professor in Software Engineering, focusing on HCI, Software Engineering and Project Management.

3) What questions come to your mind from reading this text?

How are the priorities of the issues assigned, what distinguishes medium from low priority etc.?

4) How does it affect your design practice? What applications do you see in your practice?

Designers are often not the target user population, so it is important to get as much evaluation from as many different user groups as possible.
the success of an interactive products depends doesnt depend on usability alone. Aesthetic, emotional, engaging and motivating qualities are equally important.
Evaluation methods are often used together or chained together.
It is likely that every iteration of testing/evaluating will reveal new areas where improvements can be made, so knowing when its time to move on is important.

Brief Summary of Lesson

The lesson was hosted by Oli, and he talked about what Design Fiction is, what defamiliarisation is and why it is important to evaluate projects correctly. The presentation was very interesting and served as a good recap of the texts we read. The exercise we had to do revolved around the concept of defamiliarisation. We all had to design a fridge for a specific user group, with a third of the class having to design a fridge for a dog, a third having to design a fridge for a plant, and the last third having to design a fridge for an A.I. This was interesting as it required us to think outside the box and try to throw any preconceptions we had about fridges out the window.

Because the input from Oli was shorter as he was presenting alone, Joëlle gave another input afterwards, where we did an exercise. We had to individually answer these questions related to our group project:

1) What criteria/aspect/quality is most important for you to achieve by the end?
- A good video showcasing/explaining the original concept of the project, regardless of how the prototype turns out.

2) What do you think it is for the audience? (be specific)
- An interesting experiment to try and "defamiliarize" the stressy environment

3) What do you think it is for your teachers?
- A project to see if we learned anything from the IAD Methods/Process class

4) One aspect you think the project should address but hasnt?
- What to do after you step in the sand, and you have to put your shoes back on.

It is interesting to note that our group all had completely different answers, but we had little time to compare and contrast because we had to get right back to work after the lecture. We will however talk about why our answers were different and if there are any conflicts that need to be resolved before the project ends.

Takeaways

The main takeaway from this lesson for me was that Joëlle demonstrated the difference between qualitative and quantitive feedback quite nicely, when she talked about the different "paradigms" of evaluating a course at the ZHdK. The "official" evaluation is a form where you have to answer how much you agree with a certain number of questions (i think?) such as "the amount of effort required correlates roughly to the amount of credits you get", and "i always felt respected" etc. To contrast this, Joëlle made a Miro board where we could all write our own sentences to a few categories she prepared, such as "what did you enjoy about hosting a session?" or "what did you appreciate in general in the design methods class so far?" etc. This made the difference between qualitative and quantitive evaluation of something a lot more concrete for me. Another takeaway was the brief input from claudio about the difference of Design Fiction and Speculative Design which i found quite interesting, which is that Speculative Design is often more provocative and "out there", and Design Fiction tries to stay more grounded.

One last takeaway was that i forgot my notebook at home, so i was "forced" to take notes on the computer, and it made the process of updating my blog afterwards so much easier, as i usually have to copy my notes from paper to the computer afterwards and often get too lazy and just copy over the most important things. I understand why Joëlle preferrs if we dont take notes on the computer as it is easier to get distracted and is also less ideal for the person/people presenting, but it made it significantly easier for me to elaborate on my blog this week.