Theme 5: Design research

Turn Your Mobile Into the Ball: Rendering Live Football Game Using Vibration

Réhman, Sun, Liu and Li (2008) present a design research about how vibration in a mobile phone can be used to follow the events of a football game. For this research they created a prototype that later could be evaluated through tests with users.

By doing a prototype the authors were able to practically make tests and write about the results, compared to just analyzing theory and empirical evaluations of the subject area. Generally, by creating prototypes you have a possibility to make more practical evaluations and confirm other more “soft” types of data, like theory and data from quantitative and qualitative methods. I believe that if you have the possibility and time to create a prototype it is always useful. More information can be gathered and possible difficulties and problems will also present themselves clearer.

While presenting a design research in a paper it is important to be thorough with descriptions so that the reader easily can follow along with decisions taken and the design from beginning to end. In the paper by Réhman et al. (2008) they brought up a few technical terms that I’m not very familiar with, and I would assume others also had problem grasping every detail, but it is still important to mention these details. By being detailed in the presentation of the design you direct yourself to as many research areas as possible. Some might be interested in the more technical aspects, while others might focus on more humanistic parts of the research. Even though I didn’t understand every single detail in the paper by Réhman et al. I still was able to understand how the prototype worked and how the design research was conducted. If you write this type of paper I argue that this is important. The reader should be able to follow along the design even though the person isn’t educated in every aspect of the concerning research areas.

Designing social videogames for educational uses

I chose the paper Designing social videogames for educational uses, written by González-González & Blanco-Izquierdo (2011), from the journal Computers & Education. The paper is about how to use computer games, more specifically MMORPG, as a method for learning. A lot of theory is brought up, which consist of information about video games, learning, collaborative learning with computers and examples of how different games can be seen as good for teaching various values. The theory is thoroughly described and covers many concerning areas before the presentation of the design and prototype. The theory gives a good background throughout the paper and it almost feels that the authors mention to much general theory in some parts instead of focusing on the specific areas for the design research.

The prototype was created with the help of the game Neverwinter Nights. In this MMORPG, the authors evaluated how users could learn collaboratively by playing together. Different scenarios in the game present the users, students in this case, with scenarios that each contain different tasks that has to be solved collaboratively. The authors used both quantitative and qualitative methods to gather data about how the students worked collaborate in the game environment.

Overall, I argue that the design presentation in the paper is adequate. The authors describe how they thought while designing the prototype and how they implemented and evaluated it. One aspect I would have wanted more description of is concrete information about the game and what the users’ goals were during the evaluation. The authors merely presented general information about games as learning environment and didn’t really focus specifically on the game they used in their research. That made it harder to understand some parts of how the evaluation of their prototype was conducted.

References:

González-González, C., Blanco-Izquierdo, F. (2011). Designing social videogames for educational uses. Computers & Education, Volume 58, Issue 1, January 2012, 250–262.

Réhman, S., Sun, J., Liu, L., & Li, H. (2008). Turn Your Mobile Into the Ball: Rendering Live Football Game Using Vibration. IEEE Transactions on Multimedia, 10(6), 1022-1033.

Theme 4: Post-reflections

Observation combined with content-analysis

For this week I chose a paper that used the qualitative method focus groups. During the seminar this week I realized that the two most common qualitative methods were interviews and, like in my paper, focus groups, which have some mutual parts. The other two most common qualitative methods were observations and content-analysis, which I’m not as familiar with. We discussed some regarding that observations might be challenging because of the participant’s feeling of being watched and thus leading to not performing the tasks in a natural way. This gets better if the observations are done over a long period of time, since the participants get more comfortable, but if you don’t have much time, this could be a problem.

After the seminar I thought more about this and realized that maybe content-analysis could be a solution to that problem, or at least a way around it. When conducting content-analysis, you gather information that the participant already have shared before you are collecting it (Stemler, 2001). This way you wouldn’t interfere the same way you do with having the participant video-taped or watched as in observations. I recognize that content-analysis doesn’t exactly gather the same type of data as observations do. With content-analysis you don’t get to see how the participant was sharing the information, only what the information was. But I believe that you might solve some of the problems with observations by combining this method with content-analysis. Then you would have the benefits of observations and the naturally shared information from the participant through content-analysis, which is harder to get in observations.

Developing new qualitative methods

One thing I found interesting was that there weren’t many other well-known qualitative methods except for those I have mentioned. I think that we should try to develop new types of qualitative methods and maybe think more “outside the box” while doing so, as mentioned in Creative methods –drawing, writing and photography (2012). The methods used now are good, but seem very non-creative. During the seminar it was mentioned that in an art study, the participants got to take or draw pictures of their experience during an experiment. This is creative and would be interesting to analyze. I think that more effort to try to invent new methods would help us gather more interesting and complete data and also cover more areas while doing so. Existing methods could be complemented with new and more creative ones, which could give a new dimension to the gathered data.

References:

Creative methods –drawing, writing and photography (2012). http://www.ucl.ac.uk/public-engagement/research/toolkits/Methods/Creative_methods.pdf. Downloaded 2012-11-20.

Stemler, Steve (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17).

Theme 4: Qualitative methods

Qualitative research paper

I chose the paper Honeymoon with IWBs: A qualitative insight in primary students’ views on instruction with interactive whiteboard by Sad & Özhan (2012), published Computers & Education. The paper presents a qualitative research about interactive white boards (IWB). The authors have studied how the students experience the IWB and how pedagogically beneficial it is.

The qualitative method used is focus groups, in the form of group interviews. During these focus groups the authors asked the students semi-structured questions, which the students could discuss with each other. They divided students in to homogeneous groups, with students of similar background and experience in the same group. They did this so that the discussion would flow better between the students. Sad & Özhan (2012) also pointed out that they had 10 students in every focus group. They argued that this number was small enough to have a good and interactive discussion and large enough to get comprehensive data.

Overall, I think they used the focus group method in a good way. Every decision they made when conducting the method was motivated in the paper. This qualitative method is a good way to gather deep information regarding personal experiences and opinions (Ruben & Chisnell, 2008). Compared to individual interviews, the focus group promotes discussion, which can benefit by giving the participant new ideas and a possibility to share more valuable information.

I think that the decision they made about creating homogeneous group could be questioned. If you don’t randomize the groups, and instead intentionally make them homogeneous, you take away probable disagreements and arguments in the focus group. Like-minded persons have a greater chance of agreeing, which doesn’t encourage argumentative discussions. By conducting the focus groups this way, some interesting discussions and data might be missed out on.

One other questionable aspect they mentioned is that they excluded shy and “less articulate” students from the focus groups. They did this to get a better discussion environment. I think that they have a point in that the discussion could be more giving if the participants are good at talking and expressing themselves. On the other hand, by excluding shy and less articulate students you take away a part of the students that also are affected and could have opinions regarding IWBs. These students might have interesting opinions that are typical for that group and by excluding them there is a possibility that you miss out on valuable data. By intentionally altering the sampling like this you risk having the validity of the research lowered (Cleveland-Innes, 2012).

Comics, Robots, Fashion and Programming: outlining the concept of actDresses

Fernaeus & Jacobsson (2009) write about how robots can be controlled without screens and more specifically discuss how these instead can be controlled by signs, like clothes and symbols. Different signs would make the robot behave differently and make different actions. The paper doesn’t present any specific methods used and is more an overview presentation and discussion about how the use of signs could be designed and implemented.

I liked that the authors had taken into account that we expect the robots to behave in a certain way depending on the sign. I think that the connection they made with fashion and comic is relevant because of the importance of visual representation in those areas and what we associate particular signs with in these areas.

One difficulty with using signs to control robots is that they can mean different things in different countries or cultures. One type of clothing or symbol might make the robot do something which seems natural for one culture, since it correlates to the sign in some way. That same sign might in another culture associate to a completely different thing and thus the action of the robot won’t seem that natural.

Question

Are qualitative methods better (more preferred) than quantitative methods in certain research areas (or the reverse)?

References:

Cleveland-Innes, M. (2012). Lecture, 2012-11-12.

Fernaeus, Y. & Jacobsson, M. (2009). Comics, Robots, Fashion and Programming: outlining the concept of actDresses. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. New York: ACM. 

Rubin & Chisnell (2008). Handbook of Usability Testing.

Sad, S. N. & Özhan, U. (2012). Honeymoon with IWBs: A qualitative insight in primary students’ views on instruction with interactive whiteboard. In Computers & Education.

Theme 3: Post-reflections

Deductive and inductive research

During her lecture, Marta Cleveland-Innes (2012) presented the terms deductive and inductive research, which I hadn’t been acquainted with before and that I found interesting. By doing a deductive research you start from general ideas, like theory and general studies, and work your way toward a more specific part, with more specific data. According to Cleveland-Innes, deductive research is more common for quantitative researchers since they often work with lager sampling sizes and more general data. Inductive research is the opposite, when you start from a more specific point and data and move toward more general data and analysis (SocialResearchMethods.net, 2012). Inductive research is more often used by qualitative researchers, as they use more detailed and specific data and a smaller sampling size.

The image Cleveland-Innes showed at her lecture:

Image reference: https://toknow-11.wikispaces.com/file/view/induction-deduction-1.png/284914014/induction-deduction-1.png

By using a mixed research I guess that it is hard to say if it is deductive or inductive, but it might be beneficial to do the research both ways. That way you work somewhat iteratively and can compare the starting theories from the first method with the end results from the second.

Validity and reliability

Cleveland-Innes (2012) spoke shortly about validity and reliability during her lecture. She stressed that researchers must strive for both of these to be relevant and accurate. I believe that these two terms are often mixed up, and I have at least previously been doing it. I really liked the image she used to present it, which was:

Image reference: http://explorable.com/images/validity-and-reliability.jpg

Reliability is when you get similar results for every time you do a test or research, and when the different people you test on get similar results. Validity is when the results are close to reality and when the methods to gather data have been conducted in a correct and controlled way. One example Cleveland-Innes (2012) mentioned, and that is stated on www.exporable.com (2012), is that researches should have randomly selected sample persons for the research to be valid. If some sample persons drop out or are not randomly chosen, the validity immediately gets lowered.

I believe that there are different obstacles and difficulties with conducting valid and reliable researches depending on if the research is more quantitative or qualitative. In the case of quantitative research it is important to have a big sample size for the data to be valid, while in a qualitative research it is more important to have randomized and reliable persons because of the lower sample size.

References:

Cleveland-Innes, Martha (2012). Lecture, 2012-11-12.

SocialResearchMethods.net (2012). http://www.socialresearchmethods.net/kb/dedind.php.

Explorable.com (2012). http://explorable.com/validity-and-reliability.html.

Theme 3: Quantitative methods

Both texts, Mixed research and online learning: Strategies for improvement by Lowenthal & Leech and Emotional presence, learning and the online learning environment by Cleveland-Innes & Campbell, concern online learning. The former is a presentation of mixed research and how it could be used to improve research regarding online learning. The latter is a research paper about how emotions impact online learning.

Mixed research and online learning: Strategies for improvement

This paper introduces mixed research, which is a research that uses both quantitative and qualitative sampling, methods, data and analysis. One part I found important was that the authors mentioned that for a research to be fully mixed, it has to mix quantitative and qualitative types at multiple steps (research-question, sampling, method and analysis) during the research. If these are only mixed in one or some steps, the research can only be called partially mixed. I found this paper rewarding, since I haven’t heard the term “mixed research” before and any detailed information regarding it. I believe that this type of research should be used in more research areas, since quantitative and qualitative research often complement each other, and together can gather more valuable information.

Emotional presence, learning and the online learning environment

In Emotional presence, learning and the online learning environment a questionnaire with open-ended questions and collections from conference postings were used to collect data regarding emotions in online learning. These methods of collecting data are seen as quantitative, where a questionnaire is a typical quantitative method. However, the authors mention that open-ended questions were used and that answers were collected as qualitative data, which affirms what Lowenthal & Leech (2009) say about open-ended qualitative questions. Because of that the questions were open-ended the collected answers weren’t as strict and directed as they could be, and more usually are in quantitative questionnaires. The authors got 79 completed questionnaires, however with qualitative answers. Therefore, I think it’s hard to say whether this paper leans more towards a quantitative or a qualitative research. Qualitative data doesn’t often come in great numbers (Lowenthal & Leech, 2009; Rubin & Chisnell, 2008), but since they used (quantitative) questionnaires to gather it, they were able to get responses from a lot of people.

I would say that the authors used a quantitative method to gather qualitative data, which they later analyzed through statistical and quantitative analysis. They analyzed by comparing and clustering the answers by using factor analysis, which helped them find similar answers and group them into categories. Since the authors use a mix of a quantitative method, qualitative data and quantitative analysis I believe this research could be called somewhat of a mixed research. It’s not an obvious and rich use of different types of methods and analysis in the paper and it would therefore, in my eyes, be classified only as a partially mixed research, as mentioned by Lowenthal & Leech (2009).

 

One problem with using a statistical and quantitative method to analyze qualitative data is that it is hard to precisely define the meaning of qualitative answers and, as mentioned by the authors regarding the subject, hard to “identify and label subjective and emotional experience in text-based evidence”. Qualitative responses aren’t as easy to compare since every answers could be interpreted differently and because of that the answers often differ. Qualitative data is more often used to get deeper and more detailed information compared to quantitative data, and is usually uncommon for information about general use (Rubin & Chisnell, 2008).

References:

Lowenthal & Leech (2009). Mixed research and online learning: Strategies for improvement . In T. T. Kidd (Ed.), Online education and adult learning: New frontiers for teaching practices. Hershey, PA: IGI Global.

Cleveland-Innes & Campbell (in press). Emotional presence, learning and the online learning environment. The International Review of Research in Open and Distance Learning.

Rubin & Chisnell (2008). Handbook of Usability Testing.

Theme 2: Post-reflections

After this week’s seminar discussion I got more interested in how “theory” should be defined and what could be called theory. After some discussion during the seminar, it seemed problematic to come up with one simple and assenting description of what theory really is.

One definition that was written on KTH Social was “When a theory is accepted by a majority of people, it can be regarded as fact”. I think it’s difficult to use the word fact in this sense. A whole discussion can take place just to argue about what fact is. It’s also vague to say “a majority of people”. How many people would it take? What type of people (experts, amateurs, etc.) does this majority consist of and are expert opinions valued higher than amateur opinions?

Take the evolution theory as one example. It’s certainly regarded as a theory and maybe also as a fact. But it’s hard to know if a majority of people believe in it. There are many religious views on the heritage of life that does not approve evolution. I believe theories differ, similar as opinions and ways of thinking do, depending on our culture and beliefs. Theories are only applicable in specific areas, which could be of different reach depending on the subject area and the universal view of it. As written in http://www.businessdictionary.com/definition/theory.html, the origin of theory “stresses the fact that all theories are mental models of the perceived reality”. These mental models of perceived reality may be different depending on your stance, culture or scientific background.

Another part I found interesting was that Gregor clearly states that no one, of the five theory types, necessarily is more valuable, or of “higher” theory, than another. Gregor also mentions that all types of theory consist of the analysis-theory, which makes me believe that the other theory types are analysis-theory complemented with deeper information. Therefore, I see them as improved and more detailed theory types, and thus in most cases also better. Gregor tells us that each type of theory “provide important and valuable information”, which can be true, but I don’t agree with him that no one can be of higher quality than another. All types of theory are good to implement in a paper, but I have a hard time believing that explanation & prediction theory wouldn’t be better, and more descriptive, than explanation or prediction theory separately.

Theme 2: Theory

1.

Theory should contain abstract pieces that have goals to describe, explain and enhance our understanding of the world and possibly predict outcomes. Gregor mentions in The Nature of Theory in Information Systems that theory can be broken up into seven basic components. These are representation, constructs, relationships, scope, casual explanations, falsifiable (testable) statements and prescriptive (designable) statements. Theory should, in some way, be generalized, even if the theory might focus on more specific areas. Gregor explains that there are four primary goals for theory (of information systems), which are: analysis and description, explanation, prediction and prescription. He later builds on that notion and presents five different types of theory, similar to the four goals, which all theory in this subject can be divided into:

(I) Analysis,

(II) Explanation,

(III) Prediction,

(IV) Explanation and Prediction,

(V) Design and Action.

Analysis presents what is. Explanation: what is, how, why, when and where. Prediction: what is and what will be. Design and action: how to do something. All of these types of theory consist of the analysis part (“what is”). To develop good theory you have to base it on “analysis”. It constitutes the fundamental basis to be able to express the other types of theory.

Examples of what theory is not, as mentioned in What Theory Is Not, are references, data, variables, diagrams and hypotheses. All these can be used as a foundation to develop and confirm theory, but shouldn’t itself be used as theory. Theory should mostly be verbal presentations, comparisons and explanations. These can themselves, though, be based on the implementation of various data, references, variables, diagrams and hypotheses. “Good theory is often representational and verbal”. According to Gregor, theory should be more than just experimental laws through single or few statements. Theory is more comprehensive and should be developed as a system of experimental data, statements and theoretical terms.

2.

I chose the paper Computer Supported Collaborative Learning Using Wirelessly Interconnected Handheld Computers by Zurita and Nussbaum. It is published in the journal Computers & Education. The paper was one of the most cited in this journal, which makes it probable that it is of high quality and also has theory of high quality. The main theories in the paper are collaborative learning and, more specifically, computer supported collaborative learning (CSCL). The authors mention positive effects of CSCL, as it promotes problem-solving, technical learning and social activities. They also present possible difficulties of using CSCL and what to have in mind when using it. Overall, I found the use of theory to be very good.

I would describe the theory in this paper as (IV) explanation and prediction. The authors bring up analysis, describing what CSCL is. Furthermore, they explain how it is used and why conclusions have been made about the subject. The authors also present what can be expected when using CSCL, by arguing this and referring to results in previous research. Thus, combining these presented theory types, the paper could be said to be of “explanation and prediction”-theory. This type of theory is, according to research in Gregor’s text, the most common one.

3.

By using “explanation and prediction”-theory the authors not only present the readers with descriptions of it, as an “analysis”-theory type would, but also give more in-depth presentation of where the theory comes from and what it is grounded in. I believe that an “explanation”-theory is needed for the reader to get deep knowledge of what the paper consists of. “Prediction”-theory is, in my opinion, not equally important, but gives a good groundwork, clarifying that the theory is proven and that it has been, and could be, established through research testing. One possible limitation of the theory in the paper is that it isn’t very generalized, as it doesn’t touch the more general subject of CL nearly as much as CSCL.