Human–computer information retrieval

Transcription

>> Welcome back for our first session of paper presentations. We have two kinds of papers that are going to be presented. The first are 15-minute presentations and we'll have five minutes for questions afterwards. Of course, you can always talk to people during the breaks in poster section. And then after those three presentations we're going to have a series of five-minute presentations that were shorter and there won't really be that much time for questions immediately after each of those. So, hold your questions on those and if we have a little time after that before lunch we can fill a few questions on those. So, first up I'd like to introduce Sofia and I'm not going to try to pronounce your last name. She's going to talk about "Search As You Think and Think As You Search." >> ATHENIKOS: So, my name is Sofia Athenikos and actually the one that I'm presenting today is my dissertation project, to be precise my second dissertation project and it was completed. And so, actually today my dissertation adviser, Dr. Cheryl Lin is also attending. So, I'm going to present and you can ask difficult questions to him. So, we are dividing the labor. So, but anyway before I begin I have to tell you that I have the worst flight of my life last night because I was sitting next to a baby that was crying non-stop for six hours. And then the flight arrived--took off two hours late, of course, it arrived two late so, I hardly slept. And then I came today and I was hoping to eat something when Dr. Coolis, the [INDISTINCT] that the work show was starting in a few minutes. So, I have to give up my precious free breakfast. So, as you can see I'm sick and tired and hungry but I will try to move fast. So, before I begin I really thank the anonymous of reviews who reviews my--our papers and gave helpful comments and that my regret is that I could not actually incorporate the comments into the final version of the paper because let's face it I have only four pages and if I add something I have to take out something so. But hopefully in the journal version of the paper I can elaborate. So, today my outline is introduction, research problem and conceptual framework and semantic knowledge base but of course the focus will be semantic searching interface and information retrieval evaluation and conclusion, acknowledgement and publication. The introduction just in case you are not familiar with this project. To show my objective in this project to us to demonstrate the utility and feasibility and effectiveness over entity and facts retrieval and I'm going to explain that later. And approach is I extracted semantic knowledge used in Wikipedia and based on there I constructed a semantic search into place. And application domain I used the fill in domain as a sample domain over application. And so, the products of these projects are semantic knowledge base and semantic search interface but I won't get into the details of the first parts but only focus on the interface today. And finally the results to the evaluation the effectiveness of--and to the facts extraction and retrieval [INDISTINCT] so, the research problem--actually, it also about backgrounds motivation and research problems. So, my basic point is that you must think about things and make sense with things largely by virtue of clustered by things into different classes. So, when we understand something we already know what kinds of things there are and our knowledge or understanding of the kinds of things already in front of us of the kinds of things that we ask or seek about things. So, in that sense the ontological structure of the world in the human thinking constitutes the basis of the semantic structure of sense making. And a significant kind of information seeking activities concerned with asking about things. We want to find some things that have certain attributes and when we are seek--search for some--those kinds of things we already know what kinds of things we are looking for and because we already know what kinds of things we are looking we only ask relevant questions. So in the sense that ontological structure and semantic structure constitute the basis of information seeking. But the keywords based interface are--actually separates these two process of information seeking and sense making. In the sense that the query string is decomposed into individual keywords and then as a result you get the documents not the entities or things themselves as a result. So, the query and you get the results as the list of documents. So, what I wanted to do was to go to the direct entity retrieval and not just any entity retrieval but what I called type and conditional specified entity retrieval. So, the user can specify type of the entity and also conditions to be satisfied by those entities. And as a result you get a list of our entities directly as a result of your query not just the list of documents. So, the problem is how to enable this kind of entity fact with retrieval effectively that goes away from the words based document-centric and indirect information retrieval toward the meaning based entity-centric and [INDISTINCT] and conceptual framework, so I will also explain very briefly about this part. So, by entity I mean anything of any kind that has certain attributes. Attributes is appropriative on entity and what do I mean by type. Type is a generic class in the ontology of entities under which an entity is classified. And by entity subtype, I mean the most specific clause under which entity is classified. And in my project I represented fact as entity, attribute, value and [INDISTINCT] so, this is a--there is an additional notes field to the standard subjects were they can open models. So, for notes I used to--they can also retrieve the relevant to contextual information about this entity attribute value. Okay. And by semantic condition I mean attribute and value path. So, all the things in my--in this project have been classified according to the ontology. So, the--at the top level there is a thing and then level one we have person, work, organization, blah, blah, blah and of course each of these level one clusters are also classified more so like a concept it gives out what related concept and something else. And it is also further classified, further classified and so on. So, by type I mean the level one class and by subtype I mean the [INDISTINCT] part in the ontology. So, the first part actually this project is a two wing project so one main part is the information extraction parts, knowledge extraction parts and another part is information retrieval and evaluation parts about this effectiveness of interface. So, I will not go into the details of the extraction process and I will just briefly show the results of extraction. So, I will not go into the details of the extraction process and I will just briefly show the results of extraction. So, I used the fill in pages in Wikipedia and other [INDISTINCT] all that's related pages in Wikipedia and [INDISTINCT] was developed. And also one point about this extraction in my project is that I did not use just directly extract information that is in Wikipedia but also a major part was indirectly derived knowledge based on the knowledge already extracted. But these figures actually contain those that have been directly extracted and those that have been either derived or divide. So, there are for example 209,000 entities and there are about 2 million entity-centric facts. And of course, you can also see the numbers of each different types of entities. And so, today's focus is on the semantic search interface. So, this is the interface I created and as you can see there is--there are types for different kinds of search functions and there's also page that has the user instructions. So, I implemented the--a different kind of semantic search functions for entity facts retrieval and general entity retrieval query, this it the main function that I aimed at demonstrating. And by using this--the user can retrieve the entities that specifically match the entity types, subtypes and conditions, semantic conditions that is attribute-value pairs that the user entered. But you--the user can also use a specific entity-centric query to retrieve a fact about a specific entity. And there is an entity commonality finder query where the user can find commonalities between two specified entities of the same type and subtype. And there is a direct relation finder query, the user can use this function to find relations between two entities regardless of there type and subtype. And also there is an indirect relation finder query, so the user can all find indirect relations between two specified entities that are mediated by at this entity. And finally there is browsing function category-based entity browsing, so the user can browse or fill in entities according to the hierarchal categories structure. So, I will focus on the general entity retrieval query, so how the process works. In the main menu the user selects Search Type and choose a General Entity Repair Query and then the first thing that the user does is to select Entity Type. Once the user selects Entity Type the next step is to select the Subtype on the given type. Once the user does that then the next thing to do is to Specify an Attribute and once you specify the Attribute then you need to add value and of course this process of specifying attributes and value--specifying this condition can be repeated. So, once the user is finished with that--this process then the user can submit query and then the user will get the query results. And the query results that's not just consist of a list of entity names but it also has a relevant contextual information. Furthermore, the user can also click on any entity name highlighted, not just those in the direct answers but also in the contextual information to get the information about the selected entity and of course this can also be repeated. This is the actual interface. So, let's say with general entity retrieval query, you can retrieve all entities that directly [INDISTINCT] so user clicks the button and this is what the user sees. The user does not see the whole point like but only sees the first relevant options, so the user needs to select an Entity Type [INDISTINCT] use and clicks it and then the user sees the list of Entity Types and user chooses one. Then what happens is that now the user is given the menu for the Entity Subtype Selection and the Entity Subtype Selection Menu only contains these relevant options according to the user selection of the--in the previous step. So, you will not see like some Subtypes that are relevant to Person Type Entity when you chose--when you have chosen the Concept Type Entity. And in case of person actually that uses the interface simplifies the selections so that the user can just say any. Once the user does that then you get the--so menu for the attribute selection. And again the menu only contains these attributes that are relevant to the given, chosen Entity Type and Subtype. So, once the user chooses that and in this case directed film is the attribute because there are so many films in the database. So, actually the input box first appears so that the user can type something to get the--just--but in case there are only a few values only like a few hundred values then the menu directly appears. And so, we are in this case the user is seeking to find the films that have won awards and the user wants to specify what awards it was. So, the user selects a category for best picture and then as soon as the user specifies at least one condition then the user can submit query or at this step the user can have another condition to this that is why the entities or the user can remove the last condition and specify--two minutes left? >> Two minutes left. >> ATHENIKOS: Okay. So, this is what I tell you. These are--see [INDISTINCT] anyway. So, evaluation actually I wanted to go into the Evaluation. So, the Evaluation Map that I used a comparison between Interne Movie Database and my interface for comparison and why did I use the Internet Movie Database because of the domain orientation of the project. So, all subjects had to use both interfaces and we compare the results. So, the design is that the user--well, the user was divided into six groups so they could start with my interface o IMDb first or also the questions were presented in a different orders. Okay. Some of the questions I will show you. The questions asked were like entities and so who played all of these roles but that's not the whole question. The whole question is, once we find that, give me the name of the actor and the titles of the films and so and so and so. So the user have to find not only the entities but also related entities or some related information. And in this--in this case it was--yeah, in this case also like this. So, that's why I used--when I calculate it precision I recall it was said to be based on the a weighted correct score because the user have to find of course the entity names but also some contextual of information. So, the effectiveness was measured by precision and recall based on the weighted correctness score. And the hypothesis was that the first subject ever precision I recalled will generally be higher but I was not sure whether it will be always higher. But for group precision and recall I thought it must be higher and it has been confirmed. I have no time for this. I will just show the main--where is it. So this is the main result. So the user was given ten questions and they did worked on five questions on IMDb and five questions in PanAnthropon and the results is the average precision recall was substantially higher on the PanAnthropon Interface. And first subject to average precision greater than 90% were--there were several in the group and as for all the subjects that are per subject have a average precision recall as higher for the PanAnthropon Interface versus IMDb Interface. And this is of course a tough questional responses so the--only one person said that IMDb was effective well in contrast almost all subjects graded at PanAnthropon was effective also they said that they could easily understand and use PanAnthropon and further more they were also interested in using similar interfaces. These are relative effectiveness. They all agreed that the PanAnthropon was more effective. And why they thought that the PanAnthropon was effective? The user liked the fact--the subjects like the fact that they don't need to guess the right keywords to use and they also like the step by step search process and they felt that actually they could easily understand the entity types of PanAnthropon. And they like the fact that they can actually search for specific things precisely by specified multiple conditions and so and so. >> [INDISTINCT] five minutes. >> ATHENIKOS: Yes, this a conclusion. So actually the evaluation results actually show that the interface was effective at least for the subjects and because I have to skip the parts where--actually the subjects were general subjects. They were mostly undergraduate students but they did not ask actually major in information science or a--so--but they were various groups like a biological, engineering, biology, mechanical engineering and so on. And future work I'd like to apply this approach to other domains and learning. Yeah, this research has been partially supported by the 2011 Eugene Garfield Doctoral Dissertation Fellowship. >> So I think we might have time for maybe [INDISTINCT] >> ATEHNIKOS: So sorry. >> [INDISTINCT] >> ATEHNIKOS: Yeah, I mean there was--I did not focus on actually devising--the focus of--actually they came from I device of questions but my focus was to see the kinds of queries that that can in principle be answered because IMDb has all the information about all the films. So the questions are in principle answered by IMDb but it is a hardly actually practicable when you--when you do this so I wanted to show that. So I tried to be non-biased but still I have to show the difference otherwise there is no point of creating a new kind of interface if we--it is--I just as [INDISTINCT] >> Okay. Thank you very much. >> ATEHNIKOS: Thank you. >> Next up, we have Chang Liu from Rutgers University and she's presenting a paper called "Exploring the Effect of Task Difficulty and Domain Knowledge on Dwell Times" and [INDISTINCT] work [INDISTINCT] >> LIU: Good morning. I'm Chang Liu from Rutgers University. So I'm presenting this work on behalf of our Prudo Project. This our--part of our extraordinary study on this issue. So the topic of our study is "Exploring the Effect of Task Difficulty and Domain Knowledge on Dwell Times." I know all these variable names are very familiar to our audience and of--so we are talking about dwell and domain knowledge and we are talking about task difficulty and we are also talking about dwell times. Many researcher have done this but of--I hope--we hope our research will add something new to what we already know and maybe provide some unexpected results. So let's go first to briefly go through what we already know. With respect to the task difficulty that we--some recent studies have found that users tend to issue more diverse queries and spend longer time on search result pages and use advance operators more and spend larger proportion of task time on search result pages when they are working difficult tasks. And here another study that we have ranked the examing task type and task difficulty together. So in that study we found, when user were searching for single part of finding task in difficult task they are--they tend to have longer total dwell time and longer forced dwell time on unique content pages. But there is no significant difference on the dwell time on search result pages. Well, on the other type of task especially when user are searching for multiple facts of general information gathering we have--we found the other pattern. So users tend to have longer dwell time on search result pages but there was no significant difference between dwell time on search of content pages in difficult and easy task. So our current study of--we will ask participants to search multiple information and general information gathering. We will compare these results later. This respect to domain knowledge in fact we have a lot of research and something related to what we are doing here is that users with low domain knowledge they have less efficient query terms and less efficient query tactics. And of Kelly and Kul also found that when searchers' topic familiarity increase their reading time decrease that means their dwell time on content pages decrease and their search efficacy is increased defined as the ratio of number of documents saved to the number of document they viewed in the task. And of recent study by a group of Microsoft researchers they found that it depends in different domains. So in medical and law domain they found experts had longer average page display time than non-experts. I think the display time here is very related to the dwell time on content pages. And in Finance and Computer Science domain they found the other type expert had shorter their average page display than non experts. In our study, we will have participants from medical domain so we can also compare our results with their results in medical domain. Okay. So, in order to address this question we have, we conducted user study in which we recruit 40 students from medical domain and when they first come to our experiment, we gave them a background questionnaire in which we collect their background information, computer experience, search experience. More important, we evaluate their domain knowledge through the--their briefing on their knowledge of selected matched terms. I will talk about that briefly, later. And then we give a demo of training Pascal to use our system after that we give them four search task and during each task, they are given up to 50 minutes to search and can save useful document that were related to this task. After that they have exit questionnaire and all their interactions which computers were logged by our logging system. So, this is the way that we measured their domain knowledge, we give them tree, matched tree--matched trees that's total--in total to 400 [INDISTINCT] terms associated with the topic [INDISTINCT] we will ask them to search our task so, these topics are like genetic structures, genetic processes, genetic phenomena. These are--so, for each term, we ask them to evaluate how much you know about this term from one to five that is from no knowledge to five that I can explain to others. And then we add all their ratings together that is the top. So we add all their ratings and whether some of them will not rate it. So, if they have rate that term, we add them together and then divide it by this number that is to normalize their ratings by an [INDISTINCT] the expertise who will rate all the terms as five. So, that five plus the number of terms, they rated in that--in that questionnaire and then participants were divided into two groups by the median of that's 0.4 into high domain knowledge group and low domain knowledge group. And also we controlled their task difficulty. The way that we controlled their task difficulty is defined so, hard task was defined as the number of, let me put it this way, so, hard task means there are very few relevant documents of returned of when we have their topic as a query issued into the system as preceded in nine to ten. So, you know, our task design, we controlled the task type and then in this figure shows that users rating about task difficulties and that's to confirm our design of task difficulty. So, you can see that among the five tasks we have, topic seven and topic four have their lower rating than the others. So, we--so, we have two groups of task of which is easy task and difficult task. Now, we look at the results. First, we look at the--this factors on the dwell time of on content pages and since users are searching for content page to accomplish this task and they will save some documents that were related to, that were useful to their task. So, we also considered document useful into--in this model and documents usefulness were defined as whether this page has been saved by the user. And this is the result from general model and we can see that only usefulness has main exact on their dwell time on content pages. And we also find there is an interaction [INDISTINCT] of task difficulty and domain knowledge. So, let's look at them one by one, first, surprisingly, users spend much longer time on viewed pages than on saved pages that is very quite different with what we already know. Some study have found that most of study I have say that if there's a relationship, users tend to spend longer time on useful pages than non-useful pages. They're not significant or relationship but here we found that different pattern. So, users spend longer time on non-useful pages than useful pages. And then let's look at the interaction effect, I think this is more surprising. So, the red line stands for the low domain knowledge, the blue line, high domain knowledge. So, we can see that in easy task actually users with high domain knowledge spend much longer time on content page than users with low domain knowledge. And there is not significant difference when they are working for difficult task. And in particular, if we compare dwell time on content page in difficult task, which were in easy task only focusing on users with high domain knowledge they themselves spend longer time in difficult task or easy task than in difficult task. So, after that, we continue to examine the effect of these two factors on their dwell time on search result pages. And here we only found that task difficulty and domain knowledge level have many effect here and there's no interaction effect so, if you [INDISTINCT] this figured these two lines are parallel and so for all users, when they are working for high difficult task, their dwell time on search result pages is longer than easy task. And for users with high domain knowledge, they spend shorter dwell time on search result pages than users with low domain knowledge. And we put these together so that we can compare of their dwell time on search result pages and their dwell time on content pages. So, I think if--when we look at this paper, the dot lines, represent the dwell time on surfs and the solid lines stands for their dwell time on content pages. So, if we look at only users with low domain knowledge, two lines were in red. We can see that in difficult task, users with low domain knowledge, they spend longer dwell time on content, both content page and search result pages. And then if we look at users with high domain knowledge, the blue line here we see the different pattern that the easy task, users with high domain knowledge spend much longer dwell time on content pages than on search result pages. But in difficult task, they spend longer time on search result pages than on content pages. So, that is our finding and now, let's discuss so, one mostly unexpected result is why users with high domain knowledge spend much longer time on content pages in easy task than in difficult task. We don't know the exact answer and one of the explanation could be the way we define task difficulty, so hard task were defined as task where search system returned few relevant documents using the task description and the query, so in the search result pages in difficult task they have very few relevant documents showed there. So it is possible that users with high domain knowledge, they already made the document usefulness judgment on the search result pages and then when get--go through in that particular content page they just go and save the page that is one of--one explanation but for users with low domain knowledge they still need some time to read the documents and then make the final decision. So, then we can compare our survey with previous study for some task difficulty so, in previous study that we did we found out there's no significant difference on dwell time of all content pages. So, maybe our study could help explain that maybe the [INDISTINCT] don't know that domain knowledge, it has an interaction in fact to its task difficulty. So, they did not find significant difference between easy and the difficult task but there might be some difference between people with high and low domain knowledge and with respect to the dwell time on search result pages, both these studies found users spend longer time on search result pages in difficult task than easy task and we also confirmed that. So, users tend to stay longer time on search result pages when they are searching for difficult task. So, compared with dwell time on content pages maybe dwell time on searching result page could be a better indicator of task difficulty and also domain knowledge and we compared that with the results in domain knowledge in fact actually our results were different and we are not quite sure maybe the task with difficulty has some interaction factor here, so here comes to our conclusion, that is users spend longer time on search result pages in difficult task then in easy task and users with low domain knowledge spend longer time on searching result pages than users with high domain knowledge and dwell time on content page has shows an interaction effect by the task difficulty and domain knowledge and also I have to mention that it is possible that users spend longer time in non-useful pages than useful pages, so we found that our study the result from our study may not be generalized it may depends on our design of task and the design of our system but this is what we found and we would like, to just [INDISTINCT] audience here we got very good reviews here and we would like, to continue to do this and especially examine the relationship between the dwell time on content page with the dwell time on search result page to fully understand how user search in the--with difficult task. Thank you very much. >> That was almost perfect [INDISTINCT] so we got time for some questions. >> [INDISTINCT] >> LIU: So you mean the total time of day worked on the task may affect their search behaviors, yeah, you have up to 15 minutes to search and actually this task were very difficult and in our data collection we found most of users use all the times, so 15 minutes and maybe at the end they were little bit--a little bit hurry and they want to save more and maybe that could be an effect here, thank you. >> [INDISTINCT] >> [INDISTINCT] >> LIU: Their domain knowledge--search skill we also collected information about that but I don't think we have analyzed. Do you have any, Michael? We have the data but we have not organized, thank you for the suggestion I think we, we will try to see if that has an effect. >> [INDISTINCT] >> [INDISTINCT] thank you. >> Just a suggestion that you might actually look at bearing the quality of the [INDISTINCT] as a way forcing the effectiveness of the result pages as a [INDISTINCT] of your experiment >> LIU: Uh-huh. You mean that there such result for--on the result page, right, yeah. We evaluate their search performance but the search performance is only focusing on whether they save that page but I think your question about their overall quality of the result page right? >> Specifically [INDISTINCT] >> Uh-uh. >> The [INDISTINCT] >> LIU: Yeah, so this is the content page... >> Yeah. >> LIU: They look at the abstract and this is the search result page and we have that the title alter and the name of journal so, the only information that they can get more from content page is the abstract. >> [INDISTINCT] >> [INDISTINCT] >> LIU: They just saved random document. >> [INDISTINCT] >> LIU: That's an interesting question but in fact our task were--our task were boring actually so, if you are interested I have task examples and this is easy task and this is difficult task these are from Track 2004 and I have no idea what they are about. So, they are really domain limited and very--we only recruit student from medical domain so they can understand this and they are not searching for fun they are searching for literature. >> So, this a great discussion here. I hate to cut it off but we do need to move on. So, I encourage you all to talk to [INDISTINCT] well, thank you. Next up is Jingjing Liu who is from Southern Connecticut State University and also I guess they're both partners. And she's going to talk to us about "Knowledge Examination in Multi-Session Tasks." >> LIU: Okay. Hi everyone. Yes. As we just heard knowledge did have some impact in a user search. And now here, we have another paper about it. This was user [INDISTINCT] distribution of research data but it very different analysis from what I did before. Harry, there's some mismatch here. Anyhow. In this study actually we look at how the users self assess the knowledge in a multi-session task and both before and after they work with the task and both before and after they work with each sub-topic of this multi-session task. And we just look at, you know, the relationship between the tasks--the knowledge, these kinds of knowledge that I just mentioned and see what is the pattern of them. And apparently if the users knowledge of the general task and their knowledge of the sub-task topic. Okay. They did have different values and also they have different patterns of change. And some attributes of the task knowledge but not all of them varied across the task. Just an overview. Define knowledge. We know that knowledge is obviously unavoidable aspects when we look at, when we talk about information search. And previous study have found that including the one that we did here. Okay. The users knowledge, they did affect users search behavior and their search performance also. From the starting point they choose to use, they choose the search terms, query terms, how long the query is. And to reading the results page and the even the procedure and the recall and their, you know, correctly able to answer the questions of the task. So, it's quite obvious that knowledge affect the user's behavior and their performance. And in terms of the knowledge elicitation or assessments, there have been several and different kinds of method used in previous studies. For example, according to different stage of taking a course or taking or in a program, you know the earliest stage probably corresponds to a lower knowledge and then later is corresponds with the higher knowledge or you know a person inside or outside of a domain corresponds to the novice and experts. And even they're answers to some kinds of questions in the past. Some other methods used are like including reading the terms in a Thesaurus or just simply to ask the users to self judge how familiar are you with the topic? Okay. And previous studies found that the fourth and fifth are correlated with each other. And this study that I'm talking about we used the fifth method. Okay. Now, we know that knowledge is important in affecting user's behavior. So, but we don't know that how knowledge is changed actually in the search process. And this is why we had this research also because multi-session task are often seen in the people's life and also it is a very pretty handy way--handy--convenient way to assess users knowledge gain throughout the process and we have to include research. And specific questions, we tried to answer include that, what is the pattern of users general knowledge--knowledge of the general task? And what is the pattern of users knowledge of the sub-topic of the task? And are there any differences between these two? And how are users--how different the task of type would affect user's knowledge? And also how does users knowledge change across and within the different session? Okay. First I mention this was--this data I said was coming from my dissertation research. So, those who know about it who I know [INDISTINCT] please be patient when I repeat it. Okay. So, this study we had 24 participants from undergrad journalist media study and they were paid and we [INDISTINCT] this to encouraged them to have to have a serious manner encouraging them while their working on the task. Okay. Half of them worked with the dependent task and the other worked with the parallel task. Just to explain for them what are these two sub-topics. They say that the task was that each participant was asked to write a three section article in three sections and each section with one topic. Okay. In a parallel task they were asked to focus on the three topics of the Honda Civic, Toyota Camry and the Nissan Altima. Based on other hand in a dependent task the three sub-topics were collecting from information by hybrid cars and then select three models that mainly focus on an article and they just compare and the pros and the cons of each of the model. There you see the different structure here. Why it's parallel and the other is dependent. Okay. I have click some here but I cannot see. We are now back to the study design. As I said that this was a multi-session study. Each user actually came three times. Those three sessions, each session work with one topic of the three. And before and after each session, they were asked to complete a questionnaire. And each questionnaire they were elicited--they were asked to self rate their familiarity with the topic of the general task and their familiarity with the topic of the sub-task based on the seven point [INDISTINCT] field, one is not at all and seven is extremely. So, here is how our data was elicited from, okay. As a matter of fact this study was actually a system version. So, there were two systems version involved one is called a firm suggestion. This is how it looks like, you see that on the left side there were some terms which suggest--which suggest to the users. Another version which is the non-firm suggestion it was this the blank IU window just the right side in a post side. In the study of the [INDISTINCT] actually--and based on the interview afterwards was the presentation barely used to those terms. So, we think that this system if not does not affect to what are--what are our kind of research is like and I'm going to talk more about. Okay. Now let's go to [INDISTINCT] here is the result. I'll show you in a graph about user's knowledge of the General Task, before and after each session for all three sessions. Now, we see that [INDISTINCT] side to inquiries inside of each session that we know that from the study--from the results actually in session three the difference was not significant. So, it seems like in the previous two session users gained knowledge on the general task topic but in the third session that they kind of reach the operator and they didn't--they didn't increase significantly and when we look at the between session and comparison, we see that for the pre-task and the post task it seems like both increase across sessions and the post-hoc analysis from that. The difference was the transition one and session three. The session two actually didn't significantly higher than session one but session three was. So, here is this [INDISTINCT] now, we look at the knowledge of the general task again that into different tasks one is in parallel and the other is in dependent. Now, we see that actually for the [INDISTINCT] session in the dependent task actually users knowledge gained in all three sessions. But in the parallel task which is below there in session one and session three users knowledge didn't gain but in session two we gained. So, one possibility is that, actually we look at the results. For the parallel task in session one users had a pretty high base line--base line knowledge. So, that's why they didn't seem to gain more knowledge in--after their session. And session three of the parallel task the pattern was the same as--as where we set--excuse me, as what we saw just now for the [INDISTINCT] okay, when we look at the between session scene okay, for the dependent task actually the pre-task and--the pre-task pattern propose the dependent task and the parallel task was the same pattern as what we saw just now in this side but when we look at the post session general task knowledge we see something different. So, for the dependent task actually users knowledge after the sessions, the increase across each session but in the parallel task they didn't seem to gain across any session there were no difference there. Okay, so, we did further analysis using the general model and just to have the overview of the task and the session impact. Now, we see from these results the session impact was the accordance of what we just saw in the previous two sides. And about the task impact that was not seen in the previous slide actually we saw that. The--for the pre-task knowledge a dependent task users had lower knowledge than the parallel task. Other than--after that the task the post--the session task be to reach the same. It seems to correspond with what we just saw I explain it for more later. Okay. So, now let's look at the sub-task knowledge. So, now we see that it's quite the difference from the general task knowledge and because the sub task topic they are different than--different at each time. So, the previous--the pre-task, pre-session knowledge actually they didn't seem to change at all and because they are different. But post knowledge when we see that the session what--the session wait they were a significantly increased. Possibly because the users when they had--when they get in the third session they already got some knowledge about the general task. Because they were higher in the topic of the sub topic as well. This is the same analysis in two tasks. We see that the within session the inquiries all the time and between session no difference at all. Okay. And again this is a general model analysis and we see that such an impact in was--the same as what we saw just now, and task impact no--not at all. So, pre and post the both task are the same. Now, here are some of the [INDISTINCT] first about general and the subtask, we see that they kind of different pattern. The general task knowledge users seem to increase within sessions acceptable from exceptions when they seems to be [INDISTINCT] but the--and the they seem to increase the sessions also. But for the subtask knowledge they seem to inquiries within session but not in subsequent sessions. And the--another thing that when we look at the parallel on dependent task we see that--we saw that for the general task knowledge. Dependent task users have lower knowledge than the parallel task. Probably because the parallel task seems easier side in the process okay, users bit seem to gain knowledge in the parallel task that they really gain the knowledge in the dependent task and until the end of the task they reach the same level. So, now so--what do these all suggest? We think that first, we know that--that there are two different kinds of knowledge in such a complex study or complex task. We want to be sure what we are marrying and also we see sometimes uses screw to get a petal or like [indistinct] impact when they have a pretty high knowledge already [INDISTINCT] work with another session we're not supposed to significantly increase their knowledge. And the study design we try to avoid [INDISTINCT] also we see the difference in the parallel and independent task. The parallel task is kind of like easy when you rate the task that you didn't gain knowledge in the process. Difficult at the dependent task was on the different may seem very difficult in the beginning but users actually gain knowledge. So in the end we kind of used the same, so this seem to indicate that what system want to do to help users. If they have high enough knowledge already, what can the system do to help them gain more knowledge even, and also how to help them to solve the task. So this is our [INDISTINCT] and the thought and we also are open to all other comments and additional or are additionally. Okay. Thank you. >> Okay. We have to [INDISTINCT] >> LIU: Okay. as I mentioned actually there are different ways to [INDISTINCT] different ways to measure and assess knowledge and the one that we are using was used to previously other studies, this is one thing that we also you know use to the same method and also second thing is that, this matter to us found it to be correlated with users [INDISTINCT] of their familiarity with some Thesaurus terms. We think that's a kind of not a purely objective, but, it's more objective than just the self rate my knowledge, we think this a somehow reliable way to test the users knowledge. Because the Thesaurus term we know that's not average or may I have a thesaurus, for some domain like an... >> [INDISTINCT] >> LIU: ...yeah, I hope we were able to find the thesaurus for the hybrid cars, and it has--but, obviously is that I like your suggestion and in further studies we could also look as I know, use this kind of knowledge change with the other kinds of method to asses their knowledge. >> [INDISTINCT] >> LIU: That was another study actually also out of the Rutgers Group. I'm using then study you just the heard by Chang Liu. We actually the study we ask the users the medical participants to come to rate their familiarity with the Thesaurus terms or think like by medical or--what's that genomic... >> Genomics. >> LIU: Genomics, of the--they came from mash. The medical subject [INDISTINCT] with the structure--the tree structure and we ask the users to self-asses their familiarity with those terms, for each of the terms. And we also in--on the other hand, we ask them to self rate their familiarity with the sub topic just that the as you see from the previous paper. So we got both method, and we look at the correlation between the two rating, and we found the high--very high correlation, probably 80 or 90 percent, yeah. >> [INDISTINCT] questions. Okay. So next up we're going to move into our lightning round of five-minute presentation. And we won't have time for questions after each of these, might have time for a couple of questions that we can right before lunch. The first up is Mark Smucker from the University of Waterloo. He's going to talk about An Analysis of User Strategies for Examining and Processing Ranked List of Documents. >> SMUCKER: Yeah, I guess I'm trying to put everything into the title which is appropriate for a five minute talk. Yes, so I'm Mark Smucker from the University of Waterloo. And when we have users and they come in and they do a search for the query. Then they're presented with their results and now they face of the task of examining and processing those results and this is what I'm focusing on here today. I'm interested in questions about, whether did they spend their time, similar to the records work, okay, so, whether do they spend their time on the summaries. Whether did they spend on the documents and also, with what sort of accuracies do they operate on clicking on those summaries and also the judging of those documents. Now to do this I actually had previously a 48 parts on user study and the industry was their study, we asked the users to search for relevant documents and save them if they're relevant, okay. They were given prefix lists of relevant documents, okay. So these were already manufactured for them and they were to search for them, they had 10 minutes for each searched topic that they worked on. We had a very basic user interface instructions at the top search topic on the right. There you go 10 blue links or so and query by summaries you can click on them, you get to view the full document and the documents would have query terms highlighted in them for the users, the user could then go ahead and if they wanted to, say oh yes, save it as a relevant document, this is not a required judgment, but, it is--that's how they do would do it. Then when their done with the page, simple back button to go back to the results continue searching for relevant documents and when they--if they wanted to they can certainly go 10 more results--obtain more results they could keep going for as long as they wanted to. Now, to look at this behavior, we used K means cluster analysis and we again are focusing on both time and accuracy, but, here instead of notions of just accuracy, we're actually taking signal detection, theory measures very commonly known as true positive rate and false positive rate. So we're looking at this at the--for the summaries, the accuracy there and also the times spent and again the accuracy in time on the documents. And that's--those are the variables on which we were going to do our clustering all together. So, now these are overall results for the 48 participants, this is not the cluster house this is to give you a taste for what you see oh, let's just take the whole population and compare it to the cluster group. And basically, on the summaries we are going to see, oh, they seemed to be what we would call fast and liberal, they only take about nine seconds to--before they click on a summary, and by liberal what we mean is if in doubt they are going to click on the summaries, they're operating in the manner which is very similar to, I don't want a miss a relevant document. So I'm willing to click on non-relevant documents to avoid missing the relevant documents. Now they are distinguishing between relevant and non-relevant documents here, so the false positive rate of 62%, so it's a non-relevant summary, 62% of the time they're going to click on that thing. For the relevant ones, 82% of the time you click on them. So, they are distinguishing but they're very liberal and they're biased towards where they're going to click on. For the documents, they also seemed to be quite fast and they're only spending on average 26 seconds before they leave that page or made a decision to save it. But now their decisions are quite neutral. The true positive rate and false positive rate are balanced. Now, we're going to cluster them into three groups. The paper has other--has a clustering by two groups and the paper has lots more details than this. I encourage you to take a look at it or come talk to me. Now, the best performing group that quite outdoes the other groups combines the strategy of being very fast and liberal on the summaries with actually take--being quite faster but neutral again on the documents. So, six seconds on the summaries, twenty seconds on the documents and they're really through and through in finding relevant documents the fastest. There are two slower performing groups and they basically adopt the opposite strategies. So the better throughout the two slower performing groups actually goes ahead and has a slow evaluation of the summaries. They're taking 15 seconds in comparison to this six seconds but now, we also--besides just seeing them as being slower, they're actually neutral in their decisions. Okay. They're actually balancing the--their true positive rate and the false positive rate and that's very interesting but when they get to the documents, they were able to go very quickly. The other group is--they're very fast on the summaries but then when they get to the documents, they take 40 seconds, okay, approximately. So, we really, between these two slower performing groups, see them basically allocating almost sort of the same amount of time about 45 seconds for a summary and document total evaluation. But how they spend their time is quite different. Okay. One group says, I'm going to go fast on the summaries but then I have to spend quite a bit of time on the documents, and the other group says, no, I'm going to spend quite a bit of time before I make my decision on which summary to click upon but when I get to the document I'm going be able to make that decision very quickly. And with that I look forward to your questions later because we're taking enough at the end of all of these talks. >> Perfect. Five minutes on the [INDISTINCT]. He told me it would be five minutes on [INDISTINCT]. All right. Next up is Jim Kim which supposed to be this [INDISTINCT] Smith. He's talking about the guide reading and self-browsing model plus assimilation. >> KIM: hello everyone. I'm Jim Kim from UMS-EMERST. My group is not typically associated with HCIR research but here I am. So, since our talk is about how we can improve our access to own documents, our own personal documents, I'll start by this simple question. What do you remember about your document? So, we usually--we usually don't have very good memory for documents and in that case our memories did recover this way. And if you can--you get lucky if you can come up with couple of keywords by which you can use keyword-search interface to find the documents, right? But unless you are just lucky, you'll not be able to find it but you may still remember some other information which is association with this document and other documents. And in that case, and if the system supports some sort of browsing between documents to other document, believe it, you can use this--to start the interaction by keyword search and then move to the target document by browsing, so, this is the interaction model that we're proposing and we used something like applying similar kind of interface to--regarding with this interface--interaction method. But I've read some research study in other work which I will present next week in CIK conference but today's talk should--is focusing on--so, given that interaction model, how we can evaluate that by simulation? And I think the simulation is important because it allows you to test your method with lots of variations in system configuration and you either use your parameter. And for simulation, we first proposed some sort of Probabilistic User Model which is composed of query generation and the state transition between search and browsing and link selection for browsing. So, just to--so our model starts by search and then user can click on the list of the research if they want to browse and then they can finish interaction by finding the target document. So, I give you some more repaired interaction model. So we start by taking this target document and our Probabilistic User Model takes the term, in this case, this change registration and use it for--use it to initiate the search procedure. And the search is not relevant at all. We formulate the query. But if the search looks reasonably relevant which you can't have because we have the--you can use the rank position of the tar--original target documents then it initiates a browsing by clicking one of the popped wizards and in any case, if you can't rank the target document top 10, it finishes the interaction. So, this is the interaction model of Probabilistic User Model and the point ended--hypothesis--this is re-evaluated using this simulation technique. It's a two big parameters of user. The first thing is user's browsing behavior which we divide into panels and the [INDISTINCT] approach. So, if the user is browsing the type panel--what I mean, whether user click one versus more than one item per research and this [INDISTINCT] is different by--to all in which they are browsed. So this is the behavioral part of the user model and as for the knowledge part, we had three different levels of knowledge which is Random, Inform and the Oracle and--since we have the rank position of the target document, we can--you can--we can model the user's knowledge of the--this [INDISTINCT] and finding scenario. So I don't have time for evaluation--presentation of [INDISTINCT] all over with our show that compared to the other usual study that we performed, we found that browsing is of use and is successful almost at the same rate as usual the study--research that we did. And the other thing that we found is when we looked at the successful show of the browsing we found some interaction between user's behavior and user's level of knowledge. All right. I can tell you more about this when you come to my poster later. But the summary is that we proposed this also say Browsing Motor and Evaluation File Simulation. And I just want to make a brief comment about this ongoing work. So, you can see that my previous work sort of like very simplified version but we would want incorporate human's memory model to--actually more realistically simulate user's interaction with the system. Thanks. Yeah. >> Okay. Thank you. All right. So, next up, is--let's talk about some more--you know I did [INDISTINCT] >> Quick change here. If they can do this. Okay. All right. So, like Rob said, this is the work that we did. Try to understand the stages of a search that people go through during a exploratory search session. That line sort of a standard talk format. This was specifically done in when we were looking at--we were studying fast into the library catalogs. And we've done couple of studies on this and the typical thing is we give, you know, the--it's a sort of standard lab study where we give a--we give an exploratory search task. We happen to go run through about six of those. We've been using eye tracking and--yeah, and using that to collect data. In this particular case, what we were interested in understanding is are there particular patterns--what's the overall distribution of time they spent in the various search--stages of their search and are there observable patterns? And now come back to what the stages are that we were looking at in just a minute. Just a little bit background, the study was a 18 subjects undergrads. They were--we're having them do--use exploratory searches where they were sort of grounded and writing an academic paper. Early stage research going to the library, using the library catalog to find--to find information. They happen to do six of those tasks and then what we did is we're recording the gaze data. And then at the end of the session, we actually replayed the last two of their stages--of their searches. And we hade--but we did a little retrospect with interviewed where we--we replayed the search--the video of those searches and half speed and we overlaid the gaze data so we could actually see what they're looking at slow motion. And then we ask them to do is report at each--every 10 seconds or so report what stage of the search they were in. And we captured that data. The full details of the study earned are Jay's paper. His example of the exploratory task. The--no, we used five stages and the reason we did this is there's number of different models of stages of search sessions. Actually in his model is the one that sort of drawn a lot. We had this collapse set down to something that was more useable for the end users, for the searchers because they probably didn't know--it was a lot of stages with--it was complex and then the other is that they don't--the labels and the [INDISTINCT] the stages and the models are not necessarily, gradually understandable to the end users. And so what we ended up doing is trying to collapse them and come up with more meaningful or meaningful stages, and these were the five stages. Coming up with query terms, getting an overview, extracting, deciding what to do next in their search and then deciding on the topic which was the nominal search task. So when we did that and then we physically summed up the amount of time they've spent in each stage of that task over the course of the search. You can see that in the extracting stage, they've spent significantly more time, turned out to be about 40% of their time was spent extracting and then the other was--time was spent roughly [INDISTINCT] among the other stages. And now what we did--see if I can get this to work in one minute. Just look at--what we tried to do is visualized each stage, each search in the sequence of stages that [INDISTINCT] through and present that in some visually way that gives you some way to look at an overview and drill down into the details. So, for example if we look at--zoom out a little bit. You can start to see an overview of all the--there's about 30--I think it's about 36 sessions total that [INDISTINCT] maybe three or so. And you can see how the patterns--well, some of the search--searches were no longer or shorter. We saw some interesting patterns of--you know most of them started with the query term, not surprisingly. And then--but some of the--some of the searches went, you know, through this query terms overview--query terms overview deciding on a topic, query terms overview and as you can see these different patterns of how their search progress, some of them were very lower and were deliberate perhaps. Others spent more time on the extracting phase. So that--those are sort of somebody interesting patterns that we saw and I'll stop there but there's some interesting future work that we want to do, sort of building on that. Oh, come talk about center poster. >> And last but not least is Micheal Cole. He's going to talk about the User Domain Knowledge and Eye Movement Patterns During Search. >> COLE: Okay. I'm just going to [INDISTINCT] here. So what I'm going to focus on is work we've been doing that's developing a new a methodology to take eye movement data directly from Tobii Eye Tracking [INDISTINCT] and really focus on analyzing the eye movement patterns. And what we are interested in, in particular is focusing on modeling the process of information acquisition as the person's experiencing it during the search and the way in which we operationalize that is to focus on cognitive effort that's associated with that. The connection with knowledge without getting into an awful lot of thing, the general ideas that words are indicative of concepts and concepts are more or less the same as knowledge and [INDISTINCT] questions of relationships between rules and so on and so forth. But, very importantly the process of greeting involves using knowledge in order to understand all the words that you're dealing with. And also the process--the concepts that are involved and both the acquisition of information and concepts that you're getting from text that you're reading. So, you can see that it's a very central element in all aspects of search. So, what's very important is to understand that user knowledge actually controls interaction during search. It controls it--both in the sense of selecting the words that a person chooses to read and it also importantly imposes some cognitive demands in order to understand what's going on. So, here are a couple of really key facts about eye movement. The first is that eye movements are cognitively controlled and you do not passively move your eyes around and process the information that comes in rather you looked where you want to look. And there are number of studies that show generally that where you look depends upon which your current needs are and we can make the assumption at that will [INDISTINCT] information [INDISTINCT] as well. Now really critical point and somewhat surprising point is that when eyes fixate on a particular location retrospect to reading they remain fixed until you require the meaning of the words at that point. All studies have been conducted where you take the words away and people will continue to look at the word and tell they required it so, words that are less familiar take longer. So, this really tells us why we think eye movement pattern analysis is extremely powerful. First is too obvious to mention, if you're not looking at it you can't possibly acquire the information. But, the second is I think really the key stone here. And that is that you consider one and two you realize that there's a direct causal connection between the observations that you're making of people's eye movement patterns and cognitive processing states and by extension mental states. So, that's really nice because we can think now about how we can connect that with observable use of behavior. So, this is an example of people walking out of page of highlighted in green. Let's his workers and is actually reading so, the yellow dots are the fixation and the isolated fixation points is that there are three sequences were they engaged in somewhere extent. So, what our methodology does is to take eye tracking logs on the front-end that record fixation information and then produce this representation of reading experienced. That is the [INDISTINCT] things together and it's longer with these sequences and the shorter ones. Here are the four cognitive processing measures if you come back later I'll be able to tell a lot more detail about them. [INDISTINCT] actual span, how long you will be at it and how many times you [INDISTINCT]. There's a lot of empirical support through connections between each of these. Here's a study. I've only seen this several times before. So, the results that we have are that there are strong individual correlations between the level of domain knowledge and at least three of our cognitive effort measure. First is perceptual span which is roughly the spacing and the fixations and very interestingly there's a lot of evidence showing that that's related to a conceptual processing [INDISTINCT] of human beings. Second is how long an average looking out word and then finally--and what we built is some ration and classification models that we have some success in being able to show that at least some predictive [INDISTINCT]. So, main points, methodology lots of grounding and empirical research roughly 20 or 30 years and cognitive science directly connected to the information acquisition process very nicely it's completely domain independent. Nowhere have we talked about processing the content it's there and also as a follow on it's also culturally and individually independent because it's driven by the user's experience how there actually interact with the information. Thank you very much, our Google project. >> Okay, well, thanks to all [INDISTINCT] on time and we're going to have-- [INDISTINCT] time for questions, we have five minutes [INDISTINCT] that I really encourage you--possibly during lunch and take them all the questions that you've been thinking about and hold them back. We have one public service announcement that it would take a minute for--and then--that one. >> All right. So, I prevailed upon my organizers. So let me do one public--one little PSA, HHS is proposing to change the rules for humans subjects research and they've had a--the proposal update--this going to affect anybody in sort of on the academic side who's doing subjects research. They've had a proposal out there for a while, they're looking for comments. There's a number of promising things about the proposal but, obviously the [INDISTINCT] and details and this rule making process. One of the things My first--back in August I think, sort of--said, well, let's get make sure that the HCI researchers' voices heard. So, we've look at that, there's a bunch of us that have developed--drafted a letter, it's going to a few--several revisions and it's now available if you want to look at it and hopefully sign it up here at this Google address. And it's a--it's a summary, we couldn't address all the issues so, we encourage you to look at the letter, read it. If you agree with it please sign it. We've got seven--we've got about a hundred HCI researchers. We're hoping that this makes an impact when they read their comments. If you want to see the full proposal, it's available there. There's also a link to it on this page. So, you'll remember that whole thing. And we encourage you to read that and look at that.