1. Dominikus and I will present our work The Streams of Our Lives: Visualizing Listening Histories in C as a project I did together with Frederik Seiffert, Michael Sedlmair and Sebastian Boring. THE STREAMS OF OUR LIVES > DOMINIKUS BAUR FREDERIK SEIFFERT MICHAEL SEDLMAIR SEBASTIAN BORING UNIVERSITY OF MUNICH (LMU) GERMANY VISUALIZING LISTENING HISTORIES IN CONTEXT http://www.flickr.com/photos/natita2/2565850315/
2. talk you will hear about the data space of music listening histories, see LastHistory, a tool for visual mation plus contextual data and finally, hear what we learned about aesthetics and user appreciation In this talk: The music data space Large-scale study
3. An abundance of online services gives us the chance to log almost all aspects of our lives:
4. Nike Plus captures your running behavior and helps with exercising.
5. You can use wakoopa to track what software you use on your computer and how much time you waste on the internet.
6. And this new trend seems to have no boundaries. You can track everything. Really everything.
7. One of the older services available is Last.fm. While they initially tracked a persons music consumption to provide suitable recommendations in their webradio, the resulting listening histories have become a use case on their own.
8. But once all this data has been collected, making sense of it is hard, especially as last.fm only provides chronologically sorted lists. Fortunately, they also have an API that let all kinds of statistical and graphical tools appear.
9. Fan-made tools like Last.fm Explorer or LastGraph but also Last.fms own playground tools give users an entertaining but ultimately superficial overview of their own listening habits. So, in this case study we present
10. LastHistory, a casual infovis tool for analyzing and reminiscing in ones own listening history. Several thousand people downloaded it and we received lots of feedback which Ill be coming to in just a minute.
11. First, let me talk a little bit about the data spaces characteristics that we are talking about here. To be able to provide a suitable visualization of this information we first have to be clear about the attributes of the visualized data. The space of music data 1.
12. A term that Ive mentioned several times now is listening history. In our understanding an ideal listening history describes all songs that a person has listened to, possibly in their lifetime. Whats important here is that Listening history (noun) A complete chronological collection of musical items
13. Each song is a pre-existing piece of music that has attributes such as artist, title, etc. ... Each song: (1) pre-existing piece of music ...
14. And second, each song has been heard by the owner of the history at least in parts. (2) has been heard at least partially
15. From the perspective of infovis, listening histories are multivariate time series. We can of course interpret it as univariate and time- centric, Listening history (data type) Multivariate time series data
16. as each section of time either contains music or it does not and can then start to extract e.g., listening sessions as each section of time either contains music or it does not and can then start to extract e.g., listening sessions Time
17. But much more insight can be gained once we go beyond this binary classification. Listening SessionListening Session Time
18. So, the first thing we can do about that is put the songs into the musical hierarchy of albums, artists and genres. While this classification is not perfect and oftentimes the topic of heated debates, at least its widely-known among all music listeners. One more step to overcome the downsides of a strict hierarchy is adding user-generated keywords into the mix Listening SessionListening Session Time Genre Sub-Genre Artists Albums Songs
19. that can become a stand-in for any number of different hierarchies or classifications. Ok, so once we have mapped the space of ideal music listening into this neat format were good to go building a visualization on top of it, but, unfortunately, the real world is messy Listening SessionListening Session Time Genre Sub-Genre Artists Albums Songs Tags
20. Last.fm provides the so-called audioscrobbler, a software thats running in the background and tracking all music files that are played on the computer. This procedure comes with its own limitations as the resulting listening histories
21. Real listening histories: - incomplete - noisy
22. Real listening histories: - incomplete - noisy
23. One common source is that the listener is using non-supported hardware for listening
24. Another that music comes from other sources like when shopping or being at a friends place.
25. Real listening histories: - incomplete - noisy
26. The user might leave the computer while the music keeps on playing
27. Or someone else is using the computer while the audioscrobbler is still running.
28. So much for the data space and its attributes. Next, we have to think about who our users are and what they want to do. All lifelogging applications are first of all about User Requirements 2.
29. Stroking your ego. Its about learning about yourself, understanding what you did, maybe finding patterns that you were not aware of and remembering the past.
30. We defined two larger types of tasks that should be possible with ones listening history: First, the rather impersonal analysis where Lifelogging modes Analysis Personal
31. First, the rather impersonal analysis where users are looking for patterns within the data. The nice thing about that is that all possible insights are hidden within the data, which means that people who havent created the listening history are able to understand whats going on (or at least see the patterns). Still, these insights can only be on an abstract level we see which songs are repeated are popular but dont know why.
32. To dive into the actual, underlying reasons we have to ask the creators of the data themselves, as they might be aware what intentions they had when picking a certain song. In this personal mode, a users memories form the second, complementary data source. The only thing is, we somehow have to reach the memories that the users have about a certain period in their lives and using songs with timestamps is not the best way to do it.
33. Much better memory triggers are photos or other actively created items. Therefore, to make sense of histories in this personal mode, adding such contextual information can be really helpful.
34. Ok, so with these two use cases of analysis and personal mode in mind, lets look at related work from this area. Related Work 3.
35. Understanding how people listen to music is the domain of music psychologists and music sociologists. They have uncovered fascinating aspects about human behavior in this regard but of course nothing about how to visualize that. .
36. The concept of the timeline is common in Infovis, and for personal information it has been repeatedly applied in projects like LifeLines (for medical data), and PostHistory or TheMail (for email archives). LifeLines2  PostHistory  TheMail 
37. Two projects visualize listening histories: The Stacked or Streamgraphs by Byron and Wattenberg show overviews of prominent artists in listening histories. In our own former work, Pulling Strings from a Tangle, we presented two playful visualizations for this type of data, but they, too, could only paint an abstract picture of it. Strings & Tangle  Stacked Graphs 
38. So, when we went about designing our own tool, LastHistory, we wanted to create something that allowed gaining an overview of a listening history, but also explore it in detail. Finally, we wanted to integrate contextual information as memory triggers. LastHistory4.
39. For our design requirements: We had non- infovis experts as users. So we decided to keep the interface as non-threatening as possible and make more complex tasks not obligatory. Second, the tasks that users try to fulfill with the tool were nice to have, but not vital to them. So it was important to give them an immediate benefit and keep from frustrating them. Finally, we had to work with missing and imperfect data and users should be aware of this fact and not blame the visualization. Design requirements Non-expert users Non-vital tasks Missing and imperfect data
40. With these considerations we arrived at the final version of LastHistory and Ill try to give you a quick rundown of it.
41. The largest part of the application is taken up with a 2D timeline: all songs are represented as small circles and mapped horizontally to the day and vertically to the time of day of their timestamps. This way, we can easily see daily rhythms,
42. Like at what time this user went to bed.
43. And heres another example: A user who gets up at the same time everyday and listens to music first thing in the morning. The great thing about this mapping is that users have an immediate benefit even without interacting with the visualization.
44. Each songs genre is color-coded, so the user gets an immediate overview over the variety of songs. Were of course restricted in the number of colors we can use to keep them distinguishable. classical jazz funk hip-hop electronic rock metal unknown/other
45. We also separated the whole interaction into two modes, analysis and personal. The main difference is that the personal mode also displays photos and calendar entries from the users computer to provide context and make it easier to remember what happened at what time and understanding the listening decisions. Users can simply switch between the two modes with the button in the upper left corner. Two usage modes: Analysis Personal
46. Beyond static visualization, users can navigate within the visualization by panning, triggered by dragging with the mouse X Video of Pan
47. One-dimensional zooming by using the mouses zoom wheel or the slider in the lower right corner allows them to focus on certain sections of the history. X Video of Zoom
48. Hovering over a song shows a box with user- generated keywords from last.fm, but more prominently: connects this song with all other instances of it throughout the history. So, users can easily see when they listened to this one song.
49. Preceding and succeeding repeated songs are also highlighted, so sequences such as albums or other predefined playlists are automatically highlighted.
50. Finally, in the upper right corner of the application, theres a textbox for filtering where users can enter freeform terms. Its possible to enter song or album titles or artist names to filter all other songs.
51. But the filter box can also be used for temporal queries by entering dates, or periods of time, so users can, for example, see all songs that they listened to in autumn before noon.
52. Having finished implementing LastHistory we wanted to find out how well our considerations had worked and if people would actually find it useful. Reception5.
53. First, we gave LastHistory to four participants in a lab study and let them play around with it for an hour. They analyzed their own listening history in combination with photos and calendar entries and told our experimenter what they were able to find out and what they liked or disliked about the application. 4 participants in lab study Histories at least 3.5 years Avg. 45,000 songs
54. Results were insights about daily and other rhythms: One participant stopped listening to music during the week when she started working. A few noted how artists were rediscovered after going to a concert or them releasing a new album. And gaps and noise could also be seen and identified thanks to photos: Three participants found time periods empty where they were on vacation. Cycles (Re-)discovery of artists Gaps & noise People found:
55. But much more interesting than these case studies was the feedback we received in the wild: We made LastHistory available for download and it was quickly picked up by blogs like Life Hacker or Newsweeks website, which gave us a lot of attention. LastHistory creates a spectacular visualization of your audio timeline. If you're a music junkie who loves the statistics that Last.fm gives you, you'll love how LastHistory extends that even further. newsweek.com lifehacker.com
56. In the end, about 5.000 people downloaded the application. We included a link to an online questionnaire that was shown after users had spent some time with the application and at the time of writing the paper, 243 people had filled out that form which was a valuable source of information for us. 5,000 downloads 243 filled-out questionnaires
57. Our users were predominantly male (95.1% to 4.9%) and most of them (56%) had a background in academia or technology. Age range was 16 to 67 years, with an average of 27.2 years. Gender: 95.1% male, 4.9% female Background: 56% academia/technology Avg. age: 27.2 years
58. Regarding the available data, almost all users had their own listening histories available (97.1%), but only about one third (37%) photos and 18.5% calendars. Listening history: 97.1% Photos: 37% Calendars: 18.5% Data available:
59. This, of course, led to a dominance of the analysis mode over the personal mode: 75.7% found it useful, while only 51.8% said that about the personal mode. But when you only ask people who had Analysis mode was useful: 75.7% Personal mode was useful: 51.8%
60. Photos and calendars available they reach about the same rating. Analysis mode was useful: 75.7% Personal mode was useful (people with photos + calendars): 75.8%
61. Insights that people gained also represented that: While some said about it: I like this mode the best, it should be the default mode, or clicking on a photo gallery and listening to what I was listening to at the time was very powerful others complained about the missing data. I like this mode the best, it should be the default mode! Clicking on a photo gallery and listening to what I was listening to at the time was very powerful. About the Personal Mode:
62. The analysis mode was more open for all users, and insights were mostly based on time and musical taste. One user found I rarely listen to music between the hours of 9-11 a.m., even on weekends and another one noted his commuting pattern. A third one said she found those ruts where you get stuck in listening to one particular song. Finally, users were also aware of the imperfect data. I rarely listen to music between the hours of 9- 11 a.m., even on weekends. I noted the commuting pattern. About the Analysis Mode: Those ruts where you get stuck in listening to one particular song. I listened to music for 4 straight days!
63. But the most important finding for us was: People actually wanted something like LastHistory! The positive feedback just blew us away and we were very happy that people liked it. And even though the application wasnt completely trivial, around 75% of our users found it easy to learn. It worked!
64. And with that, I want to talk a little bit about what we learned and where to go from here. Beyond music 6.
65. First, of course, our greatest fear as Infovis researchers is that our work is labeled as pretty pictures. But if you can build a pretty visualization without sacrificing functionality or making reading it more difficult why not do it? Users always prefer an aesthetically pleasing application to a dull one. And yes, even expert users. Thinking about users If you have to do it anyway, why not make it pretty?
66. One new aspect that this very personal data type let us do was integrate a non-tangible data type into the visualization: the users memory. Depending on the use case, this can be a powerful source for understanding and learning. Providing suitable memory triggers is essential here.
67. Finally, lifelogging wont go away and it will probably only get worse. Especially as all media consumption becomes digital (even books!) tracking every little aspect of reading, watching and listening will become commonplace. The age of lifelogging is upon us.
68. The concepts that we used in LastHistory would of course also work for these other media and we could basically add all the data streams we could think of to the timeline. But I think whats more in this regard is that people (1) actually want something like that and (2) that information visualization can help them for understanding their behavior and reminiscing. That means that their data is no longer only useful to companies and governments: These visualizations can give the data back to their creators. And we as visualization researchers and practitioners are the people to build those tools.
69. Thank you! > DOMINIKUS BAUR FREDERIK SEIFFERT MICHAEL SEDLMAIR SEBASTIAN BORING UNIVERSITY OF MUNICH, GERMANY Download LastHistory (open source) at: http://www.frederikseiffert.de/lasthistory email@example.com twitter: @dominikus
70. Image credits: Dear Diary: http://www.flickr.com/photos/hippie/2475835909/ Nike Plus: http://www.flickr.com/photos/irisheyes/327069125/ Last.fm Logo: http://www.flickr.com/photos/dekuwa/3383948292/sizes/o/in/photostream/ Audio cassette: http://www.flickr.com/photos/steve-maw/4234388137/ Notebook scribbles: http://www.flickr.com/photos/cherryboppy/4812211497/ Messy Pudding Kid: http://www.flickr.com/photos/ripizzo/2310929170/ Empty computer: http://www.flickr.com/photos/nycgraeme/2287826242/sizes/l/ Grammophone: http://www.flickr.com/photos/jenik/2861494379/sizes/l/ Shopping Cart: http://www.flickr.com/photos/spijker/3273982099/sizes/l/ Computer cat: http://www.flickr.com/photos/kevinsteele/1507196484/sizes/l/ Photo Girl: http://www.flickr.com/photos/blythe_d/1451273161/sizes/o/ Scientists: http://www.flickr.com/photos/marsdd/2986989396/sizes/o/ Reading girl: http://www.flickr.com/photos/12392252@N03/2482835894/ Old photos: http://www.flickr.com/photos/soaringhorse/419617753/ Gallery reception: http://www.flickr.com/photos/bricolage108/319671818/ Light at the end of the tunnel: http://www.flickr.com/photos/mercedesdayanara/881056652/ Papers:  Taowei Wang et al.: Aligning Temporal Data by Sentinel Events: Discovering Patterns in Electronic Health Records, CHI 2008  Viegas et al.: Digital Artifacts for remembering and storytelling: Posthistory and social network fragments, HICSS-37, 2004  Viegas et al.: Visualizing email content: portraying relationships from conversational histories, CHI 2006  Byron et al.: Stacked graphsgeometry & aesthetics, InfoVis 2008  Baur et al.: Pulling Strings from a Tangle: Visualizing a Personal Music Listening History, IUI 2009