Teaching and learning computer science sorting algorithms with mobile devices: A case study

  • Published on

  • View

  • Download


  • Teaching and LearningComputer Science SortingAlgorithms With MobileDevices: A Case StudyIVICA BOTICKI,1 ANTE BARISIC,1 SERGIO MARTIN,2 NEVEN DRLJEVIC1

    1Faculty of Electrical Engineering and Computing, Department of Applied Computing, University of Zagreb, Zagreb,


    2Electrical and Computer Engineering Department, UNED (Spanish University for Distance Education), Madrid, Spain

    Received 3 December 2011; accepted 20 February 2012

    ABSTRACT: In this article, we present a semester-long study on using smartphone devices in computingengineering education. We developed Sortko, an Android-based smartphone application for learning sorting

    algorithms, an important undergraduate computer science topic. The application consists of four main compo-

    nentsthe module for interactive sorting, the scaffolding module, the motivational module, and the graphical

    user interface module, each with a distinct role of helping students in learning sorting algorithms. Our re-

    search methodology included data collection with administered two surveys, collected exam results and

    recorded application usage data. Analysis of the collected data shows our approach is an effective way of

    learning sorting algorithms. 2012 Wiley Periodicals, Inc. Comput Appl Eng Educ; View this article online atwileyonlinelibrary.com/journal/cae; DOI 10.1002/cae.21561

    Keywords: mobile learning; engineering education; sorting algorithms


    This study examines educational benets of smartphone devices

    in higher education. In our approach students use Android [1]

    devices as a support in the process of learning sorting algo-

    rithms. Sorting, or putting series of elements in appropriate or-

    der, is an important computer science topic and presents an old

    and well-dened computer science problem. Mastering sorting

    algorithms is especially important for future software engineers

    regardless of the specic domain they work in.

    The growing popularity of smartphone devices can be

    clearly observed in everyday student life. Although they are

    mostly used for fun and informal ways of learning (e.g., access

    to Wikipedia, YouTube, etc.), scientic studies have shown

    the educational value of these devices. Based on an informal

    observation of our students interest in learning with smart-

    phone devices and by taking into account behaviorism and so-

    cialconstructivist theory of learning, we are suggesting a

    design of a smartphone application for learning sorting algo-

    rithms which, amongst others, includes a scaffolding module

    designed to assist students while performing sorting and a moti-

    vational module to additionally motivate students in using the


    Normally, students learn sorting by using a paper and pen-

    cil while simultaneously watching sorting algorithm program

    code. This tiring and ineffective procedure often results in dif-

    ferent kinds of errors such as students learning their own ver-

    sion of a sorting algorithm which does not match the real one

    (the sequence of steps while sorting elements is not adequate).

    We believe that technology can restructure this troublesome and

    ineffective process of learning thereby improving students

    knowledge of sorting algorithms. To verify our assumptions we

    are exploring these research questions:

    The students who used the application achieve betteracademic results than the students who did not use the

    application. The scaffolding module is able to assist students in theprocess of sorting.

    The motivational module motivates students on furtherapplication use.

    These research questions will be explored by analyzing

    results of standard exams, usage logs, and specially designed

    Correspondence to I. Boticki (ivica.boticki@fer.hr).

    2012 Wiley Periodicals, Inc.


  • questionnaires querying our students perception on usage and

    usefulness of the Sortko application and the approach in


    Learning and Playing With Mobile Devices

    An early research conducted in England as early as in 2005 had

    shown that more than 95% of students possess at least one mo-

    bile device [2], which opens up a space for their use in the

    process of educating young people. As for the effectiveness of

    the approach, another study presents a learning trail in which

    students were equipped with 100 Palm Pads stating that 90% of

    teachers had a positive attitude regarding the impact of this

    type of technology on students and on their apprehension of

    learning content [3,4].

    Apart from mobile learning [5,6], game-based learning

    examines the process of studying which is catalyzed and stimu-

    lated with the elements that are considered to be the standard

    parts of games (i.e., awards, collecting points, reaching higher

    levels, and achieving high scores in computer games). These

    motivate players to be involved in the game more thus enhanc-

    ing and accelerating their learning process through repeated

    game play [7].

    The crucial part of such approaches is the united goal of

    the game and of the learning process [8,9]. Once that is

    achieved, learning becomes entertainment and students get mo-

    tivated to learn at home and to take true interest in the course

    contents making the educational process more efcient and

    self-regulated [10]. What is more, by employing new pedago-

    gies which integrate learning with game play, teaching becomes

    more fun as well.

    A Paper and Pencil Approach to Learning SortingAlgorithms

    The topic of learning sorting algorithms is a part of the Algo-

    rithms and Data Structures course at the Faculty of Electrical

    Engineering and Computing, University of Zagreb, Croatia and

    has so far been taught in a paper-and-pencil fashion. Typically,

    various types of sorting algorithms (such as the Bubble sort

    algorithm which bubbles out the greatest number in each

    algorithm pass) [11,12] are visualized by drawing a series of

    algorithm passes, where each pass is drawn as an array of num-

    bers on the blackboard or a sheet of paper. After each algorithm

    pass, a new sequence of numbers is written down again

    (Fig. 1). As a consequence, during exams, students knowledge

    of sorting algorithms is tested in the same way.

    This repetitive and cumbersome way of learning often

    results in errors (i.e., a wrong sequence of algorithm steps is

    learned) and distracts students from actually understanding the

    algorithm big picture. Another problem that arises from this

    type of learning is that there is no easy way a student can check

    the accuracy of her proposed sorting algorithm passes. What is

    more, students sometimes get a correct (sorted) sequence even

    when a wrong set of algorithm steps is applied. For example,

    while applying Quick sort algorithm and identifying the pivotal

    element, a student can decide not to follow the step-to-step ap-

    proach of the algorithm and just position all smaller numbers

    on the left-hand side of the pivotal element and all greater num-

    bers on the right-hand side of the pivotal element. Although

    conceptually correct and with a correct nal solution, this way

    of sorting does not indicate whether the student actually knows

    the precise sequence of steps of the Quick sort algorithm and

    hence is graded with lower marks.

    The Use of Animations in Learning Sorting Algorithms

    The use of animation materials as learning resources has been

    advocated by many researchers [13] and there are a number of

    studies dealing with overcoming the typical difculties in learn-

    ing sorting algorithms with the use of animations [14,15]. Thus,

    many authors [1619] have provided animated resources to

    their students in order to enhance the concepts acquisition.

    Sutinen [20] introduced a Web-based tool, Jeliot, providing

    simple animations helping students in learning and understand-

    ing algorithms and data structures. Kordaki [21] performed a

    pilot where secondary school students used a web-based appli-

    cation helping them in understanding different sorting algo-

    rithms during in-class learning.

    An algorithm animation helps students in the visualization

    of a specic sorting algorithm but does not allow them to try

    out sorting on their own [22]. For example, www.sorting-algo-

    rithms.com web page [23] with a parallel animation of several

    sorting algorithms on the same unsorted number sequence.

    With this simulation several sorting algorithm behaviors can be

    observed on a randomly generated, almost sorted, reversed or a

    sequence with repeated numbers. Figure 2 shows the Sort Ani-

    mation web page which, in addition to the sorting algorithm

    operation animation, allows a parallel view of the programming

    code being executed during a specic animation step. The

    application also shows temporary algorithm variables and pro-

    vides users with a possibility of adjusting the sorting sequence


    In general, animations help viewers track patterns and ob-

    serve relationships, which is claimed to improve learning [24].

    However, some researchers challenge this stating that the use of

    PC-based animations for algorithm visualization does not nec-

    essarily lead to better concepts acquisition than with static dia-

    grams, especially in the case of students needing to make

    Figure 1 Traditional visual representation of the Quick sort sorting

    algorithm operation while sorting an array of numbers. [Color

    gure can be viewed in the online issue, which is available at



  • predictions of the coming animation sequence [13,15,25,26]

    and even suggest that animations distract students. Building on

    the rich amount of knowledge in learning with the help of ani-

    mations we propose a design in which instructional agency

    shifts toward the students side. By allowing students to explore

    the rich space of both incorrect and correct solutions while sort-

    ing, we aim to utilize technology capabilities to provide them

    with scaffolds in an experience-driven learning approach.


    Sortko System in Use

    The smartphone mobile learning application Sortko, designed

    as a part of this study, provides students with a graphical user

    interface with an option of choosing a sorting algorithm (e.g.,

    bubble sort, shell sort, quick sort, insertion sort, etc.) and gener-

    ates a random sequence of numbers to be sorted by applying a

    sequence of specic interaction gestures. During the sorting

    process, the system informs students about their success in per-

    forming so-called switches (element exchanges) and pro-

    vides them with textual help messages facilitating further

    progress. For each game a student gets rewarded with points

    that are then summarized on a joint public list.

    The graphical user interface consists of three main views:

    initial view, sorting view, and the result view, all made in hori-

    zontal mode to maximize display space and usage ergonomics

    (left-hand side of Fig. 3a).

    The sorting view contains a sequence of randomly gener-

    ated numbers (Fig. 3a, callout 1). While students sort by apply-

    ing gestures (Table 1), the system displays contextual textual

    help messages (Fig. 3a, callout 3). After each step (pass) of a

    sorting process, the textual help message changes therefore

    directing students toward the next required action. The current-

    ly used algorithm is shown to users as well (Fig. 3a, callout 4)

    indirectly determining the number of points a student will be

    assigned according to the predened formula (details of the mo-

    tivational module are described in the Motivational Module


    The Sortko application contains an auxiliary visual ele-

    ment positioned separately from the sorting array (Fig. 3a,

    callout 2). It is used to hold a variable with a temporarily

    element value stored until certain algorithm condition gets


    After sorting has completed, the results are sent to the

    server side and the complete ranking list is shown to the sorter

    (Fig. 3b).

    Sortko Application Model

    Sortko System Architecture. Sortko application is built for An-

    droid operating system which includes a set of programming

    libraries corresponding to the elementary Java programming

    language library [1]. Every Android application runs in its own

    process, with its own instance of the Dalvik virtual machine

    (VM). Dalvik [27] is a purpose-built VM designed specically

    for Android taking into account inherent mobile device limita-

    tions such as short battery life and processing power. Sortko

    application is created using Eclipse integrated development

    tools (IDE) equipped with the Android Development Tools

    plugin (ADT plugin) which allows for an easy setup of Android

    projects, application UIs and for the Android Framework API

    component reuse [28].

    The Sortko system is based on the clientserver architec-

    ture model and is divided into the following components: data

    access layer, the domain model, web component, service com-

    ponent, and the smartphone client application (Fig. 4). The

    server side of the system consists of a database, data access

    layer, web services, and a web site. The web part resides on the

    data access layer and can be accessed via desktop computers

    and mobile devices.

    The client part of the system is developed as an Android

    smartphone application and consists of the following modules

    (Fig. 5):

    The Interactive sorting module; The Scaffolding module providing help messages;

    Figure 2 Sort Animation web page with a parallel view of a sorting

    animation and the appropriate algorithm execution (http://www.cs.oswego.

    edu/mohammad/classes/csc241/samples/sort/Sort2-E.html). [Color gurecan be viewed in the online issue, which is available at wileyonlinelibrary.


    Figure 3 Graphical user interface of the Sortko Android smartphone application: (a) sorting view and (b) results

    view. [Color gure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


  • The Motivational module for awarding students; The User interface module.

    Graphical User Interface Module. The Interactive sorting

    module veries whether sorting steps provided by a student are

    valid for a given sorting algorithm. The Scaffolding module

    helps students in solving typical dilemmas in the sorting pro-

    cess and displays help messages after an error in sorting has

    been identied. It directs users toward choosing the right sort-

    ing solution by illustrating the logic behind a sorting problem,

    which is crucial in understanding sorting algorithms. The Moti-

    vational module tracks the successfulness of sorting according

    to different parameters (speed, accuracy, etc.) and awards points

    to users accordingly. The summary points and ranks are pre-

    sented in form of a web page accessible to all students.

    The client side application architecture follows the Model-

    View-ViewModel pattern [29]. It consists of a view, which col-

    lects user input and forwards it to the viewmodel and which

    actively pulls the data from the viewmodel by using the data

    binding technique. This pattern facilitates desirable software de-

    velopment techniques such as the separation of concerns

    (decoupling), unit testing, and test driven development. View-

    model encompasses all the benets of the patterns such as Mod-

    el-View-Controller and in the same time has an additional

    benet: the simplication coming from the declarative binding

    transferring data bidirectionally between viewmodel and the


    Interactive Sorting Module. The Interactive sorting module

    contains specialized implementations of sorting algorithms used

    to verify whether a user has chosen the accurate algorithm

    steps. The base class inherited by all specialized algorithm

    implementations is the abstract Algorithm.java class. Concrete

    implementations of algorithms are (1) BubbleSort.java, (2)

    InsertionSort.java, (3) QuickSort.java, (4) SelectionSort.java,

    and (5) ShellSort.java (Fig. 6).

    Scaffolding Module. The Scaffolding module helps students in

    sorting by giving them contextual textual help messages with

    the instructions on how to continue or on how to correct an

    error. It is linked with the Interactive sorting module and pre-

    serves the current state of the sorting algorithm currently in use

    (Fig. 7).

    The contextual text help messages are implemented

    according to each algorithms specics and capture current al-

    gorithm position. For example, textual help messages for the

    Bubble sort algorithm are provided by the BubbleSortPosition.-

    java class.

    Motivational Module. The Motivational module assigns points

    to students which are then sent to the server-side and there get

    summarized. The amount of points a student is awarded

    depends on the accuracy and the speed of her actions and the

    type of algorithm used. At a microlevel (one user action or one

    element exchange), points for a successful user action are

    awarded according to the following formula:

    User Action Points 1; 000 Alg Weight 10; 000 12 Alg Weight=10Switch Duration

    The formula is designed specically for this project in or-

    der to reward the use of more difcult algorithms. It consists of

    an exponential decreasing function starting with the maximum

    number of points, which decreases over time. For more

    Table 1 Typical Sorting Gestures

    Sorting gesture Sorting gesture description

    Sorting gesture

    algorithm coverage Note

    Element switch Two sequence elements switch their positions Bubble sort, Selection sort Action sequence not important

    Element switch

    (with itself)

    Element gets switched with itself, no visible

    sequence change afterwards

    Quick sort, Selection sort

    Element copy One element value is copied into another element Shell sort, Insertion sort Action sequence is important

    Figure 4 Sortko system architecture. [Color gure can be viewed in

    the online issue, which is available at wileyonlinelibrary.com.] Figure 5 Client side of the Sortko system.


  • complex algorithms, the function is declining more slowly but

    starts from a much larger number of points, therefore leaving

    more time for choosing an algorithm step to students. In case

    of simple sorting algorithms, the function is adjusted so that it

    rewards speed and accuracy, and has a smaller contribution to

    the overall amount of collected points. Such a reward system

    should direct users toward using more complex algorithms.

    The duration of one element switch (Switch_Duration) is

    the time from the last successful element switch till the current

    element switch. The formula constant 1,000 represents minimal

    points a student can get per one action (one element exchange),

    while the constant 10,000 represents the maximum number of

    points a user can get per one action. Constants 2 and 10 are

    used to model exponential function used to reduce the number

    of points awarded per user action as the exchange duration

    increases. As a consequence, a student gets a minimum of

    1,000 points for a successful action no matter how long it took

    to complete, and a maximum of Alg_Weight 10,000 pointsfor a successful action with the shortest possible element ex-

    change duration (if her action happened really fast). The algo-

    rithm weight (Alg_Weight) is a constant specic to each sorting

    algorithm used to reward students with more points when sort-

    ing with Quick sort and Shell sort algorithms. For an unsuc-

    cessful action users get penalized with a xed amount of 1,000

    points. The nal game points are calculated by summing up all

    points for successful actions and by deducting all points coming

    from the unsuccessful actions, therefore forming total single

    game points.

    Figure 8 shows two graphs with the visualization of the

    User_Action_Points formula for the two distinct algorithm

    weight (Alg_Weight) constants 1 (Bubble Sort, Insertion Sort

    and Selection Sort) and 5 (Quick Sort and Shell Sort algorithms).

    The graphs show that the function gives more points to the

    more demanding algorithms. For example, for a switch interval

    of 5 s, a student is awarded with around 1,300 points in the rst

    case (Alg_Weight 1) and with around 7,500 points in thesecond case (Alg_Weight 5).


    Sortko was used in the summer semester of the academic year

    2010/2011 of the Algorithms and Data Structures course

    taught as a part of the undergraduate study at the Faculty of

    Electrical Engineering, University of Zagreb. Prior to the

    students being formally introduced with the topic of learning

    sorting algorithms, the Sortko system was presented and the

    initial survey was administered.

    During the classes covering sorting algorithms students

    were able to download and install the Sortko application onto

    their Android smartphones from the Sortko website [30].

    They were enticed to do so by the teacher, but there was no

    curriculum integration of the Sortko activities, neither were

    there additional merits for students. Sortko was deployed as an

    add-on and its value is examined from that perspective

    throughout this section of the paper. Although, initially, the

    application was available for download as an Android-specic

    apk le, the new Sortko version can be found on the Android

    Market and downloaded as any other Android smartphone

    application [31].

    After the topic of sorting algorithm was completed, the

    students took their midterm exams. In addition to the midterm

    exam results, the log information on students activity in the

    Sortko system was collected and analyzed. As the nal data

    collection mechanism in this study, another survey was admin-

    istered in order to gather the overall students experiences.

    The Initial Survey

    Prior to the Sortko application release, the initial anonymous

    survey was administered. One hundred twenty-two out of 596

    students enrolled in the course took part in this survey. It con-

    sisted of 20 questions mainly targeting students afnities to-

    ward the use of technology in learning, availability of Android-

    based smartphones, etc. (Tables 2 and 3).

    Figure 6 Class diagram showing the main classes of the Interactive sorting module. [Color gure can be viewed in

    the online issue, which is available at wileyonlinelibrary.com.]

    Figure 7 Class diagram showing the main classes of the Scaffolding module. [Color gure can be viewed in the

    online issue, which is available at wileyonlinelibrary.com.]


  • The survey results show that sorting algorithms present a

    new topic to the students. Although the majority has heard

    about sorting, as much as 69% claim they do not possess opera-

    tional knowledge of sorting. This nding eliminates the tertium

    quid effect of the prior knowledge of sorting that might inter-

    fere with our study results. Additionally, we were interested in

    nding out whether our students were keen on accepting the

    new way of studying with smartphones. As the survey shows,

    they feel they learn best when they do it home alone studying

    course literature (67%) with only 28% applying some form of

    collaborative learning. This does not come as a surprise consid-

    ering half of the students are not satised with the formal lec-

    tures as they are currently done by the teachers in classrooms.

    This nding not only opens up a new space for curriculum in-

    novation, but also cautions us as our students are generally not

    satised with the current learning and teaching system.

    As a digital teaching tool, Sortko differs from the tools

    students are used to in their prior education. This does not pres-

    ent a problem to our students who believe that the technology

    can be used for learning (93%) and that it can enhance educa-

    tion (96%). They also believe sorting algorithms can be learned

    by using computers (90%) and as much as 85% believe that it

    can be done better than without computers. As far as smart-

    phones are concerned, the students are unsure whether they can

    help in learning. However, the students who already own a

    smartphone gave more optimistic responses (AVG 3.41)compared to the students who do not have a smartphone device

    (AVG 2.86), with the mean difference of 0.55 being

    signicant t(120) 2.08, P < .05 two-tailed, r .18, repre-senting a small-sized effect. Similar results were acquired for

    the general questions on use of technology in education.

    In total 39% of the surveyed students have their own

    smartphone device and 62% of smartphone users own an

    Android-based device, with version 2.2 being the most frequent


    Sortko Usage and Midterm Results Analysis

    After each successful sequence sorting, Sortko sends student

    identication, sorting algorithms applied, and the score awarded

    to the student to the server-side storage. That data is used to

    analyze what kind of sorting algorithms are most frequent and

    whether usage correlates with the standard exam results. The

    log data shows the most frequently used sorting algorithm was

    Bubble sort (31% of the total sequence sorting done), followed

    by Selection Sort (27%), Quick Sort (17%), Insertion Sort

    (15%), and the least used Shell sort (10%). We explain this

    difference with the two main reasons: Bubble sort is the sim-

    plest and usually the rst algorithm taught.

    The course Algorithms and data structures includes two

    midterm exams and a nal exam. Sorting algorithms are taught

    as the middle part of the course and are assessed with an as-

    signment in the second midterm exam, such as this:

    A sequence of numbers 8, 6, 4, 9, 2, 7, 5, 0, 3, 1 needs

    to be sorted using the Quick sort algorithm. Illustrate the

    sorting algorithm behavior by producing a new sequence

    after each element exchange and specially mark the

    numbers relevant for the next step (exchanges, pivotal

    elements, median approximation). The pivotal element

    needs to be chosen with the method of median approxi-

    mation of the rst, middle, and last sequence element.

    Figure 8 User_Action_Points formula graphs for the two distinct

    Alg_Weight constants.

    Table 2 Initial Survey Items and Summarized Students ResponsesLikert Scale Question Items (N 122)Survey item AVG SD

    Before the sorting algorithms lectures started I already knew sorting algorithms well 1.98 1.083

    During the ASP course lectures I am able to learn course contents well 3.17 1.197

    I learn the most by studying literature (e.g., slides, books, textbooks, exams) at home 3.88 1.182

    I learn the most when studying with colleagues in a group 2.66 1.270

    I like the idea of using technology in learning programming 4.28 0.884

    I believe the technology can help in improving education 4.62 0.647

    I believe the technology can be used for learning 4.56 0.793

    I believe there are advantages to using smartphones in education compared to classic desktop computers

    due to their characteristics (easy to carry, instant connectiveness, etc.)

    2.99 1.269

    I believe sorting can be learned with the help of computers 4.41 0.747

    I believe sorting can be better learned with the use of computer than without the use of computers 4.29 0.922

    I would like to participate in the research on learning sorting algorithms using smartphones 3.09 1.477

    Table 3 Initial Survey Items and Summarized Students

    ResponsesYes/No Question Items (N 122)Survey item Yes No

    I own a mobile phone (of any kindcell

    phone, smartphone, iPad, etc.)

    120 (98%) 2 (2%)

    I own a smartphone (Android, iPhone,

    Windows Phone 7, Blackberry, etc.)

    47 (39%) 75 (61%)

    I own a smartphone with an Android

    operating system

    29 (24%) 93 (76%)


  • In our study we were interested in nding out the correla-

    tion between the three main data sources: the total points stu-

    dents had on the Sortko system after the semester was done, the

    total points students had on the Quick sort algorithm on the

    Sortko system after the semester was done, and the results from

    the above presented assignment (standard midterm exam task

    4). The correlations are given in Table 4.

    Since the standard exam assignment included Quick sort

    algorithm, we calculated the correlation between the application

    usage frequency and the standard exams assignment results and

    got a fairly high value of 0.47 with a signicance of P < 0.05.Furthermore, the correlation between the Sortko total rank and

    the Quick sort assignment results gives a similarly high value

    of 0.53. Therefore, we became condent that Sortko usage

    means better standard exam results and explored this hypothesis

    by applying the independent sample t-test to compare between

    groups according to the mean values of usage frequency, total

    Sortko points, and the Quick sort Sortko points (Table 5).

    The analysis shows there is a signicant difference be-

    tween exam results when comparing Sortko users and Sortko

    non-users of around 30% according to all three criteria. The

    effect was large-sized of r 0.52, 0.70, and 0.65 for total us-age frequency, total Sortko points, and Quick sort Sortko points

    respectively. As a control mechanism, there are no signicant

    differences between the Sortko users and Sortko non-users

    when comparing their second midterm and rst and second

    midterm exam results. This proves our initial hypothesis that

    Sortko users achieve better standard academic results than

    Sortko non-users.

    The Final Survey

    The nal survey was done after the lectures and the second

    midterm exam were completed. It was conducted on a volun-

    tary basis attracting 29 students who used the Sortko system.

    They were asked how practical the application is when learning

    sorting algorithms, how well individual application modules

    were structured, etc. (Table 6). Additionally, as a part of the

    survey, we were interested in nding out whether and how often

    our students used the application. Sixteen out of 29 students

    claim they used Sortko once, while 27 out of 29 responded they

    used it more than once during the semester.

    In the analysis of the nal survey, we separated the stu-

    dents into two groups according to the academic success and

    the usage frequency criteria. We wanted to identify survey

    items for which these groups of students gave signicantly dif-

    ferent responses.

    Generally, the students feel Sortko helped them in learning

    sorting algorithms and in achieving better results in standard

    exams which is also conrmed by the quantitative analysis pre-

    sented in the previous section. They feel the Sortko Scaffolding

    module helped them in dealing with doubts while sorting, while

    the motivational module attracted a diverse set of responses

    which is analyzed in detail in the following sub-section.

    In order to further elaborate on the students answers to

    the survey questions, they were separated into two groups

    according to two main criteria: Their academic success and

    their Sortko usage frequency. By doing this we were able to

    explore whether the difference in students success and the

    number of times they used Sortko affects their perception as

    expressed through the administered survey. When comparing

    students according to their academic ability we found out there

    is a signicant difference between more and less academically

    successful students on only one survey item:

    The Sortko help message system helped me to learnsorts better (P < 0.05, one-tailed signicance level,r .19, small-sized effect).

    When comparing students answers to the survey questions

    according to the Sortko usage frequency, we found out there is

    a signicant difference between students who used the applica-

    tion a lot and the ones who used it less on the following survey


    Sortko application helped me in learning sorts(P < 0.05, two-tailed signicance level, r .40, large-sized effect).

    Sortko application helped me in achieving better resultson exam/quizzes/short lecture assessments (P < 0.05,

    two-tailed signicance level, r .33, medium-sized effect). The Sortko rank list motivated me to compete and usethe Sortko application multiple times (P < 0.05, two-tailed signicance level, r .70, large-sized effect).

    Motivational Module Effectiveness Analysis

    Sortkos motivational module was used to introduce a game ele-

    ment into the sorting activities. The rewards system was set up

    so that it stimulates using more complex algorithms such as the

    Quick sort or Shell sort by simply awarding such sorts with

    more points compared to the simpler algorithms.

    Table 4 Correlations Between Sortko Points, Sortko Usage, and

    Midterm Exam Results

    Total Sortko


    Quick sort

    Sortko points

    Midterm 4th

    exam points

    Quick sort sorting frequency 0.751 0.802 0.473Total Sortko points 0.788 0.528Quick sort Sortko points 0.499

    P < 0.05 (two-tailed).P < 0.01 (two-tailed).

    Table 5 Independent Sample t-Test Exploring the Difference in Academic Results Between Sortko Users and Sortko Non-Users (N 24)Total usage frequency Total Sortko points Quick sort Sortko points

    Fourth standard exam results (mean difference

    between groups in % of the assignment total)

    29,076 36,333 31,143

    Second midterm standard exam 4,711 3,833 3,657

    First and second midterm exam results 3,956 3,167 3,171

    P < 0.05 (two-tailed).


  • The correlation of students responses on the survey ques-

    tion Rank list motivated me to compete and use Sortko multi-

    ple times (AVG 4.29) and the usage frequency for thestudents who actually used the application is r .35, present-ing a medium-sized effect. Sortko power users gave more posi-

    tive answers to this question which conrms the rewards

    system motivated students to compete and use Sortko multiple


    An interesting issue arises in the nal survey analysis

    which shows that the application use frequency negatively cor-

    relates with the perception of adequacy of the rewards system

    (r .37). However, by taking into account the nal surveyresults analysis on the question item The Sortko rank list mo-

    tivated me to compete and use the Sortko application multiple

    times, we were able to identify there is a signicant difference

    of 1,894 between frequent and non-frequent Sortko users

    (P < 0.001, two-tailed signicance level). The explanation forsuch responses was looked for through informal personal inter-

    views in which we found out that in some cases points get

    awarded depending on luck. The main reason for that is the

    initial sorting sequence generation algorithm which in some

    cases generates sequences which are easy to sort. This leads to

    an interesting situation where a couple of Sortko power users

    had an extremely negative opinion regarding the rewards sys-

    tem. Regardless of the perceived unfairness, the correlation co-

    efcient between the usage frequency and the perceived

    motivational effects of the rewards system is relatively high

    (r 0.33).A surprising effect of the motivational module emerged

    after the nal survey was administered and the study was for-

    mally completed with additional 519 sequences sorted (27% of

    the totally sorted sequences). In addition to the pre-exam time,

    there is a usage peak between May 12, 2011 and May 22, 2011,

    a period after the nal exam (Fig. 9). This sustained usage tells

    us the standard exam is not the only motivational element for

    using Sortko and conrms the benets of the Motivational



    This article described a case study exploring the benets of

    using mobile devices in supporting teaching and learning of

    sorting algorithms, an important computer science topic. We

    proposed an approach in which an Android mobile application

    was designed and made available to students to be used in and

    out of classrooms. As a contrast to the conventional mode of

    teaching and learning, this application allowed students to en-

    gage in learning sorting algorithms in a location-independent

    wayboth in and out of classrooms. By leveraging the existing

    body of knowledge dealing with the use of animation in teach-

    ing and learning, we designed an interactive application which

    scaffolds students in trying out sorting on their own. Our design

    Table 6 Final Survey Items and Summarized Students ResponsesLikert Scale Question Items (N 29)

    Survey item AVG SD



    ac. succ.






    usage freq.




    Before the lectures on sorting in ASP course started I knew sorts well 1.86 0.953 0.128 0.543 0.850 0.071Sortko application helped me in learning sorts 4.28 0.922 0.735 0.119 0.037 0.727Sortko application helped me in achieving better results on

    exam/quizzes/short lecture assessments

    3.86 1.246 0.547 0.286 0.002 1.247

    By using Sortko application throughout a longer period of time,

    the sorting course contents are better mastered

    4.21 1.146 0.548 0.262 0.220 0.545

    The Sortko help message system helped me to learn sorts better 4.31 0.761 0.098 0.462 0.435 0.232The Sortko help message system helped me in clearing some

    doubts regarding sorts

    4.00 0.845 0.192 0.414 0.179 0.439

    Sortko help messages clearly direct on the application usage and

    identify errors during sorting

    3.72 1.066 0.466 0.295 0.490 0.288

    The Sortko rank list motivated me to compete and use the

    Sortko application multiple times

    3.55 1.639 0.778 0.176 0.000 1.894

    The Sortko reward (points) system is adequate and fair 3.48 1.379 0.950 0.033 0.428 0.485The Sortko reward (points) system motivates me to

    additionally use the application

    3.72 1.306 0.382 0.433 0.384 0.444

    I like Sortkos graphical design 4.45 0.827 0.751 0.100 0.159 0.449

    The Sortko application is easy and intuitive to use 4.34 0.936 0.947 0.024 0.195 0.470

    Now many points did you collect at the ASP course so far? 33.81 10.47

    P < 0.05 (one-tailed).P < 0.05 (two-tailed).P < 0.001 (two-tailed).

    Figure 9 Sortko usage peaks (summer 2011 semester).


  • acknowledged the role of motivation in learning by awarding

    student with points which were their accumulated on the com-

    mon rank list stimulating prolonged application use.

    We found out that student like using technology for learn-

    ing, although they are not all sure that smartphones specically

    can help in learning. However, we were able to nd out that the

    students who already own a smartphone recognize its educa-

    tional benets better than the non-owners. As far as the Sortkos

    motivational module is concerned, we are able to report that

    our students found it effective and that it enticed students on a

    prolonged system use. On the other hand, standard exam results

    analysis shows Sortko users achieve better results compared to

    Sortko non-users.

    We plan to upgrade Sortko for the upcoming academic

    year, focusing on improving the Motivational module by reduc-

    ing the luck factor in the initial number sequence generation as

    well as creating a skill-based ranking mechanism in addition to

    the purely cumulative one. This should make the Motivational

    module more effective in motivating students on using Sortko

    and neutralize the annoyance exhibited by power users who fa-

    vor skill-based competition, as shown by the analysis. Due to

    the rapid growth of the smartphone market porting Sortko onto

    other prevailing mobile platforms, such as iPhone or Windows

    Phone 7, should allow us to reach wider audiences and afrm

    Sortko as an educational training tool for mastering sorting


    Although the application and the results were encouraging,

    we cannot emphasize the role of curriculum integration more.

    Since the Sortko application and this study were done as an

    add-on onto the curriculum, we were not able to motivate the

    majority of students to participate. It is our belief that similar

    educational interventions have to be employed as well integrat-

    ed curriculum elements and not as add-ons to achieve the full

    and sustained educational power.

    To sum up, mobile learning applications such as Sortko

    are able to engage students and can serve as a motivational

    vehicle in computer science engineering motivation. Students

    are open to such interventions and educators should leverage on

    the intrinsic benets of mobile technologies in developing their

    teaching and assessment strategies.


    [1] Android-Online, Android developers guide, Retrieved February

    15, 2012, From http://www.android.com/developers/.

    [2] L. Naismith, P. Lonsdale, G. Vavoula, and M. Sharples, Literature

    review in mobile technologies and learning, report no. 11, Future-

    lab, Bristol, United Kingdom, 2005.

    [3] H. B. Andreu and M. Nussbaum, An experimental study of the

    inclusion of technology in higher education, Comput Appl Eng

    Educ 17 (2009), 100107.

    [4] J. Roschelle, Keynote paper: Unlocking the learning value of wire-

    less mobile devices, J Comput Assist Learn 19 (2003), 269272.

    [5] C. Romero, S. Ventura, and P. de Bra, Using mobile and web-

    based computerized tests to evaluate university students, Comput

    Appl Eng Educ 17 (2009), 435447.

    [6] G. Vavoula and M. Sharples, Challenges in evaluating mobile

    learning, Proceedings of the mLearn 2008 Conference, Shropshire,

    United Kingdom, 7th October 2008.

    [7] M. Ebner and A. Holzinger, Successful implementation of user-

    centered game based learning in higher education: An example

    from civil engineering, J Comput Educ 49 (2007), 873890.

    [8] A. A. Deshpande and S. H. Huang, Simulation games in engineer-

    ing education: A state-of-the-art review, Comput Appl Eng Educ

    19 (2011), 399410.

    [9] M. Milrad and M. Jackskon, Designing and implementing educa-

    tional mobile services in university classrooms using smart phones

    and cellular networks, Int J Eng Educ Mobile Technol Eng Educ

    23 (2007), 8491.

    [10] T. A. Philpot, R. H. Hall, N. Hubing, and R. E. Flori, Using games

    to teach statics calculation procedures: Application and assess-

    ment, Comput Appl Eng Educ 13 (2005), 222232.

    [11] A. Drozdek, Data structures and algorithms in C, Course tech-nology, Boston, 2005.

    [12] M. A. Weiss, Data structures and algorithm analysis in C, Addison

    Wesley, Reading, MA, 1997.

    [13] M. D. Byrnea, R. Catramboneb, and J. T. Stasko, Evaluating ani-

    mations as student aids in learning computer algorithms, Comput

    Educ 33 (1999), 253278.

    [14] S. R. Hansen, N. H. Narayanan, and D. Schrimpsher, Helping

    learners visualize and comprehend algorithms, Interact Multimedia

    Electron J Comput Enhanced Learn 1 (2000).

    [15] C. Kann and C. Hundhausen, Integrating algorithm visualization

    technology into an undergraduate algorithms course: ethnographic

    studies of a social constructivist approach, Comput Educ 39

    (2002), 237260.

    [16] R. M. Baecker, Sorting out sorting. Film. University of Toronto,

    M. Kaufmann Publishers, San Francisco, USA, 1981.

    [17] B. Deliktas, Computer technology for enhancing teaching and

    learning modules of engineering mechanics, Comput Appl Eng

    Educ 19 (2011), 421432.

    [18] H. L. Dershem and P. Brummund, Tools for web-based sorting

    animation, ACM Special Interest Group Comput Sci Educ Bull

    Inroads 30 (1998), 222226.

    [19] T. L. Naps, Algorithm visualization in Computer Science Labora-

    tories, Proceedings of the 21st SIGSE Technical Symposium on

    Computer Science Education Special Interest Group for Computer

    Science Education Bulletin Inroads, Washington, USA, February


    [20] E. Sutinen, J. Tarhio, and T. Tera, Easy algorithm animation on

    the Web, Multimedia Tools Appl 19 (2003), 179194.

    [21] M. Kordaki, M. Miatidis, and G. Kapsampelis, A computer envi-

    ronment for beginners learning of sorting algorithms: Design and

    pilot evaluation, Comput Educ 51 (2008), 708723.

    [22] S. Price and Y. Rogers, Lets get physical: the learning benets of

    interacting in digitally augmented physical spaces, J Comput Educ

    15 (2004), 169185.

    [23] D. R. Martin, Sorting algorithm animations, 2007, From http://


    [24] G. G. Robertson, S. K. Card, and J. D. Mackinlay, Information

    visualization using 3D interactive animation, Commun ACM 36

    (1993), 5771.

    [25] J. S. Gurka, Pedagogic aspects of algorithm animation, Unpub-

    lished PhD Dissertation, University of Colorado, Colorado,


    [26] J. Stasko and E. Kraemer, A methodology for building applica-

    tion-specic visualizations of parallel programs, J Parallel Distrib

    Comput 18 (1993), 258264.

    [27] D. Bornstein, Presentation of Dalvik VM Internals, Google,


    [28] Eclipse-Online, About the Eclipse Foundation, Retrieved February

    15, 2012, From http://www.eclipse.org/org/.

    [29] S. F. Allam, SOA Design pattern for distributing the object model

    in n-tier applications, Innovations and advanced techniques in sys-

    tems, computing sciences and software engineering, Springer,

    Netherlands, 2008, pp 182185.

    [30] FER-ZPR, Sortko website, 2012, From http://nihao.fer.hr/sortko

    (alternate url

    [31] FER-ZPR, Sortko Android market web page, 2012, From https://




    Dr. Ivica Boticki is a researcher and lecturer

    in the Department of Applied Computing,

    Faculty of Electrical Engineering and Comput-

    ing, University of Zagreb, Croatia. His

    research interests include mobile learning,

    software engineering and engineering educa-

    tion. He teaches courses in software engineer-

    ing and does research in technology enhanced

    learning focusing on two main areas: primary

    school education and the enhancement of edu-

    cational practices in software engineering education. He is an author

    and co-author and reviewer of publications in several journals and


    Ante Barisic got his masters degree in com-

    puter science and is currently pursuing another

    masters degree in economics at the University

    of Zagreb, Croatia. Passionate about product

    design and development, he started his career

    with developing ICT products in telecom

    industry. His main area of interest is the devel-

    opment of web and mobile applications.

    Dr. Sergio Martin is an assistant professor

    and researcher in the Electrical and Computer

    Engineering Department at UNED (Spanish

    University for Distance Education) since 2006.

    He has a PhD in Computer Engineering. He

    has participated since 2002 in national and

    international research projects.

    Neven Drljevic is currently pursuing his mas-

    ters degree in computing at the University of

    Zagreb. Enthusiastic about new technologies

    and opportunities they open up for young

    people, several years ago he started the Web

    Start Contest, a successful local competition

    and educational course in developing student

    web start-up projects, organized by the student

    association eSTUDENT.