Analyzing Students’ Computational Thinking Practices in a First-Year Engineering Course

This study investigates the relationship between two related computational thinking practices: data practices and computational problem-solving practices in acquiring related computational thinking practices during a first-year undergraduate engineering course. While computational thinking theory is still developing, empirical studies can help further understand how students demonstrate this knowledge and their progression in attaining the practices. Therefore, with this empirical study, the following questions are addressed. RQ1: What are the differences in students' computational thinking practices at the beginning of an undergraduate introductory programming course? RQ 2: How do these differences correspond to the acquisition of more advanced computational thinking practices? The use of a descriptive non-experimental design that aims to understand the correlation between practices related to data and computational problem-solving is presented. A machine learning technique is employed, utilizing historical data from introductory programming for a problem-solving course with more than 1000 first-year engineering students. After identifying groups of students defining different profiles, data from posterior performance in more advanced programming topics were descriptively analyzed. This study supports the characterization of four different student profiles demonstrating differences in their performance at the beginning of the semester. From these four profiles, two of them show a subsequent differential progression besides their similarity at the beginning of the semester. In this particular case, troubleshooting and debugging appear as a relevant competency when distinguishing these two learners' groups. These findings suggest that previous knowledge or exposure to different practices can result in different progressions of more complex computational practices, emphasizing the relevance of troubleshooting and debugging as a practice required for a successful and timely progression on the acquisition of other computational thinking practices.

SECTION I. Introduction

Identifying solutions to complex problems in our society demands engineering graduates who possess not only the technical knowledge and skill set of the discipline but also the professional mindset needed to shape the future. For example, the Engineer 2020 proposes traits such as strong analytical skills, creativity, ingenuity, professionalism, and leadership as aspirations needed to operate in societal, geopolitical, and professional contexts within which engineering and its technologies will occur [1]. Such ambitions and traits relate to students’ “habits of mind,” previously defined as “modes of thinking required for STEM students to become effective problem solvers capable of transferring such skills to new contexts” [2].

A skill that contributes to the development of analytical skills is students’ ability to use computation [3]. Our preliminary work has investigated aspects of teaching and learning with computation in the context of engineering education. This work has identified that undergraduate learners often face obstacles when engaging in computation assignments. The identified barriers have primarily been related to students’ inability to conduct mappings between the physical phenomena, the mathematical representation, and the computational representation [4], [5]. However, after semester-long exposure, undergraduate students have been able to demonstrate (a) understanding of foundational computing concepts and procedures for solving well-structured engineering problems [6], [7], and (b) changes to positive attitudes toward their ability to interpret computational representations, [8].

The computational aspects of teaching and learning investigated before could also be expressed as computational thinking practices. Computational Thinking (CT) is a mindset required for this new century [9]. The International Society for Technology in Education (ISTE) and Computer Science Teacher Association (CSTA) [10] define CT as a problem-solving process that includes but is not limited to six competencies. (1) Formulating problems in a way that enables us to use a computer and other tools to help solve them, (2) Logically organizing and analyzing data, (3) Representing data through abstractions such as models and simulations, (4) Automating solutions through algorithmic thinking, (5) Identifying, analyzing, and implementing possible solutions to achieve the most efficient and effective combination of steps and resources, and (6) Generalizing and transferring this problem-solving process to a wide variety of problems.

The processes involved in the formulation of problems and solutions using computational thinking are complex, as there are multiple competencies simultaneously in-use [11], [12]. As a complex mindset, CT has different components, which are called computational thinking practices [13]. These practices are more granular abilities that, in conjunction with each other, produce CT competencies. CT latest calls identified one of the goals for CT education as the integration of CT into disciplinary education [14], yet all of those new features, coupled with the complexity of what CT, make the design of courses and assessments an onerous task.

The introduction of concepts and experiences that help students acquire CT has been discussed broadly. Introduction of a specific programming language in K-12 [15] to the incorporation of robots and game design for young adults [16] and children and adolescents with Autism Spectrum Disorder (ASD) [17]. With some authors reporting a considerable variation in the attainment of the competencies when in an introductory course [18]. In addition, some researchers have also focused on developing frameworks to evaluate the attainment of CT competencies when students are under certain experiences, as it is game design workshops [19].

In terms of relationships, CT and other aspects of learning have been empirically evaluated. For example, research has explored the relationship between CT and the use of computational tools, student’s engagement in robotics, the effect of previous mathematics classes, and the role of creativity [20]. Some empirical studies have also addressed the differences in the achievement of students within specific backgrounds. For example, Peteranetz et al. [21] evaluated the differences in the achievement of students with similar patterns of motivation, engagement, and self-regulation. In comparison, Shell and Soh [22] evaluated the differences between students that take a CT class as part of a major or a non-major course.

The CT literature has addressed relationships of CT with external approaches to it, as well as background and emotional treats, yet there is a lack of understanding of the relationships between the practices that conform CT and student performance. Therefore, a careful evaluation of the practices that compose CT is required to develop the curriculum and assessment of CT courses. Understanding the interactions that can occur between the different practices and how they relate to each other can inform more effective ways of teaching computation. Knowing that CT is complex and therefore composed of multiple competencies, empirical studies will need to start with some of these competencies.

In order to further understand how CT practices interact, it is crucial to study their progressions empirically. Therefore, the purpose of this study is to further the understanding of the interaction between CT foundational and more advanced practices. Our research questions are: RQ$I$ : What are the differences in students’ computational thinking practices at the beginning of an undergraduate introductory programming course? RQ 2: How do these differences correspond to the acquisition of more advanced computational thinking practices? Specifically, our goal is to identify and analyze patterns of the initial ability of the students in CT practices and their relationship with the progress of other CT practices in a first-year engineering course.

SECTION II. Background

Weintrop et al. [13] reviewed the existing literature on CT. They performed open coding on 32 in-classroom activities and used external reviewers to propose a taxonomy that includes four major categories: (1) Data practices, (2) Modeling and Simulations Practices, (3) Computational problem-solving practices, and (4) System thinking practices.

Although Weintrop and colleagues’ framework was proposed for the K-12 settings, it is relevant to a first-year engineering setting given that incorporation of CT to K-12 is still in progress. Therefore, it is likely that first-year engineering students enter the program without previous CT experience. Additionally, it has been pointed out that the literature on CT higher education is minimal. The vast existing literature on CT addresses the K-12 setting, and some authors have highlighted the importance of refining CT literature into higher education [20]. In addition, Weintrop and colleagues’ framework was chosen because it is comprehensive and grounded on evidence, that unlike many other taxonomies, has been based on collected data from classrooms and expert reviewers.

For this study, the taxonomy proposed by Weintrop et al. [13] was chosen. Weintrop and colleagues developed the taxonomy for science and math teachers to use it for more specific characterization of the practices applied in their specific domain, rather than computer science. Specifically, this study focuses on two of the four major practices proposed by Weintrop and colleagues: Data practices (DP) and Computational problem-solving practices (CP).

DP refers to the role of data in the inquiry. DP, as the authors defined it, refers to the experiences of experts dealing with raw data, manipulating it, and setting questions that can be answered using the available data. It is composed of five computational thinking practices: collecting data, creating data, manipulating data, analyzing data, and visualizing data.

Modeling and simulation practices refer to the ability to create, refine, and use models to understand phenomena. This category is composed of five different practices: using computational models to understand a concept, using computational models to find and test solutions, assessing computational models, designing computational models, and finally constructing computational models.

CP refers to the methods that are effective when trying to solve a problem with a machine or other computational tools. This category is composed of seven practices that support critical thinking with computation, preparing problems for computational solutions, programming, choosing useful computational tools, assessing different solutions, developing modular computational solutions, creating computational abstractions, and troubleshooting and debugging.

Finally, systems thinking practices refer to the ability to understand the different behaviors of the elements of a system, the interactions between elements, and the aggregate dynamic system. Systems thinking is composed of five practices: investigating a complex system as a whole, understanding the relationship within a system, thinking in levels, communicating information about a system, and defining systems and managing complexity.

In engineering, the first-year engineering curricula in some institutions offer a project-based introductory class, in which concepts of CP and DP are presented to the students [23], [24]. These two practices may be chosen due to their developing closeness to the knowledge acquired in K-12.

SECTION III. Methods

A. Contexts and Participants

This study was set in a large 16-week required first-year engineering course offered in Spring 2018. The course was divided into 14 sections of no more than 120 students, 1559 students in total. The characterization of the population studied was based on years in the program, gender, race, and nationality. 91% of the students were freshmen, 6% were sophomore, 2% were junior, and the remaining 1% were from other years. 25% of the students identified themselves as women, 73% as male, and the remaining 2% identified with another gender. 62% of the students were White, 18% Asian, 10% Hispanic, 2% Black, 0.2% Native Americans, and the remaining 7.8% marked other or declined to answer. Finally, 85% of the students were domestic students, and 15% were international.

Each section met for 110 minutes twice a week. All sections followed the same curriculum with common lecture material and resources, assignments, and exams. There were 18 activities in the course that corresponded to three exams, five deliverables for the main project, and ten problem sets. All the sections had different times for the activities, but they all converged in the order in which the activities took place

B. Learning Objectives

The course goals centered on developing students’ abilities to: (1) apply basic (MATLAB) programming concepts to the solution of engineering problems, (2) represent and interpret data in multiple formats, (3) develop, select, modify, and justify mathematical models to solve an engineering problem, (4) function effectively as a member of a team, and (5) demonstrate habits of a professional engineer. The first three goals of the course were translated into computational thinking practices. Goal number (1) refers to the familiarity with the programming language but also to the problem-solving process that involves the iteration between information-processing agents. This goal refers, particularly, to the computational solving practices proposed by [13]. Language is usually a minor part of this goal; in other words, computational thinking ideally would be separated from the programming language [25]. Goal number (2) refers to the data aspects of the taxonomy, data as the manipulation, analysis, and representation of data. Goal number (3) refers to the interaction between data and pattern recognition, aiming for structures to characterize a group of points in space. Goal number three refers to modeling and simulation practices; however, it was only assessed in team assignments.

This research work will focus on goals (1) and (2), basing the decision on the research questions and the main objective. The learning objectives (LOs) that addressed these goals were classified using the CT taxonomy proposed by [13]. The following lines present a review of CT practices that were assessed in the different stages of the course is offered.

C. Assessment

The assessment of the activities was at the level of learning objective (LO). The structure of the course allowed multiple assessments for every LO. Additionally, the LOs’ were evaluated at a more granular level, and based on those more granular levels, a score of competence was assigned. The four scores of competencies were: Proficient, developing, emerging, and insufficient. Two additional levels indicated if the student did not attempt the question or the entire activity. An example of a rubric used can be found in Appendix $A$ . Different graders performed the grading, with each grader grading approximately 30 students for the semester. Before they began to grade, they completed and passed an online training module for TAs in which they applied the rubric for scoring samples of students’ work and received feedback accordingly. Additional spot checks were given by TAs, randomly pulling out student’s work and checking the graders’ assessment to correct those that seemed unfair. Finally, a final check was done using the grades’ distribution by the grader to look at similarities and deviances in grading.

The LOs chosen for the current study were assessed more than three times during the semester on an individual basis. Table 1 presents the alignment between LOs’ and DP and CT practices. DP included objectives such as manipulating, visualizing, and analyzing data. CP practices included objectives such as troubleshooting and debugging, developing modular computational solutions, preparing problems for computational solutions, and programming. Besides, Table 1 presents the times in the semester as well as the number of times a LO was assessed.

TABLE 1 Number of Assessment Items Per LO and Time of the Semester

The construct validity consisted of aligning specific learning objectives of the course with computational thinking practices derived from our theoretical framework (see Table 1). To perform this alignment, the four investigators with expertise in computing and engineering education individually performed the alignment in an initial iteration. The four investigators then met during a working session to compare and contrast their alignments. When instances of different categorizations were encountered, a discussion was held where each investigator provided a rationale for their thinking. This process was followed until an agreement was made among all researchers.

D. Data Analysis

The semester was divided into three stages: early, middle, and late, based on its organization. The division of the semester corresponded to the course content. The first stage of the semester was considered an “exploratory stage,” where students would demonstrate their knowledge in different foundational practices. In comparison, the middle and late in the semester represent the different practices that the students acquired throughout the course. The early, middle, and late in the semester stages were structured as follows. Early was the combination of six problem sets and the first exam, the middle stage four problem sets, and the second exam, and the late stage was composed by the final exam and the project assignments. A sample of LOs related to each practice can be found in Appendix A.

1) Profile Method

To form student profiles, only the activities that occurred at the beginning of the semester were considered in the exploratory stage. Nine LOs were assessed more than three times early on in the semester (see Table 1). The feature vector for those consisting of 54 variables represented the proportion of assessments corresponding to each competency level (Proficient, developing, emerging, insufficient, problem not attempted, an activity not attempted). Accurately, each student was modeled as a feature vector. The feature vector consists of the proportion of assessments on a certain level of competency for a given LO. This step is required as it represents better the information that was stored as categorical. Practically, there are now percentages of every category for every LO per student. [26].

The N-TARP clustering method proposed in [27] is used to identify the essential features that define differences among students early in the semester. Subsequently, the N-TARP method was used to uncover binary clusterings among the feature vectors. The N-TARP method consists of the projection of the feature vectors (in this case the row vector containing student assessment information) on a random line through the origin, seeking a binary clustering within the projected data. The extent to which the projected data was clustered is measured using a quantity labeled W, which quantifies how the variance of the projected data compares to the variance of the best two clusters present in the data. A small value of W indicates a useful clustering. The data was projected N times (using N random vectors), and the projection that yielded the best clustering among those N was selected. Different runs of the algorithm yielded different clusterings of the data, performing 14 runs of the experiments (one per section), and setting $\text{N}=10000$ , to obtain 657 clusterings of the data.

Although there are many clustering methods in the literature, the N-TARP method was chosen for three main reasons. (1) As explained in [28], real data is often the result of many different and potentially unrelated phenomena or processes corresponding to different groupings; choosing a projection effectively consists of picking up one of these phenomena, viewing it as the signal, and discarding the other phenomena as noise. In contrast with other methods, which pick a single phenomenon and completely discard the other ones, N-TARP allowed us to uncover and collect several different clusterings. (2) There were ten individual learning objectives assessed over the semester, the combination of those learning objectives with the competency levels and the time in which they were assessed put the problem in a high dimensional space. N-TARP is both practical and fast in high-dimensional spaces. (3) The fact that the clusters were found in a one-dimensional space rather than a higher dimensional space made it easier to test for the statistical significance of the clusters found. (4) By using the current technique, the probability of putting two students in the same clustering when they do not belong together is lower than with the most commonly used methods. The likelihood of incurring in type I error, meaning the probability of having a false positive, is reduced significantly. Additionally, N-TARP has also been compared with other more commonly used algorithms to prove its advantages in high dimensional conditions [29].

Nevertheless, to test the statistical significance of the clusters found with N-TARP, the 14 sections of students within the course were exploited. Specifically, N-TARP was performed on the first section 10, 000 times, so to identify 10,000 binary clusterings of the students in the first section. Each of these clusterings is associated with a line on which the data is projected. To check whether the obtained clustering was significant, the 13 remaining sections of students were projected using the same procedure, looking for the persistence of the clustering structure in those sections. If the clustering found in the first section was a random fluke, then it was unlikely that clustering would be found in all 13 other sections as well. The randomness was quantified using the quantity W evaluated on each of the remaining 13 sections using the relevant projection line. Especially, clustering whose values of W were less than 0.36 for all 13 sections were selected. This criterion was interpreted as follows: if the null hypothesis that the data in the original high-dimensional space was normally distributed (no cluster), then since the probability that $\text{W} < 0.36$ was 1/2 [26], the probability that the null hypothesis was valid is 2⁻¹³, and thus one could safely reject the null hypothesis. Doing so allows us to select many statistically significant binary clustering of the student data.

These statistically significant clusterings were used to construct student profiles as follows: for each clustering, a record was kept on whether a given student belonged to the first cluster (using the digit ‘0’) or the second cluster (using the digit ‘1’). Then a vector containing the sequence of digits representing all the cluster memberships of the student was constructed. For example, a student belonging to the first cluster concerning all the clusterings would be represented by a vector of zeros: (0,0,…,0). Then students were grouped with the same profile together. A detailed flowchart of the method can be found in Figure 1.

FIGURE 1.

Flowchart of the method of analysis for the exploratory stage illustrates the method used for obtaining the profiles of the students with the information obtained early on in the semester.

2) Profiles Interpretation

After obtaining the profiles, some of them were selected for further analysis. The analysis focused on groups containing the most students, which lead to the selection of profiles that shared more than 1% of the students. Although profiles were obtained with the assessment data of the students, understanding its meaning required a subsequent analysis. An ANOVA and a simultaneous inference procedure were used on the grade point averages per LO. The simultaneous inference procedure using a sandwich estimator was reviewed by [30]. The ANOVA is an omnibus test that would indicate whether at least one pair of profiles differed for a given LO. The simultaneous inference procedure helps discover the pattern of those differences. The simultaneous inference procedure is a test that compares the means of every group for a given variable and uses the mean squared error from the ANOVA to evaluate the significance of the differences. In comparison, the ANOVA gives a general significance of the difference, and it does not allow the location of differences in specific groups. On the contrary, the simultaneous inference procedure gives the specific groups in which statistically significant differences are found.

The grouping of the profiles per learning objective was given by those profiles that showed significant differences between one and the other, using the results from the simultaneous inference procedure. In cases in which there were ties, the group chosen was that with the smallest p-value. After obtaining the different significant groups per LO, a radar plot was used to interpret the differences and give meaning to the profiles (see Figure 2).

FIGURE 2.

Radar plot with the LOs in which statistically significant differences were observed for profiles different from profile 4. The plot has been rescaled to have Profile 4 as a reference point and have a better visualization of the differences for the other profiles.

3) Progressions

After obtaining the profiles, using the data from early on the semester, an analysis of the progression trends of the students in the rest of the semester, as measured by their performance (grade point average) on subsequent activity, is performed. Each activity was viewed as a point in time, with the second exam being the first point in the progression (T1), followed by four problem sets (T2, T3, T4, T5), and finishing with the third exam (T6). Every individual activity assessed after the periods considered for the construction of the profiles is included for the progressions. The performance of the profiles was compared concerning each LO separately. The standard error at every time of the semester is included to measure the differences relative to the variance.

SECTION IV. Results

A. RQ I: What are the Differences in Students’ Computational Thinking Practices at the Beginning of an Undergraduate Introductory Programming Course?

N-TARP resulted in 18 statistically significant clusterings. Remarkably, the 1559 students were found to belong to a total of merely 657 profiles (out of the $2^{18}=262144$ possible profiles). An analysis of the profiles shared by more than 1% of the students took place, which resulted in four main profiles of interest. Those four profiles, viewed as “dominant,” describe 485 students from the 14 sections. The remaining students were too inhomogeneous to be accurately studied with this framework.

The student performance on the nine LOs for the four groups was compared using ANOVA applied to the beginning of the semester performance data (Table 1). Five of those LOs belonged to the DP major practice (manipulating data and visualizing and analyzing data). In contrast, four of them belonged to the CP major practice (Troubleshooting and debugging I and II and developing modular computational solutions I and II.) The ANOVA showed an F value that was significant for all the LOs. A simultaneous inference test was developed to understand the differences between profiles for every learning objective (Figure 1). However, for three LOs (LO1, LO2, and LO3), the difference that was shown significantly corresponded only to Profile 4. This profile was observed to correspond to learners that were low performing for all the LOs. The remaining six objectives showed differences between the other profiles.

The simultaneous inference test (see Table 2) encountered that for LO1, Profile 1 and Profile 2 were not different, while Profile 3 and Profile 4 were different, with the group conformed by Profile 1 and Profile 2 being the group that performed better on average. A similar interpretation can be obtained for the rest of the LOs.

TABLE 2 Significant Groupings Using the Multiple Inference Procedure for Heteroscedasticity Proposed in [28]. The Alphabetical Order Indicated the Performance Order of the Groups

TABLE 3 Four Main Profiles With Description

Based on the results from the multiple inference procedures (Table 2) and the radar plot (Figure 2), the profiles are described as follows: Profile 1, the most substantial profile with 299 students, scored well overall, but students in that group performed lower than Profile 2 on troubleshooting and debugging II (LO4) of the CP category. Profile 2 characterizes students that showed a good performance on all the objectives, with similar behavior to Profile 3 except for troubleshooting and debugging I (LO1) and Troubleshooting and debugging II (LO4) in which Profile 2 had a better performance than Profile 3. Profile 4 were students that performed lower on all the practices evaluated early in the semester. From the results of the interpretation, one can observe two strong relationships between some of the profiles. (1) Profile 1 and Profile 4 are opposite profiles. While Profile 1 is composed of students that can be considered high-performance students, Profile 4 are students that in all LOs evaluated performed lower than the rest of the groups. (2) Profile 2 and Profile 3 were students that behaved similarly in all the learning objectives, but in those that pertained to troubleshooting and debugging, in which Profile 2 showed mastery while Profile 3 showed a lower performance.

The assessment behavior of these learners after the first part of the course, where more advanced topics were taught and evaluated, is then analyzed. It is important to note that in the middle and end of the semester, the tasks were related to computational solutions and programming. Practically, the progressions of these learners on some other practices that were also included in the CT taxonomy are analyzed.

B. RQ 2: How Do These Differences Correspond to the Acquisition of More Advanced Computational Thinking Practices?

For the comparison, the trends for all four profiles are analyzed in the middle and end of the semester. There were three assessed computational practices in the middle and end of the semester. Developing modular computational solutions (LO11 in Figure 3), Preparing problems for computational solutions (LO15 in Figure 4), Programming I (LO16 in Figure 5). The obtained results are analyzed for the progression of these practices, starting with the development of modular computational solutions.

FIGURE 3.

Developing modular computational solutions progressions (LO11). The vertical lines at every time point show the standard error.

FIGURE 4.

Preparing problems for computational solutions progression (LO15). The vertical lines at every time point show the standard error.

FIGURE 5.

Programming I progressions (LO16). The vertical lines at every time point show the standard error.

When evaluating LO11, it is possible to observe that all the sections show a declined progression (see Figure 3), which can be possibly attributed to an increasing level of difficulty of the tasks proposed to evaluate this LO. Specifically, the involvement of new tools and more complex problems were introduced as the semester progressed. Additionally, although Profile 2 and Profile 3 were similar during the early stage of the semester, the difference between their demonstrated ability in troubleshooting and debugging related objectives seems to have had a substantial effect on the progressions on LO11 during the middle and end of the semester. It is possible to see in Figure 3 how the gap between Profile 2 and Profile 3, both profiles that were intermediate performers, changed during the rest of the semester. The progression shows an enhancing difference between Profile 3 and Profile 1 (high-performance group) at the same time that Profile 2 behaves similarly to Profile 1, which is a high-performance group at the beginning of the semester.

From Figure 4, it is possible to observe that there is a low decline in the performance for this objective for all profiles, with Profile 1, Profile 2, Profile 3 merge into a unique group at the end of the semester, while Profile 4 has a lower performance at this same time. Additionally, a gap between Profile 3, and Profile 1 and Profile 2 in T1, T2, and T3 started to diminish after T4. Considering that Profile 3 are learners that in the early stage of the semester had a lower performance in the objectives related to troubleshooting and debugging, the gap observed may correspond to the importance of this objective in the progression and the possible delay in the attainment of this practice for these learners.

From Figure 5, a similar gap to the beginning of the progression of LO15 is found here with a shorter length. Profile 3 shows a later improvement in the performance of this competency as compared to their peers. However, all students from the different profiles reached a high performance in programming I by T6. In addition to these results, a lower performance for Profile 4 is observed in all progressions. This lower performance can be explained by a possible student’s lack of motivation from the beginning of the semester that is continuous and translated to low performance in the rest of the semester. Surprisingly, the students in profile four reached the students in the other profiles in T6. There exist at least three factors can lead to these phenomena (1) Although T6 corresponds solely to the final exam, which was assessed individually, the teamwork that occurs in the late stage of the semester might have some contribution to the higher performance of students in the LOs evaluated late in the semester. (2) It is also possible that there is extra motivation for students that are having a low performance to do well in the last exam (3) Finally, the assessment of this LO shows a minimal variation and can be characterized by some particularly easy activities.

SECTION V. Disscussion

This study investigated the progression of computational practices in the context of a first-year engineering course. In this study, the characterization of four different student profiles according to this foundational knowledge is obtained. Whereas the findings are descriptive, they allow to identify how students performed differently throughout the semester based on their performance previously. While there is literature asserting that CT is complex and dynamic, requiring multiple competencies to work simultaneously [11], previous much of the literature has identified the composition of CT through expert opinion and conceptual frameworks [12], [13]. This article extends their work by applying the conceptual framework of [13] to empirically study students’ development of CT in an introductory course.

Similar to previous research, there was considerable variation in student practices at the beginning of an introduction to programming course [17]. In addition, students in this study had an essential connection between troubleshooting and debugging and further progressions in the next stages of the semester. Mainly, the primary difference between performance in this practice shows delays in the achievement of continuous gaps in the attainment of more complex CT practices. This finding corresponds to previous work, which described debugging as an essential ability that students need to master in order to accurately correct problems encountered when working on the solution of problems with computational tools [13].

The assessment of computational practices at the beginning of a first-semester programming course is vital for the study of computational thinking (CT). Although there is little to no agreement on its composition or assessment of CT [15], empirical studies will help build the CT literature by choosing a theoretical framework and adding content to what until now has been purely theoretical. At the assessment level, this study builds on the future assessment of CT amplifying the need for a complex in-classroom assessment system that enables the assessment of the CT practices at a granular level and at multiple times.

Implications for assessment relate to the need for new methodologies to analyze learning data so that it is possible to understand how learning materials can be made more accessible to different groups of students. That is, with enough data and more comprehensive analytical approaches, it is possible to understand better the different groups of students and possible ways to foster the CT mindset in these specific groups. It is vital to systematically collect information and use learning analytics tools, as it is data visualization, to understand data in the first stage. Besides, the cleaning and coding procedures need to be well documented so that further decisions could be translated into the results. Finally, the choice of techniques for data exploration needs to show evidence of reliability.

This study has some limitations. One limitation is that the data analyzed was classroom assessment data. Thus, the measures are dependent upon how the course instructors formatted the assignments and how they elicited students’ skills and knowledge. Another limitation is the possible noise created by the use of multiple teaching assistants and that it is not accounted for in this study. However, all graders and teaching assistants were required to participate in training and calibration activities before grading. Future research can provide an experimental design to offer less external variability in the classification.

SECTION VI. Conclusion

While the importance of CT has been discussed widely, empirical studies provide a unique perspective of how CT can be characterized and assessed. Typical of what most instructors observe, this study’s findings (presented in Figures 2–​5) supported that some students performed well on all assignments throughout the semester, and other groups of students consistently performed poorly on all assignments. As expected, the importance of acquiring a strong foundational knowledge of computational problem-solving practices is critical to support the learning of advanced practices. In particular, students who did not master the beginning practices associated with preparing the problems for computational solutions later struggled to develop practices related to the computational solutions’ actual implementation. However, our study goes beyond this finding to also identify a specific practice, troubleshooting and debugging, that created a critical gap and delay in students that had a similar performance at the early stage of the course (Profile 2 and Profile 3). Therefore, this finding suggests that students must grasp the computational problem-solving practice of troubleshooting, debugging early in introductory courses.

Additionally, with information from early in the semester, it was possible to identify students that belonged to a low-performance group reliably and that showed low-performance progressions in posterior evaluated practices. The identification of these students opens up opportunities for research to identify and further understand the reasons for these students to have low performance and possibly create possibilities for them to engage. Considering the previous experiences of students in a first programming class will help instructors to leverage the knowledge of individual practices and design courses that can include different students’ experiences.

National and international calls have emphasized the need to prepare engineering students with data science and computational thinking [9]. In a rush to prepare students, time must be given for students to master the computational practices some may consider as rudimentary or unnecessary in an undergraduate course. Given that students have a variety of K-12 data science learning opportunities, instructors and programs should consider developing courses for groups of learners or how to differentiate instruction within large classes, rather than having all students go through the same course materials.

Finally, this study was possible because of the implementation of a very complex grading system that allowed the assessment of every computational practice multiple times, based on rubrics and levels that provide enough information for characterizing the profiles [31]. This type of assessment presents a possible solution for the empirical studies of computational practices in the future.

ACKNOWLEDGMENT

The authors would like to especially thank Prof. Heidi Diefes-Dux for the design and implementation of the assessments and sharing the data collected. This work was supported by the U.S. National Science Foundation under Award EEC #1826099 and Award EEC #1449238. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Science Foundation.

Appendix A

See Figure 6.

FIGURE 6.

Example of a grading rubric for LO 4.

Appendix B

See Table 4.

TABLE 4 Learning Objectives and Corresponding Practices