Normalizing Class XII Marks

3 Comments

Recently it was reported in newspapers that CBSE moderates its marks by effectively increasing them – largely to “compete” with other Boards. It was also reported recently that some of the top colleges in Delhi have a majority of students from one Board in south, and a large number from one school.

Both of these anomalies are due to one reason – admission being based on Board percentage without normalizing the marks of different Boards. Due to this, Boards realize that their students will benefit if they have a higher percentage – so there is a race to give more marks. Besides distorting the admission process, this race is unhealthy for education and learning and gives a false sense of achievement to students.

Of course the natural course of action is to normalize marks from different Boards. Normalization across Boards can be easily done – all it requires is a little extra information from each Board. For some strange reason, it has not become regular practice and Boards do not provide sufficient information for normalization.

Let us first understand the normalization problem. Each Board gives marks between 0-100%. Normalization requires that marks between 0-100 given by different Boards be converted to a “normalized score”, also between 0 and 100, which provides a common reference where X marks mean the same, regardless of whether the Board was “tough” or “easy” in its marking.

One approach to normalize, which CBSE also used for JEE, is to base it on the percentile of a student in the Board. The percentile score of a student reflects what percentage of students in the Board have marks below that of the student. I.e. a student with 90 percentile means that 90% of the students have received marks that are below this student’s marks. To convert marks to percentile score, students are ordered in the order of marks they received, and then divided in 100 equal groups – the top 1% students fall in 99-percentile, the next 1% fall in 98-percentile, etc. (This can, of course, be done at 0.1 or 0.01 percentile granularity for finer resolution.) Once percentile is given, then there are ways to normalize, with the assumption that top N% of the students in a Board are essentially similar to top N% students in another Board, i.e., a 99 percentile student in one Board can be considered similar to a 99 percentile student of another Board. (If this assumption cannot be made, then it will require calibrating different Boards – an exercise that is unlikely to be undertaken, and if initiated, unlikely to culminate in an acceptable calibration.)

With a percentile score, one way to normalize across Boards is simply to use the percentile score. In this case, a student with 99.5 percentile (from any Board) will be ranked higher than a student in 99.3 percentile (from any Board). The percentile score can have finer granularity, if desired, and within each percentile, there can be tie-breaking rule.

The ranking with percentile is sufficient if the decision of admission is based only on class XII score, i.e. one needs to rank or order students only on class XII score. However, if admission is based on sum of multiple scores, in which one of them is the class XII marks (as was the case in JEE, where 60 marks came from JEE exam and 40 from class XII), then the situation is more complex and percentile will have to converted to a normalized score to be added to the other scores.

There are techniques to convert the percentile score to a normalized score. For this conversion, a desired target distribution is needed, which gives what fraction of students should be at each mark in the normalized marks between 0-100. The target distribution for normalized marks is a choice to be made, and any reasonable distribution can be chosen – the preferred distribution for exams is Normal Distribution, in which the largest fraction of students is at the mean and then the fraction at each mark reduces as we move on the two sides of the mean. If the target distribution is taken to be Normal Distribution with mean of 50 and the standard deviation (a statistical attribute indicating the variability in scores) of 15, then 99 percentile will translate to 85 normalized marks, 98 percentile to 81 marks, 95 percentile to 75 marks, 90 percentile to 69 marks. There are standard tables available for this conversion. (This conversion will be different if a different mean and/or standard deviation are selected.)

It should be clear that with percentile based normalization, inflating marks in a board does not help students – the top 1 % students of all boards will be mapped to the same normalized marks. So, in a Board which has inflated marks such that a large fraction of its students get above 90% marks, only the top 1% students will get the same normalized score (99 percentile, or 85 marks in the above example), which will be same as the top 1% students from a Board which has a much smaller fraction of students above 90%.

Overall, normalization can be done easily and transparently if information about percentile of a student is provided by the Boards.  If Boards provide the percentile, normalization is straightforward.

Normalization is not possible if Boards only give the percentage marks to students, as they do now. Though determining percentile is trivial and Boards can easily do it, for some reason, it is not being done. Perhaps because just with marks a Board can have as many students above 90% as it wants and let the students and their parents feel good. With percentile only 10% of the students can be in the 90 percentile – which will give a clear picture to the student about his/her relative standing in the Board. Given this situation, probably MHRD will have to mandate that percentile information must also be provided to the students. And to support this move, all universities that use class XII marks for admission can declare that they will normalize marks of different Boards and will therefore not admit students from a Board unless the percentile scores are also made available – if this is done all Boards will have to provide this information.

If normalization is done, besides fairness in admissions, it will also lead to curtailing of the unhealthy exercise of marks inflation that the boards seem to have gotten into. If normalization is not done, given the publicity received about DU’s admission this year, we will see an unhealthy race between Boards to give easy marks, resulting in a complete failure of merit based admissions, with less deserving students from some Boards getting admission into the best colleges in the country at the cost of more deserving students from some other Boards being denied admission.

Widen the Entrance Criteria in Higher Education Institutions

6 Comments

It is well established that good quality higher education is the best way to open doors to a variety of opportunities – that is why world over students vie to get into the best universities and colleges. Due to this, while school education is meant to lay the foundation for a broad development of the individual, the single most important goal of school education becomes getting admission in a high quality higher education institutions (HEI).

Admission to our HEIs is based almost exclusively on performance in exams – class XII or entrance test. Most engineering institutes admit students through entrance test, though now class XII marks are also given weight, and most universities like Delhi University give admission based on class XII marks (though have some seats for sports etc). So, regardless of what educationists may like to see, students, parents, and teachers all eventually align to a single goal as outcome of school education – doing well in class XII exams and competitive entrance tests. As nothing else matters for achieving the important goal of getting into a high quality HEI, other aspects of development that the school education is supposed to provide, are mostly ignored.

As a result of  this exclusive focus on exams, a student who does innovative projects in schools demonstrating innate talent and interest for engineering is precisely the one who may not make it to the best engineering institutions as he/she “wasted” time doing these projects – time which could have been more optimally used in coaching classes. Similarly, a student who does internship in some company and writes a report on the economics of a sector – perhaps the ideal candidate for an economics program – may not be able to get into a good economics program as others who spent all the time preparing for exams get higher marks. Similarly, students who engage in school debates, participate in social work, sports, or other activities that can broaden their development and horizons, are often at a disadvantage for getting admitted to HEIs as compared to those who spend their time preparing for tests. This uni-focus on attaining high test scores also inevitably leads to shallow learning styles which maximize performance in tests but prevent deep understanding of subjects.

This focus on exams cannot be changed just by exhortation or changing the pattern of the exam or bemoaning the state of affairs. We have to squarely accept the fact that the most important goal for a student is indeed getting admission into best colleges, and if we want students to have wider development in schools, we have to widen the criteria for admission to include achievements and efforts outside tests.

One direct approach can be to assign some marks (say 20 out of 100) for achievement in other spheres while the remaining 80 can remain based on results of class XII and entrance test. With this, the problem reduces to developing sound procedures for assigning marks out of 20 for achievement in other spheres. This will be a challenge but not one that is unsurmountable – PG/MBA programs or public service exams routinely do this, by having an interview and assigning some weight to it.

IIIT-Delhi has been following another approach for the last few years for this. In IIIT-Delhi, for admission in BTech program, up to 10 bonus marks (on a base of 100) are given for achievements in various spheres, through a published criteria. For example, bonus marks are given to students who reach final stages of various Olympiads, participate in national school games, have Chess FIDE rating, get an award in the INSPIRE or IGNITE program, win prize in programming contests, have ministry of culture’s scholarship for talent, etc. The program was slow to start, but in the previous two batches, over 10% of the students admitted are ones who have received bonus marks.

We have also done some analysis of how these students perform in our Institute. As we had anticipated, the first year performance in the Institute of the students who had received bonus marks was significantly better than the performance of students without bonus marks (the average CGPA was higher by more than 1.) This clearly demonstrated that students with broader base are likely to be better prepared for higher education.

Most US universities, while giving a considerable weight to SAT scores and performance in high school, consider a host of other factors and achievements for admission. In fact, in top universities it is now known that just good grades and SAT scores are not sufficient, and students must show other achievements. This hugely motivates families to develop other aspects of a students’ personality – sports, culture, social work, volunteering, etc. If we start incorporating achievements and contributions in other spheres in admission to most of our top HEIs, we may also see an increase in motivation and drive to undertake such activities in school – this can only be good for our students and their development.

Assessing the Value of High Quality Education

4 Comments

How good is the education being provided by an academic Institution? While there can be sophesticated ways of assessing the quality (e.g. by having peope who understand education and its goals assess the quality by looking at the faculty, courses, method of teaching, etc.) often quality of the overall education is assessed by how well their graduates do in their careers. And an indicator of this is how good a start the graduates get, i.e. the placement record of Institute.

Most good institutions have good placement records. While good institutions have good placement/careers statistics, there has been a perpetual debate/doubt whether this outcome is due to the fact that these institutes, being higly selective, take in the brightest of the students (if you take the top 5% of the students, of course they will do well – the argument goes), or whether it is due to the value of education they provide.  All of us academics in top institutions believe that while high quality of intake has a role to play, the quality of education provided at these institutions is the key determining factor in the success of these students. Unfortunately, this point cannot be supported by data as it is not possible get the desired data (e.g. average students taken in the top institutions and provided good education and then seeing how they do.)

At IIIT Delhi, as an Institute which has now evolved into a sought-after, high quality institution, we have a limited data from the first batch to shed some light to this debate. In 2008 the Institute was started – I was appointed the Director (employee #001) in Aug, and we had to start the new session in Sept. As all the admissions were over, the “best” students were already gone – even if they were not, why will they join an Institute that just started and had no faculty or facilities. In any case, we did an entrance test with an eligibility criteria being 60% in class XII.  About 350 students applied, finally around 250 or so appeared, of which we selected 60 – and many of those who were offered did not accept it. Effectively, almost one in every three students who wanted was admitted.   (Now that we have established ourselves as a top Institution,  of those who apply – and only those above 80% in class XII can apply – about 6% are selected.)

So, clearly the first batch was hardly selective by Indian standards. More so, given the fact that admissions in almost all major Institutions were over when we started our process (our exam was held in Aug!).

After admisison, this batch was taught in their first by guest faculty in borrowed facilites, but using our program and curriculum. In second year, many of the courses were taught by our faculty. It is only by third year when almost all the Computing courses were being taught by our faculty. In other words, whatever is our quality of education now, the first batch did not get it fully.

Fast forward to 2012, when this batch graduated. The Institute had no track record and no placement history – making it extremely hard to convince companies to visit for campus placement, particularly since we wanted to focus on technology companies as we felt that our students were far too well trained for the “regular” software jobs (by this I mean the entry level jobs offered by the large software houses in the country and where perhaps 90% of the CS graduates in the country get their employment. These jobs, it is well know, have a starting package of about Rs 3 to 3.5 Lac). With effort, we were able to convince some of the companies to come.

Almost every company that visited, despite their initial doubts, recruited some students.  Here we are talking about tech companies that generally have a rigorous selection process – test, multiple rounds of intereviews, etc. Some companies were so happy by the quality of graduating students that they recruited far more than what their best case scenario was (as we were told.) One company, which came for recruiting for their research lab, then called in their other product divisions to recurit. Some highly selective companies which visit few campuses in the country told us that they felt that the graduates are as good as other places they visit.

The final outcome in numbers: The average salary offer is about Rs 7 Lac (twice that of a regular software job), with majority getting offers of more than Rs 6 Lac, and about 10%  getting offers of more than Rs 10 Lac.

Personally also I have interacted with many (and have taught them some courses) and I can see that the capability of these students, their confidence, and their aspirations are so much higher than the CS graduates we often see in IITs for MTech admission – which are arguably the best from the engineering colleges. In fact, when the batch was in final year, some of my colleagues would sometimes say that the quality of these students is comparable or better than the later batches.

I believe this unique data point, which even we cannot now replicate, provides a limited argument for the intrinsic value of education. For the first batch, which was not very selective in admission, purely by providing good quality education, we have changed lives of many of them. It can be easily argued that these students, if they had not joined IIITD, would have ended up in some engineering college (I have collected this data – most of the students would have otherwise joined a college in Delhi or NCR) from where they would have graduated and most likely landed the “regular software job”. Our education has transformed them and has changed their career trajectories substantially by giving most of them a good start for buidling strong and successful careers.

So, while high quality of students intake is desirable and all Institutes vie for the best students, it is the quality of education provided by Institutes that makes the graduates what they are capable of. Quality of education matters and matters hugely!

Entrance Tests and Admissions to Colleges

3 Comments

When there is a large pool of applicants for an education program, job, etc. and only a few seats, one needs some selection method to select the required number from the large pool. The basic goal of a selection method is to identify those candidates who are most suited for the program/job, i.e., they are most likely to perform well in the position.

Tests are often used as an instrument in this selection process – the scores of the test provides one of the parameters in the selection. For admission to many higher study programs as well as many jobs, besides scores in some test, performance in interviews, statement of purpose, performance in undergraduate studies, etc are also used in the selection method.

When it comes to admission in colleges for undergraduate programs, although a multi-dimensional criteria can provide a better method for selection, we often use a single dimension approach: score in some admission test, or percentages in class XII. This method provides transparency and fairness, but is unlikely to be the best method of selecting the most suitable candidates.

At IIIT Delhi, we have been trying to implement multi-dimensional criteria. This note describes the evolution of the approach for entrance used by IIITD.

Initial Approach. We noted that all engineering entrance tests focused on Physics, Chemistry, Math (PCM), but did not give any importance to aptitude. In the US the SAT exam, whose scores are an important input for admission, is a general aptitude and thinking ability test.  We believed that students who are good in PCM subjects as well as aptitude are the best for a rigorous education system that we had put in place in IIIT Delhi.

Hence our initial admission criteria was – only students who have more than 80% in PCM in class XII will be eligible to apply for admission to IIIT Delhi. These cutoffs ensured that the students have a decent background in these subjects. These students had to take an entrance test, which was primarily an aptitude test examining comprehension, logical thinking, and quantitative thinking. Of the eligible students, we selected those who performed well in this test, giving us those students who did well in PCM in school, and who scored well in the aptitude test.

Data Analysis. We used data from the last few years to study how well various criteria at entrance time predict the performance of the student in our rigorous program in first year. We studied the predictive power of AIEEE, class XII marks,  and scores from our aptitude test.  In other words, we studied how strongly the performance in the entrance exams or class XII predicts performance in the first year of our CS program.  We also used data from a few well established and respected IT Institutes for the study.

What we found is interesting.  We found that the strongest correlation (of more than 0.5) with performance in IIITD (and other institutes) was of Class XII marks. That is, class XII performance is a reasonable (but not perfect) predictor of how well a student does in these Institutes.  We also found a decent correlation between the performance in the aptitude test that we conduct for entrance and the performance in the Institute. Of the three parameters we studied, the rank correlation of AIEEE with performance in the Institute was the lowest.

We further studied the impact of two main factors with decent correlations – the class XII score and score in the aptitude test.  We identified two groups of students – those who had a CGPA of 8.0 or above, and those who had a CGPA of 6.0 or below. We found that the average class XII score of the first group was almost 10% higher than the average of the second group. Similarly, we found that the average score in the aptitude test was also about 10% higher for the first group.

Refinement. The analysis indicated clearly that students who are high performers in class XII and have high aptitude scores are likely to do well in our education system. To select these students it was clear that class XII marks should be given more weight for admission.  Based on this analysis, we decided to give 50% weight to class XII marks, and 50% weight to an aptitude based entrance test.  This is the entrance method we used for 2012.  In other words, though giving importance to class XII marks for admission may a desired objective from the point of view of strengthening the school education, we gave weight to class XII marks mostly because our data showed it as the strongest predictor of performance.

Scope for further refinement. We were able to include in our selection two criteria which we thought were the most important and which could be included in a transparent manner – class XII marks and score in an aptitude test.  This year we also gingerly took some step towards including other factors – we decided to give up to 10 bonus marks for those selected for the Math or Informatics Olympiad in India. (In these two, a total of about 50 students are selected – this time none of these students completed the application process fully for IIIT Delhi.) We hope to expand this approach to find ways to include some other important factors in a transparent manner which will help IIITD select better students.