Accuracy of Pupil Reporting
- Author:
Lewis, K. J. Hoke, J. B. Welles, and G. M. Wilson,
Teachers College, New York City, N. Y.
During the past few years many studies in education, particularly those concerning retardation and promotion, have been based upon the answers given by children to questions asked by the investigators. While this method of collecting data has been known to be inaccurate, no one has attempted to see how much the error was nor in which direction it lay, that is, whether it was in favor of the school examined or not. In order to get some idea of this problem, a study was made among the school children of a city near New York,* who had attended the schools for the past six years and whose cumulative record cards for this period were complete. The children were a selected group in that they had spent their entire school life, six years in extent, in this particular school system. It was found however on careful examination that the selection with regard to race and financial circumstances was really a random one. The age distribution as well as the grade distribution was normal. The distribution by grades was,? grade three 9, grade four 33, grade five 54, grade six 74, grade seven 5, grade eight 1, total 176. The distribution by ages was,? 5 nine years old, 41 ten years old, 71 eleven years old, 42 twelve years old, 12 thirteen years old, 4 fourteen, 1 fifteen; total 176. There were 63 boys and 93 girls, total 176. The pupils were asked to fill out the following blank, and the returns were secured from 137 of the 176 children in the six year group:
Name 1. Have you ever skipped a grade? 2. What grade or grades have you skipped? 3. Have you stayed two or more years in any grade? 4. In what grade or grades have you stayed more than a year? * The authors desire to recognize their indebtedness to Dr G. D. Strayer, Department of School Administration, Teachers College, under whose direction this study was undertaken, and whose co-operation made it possible. The original data for this study are on file in Dr Strayer’a office.
To avoid too much disturbance and to have the questions answered under normal conditions, the teachers gave them out and returned them to the investigators. In one school the children were assisted by the teachers in recalling their school-life, and in another school the children had been cooperating with the principal in the correction of their record cards just previous to this study. These facts would tend to make a slightly larger percentage of the answers correct. The investigators carefully examined the cumulative record card of each pupil and checked the answers accordingly. In checking up the answers, if the pupil said that he was retarded one year in the second grade, but was recorded as having been retarded one year in the first grade, his answer was accepted as correct. This liberal construction tends further to make the percentage of correct answers too large. If the pupil reported an answer in any other way than that shown above, it was considered incorrect. Table I shows the results in detail.
TABLE I. ANSWERS OF 137 SCHOOL CHILDREN. Normal Children o O 51 Incorrect Say ret. Say accel. Accelerated h o O Incorrect Say normal Retarded a o O 31 Incorrect Say normal 27 Say accel. 17 1 yr. too few 2 yrs. too few No ans.
A double scoring was used in the case of double retardations and promotions. By this method the actual number of pupils’ answers will be found to exceed the number of pupils by eleven. Reading the table from the left we find that of the normal children fifty-one gave correct answers, four incorrect; two of these saying that they had been retarded and two accelerated. Of the accelerated children three answered correctly and one incorrectly, the one saying that he was normal. Of the retarded children, we find that thirtyone answers were correct, and of those answering incorrectly, twentyseven said that they were normal, seventeen that they were accelerated. Four reported that they had been retarded, but reported one year less than the records showed and four two years less than the records. Finally we find that six gave no answer at all. Reducing this table to a percentage basis, and charting it, we find that 93 per cent of the children who were normal answered correctly, while 3.5 per cent said that they were retarded. Among the accelerated children the small number of cases distorted the percentages; one child said that he was normal. The most significant facts were found in the answers of the retarded children. Sixty-five per cent of the answers were incorrect.
This percentage was distributed among the different mistakes as follows: 30 per cent reported normal progress, 19 per cent said they were accelerated, 4.5 per cent reported one year less than they bad actually been retarded. Another 4.5 per cent reported twTo years less retardation than they had actually had, and 7 per cent gave no answer. It may be evident therefore that normal and accelerated pupils usually know what happened to them, but retarded pupils do not recall readily their past life in school. The fact is brought straight home to every school man that no accurate study of grade progress can be made without fully kept cumulative record cards. Let us apply the results of this inquiry to a well known study based upon the answers of pupils as to retardation or failure,?” The Incidence of Retardation” by Dr Louis B. Blan (Teachers College, 1911). This appealed to the writers as a splendid study; but their scientific interest led them to the plan as described above for checking up, for proving or disproving the results.
Blan studied 4579 children as follows: a New York City district, 1312 cases; Elizabeth, N. J., 1088 cases; Paterson, N. J., 1246 cases; East Orange, N. J., 448 cases; Plainfield, N. J., 485 cases. By a method identical with our own as shown by an examination of Blan’s paper, pp. 22, 32, and 36, the 4579 selected pupils in the five cities were studied as to the grade distribution of the 3947 cases of retardation. These were reduced to a percentage basis for each group and then the median of the five groups calculated. This median for the fifth grade pupils is:
Per cent of failures in the fifth grade 11.2 ” ” ” ” ” fourth ” 9.2 ” ” ” ” third ” 7.2 ” ” ” ” ” second ” 6.6 ” ” first ” 7.2 Since our study tends to show that in the lower grades these figures are much too small, it is only fair to call attention to the evidence in Blan’s own study tending to support the conclusion which we reach. We are surprised that so careful a student should have neglected this evidence, tending as it does to prove the unreliability of a pupil’s report as to his own failure, a conclusion which if substantiated would invalidate his entire study.
Blan says (p. 31): “Each pupil was asked to state in what grade or grades he had been kept back for a second term. In the event of the pupil’s inability to remember accurately, a note was made of such inability and recourse was had to the record card filed in the office of the principal. Such action was necessary in 32 out of 1312 cases and the writer was particularly fortunate in obtaining full records of same.”
The above quotation shows how little use was made of the cumulative record cards in checking up pupils’ answers. Blan (p. 52) notes that the higher the grade the fewer failures there are reported for lower grades, but it does not seem to occur to him that this failure to report may have been due to the inability of pupils to remember the failures that occurred so many years before. Blan says: “In fine, then, it may be said that the fourth or fifth grade pupil is left back in the third grade more than twice as often as the eighth or seventh grade pupil.”
In Plainfield, N. J., there were cumulative record cards which Blan could have used, but of which he made little use. It appears however that they had some effect in helping the memory of pupils. The 485 pupils in this system reported 616 failures, or a larger proportion than in any of the other systems involved in the study. The first grade failures reported in this system are from five to eight times the median per cent of the other systems, and range from 30 to 39.2 per cent. In connection with this system Blan says: “The writer, however, visited each class room of the fifth, sixth, seventh, and eighth grades and questioned the pupils individually as in the other cities… . Their memory seemed to wane only in the case of the primary grades… . Wherever it was found that pupils deliberately misstated the facts, their records as read from individual history cards were invariably somewhat worse than they cared to admit. For example, when pupils replied that they were left back a given number of times, on checking up their statements it was found that in no case were they left back less than they stated. On the other hand, in quite a number of instances their history cards showed one or more retardations above the number admitted by them in class.”
The values to be attached to the expressions “somewhat” and “quite a number” in the above quotation can be best estimated from the details of table I of this study. It must be evident that if 65 per cent of pupils’ answers as to retardation are incorrect, the only profitable way in which to study this question is through an examination of the individual cumulative record cards. Let us proceed a step further by distributing the retardations as actually occurring and as reported by the children, and then apply these facts to Blan’s study. Table II shows the distribution of reported and unreported failures, and the total of the whole number actually occurring.
TABLE II. Grade. Number of Retardations reported V IV III II I IV. III. 6 7 5 12 Number of Retardations not reported IV 1 1 III 3 1 2 II 16 I 14 22 4 13 Total Retardations. V 1 IV 4 2 III 6 2 3 II 14 21 5 I 21 34 13
Taking the first line, table II reads, beginning at the left, the fifth grade pupils reported 1 retardation in grade five, 3 retardations in grade four, 3 in grade three, 6 in grade two, and 7 in grade one; and they failed to report 1 retardation in grade four, 3 in grade three, 8 in grade two, and 14 in grade one; the distribution of total failures among fifth grade pupils being 1 in grade five, 4 in grade four, 6 in grade three, 14 in grade two, and 21 in grade one. The next line of table II shows the distribution of fourth grade retardations; the last line, the distribution of third grade retardations. It will be seen from the table that fourth grade pupils report more inaccurately than fifth grade pupils, and that the third grade pupils do poorest of all.
The only place where this table overlaps Blan’s is in the fifth year. It will be interesting therefore to calculate the incidence of retardation for the fifth grade and compare the results with the figures in Blan’s table. Since Blan used pupils’ reports, however, it wall be proper for us to use pupils’ reports in making the comparison. This comparison is shown in table III. This table is derived directly from table II. The 4.7 per cent under grade I of table III, simply means that 7 in table II is 4.7 per cent of 148,?148 being the total number of cases involved, including normal, accelerated, and retarded, each retardation being counted as a separate case. Likewise 6 in table II is 4.0 per cent of 148, etc.
TABLE III. SHOWING THE INCIDENCE OF RETARDATION FOR FIFTH GRADE PUPILS ACCORDING TO THEIR OWN REPORTS. Grade As per pupils’ reports. .7 IV 2.0 in 2.0 II 4.0 4.7 Blan, based on pupils’ reports. 11.2 9.2 7.2 6.6 7.2
Table IV is derived from table II in the same maimer, but the totals for grade V are used. That is, 21 is 14.2 per cent of 148; 14 is 9.5 per cent of 148, etc. The fifth grade incidence of retardation thus derived is based upon the total retardations, the retardations reported by the children plus those not reported. The figures showing the corrected incidence of retardation for Blan’s study are TABLE IV. SHOWING THE CORRECT FIFTH GRADE INCIDENCE OF RETARDATION FOR THIS STUDY, AND THE CORRECTED INCIDENCE FOR BLAN’s STUDY. Grade This study as per record cards .7 IV 2.7 III 4.1 II 9.5 14.2 Blan,allowing for errors of pupils’ reports 11.2 12.4 14.7 15.7 21.7 secured by simple proportion in each case, and assume that the reporting of pupils in his study was inaccurate to the same degree and in the same direction as the reporting of the children in this study. If this assumption is correct, and Blan’s own study bears evidence that it is, the conclusions reached by Blan are equally unreliable. The situation as to failure in upper as compared with lower grades is exactly reversed.
It is not necessary to pursue the discussion further. While this study is not conclusive, it points strongly towards the following conclusions:
First.?Pupils’ reports as to progress through the grades are likely to be very inaccurate.
Second.?The inaccuracies are greatest in the lower grades and where failure is involved.
Third.?Conclusions as to the progress of children through the grades when based upon the reports of children are therefore more or less inaccurate and may be directly contrary to the facts. Fourth.?As the cumulative record card is being moie and more generally introduced, we are justified in insisting that future studies as to the progress of children through the grades shall be based upon the evidence furnished by such card.
Disclaimer
The historical material in this project falls into one of three categories for clearances and permissions:
Material currently under copyright, made available with a Creative Commons license chosen by the publisher.
Material that is in the public domain
Material identified by the Welcome Trust as an Orphan Work, made available with a Creative Commons Attribution-NonCommercial 4.0 International License.
While we are in the process of adding metadata to the articles, please check the article at its original source for specific copyrights.