The author concludes (IMO, but you should read it yourself) that while tutoring does have positive effects, the 2 Sigma effect size measured by Bloom was probably an outlier.
Keep in mind that Benjamin Bloom attributes much of the success of tutoring to the affective learning components. That is not addressed in the meta analysis.
Bloom described how the tutor and the student achieved an emotional connection that is often difficult to achieve with a class of students.
This is so critical, because the barrier to learning for many students is emotional. It’s not that they are really trying and just don’t get it; it’s that they don’t have the capacity to care enough to engage over time. Emotional barriers to learning are super widespread — being “bored” for instance is an emotional response.
Transformational learning takes place when there is an authentic emotional motivation to succeed. Human contact can support that. It’s also why stuff like ALECKS only goes so far. There is not emotional resonance, like with a human.
I think you could combine both models: Students use tutoring software and their teacher spends time with each student celebrating their progress and encouraging and teaching them when they struggle. The teacher's primary job would be to teach the children how to learn and succeed in academics. Specifically, teachers would teach students how to make plans, follow plans, focus, how to think about success, how to think about failure, determine the cause of failure, update their plans, develop determination, evaluate their own mood, and recognize their mental habits. Teachers would also assign, grade, and give feedback on student projects. Projects would have multiple iterations before a final grade.
> being “bored” for instance is an emotional response
Don't be so patronising. People are bored (not "bored") because they're being asked to do something they don't care about. That's an extremely common experience. About a third of high school students are bored every day in every class, and another third report being bored in at least one class every day. Most people have no interest in intellectual pursuits. Their preferences are completely valid.
I’m sure we can sit here all day and debate what material kids should learn. My point is that 1. people do well in any subject when they care about it and 2. tutors often help kids care. I’m trying to distinguish this emotional effect of tutoring from the cognitive effect —- otherwise it is difficult to explain the 2 sigma findings.
> The history of the educational research literature is one plagued with low quality small sample size studies that were done decades ago, with less work being done now. It can be that now researchers are focusing on studying other instructional methods. Still, the fact that most large RCTs tend to find little effects should make us have a sceptical prior when presented with a new educational method.
Thanks. My takeaway from the Nintil article is that nobody has performed a good study on the effectiveness of mastery learning vs traditional teaching. All of the studies have some fatal flaw: not randomized, small sample size, study duration too short, interval between exams too long, not providing specialized remedial content to students, or not actually requiring mastery.
I think the massive effects shown by software tutoring in the DARPA studies point to the mechanism: frequent exams and specialized remedial content. Good tutoring software continually tests students for mastery, identifies specific misconceptions, and provides specialized remedial content for each misconception. The automated software can perform this iteration for each core concept, multiple times per hour. Students frequently get feedback on problems so they waste little time trying to learn material when they don't have the pre-requisite concepts. Students also frequently pass section mini-exams and enjoy feelings of accomplishment. These positive feelings help with learning.
Compare that to the mastery learning studies performed. The studies gave exams once a week or once every 4 weeks. A student with a crucial misconception will struggle for weeks before the they finally understand the content. During that time, they feel frustrated and unmotivated.
We need a good study of mastery learning.
We also need researchers to design their studies better.
IDEA: A new kind of journal with an open study design process. Researchers submit their study proposal, experimental procedures, example raw data, code for cleaning and filtering the raw data, code for statistical analyses, code for generating tables and graphs from data, and a paper template that includes different conclusions based on the values produced by the code. The paper template pulls in the tables and graphs generated by the checked-in code. All of this content is public. Anyone may register an account and provide feedback. Vetted researchers volunteer to review the proposal and code. They receive credit in the resulting paper. When reviewers give LGTM, then the journal and researchers commit to publishing the paper, regardless of the results, and before they have done any experiments. A separate LGTM is required from an experienced statistician. The code includes assertions for sample sizes and valid data ranges.
The researchers must record video of themselves as they perform the experiments. They must also record raw data from their instruments. They must upload these recordings and raw data. The reviewers must LGTM the recordings and any PII redactions. The researchers must get LGTM for all changes to the code and paper template. The journal's servers execute the template and generate the final paper. When someone later discovers an error in the analysis or code, they can file a ticket or send a pull-request with a proposed change. The researchers commit to reviewing every issue and PR within a time limit. If they fail to do that, then the reviewers must handle it. If the reviewers also fail to do it, then the journal assigns another qualified volunteer as a new reviewer to handle it. After making a change, the system generates a new version of the paper.
Anyone may "star" the paper and receive notifications whenever it changes or there is a change to any of the papers it references. If a paper is withdrawn, the system automatically adds warnings to all papers that reference it.
I've seen middle schools integrate Aleks (https://www.aleks.com/) online learning along with regular math classes. I don't have the data to know how successful it was at bringing the lower tail of the distribution up, but it was great for kids who were smart and motivated by the material. Some were even able to complete two years' worth of curriculum and test out of their next math class.
Schools are set up for the convenience of the people who work there. Mastery learning is a nightmare. Differentiated learning is really hard because you have to teach what amount to three classes at the same time. Mastery learning is worse because you have to try and teach as many classes as there are students. I exaggerate but you get the point. I know of absolutely no examples of schools using mastery learning. The school I'm in moved to streaming English classes by ability for grades 6,7,8 (Foundation) into A,B,C,D and 9 into A,B,C. Teaching the Foundation classes works ok but I can see why so many teachers hate streaming. Teaching lots of low ability/motivation students is unbelievably dispiriting. Low ability students you can at least help but if you're teaching them something they don't care about, like most teachers do for most of every class then low motivation students are hell to teach.
Schools are mostly child care and ranking by academic ability. Education is secomdary. Mastery learning is really hard and would reqire a wholesale reorganisation of schools. That's a large part of why it isn't done.
Mastery learning doesn't have to be harder than regular teaching. It just requires the teacher to split the class after each quiz and spend some time teaching remedial content to the under-performing group.
I think your point is that teachers don't like teaching unmotivated students so they ignore mastery learning programs.
Many Charter schools are specifically set up to teach under-performers. Why do those schools ignore mastery learning?
> Mastery learning doesn't have to be harder than regular teaching. It just requires the teacher to split the class after each quiz and spend some time teaching remedial content to the under-performing group.
This very easily leads to teaching three different grade levels of material. If you have a larger class the spread can very easily be even bigger. This is why ed schools emphasise enrichment over acceleration. You cannot teach a class where most of the class is doing Grade 6 material, three are doing Grade 9, two aren't really ready for Grade 5 and one kid is doing Grade 11.
> I think your point is that teachers don't like teaching unmotivated students so they ignore mastery learning programs.
Not really. I don't think mastery learning is necessarily harder than age based classes but it would require at minimum a complete reorganisation of how schools work. You would have to abandon age based classes altogether.
> Many Charter schools are specifically set up to teach under-performers. Why do those schools ignore mastery learning?
It's weird. The teachers don't want it. There's no demand for it.
Are we talking about the same Mastery Learning as described in the article? Can you please explain why Mastery Learning requires teaching different grade levels to the same class? My understanding is that all students study the same material, take a quiz, then the ones who aced it get to study whatever they want for the rest of the period and the ones who didn't ace it must listen to more lecture and discuss the content with the teacher and take another quiz. Where are the multiple grade levels of material?
> it would require at minimum a complete reorganisation of how schools work. You would have to abandon age based classes altogether.
I had it for a class in college and it was an absolute nightmare for me. This was from someone well known in mastery learning so I don't think it was poor implementation.
The only thing I learned was empathy for people who struggle that way with the typical mode of instruction (which works very well for me).
Yet despite this being completely obvious to anybody who has any experience with tutoring, I see people rolling out the same old discredited hypothesis of genetics having the largest influence on academic achievement in every HN thread tangentially related to intelligence.
Improved environment actually exacerbates genetic differences. One obvious example is height. Back when malnourishment was common most everyone was rather short and nutrition was the best predictor of height. In an environment where adequate nutrition is widespread, genetics is the best predictor of height.
I’m distrustful of analogies, but I wouldn’t rule out the possibility that we’ll see a similar effect with improving education. Although solving that is going to be trickier than nutrition I expect.
Right, today everyone gets to go to a decent school for many years. Historically people didn't go to school.
Take my case, I was neglected as a kid, my mom often didn't have a place to live so we moved between her friends places. I was raised on fast food and cookies, never did homework or studied. Yet I still did much better than average people in school and later got into Google as a software engineer. Most people who fail to get good grades in math had a much better home environment than I had, if they had my genes they would have gotten perfect grades in it like I did.
So from my perspective, the only thing I can say is that they are stupid privileged rich kids (a kid whose parent has a stable place where they can live is privileged from my perspective) who didn't have any difficulties in life and still can't even do something as simple as calculus even after having listened to it in class for like 100 hours.
"Outliers doesn't matter"
They sure do in this kind of discussion. If you just ignore all the kids with good genes who came from bad environments then of course it is easy to say that good genes doesn't matter, but there are a lot of such kids. And the kids from poor environments who don't do well just didn't have good enough genes to make up for the environment. I can say that if every kid had my level of talent for math then basically every single kid would get straight A's on every math course given today that a typical software engineer takes without studying. I did that, and most people have better upbringings than me. Of course in that case the curriculum would adapt, but still.
(Note that we shouldn't base education decisions on outliers however. I can't give people any advice on what to do, others can't do the same thing as me, I just understood everything in class from sitting there and listening, other kids obviously doesn't do the same so they need help. But still you can't say that genes don't matter)
Edit: Also note that I have two brothers who will likely go through their entire lives without ever having a stable job. There wasn't some kind of hidden special thing in my upgringing.
Couldn't it simply be that you actually did have the right environment to learn math? Being poor might mean you couldn't have a tutor, but it doesn't mean you couldn't concentrate. I always found distraction to be the real problem, and that can happen to rich kids as well.
You mean sitting in a classroom and listening/talking to people? Because that is all it takes.
Edit: It is really irritating to me how so many people thinks it is impossible to grasp math by just sitting there. Look, there are so extremely few concepts to learn and you have many hundreds of hours of class time to learn it, how can you not learn during that time? Nobody told me to do work during class, so I didn't. Nobody told me to do homework, so I didn't. I just sat there, when they gave me a test I did it, the tests was the first time I ever tried to solve a problem and I solved them because I had seen how to solve similar problems during class. This strategy worked up to a masters level in physics/maths, so not just high school.
Did people really explain things to you at university? Where I went you were expected to read a lot on your own, and then you had a few hours of tutorials where you could ask some questions.
For high school, you're basically right, there isn't that much content. But it's not like you're in a math class particularly long either. Few hours a week for a few dozen weeks a year, for a few years, and a fair bit of each class is getting everyone seated, telling people the homework, and reviewing the previous lesson. If you look at a math/science high school curriculum, you tend to just follow some book that's a few hundred pages, and it takes the whole year to do it. An engaged kid could read it all over a few evenings, per class.
> It is really irritating to me how so many people thinks it is impossible to grasp math by just sitting there
It's very easy to think you really understand something when someone's explaining it to you. Kinda like watching a show on Discovery. Then when it comes to solving problems, there's a fair chance that you're exposed. I found this a lot at university, there'd only be 10 questions but they were good ones that necessitated looking at a variety of sources and getting a real understanding. Often nobody I knew would be confident on all the answers, and this is kids that got into a world famous university.
> It's very easy to think you really understand something when someone's explaining it to you. Kinda like watching a show on Discovery. Then when it comes to solving problems, there's a fair chance that you're exposed.
I didn't think about it though, I didn't assume I understood anything. Just that one day there is a test, I went to the test and answered everything as well I could. I didn't get everything right, I did a lot of stupid mistakes, but I got enough things right.
I think that if you need to practice to do well on the tests then you don't understand the subject well. There comes a point where you feel you have perfect clarity of a topic, I just reach that level very fast. I am not always correct when I feel that way, but it was good enough.
> Did people really explain things to you at university? Where I went you were expected to read a lot on your own, and then you had a few hours of tutorials where you could ask some questions.
Right, higher level courses makes you do more independent work. But the courses people take for physics, CS or engineering are still very well explained. The courses you take to prepare for math research didn't go as well for the reasons you explained, I still passed them but with worse grades and started doing math research, but after I solved my first research problem and wrote about it I realized I didn't really like math that much and learned programming and got into Google instead. I figured that if I have to work on something I don't care about I can just as well earn a lot of money doing it.
Just explaining things here to make my position more understandable. I always hated teachers who grades after how hard you work or how many boring practice problems you solve etc, so I have a strong distaste for any argument along those lines. "But you need to work hard to learn!", no you really don't. That is my point. Don't make me do boring stuff because you had to.
I don't think I ever suggested grading based on effort, it makes no sense to me either.
But most people do need to do something other than just show up to class to learn things. That's got to somehow be obvious, based on everyday observation of your peers? Certainly my main experience of university was that kids who were diligent understood things, and those who weren't didn't. Plenty of kids thought they could either quickly catch up, or get an overview at the lectures and just be smart and figure it out. It might have worked for one or two, but for the large mass of kids, they needed to sit and look at things themselves, and take some time.
Also I don't understand how you think courses were well explained. Where did you study? I never thought anything was explained much at all, lectures were mostly a summary of major ideas, with a lot of glossing over the details by professors who were long past the point where it mattered how well they understood the stuff.
> Also I don't understand how you think courses were well explained. Where did you study? I never thought anything was explained much at all, lectures were mostly a summary of major ideas, with a lot of glossing over the details by professors who were long past the point where it mattered how well they understood the stuff.
My point is that to me they were well explained. To most others they are not, but that wasn't what we were discussing. A guy above said that nature is completely dwarfed by nature even today. I disagree with that. A bottom 10% kid with respect to nurture can still get great results if their nature just is good enough.
Most of my professors in college called me a genius. I got some research projects after my first year, but I didn't have any work ethics so I didn't do anything with it, I mostly just sat there and didn't do anything just like how I spend most of my time in class. I solved the original problem quickly, but all time after that was just sitting. I know I am not normal, I just want to say that people like me exists, and I get angry when others argue that people like me doesn't exist, that we just fake it or are privileged etc. It is just nonsense and it irritates me a lot.
I'm a total mental wreck, I could barely do anything, arguing that the reason I did so much better at math than median persons was because I worked hard, got tutoring, had a better home environment, had better focus etc, all of that is just nonsense. I'm a bit better today, there is no way I could even hold a job back then as just sitting and not doing shit doesn't fly at work, but I still lived through all that shit.
A simple model is that some people will do well regardless, some will do poorly regardless, and the middle will perform dependent on what environment they're put in.
I'm not saying really smart people don't exist, obviously we've all met mentally disabled people who cannot be taught higher math so why wouldn't there be people who get it really easily?
But I do think most people by far are in the middle group. Teach them well, encourage them, and they do well. Fail to do so and they do badly.
But if you agree with such a model you must see that in aggregate, nurture wins. There's a bunch of people who wouldn't know any calculus or whatever if they hadn't been set to do certain problems while being educated.
If you look at something like private education in the UK, you see this very thing. Kids are coached and they do well, on average. They're confident and they're well drilled. Not every kid in those schools does well, and not every kid who does well went to one of those schools. Yet stats show that the 7% of kids who went to private schools get about 50% of the Oxbridge places.
The existence of so-called gifted persons such as yourself does present some difficulty to the blank slate crowd. Sadly, they appear to have settled on reducing or eliminating testing as well as tracking into gifted programs as a way to cut that gordian knot. Their alternative is more subjective measures, where I imagine at least some of your teachers would have been inclined to grade you poorly despite your grasp of the material.
It’s disappointing, but also amusing. When I was in high school Harrison Bergeron was assigned reading. And it’s a great loss too, but one that most people won’t even notice. Observe how much skepticism your personal testimony gets from those who cannot even imagine what it’s like to have extremely high intelligence.
I had the same experience, never studying at all, passing all the tests, and winning academic competitions. The difference is that I understand the reason for my apparent giftedness is being taught to think properly at a young age. I have taught others to do things that I had done at almost half the age because I have a deeper appreciation of pedagogy than my own teacher.
I agree (having taught), but I disagree it's somehow genetic (which implies inevitable and irreversible). Most of the time it was a mismatch in prerequisites and a mental block like anxiety.
It can’t be not genetic. Just not totally I guess you meant. Or determinist as you said. As above said many factors but push genetic out is also not good.
Phineas Gage[1] would have limited ability, it doesn't mean he was genetically destined to get a rod through the brain. Weston Price[2] photographed and studied people around the world and wrote the book "Nutrition and Physical Degeneration" arguing that modern diets lead to physical development problems.
Not that it can't be influenced by genetics, or that people can all be the same; very few people will be Von Neumann reading a 46-volume world history series in his fifth language at age 8.
But saying "these people are failing at school, it's genetic" is a handwavy dismissal; a giving up on them. If it is, what should it change about teaching?
If you have tutored one-on-one, you will surely have noticed that the difference in ability is miniscule compared to what can be taught by tutoring. Indian children do not have a genetic predisposition to spelling English words correctly.
At the top end, performance levels are determined by intrinsics perhaps genetics, perhaps not). If I had been coached in basketball every day since birth I could have become a very good basketball player. I would have probably been the captain of my high school team despite being 5 foot 6. However there is ~zero chance I would gotten to the level of playing in the NBA.
Anyone can get better at anything, but they will ~always be worse than someone.
Three of those are genetically inluenced. The better your educational sysyem is at imparting non-cognitive skills and at removing children from abusive and unsafe environments the more the generits will matter for the same reason genetics mattered more for height in 1900s US than UK, saturation of inputs.
As for the one that is in no way genetic
> Small class size has at best a small effect on academic achievement, and may harm some students
> Reducing class size is seen as a way of improving student performance. But larger class sizes help control education budgets. The evidence suggests at best a small effect on reading achievement. There is a negative, but statistically insignificant, effect on mathematics, so it cannot be ruled out that some children may be adversely affected.
Perhaps the point is most people are limited by unfulfilled potential - not that they've reached their genetic max. I do t think anyone would argue that a really nurtured lettuce plant would ever match ability to learn with a slightly neglected human.
> The high heritability of educational achievement reflects many genetically influenced traits, not just intelligence
> Because educational achievement at the end of compulsory schooling represents a major tipping point in life, understanding its causes and correlates is important for individual children, their families, and society. Here we identify the general ingredients of educational achievement using a multivariate design that goes beyond intelligence to consider a wide range of predictors, such as self-efficacy, personality, and behavior problems, to assess their independent and joint contributions to educational achievement. We use a genetically sensitive design to address the question of why educational achievement is so highly heritable. We focus on the results of a United Kingdom-wide examination, the General Certificate of Secondary Education (GCSE), which is administered at the end of compulsory education at age 16. GCSE scores were obtained for 13,306 twins at age 16, whom we also assessed contemporaneously on 83 scales that were condensed to nine broad psychological domains, including intelligence, self-efficacy, personality, well-being, and behavior problems. The mean of GCSE core subjects (English, mathematics, science) is more heritable (62%) than the nine predictor domains (35–58%). Each of the domains correlates significantly with GCSE results, and these correlations are largely mediated genetically. The main finding is that, although intelligence accounts for more of the heritability of GCSE than any other single domain, the other domains collectively account for about as much GCSE heritability as intelligence. Together with intelligence, these domains account for 75% of the heritability of GCSE. We conclude that the high heritability of educational achievement reflects many genetically influenced traits, not just intelligence.
The author concludes (IMO, but you should read it yourself) that while tutoring does have positive effects, the 2 Sigma effect size measured by Bloom was probably an outlier.