The unavoidable reality is that we won't get a real sense for how well these kinds of innovative programs work until we try them on large populations over long periods of time. In doing so, are we experimenting with our kids? Yes. The key to breaking our educational paralysis is to recognize that we are also experimenting with them if we continue traditional teaching methods in the face of a changing modern world.
Yes this is an awful design with a completely predictable outcome, and I hope for ethical reasons the control students will get chromebooks and access to the blended learning stuff afterward at least. And using the same teacher in both conditions doesn't solve problems - the teacher expects students in the experimental group to do better, and will influence the result even if not actively trying to.
But unfortunately A) this is the kind of studies the department of education wants and funds (randomized experimental designs - see the 'what works' database, where they ignore anything not using that design). And this is the case even when the control group is obviously in an intentionally deprived learning environment. And B) the opposite design, tightly equalizing every aspect of the control and experimental groups, isn't good either, because it wouldn't take advantage of the new things blended learning / software afford (like animations, interactive feedback) that traditional paper-based instruction doesn't. See page 2 of this article from the Concord Consortium about the 'hobbled horse race' and so forth: https://docs.google.com/viewer?url=http%3A%2F%2Fwww.concord....
There are 2 alternative research designs I would have done in this case:
1) Design-based research: http://edutechwiki.unige.ch/en/Design-based_research
Do the best you can with the Khan Academy curriculum in a class. Pre-test and post-test, interview and observe and so forth. Measure what they learned, their engagement, etc. THEN you try to figure out things that didn't work out so well, things you missed...try to apply a theory or framework to explain what happened. THEN try it out a second time making revisions, and see if you can increase the learning gains and engagement even more the next time. Use of pre-test / baseline measures makes it possible to do comparisons.
2) Do short studies comparing different designs of the Khan Academy software and curriculum. This is traditional, individual lab-based studies. Can be useful if there is a valid issue where the designers / researchers don't know the best path, and want to test/compare different options.
Method (1) seems like it could provide some valid insight. To me, however, collecting information from a number of schools seems like the best approach.
I'm not terribly familiar with how studies are done in schools. Is it incredibly difficult to collect cross-section which is large enough to control for differences in teachers/students? Maybe my inner economics student is showing through, but this seems like a prime target for good old-fashioned regression analysis.
Education is not so special that it can't be subjected to the scientific method. Medicine is similarly complex, and benefited immensely from randomised trials. There may be challenges (the placebo effect of new methodologies, and the impact which observation has on behavior), but it's not so special that big gains can't be made by studying what works in an empirical manner.
When education catches up to where medicine was in the 1937 (with the introduction of placebo trials), or even the 1950s (with double blind trials) they can start looking into methodological improvements.
Until then, there's lots of low-hanging apples.
I guess the best way forward is for the education departments to stop trying to control everything. They send out instructions, telling teachers what they are meant to do. Why not just AB test the instructions they are sending principals? That way, they can learn which of their decisions actually work.
I suppose all this experiment design will be helpful for academic theorists. In the real world classroom, Khan Academy is a HUGE step forward for almost any learning environment children are put in.
I know, because I have experienced it myself -- watching a teacher understand they can see exactly which math subjects each student struggles with -- and watching students leap ahead when given the chance to work on topics interesting them and attach fun bragging rights for 'success' in a social computer classroom environment.
The leap in engagement and skill building is so hugely marked that I don't believe it's ethical to keep children from such an experience unless there is strong evidence that it is actively harmful.
All this is anecdote, not statistics of course. The kids I worked with using Khan Academy last year would say "Can we stay in from recess to do more math?? Are we doing Khan Academy today? YES!!" This level of response just isn't present with all but the most magnificent teacher in a more traditional classroom setting, and it makes me believe that there's something real and beneficial there.
This is clearly quite a difficult task -- successfully controlling variables in an educational environment is always going to be difficult. There are so many students of varying levels, learning styles and parental support structures that it will be a few years before tools like Khan Academy can be fully assessed in a real classroom settings.
However, blended learning environments are clearly the future of learning and the only hope for personalizing education in a system that is under constant budgetary pressure with ever increasing class sizes. It will be exciting to see the outcomes of these and future studies. Kudos to Khan Academy for really driving this forward.
Thanks for the link, edutechdev. I am speaking on a panel for edtech entrepreneurs and educators in SF in October on exactly this thread topic, experimentation. I will reference your design-based research guide. It is very important that those of us who know something about social science experimentation weigh in on this national conversation.
