In the US, the ESSA framework sorts EdTech evidence into four tiers. Tier 4 is the lowest. It only asks for a "research-informed rationale", and it gives little guidance on how to build one. So quality varies wildly. A new paper by Kucirkova, Vahey, Patterson and Dockterman, summarised by EdTech Partnerships, argues that "Tier IV could be much more than the bottom rung of an evidence hierarchy." We think they are right, and the reason applies far beyond the US.
The problem: reverse engineering evidence
The paper names something every EdTech company has been tempted to do. You build a feature, then look for research that seems to support it. "It has feedback, therefore it is based on learning theories." The authors call this "a kind of reverse engineering of evidence." It looks scientific. It changes nothing.
Looking for research after building a feature is reverse engineering, not evidence.
The method they propose
Five steps. Do a rapid review of the literature. Accept different kinds of research, qualitative and quantitative, conceptual and empirical, as long as it is rigorous and relevant. Weigh how strong and how applicable the evidence is. Check where the product lines up with the research, and where it does not. Then turn that into a theory of change and a research roadmap.
The step that matters most
Step four is the honest one. Where does the product not match the research? In the paper's case study, a reading app called I Can Read lined up well on reading components and less well on vocabulary. So the team changed the activities. That is the whole point. Evidence that never changes the product is decoration.
More frameworks will not fix it
There are more than 75 evidence frameworks and certification schemes around the world. Another badge will not help a school choose. What helps is a company that can show its reasoning, including the parts where the research says it is not there yet, and a plan to find out.
What to ask
If you are a school or a parent, ask any EdTech company three things. Which research is your product based on? Where does your product not match it yet? And what did you change because of it? A good answer to the second question tells you more than any badge. It is also the standard we want to be held to.
Take these with you
- 1Looking for research after building a feature is reverse engineering, not evidence.
- 2The useful step is finding where a product does not match the research.
- 3Evidence should lead to changes in the product.
- 4Ask any EdTech company what it changed because of the research.
