I am writing a paper about one of my dissertation analyses, the one that most clearly distinguished the two groups in my study. It looks at kids' use of unusual words, including specimens like "ziggurats."
One of the hassles of studying discourse-level language -- conversation and storytelling -- is dealing with the argument that your work needs more standardized test scores to back it up. Because hey, maybe you're using "ziggurats" appropriately in conversation, but without those standardized test scores we can't know much about your language skills. [snort]
It might surprise you to learn that standardized test scores don't actually hang together very well with discourse-level language measures: the kids with complex sentences in conversation might freeze up in a test scenario, or the kids who are good at tests might tank in conversation. Even though I knew this, I was optimistic that my shiny new measure would correlate well with the test scores available to me.
I looked at standardized test scores and pulled out a group of high-scoring kids and a group of low-scoring kids. There was, like, an ocean of difference between their scores on that test. Next I got raw frequency counts for their use of unusual words. I was astonished to see that the mean was the same to TWO decimal places. I looked back to see if I'd made a dumb mistake, like entering the same information twice: nope.
I thought, well, that's weird but I bet it will shift around a bit. I removed all the low-scoring kids whose parents weren't concerned about their language use, because some kids just don't want to be bothered with a standardized test that goes on and on. And I recruited two of the lab crew to go through the transcripts and look at word use in context. I wondered if the low-scoring kids might be more likely to throw in words that didn't belong: maybe a sentence like, "and my cousin's pet bunny is named ziggurat or --wait, Sigmund." They don't get credit for a word unless they use it with a measure of semantic appropriateness.
This morning I went through and incorporated the students' judgments. I re-ran the calculations and was astonished -- FLOORED -- to find that the group means both dropped by exactly the same amount. They're still identical, to two decimal places. It's definitely not a computational error, or a stupid mistake like giving the spreadsheet the wrong range of cells.
That is just plain freaky. Language is a strange and slippery beast.
Recent Comments