Here's an insanely simple argument for contextualism about knowledge. I think it's sound, although I'm not sure I'd expect many people to be persuaded by it. I'd be interested in hearing about how readers might think it best to resist it.
Here's premise one. Epistemic modals are intimately connected to knowledge in something like the following way: it might be that p iff the relevant base of knowledge doesn't entail that not-p. This is pretty much standard, I think, although of course people argue about just which knowledge base is the relevant one. This much looks like common ground, for instance, in the debate between contextualists and relativists about epistemic modals. What's at issue there is how the relevant knowledge base gets fixed -- whose knowledge counts. If you need an argument for this connection, just reflect on the absurdity of "it might be that p, but I know that not-p" and "I don't know that p, but it must be that p". (I'm assuming the duality of might and must.)
Here's premise two. In many situations, both of the following obtain: (a) were someone to say "I know that p", that utterance would be accommodated and accepted as true; (b) were someone to say "it might be that not-p", that utterance would be accommodated and accepted as true. For example, in my current situation, I could truly assert "I know that Derek will respond to Paul (because that's what the workshop schedule indicates)". Alternatively, I could truly assert "Derek might not respond to Paul (because it's possible that he'll get sick during the lunch break and have to go home)." (Of course, I can't go both ways; in no context can I say both things; the point is that in some contexts I could say either.)
These two premises, all by themselves, put the invariantist in hot water. Take one of the situations described in premise two, and suppose invariantism is true. In that situation, by premise two, "S knows that p" expresses a truth in some context; therefore, by invariantism, it expresses a truth in all contexts. But, by premise one, "S knows that p" is inconsistent with "it might be that not-p". So this modal claim must be false in all contexts—against the stipulation of premise two.
The obvious solution, from my point of view, is contextualism about 'knows'. Then we can maintain the connection between 'knows' and 'might' in all contexts, and have each sentence true in some context. I don't see any option nearly so appealing for the invariantist. But this argument is so simple that it can't be decisive. So what should the invariantist say?