It's a little bit natural to think that 'knows' contextualism and the shifty kind of invariantist that's sometimes called an 'SSI theorist' or an 'IRI theorist' come to a bit of an intuitive draw considering two kinds of third-person knowledge attributions. High Howie has whatever features you think makes it harder to know, or makes 'know' express a stronger relation: he's thinking about skeptical possibilities, it's really important to him whether p, or whatever. Low Louie is just the opposite: to Louie, p's no big deal, he's not worried about it, whatever. High Howie says things like "I don't know that p," while Low Louie says things like "I know that p," and both utterances look pretty good, even though in some sense Howie and Louie look to be in identical epistemic situations -- they have the same evidence, or something like that.
(Now I happen to think that it's not at all clear how to make sense of that last stipulation. This basically amounts to a worry whether there is any correct generalization characterizing the difference between the shifty SSI-like views from 'classical invariantism'. But I'm setting that aside for now, assuming, as is usual in this discussion, that the sense in which Howie and Louie are in the same 'epistemic position' is tractable -- and does not at least really trivially entail that they're alike with respect to knowledge. I'll here use 'epistemic position' technically to mean the stuff that traditional invariantists affirm, but shifty people deny, comprise a supervenience base for knowledge.)