Poem interpretation (!)

One of the pieces I've been working on in my voice lessons is a setting of a Thomas Hardy poem by Gerald Finzi. The poem is entitled, "To Lizbie Browne". I've been singing it for several weeks now, and yesterday I learned that my teacher subscribes to what seems to me to be a very odd interpretation. I have very, very little confidence in my ability to interpret poetry, but I feel right to me. Please tell me what you think. Following is the poem. My teacher thinks that the speaker was married to Lizbie Browne. I think that he was not -- in fact, that she was never even very aware of him.

"To Lizbie Browne" Thomas Hardy Dear Lizbie Browne, Where are you now? In sun, in rain? - Or is your brow Past joy, past pain, Dear Lizbie Browne? Sweet Lizbie Browne, How you could smile, How you could sing! - How archly wile In glance-giving, Sweet Lizbie Browne! And, Lizbie Browne, Who else had hair Bay-red as yours, Or flesh so fair Bred out of doors, Sweet Lizbie Browne? When, Lizbie Browne, You had just begun To be endeared By stealth to one, You disappeared My Lizbie Browne! Ay, Lizbie Browne, So swift your life, And mine so slow, You were a wife Ere I could show Love, Lizbie Browne. Still, Lizbie Browne, You won, they said, The best of men When you were wed Where went you then, O Lizbie Browne? Dear Lizbie Browne, I should have thought, "Girls ripen fast," And coaxed and caught You ere you passed, Dear Lizbie Browne! But, Lizbie Browne, I let you slip; Shaped not a sign; Touched never your lip With lip of mine, Lost Lizbie Browne! So, Lizbie Browne, When on a day Men speak of me As not, you'll say, "And who was he?" - Yes, Lizbie Browne.

For the record, I like the song.