An excerpt from The Inside Story on English Spelling by Paquita Boston explains that the silent "w" in answer is a case of consonant cluster simplification. In simpler terms, certain sounds in the word didn’t blend well together, leading some to either merge with neighboring sounds or disappear entirely.
Because it is short, [w] is silenced in difficult blends. Follow the sounds of sword: make [s] behind top teeth, move forward to [w] with lips, right back to [or] at the back of the throat, and forward again to [d] just behind the teeth. The mouth finds going from middle, to front, to back, and forward again too much, too difficult, and misses out the front sound, taking a short cut to the back of the mouth. That front sound, you'll remember, is [w], which is silenced. Although W is silenced in sword, it's not silenced in swim, because [i] is not made right at the back of the mouth. There's a Silent W in answer but not in swear. In the first, Silent W comes between a long consonant and a back vowel. The second does not have a back vowel and so a short cut is not required. Once upon a time we didn't look for short cuts. We didn't shorten our words, back when instead of 'Answer me', the demand was 'Speak, and swear it is the truth'. That statement then reduced to 'and swear', until finally, without D, it became answer. Well, that's the story — and to help littlies remember the Silent W in answer, it is pretty close to the truth, because Old English andswaru came from and, meaning 'against or ante' and swaru meaning 'affirmation', from swerian — 'to swear'. In other words, it means 'to make a sworn statement rebutting a charge, to swear you were not to blame'.
