See a question on TeX.stackexchange, and especially Khaled Hosny's answer. It appears that XeTeX is not using the correct font to decide how to turn a math list into a horizontal list. Namely, it seems to be using the last used font (or last defined? or last selected with a font command?) rather than the font in which the math formula should be typeset (typically controlled by \textfont etc.).
The problem occurs when mixing OpenType and non-OpenType fonts, more particularly when nesting math mode (even within a \setbox...\hbox{$...$} construction, which should not have global consequences).
\global\font\xxx"[latinmodern-math.otf]"\relax
\global\textfont 0\xxx
\newbox\mybox
\def\ignore#1{\setbox\mybox\hbox{#1}}
\def\showhbox#1{\setbox0\hbox{#1}\showbox0\relax}
\tracingall
\showhbox{${\ \over\ }\ignore{a}$}
\showhbox{${\ \over\ }\ignore{$\mathchar`a$}$}
\showhbox{${\ \over\ }\ignore{a}$}
\bye
Anonymous
Should be fixed by 9785737384bfe6080b8231f666ddf5b3861c957a, I think; please test.