Previous studies on face processing have revealed an asymmetric overlap between identity and expression, as identity is processed irrespective of expression while expression processing partly depends on identity. To investigate whether this relative interaction is caused by dominance of identity over expression, participants performed familiarity and expression judgments during task switching. This paradigm reveals task-set dominance with a paradoxical asymmetric switch-cost (i.e., greater difference between switch and repeat trials when switching toward the dominant task). Event-related potentials (ERPs) were recorded to find the neural signature of the asymmetric cost. As expected, greater switch-cost was shown in the familiarity task with respect to response times, indicating its dominance over the expression task. Moreover, a left-sided ERP correlate of this effect was observed at the level of the frontal N2 component, interpreted as an index of modulations in endogenous executive control. Altogether, these results confirm the overlap between identity and expression during face processing and further indicate their relative dominance.