Machine learning (ML) algorithms play an increasingly prominent role in the distribution of benefits and burdens in sensitive domains. However, ML systems risk introducing biases that undermine values of justices and fairness, for instance, by perpetuating or even exacerbating unjustifiable harms against vulnerable communities. In response to this concern, a burgeoning field of fair ML has emerged, with researchers developing various fairness measures and methodologies for quantifying and mitigating algorithmic harms. From this perspective, philosophical theorizing about ML tools is often constrained to clarifying the normative underpinnings of fairness measures to help resolve disagreements arising from impossibility results and formal trade-offs. In this talk, I argue that this type of fair ML strategy is usefully characterized as a problematic form of ideal theorizing about justice, and thus suffers from limitations known to plague that approach more broadly. Building on this conceptual framing, I show how the strategy limits our ability to anticipate critical failure modes or propose useful policy remedies. These arguments undermine the ethical grounding of standard auditing and debiasing toolkits, and highlight an urgent need to develop alternative normative frameworks. I provide some suggestions about what these alternative approaches could look like.