As large language models integrate into daily workflows—from personal assistants to workplace tools—they handle sensitive information from multiple sources yet struggle to reason about what to share, with whom, and when. In this talk, we explore critical gaps in LLMs' privacy reasoning through complementary benchmarks. First, ConfAIde reveals that even advanced models like GPT-4 inappropriately disclose private information in contexts where humans would maintain boundaries. Second, we extend this analysis to persistent memories—an increasingly adopted personalization feature—showing failures in handling compositional secrets with multiple attributes and contextual cues. We then present a data minimization framework that formally defines the least privacy-revealing disclosure that maintains task utility. Our experiments show frontier models can tolerate up to 85% data redaction without losing functionality, yet they lack awareness of what information they actually need—leading to systematic oversharing. We conclude with techniques for restoring performance when privacy measures are applied, offering a path toward AI systems that respect contextual privacy norms while remaining useful.