The rapid growth of digital English listening platforms has intensified the need for intelligent personalization mechanisms that adapt to learner progression while preserving data privacy. Existing adaptive systems primarily rely on static difficulty scaling or centralized learning architectures, often neglecting learner engagement dynamics and raising concerns about sensitive data exposure. To address these limitations, this study proposes PrivAURAL, a privacy-preserving and affect-aware adaptive English listening framework that models listening instruction as a sequential decision-making problem. The objective is to dynamically personalize listening tasks by jointly considering comprehension performance and engagement trends, without transmitting raw learner data. PrivAURAL integrates HuBERT-based semantic–acoustic representations with affective proxy signals derived from learner behavior and employs a Federated Deep Q-Network to adapt task difficulty, playback speed, and assessment frequency. The model is implemented using PyTorch, HuggingFace speech models, and a simulated federated learning environment with secure aggregation. Experiments conducted on the TED-LIUM dataset demonstrate a 32.7 reduction in Word Error Rate over ten sessions, a 21.9 decrease in task completion time, and an improvement in listening accuracy from 86.1 to 87.3 compared with non–affect-aware baselines. Federated training further ensures stable convergence, while maintaining strict privacy constraints. The results confirm that reinforcement-driven, affect-aware personalization can significantly enhance listening efficiency and engagement, positioning PrivAURAL as a scalable, ethical, and privacy-conscious solution for next-generation digital language learning systems.