Generative AI platforms such as ChatGPT have found prominence in recent times with their ability to generate texts, images, etc. Several questions pertaining to ethical and legal issues surrounding ChatGPT have arisen. In this paper, I discuss the nature and background of generative AI, situating its development in the historical context of AI. I then discuss my primary research questions: is sufficient attention paid in the literature on ethics of AI to the principle of beneficence and is there theoretical clarity on its meaning? I highlight that while there is great deal of discussion on what harms can arise from generative AI and how to stop them, there is very little discussion on what amounts to AI-for-good, particularly in the literature of AI ethics and regulation. To the extent that such discussion exists, it pushes ahead with suggesting specific solutions, without fully addressing the underlying question of what makes their prescriptions beneficent. I demonstrate that the principle of beneficence as understood in biomedical ethics and human rights frameworks has limited utility for the ethics of generative AI. These gaps in the AI regulation are prominent, as they can derail long term progress of generative AI and the realization of its full potential.