This study aims to identify the critical factors influencing the user experience of contact tracing apps and the sentiments around them. For this purpose, we used Google play reviews of Aarogya Setu, a contact tracing app developed in India. First, we establish the relationship between review sentiment and review rating using regression between sentiment polarity and review rating. Then, we used a hybrid aspect-based sentiment analysis approach that uses unsupervised linguistic techniques to determine statistically significant concepts present in the review texts and cluster them into representative aspects that were then tagged under human supervision. Finally, supervised deep learning methods were applied for exhaustive extraction of the aspects and associated sentiments from the reviews. The final exercise of determining the key influencing factors was done by grouping these aspects under factors identified by marketing experts. A total of nine factors were identified, with the usefulness of the app being the most important factor. The findings of this study are essential for the development team and government to improve the application and increase adoption.