Speaking from personal experience, I would strongly recommend against seeing a job as income solely and pursuing your passion in your free time.

You spend a huge amount of time in your work. We are mostly privileged enough to choose the job we take. Don't stay where you're unhappy. Don't settle for a job you don't find fulfilling.