Indepentent samples t-test with very different sample sizes

I have two groups from the same sample of 117 that answered either "yes" or "no" to several questions. Some of the "yes" answer groups are very small (4 to 10). When I do a t-test to look for differences between groups on the continuous variable, some results fail Levene's test. Can I trust the Welch values to be statistically sound?