Do I have to lose fat to build muscle?

There are several different things you need to understand about this.
1. You need to understand Progressive Overload. What that means is, your body adapts to stress. So let’s say you started doing maybe 20push-ups. So, from zero push-ups, your body then got stressed when you did 20 push-ups and consequently, adapted to that by getting stronger (and slightly bigger). Then say you went to 30 and 40 and then to 50 push-ups. Your body adapted to these increases and got stronger and slightly bigger each time. However, now, since you are always doing 50 push-ups everyday, given the fact that you have already got adapted to 50 push-ups, your body needs to do nothing more. You understand, how this works now. So, you might try to do higher reps and might see more results.
2. One other thing you need to understand is that, these are the limitations of Body Weight Exercises. You might try to up the reps, you might try to make it more difficult by placing weights on your back, maybe doing inclined push-ups… Consider training with Free Weights. You can adjust the weights up/down as little as required…they work your whole body… You can therefore, Progressively Increase the Weights ever so slightly…so, your body gets stressed more than the last time…and your body therefore, now has a reason to get stronger and bigger.
3. With all the above, you still need to ensure that you get adequate rest (a minimum of 8 to 9 hours sleep is absolutely essential)…
4. And, you need Adequate/Appropriate Food. Protein is the main fuel for muscles…so, make sure you get Plenty of Good Quality Protein with Every Meal. Do NOT skip any MEAL. Plus, you will need to be in a Calorie Surplus status to grow.
All above things working in Tandem…you will get Stronger and Bigger.
Now, to address your topic heading… The reverse is true…you will put on fat, in the process of putting on more muscle. However, if you do it right (following things discussed above), you can skew the growth, to happen More towards Muscle Growth and Less towards Fat Accumulation.
I am assuming you are not Obese. If you are very Fat/Obese, then, your priority might be to cut down on some of the Fat initially, and therefore, might need to take note of your Calorie Intake (How much you Eat).
Once, you have shed a lot of your excess fat (that is, if you are Obese, to begin with), then, slowly start increasing your Calorie Intake…all the while, working out as discussed above.
Hope this helps!