To increase the size of muscles, should we increase the number of reps in a set or should we increase the number of sets?

If you are training for Hypertrophy (Increase in Size) in particular, increasing number of reps per set is the way to go. Generally, about 8–15 reps per set is used for training towards Hypertrophy.

On the other hand, if you are training for gaining Strength specifically, you need to train with Higher Weights and Lower Reps per set. Typically, 5 reps or lower for gaining Strength.

Ideally, with 5 reps per set, you will improve your Strength and will also gain some Hypertrophy (you will get bigger).

Of course, the actually weight you will use for these sets will depend upon the number of reps per set. The Higher the Reps, Lower the Weights for that set (Hypertrophy Training) and the Higher the Weights, Lower the Reps for the set (Strength Training).