This can be solved by simultaneous equations, also called a system of equations. Let x be the number of Rs 8 books and y be the number of Rs 3 books he buys. Then:
x + y = 20. (Total books purchased)
8x + 3y = 110. (Price of each book)
From here there are several strategies, but perhaps one of the easiest is called the elimination method, which is the basis of linear algebra and matrix manipulation. Multiply the first equation by -3 to produce
-3x - 3y = -60
Then add that equation to the second equation to get
8x + 3y - 3x - 3y = 110 - 60
5x = 50, then x = 10.
So he bought 10 books costing 8 rupees each. We cab use substitution next, but it should be obvious that he had to buy 10 books costing 3 rupees each to get to the 20 books he bought in total.
To recap, 10 books at 8 rupees each (80 total rupees) and 10 books costing 3 rupees each (30 rupees) for a total of 20 books that coat a combined 110 rupees.