A niche market is a focused, targetable portion of a market.
By definition, then, a business that focuses on a niche market is addressing a need for a product or service that is not being addressed by mainstream providers. You can think of a niche market as a narrowly defined group of potential customers.
For instance, instead of offering cleaning services, a business might establish a niche market by specializing in blind cleaning services.
To my way of thinking, animation history would fit the definition of a niche market.
How big is such a market? You would really need to do some deep research before even thinking of starting to write a book on such a topic, I would presume. Who knows, it may be a big success.
Here is a page on animation history that you might find interesting:
Another question to ask yourself is, how do you go about finding a publisher that would publish such a book, by an unknown, unproven author?
You may have written a fantastic book, but if you can't find anyone to publish it, it will collect dust.