I have been experimenting with Redis and MongoDB lately and it would seem that there are often cases where you would store an array of id’s in either MongoDB or Redis. I’ll stick with Redis for this question since I am asking about the MySQL IN operator.
I was wondering how performant it is to list a large number (300-3000) of id’s inside the IN operator, which would look something like this:
SELECT id, name, price FROM products WHERE id IN (1, 2, 3, 4, ...... 3000)
Imagine something as simple as a products and categories table which you might normally JOIN together to get the products from a certain category. In the example above you can see that under a given category in Redis ( category:4:product_ids ) I return all the product ids from the category with id 4, and place them in the above SELECT query inside the IN operator.
How performant is this?
Is this an “it depends” situation? Or is there a concrete “this is (un)acceptable” or “fast” or “slow” or should I add a LIMIT 25, or doesn’t that help?
SELECT id, name, price FROM products WHERE id IN (1, 2, 3, 4, ...... 3000) LIMIT 25
Or should I trim the array of product id’s returned by Redis to limit it to 25 and only add 25 id’s to the query rather than 3000 and LIMIT-ing it to 25 from inside the query?
SELECT id, name, price FROM products WHERE id IN (1, 2, 3, 4, ...... 25)
Any suggestions/feedback is much appreciated!
Advertisement
Answer
Generally speaking, if the IN list gets too large (for some ill-defined value of ‘too large’ that is usually in the region of 100 or smaller), it becomes more efficient to use a join, creating a temporary table if need so be to hold the numbers.
If the numbers are a dense set (no gaps – which the sample data suggests), then you can do even better with WHERE id BETWEEN 300 AND 3000.
However, presumably there are gaps in the set, at which point it may be better to go with the list of valid values after all (unless the gaps are relatively few in number, in which case you could use:
WHERE id BETWEEN 300 AND 3000 AND id NOT BETWEEN 742 AND 836
Or whatever the gaps are.