Various factors can play a part in how your directory performs on your chosen hardware and environment. We will attempt to discuss these here.
Scale your cache to use available memory and increase system memory if you can.
More info here.
Use fast subsystems. Put each database and logs on separate disks.
Example showing config settings
https://www.openldap.org/faq/data/cache/363.html
Drawing here.
Reference to other sections and good/bad drawing here.
Discussion.