Premature optimisation often makes things slower rather than faster. E.g. if something's written to have the theoretical optimal Big O complexity class, that might only break even around a million elements, and be significantly slower for a hundred elements where everything fits in L1 and the simplest implemention possible is fine. If you don't know the kind of situations the implementation will be used in yet, you can't know whether the optimisation is really an optimisation. If it's only used a few times on a few elements, then it doesn't matter either way, but if it's used loads but only ever on a small dataset, it can make things much worse.

Also, it's common that the things that end up being slow in software are things the developer didn't expect to be slow (otherwise they'd have been careful to avoid them). Premature optimisation will only ever affect the things a developer expects to be slow.