Is as simple as that. Not only does it help you to get better organic seearch results, because Google likes fast sites and even punishes slow sites, also real people will more likely stick around on your site when it is not as slow as waiting in the line in the supermarket standing behind an lovely old lady looking for her purse, while the other row (you just stood in) runs twice as fast.
Nobody likes waiting.
Now go get yourself a fast WordPress theme. And please, do not put to much energy in tweaking it. Let your content talk for itself.