Figured it out. Didn’t have the correct order for the rules. It’s not as obvious when reading the documentation and maybe some further examples would be helpful for others.
Options +FollowSymLinks
IndexIgnore */*
RewriteEngine on
# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# otherwise forward it to index.php
RewriteRule . index.php