This is because the default theme of retropie (carbon) is really not adapted for small LCD. You can install “GBZ35” or “pixel TFT” themes which have bigger fonts, and better rendering on small LCD.
Still the Raspiboy use the composite signal of the raspiboy, so the picture will always be a little bit flickering. But this is mainly in emulation station main menu. In games it should not be much flickering. It’s because emulation station native resolution is for big TV and it does not scale down perfectly.