Ask a HVAC guy that knows his air flow stuff well?
Even without regards to air flow, I would likely increase to 6 inch and avoid flexible duct as much as possible. If a typical in ceiling fan, you get more fan noise if there is too much restriction. If an inline fan and the blower is further away you usually won't hear increased noises because of duct size, plus flex duct becomes somewhat a muffler in those kinds of applications, though it does lessen air flow.
I believe my HVAC buddy once told me 6 inch was what is recommended for ~110 CFM as the base rule when they are sizing their duct runs, sort of like we automatically select 12 AWG for a 20 amp circuit, there can be factors that cause adjustments though.