The passages in a contol valve are quite large and it would take a large piece of debris to remain there to cause problems, and those will usually show up as restricted movement of the control handle.
Debris may get caught up in the hoses or lines to that cylinder and restrict oil flow. Disconnect the hoses to the cylinder and place the ends in a clean bucket. Start the engine at idle and cycle the valve both ways for a couple of seconds and observe the relative flows. If nothing obvious shows, do the same for another cylinder that is working at the correct speed and compare the flow rates. If you find a marked difference in flow rates, flush the appropriate line in both directions into the clean bucket and check for chunks.
If the flow rates match, go back to what was replaced since the last time the loader worked correctly and check the flows at that point. The various flows should all be the same, or very close, as what comes from the pump itself.
This should isolate where the restriction is and appropriate servicing can be done to that component. It may be that you got a bum diverter valve, but not likely. At the moment, the lines seem the most likely place for debris to be the cause.