Outer Limits 4x4 Board

Pulled out my lh drive shaft out to replace my cv's, the lh inner had dammage where its been spinning inside the housing.
There is obviously a bush there which did seem a little rough. All i had time for was to clean it with some emery tape and reasemble the new cv into the housing.
Is this a huge concern, should i be stripping the centre and replacing the bush, free wheel hubs or just turning a blind eye to what i saw.

there is supposed to be a neddle roller bearing in there. they are usually buggered and only about $30 from mitsubishi
cheers murray