Don't stress - i would just leave it, but if you give your bike a real hammering, just get it welded/filled.
About two years ago i rode 60km with a few loose sprocket bolts. I actually thought i'd bent the brake disc or had something jammed in the spokes because the feeling i was very abrupt and jerky. I put up with it for a while because it was a timed event.
I eventually stopped to find i had lost 3 bolts, had two just hanging in there (no nuts), and one bolt doing its job. I had carved a nice 10-15mm groove into my swingarm - luckily there is a weld/join in that spot or it would have snapped off for sure.
I had it filled/welded, and haven't had an issue with it since.