I haven't experienced fatty tumors with my chin (she's my first and I've had her for less than a year), but I have encountered them with rats, dogs, and horses. In my experience with those species, they did seem to be prone to coming back after removal, so I usually only bothered with having them removed if they got rather large and/or were in a location that caused discomfort or hampered movement/mobility. Basically, if it starts to negatively impact quality of life, then I'd have it removed, but if not, it didn't seem worth putting the animal through the stress and pain of surgery and recovery for something that was likely to grow back anyway. It only seemed to be a net gain in cases where the size and/or location of the tumor was causing issues for them.
In other words, I'm inclined to agree with your vet. Definitely keep an eye on it, as the location could hamper mobility if it gets larger, but if it's not cancer and doesn't seem to be bothering him, it's probably not worth removing, at least for the time being.