I personally find with a little practice 6x is fine offhand, in most any condition. Then again silhouette shooters use 24x or more offhand, though I think that differs a bit from plinking/hunting use. Myself certainly by 12x, shooting normal offhand while doable extracts enough concentration that I don't enjoy it much. Though probably more practice offhand might change that.
I recall one reason the Army in WWII settled on 7x for binoculars was any more power was too hard to hold steady. Sounds about right for scopes to me as well.
For benchrest, I find 12x pretty good. And 16 or 20x plenty. Serious benchresters cannot get enough magnification I suppose.
So 4-12x might be okay, though not much more than your 3-9x. 6.5-20x or 6-24x I think is the upper end. 5-15 or 4-16 might be the best compromise all the way around.
So long way around saying, sure I think your experience trying 4x or 6x is about right.