My husband and I are similar travelers. We don't do much of the adrenaline pumping stuff either. As you mentioned, the best travel partners are ones that respect each other's interests. It is wonderful to find that.
You have certainly done more than we have thus far. The hot air ballooning sounds amazing. I'd like to camp out somewhere that has pitch blackness to view stars as well. My kids are 3 and 5 now though, so we are closer to having some extra travel buddies at a decent age soon.