No contest at all- Baseball is much, much, much harder.
The reason why Baseball is so hard is because you are hitting a round object (the baseball) with another round object (the bat). In Golf, you are hitting a round object with a flat object, much easier.
In baseball, you need to be (or close enough) physically fit so you can run the bases, hit the ball for a base hit, and hit a home run if you are a power player.
In Golf, anyone can play the sport, regardless of age.