Are potatoes hard to grow?

My soil isn't particularly fertile-- It's clay mixed with a ton of manure and kitchen waste like banana peels. It still has lumps of clay but it is pretty decent... I was thinking about growing some potatoes, probably the yukon-gold variety. I know sweet potatos are a completely different plant, but are they easier or harder to grow than regular potatoes? Not only that, but are potatoes a high-yielding crop or are they not worth the space?