I accidentally misspelled the name of an artist as I created the artist, and now I want to fix that mistake.
Can I rename the artist, or do I have to create a new one with the correct name?

Just make a new tag that's spelled right, and maybe alias the old mispelled tag to the new spelled tag

What dripen arn said, but you don't have to make a new tag, simply alias request in the forums for the new name.

You don't need to make an alias if it was just a one-time accidental misspelling. Just put in the correct tag, delete the goof tag from the post(s), and continue on as usual.