setValue:forKey: is part of the NSKeyValueCoding protocol, which among other things, lets you access object properties from the likes of Interface Builder. setValue:forKey: is implemented in classes other than NSDictionary.
setObject:forKey: is NSMutableDictionary’s reason to exist. Its signature happens to be quite similar to setValue:forKey:, but is more generic (e.g. any key type). It’s somewhat of a coincidence that the signatures are so similar.
What adds to the confusion is that NSMutableDictionary’s implementation of setValue:forKey: is equivalent to setObject:forKey: in most cases. In other classes, _setValue:forKey: _ changes member variables. In NSMutableDictionary, it changes dictionary entries, unless you prefix the key with a ‘@’ character — in which case it modifies member variables.
So in a nutshell, use _setObject:forKey: _ when you need to work with dictionary keys and valus, and setValue:forKey: in the rarer cases where you need to tackle KVP.