I agree with you 100%. To paraphrase (iirc) Larry Wall, the primary difference between programming languages is what they make easy. To adapt to benchmarking, we need to consider which algorithms the language's constructs, standard library and compiler make fast.
The question that should be answered in such comparisons is: when written using a language's idioms for time-efficient code, which language offers the best performance?