Are you sure that poor speed is acceptable on desktop applications? What about applications that need to load complex forms where opening such form can take half a minute or more? Or when you need to create plenty of business objects, or when you need to do some textual processing. Good compiler means a lot in such cases. In last few Delphi releases @Stefan Glienke has continuously submitted hand crafted code to improve speed of some widely used RTL routines that have impact on overall performance. With better compiler that would not be necessary. Not only that, but with better compiler all other code we write would be faster and when speed matters we wouldn't have to bend over backwards, writing horribly unreadable code just to get the speed we need. We could just write the clean and readable and maintainable code and leave the dirty work to the compiler.
Speed always matters. Now, Delphi still might be fast enough for many purposes, but for many that is simply not enough. Once it was state of the art compiler, now it is seriously falling behind.