The problem with using the orange diamonds is the timeline is clicked/dragged a lot and we don't want to make that require precise mouse movements.
You can use the dopesheet below the graph, so you still have access to the white keys. In the screenshot below I clicked the white dopesheet key and it selected the graph keys too.
Loading Image
The Sync
button is specifically for that configuration and ensures the dopesheet and graph keys line up horizontally. In that screenshot I used the dopesheet view settings to hide the dopesheet toolbar. The dopesheet view could be made even smaller, so only the first row is visible. Maybe we could provide a way to hide the timeline, too.
We considered showing something like the keys in the first row of the dopesheet somewhere in the graph, but it made the graph a bit cluttered. We may still do it in the future, but not for 4.0.
We could use modifier keys so dragging a key would drag all keys on that frame. I'm not sure if that is the best use of modifier keys though, as there are only so many available. Also it's not great to hide a relatively common action without any UI counterpart, making it only available if you know the modifier keys.