Delete - Step Input    ⌃⌫

There is no delete button available on the Step Input Keyboard. When the keyboard is open, actually has focus, there are no focused areas, so the normal delete key should be available.

In any event (see what I did there) control-delete is used to delete the events that are being edited.

Use step input recording in Logic Pro — Apple Support

Step input allows you to insert MIDI notes when you’re not recording in real time. You can use step input to create note runs that may be too fast for you to play or to replicate sheet music that’s too difficult for you to play.

I find it interesting that the documentation doesn’t tell you how to remove the things that you have inserted. That leaves us to think that we have to switch to an eraser tool, or some other method.