R&D funding for rocketry has been very sparse, and large scale rocketry has consistently been extremely risk averse. Because of that there have only been a few different rocket engine designs which have been advanced towards production suitable for use on orbital launch vehicles. In the US, for example, there was a period of development on LOX/Kerosene engines in the US during the Apollo program but then the Hydrogen fad came along with the Shuttle program and that largely stopped until SpaceX came along. The SSME, RS-68, and SpaceX's engines are the only rocket engines with more than 500 kN of thrust that have been developed in the US in decades. That gives you a sense of how sparse the development landscape for rocket engines has been.
We simply have not explored much of the design space for rocket engines as of yet. We haven't even explored many of the most promising propellants (LOX/Methane being one example), let alone most of the rocket engine designs. Pursuing a new engine design and scaling it up to a size suitable for orbital rocketry is typically incredibly expensive and difficult, which is why it has been done so rarely.
Additionally, government backed R&D has often suffered from a bit from "Wile E Coyote" engineering. If a project works on something new and that project fails, or sometimes merely ends, then whatever was being worked on is often abandoned and treated as though it's impossible. Just look at the failure to followup on the DC-X work, despite the program overall being quite successful. Yet now there are several companies who are following up on those design principles (VTVL reusability) such as SpaceX, Blue Origin, and now Firefly. Aerospikes have a lot of potential but also a lot of difficulties, nobody has ever put in sufficient work to figure out whether those difficulties are surmountable.
We simply have not explored much of the design space for rocket engines as of yet. We haven't even explored many of the most promising propellants (LOX/Methane being one example), let alone most of the rocket engine designs. Pursuing a new engine design and scaling it up to a size suitable for orbital rocketry is typically incredibly expensive and difficult, which is why it has been done so rarely.
Additionally, government backed R&D has often suffered from a bit from "Wile E Coyote" engineering. If a project works on something new and that project fails, or sometimes merely ends, then whatever was being worked on is often abandoned and treated as though it's impossible. Just look at the failure to followup on the DC-X work, despite the program overall being quite successful. Yet now there are several companies who are following up on those design principles (VTVL reusability) such as SpaceX, Blue Origin, and now Firefly. Aerospikes have a lot of potential but also a lot of difficulties, nobody has ever put in sufficient work to figure out whether those difficulties are surmountable.