I wish this could be possible.

Eventually you'll have to track every minor version of every library when not every site stays up to date. At that point it's (almost) effectively a CDN.