> just run the output through a completely separate model that checks whether the string that's about to be returned violates the prohibitions
The big names do do this. Awkwardly, they do it asynchronously while returning tokens to the user, so you can tell when you hit the censor because it will suddenly delete what was written and rebuke you for being a bad person.
This is super funny when it's hitting the resource limit for a free tier. Like... I see that you already spent the resources to answer the question and send me half the response...
It's kinda hilarious because if a movie had the AI start giving out an answer and then mid-way censor itself I would call the movie bad. Truth is stranger than fiction I suppose.
The big names do do this. Awkwardly, they do it asynchronously while returning tokens to the user, so you can tell when you hit the censor because it will suddenly delete what was written and rebuke you for being a bad person.