RELU would have made it flat below zero ( _/ not \/). Adding the abs first just ...

		fancyfredbot 5 months ago \| parent \| context \| favorite \| on: Language models pack billions of concepts into 12k... RELU would have made it flat below zero ( _/ not \/). Adding the abs first just makes RELU do nothing.