The article is accurate

I misunderstood and thought this was a bill paid q1 2023 (after the new 3 year contract) not in q1 2022

They did build a self hosted alternative based on grafana, Loki, Prometheus

Had a whole team of 10+ engineers working on it for 2 quarters, then scrapped it because it performed terribly

The only thing that came of it was negotiation leverage with datadog ("give us X% off or we go self hosted")

What kind of scale of logs are we talking here? The company I work for run a self-hosted Grafana LGTM stack ingesting about 1TB of logs per day, it’s pretty snappy and works well enough, and only costs a few thousand dollars per month in GKE costs for the entire observability stack.

How much logging are we talking here?

GitHub has over 21TB of source code. Applications consistently pour through this data and emit logs and events. 1TB of data by breakfast maybe? In reality, we're not pushing logs to datadog, just metrics and event tags. Our level of cardinality, however, requires a lot of horsepower on the backend. Our attempted Prometheus transition was just not cost effective when attempting to view large sets of data over a large-ish period of time. Combined with the heavy lift of integration (we depended heavily on dogstatsd) it just didn't seem efficient to move to Prometheus, support the infrastructure required, all while migrating to microsoft's inhouse product.

I worked at Coinbase until very recently and can confirm this is Coinbase

They paid upfront for 3 years of usage, and yes they were burning > $20m/year on datadog

Dear Lord...the commission on that hog. The sales team eating good.

I was previously in sales and SaaS is considered the pinnacle of industries to be in for sales people. Medical device sales is the only real competitor when it comes to earning potential and my understanding is that's US specific and also comes with atrocious work life balance.

Back in the day I was auditing our support contracts for a place that I worked at. Basically figuring out if we were getting what we were paying for.

My favorite "overpriced support contract" was for an Oracle product. The cost of support was seven figures, and in the entire year a single phone call had been placed to support.

The best bit when I worked at one of those companies that had an expensive Oracle contract was this dynamic:

1. Can we use MySQL for <new product>?

No, use Oracle, we have a support contract

2. <oracle related issue occurs> Can we call that support contract in now?

No, let's try the inhouse expertise first.

3. <inhouse expertise comes up with barely passable hack> Can we check if they have any better solutions?

No, it's "solved" now

Like, what were we paying for? I have to assume there's per-engagement costs as well as the ongoing costs, given how hestitant our contract owning team were to let us anywhere near Oracle.

Did that end up being a “support” call where they sold you more services?

> 3 years

Okay, it seemed like it was 1 year.

So, only insane, not insane^2.

So you’re saying it’s $65 million over three years?

The headline seems misleading, like it’s for a single month.

yes but 20 million dollars a year.

20 fucking million

getting splunk with 1pb a month ingest isnt that expensive.

Please tell me I am allowed to say "you could just use grep and rsync". I must be allowed!

but only if you have "as-a-service" attached to the end of it.

or something with chatgpt

They'll probably throw in SignalFX for free if you ask them nicely.

Headline is slightly misleading

Earnings call said it was an upfront cost