Billed for wall time. whichever plan you are on you get in credits, so hobby plan gets $50 of credits and beyond that billed on per CPU wall time.

CI Builders/QA Agents can do this very well. User session starts, bring VM up with content + dependencies, when session is done throw it away. Keeps it clean, debuggable, fast and cheap.

Freestyle has really built with this in mind. We propose a primary architecture built around declarative configuration of the vm with a git repo as external source of truth.

If the VM crashes/you have another idea/you want to try something else it should be reconstructable from outside of the VM.

However, I think this is potentially unrealistic. While it is the ideal architecture, I hear more and more every day people who just want to have the VMs run for months at a time.

Yessir, we haven't mastered it yet but we've compiled the kernel with enough flags for stuff like nftables and KVM to make it possible.

Proxmox forking in a few seconds is a miracle!

These are likely only a better value for you at large scale/if you start wanting to run hundreds.

Generally compared to those two more powerful. Freestyle VMs are full Debian machines, with support for sysd, docker in docker, multiple users, hardware virtualization etc. Daytona and E2B are both great "sandbox" providers but don't really feel like VMs/you can't run everything you can in an EC2.

We also support the forking/snapshotting/long running jobs that they struggle to.

50 is not heavy, what is heavy is 1000 VMs that can be paused/brought back 50 in 1 second.

Though generally ya, handrolling this stuff can work at the scale of 50 VMs, it becomes a lot harder once you hit hundreds/thousands.

So fork time is actually O(1) with VM size, its 500ms even for 64gb + disk. We're using some pretty weird COW techniques to pull it off.

Insane. Does it possible to fork to another bare metal machine? Maybe multi region as fly io. If not, I bet you have huge disk sizes on your machines to store all the snapshots (you said, you store them and bill only for disk space).

So forking across multiple nodes in that speed is not possible — we run extremely beefy nodes in order to avoid moving VMs across nodes as much as possible.

We are researching systems of hot moving VMs across VMs but it would have very different performance characteristics.

Yeah, I see. Is it possible to get a corrupted state? Let’s say we had realtime database actively writing at that moment?

It is impossible.

Our tech is not decades old so there is a chance we've missed something but our layer management is atomic so I'd be shocked if you'd be able to corrupt state across forks/snapshots.

Yes! You can def run something like K3s in these VMs.

Never tried them, I think the weird thing about VM providers is the difference really all is in the execution. These guys seem great in concept but I don’t know enough about how they properly work.