Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Storage and compute separation is key to scaling data workloads. Here, scaling could be w.r.t volume/shape of data, number of concurrent jobs on the same dataset, complexity of each job etc. In such an architecture, network access is unavoidable. And, to if you have multiple jobs competing for access to the same dataset concurrently, your sequential access can turn into semi-random access. You also have concerns about utilization of resources while being scalable w.r.t arbitrary bursty contentious workloads. These are the things that make it complex w.r.t managing IO resources.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: