What is https://httpyac.github.io/ missing in comparison to postman, insomnia or hopscotch? I really like being able to run .http files as unit tests in ci as well as during debugging.
I would be happy to hear what you think of ownCloud Infinite Scale, a go microservice implementation of the ownCloud WebDAV and sharing APIs. We are limiting ourselves to file sync and share, but integrations with collabora and OnlyOffice are there. Just don't expect server side encryption. It was only added to the oc10 codebase to prevent storage admins to read the data. It does not protect against evil sysadmins in the least. Imo an encrypted filesystem or overlay filesystem can be used to solve that use case.
If you need E2EE cryptomator works just fine.
That being said, oCIS does add other things to the table: for an out of box experience we bundle a small OpenID Connect Identity provider that should suffice for a small home lab.
And we now support parts of the libre graph API, which is an effort to build a federated graph API closely following Microsoft Graph. Obviously, our team focuses on file sync and share, but e.g. the Kopano guys are working on the PIM parts.
The team is busy preparing the next release and customer projects based on the new stack, but we are happy with what we have achieved so far.
I am definitely interested! I have some initial thoughts, and I'll post them on rocketchat as well later.
* It looks very competitive for people who use Nextcloud just for sync/share and don't want the cruft
* To be competitive in the homelab space, you really need a docker container!
* LDAP auth is a good start, but SSO would be even better; SAML is only available on enterprise, but not many selfhosters use that anyways
* I don't think there's much need for another OIDC provider in the selfhosting space; people are going to use Authelia, Authentik, or Keycloak anyways.
* Having an OIDC client that can use the above, which I already have set up, would be the best scenario
* Finding stuff on your website was not super intuitive; I wanted to see what's available in the community version (which I eventually found in pricing), and then I wasn't quite sure if that page is up to date for OCIS (ditto for the feature comparison page)
This is a great slide deck and directly follows what I learned about this over the years. In nearly every case I think you want a flattened table of the hierarchy. I hadn't heard the name closure table before so thanks.
Not sure the age on that slide deck but Sqlite has definitely supported recursive table expressions for years now.
In some ways it is more permissive with syntax allowed in the recursive portion than Postgres.
Recursive CTEs are almost never what you want. They're the database equivalent of pointer-chasing in code. If you design your schema such that you need a recursive CTE to query it, be prepared for bad performance unless your CTE only ever does a handful of iterations over a handful of rows.
I like paths for representing hierarchy, but closure tables can also be a good idea, depending on what you're modelling and how you query it.
> They're the database equivalent of pointer-chasing in code.
That's the general case, but more specific CTE queries can be optimized, e.g. by adding database indexes. Recent versions of Postgres have greatly improved wrt. not making CTE's overly inefficient.
I said recursive CTEs. CTEs being an optimization barrier is a different issue - and often desirable with Postgres with its lack of optimization hints. Hence with not / materialized etc.
I'm a big fan of the nested set representation. It hits the sweet spot of compactness and expressivity - queries for parents, children, ancestors and descendants are simple integer comparison and pretty fast with the right indices.
The only downside is that this requires updates of a large number of rows whenever the tree changes (not all rows though). If you have a sharded table that could be problematic.
Back in 2011 we built yatta solutions as a startup around template based roundtrip engineering. UML lab would use templates to not only generate the source code from UML diagrams but also to 'parse' the code.
I don't know the current state of https://www.uml-lab.com/en/uml-lab/ because I left for other adventures, but if you are looking for a tool that can actually keep code and UML in sync give it a try.
Hm, let me give a short history of owncloud reva and ocis.
Many years ago CERN chose ownCloud over other solutions because of the state based sync. They could use that to let researchers sync petabytes of data residin on EOS, their custom built storage solution. That ownCloud supports custom storage implementations made this a lot easier.
The did suffer some database bottlenecks and decided to extend EOS with features that the sync clients needs: tree modification time propagation (so that the etag of the root changes when anything in the tree changes) and size tree accounting (so you can see how many bytes are hidden in a folder including all children).
The basically maintained a fork that was half PHP, half c++. Together, we added APIs and interfaces to the codebase to make the file cache implementation exchangeable.
They went ahead and implemented a golang service that could serve the api requests, while the web ui was rendered in PHP. It should be possible to dig into the details by looking at the public CernBox repo. Code archeology ;)
Anyway, all that was before the nextloud fork.
To be honest, CERN has tried to convince ownCloud to switch to a different architecture for years. With some long held opinions leaving the company we were free to reassess our options.
ownCloud has long had problems with long running operations being killed by eg. php-fpm timeouts. So I was evaluating and comparing different PHP frameworks like reactphp, swoole and amphp. I wanted to be able to offload workflows that are triggered after an upload has finished to a proper background job. And I really wanted to stay in PHP land because of all the already written code and existing apps.
But I noticed that they all had one thing in common. The all reimplemented a redis and a mysql library which made me wonder why?The existing drivers would block network IO, killing every concurrency gains you could gain by using a reactive framework or the go like coroutines of swoole.
It finally dawned on me that PHP may not be the best language to implement a service that has to deal with file IO. A systems language is more fitting.
Go or rust? Two years ago that was way easier to answer. Furthermore, CERN had a working server side API of the ownCloud webdav and ocs endpoints written in golang.
We sat down and discussed how o file sync and share solution would have to look like at the protocol level. What services are necessary and how could we make the existing code more modular to support other storage backend a than EOS? How can we get rid of the centralised database?
The result were changes to the CS3 api, making reva a reference implementation and using ocis to tie it together with user management openid connect and thinking about migration strategies. There is still the possibility to wrap PHP in a sandbox like service using roadrunner.
I would not have dared to start from scratch. But with an existing codebase that was used in production the decision became easier. The story is not over, yet.
Yes, we are leaving things behind. But we can embrace new things as well. And I am happy to be able to work with an awesome team to see this through. Every helping hand is welcome.
Lukas! Yep, for now, we focus on file sync and share.
Personally, I am looking into kopano or etesync to replace calendar and contacts. We already embed kopano konnectd as the OpenID provider so, there are synergies.
We do maintain backwards compatibility for the WebDAV, OCS and OCM apis and the clients are all working. The inner hook system of oc 10 obviously is gone, just as well as the internal PHP APIs. Instead we now rely on the CS3 APIs that describe a set of services in a microservice architecture to extend or customize functionality. Something that has always been fragile, because a bug in an oc/nc app could affect all other services.
Furthermore, we did not have to start from scratch but contributed to reva, which has been powering CernBox for a few years. It is the reference implementation for the CS3 api.
So, we are very much Protocol driven, which makes changing the implementation language possible if needed. And everyone is free to implement his own service replacement if he needs to. The extensive acceptance testsuite will tell you what is not working compared to oc 10 or ocis. For now, we are pretty happy with go and the initial benchmarks.
One more thing regarding core developers having moved from oc to nc. The server is just one piece of the puzzle. None, literally, none of the desktop, iOS or Android developers paid by ownCloud moved to the fork. It feels as if they were not even considered important. The sync protocol and the end user experience on his daily driver really is what matters. I am very happy that we as ownCloud finally grabbed the chance to really tackle the file sync and share part with the right tools for the job. Without being pulled in a gazillion of directions. I cannot express my gratitude for the rest of the oc sales and support team that keeps oc 10 well running and bought us the time to actually make this step. Truly exciting times
Keep in mind this is still tech preview. There is still a lot of work to do. Helping hands welcome. And yes we need to work on our communication skills...
Full disclosure: oc employee no 7 and ocis tech lead here. Originally hired to work on full text search in ownCloud 10. Good old times. Cheers to all the oc and nc devs: it should be possible to implement the internal PHP API of oc/nc using roadrunner. That might allow existing PHP apps to be wrapped in a dedicated microservice. But that is not something we are investing time in. Feel free to ping me on https://talk.owncloud.com
Are you looking into leveraging Syncthing for filesync? I always had the experience that, even recently, it was more reliable and faster. Its also written in Go.
No, but thx for the pointer. AFAICT the difference in the sync protocol is a block (syncthing) vs file based (owncloud) protocol. There are several reasons why we prefer the file based protocol.
Currently, all our clients sync using webdav. Changing the protocol would require rewriting that part of every client.
Another reason is that ownCloud as well as ocis are used to access files that resides in other existing storages, eg. S3 or CEPH. Translating those protocols to a file based sync is a lot easier than adapting syncthing (just my gut feeling).
Another aspect is that the file based sync is also state based. We can use the etag to detect changes and immediately start syncing in a breadth first approach. We are not doing that, yet. But the windows cloud vfs we implemented for the desktop clients makes this one of the next steps.
The sync in ownCloud feels slow because the clients are polling. There is currently no persistent connection that the server could use to push changes. But to be honest we have been bitten by firewalls between server and client so often that I doubt we can do real push notifications. Which is why I personally am aiming for long polling. For mobile devices we have to rely on the existing notification infrastructure from Google and apple anyway. Anyway, there are ways to speed up sync. Which switching to go made a lot more implementable. But one step after another.
We already implemented TUS for uploading files in ocis and all clients. It is a well designed extendable upload protocol that covers a lot of corner cases we have experienced first hand with all our clients. We contributed our experiences and are planning a batch extension that we would use to group lots of small files in a single upload. Delta sync should be a TUS extension as well, IMO. But that is mostly in my head.
First, we want to get the basic file sync and share features fully implemented.
So ... Yeah ... Changing the language and the architecture kind of made a knot disappear and we are moving forward.
In that regard, I personally have an eye on the Microsoft graph api which we could use for file management. Input welcome.
