Still don't understand how Android doesn't have built-in support for CardDAV/CalDAV. It's built-in on iPhone. And the built-in ActiveSync support has been left to wither and die.
It's surprising that it's built-in on iPhone (though unsurprising that e.g. the support built into iCloud is so poorly documented).
Generally, supporting open interoperable standards that make switching providers easy is bad for business, so I'm generally a lot more surprised at where it is supported, rather than at where it isn't.
FWIW I sync my Android data with my iPhone (personal -v- work devices) using the FDroid DAVx⁵ and ICSx⁵ apps, rather than the author's one-time export solution, and they seem to work ok.
> Generally, supporting open interoperable standards that make switching providers easy is bad for business
It was one of the reasons I switched from Android to an iPhone. It's really annoying to use Android if you don't want to use Gmail and Google Calendar, while on iOS it's super easy to use a third party for email, calendar and password manager.
But that's of course just an anecdote, I'm sure you're right and it in general bad for business.
100% agree. Apple gets a lot of flak for lock-in, and they are bad, but nowhere near as bad as Google.
I've run an Android phone without any Google account at all (something that's not possible on iOS), using a combination of FDroid and Yalp to work around the lack of Play. The fact that this is possible lulls people into this idea that Google is more open, but the reality is it's pain. You're locked out of many basic phone features, notifications are flakey, many updates don't work, and once you do opt into a Google account it invades every part of your phone's experience.
At least with iOS I can sign up for a fresh empty apple account and it leaves the rest of the (admittedly limited and very locked down) phone to me.
It was the same for me. Somehow, once you look past the fact that everything is at Apple's mercy and I can't make an app that runs on my phone unless I buy a computer I don't want to run a desktop OS I don't like (boo!), iOS actually better implements Android's promise of openness.
Apple is in a position where they can't expect everyone to be happy with iCloud, which is great because it means they actually put effort into supporting (decently!) a range of third party services. Siri generally isn't as smart as Google Assistant, but it also isn't completely obsessed with Google services. That is to say, unlike Google Assistant, every app on my phone - without exception - actually knows what is on my NextCloud calendar. And it actually gets useful information from my email to do helpful things with.
On the other hand, if you look at how Android has progressed over the years, I think Google has been trying very hard to forget about Android's past as a flexible operating system that exists outside of Google's ecosystem.
> On the other hand, if you look at how Android has progressed over the years, I think Google has been trying very hard to forget about Android's past as a flexible operating system that exists outside of Google's ecosystem.
They don't even try with the stuff they extended android with, Android Wear and android's Daydream: no third party stores allowed, only loading via ADB.
Installing an app is no problem, and I was using a calendar app that worked okay. But I was unable to find an email app that I liked and all the password manager apps worked (or work) by (ab)using an accessibility feature that had a pretty severe impact on the performance of my phone.
And uninstalling an app is impossible on stock Android, if that app is made by Google.To achieve that I had to unlock the boot loader of my phone, for which there was no official method.
Meanwhile on iOS I can use the default mail and calendar apps that work fine and my password manager integrates perfectly with the os, while I don't need to use iCloud at all if I don't want to.
Those 3 apps weren't the only reason to switch from os, but they did make the choice for, and switching to, iOS a lot easier.
You need to unlock your bootloader, install custom recovery (TWRP) and then install a custom ROM (I use LineageOS). It's optional to install Google into AOSP-based ROMs.
You need to find instructions based on your specific device. Each one is different.
I personally really disliked how iOS hides the underlying filesystem and directory from me so I never went back to iOS. I have an iPad I received as a gift and while it's really nice in terms of usability, I don't like how cumbersome transferring files is in 2022. The weird quirks of Android are smaller issue if I can just mirror the entire internal storage to my installed sd card. I also dislike how the web browser on iOS is so locked down. Having an operating system behave really similarly to a desktop operating system makes me stick with Android in the long term. The open nature of Android means that the "missing" features can be added relatively easily, like the DAVx⁵ and ICSx⁵ apps. I also use FolderSync to sync phone/sd storage with cloud backup. In fact FolderSync can talk directly to NextCloud.
> Generally, supporting open interoperable standards that make switching providers easy is bad for business, so I'm generally a lot more surprised at where it is supported, rather than at where it isn't.
Exactly this. Google makes it very, very easy to have everything synced to a Google account on Android devices: just log in and it just works. You contacts, mail and calendar are all immediately available on your new device. If they'd implement CalDAV in an easy manner, it would probably be a more viable option for less tech-savvy people.
What the parent commenter said: from a business perspective it doesn't make sense to implement it.
Fortunately there are some good, relatively simple to use apps available to implement it for us.
>rather than the author's one-time export solution, and they seem to work ok.
The reason why a simple one-time export was my go-to solution, is because I really don't want my data on Google's servers. In the blogpost I explain how to import the data to Nextcloud and use that as a 'single source of truth'. Other devices can then sync up to your nextcloud instance.
I don't have an iPhone, so I'm not sure how/if this would work on iOS, but I am sure that it's possible to sync it up in a similar manner.
you're right, iphone can have carddav accounts, but you can't migrate easily to a selfhosted carddav account even on Apple.
If you want to transfer contacts from the main addressbook you have to go through iCloud and export to vcf for a full take, last time I checked on iOS 12 I wasn't able to copy between addressbooks. Sorry if this has changed.
But while Android can't do other carddav accounts by itself, you can export the .vcf on-device. So while Apple offers to embed another carddav account, the migration story is not better but worse.
It's at least seamless by installing a single app, Davx5. Alternate Android distros like Calyxos come with Davx5 pre-installed, so it might as well be built in there.
Yeah, though I've found with Pixel software, it's still a bit annoying because Contact/Calendar apps will always show your Google account first, then keep prompting you to turn sync back on if you turn it off, etc. Flash your own ROM and you get a perfect experience there, but you get a whole new set of problems. These tradeoffs are a lot of why I moved to an iPhone this month for the first time since the 4S. Say what you will, but iOS integrates with Fastmail like it was built for it.
Google doesn't want you using anything besides Gmail, plain and simple. Adding an ICS feed is ridiculously complicated--you have to add it on your desktop in Gmail, and the on each phone you have to manually enable the calendar. Why wouldn't they just have the ICS feed enabled by default?!
I've had to do 'subscribe to calendar' work on a few projects. With iOS, I can just give someone a link to ical data, and it checks whenever they check. It seems, in the google world, you giving a google product your ics/ical link, it gives it to google, which may check it then, then... periodically polls that link on google's schedule (which may be hours or days in between) with no ability to force the google products to poll for updated info.
Is there some trick to get google products to regularly poll (or poll on demand) for updated ical/ics info?
There is CardDAV sync built in to iOS Contacts, but I don't think the person who programmed it actually tried using it. When you create a contact on the iPhone, there's no reasonable UI for putting it in a group (it can be done, but it's ridiculously hard); and only contacts in groups are synced via CardDAV. So in essence you can't create contacts on the phone.
I can very easily, using only built in apps and set up, share a calendar, a todo list, a photo album, a music playlist, a note, an e-mail account etc. with my wife.
But I cannot share a contact list with her without resorting to complicated CardDAV setups.
Wait before you see the implementation of tasks on the google side. If I add a nextcloud tasklist to IOS. Siri can add tasks to it, ez pz. On a google phone there is no way to add tasks to nextcloud tasklists with voice commands. Not so OK google.
Same with the built in Mail and Calendar apps in Windows. On my Mac I can connect to my NextCloud/Mail-in-a-box natively but on my gaming PC running Windows I had to install Thunderbird.
iPhone has *DAV it because iCloud uses it for Calendar and Contacts as well. Android has only EAS because most companies use MS Exchange. There is no need to support CalDAV for the Android develops.
Nice, I did this too. It's brilliant as backend for my iPhone, except that they are not putting imagemagick into the docker image and so your .heic pictures have no thumbnails and you can't view them. Still looking for solution to that.
One thing, you don't have to expose that mariadb port, all services in a docker-compose file are on the same network unless otherwise specified (and you can address them by their container name, build-in DNS!) :)
Oh and you can super-power it by Using Traefik for https, you just need to add some stuff to your docker-compose.yaml file. Although OP mentions a reverse proxy already, outside of this compose file.
I wrote a bit about this too with more detail, but I'm too afraid that HN will smash my poor Corei3 server :p. I want to get all of it on GitHub at some point, and then share it.
Hi Teekert! Thank you for your feedback; I will 'unexpose' the mariadb port in this example. That is one less service open for exploiting! :)
I now about Traefik, but I use the Synology reverse proxy at the moment. I felt that it is not fair to include it in this blogpost, since it is not an open source solution and not truely in the spirit of selfhosting. I plan to make the switch in the future to another reverse proxy like Traefik, Caddy or maybe just Nginx, not sure yet.
Could you give me the link to the blogpost? I might link to it :) Quality conten t must be shared and I think we as selfhosters should make it as approachable as possible for newcomers.
I really have to think about the post and update it, perhaps I'll get back to you and put it on a vps, not on "the hub of my digital life", before I share it.
Actually almost everything I learned and use I heard about on the Self-hosted podcast [1], and on one of the host's GitHub repos [2] (who takes personal infra as code to the next level).
Yeah, I know, frustrating because I just use it for the family, I don't think they will be uploading malicious SVGs. If I wouldn't use the container, it would be quite easy to install it myself.
As a note for anyone interested in doing this, imagemagick by default doesn't support heic, you need to build it from scratch yourself for it to work (at least I needed to on ubuntu 20.04) literally did it yesterday for a little sideproject:
Yep I'm using Traefik as a proxy for multiple docker containers. Guacamole, pihole, grafana, nextcloud, portainer. Admittedly, I'm not using nextcloud for anything but testing my pixel with CalyxOS right now.
I've used Nextcloud for my Contacts and Calendar for several years now. Works great on my iphone and linux computers (evolution + gnome-contacts/gnome-calendar).
I also use Nextcloud for my Notes/Task Lists (also works great with iphone and evolution), bookmarks, and recipe management.
My only issue with Nextcloud, is I really with they had an LTS release with at least 2 years of support. I have to keep my server moving between major versions pretty much every 4-6 months (you can't skip versions), or the clients which tend to get auto updated (especially on iOS) stop supporting the older server versions.
Just today, I received a text message from Google that my account has been disabled. The stated reason is "Spamming". I've been actively using this account as my main email for more than 14 years - back when you Gmail was intive-only. I rarely send any emails, and I'm sure my accounts has sent less than 50 emails in the past year. I don't know how an account with so few sent emails can be marked as spammer by any AI. But it was just the last day that I read the tweet saying files containing a "1" are marked as copyright infringement, so I shouldn't be so surprised.
Like the author I've also seen many such horror stories before, but always thought that wouldn't happen to me. I'm setting up my new email address on my own domain now, and encourage everyone to do the same. I'm still waiting for my appeal, so have not yet exactly checked how many sites I've lost access to because of Google login. Anyways, I will not make the mistake of using Google login or an email with domain not under my control for registering on any site again, even the least important ones.
I worked for an identity company for years and preached the dangers of social auth.. especially when there is little chance/process for appeal.
If Google kills your account, gmail and youtube are gone. Every social auth account is frozen. No clue what happens to your Android devices but your Play purchases are gone. Your Google Voice number disappears. It's a bad place to be. The story isn't any different for Facebook (Whatsapp, Instagram, Occulus), Apple (icloud, app store), and many many others.
And even then, I'm starting to see the same dangers for any centralized auth provider.
Social auth isn't that much better than email auth which is what social auth replaces. If you lose your primary @gmail there goes your accounts to everything. Some services let you change your email but most don't and your email is your account. Having all your accounts exist in a dead-man walking state isn't that much better.
Just a reminder for anyone not ready to ditch gmail...everyone should at a bare minimum be using https://takeout.google.com/ to export and backup their data if you use any google service.
Are there any services that can automate the Takeout process on a cadence? For example, every month or quarter initiate the takeout process and upload the results to Backblaze, Dropbox, OneDrive, etc?
I landed on a similar approach. Nextcloud is heavy for my purposes, but integrating everything into one service has benefits.
The only feature I miss from Radicale is the built-in support for versioning in Git. It's definitely handy to keep track of/roll back changes to my calendars, contacts, todos, etc on the server.
I wrote a small script [1] that periodically syncs changes from my Nextcloud cal/cardDAV feeds into a Git repository. Sharing it here as a bonus in case it's useful to others.
I tried to set Nextcloud up with Collabora a while ago with docker. The connection was not obvious and had many issues with SSL, networking and DNS. The examples they provide still need a lot of tweaking. Setting up the right Traefik labels was also a big trial and error.
Performance on document editing was a little laggy as well. Its really noticeable when your cursor lags behind what you type.
I'd love if the great folks over at NextCloud could focus on the following, which would really ease adoption:
* A polished, extensive example of a deployment in docker with all the bells and whistles (Collabora, cron, backups)
* An env var / config file way to link and setup collabora with nextcloud. No more installing from a gui and typing in endpoints. Please let me config manage this as code!
* a more polished initial setup wizard. I remember getting errors trying to specify all the env vars needed to set up the database. It wasn't obvious what was missing.
* A polished config for common proxies. Traefik is the obvious choice for docker I think.
I don't want this to come across wrong, I think the Nextcloud project is great! I feel like they could add a little time to the things mentioned above and give it a fantastic on-boarding experience.
This is great. I noticed however that they mention that getting kicked off of Google (which apparently is happening *A LOT* now, as nearly every day there's been multiple HN posts stating they have been locked out) - and this is the (much needed) thrust towards self hosting these services.
However, I noticed they left out making a mail solution. This is important and shows just how enormous of an opportunity there is in this space to make a quick, easy simple, self hosted mail solution.
Any protonmail-like self hosted *simple* solutions out there?
While it isn't hosting your own mail, Nextcloud does have some nice webmail interfaces, like rainloop, that can be configured using any imap/smtp server, like a desktop mail application.
My hosting provider has hosted my email for 15+ years (with my own domain), but their webmail isn't super great, especially for my non-techy family. They mostly use the mail app on their iphone, but Rainloop through nextcloud made it really easy for them to check through the web.
The issue with mail is that it is supposed to be a reliable service where people can contact you on a multi-year or multi-decade span.
Sure it is fun to self host mail in your Raspberry in the garage. But for many people the reliability of standard providers is a necessity. And that's even without accounting the anti spam filters
Author here. I have had some suggestions for this, because of this post. I have experimented a little, but I first want to fully understand what is going on in the mail landscape.
I want to write a beginner friendly guide on selfhosted mail in the future, so if you happen to have some tips, or things you want to know about, let me know!
Interesting; I am quite hesitant to move from Protonmail to a selfhosted mail. How long have you been selfhosting your mail? How do you deal with spam? Are your mails always received at the other end?
I moved to selfhosting email about 2 years ago from google gsuite. I use mailcow[1] and have it on Hetzner Cloud. I don't send much but they have always gotten to their destination. Spam is very minimal, next to zero. Mailcow has built in spam detection with rspamd and lets you train it if any spam does get through just by moving the email to your junk folder.
Another big thing I love about mailcow is that it has sync jobs so I can create mailbox's within it and have it sync my emails from other gmail accounts I have or really any mail host that you can connect to with IMAP. You can even set it so it deletes the email on the source account, which is great as most of my extra gmail accounts are just used to receive.
The dashboard is pretty great too and you only really need to use the shell, apart from the initial install, to update it every now and then. So many great things I have to say about it and how you can super power it like hosting custom domains and nextcloud among other things. I'm glad I made the change to selfhost. If you know what you're doing I'd 100% recommend you test out running your own mail server for a bit to test the waters.
Late reply but it's around 15 euro. I'm using the CPX31 on Hetzner Cloud[1]. It holds my main setup and the extras like nextcloud, custom domains, etc.
I'm self hosting with mail-in-a-box as well. It comes with a nextcloud install. It hasn't been flawless but it's been good and I'm happy to have some control.
With e-mail, I haven't had any problems with one exception. When e-mailing the local school system, they reject my e-mails. I looked into it and it turns out that their spam provider was blocking me because I was a private domain or something like that. It was a configuration on their side. Their tech support told me that I should "get an e-mail address with a normal extension."
Outside of that issue, my e-mails have gotten delivered. Between graylisting and the built-in spam filter software, I haven't had any spam issues. It's been smooth as far as that goes. The webmail (roundcube) isn't as nice as gmail but desktop and mobile clients are good in any event.
The mail-in-a-box nextcloud install does use sqlite which means that you should make sure to backup contacts in case sqlite breaks. It broke for me once but I was able to copy my contacts from Thunderbird back into the system without any real problems.
Calendaring works pretty well with Nextcloud but I haven't found any calendar software that I really love. The web software is good but not super fast. Lightning has gotten better but still feels bolted on. Kontact calendar is too groupware-oriented for my personal use. Evolution never quite felt right to me. The built-in Apple calendar and Samsung (Andriod) calendar apps work fairly well.
I’ve been running MIAB (mailinabox) for business and personal since 2016. I haven’t seen any major issues with deliverability but YMMV. You have to register with all the feedback loops. Once you do, you get feedback when something you send gets marked as spam. Spam on MIAB is handled with postgrey and with spamassasin. It’s pretty good, although I’ve had recent issues with spam coming from gmail and hotmail. I’ve also customized some of the config to bounce Microsoft Sharepoint and Google Docs since neither companies control spam coming from their networks.
I self-host on a small-ish VPS. I don't really have to deal with spam, mail-in-a-box comes with pre-configured spamassassin. I haven't had issues with deliverability. When I created the box I did a scan of the IP and domain, found out that one antispam provider was blocking my IP, I sent a request to unblock it explaining this is my new IP I just acquired, they unblocked it. That was the last time I had to deal with deliverability issues, in any case I have an automated warning that I set up on MxToolBox. Let me know if you have more questions.
I use nextcloud for my card and cal DAV, but I no longer need a lot of the other features of Nextcloud, has anyone else run something like https://radicale.org/v3.html on serverless or something?
I find NextCloud in itself to be pretty bloated and trying to do too many things with a lot of poor implementations. Syncthing is great for file syncing and there are a few good Cal/CardDAV options. I also find PHP applications incredibly frustrating compared to more simple tools. Vdirsync isn't perfect but being able to backup and sync across providers is awesome and I love that I can view/sync the files natively vs having to routinely backup an entire database and use specialized tools to examine contacts and calendar entries.
I'm hopeful someone will release a simple Go CalDAV and CardDav server, or that someone will contribute to making official packages for CyrusIMAP.
* https://sabre.io/baikal/ - written in php...which i see that you are not so crazy about...but i note it only because it is quite solid reputation.
I'm curious why would you want a calDav, cardDav server written in Go? Is it for scalability? Or, ease of deployment? I am not judging your preference at all; i'm genuinely curious? Also, separately, i first learned of cyrus imap from a blogpost that FastMail folks posted, but do not much about it (other than it is highly respected as a platform for mail, calendar, contacts)...Is it built in Go?
> i first learned of cyrus imap from a blogpost that FastMail folks posted, but do not much about it (other than it is highly respected as a platform for mail, calendar, contacts)...Is it built in Go?
Cyrus predates Go by quite a while! The project started in 1994. It's implemented in C.
I would suggest EteSync [1], they do have an adapter [2] that would remove the e2e encryption, but I recommend running that adapter locally and not over the network to actually benefit from the e2e.
>I also find PHP applications incredibly frustrating compared to more simple tools.
Really? I find them so much simpler, just stick everything in a folder and point Apache at that folder. No messing about with extra layers like docker.
Export into vcard will cause loss of contact photos. Google takeout saves the photos as extra files (you don't want to have to re-add them manually). The easiest way to move contacts between accounts I have found is with cardbook for Thunderbird.
I've found NextCloud a little buggy. OwnCloud seemed a lot more stable (used it for years). NextCloud has more features and is more secure I guess? But sometimes stuff doesn't work quite right. Auto Upload for phone for instance is hit and miss.
It does seem there are less bugs over the past year or so or maybe I'm just learning to deal with it, but it does seem OwnCloud was more stable.
I recently set up NextCloud on a Raspberry Pi and found the process a little more convoluted than I would have liked - but that was mostly because the linuxserver.io image for MariaDB seemed to have some problems during database initialisation which I couldn't figure out. I switched over to use the yobasystems/alpine-mariadb image and that worked flawlessly. I haven't tried syncing contacts or calendar yet but that wasn't really my primary motivation - I just wanted self-hosted files with a nice interface. The NextCloud UX is a little clunky imo but I'm not complaining - it's a great piece of software especially considering the price tag! I also am trying to do open-source everything so it is very attractive in that regard.
the popularity is deserved as there exist adjacent apps and community forums for help. The minimal chat/webrtc solution "Talk" is great for family and friends, the (Android) Talk App itself made leaps last year. It will also write plaintext rows per chatline to the db, but that's okay. Things transferred via webdav can be clientside encrypted, cal- and carddav again is cleartext. Etesync can do it encrypted but then you'd leave the all-in-one package. It's even more fun with customroms having davx and a webdav sync solution already integrated.
What i miss un articles like this is the server/hardware choice and reasoning for it. The instructions how to setup nextcloud are on the nextcloud homepage, so no reason to repeat them.
Author of the blogpost. Thank you for your feedback. The problem with hardware choice, is that many selfhosters have really low-power, budget, powersaving hardware, including myself. The point of this article was for me to discover how I could move some of my data away from Google, of which this is one piece.
I had this running on a Raspberry Pi 4 in this blogpost, which functioned just fine; although only with one user and little apps installed. I might do a follow up when I have used Nextcloud more extensively and explore different hardware options.
I switched from dropbox to nextcloud recently. I use pis for VPN gateways and DNS servers, but don't trust them for things like data storage.
I host nextcloud via TrueNAS SCALE running on a dual E5 + 4x2TB NVMe. TrueNAS handles the docker images with a kubernetes cluster managed by middleware. Straying from default settings is a time-consuming path, but the defaults work great.
Nextcloud doesn't feel perfect with lots of UI jank (initial photo upload was a real headache), but I trust the data put there will not delete itself. I love having the ability to selectively view and sync files with native OS explorers on windows, mac, and iOS. iOS photo backup with live photos is also something no other solution I've seen can compete with.
I'm also running NC on a RPi 4 (8GB), with an NFS share from my QNAP NAS as its data store. I didn't go the docker route, but rather elected to just do a local install.
I have ~12 users, ranging from a few with less than 5GB of data to a couple power users with ~100GB each.
Overall, I'm quite happy with NC. Sure, there are a couple of rough edges, but for the most part, having a single install handle file, contacts, calendar, chat, etc. has been great.
Before everyone moved to cloud services, self-hosting was a goto choice for nerds who wanted their own stuff, including me. One challenge that self-hosting has not really conquered is availability. If your rpi dies, so does your self-hosting. You might want to explore (and blog about) setting-up a failover device. I'd definitely read about that.
I am running most of my webservices of my unraid nas it runs a cheap passive cooled ASRock J3355B-ITX with 16GB RAM and 2TB SATA Cache with a 80TB JBOD array. All in a really nice compact Fractal Design Node 304. When all HDDs are running it sips 90W at standby 15W.
The nice thing about unraid is the array needs to only spin up the disks that are being accessed AND the parity disks (only while writing). I run two parity disks so that I could lose two disks in total before the array fails. While running a parity check all drives get spun up and the total noise level is arround 35db but this totally depends on the drives you are using. For me it's mostly Western Digital Ultrastar DC HC530 shucked from WD Elements because of cost savings.
