This idea of getting historical data fused from multiple private sources, to be able to remember properly what happened, was at the core of Memacs[1], an Emacs (org-mode-based) data-fusion framework feeding into calendar.

Here's a diagram from the Memacs repo:

   emails              -> memacs-maildir.py  \
   firefox history     -> memacs-firefox.py   |
   SMS                 -> memacs-sms.py       |
   RSS-feeds           -> memacs_rss.py       |
   bank statements     -> memacs-easybank.py  |>  Memacs
   postings            -> memacs-slrn.org     |
   git repository logs -> memacs_git.py       |
   svn repository logs -> memacs_svn.py      /
   |_________________|   |_________________|     |______|
   your personal data      Memacs modules        Org mode

This relied on org-mode having a good date-links-to-calendar and agenda system.

See the whitepaper, called "What really happened on September 15th 2008? Getting The Most from Your Personal Information with Memacs"[2].

This project is now a little bit dead, but the concept of private data fusion was fantastic, and transformed my view of calendars.

Agree that calendars are a little underused in that way, and would love to see more work towards that private calendar data usage.

[1]: https://github.com/novoid/Memacs [2]: https://arxiv.org/abs/1304.1332