Librevents: “liberating” data from Big Tech

⏸️ The project is temporarily on hiatus.

daline@proton.me (Data Liberation Network) — Wed, 15 Mar 2023 00:00:00 +0000

At the moment there is not the energy to develop the necessary progress, we need to strengthen the community to support it. There will be new updates when we have resolved this issue and can announce an updated roadmap.

Check the contacts section if you want get in touch.

🌱 Developer Meetup n.2; Report

daline@proton.me (Data Liberation Network) — Wed, 22 Feb 2023 00:00:00 +0000

Highlight: CURRENT VERSION ISN'T WORKING, afterall challenging Facebook has some complexities. Whatever: ✊

In this report we talk about: fixes, errors, fundraising. The meeting was about defining what should happen between February 15th and March 15th, the next of our online monthly meeting.

What to do in the next month

"keep promoting and perhaps fundraise", the same goal of the previous meeting. We need to continue and make this community grow, or, well, exist.
Facebook changed its HTML structure. It is expected, we know how to handle it, but read more below on how to reduce the impact of their obfuscation.
Complete a gathio installation, and eventually support the cross-posting there. It should be easy to add in mongodb a new event.
By inspecting some of the errors triggered in the last month, we need to better report errors, because, for example, if someone provide a wrong login and password, should be consider a different error rather than “the server didn’t properly worked”. reminder it was handy check the mobilizon version via API.

Fundraising narratives

Check out this (read only) cryptpad notes, it describes some of the narratives usable in fundraising.

Note🍀 this is our biggest weak spot. There is not yet any concrete effort, and often, fundations, supports the creation of new tools rather than adversarial approaches. It makes sense, but it is an additional vulnerability this project faces.

On Facebook HTML obfuscation

HTML randomization is one of the ways by which facebook wants to make, every scraping action, more expensive. An update for them may be at no cost, and if the technology is to be maintained, for scrapers it means having to do an update every time.

Above you’ll see HTML from facebook.com/events/$EVENT_ID page.

No meaningful classes
Nesting with multiple layers meant to randomize this
Usage of span div h-* and p in an arbitrary fashon, relying on CSS to produce the intended result

The more mature and robust the scraping technology is, the more it will remain viable at the expense of upgrades. It may seem counterintuitive, but Facebook’s final interface does not change very often, and if you build a scraper attono to those elements, if span becomes div and h1 becomes h3, it will only break the most delicate scrapers.

Unfortunately, despite knowledge of this theory, the extension underwent major changes in 2022 to move the prototype forward, so right now it is in the “weak” stage and updating the HTML causes the title and description to be missing:

This is a backend problem, as it should not allow posting of events with no title/description, and should be reported as an Error. And also, scraping reliability problem cannot be, in 0.3.x version, solved once and for all. It is an improvement part of a larger refactor and redesign that would come along with Manifest V3.

(at least) three kinds of `Errors`

At the moment the error collecton in mongodb looks like this. Below you’ll find an error triggered when I changed the username in something invalid:

{
  "message": "User not found: {\"response\":{\"data\":{\"login\":null},\"errors\":[{\"code\":\"user_not_found\",\"field\":null,\"locations\":[{\"column\":7,\"line\":2}],\"message\":\"User not found\",\"path\":[\"login\"],\"status_code\":404}],\"status\":200,\"headers\":{}},\"request\":{\"query\":\"mutation Login($email: String!, $password: String!)\\n    { login(email: $email, password: $password)   \\n    { accessToken refreshToken user  \\n    { id  \\n      email    \\n      role   \\n      __typename }  \\n    __typename}\\n}\\n\",\"variables\":{\"email\":\"mobilibr@mt2015-\",\"password\":\"experiment\"}}}",
  "publicKey": "E1CdKsPZLCrfhsrRYYR8t8snahxx1LAaDNkSX9DJvWEh",
  "when": {
    "$date": {
      "$numberLong": "1674805040484"
    }
  }
}

But we should not tread all the errors as the same, we might need to identify:

Login/Password errors, coming from authentication failures in Mobilizon backend. This require an adopters intervention.
Inconsistecy errors, like the faulty scraping, that might provide inappropriate data. This require us to fix it, or potentially the adopter editing the event and manually fixing it.
Any other kind of failures as unexpected bug. This require us to further investigate.

The harvest 🍂 after January'23 planning

daline@proton.me (Data Liberation Network) — Fri, 10 Feb 2023 00:00:00 +0000

🎉 Librevents has a Mobilizon group. Check out the 🌱 Developer Meetup n.2 event announcement. It would be in February 15th at 18:00 CET; Join the Matrix channel to introduce yourself and get a Jitsi link.

Read more to get a summary of the achievements after the 🌱 Developer Meetup n.1.

Software and site updates

Version 0.3.6 published - due to a critical bug detected immediately after the first series of announcements - as usual. The bug was discovered because the statistics were not showing any updates, so they actually worked ;)
The software was contextually extended, and so were the statistics, to see total activities, errors, and successfully released events. This should help to understand the overall stability.
The pages about and who are we talking to have been revised.
Added an API to retrieve errors. Useful for debugging, but currently requires an admin key to access it because it may contain information we don’t want to get out.
Presented librevents at OFFDEM, got fascinating contacts and a fully new context where to talk about it!

Heard from the Fediverse

Friendica visualizes events in this very readable format great for integration!
We also get to know about freeCodeCamp/chapter which is a software that handles events, but does not seem to handle activitypub yet?.
From that issue you can find this detailed analysis on Event Sharing In The Fediverse at socialhub.activitypub.rocks, among them also:
Wordpress PHP ActivityPub support for events.

🌱 Developer Meetup n.1; Report

daline@proton.me (Data Liberation Network) — Sun, 22 Jan 2023 00:00:00 +0000

This is the report of the first librevent developer meeting; a hybrid solution between physical and digital meeting to discuss what to do between Jan. 15 and Feb. 15 at 6 p.m., the date of the second and next meeting.

Main outcomes

The priority is to spread the concept (of both data liberation and librevents). To spread the extension that we have. Talk about it in the fediverse. Improve the explanation of what is going on, and measure how much, how, and whether this tool is appreciated and used.

Therefore, if you more than once per year access facebook.com to look at an Event invitation, even if unlogged because you don’t have an account, please consider installing:

Chrome Firefox

In the next month will be secondary to work on new features and new platforms (source and target), so gath.io and eventbrite are postponed.

Several “gitlab issues” have been opened to keep track of the most anticipated improvements and the most trivial bugs. These are features that could help in the months ahead to make the data liberation process more pleasant, complete, and robust. But more prioritarian than improvements we need to prioritize outreach and team consolidation.

There should be no programmer-single-point-of-failure in development, and this condition should be addressed before making the tool more sophisticated and complex. Considering the code is has 6 years of baggage (being a heavily modified fork of facebook.tracking.exposed) and considering the looming migration to manifest V3, rewriting the extension and reorganizing the backend is a refactor to consider.

Goals for the next meeting

Raise awareness about the project. We should find early adopters, developers, and moral support. In support of this we should spread the current produced material and produce new media.

The talk gave at HIP'22 is now on peertube

Slides: 7 slides (first part), 17 slides (second part).

How to measure progress

Since one way to measure success is to know how many people are using it, an important thing is to know how many users there are, so statistics must be prioritized. So an API (https://libr.events/api/stats) has been implemented to export aggregate counters.

👉 data is consumed and visualized in this observable.

Minor bugfix

The incorrect text strings at the end of text has been fixed.

Recurring developers meetup: how and what

daline@proton.me (Data Liberation Network) — Sun, 15 Jan 2023 00:00:00 +0000

From 2023 this project will have a recurring meeting: on the 15th of each month, regardless of the day of the week or time zone, at 18:00 CET.

This is the first developer meeting event. To participate remotely, join our matrix channel, a jitsi link will be shared before the event.

A report should be posted on the website the days after the meeting.

The meeting is online, as it should be attended by people in different time zones. If some of the participants are in the same city, they can connect from the same space.

Agenda points

Fix the upload of images. It is necessary to download the main image of an event and publish it.
Fix the HTML sent to mobilizon, because it is dirty from the metadata at the end of the post.
Tell about the event at HIP on the fediverse, and try to embed material from the fediverse into this site (made on hugo).
Design a unit test for a robust date extraction.

Software operation flow for version 0.3.x

daline@proton.me (Data Liberation Network) — Sun, 25 Dec 2022 00:00:00 +0000

This post has been written In preparation for the HIP talk (7 slides first part, 17 slides second part, video on peertube), and contains a quite deep explanation on how the Librevents workflow connects: Facebook → Extension → Backend → Mobilizon.

In this post you’ll also read about some current design limits in version 0.3.x, and the digital spaces where we can coordinate our efforts toward a stable 1.x release.

Software operation flow

The first figure takeaways:

You should use the extension without being logged in
The Rocket icon only should work in Event page. Meaning the URL has to match this format https://facebook.com/events/NUMERIC_EVENT_ID, optionally with some ?parameter after the NUMERIC_EVENT_ID, and those parameters are ignored. (for example: https://www.facebook.com/events/456737733157167).
The event need to be expanded. Facebook often cut the text and you’ve to press See More to see the complete event. This need to be clicked before liberating the event.

Once the event have been acquired, a few colored border would appear in the facebook page. It talks about the section captured and sent.
The backend is an intermediary step between the extension and the mobilizon instance. This is a design issue/feature that would likely to change in next version, but in this phase of development is necessary to pass through this approach (even if not privacy nor security wise)

This graph display the connections made between the mobilizon-poster library and the Mobilizon instance; Follow the git repository to know more.

Design limits

At this time, the extension must be aware of a backend server and have the necessary login and password to send the event to mobilizon. It means that a password is sent to a third party to do the authentication. This is not a clean or at the very least appropriate method, but there are reasons why I have come to this point:

mobilizon supports the OAuth protocol but need to be configured at instance level, so not all the instances can be supported. It would be better to use that, but we haven’t followed that implementation yet. It will happen in future versions, and is still unclear if the token exchange should be done either with libr.events website or from the extension.
having a server that can keep track of freed IDs is useful to avoid duplication in the fediverse, although this should be solved with a DHT or other non-centralized technology.
at the moment the mobilizon interaction library is very sparse, it can barely do event updates, thinking about how to improve is part of the role separation between mobilizon support (which should be coupled with support for gath.io and hook) and the necessary registration one must have to interact with instances.

Technical details on how the reposting works in the backend

This is a commented description on what happen when you run npm run watch. Assume you just clicked on the event with URL https://www.facebook.com/events/4760265773991156:

This happens in the backend, comments below the piece of logs:

lib:api  * Supporter papaya-kumquat-succotash [active on 22:59] last activity 22:58 26/12 (a minute ago) +1m
lib:api Saved for debug /tmp/1203.json +1ms

The backend identify a supporter (someone with the extension installed) by their publicKey. The publicKey is normally not display in bytes, but with a pseudonym made of three random food words.

lib:api Time travel problem! +2ms
lib:api dissectDates from [SATURDAY, DECEMBER 31, 2022 AT 7:00 PM – 2:00 ] start: 2022-12-31 23:59 end 2022-12-31 23:59 +0ms

The dates might take different shape and formats. At the moment we didn’t yet test all the possible conditions, but in cases like this (7pm on the last day of the year, till 2am of New Year Eve) and other complex combination, the software might fail in understand the meaning.

Every time the end date seems to be earlier than the start date, it is printed Time travel problem! and it is a bug.

mobi:login perform: Connecting to graphQL endpoint [https://mobilizon.libr.events/api] +1m
mobi:login getIdentityInfo: Connecting to graphQL endpoint [https://mobilizon.libr.events/api] +2s

mobilizon-poster connects to the mobilizon server, to get a valid authentication token. The token is necessary to post a new event.

lib:core Trying to resolve location in [ 'Market Grounds', 'Pub', ' ', ' · Restaurant' ] +3s
mobi:shared Connecting to https://mobilizon.libr.events/api +1m
mobi:shared:IO Status: 200 +1m
mobi:location Available 1 locations, selected [Market Grounds](115.855594;-31.949454) +1s
mobi:shared Connecting to https://mobilizon.libr.events/api +1s
mobi:shared:IO Status: 200 +814ms
mobi:location Available 10 locations, selected [PUB](18.061288438256224;59.33456245) +815ms
mobi:shared Connecting to https://mobilizon.libr.events/api +815ms
mobi:shared:IO Status: 200 +726ms
lib:core Not found a location for [ ] +3s
mobi:shared Connecting to https://mobilizon.libr.events/api +726ms
mobi:shared:IO Status: 200 +924ms
mobi:location Available 10 locations, selected [Neubau (Metzgerei, Hotel, Restaurant, Tagungsräume) Hotel Wittmann](11.458678257358994;49.274438599999996) +923ms
lib:core Unusual condition where both of the label worked [ 'Market Grounds', 'Pub', ' ', ' · Restaurant' ] +923ms

This piece of log need a few clarification:

Mobilizon expect a location resolved with a determined format, the same format of Open Street Map; every mobilizon instance wrap a query to OSM to so interactively offer the right spot
During the scraping you might acquire one or more possible location, this is a scraping problem, but this is the reason why three labels have been searched in this case.

mobi:poster Successful event created: {
mobi:poster   __typename: 'Event',
mobi:poster   attributedTo: null,
mobi:poster   beginsOn: '2022-12-31T23:59:59Z',
mobi:poster   category: 'MEETING',
mobi:poster   description: '<p>Motif and Market Grounds presents...[...]
mobi:poster   draft: false,
mobi:poster   endsOn: '2022-12-31T23:59:59Z',
mobi:poster   id: '165',
mobi:poster   joinOptions: 'FREE',
mobi:poster   local: true,
mobi:poster   onlineAddress: 'https://www.facebook.com/events/4760265773991156/',
[...]
mobi:poster   title: 'The Night Market NYE 2023',
mobi:poster   url: 'https://mobilizon.libr.events/events/df6791ef-f3e7-4234-92a9-c26ab8a8c6f1',
mobi:poster } +2m
lib:core createEvent [The Night Market NYE 2023] successful! +1s
lib:api Written event post [SUCESSFUL] (https://mobilizon.libr.events/events/df6791ef-f3e7-4234-92a9-c26ab8a8c6f1) supporter papaya-kumquat-succotash, took a few seconds +10ms
librevent:server API success, returning JSON <31 bytes> +0ms

The event has been created and inserted in the DB.

From the popup you can click “Access the Personal Page” where you can see the link of the event liberated, the mobilizon link just created, and few more information.

⚠️⚠️⚠️ In version 0.3.x the scraping/reposting bug can only be solved by editing the event in Mobilizon.

And below you can see how the event liberated looks like on mobilizon. The user experiment is the one provided by default in the browser extension.

Coordination spaces

We use the Matrix Protocol, to chat. And please don’t propose a slack/mattermost solution, thank you.

Use matrix and then connect to our chatroom: #mobilize.berlin:systemli.org.

On the 15th day of every month, at 18:00 CET we meet to chat and discuss last month progress and next month goal. This habit would be true for the 0.x release and perhaps if the project proceed would be updated.

Repository summary (extension and backend)

The command below have been tested on Ubuntu 20+ Linux. Also on MacOSX and Windows, potentially, this codebase can work. You might need to open the buildall.sh script and execute command line by line. It requires mongodb, nodejs, npm.

git clone https://0xacab.org/daline/librevents.git
cd librevents
./buildall.sh

If everything has worked, you can then:

Load the extension locally. The extension for DEVELOPMENT mode should be in librevents/extension/build. If you’re new with this, it might help Chrome install extension locally or Firefox install extension locally. When the extension is loaded, please access a Facebook event at your choice and check if the rocket icon is appearing on the left.
from librevents/backend run npm run watch this would load the backend locally. the extension you just built would send data there, and because of the verbosity, you’ll notice the activities.

Compile the site, try to edit it, and potentially make a pull request

It requires Hugo, as it is a static website generator, which is normally installed via snap, flatpak, and other package manager. The repository also has a submodule, which is the hugo-theme-texify.

git clone https://0xacab.org/daline/librevents.git
cd librevents/site
git submodule sync
hugo -D server

Then you can change the librevents/site/content/** files, and by accessing http://localhost:1313, see the website preview.

The 0.3.4 release; bugfix and new iconography

daline@proton.me (Data Liberation Network) — Tue, 20 Dec 2022 00:00:00 +0000

This is the new extension icon: 🚀 What’s better than a rocket to shoot free events in the Fediverse?

0.3.4 version for Firefox and Chrome. Or manually install the extension from the git release.

Fixes

To get better insight, follow the list of the closed issues.
fix: updating settings via popup is correctly recorded. So to test the tool now you can use your login and password.
fix: new iconography. The rocket appears on the right of the Facebook Event pages, and should be clicked.
fix: typos and trimming in the popup, give a look on how it looks like:

What do the options mean?

Mobilizon: this is the mobilizon server, you should add /api at the end, because what is needed is the API endpoint.
Login and Password: login and password for the Mobilizon account.
Backend server: keep libr.events as it is the only backend I know of, but if you run the librevents backend you can configure your own.
💡 Access the Personal page: this link brings you to the list of events you liberated, so you can review if they are correct and access to the Mobilizon entry where they can be updated if any detail was wrong.

30 minutes talk at HIP 2022

daline@proton.me (Data Liberation Network) — Wed, 07 Dec 2022 00:00:00 +0000

The talk proposed for HIP 2022 (Hacking in Parallel, the covid-safe replacement of the Chaos Computer Congress) was accepted! and would happen on the 28th of December.

TODO slides link

Abstract

Librevents is a browser extension allowing any user to copy and republish (“scrape”) data about events posted on proprietary platforms onto free libre and open source decentralized networks.

For now, Librevents focuses on liberating events data (description, date and time, location) from Facebook onto Mobilizon, an event-management platform alternative part of the Fediverse*.

The intention behind Librevents is to feed alternative ethical platforms like Mobilizon with content, in order to help them counter the “network effect” (users staying on Facebook because the information is only available there). The concept of “data liberation” could later be applied to other types of contents and platforms.

The data we liberate is initially posted as a “public event” by the organizer. We make this data truly “public” and accessible on free platforms, without violating the organizer’s original intentions.

* Fediverse: a federated network of free libre and open source alternatives to Twitter, Facebook, Youtube. The networks within the Fediverse communicate together through the ActivityPub protocol.

In this talk we will present the browser extension Librevents, ready to be used, and allowing any user to scrape and republish themselves events onto a Mobilizon instance.

Our goal is to grow our community of developers, event organizers, and Mobilizon instance administrators in order to improve Librevents, to promote Mobilizon and the Fediverse, and make data liberation happen.

keywords: Fediverse, Mobilizon data liberation, adversarial interoperability.

Private activity log

daline@proton.me (Data Liberation Network) — Fri, 18 Nov 2022 00:00:00 +0000

🏠 Private activity log

Your profile

First access	Last access	Events liberated	Errors

Event you liberated

When	Title	Event Time	Links

Discuss improvements in the git repository.

The 0.3.3 release and Lisbon Privacy Cafè

daline@proton.me (Data Liberation Network) — Fri, 18 Nov 2022 00:00:00 +0000

The upcoming presentation of Librevents in Lisbon, at the Privacy Cafè, has pushed to deliver some of the progress foreseen at the last Hackathon. Slides.

The 0.3.x release branch is the beginning of many parallel improvements! Get the browser extension for (for Firefox, for Chrome.)

What’s new?

A new “conceptual space” hosts this software. The name is Data Liberation Network or dalina for those in the family; Librevents was born under the Tracking Exposed Manifesto two years ago. Then, between 2021-2022 Tracking Exposed grew significantly, and was able to focus in algorithm analysis. It became natural, to highlight the different focus of Librevent, to move it to a more precise space.
There is a new dedicated git repository hosted on riseup.
The new extension has a calendar as an icon, and some scribbles that appear to the left of Facebook pages need to be clicked on. The user experience is still problematic. We can aspire that by version 0.4 we will have fixed most of these shortcomings.
The mobilezon-poster has been updated to version 1.1.5.
This website still runs under hugo, like the previous one, but the LaTeX-inspired style gives it a whole other tone. Perhaps inappropriate with the revolutionary concept that data liberation might expect to be, which perhaps it is this dissonance to make this appealing.
As part of the stylistic experiment, the extension popup also has this font and has abandoned the psychedelic animated gif found at the bottom of the page.

Commitment

We’ll have to build an international community of people who want to liberate data. The monopolistic social media model would keep frustrating us.

Of course, would have been more exciting if, for example, Twitter had gone out of business because people go on the fediverse, and not because some crazy billionaire drives you to there.

We would have all wanted communities to leave Facebook because it is a toxic and exploitative environment, and not because there are only bots, hate-filled boomers, and poorly targeted advertising.

If this is a phase of transition, data liberation can be a bridge, to help breaking the critical mass constraint, and leave those blazing dumpster once and for all.

References

The Privacy Cafè invite:

To not forget the thousands, if not the millions, of you lusting over an image morphing made by deep neural networks, mashed up to build a 90s-style banner for a web site.

Librevents: “liberating” data from Big Tech

⏸️ The project is temporarily on hiatus.

🌱 Developer Meetup n.2; Report

What to do in the next month

Fundraising narratives

On Facebook HTML obfuscation

(at least) three kinds of Errors

The harvest 🍂 after January'23 planning

Software and site updates

Heard from the Fediverse

🌱 Developer Meetup n.1; Report

Main outcomes

Goals for the next meeting

The talk gave at HIP'22 is now on peertube

How to measure progress

👉 data is consumed and visualized in this observable.

Minor bugfix

Recurring developers meetup: how and what

Agenda points

Software operation flow for version 0.3.x

Software operation flow

Design limits

Technical details on how the reposting works in the backend

⚠️⚠️⚠️ In version 0.3.x the scraping/reposting bug can only be solved by editing the event in Mobilizon.

Coordination spaces

Repository summary (extension and backend)

Compile the site, try to edit it, and potentially make a pull request

The 0.3.4 release; bugfix and new iconography

Fixes

What do the options mean?

30 minutes talk at HIP 2022

Abstract

keywords: Fediverse, Mobilizon data liberation, adversarial interoperability.

Private activity log

🏠 Private activity log

Your profile

Event you liberated

The 0.3.3 release and Lisbon Privacy Cafè

What’s new?

Commitment

References

(at least) three kinds of `Errors`