Replying to
First version of the new #Fedifinder. I redesigned the frontend to make it easier to use for people who are joining the #fediverse.
You can try it here: https://fedifinder-backup2.glitch.me/
Let me know, what you think.
Remote
Luca Hammer EN
@Luca@vis.social
mastodon 4.7.3Scientific programmer @sfb1472@wisskomm.social at University of Siegen. Neurodivergent.
Helped thousands of people migrate their social graph to the Fediverse with #fedifinder.
This account focuses on my work and projects. For everything else, follow @luca@social.luca.run.
2985 Followers
1594 Following
50 Posts
Joined January 23, 2018
Pronouns:
he/him
Web & Blog:
FediFinder:
Send me money:
Replying to
And #Fedifinder is back. The API wasn't turned off for good, but they removed my app from my project and as a result I lost access to v2.
Find your friends from Twitter in the fediverse and download a copy of their profiles to find them after deleting your Twitter account: https://fedifinder.glitch.me/
Replying to
Full speed! Thanks to the latest update, #Fedifinder scans my 2k followings within 10 seconds. From login to exported file.
The solution to a slow databases wasn't a fast database, but no database at all. Finally, I can tell you to share fedifinder with all of your contacts without fearing that the
service will go down because of it.
Go and find your people! https://fedifinder.glitch.me/
Replying to
The important thing: #FediFinder is back!
While https://fedifinder.glitch.me seems to work fine again, I added two backups that you can use as well:
https://fedifinder-backup.glitch.me and https://fedifinder-backup2.glitch.me
New feature: You can now click on the usernames for the fediverse (may not always work because of different formats of software) and Twitter.
Replying to
Every few days I compare the #fedifinder and #debirdify results to find issues as well as to improve the handle detection. This time I realized, that fedifinder had a bug with pinned Tweets. It's fixed now.
For my own account:
566 handles with fedifinder
534 handles with debirdify
Let me know in the replies which handles fedifinder missed for your contacts so I can fix it.
https://fedifinder-backup.glitch.me/
https://debirdify.pruvisto.org/
I will not collaborate with @Tibor@mastodon.social and won't recommend his tools (#movetodon, whotofollow) because of his unwillingness to reflect his behavior.
CW: Genocide.
@melinaborcak@dju.social informs about the #SrebrenicaGenocide (follow her). Tibor replied with disrespectful "jokes" to her Tweets, after blocking her. When people called him out on it, he blocked them as well.
https://nitter.net/MelinaBorcak/status/1559578560083509248#m
Replying to
Updates to #fedifinder
- Accounts are now sorted by fediverse instance
- Strings that look like mail addresses are included again eg. name@domain.tld
- domains are checked for /.well-known/host-meta (if it exists, the domain is more likely to be part of the fediverse; someday I may look at the content as well; currently just HEAD requests) Thanks @yetzt for the idea.
- domain checks are cached to make them faster and don't put unnecessary load on servers
- offline domains excluded
Replying to
It seems like the #TwitterAPI has been turned off for real this time. #Fedifinder doesn't work anymore.
I don't know yet, if I will keep working on it. I don't even have the energy to think about it at the moment.
Replying to
Support for lists arrived in #fedifinder.
It shows you your own lists as well as the lists you are subscribed to. You can scan each of them and the found handles are added to the visible list as well as the download. (False positives of the previous scan are removed from the visible list with each additional scan.)
Replying to
You can now check if the handles in your own Twitter profile are found. They are not added to the export. If there are issues with your profile, let me know.
Thanks for the idea @nelson. #twittermigration #neuhier #nieuwhier
Replying to
Today wasn't fun. While I was outside, I got many messages that #FediFinder didn't work. I checked it and saw that it was overloaded and crashed every few minutes. I then tried to fix it from my mobile, but with each line of code it seemed like I added multiple new bugs. Not good. So I added a message that its broken and to check back later.
I spent the whole afternoon and evening trying to fix the mistakes of past me and clean up the code to make it at least slightly easier in the future.
Replying to
Tiny changes on the frontend, massive ones in the backend. #fedifinder
-
Option to scan the profiles of your followers
-
Twiter handle where the fediverse account was found
-
Handles of failed servers can be displayed
-
Switched to API v2
-
Pinned Tweets are included
-
You can now scan up to 15k followings and 15k followers
-
Better handling of rate limits
-
Caching of reason of unreachability of servers (and limited amount of retries for servers that timed out)
-
better regex
Replying to
There are people who had problems using #fedifinder. Not everyone knows what a CSV is or where to find the import. But I have a solution: Follow buttons!
If one or more #fediverse handles are identified on the profile of the person using fedifinder, they can now toggle on follow buttons to get directly to the follow request on their own instances. This should make it much easier.
(At the moment they only work for mastodon instances. I am open to pull requests for other software.)
Replying to
I updated the ReadMe to help people run their own #Fedifinder (no coding skills needed) and how to contribute.
https://github.com/lucahammer/fedifinder
And fedifinder got a new API endpoint. api/check. You can use it with a domain and/or handle as query parameters and get back the cached information fedifinder has for that: https://fedifinder.glitch.me/api/check?handle=@luca@lucahammer.com
Replying to
More improvements for #fedfinder.
The results now display basic stats for each instance and if the registration is open. Thanks for the hint @pfefferle.
Better check if something is part of the #fediverse.
Data handling is still a mess, but less messy now, which results in fewer requests to the server and less jumping text.
Replying to
For half an hour full of stress I thought, I exposed the sessions database. I noticed a data folder in the git repository that should not have been there (wrong path). First I deleted the folder (to make it less visible), then I revoked the Twitter app tokens, filtered the data from the git history (to actually remove it) and put a notice on the #fedifinder homepage. Finally, I felt like I did, what I could. I relaxed and realized that user data was never at risk. The was the staging database.
Replying to
I was able to reduce the wait time by around 40% by moving the domain checks to after the page got rendered. I fear some rate limiting issues came from people reloading before receiving a response.
False positives are kept in the visible list, but striked through. They are still removed from the exported CSV to reduce server load on imports.
Next up: Lists. They are the most requested feature. #fedifinder
Replying to
#fedifinder did not get suspended, but authorization with Twitter API V2 doesn't work at the moment. Please use https://fedifinder-v1.glitch.me/ (uses API V1) until Twitter resolves the issue or falls apart.
Replying to
Finally felt confident enough to switch the main server to the new #fedifinder version: https://fedifinder.glitch.me/
Because I know how much people hate change, you can still access the old version at https://fedifinder-v1.glitch.me/
Replying to
The sleepless #fediFinder update.
Authorize your Mastodon account to see which Fediverse accounts of your Twitter followings, you already follow.
Because the feature isn't finished yet and still confusing (you have to authorize, then click hide followings, then scan followings; no export yet) it's currently only available on the backup server: https://fedifinder-backup.glitch.me/
I am looking forward to your feedback (cc @illionas@mstdn.io @ColinSch @botolo86) And to my bed. Good night.
Replying to
Thanks for all the feedback. I believe, I fixed most of the issues with the new #fedifinderhttps://fedifinder-backup.glitch.me
Please let me know, if something still doesn't work. Else I will move it to the main link soon and put the current one somewhere else. For people who don't like change.
Replying to
Making it easier to run your own #fedifinder.
Until now, fedifinder needed a Twitter developer app with elevated access because the authorization flow relied on Twitter API v1. I finally switched it to oAuth2. Now, you only need essential API access to run it and it gets access to fewer things than before thanks to scopes.
If you have essential access, please test it by clicking Remix https://glitch.com/edit/#!/fedifinder configuring the .env and adding the callback URL to your Twitter app.
Replying to
Only a small update for #FediFinder today: Splitting up functionality to multiple servers.
The main server is only responsible for getting data from Twitter from now on. A second server does the handle and domain lookups. Because the lookups locked the server up from time to time, this should make the service more stable and faster. If no lookup server is added to the .env, fedifinder is fully functional as a single server.
Replying to
But Luca, not everyone is in the fediverse. The export is useless for the other accounts. And in my Twitter archive are only the IDs of the people I followed. Not their names or websites. What about them?
If you scroll to the bottom of #Fedifinder, you can now view a table of the accounts that you have scanned. Even if they have no handle in their profile. Or one that wasn't correctly detected.
Replying to
I added an API (kinda) to #FediFinder. The only endpoint is /api/known_instances.json. It returns all instances it knows at the moment.
Currently, It is used to share data with @debirdify.
I don't know if it will stay public/available and/or if the structure will change in the future.
Replying to
#Fedifinder still works to find your remaining followings from #Twitter in the #fediverse: https://fedifinder.glitch.me/
It didn't at first when I tested it today, but after seeing that the old version (https://fedifinder-v1.glitch.me/) works, I gave it another try and it went through. Down to 538 from 617 accounts. Will have to look into what happened with the rest (deleted?).
#Debirdify sadly got suspended by Twitter last week.
#twitodon is still functional as well: https://twitodon.com/
Replying to
This is one of the steps to make #fedifinder more resilient against being shut down. Getting essential access is easier than getting elevated access. Everyone can spin up their own fedifinder instance on #Glitch within minutes. If my account gets suspended, everyone can take over.
At the moment, there is no indication that fedifinder will get shut down. I am careful to follow the Twitter developer terms.
Replying to
I am currently looking through the #fediverse software options to find more URL patterns of profiles. Here is what I have at the moment. Most of them are easy to detect. But for pleroma I saw some servers with and some without the @ in the username. Makes it harder.
domain.tld/@name
mastodon, misskey, pleroma, funkwhale
domain.tld/profile/name
friendica, hubzilla
domain.tld/u/name
diaspora
domain.tld/c/name/videos
peertube
domain.tld/name
pleroma, gnusocial
Replying to
Instead of building a better export, I made the sessions persistent, so that not everyone re-logins when I change something and the server restarts.
You can now watch as #FediFinder checks the domains. And even retry if it gets stuck (should add a 10 second timeout there as well).
And I added links to the backup servers and info about donations to the homepage.
Replying to
Start of the week test. https://fedifinder.glitch.me/ still works. #fedifinder
Down to 524 found handles, from 538 two weeks ago.
Replying to
Starting today, #fedifinder collects minimal usage stats. Not linked to accounts, only a timestamp and the number of Twitter accounts that got checked.
This will allow me to decide if crowdfunding the upcoming $100 tier (https://social.luca.run/@luca/110110514116861680) would make sense and if there is enough usage to motivate me to write the necessary code changes.
Replying to
I made the scan of the followings optional in #fedifinder to improve speed, reduce the amount of rate limiting errors (eg. users reloading the page) and make it easier for people who only want to scan lists or maybe something else that gets added in the future.
Rate limiting errors aren't displayed as json anymore, but tell you how to fix them (wait, wait and wait).
Replying to
Several people reported issues with revoking API access after using #fedifinder with oauth2 authentication. When they go to Twitter settings and revoke access, they get an error or nothing happens.
Sadly, I can't do anything in those cases, because it all happens on Twitters side and you have to contact their support (good luck?).
But there are reports, that it still worked and access is gone after a refresh. Feel free to share your experience with others in the replies.
Replying to
#FediFinder Day 9(?). I moved the handle identification from the backend to the frontend to improve performance. The server is only responsible for getting the data from Twitter and checking domains. No changes for the users.
With the functions, I moved some tests to the frontend. Only visible on the staging server or the code.
If handles are detected on the user profile, I show links to their import. Thanks @nart for the idea.
JSON export (including accounts without handles).
Fixed bugs.
Replying to
The #fedifinder blank screen bug should be fixed now. Let me know, if you still have issues. https://fedifinder.glitch.me/
I started to add improvements for less common setups (eg. webfinger and instance on different domains), so they should show up as valid fediverse handles. But it isn't finished and needs testing. No energy at the moment.
I am close to another refactoring, but know that I don't have the energy to finish it.
617 handles identified for my own followings.
Replying to
I fixed a tiny bug in #Fedifinder that resulted in a fatal error. Everything works again. But there are only 507 handles in the profiles of the accounts I follow. Down from 524 a month ago.
(Don't ask me why I still have API access.)
Replying to
I already tried to subscribe to basic access, but there is no subscribe button. Probably because I am already subscribed to business and elevated. Let's see what happens at the end of April. #TwitterAPI
Replying to
Currently, #fedifinder checks around 100k Twitter accounts per day for linked #fediverse accounts.
/users/:id/following returns 1k accounts per request. 500 requests per day. That would be enough without any other changes than paying for basic access tier.
/2/users (aka lookup) only returns 100 accounts per request, but could be scaled by using cached results and allowing users to check their followings from the Twitter data export. 500 requests means 50k unique accounts per day. 1.5M/month.
@Mareike2405@fedihum.org @spinfocl@fedihum.org Ja, funktioniert noch. Ich weiß aber selbst nicht warum. Es wurde nur ein Teil meiner Apps abgeschalten. Einmal wurden Einstellungen zurückgesetzt, sodass es zwischendurch nicht funktioniert hat. Ich warte jeden Tag darauf, dass es endgültig vorbei ist.
@meikerenschbergner@norden.social @hdrmeurer@photog.social Wenn du den Account löschst, sind die Tweets auch weg. Ich halte es nicht für notwendig erst die Tweets und dann den Account zu löschen.
Tweets löschen ist notwendig, wenn man den Account noch behalten möchte, aber eben nicht mehr die Inhalte dort haben will.
@forteller@tutoteket.no Should be fixed now.
@max@peering.social Die alte Version, https://fedifinder-v1.glitch.me , lädt Followings nicht automatisch, somit ist das dort möglich.
@elan@publicsquare.global Currently on vacation. Should be in the glitch branch.
Mastodon seems to have changed that URL with some version because I recently got feedback that the 's' is needed. https://elk.zone/mastodon.social/@sikachu/110600781654777242
Replying to
@Variety@tooting.ch @luca@lucahammer.com Ich bin mir gerade nicht sicher, was du meinst.
@sstoneb@retro.pizza I worked on that but never finished it because of the close shutdown of the API and my limited energy.
Replying to
@kristinHenry@vis.social Thanks. (Makes me feel slightly less bad for not posting any visualizations or related content in months.)
@Variety@tooting.ch @luca@lucahammer.com @feditips@mstdn.social Da ist dann etwas beim Upload schief gegangen.