Victoria

Channel Apps

Kanal Über Fotos Dateien Kalender Galerie

System Apps

Fehler melden Hilfe QRator Sprache Suche Verzeichnis Zufälliger Kanal

Zotum outage

Fri, 24 Mar 2023 08:20:57 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Zotum @Hubzilla Support Forum @Max Kostikov @Maria Karlsen @Kristian @Jupiter Rowland @Mario Vavti

All - please can whoever receives this try to get a message to the administrator of Zotum? Connectivity with both ActivityPub and other Hubzilla instances appears to have been offline for around 36 hours.

show all 68 comments

Fri, 24 Mar 2023 09:29:21 +0100

Mario Vavti

mario@hub.somaton.com

Seems OK to me now though...

Fri, 24 Mar 2023 09:32:04 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Nothing I posted since Wednesday has arrived on the clone of the channel or on ActivityPub yet. I don't think it has recovered yet, although Max does seem to be aware now.

Fri, 24 Mar 2023 09:43:21 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 09:43:41 +0100

Mario Vavti

mario@hub.somaton.com

hub.netzgemeide.eu is delivering slowly currently. Might have something todo with that...

Fri, 24 Mar 2023 10:01:58 +0100

Kristian

z428@hub.netzgemeinde.eu

It _seems_ to slowly get better at the moment. Good. ?

Fri, 24 Mar 2023 10:16:01 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mario Vavti Right now, new messages are going through to ActivityPub immediately. But the posts I submitted during the outage haven't arrived, and their delivery reports are completely blank. Does that mean they're in a queue?

Fri, 24 Mar 2023 10:28:20 +0100

Mario Vavti

mario@hub.somaton.com

@Victoria which hub?

Fri, 24 Mar 2023 10:35:30 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

My guess is that it had to do with the 8.2 upgrade, that it created a gigantic queue of messages to send. hub.netzgemeinde.eu slowed down dramatically outbound while still being responsive in-bound two days ago. Messages took hours to even create delivery reports, and the most recent to be delivered showed delays between posting and nomadic federation of six hours and between posting and delivery through ActivityPub of over 19 or 20 hours.

I had to resort to contact the admin through my clone which, albeit upgraded to 8.2, wasn't affected, to be on the safe side. He found over 70,000 out-bound messages queued up, adjusted the queue worker, and things cleared up within an hour yesterday.

Zotum might actually be even bigger, causing even longer lag.

Fri, 24 Mar 2023 10:36:40 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 10:37:41 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mario Vavti I'd rather use this channel for discussion about Hubzilla issues but the posts I'm referring to were sent on @Victoria channel on Zotum - everything since the one hashtagged "SaphalaEkadashi". The channel is cloned to Netzgemeinde but I sent them from Zotum, not Netzgemeinde. They have not appeared in Mastodon. Netzgemeinde hasn't synchronized them yet. The delivery reports remain blank.

(You can see the Sephala Ekadashi post is the most recent one being shown on the Nezgemeinde clone here: https://hub.netzgemeinde.eu/channel/vsrj)

Fri, 24 Mar 2023 11:03:23 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Jupiter Rowland Not sure I should ask why 8.2 caused such a big queue, but if it did, it did. It could explain some of the problems we've seen this week.

Max said there were around 22,000 messages queued on Zotum. I've no idea how long clearing that would typically be expected to take. I've been thinking it might be useful if the size of the queue was visible somewhere, for users and administrators alike. Problems might be easier to spot and people would know if their posts have really never been delivered or if there's just still a queue.

It would be helpful to know if the posts I sent from Wednesday are ever expected to arrive - whether I should try to redeliver them or if that will only make things worse, etc. I'm happy to wait if I know they will eventually appear. I'd just rather not spend 2-3 days waiting only to find they never did and that those days could've been spent figuring out why or reposting them.

Fri, 24 Mar 2023 11:52:56 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 11:54:21 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Also - @Mario Vavti If the issue on Zotum is queueing, and that's why the older posts haven't appeared in ActivityPub or on the Public Stream, how come this test message I just sent from this channel here on Zotum appeared immediately in ActivityPub and the Public Stream? Shouldn't it be at the back of the queue?

Fri, 24 Mar 2023 11:58:07 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Victoria @Mario Vavti Similarly, why are there complaints of posts from Netzgemeinde still queueing up while my personal queue disappeared fully less than two hours after @Mark Nowiasz has tuned the queue worker?

Do they give you VIP lanes if you complain loudly enough?

I mean, I used to run my own private Hubzilla hub back in the day, and I don't recall there being a VIP lane.

Fri, 24 Mar 2023 12:03:25 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 12:06:16 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mario Vavti @Jupiter Rowland Are the complaints about new posts or undelivered old posts?

(Edit: Sorry, hi @Mark Nowiasz , didn't intend to drop you off the list ... apologies if the tags are unnecessary, still getting to grips with how this all works)

Fri, 24 Mar 2023 12:12:40 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Victoria @Mario Vavti They have been about old posts that don't even have a delivery report yet.

To drag someone else into the discussion, @Tokyo Outsider (337ppm) has been waiting for posts from Zotum to be delivered for days already, but at the same time, we can basically use Hubzilla as a chat.

I'd say that either new posts bypass the old queue, or the old queue has been stopped entirely. But that wouldn't explain how my Netzgemeinde backlog went through within an hour or two yesterday. And I know it did because it was followed by a flood of reactions.

Fri, 24 Mar 2023 12:14:49 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 12:15:26 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

I'm thinking a couple of things could be going on here. For some reason, Hubzilla could put posts into a queue only if it can't deliver them straight away (like some kind of retry queue). That would explain why the new posts now are arriving so quickly while old ones are queued.

Alternatively, the old ones for which I'm awaiting delivery aren't in the queue. The question becomes, "why not?". With their blank delivery reports, they do look exactly like the ones that arrived an hour or so later after the outage Sunday/Monday, though, so I am still hoping they're just somewhere in the queue. I took care to only post from the primary instance during the outage this time.

The other possibility is that somehow a notice is being sent to ActivityPub during the outage, but because of the outage ActivityPub is unable to collect the post during its usual time frame. One of the Mastodon admins told me this week that Mastodon usually only retries collection once, and that's within 24 hours.

If that happens, it won't matter whether the post is queued or not, or even perhaps if it's redelivered (given that all the IDs will be the same). It will just never be recollected, because it has already exceeded the timeout for ActivityPub collection.

We'd really need more information from @Mario Vavti @Max Kostikov or @Mark Nowiasz, or anyone else who knows the internals of the system, to know which of these things might be happening.

Fri, 24 Mar 2023 12:24:16 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Victoria Additional theory, based on yours: Even when retrying after 24 hours, the queue might be so huge that the worker can't go through it all before the next time-out. If it's big enough, some posts might get stuck for days.

Mine were simply lucky to go through at the first retry and even that maybe only after Mark's tuning.

But I suspect there also being an hourly retry. This explains why that one post I've examined took 19 hours and a bit to reach some ActivityPub destinations and 20 hours and a bit to reach the rest.

Fri, 24 Mar 2023 12:32:09 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

That's also possible. What I was told was that there was only one retry, although that didn't match what I'd heard people discussing before. Nonetheless, your idea of the queue being too huge to process within 24 hours could still explain the failure on the first retry, either way, and then if that was the only retry the posts would never be picked up.

A lot of this depends on when the ActivityPub notification is sent. If it's sent by some independent process while other parts of the system are queued up, that could explain why posts are then never collected (because they never get through the queue into the ActivityPub outbox on Zotum before various ActivityPub apps receive the notification and try or retry to collect the post).

Or, if the ActivityPub notification is sent after the queue processing, that would explain why some posts would take 19 hours while others might take 20 hours, because when the notification is sent will depend on whether they emerge from the queue. In that case, the ones that are never delivered might never be going into the queue or are being cleared somehow from the queue without processing - because otherwise at some point they would emerge.

Fri, 24 Mar 2023 12:33:32 +0100

Shimriez

shimriez@zotum.net

I had four messages with blank delivery reports. Just now I tried "redeliver" and everything went back to normal it seems:-)

Fri, 24 Mar 2023 12:34:48 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Maria Karlsen Thank you - I will try a redeliver on the first one. I don't want to clear down all of them just yet in case @Mario Vavti or @Max Kostikov are using them to figure out what has gone wrong.

Fri, 24 Mar 2023 12:40:44 +0100 zuletzt bearbeitet: Fri, 24 Mar 2023 12:41:08 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

I hit redeliver (just on the first one). The delivery report lit up like a Christmas tree... it now says the message was posted at zotum.net and also that it was "accepted for delivery" by a whole bunch of Mastodon servers. But it hasn't appeared on any of them.

Sat, 25 Mar 2023 03:01:06 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Just checked this morning and none of these messages have been delivered either to the clone on Netzgemeinde or to anyone on ActivityPub.

The one I used "redeliver" with shows that it was "accepted for delivery" at many ActivityPub instances (see attached screenshot), but the post is not visible on them.

When Zotum recovered from the outage Sunday/Monday, posts I had sent from Zotum and that had, during the outage, shown blank delivery reports, appeared on ActivityPub within a few hours.

@Max Kostikov @Mark Nowiasz How many items are still queued today?

Sat, 25 Mar 2023 04:08:21 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Just remembered that I have a second clone of the channel, on hubzilla.eskimo.com. That is showing all the posts that were sent from Zotum during the outage. So their non-appearance on Netzgemeinde is a Netzgemeinde-specific issue (cc @Mark Nowiasz).

From a Zotum perspective, the only issue is their ongoing non-delivery to ActivityPub.

Sat, 25 Mar 2023 07:52:03 +0100

Mario Vavti

mario@hub.somaton.com

@Victoria the non delivery to AP in this case has possibly something to do with this issue:
https://framagit.org/hubzilla/core/-/issues/1722

Mastodon and the likes do not support nomadic identities. It is advised to connect to those contacts from the clones for delivery to be reliable.

Sat, 25 Mar 2023 08:16:47 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mario Vavti The posts weren't sent from the clone, they were sent from Zotum. I don't understand how the issue you linked would be relevant in this case.

Sat, 25 Mar 2023 08:19:51 +0100

Mario Vavti

mario@hub.somaton.com

@Victoria zotum is your primary?

Sat, 25 Mar 2023 08:23:02 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mario Vavti Yes.

Sat, 25 Mar 2023 08:26:52 +0100

Mario Vavti

mario@hub.somaton.com

Ok, but now if you post from zotum the messages get delivered right?

Sat, 25 Mar 2023 08:27:31 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Now, yes. The issue is that the messages I sent during the outage were never delivered.

Sat, 25 Mar 2023 10:41:09 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Victoria Well, if you have clones, messages to outside the Zot/Nomad network, e.g. to ActivityPub participants, are only ever sent from the main, never from the clones.

Otherwise each ActivityPub user would get as many copies of your messages as there are instances you've got your channel on.

Sat, 25 Mar 2023 10:58:12 +0100 zuletzt bearbeitet: Sat, 25 Mar 2023 11:06:10 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Well, if you have clones, messages to outside the Zot/Nomad network, e.g. to ActivityPub participants, are only ever sent from the main, never from the clones.
Otherwise each ActivityPub user would get as many copies of your messages as there are instances you've got your channel on.

That is how I understood it to work initially.

I think perhaps the instance the post is sent from, rather than the primary instance, may currently be the one that sends the notification to ActivityPub. Earlier in the week (after the first outage) I had a message appear from a clone identity that never should have, because it was sent "from" the primary identity on Zotum. I've made a note of that to look into later.

The only way I can think the issue @Mario Vavti linked might prevent message delivery is if the primaries and the clones all send the notification to ActivityPub, ActivityPub sees them as separate identities, but for some reason (maybe due to a post ID?) it believes there is only one message, and so whichever instance gets its message to ActivityPub first takes precedence and the others are never displayed after, because ActivityPub thinks they're duplicates.

That would depend on Hubzilla working in a very specific way with ActivityPub, though, and I think you'd see more issues all of the time with messages showing up from the wrong identity. That wouldn't be visible to ActivityPub users who only follow the primary identity, but any of them that are connected to multiple identities would see the messages coming from multiple places. (It would be quite a security hole in ActivityPub, too.)

My messages still haven't arrived in ActivityPub, although I did get some late notifications about an hour ago so perhaps something is still working through a backlog.

Sat, 25 Mar 2023 11:08:25 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

You know, it might just be better to ask @Mario Vavti how it works - Mario, which instance is supposed to be sending the "new post" notification to ActivityPub? Always the primary? Or only the instance that sends the message? Or the primary and all its clones?

Also, @Max Kostikov, is it possible to see how many items are in the queue/backlog now?

Sat, 25 Mar 2023 11:10:30 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Victoria

That is how I understood it to work initially.

I think perhaps the instance the post is sent from, rather than the primary instance, may currently be the one that sends the notification to ActivityPub. Earlier in the week (after the first outage) I had a message appear from a clone identity that never should have, because it was sent "from" the primary identity on Zotum. I've made a note of that to look into later.

That could be possible, too.

Of course, the downside of it would be that e.g. Mastodon users receive messages from you through an entirely different address. ActivityPub doesn't know nomadic identity, and 99.99% of all Mastodon users don't know nomadic identity either, what with many still believing Mastodon is alone in the Fediverse. I mean, even many Friendica users have never heard of Hubzilla.

They'll simply assume that you've got a new account, that you've moved instances, and they'll follow your clone along with your main.

The only way to mitigate this in the long run would be ActivityPub and its implementations at least understanding and correctly handling nomadic identity as implemented in Zot and Nomad.

Sat, 25 Mar 2023 11:25:02 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

Having messages arrive in ActivityPub from multiple Hubzilla instances is confusing. You've made some interesting points so I will split them off into a different thread, so we can keep this one focused on the missing posts.

@Max Kostikov , is it possible to see how many items are in the Zotum queue/backlog now?

Sat, 25 Mar 2023 11:41:53 +0100

Max Kostikov

kostikov@zotum.net

@Victoria

is it possible to see how many items are in the Zotum queue/backlog now?

21k

Sat, 25 Mar 2023 11:48:52 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

21k

Seriously? ?

Ok, thank you for the information. Let's come back to this and see if the problem is resolved in a few weeks' time....

Sat, 25 Mar 2023 11:53:19 +0100

Max Kostikov

kostikov@zotum.net

@Victoria I have initiated forced queue processing. I think that should help push up the older supplies.

Sat, 25 Mar 2023 12:03:10 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

I have initiated forced queue processing. I think that should help push up the older supplies.

That's great - thank you very much. It will be interesting to see if these posts finally emerge! ?

Sat, 25 Mar 2023 16:23:06 +0100

Mark Nowiasz

buckaroo@hub.netzgemeinde.eu

@Victoria 18950 right now

Sat, 25 Mar 2023 16:44:57 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mark Nowiasz Thank you for letting me know, Mark - much appreciated!

Sat, 25 Mar 2023 16:47:45 +0100

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Mark Nowiasz In fact, I just checked my clone on Netzgemeinde and it's fully up to date!! Sorry, I should have checked before I replied to you just now. Thank you again for letting me know about the queue. It looks like the issue on my clone is resolved now. Hope everything catches up for you soon.

Sat, 25 Mar 2023 16:48:12 +0100

Mark Nowiasz

buckaroo@hub.netzgemeinde.eu

@Max Kostikov How do you do this? I remember that there was an hidden option for the queue so you could empty queues/mark queues, but I can't remember what it was ?

Sat, 25 Mar 2023 16:59:32 +0100 zuletzt bearbeitet: Sat, 25 Mar 2023 16:59:47 +0100

Max Kostikov

kostikov@zotum.net

@Mark Nowiasz

How do you do this? I remember that there was an hidden option for the queue so you could empty queues/mark queues, but I can't remember what it was

I don't remember about the options, but did it this way.
1. Stopped PHP-FPM and cron.
2. Killed all PHP processes.
3. Deleted data from "workerq" table.
4. Made OPTIMIZE TABLE for "workerq" and "outq".
5. Performed "UPDATE outq SET outq_priority = 0, outq_scheduled = NOW();" query.
6. Started PHP-FPM and cron.

Sat, 25 Mar 2023 17:13:28 +0100

Mark Nowiasz

buckaroo@hub.netzgemeinde.eu

@Max Kostikov Thanks, this is currently a little bit too sophisticated to do it using my phone, being currently on Orlando ?

Sat, 25 Mar 2023 21:39:11 +0100

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Mark Nowiasz @Max Kostikov I mean, I've ssh'd from a mobile phone in the past. But the phone in question had a hardware keyboard.

Nowadays, I'd rather ssh into mobile phones.

Sat, 25 Mar 2023 21:40:35 +0100

Max Kostikov

kostikov@zotum.net

@Jupiter Rowland

I've ssh'd from a mobile phone in the past

Yeah, no problems except screen size and onscreen keyboard.

Sat, 25 Mar 2023 21:41:03 +0100

Mark Nowiasz

buckaroo@hub.netzgemeinde.eu

@Jupiter Rowland Well, I did perform a kernel upgrade from my phone while queing for my rollercoaster ride ?

Sat, 25 Mar 2023 21:45:02 +0100

Mark Nowiasz

buckaroo@hub.netzgemeinde.eu

@Max Kostikov it's perfectly fine for emergency work (and some routine stuff) but for regular work I prefer a real keyboard, preferably a ergonomic one.

Sun, 26 Mar 2023 09:04:36 +0200 zuletzt bearbeitet: Sun, 26 Mar 2023 09:05:50 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov Microsoft used to do a "wedge" mobile bluetooth keyboard that's really good - small enough to throw in almost any bag but big enough that you can touch type on it. Works with smartphones or tablets.

How's the queue today? I don't expect it to be clear yet, but it'd be interesting to know if the number is going down quicker. (My posts have not appeared yet in ActivityPub)

Sun, 26 Mar 2023 10:08:53 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

How's the queue today?

Feels good :-D

Sun, 26 Mar 2023 10:09:59 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

My posts have not appeared yet in ActivityPub

What server?

Sun, 26 Mar 2023 10:20:29 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov Zotum... the same one as yesterday ?

Sun, 26 Mar 2023 10:22:18 +0200

Max Kostikov

kostikov@zotum.net

@Victoria I meant ActivityPub server.

Sun, 26 Mar 2023 10:29:28 +0200 zuletzt bearbeitet: Sun, 26 Mar 2023 10:30:13 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov All of them. Mastodon.social specifically, but I just checked mas.to, mastodon.online, universeodon.com, masto.ai and mastodon.world as well and the posts haven't appeared on any of them.

There are five missing posts (the most recent five sent from @Victoria - all the posts sent since last Wednesday). I tried a redeliver on one of them, but it made no difference. The delivery reports showed nothing initially. Now they show the synchronisation with Hubzilla servers to which I've cloned the instance, but nothing about ActivityPub.

Sun, 26 Mar 2023 10:43:44 +0200

Max Kostikov

kostikov@zotum.net

@Victoria
No submissions to these servers are found in the queue, so I assume it has been delivered.

Sun, 26 Mar 2023 10:44:31 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov It has not.

Sun, 26 Mar 2023 10:46:02 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

It has not.

Well, I can do nothing about it.

Sun, 26 Mar 2023 10:47:31 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov So I waited for four days with you telling me "it's in the queue" and now you "can do nothing about it"...

@Mario Vavti What's the way forward here? How can these posts have been sent but not in the queue?

Sun, 26 Mar 2023 10:48:40 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

So I waited for four days with you telling me "it's in the queue" and now you "can do nothing about it"...

And?

Sun, 26 Mar 2023 10:49:18 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov If posts are being lost that seems like a problem. It seems like something someone running an instance would be interested in.

Sun, 26 Mar 2023 10:53:11 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

If posts are being lost that seems like a problem. It seems like something someone running an instance would be interested in.

Currently I see no issues on the @Zotum side.
If some of your deliveries are not displayed on other servers for whatever reason, there is nothing I can do about it. Try to re-initiate the re-delivery yourself.

Sun, 26 Mar 2023 10:57:51 +0200 zuletzt bearbeitet: Sun, 26 Mar 2023 11:03:07 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

Currently I see no issues on the @Zotum side.
If some of your deliveries are not displayed on other servers for whatever reason, there is nothing I can do about it. Try to re-initiate the re-delivery yourself.

I've already said that I tried that and nothing happened (nothing was delivered, to any of the six Mastodon servers that normally collect posts from the channel and should have collected the re-delivered post).

I have just tried it again (with two posts - the one I tried to redeliver before, and another that hadn't been touched), and again nothing happened.

There's no way there is no issue on the Zotum side, and all six independently hosted Mastodon servers have simultaneously failed for the last five days.

Sun, 26 Mar 2023 11:04:41 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

There's no way there is no issue on the Zotum side, and all six independently hosted Mastodon servers have simultaneously failed for the last five days.

I just checked and mastodon.social works for me. There is nothing to investigate on my side until we'll have some new inputs.
BTW, I don't see any contacts from the Mastodon servers in your profile.

Sun, 26 Mar 2023 11:12:35 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

I just checked and mastodon.social works for me. There is nothing to investigate on my side until we'll have some new inputs.

What you're saying here is that posts that are sent while ActivityPub connectivity is down and are never delivered, invisibly, are of no concern and won't be investigated.

If that's the case, Hubzilla becomes unusable for anyone who wants to use it with ActivityPub. It's not what I had understood the system to be, from @mike or @Mario Vavti.

BTW, I don't see any contacts from the Mastodon servers in your profile.

There is one contact on a Mastodon server on this profile. There are seven contacts on Mastodon servers on the profile where this issue (the missing posts) is being seen, which I linked a few replies ago. I'm not sure where you're looking.

Sun, 26 Mar 2023 11:16:38 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

while ActivityPub connectivity is down

Why did you turn it off? You shouldn't have done that, of course. It is impossible to deliver what should not have been delivered.

Sun, 26 Mar 2023 11:18:09 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

Why did you turn it off? You shouldn't have done that, of course. It is impossible to deliver what should not have been delivered.

I never turned it off. It stopped working.

You know this because I already told you, multiple times.

Sun, 26 Mar 2023 11:18:56 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

I never turned it off. It stopped working.

Works for me.

Sun, 26 Mar 2023 11:20:35 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

Works for me.

Ok Max.

It's clear you have absolutely no interest in investigating the issue here, or making Hubzilla work.

Thanks for clarifying.

I actually thought about supporting this server the same way @Maria Karlsen has, but I'm very glad right now I hadn't done that yet.

I'll move my data off the instance.

Sun, 26 Mar 2023 11:21:16 +0200 zuletzt bearbeitet: Sun, 26 Mar 2023 11:22:31 +0200

Max Kostikov

kostikov@zotum.net

@Victoria

It's clear you have absolutely no interest in investigating the issue here, or making Hubzilla work.

Once again, there is nothing to investigate at the moment.
Good luck.

Sun, 26 Mar 2023 11:29:25 +0200

Victoria

vsrjaccount@hub.netzgemeinde.eu

@Max Kostikov

there is nothing to investigate at the moment.

You have the posts. You have all the database entries related to them. You have the server logs. You can see every part of the system. And you are familiar with the code, because I've seen your contributions on GitLab.

If you don't want me to use your server, that's absolutely fine.

But I would like to be absolutely clear: There is an issue here that affects not only your instance but potentially all others using Hubzilla software that you have the capability to investigate, and are choosing not to. Anything else is a lie, or an incredibly strange misunderstanding for a man who runs a server and develops the code for it.

Goodbye Max.