Hi there,
Unfortunately the site was down for about 2 days over the 26-27 May this month.
This was due to the datacentre we use being affected by a storm/tornado: (https://apnews.com/article/tornadoes-weather-texas-oklahoma-93001e1d81f120d0d55c8dfa24938250) which took out power to the facility for just under 2 days. At no time was there any risk of loss to the server due to off site backups, however we did have to wait for the centre to come back online.
While this server is run extremely cheaply, at just $20 USD per month (approx) which includes paying for the compute, domain, email, and off site backup as well as object storage, it does mean that there is not redundancy when a site wide outage occurs. The low cost means that there is no risk of this server ever being unable to meet its costs and be paid for, but it does mean we sometimes can have an outage if a major event happens such as what happened and a delay before being brought back online.
The decision for this was because this server is designed to help reduce load on the rest of the fediverse and does not store much local content so a couple days a year the server is out was deemed acceptable during the design phase.
If you have any questions or comments feel free to post and I will try my best to answer them :)
- P
Thanks for keeping us posted!
Looking at Lemmy posts right now, it seems like many servers were affected - I see a lot of beehaw posts and not a lot else at the moment. I’m curious how big of an impact this was.
Thank you, I am busy looking at the federation issue right now and hope to resolve it soon. This instance is just a single server so if its turned off then the impact is total, but in terms of the datacentre the entire site was down and power was slowly restored to parts of it and is still ongoing, we are luckily back online but some parts are still down and damaged. The site is 1515 round table drive Dallas TX.
I also purged the post you reported.
Federation is returning to normal and should be working again fully by tomorrow. There is some catch up to do.
Good news! Thank you for managing the server!
Just to let you have an update, we are caught up with all instances except lemmy.world which is still 3 days behind. You can monitor or track the federation progress here: https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=lemmy.myserv.one&from=now-12h&to=now
All good, sorry for the downtime :)