Type: Incident

Duration: 26 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>14:41:07</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>14:47:11</var> GMT+0</small><br><strong>Identified</strong> - It's DNS..</p> <p><small>Aug <var data-var='date'> 19</var>, <var data-var='time'>15:06:48</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p>

Type: Incident

Duration: 1 day, 2 hours and 14 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>15:07:38</var> GMT+0</small><br><strong>Investigating</strong> - There appears to be an issue with our Elasticsearch cluster that is causing the instance to fail. We are currently investigating the issue..</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>16:00:43</var> GMT+0</small><br><strong>Identified</strong> - We identified the source of the abnormal search trafic and are working on a solution.</p> <p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>17:43:17</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>17:21:52</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p>

Type: Incident

Duration: 1 hour and 2 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>22:26:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>21:24:41</var> GMT+0</small><br><strong>Investigating</strong> - There appears to be an issue with our Elasticsearch cluster that is causing the instance to fail. We are currently investigating the issue..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>21:46:41</var> GMT+0</small><br><strong>Identified</strong> - The issue seems to be incredibly high timeouts on Hetzner volumes, causing components with high-throughput volumes like Elasticsearch to degrade and fail. Search has temporarily been disabled while we investigate this issue and attempt to work around it..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>22:26:45</var> GMT+0</small><br><strong>Monitoring</strong> - Search functionality has been restored and volume timeouts seem to have been fixed. We will continue to monitor the situation..</p>

Type: Incident

Duration: 5 days, 21 hours and 24 minutes

Affected Components:

<p><small>May <var data-var='date'> 28</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are having issues with the caching server used by mastodon.social, which is causing increased latency and intermittent errors. We are working on implementing a solution..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>19:56:29</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>12:23:39</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p>

Type: Incident

Duration: 20 hours and 6 minutes

Affected Components: , , ,

<p><small>Apr <var data-var='date'> 24</var>, <var data-var='time'>05:54:04</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 23</var>, <var data-var='time'>09:48:33</var> GMT+0</small><br><strong>Identified</strong> - We are (again) targeted by a DDoS attack and [mastodon.social](http://mastodon.social) may not be available while we mitigate it..</p> <p><small>Apr <var data-var='date'> 23</var>, <var data-var='time'>10:05:56</var> GMT+0</small><br><strong>Monitoring</strong> - We have enabled DDoS protection methods, and services is mostly restored. We still continue to monitor the situation..</p> <p><small>Apr <var data-var='date'> 23</var>, <var data-var='time'>10:40:45</var> GMT+0</small><br><strong>Monitoring</strong> - The attack seems to have stopped. We are continuing to monitor the situation..</p>

Type: Incident

Duration: 21 hours and 15 minutes

Affected Components: ,

<p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>10:58:33</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing a DDoS attack. We are actively investigating the issue..</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>13:05:28</var> GMT+0</small><br><strong>Monitoring</strong> - We have implemented countermeasure against the DDoS attack, and the site is accessible. We are continuing to monitor the situation..</p> <p><small>Apr <var data-var='date'> 21</var>, <var data-var='time'>08:13:37</var> GMT+0</small><br><strong>Resolved</strong> - The attacks seem to have stopped, all operations have returned to normal..</p>

Type: Incident

Duration: 27 minutes

Affected Components: , ,

<p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>21:08:32</var> GMT+0</small><br><strong>Investigating</strong> - Hetzner, our cloud provider, appears to be having a significant outage, leaving many of our resources unavailable. We are monitoring the issue..</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>21:16:30</var> GMT+0</small><br><strong>Monitoring</strong> - Services have been restored and Hetzner connectivity seems to have returned. Will continue to monitor the situation..</p> <p><small>Feb <var data-var='date'> 25</var>, <var data-var='time'>21:35:12</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p>

Type: Incident

Duration: 12 days, 18 hours and 22 minutes

Affected Components: ,

<p><small>Dec <var data-var='date'> 17</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Investigating</strong> - There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue..</p> <p><small>Jan <var data-var='date'> 18</var>, <var data-var='time'>09:22:00</var> GMT+0</small><br><strong>Resolved</strong> - Sidekiq has been stable, so considering this incident resolved..</p> <p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>06:42:57</var> GMT+0</small><br><strong>Monitoring</strong> - We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue..</p>

Type: Incident

Duration: 8 minutes

Affected Components: ,

<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>17:20:27</var> GMT+0</small><br><strong>Identified</strong> - [mastodon.social](http://mastodon.social) is currently unresponsive to a database issue. We identified the problem and are working on fixing it..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>17:28:47</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, everything is now operational again..</p>

Type: Incident

Duration: 6 days, 2 hours and 35 minutes

<p><small>Jun <var data-var='date'> 17</var>, <var data-var='time'>13:50:20</var> GMT+0</small><br><strong>Identified</strong> - We paused sending emails from [mastodon.social](http://mastodon.social) (including password recovery and account confirmation emails) due to an identified issue that sent duplicate emails about our upcoming Terms of Service changes to some users. We are working on fixing this and avoiding more duplicate emails to be sent..</p> <p><small>Jun <var data-var='date'> 17</var>, <var data-var='time'>17:42:30</var> GMT+0</small><br><strong>Monitoring</strong> - Email sending has been removed. Servers are working through the backlog, and any pending email should arrive in the next few hours..</p> <p><small>Jun <var data-var='date'> 18</var>, <var data-var='time'>05:40:36</var> GMT+0</small><br><strong>Identified</strong> - The backlog of emails has been sent, and emails are reaching out their recipients now. Some emails sent in the last 12 hours are stuck with our email provider and we are working with them to fix this situation..</p> <p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>16:25:19</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved last Thursday, and email have been delivered correctly since then..</p>

Type: Incident

Duration: 2 hours and 40 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>22:33:26</var> GMT+0</small><br><strong>Investigating</strong> - We are experiencing issues with our redis caching system, as well as kubernetes node errors. We are currently investigating the issue..</p> <p><small>Jun <var data-var='date'> 11</var>, <var data-var='time'>22:42:52</var> GMT+0</small><br><strong>Identified</strong> - Website is operative again. Issue seems to be related to kubernetes pods becoming completely unresponsive. Continuing to investigate..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>01:13:29</var> GMT+0</small><br><strong>Resolved</strong> - Kubernetes node issues have been resolved, and service is now fully restored..</p>

Type: Maintenance

Duration: 16 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Identified</strong> - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>06:46:24</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p>