Short outage

Closed | Sep 25, 2024 | 10:38 GMT+03:00

Howspace had short (under 5 minutes) outage on Monday 23rd of September. This has been investigated and problem has been identified. Some of our widgets have technical implementation, that can cause heavy load for whole platform, if used a lot in a workspace with a large amount of users online.

We are sorry for any inconvenience this have caused and we are already working on a fix to avoid similar issues in the future.

Workspace login page not functional

Closed | Jun 05, 2024 | 09:03 GMT+03:00

This morning 9.02 EEST we deployed a version of Howspace that had an coding error which caused login pages to not load properly. Error was noticed quickly after the deployment and we immediately started working for a fix. Fixed version of Howspace was deployed 10.09 EEST and login pages are working again.

Issue affected all the users that tried to access workspaces without session and didn't came through login link.

To prevent this happening again we are doing few things:

- Add automated tests for login page (we only tested the login links actually works before this)
- Discuss internally about manual testing process

We are deeply sorry for any inconvenience this have caused and do our best to try avoid this happening in the future.

Affected components

• Howspace (in.howspace.com)

We are experiencing issues with sending emails from workspaces

Closed | Nov 30, 2023 | 16:06 GMT+02:00

During 29.11.2023 12.35 - 30.11.2023 8.45 Howspace had an incident that prevented all email deliveries.


November 27th Spammy account noticed

On Monday 27th our employee noticed from our monitoring, that there was an exceptional spike in email delivery amounts on one of the Free accounts. We immediately investigated the issue and noticed that this account was used to deliver emails that were obviously spam. User limitation was circumvented by always deleting the users and adding new ones. Emails were sent during the weekend, from Friday to Sunday. When we investigated this on Monday, the last emails were sent 6 hours ago.

We immediately closed the account and went to look through if there were similar accounts. We found 3 other accounts that were used for similar activity and closed those too.

In case to have a better view of a situation like this, we implemented some fine-tuned monitoring and alerts for Free email sending and decided to manually pay close attention to every new Free account that is created.


November 29th Email delivery stopped

At 12.35 (GMT) our support received a phone call from our client, that they cannot receive any emails from Howspace. Something that is usually a common support case (email address can be mistyped, the spam filter is working too eagerly, etc) turned much more severe when our technical support noticed that AWS (our hosting provider) had paused our email deliveries. That was immediately escalated to our developers, who then noticed that AWS had created a support issue on Friday 24th, and warned us about the increasing amount of email complaints we are receiving that could lead to pausing our email deliveries. The problem with this issue was that we never received an actual email from that issue, so no one in our company knew about this.

This issue was already updated with the information, that our email sending is paused until we take actions that prevent email complaints from increasing.

We then verified that most of the complaints were caused by the spammy account we identified and closed on Monday. After that, at 14.21, we gave detailed answers about what had happened and the actions we had already taken on Monday. Since email delivery is a critical aspect of our platform (the most common login method is a login link delivered by email), we contacted our Account Manager, who promised to increase the importance of our issue.

We didn't receive any information back from AWS, so we decided to open another support ticket to them at 19.21 (they provide 4-hour response times for issues with "Production system impaired" severity). We highlighted the importance of emails for our platform and asked for a quick resolution. We got a response for this issue on 00.20 where they closed the duplicate issue raised the severity of the original issue - and promised that the engineer will be reaching out to us soon.


November 30th Issue resolved

On 4.26 Thursday morning we still haven't received any update on the original issue, where AWS notified us they had paused our email service and we had already provided the list of actions we had taken to prevent the issue. We asked for an update and 2 minutes later received an (probably automated) update to the issue, that asked for the same evidence again. At 4.38 we provided them again.

Then at 6.10, we received the next update which once again asked for more information about what actions we had taken to prevent this, but this time also said that "your appeal did not provide the information we need to make a decision". So we answered for 3rd time at 6.42 and this time added also new action where we closed our Free form, which can be used to open new Free accounts on our platform.

We started to feel a bit desperate about the poor communications and the lack of urgency we were receiving from AWS, so we started thinking about alternatives - although we knew there wasn't any fast shortcut available. Then at 8.45, we received an update on the issue, that the review period and sending pause was over and email delivery was working again.


Aftermath

Our platform is sending approx. million emails a month and the incident that led AWS to halt our full email deliveries was caused by one spam account, that sent ~2000 spam emails through our platform. We never received any email, phone call, or other notification about that this is leading to halting our whole email delivery. While we are very unhappy about how our hosting partner handled this topic, we can only focus and take responsibility for our actions. Therefore we have decided to improve our email deliveries in the future by:

- Adding hard limitations to the amount of emails Free accounts can send
- Enhance our monitoring and alerting so we get ahead of the issues before they escalate
- Implement an alternative email delivery system, so that in case of failure, we can quickly change the way our platform sends emails. This should also help us to improve the email delivery percentage.

We are deeply sorry for any inconvenience this has caused for your Howspace usage and we do our best to make sure this won't happen again.

Affected components

• Howspace (in.howspace.com)

+Show history

• Resolved | Nov 30, 2023 | 10:49 GMT+02:00

  Emails are working again.

• Open | Nov 30, 2023 | 09:29 GMT+02:00

  We have identified the underlying issue behind the email sending issue and are working towards a resolution. We thank you for your patience.
  
  Our alternative methods of logging in are not affected by this issue. SSO login and registrations through social media accounts can be used to access workspaces. Also most of the login links in older emails should work.

Unexpected outage

Closed | Oct 17, 2023 | 00:00 GMT+03:00

The issue has been resolved and closed.

Affected components

• Howspace (in.howspace.com)

+Show history

• Resolved | Oct 12, 2023 | 14:56 GMT+03:00

  Yesterday, from 10:28 AM EEST to 2:50 PM EEST, the Howspace platform experienced an unexpected outage, resulting in unavailability and delays for some users. The outage was attributed to an unstable WebSocket connection.
  
  We are pleased to inform you that the implemented fixes have successfully stabilized the platform. We are actively monitoring the system and will allocate development resources to enhance the reliability of our WebSocket implementation, thus preventing such incidents in the future.
  
  We appreciate your patience and understanding throughout the outage.

• Monitoring | Oct 11, 2023 | 15:26 GMT+03:00

  We've deployed fixes that have significantly improved the platform's stability. We're maintaining continuous monitoring to ensure its continued reliability.

• Investigating | Oct 11, 2023 | 15:15 GMT+03:00

  An update has been launched that should help alleviate the issue but it is not a final resolution to the issue. Our work towards finding and solving the root cause continues.

• Investigating | Oct 11, 2023 | 14:07 GMT+03:00

  The issue remains and our work to resolve it continues. There was momentary respite after we took certain steps towards fixing the problem. It seems the root cause still hasn't been solved.
  
  We apologize for the inconvenience and thank you for your understanding.

• Investigating | Oct 11, 2023 | 12:12 GMT+03:00

  We are actively monitoring the situation and working on finding the root cause behind the issue. The issue seems to have alleviated and performance has improved during the latest 30 minutes.

• Investigating | Oct 11, 2023 | 10:51 GMT+03:00

  Howspace is experiencing an issue with the WebSocket connection. We are investigating the reasons behind the issue and are working on resolving it as soon as possible. We apologize for any inconvenience caused and appreciate your patience during this time.

Websocket connection issue

Closed | Jun 08, 2023 | 12:00 GMT+03:00

The issue has been resolved and closed.

The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

We apologize for any inconvenience it might have caused and thank you for your patience and understanding.

Affected components

• Howspace (in.howspace.com)

+Show history

• Resolved | Jun 01, 2023 | 16:00 GMT+03:00

  The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.
  
  We are pleased to inform you that our development team swiftly identified and resolved the issue. We are actively working to prevent similar challenges from occurring in the future. We apologize for any inconvenience caused and appreciate your patience during this time.

• Monitoring | Jun 01, 2023 | 14:15 GMT+03:00

  The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

• Resolution in progress | Jun 01, 2023 | 13:48 GMT+03:00

  We are deploying an update to production that will hopefully resolve the connection issue.
  
  

• Open | Jun 01, 2023 | 13:10 GMT+03:00

  There is currently an issue with the Websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.