Howspace (in.howspace.com) logo

All systems operational

Refreshed

This page contains the status of Howspace services in the EU region (in.howspace.com).

The status of Howspace services in the CA region can be found at https://status.ca.howspace.com.

Components

Howspace (in.howspace.com)

Operational

Apr 17, 2024
Uptime 100.000 %
Apr 18, 2024
Uptime 100.000 %
Apr 19, 2024
Uptime 100.000 %
Apr 20, 2024
Uptime 100.000 %
Apr 21, 2024
Uptime 100.000 %
Apr 22, 2024
Uptime 100.000 %
Apr 23, 2024
Uptime 100.000 %
Apr 24, 2024
Uptime 100.000 %
Apr 25, 2024
Uptime 100.000 %
Apr 26, 2024
Uptime 100.000 %
Apr 27, 2024
Uptime 100.000 %
Apr 28, 2024
Uptime 100.000 %
Apr 29, 2024
Uptime 100.000 %
Apr 30, 2024
Uptime 100.000 %
May 01, 2024
Uptime 100.000 %
May 02, 2024
Uptime 100.000 %
May 03, 2024
Uptime 100.000 %
May 04, 2024
Uptime 100.000 %
May 05, 2024
Uptime 100.000 %
May 06, 2024
Uptime 100.000 %
May 07, 2024
Uptime 100.000 %
May 08, 2024
Uptime 100.000 %
May 09, 2024
Uptime 100.000 %
May 10, 2024
Uptime 100.000 %
May 11, 2024
Uptime 100.000 %
May 12, 2024
Uptime 100.000 %
May 13, 2024
Uptime 100.000 %
May 14, 2024
Uptime 100.000 %
May 15, 2024
Uptime 100.000 %
May 16, 2024
Uptime 100.000 %
May 17, 2024
Uptime 100.000 %
May 18, 2024
Uptime 100.000 %
May 19, 2024
Uptime 100.000 %
May 20, 2024
Uptime 100.000 %
May 21, 2024
Uptime 100.000 %
May 22, 2024
Uptime 100.000 %
May 23, 2024
Uptime 100.000 %
May 24, 2024
Uptime 100.000 %
May 25, 2024
Uptime 100.000 %
May 26, 2024
Uptime 100.000 %
May 27, 2024
Uptime 100.000 %
May 28, 2024
Uptime 100.000 %
May 29, 2024
Uptime 100.000 %
May 30, 2024
Uptime 100.000 %
May 31, 2024
Uptime 100.000 %
Jun 01, 2024
Uptime 100.000 %
Jun 02, 2024
Uptime 100.000 %
Jun 03, 2024
Uptime 100.000 %
Jun 04, 2024
Uptime 100.000 %
Jun 05, 2024
Uptime 100.000 %
Jun 06, 2024
Uptime 100.000 %
Jun 07, 2024
Uptime 100.000 %
Jun 08, 2024
Uptime 100.000 %
Jun 09, 2024
Uptime 100.000 %
Jun 10, 2024
Uptime 100.000 %
Jun 11, 2024
Uptime 100.000 %
Jun 12, 2024
Uptime 100.000 %
Jun 13, 2024
Uptime 100.000 %
Jun 14, 2024
Uptime 100.000 %
Jun 15, 2024
Uptime 100.000 %
Jun 16, 2024
Uptime 100.000 %
Jun 17, 2024
Uptime 100.000 %
Jun 18, 2024
Uptime 100.000 %
Jun 19, 2024
Uptime 100.000 %
Jun 20, 2024
Uptime 100.000 %
Jun 21, 2024
Uptime 100.000 %
Jun 22, 2024
Uptime 100.000 %
Jun 23, 2024
Uptime 100.000 %
Jun 24, 2024
Uptime 100.000 %
Jun 25, 2024
Uptime 100.000 %
Jun 26, 2024
Uptime 100.000 %
Jun 27, 2024
Uptime 100.000 %
Jun 28, 2024
Uptime 100.000 %
Jun 29, 2024
Uptime 100.000 %
Jun 30, 2024
Uptime 100.000 %
Jul 01, 2024
Uptime 100.000 %
Jul 02, 2024
Uptime 100.000 %
Jul 03, 2024
Uptime 100.000 %
Jul 04, 2024
Uptime 100.000 %
Jul 05, 2024
Uptime 100.000 %
Jul 06, 2024
Uptime 100.000 %
Jul 07, 2024
Uptime 100.000 %
Jul 08, 2024
Uptime 100.000 %
Jul 09, 2024
Uptime 100.000 %
Jul 10, 2024
Uptime 100.000 %
Jul 11, 2024
Uptime 100.000 %
Jul 12, 2024
Uptime 100.000 %
Jul 13, 2024
Uptime 100.000 %
Jul 14, 2024
Uptime 100.000 %
Jul 15, 2024
Uptime 100.000 %

Uptime over the past 90 days 100.000 % uptime

Monitors

in.howspace.com https

Uptime 656 days 33 minutes 47 seconds

Operational

24 hours

100.000%

Week

100.000%

Month

100.000%

Incident history

Workspace login page not functional

Closed | Jun 05, 2024 | 09:03 GMT+03:00

This morning 9.02 EEST we deployed a version of Howspace that had an coding error which caused login pages to not load properly. Error was noticed quickly after the deployment and we immediately started working for a fix. Fixed version of Howspace was deployed 10.09 EEST and login pages are working again.

Issue affected all the users that tried to access workspaces without session and didn't came through login link.

To prevent this happening again we are doing few things:

- Add automated tests for login page (we only tested the login links actually works before this)
- Discuss internally about manual testing process

We are deeply sorry for any inconvenience this have caused and do our best to try avoid this happening in the future.

Affected components

  • Howspace (in.howspace.com)

We are experiencing issues with sending emails from workspaces

Closed | Nov 30, 2023 | 16:06 GMT+02:00

During 29.11.2023 12.35 - 30.11.2023 8.45 Howspace had an incident that prevented all email deliveries.


November 27th Spammy account noticed

On Monday 27th our employee noticed from our monitoring, that there was an exceptional spike in email delivery amounts on one of the Free accounts. We immediately investigated the issue and noticed that this account was used to deliver emails that were obviously spam. User limitation was circumvented by always deleting the users and adding new ones. Emails were sent during the weekend, from Friday to Sunday. When we investigated this on Monday, the last emails were sent 6 hours ago.

We immediately closed the account and went to look through if there were similar accounts. We found 3 other accounts that were used for similar activity and closed those too.

In case to have a better view of a situation like this, we implemented some fine-tuned monitoring and alerts for Free email sending and decided to manually pay close attention to every new Free account that is created.


November 29th Email delivery stopped

At 12.35 (GMT) our support received a phone call from our client, that they cannot receive any emails from Howspace. Something that is usually a common support case (email address can be mistyped, the spam filter is working too eagerly, etc) turned much more severe when our technical support noticed that AWS (our hosting provider) had paused our email deliveries. That was immediately escalated to our developers, who then noticed that AWS had created a support issue on Friday 24th, and warned us about the increasing amount of email complaints we are receiving that could lead to pausing our email deliveries. The problem with this issue was that we never received an actual email from that issue, so no one in our company knew about this.

This issue was already updated with the information, that our email sending is paused until we take actions that prevent email complaints from increasing.

We then verified that most of the complaints were caused by the spammy account we identified and closed on Monday. After that, at 14.21, we gave detailed answers about what had happened and the actions we had already taken on Monday. Since email delivery is a critical aspect of our platform (the most common login method is a login link delivered by email), we contacted our Account Manager, who promised to increase the importance of our issue.

We didn't receive any information back from AWS, so we decided to open another support ticket to them at 19.21 (they provide 4-hour response times for issues with "Production system impaired" severity). We highlighted the importance of emails for our platform and asked for a quick resolution. We got a response for this issue on 00.20 where they closed the duplicate issue raised the severity of the original issue - and promised that the engineer will be reaching out to us soon.


November 30th Issue resolved

On 4.26 Thursday morning we still haven't received any update on the original issue, where AWS notified us they had paused our email service and we had already provided the list of actions we had taken to prevent the issue. We asked for an update and 2 minutes later received an (probably automated) update to the issue, that asked for the same evidence again. At 4.38 we provided them again.

Then at 6.10, we received the next update which once again asked for more information about what actions we had taken to prevent this, but this time also said that "your appeal did not provide the information we need to make a decision". So we answered for 3rd time at 6.42 and this time added also new action where we closed our Free form, which can be used to open new Free accounts on our platform.

We started to feel a bit desperate about the poor communications and the lack of urgency we were receiving from AWS, so we started thinking about alternatives - although we knew there wasn't any fast shortcut available. Then at 8.45, we received an update on the issue, that the review period and sending pause was over and email delivery was working again.


Aftermath

Our platform is sending approx. million emails a month and the incident that led AWS to halt our full email deliveries was caused by one spam account, that sent ~2000 spam emails through our platform. We never received any email, phone call, or other notification about that this is leading to halting our whole email delivery. While we are very unhappy about how our hosting partner handled this topic, we can only focus and take responsibility for our actions. Therefore we have decided to improve our email deliveries in the future by:

- Adding hard limitations to the amount of emails Free accounts can send
- Enhance our monitoring and alerting so we get ahead of the issues before they escalate
- Implement an alternative email delivery system, so that in case of failure, we can quickly change the way our platform sends emails. This should also help us to improve the email delivery percentage.

We are deeply sorry for any inconvenience this has caused for your Howspace usage and we do our best to make sure this won't happen again.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Nov 30, 2023 | 10:49 GMT+02:00

    Emails are working again.

  • Open | Nov 30, 2023 | 09:29 GMT+02:00

    We have identified the underlying issue behind the email sending issue and are working towards a resolution. We thank you for your patience.

    Our alternative methods of logging in are not affected by this issue. SSO login and registrations through social media accounts can be used to access workspaces. Also most of the login links in older emails should work.

Unexpected outage

Closed | Oct 17, 2023 | 00:00 GMT+03:00

The issue has been resolved and closed.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Oct 12, 2023 | 14:56 GMT+03:00

    Yesterday, from 10:28 AM EEST to 2:50 PM EEST, the Howspace platform experienced an unexpected outage, resulting in unavailability and delays for some users. The outage was attributed to an unstable WebSocket connection.

    We are pleased to inform you that the implemented fixes have successfully stabilized the platform. We are actively monitoring the system and will allocate development resources to enhance the reliability of our WebSocket implementation, thus preventing such incidents in the future.

    We appreciate your patience and understanding throughout the outage.

  • Monitoring | Oct 11, 2023 | 15:26 GMT+03:00

    We've deployed fixes that have significantly improved the platform's stability. We're maintaining continuous monitoring to ensure its continued reliability.

  • Investigating | Oct 11, 2023 | 15:15 GMT+03:00

    An update has been launched that should help alleviate the issue but it is not a final resolution to the issue. Our work towards finding and solving the root cause continues.

  • Investigating | Oct 11, 2023 | 14:07 GMT+03:00

    The issue remains and our work to resolve it continues. There was momentary respite after we took certain steps towards fixing the problem. It seems the root cause still hasn't been solved.

    We apologize for the inconvenience and thank you for your understanding.

  • Investigating | Oct 11, 2023 | 12:12 GMT+03:00

    We are actively monitoring the situation and working on finding the root cause behind the issue. The issue seems to have alleviated and performance has improved during the latest 30 minutes.

  • Investigating | Oct 11, 2023 | 10:51 GMT+03:00

    Howspace is experiencing an issue with the WebSocket connection. We are investigating the reasons behind the issue and are working on resolving it as soon as possible. We apologize for any inconvenience caused and appreciate your patience during this time.

Websocket connection issue

Closed | Jun 08, 2023 | 12:00 GMT+03:00

The issue has been resolved and closed.

The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

We apologize for any inconvenience it might have caused and thank you for your patience and understanding.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Jun 01, 2023 | 16:00 GMT+03:00

    The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

    We are pleased to inform you that our development team swiftly identified and resolved the issue. We are actively working to prevent similar challenges from occurring in the future. We apologize for any inconvenience caused and appreciate your patience during this time.

  • Monitoring | Jun 01, 2023 | 14:15 GMT+03:00

    The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

  • Resolution in progress | Jun 01, 2023 | 13:48 GMT+03:00

    We are deploying an update to production that will hopefully resolve the connection issue.

  • Open | Jun 01, 2023 | 13:10 GMT+03:00

    There is currently an issue with the Websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.

Websocket Connection issue

Closed | Dec 02, 2022 | 11:33 GMT+02:00

The issue has been resolved and closed.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Monitoring | Dec 01, 2022 | 16:25 GMT+02:00

    The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

  • Resolution in progress | Dec 01, 2022 | 16:03 GMT+02:00

    We are deploying an update to production that will hopefully resolve the connection issue.

  • Open | Dec 01, 2022 | 15:19 GMT+02:00

    There is currently an issue with the websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.

    We'll update here as we get more information.

Maintenance

No maintenance reported.

Status Page powered by Admin Labs

Subscribe to updates

Get email notifications about status page updates.

Thanks for subscribing.

Something went wrong with your email address, please try again.