Howspace (in.howspace.com) logo

All systems operational

Refreshed

This page contains the status of Howspace services in the EU region (in.howspace.com).

The status of Howspace services in the CA region can be found at https://status.ca.howspace.com.

Components

Howspace (in.howspace.com)

Operational

Sep 11, 2024
Uptime 100.000 %
Sep 12, 2024
Uptime 100.000 %
Sep 13, 2024
Uptime 100.000 %
Sep 14, 2024
Uptime 100.000 %
Sep 15, 2024
Uptime 100.000 %
Sep 16, 2024
Uptime 100.000 %
Sep 17, 2024
Uptime 100.000 %
Sep 18, 2024
Uptime 100.000 %
Sep 19, 2024
Uptime 100.000 %
Sep 20, 2024
Uptime 100.000 %
Sep 21, 2024
Uptime 100.000 %
Sep 22, 2024
Uptime 100.000 %
Sep 23, 2024
Uptime 100.000 %
Sep 24, 2024
Uptime 100.000 %
Sep 25, 2024
Uptime 100.000 %
Sep 26, 2024
Uptime 100.000 %
Sep 27, 2024
Uptime 100.000 %
Sep 28, 2024
Uptime 100.000 %
Sep 29, 2024
Uptime 100.000 %
Sep 30, 2024
Uptime 100.000 %
Oct 01, 2024
Uptime 100.000 %
Oct 02, 2024
Uptime 100.000 %
Oct 03, 2024
Uptime 100.000 %
Oct 04, 2024
Uptime 100.000 %
Oct 05, 2024
Uptime 100.000 %
Oct 06, 2024
Uptime 100.000 %
Oct 07, 2024
Uptime 100.000 %
Oct 08, 2024
Uptime 100.000 %
Oct 09, 2024
Uptime 100.000 %
Oct 10, 2024
Uptime 100.000 %
Oct 11, 2024
Uptime 100.000 %
Oct 12, 2024
Uptime 100.000 %
Oct 13, 2024
Uptime 100.000 %
Oct 14, 2024
Uptime 100.000 %
Oct 15, 2024
Uptime 100.000 %
Oct 16, 2024
Uptime 100.000 %
Oct 17, 2024
Uptime 100.000 %
Oct 18, 2024
Uptime 100.000 %
Oct 19, 2024
Uptime 100.000 %
Oct 20, 2024
Uptime 100.000 %
Oct 21, 2024
Uptime 100.000 %
Oct 22, 2024
Uptime 100.000 %
Oct 23, 2024
Uptime 100.000 %
Oct 24, 2024
Uptime 100.000 %
Oct 25, 2024
Uptime 100.000 %
Oct 26, 2024
Uptime 100.000 %
Oct 27, 2024
Uptime 100.000 %
Oct 28, 2024
Uptime 100.000 %
Oct 29, 2024
Uptime 100.000 %
Oct 30, 2024
Uptime 100.000 %
Oct 31, 2024
Uptime 100.000 %
Nov 01, 2024
Uptime 100.000 %
Nov 02, 2024
Uptime 100.000 %
Nov 03, 2024
Uptime 100.000 %
Nov 04, 2024
Uptime 100.000 %
Nov 05, 2024
Uptime 100.000 %
Nov 06, 2024
Uptime 100.000 %
Nov 07, 2024
Uptime 100.000 %
Nov 08, 2024
Uptime 100.000 %
Nov 09, 2024
Uptime 100.000 %
Nov 10, 2024
Uptime 100.000 %
Nov 11, 2024
Uptime 100.000 %
Nov 12, 2024
Uptime 100.000 %
Nov 13, 2024
Uptime 100.000 %
Nov 14, 2024
Uptime 100.000 %
Nov 15, 2024
Uptime 100.000 %
Nov 16, 2024
Uptime 100.000 %
Nov 17, 2024
Uptime 100.000 %
Nov 18, 2024
Uptime 100.000 %
Nov 19, 2024
Uptime 100.000 %
Nov 20, 2024
Uptime 100.000 %
Nov 21, 2024
Uptime 100.000 %
Nov 22, 2024
Uptime 100.000 %
Nov 23, 2024
Uptime 100.000 %
Nov 24, 2024
Uptime 100.000 %
Nov 25, 2024
Uptime 100.000 %
Nov 26, 2024
Uptime 100.000 %
Nov 27, 2024
Uptime 100.000 %
Nov 28, 2024
Uptime 100.000 %
Nov 29, 2024
Uptime 100.000 %
Nov 30, 2024
Uptime 100.000 %
Dec 01, 2024
Uptime 100.000 %
Dec 02, 2024
Uptime 100.000 %
Dec 03, 2024
Uptime 100.000 %
Dec 04, 2024
Uptime 100.000 %
Dec 05, 2024
Uptime 100.000 %
Dec 06, 2024
Uptime 100.000 %
Dec 07, 2024
Uptime 100.000 %
Dec 08, 2024
Uptime 100.000 %
Dec 09, 2024
Uptime 100.000 %

Uptime over the past 90 days 100.000 % uptime

Monitors

in.howspace.com https

Uptime 802 days 15 hours 16 minutes 20 seconds

Operational

24 hours

100.000%

Week

100.000%

Month

100.000%

Incident history

Short outage

Closed | Sep 25, 2024 | 10:38 GMT+03:00

Howspace had short (under 5 minutes) outage on Monday 23rd of September. This has been investigated and problem has been identified. Some of our widgets have technical implementation, that can cause heavy load for whole platform, if used a lot in a workspace with a large amount of users online.

We are sorry for any inconvenience this have caused and we are already working on a fix to avoid similar issues in the future.

Workspace login page not functional

Closed | Jun 05, 2024 | 09:03 GMT+03:00

This morning 9.02 EEST we deployed a version of Howspace that had an coding error which caused login pages to not load properly. Error was noticed quickly after the deployment and we immediately started working for a fix. Fixed version of Howspace was deployed 10.09 EEST and login pages are working again.

Issue affected all the users that tried to access workspaces without session and didn't came through login link.

To prevent this happening again we are doing few things:

- Add automated tests for login page (we only tested the login links actually works before this)
- Discuss internally about manual testing process

We are deeply sorry for any inconvenience this have caused and do our best to try avoid this happening in the future.

Affected components

  • Howspace (in.howspace.com)

We are experiencing issues with sending emails from workspaces

Closed | Nov 30, 2023 | 16:06 GMT+02:00

During 29.11.2023 12.35 - 30.11.2023 8.45 Howspace had an incident that prevented all email deliveries.


November 27th Spammy account noticed

On Monday 27th our employee noticed from our monitoring, that there was an exceptional spike in email delivery amounts on one of the Free accounts. We immediately investigated the issue and noticed that this account was used to deliver emails that were obviously spam. User limitation was circumvented by always deleting the users and adding new ones. Emails were sent during the weekend, from Friday to Sunday. When we investigated this on Monday, the last emails were sent 6 hours ago.

We immediately closed the account and went to look through if there were similar accounts. We found 3 other accounts that were used for similar activity and closed those too.

In case to have a better view of a situation like this, we implemented some fine-tuned monitoring and alerts for Free email sending and decided to manually pay close attention to every new Free account that is created.


November 29th Email delivery stopped

At 12.35 (GMT) our support received a phone call from our client, that they cannot receive any emails from Howspace. Something that is usually a common support case (email address can be mistyped, the spam filter is working too eagerly, etc) turned much more severe when our technical support noticed that AWS (our hosting provider) had paused our email deliveries. That was immediately escalated to our developers, who then noticed that AWS had created a support issue on Friday 24th, and warned us about the increasing amount of email complaints we are receiving that could lead to pausing our email deliveries. The problem with this issue was that we never received an actual email from that issue, so no one in our company knew about this.

This issue was already updated with the information, that our email sending is paused until we take actions that prevent email complaints from increasing.

We then verified that most of the complaints were caused by the spammy account we identified and closed on Monday. After that, at 14.21, we gave detailed answers about what had happened and the actions we had already taken on Monday. Since email delivery is a critical aspect of our platform (the most common login method is a login link delivered by email), we contacted our Account Manager, who promised to increase the importance of our issue.

We didn't receive any information back from AWS, so we decided to open another support ticket to them at 19.21 (they provide 4-hour response times for issues with "Production system impaired" severity). We highlighted the importance of emails for our platform and asked for a quick resolution. We got a response for this issue on 00.20 where they closed the duplicate issue raised the severity of the original issue - and promised that the engineer will be reaching out to us soon.


November 30th Issue resolved

On 4.26 Thursday morning we still haven't received any update on the original issue, where AWS notified us they had paused our email service and we had already provided the list of actions we had taken to prevent the issue. We asked for an update and 2 minutes later received an (probably automated) update to the issue, that asked for the same evidence again. At 4.38 we provided them again.

Then at 6.10, we received the next update which once again asked for more information about what actions we had taken to prevent this, but this time also said that "your appeal did not provide the information we need to make a decision". So we answered for 3rd time at 6.42 and this time added also new action where we closed our Free form, which can be used to open new Free accounts on our platform.

We started to feel a bit desperate about the poor communications and the lack of urgency we were receiving from AWS, so we started thinking about alternatives - although we knew there wasn't any fast shortcut available. Then at 8.45, we received an update on the issue, that the review period and sending pause was over and email delivery was working again.


Aftermath

Our platform is sending approx. million emails a month and the incident that led AWS to halt our full email deliveries was caused by one spam account, that sent ~2000 spam emails through our platform. We never received any email, phone call, or other notification about that this is leading to halting our whole email delivery. While we are very unhappy about how our hosting partner handled this topic, we can only focus and take responsibility for our actions. Therefore we have decided to improve our email deliveries in the future by:

- Adding hard limitations to the amount of emails Free accounts can send
- Enhance our monitoring and alerting so we get ahead of the issues before they escalate
- Implement an alternative email delivery system, so that in case of failure, we can quickly change the way our platform sends emails. This should also help us to improve the email delivery percentage.

We are deeply sorry for any inconvenience this has caused for your Howspace usage and we do our best to make sure this won't happen again.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Nov 30, 2023 | 10:49 GMT+02:00

    Emails are working again.

  • Open | Nov 30, 2023 | 09:29 GMT+02:00

    We have identified the underlying issue behind the email sending issue and are working towards a resolution. We thank you for your patience.

    Our alternative methods of logging in are not affected by this issue. SSO login and registrations through social media accounts can be used to access workspaces. Also most of the login links in older emails should work.

Unexpected outage

Closed | Oct 17, 2023 | 00:00 GMT+03:00

The issue has been resolved and closed.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Oct 12, 2023 | 14:56 GMT+03:00

    Yesterday, from 10:28 AM EEST to 2:50 PM EEST, the Howspace platform experienced an unexpected outage, resulting in unavailability and delays for some users. The outage was attributed to an unstable WebSocket connection.

    We are pleased to inform you that the implemented fixes have successfully stabilized the platform. We are actively monitoring the system and will allocate development resources to enhance the reliability of our WebSocket implementation, thus preventing such incidents in the future.

    We appreciate your patience and understanding throughout the outage.

  • Monitoring | Oct 11, 2023 | 15:26 GMT+03:00

    We've deployed fixes that have significantly improved the platform's stability. We're maintaining continuous monitoring to ensure its continued reliability.

  • Investigating | Oct 11, 2023 | 15:15 GMT+03:00

    An update has been launched that should help alleviate the issue but it is not a final resolution to the issue. Our work towards finding and solving the root cause continues.

  • Investigating | Oct 11, 2023 | 14:07 GMT+03:00

    The issue remains and our work to resolve it continues. There was momentary respite after we took certain steps towards fixing the problem. It seems the root cause still hasn't been solved.

    We apologize for the inconvenience and thank you for your understanding.

  • Investigating | Oct 11, 2023 | 12:12 GMT+03:00

    We are actively monitoring the situation and working on finding the root cause behind the issue. The issue seems to have alleviated and performance has improved during the latest 30 minutes.

  • Investigating | Oct 11, 2023 | 10:51 GMT+03:00

    Howspace is experiencing an issue with the WebSocket connection. We are investigating the reasons behind the issue and are working on resolving it as soon as possible. We apologize for any inconvenience caused and appreciate your patience during this time.

Websocket connection issue

Closed | Jun 08, 2023 | 12:00 GMT+03:00

The issue has been resolved and closed.

The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

We apologize for any inconvenience it might have caused and thank you for your patience and understanding.

Affected components

  • Howspace (in.howspace.com)

+Show history

  • Resolved | Jun 01, 2023 | 16:00 GMT+03:00

    The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

    We are pleased to inform you that our development team swiftly identified and resolved the issue. We are actively working to prevent similar challenges from occurring in the future. We apologize for any inconvenience caused and appreciate your patience during this time.

  • Monitoring | Jun 01, 2023 | 14:15 GMT+03:00

    The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

  • Resolution in progress | Jun 01, 2023 | 13:48 GMT+03:00

    We are deploying an update to production that will hopefully resolve the connection issue.

  • Open | Jun 01, 2023 | 13:10 GMT+03:00

    There is currently an issue with the Websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.

Maintenance

No maintenance reported.

Status Page powered by Admin Labs

Subscribe to updates

Get email notifications about status page updates.

Thanks for subscribing.

Something went wrong with your email address, please try again.