Websocket connection issue

Resolved | Jun 01, 2023 | 16:00 GMT+03:00

The Websocket connection issues started when our updated Websocket authentication process closed some connections ungracefully. This triggered a flood of retry attempts from the disconnected clients, which overloaded our servers and propagated the problem to other connections.

We are pleased to inform you that our development team swiftly identified and resolved the issue. We are actively working to prevent similar challenges from occurring in the future. We apologize for any inconvenience caused and appreciate your patience during this time.

Affected components

• Howspace (in.howspace.com)

+Show history

• Monitoring | Jun 01, 2023 | 14:15 GMT+03:00

  The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

• Resolution in progress | Jun 01, 2023 | 13:48 GMT+03:00

  We are deploying an update to production that will hopefully resolve the connection issue.
  
  

• Open | Jun 01, 2023 | 13:10 GMT+03:00

  There is currently an issue with the Websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.
  

Websocket Connection issue

Resolved | Dec 02, 2022 | 11:33 GMT+02:00

The issue has been resolved.

Affected components

• Howspace (in.howspace.com)

+Show history

• Monitoring | Dec 01, 2022 | 16:25 GMT+02:00

  The websocket connection issue has now been fixed. We are still monitoring the situation and analyzing the root cause behind the issue.

• Resolution in progress | Dec 01, 2022 | 16:03 GMT+02:00

  We are deploying an update to production that will hopefully resolve the connection issue.

• Open | Dec 01, 2022 | 15:19 GMT+02:00

  There is currently an issue with the websocket connection. We are investigating the reason behind the issue and are working on resolving it as soon as possible.
  
  We'll update here as we get more information.
  

Partial outage

Resolved | Nov 28, 2022 | 13:31 GMT+02:00

The partial outage is now over and all functions are back to normal. The root cause behind the issue has been identified and our developers are working on a more permanent solution to prevent similar outages in the future.

+Show history

• Open | Nov 28, 2022 | 13:14 GMT+02:00

  Howspace is experiencing a partial outage of certain features.
  
  Duplication, addition of users and AI features are experiencing slowness for the time being.
  
  Workspaces are still accessible and most features remain unaffected.
  
  We are investigating the root cause and working on a resolution.
  

Unexpected partial outage

Closed | Nov 17, 2022 | 16:07 GMT+02:00

The issue has been resolved and closed. It was due to a spike of activity which caused an overload to a specific feature in Howspace.

After identifying the specific feature behind the issue, steps have been taken to prevent any further incidents caused by large activity spikes.

Our server autoscaling setup allowed for a swift automatic resolution within 10-15 minutes after the issue began.

The separate issue with the status page has been resolved by the service provider. It was an unfortunate coincidence that both issues happened simultaneously.

Affected components

• Howspace (in.howspace.com)

+Show history

• Monitoring | Nov 17, 2022 | 10:43 GMT+02:00

  Most of the performance issues have been resolved but we are still monitoring the situation and investigating the root cause.
  
  The status page provider has an issue currently which might make the page inaccessible. They have been made aware of the issue and are working on resolving it.

• Open | Nov 17, 2022 | 10:17 GMT+02:00

  We are experiencing some unexpected performance issues.
  We are actively investigating the situation and will update the situation in 15 minutes.

Workspaces with FI and SV translations unavailable

Closed | Nov 16, 2022 | 08:10 GMT+02:00

The issue has been resolved and closed.

During planned maintenance, we encountered unexpected complications with the product due to corrupted FI and SV translations. The issue was in production from 07:42 EEST until 08:01 EEST. The issue made all workspaces with FI and SV UI language unavailable. After noticing the issue, corrective actions were made immediately.

We apologize for any inconvenience it might have caused and thank you for your patience and understanding.

Affected components

• Howspace (in.howspace.com)

+Show history

• Monitoring | Nov 16, 2022 | 08:03 GMT+02:00

  During planned maintenance, we encountered unexpected complications with the product due to corrupted FI and SV translations. The issue was in production from 07:42 EEST until 08:01 EEST. The issue made all workspaces with FI and SV UI language unavailable. After noticing the issue, corrective actions were made immediately.
  
  We apologize for any inconvenience it might have caused and thank you for your patience and understanding.

Unexpected outage

Closed | Sep 28, 2022 | 15:15 GMT+03:00

The issue has been resolved and closed.

We update Howspace many times a week. Sometimes it also includes database
maintenance. These are normal everyday operations.

We had detected the database performance could be improved by altering an
index. With the change in this update, sorting would be more efficient, and each
Howspace page would load faster.

However, this time, the wrong order of operations during database maintenance
caused a premature deletion of a database index. The old index was dropped
before the build of the new one was completed. The system was unresponsive
because of excessive query times during the period needed for the new index to
rebuild.

There was not much we could do but wait for the index rebuild to finish. We
tried to prevent further slow queries by letting the database finish
indexing. Once the new index was in place, everything started to work normally
again. Now, based on some initial checks, the pages load faster.

We're very sorry for the downtime. We learned this could happen, and we will
implement precautions for the future.

Affected components

• Howspace (in.howspace.com)

+Show history

• Resolved | Sep 28, 2022 | 10:41 GMT+03:00

  Howspace services is fully operational again.
  
  We will write and publish an incident report soon.
  

• Investigating | Sep 28, 2022 | 10:21 GMT+03:00

  Howspace workspaces are unavailable at the moment. The reason seems to be database re-indexing issue causing temporary failure to use the data.
  
  We are investigating the situation.
  
  

Temporary issue

Closed | Aug 26, 2022 | 14:11 GMT+03:00

The incident was swiftly resolved by our developer team once it was noticed. Steps have been taken to improve our internal alerts to prevent issues of this type in the future.

+Show history

• Resolved | Aug 12, 2022 | 09:21 GMT+03:00

  Steps have been taken that should resolve the issue. Howspace is up and running. We are monitoring the situation and will write and publish an incident report.

• Monitoring | Aug 12, 2022 | 08:10 GMT+03:00

  Steps have been taken that should resolve the issue. Howspace is up and running. We are monitoring the situation and will write and publish an incident report.

• Investigating | Aug 12, 2022 | 08:40 GMT+03:00

  The source of the issue has been identified. The problem is being actively worked on by our team.

• Open | Aug 12, 2022 | 07:55 GMT+03:00

  An issue has been identified that interferes with access to Howspace. Investigation into the issue has begun and our team is working on resolving the issue .

Single sign-on feature not available

Closed | Feb 01, 2022 | 00:28 GMT+02:00

During planned maintenance, there were unexpected complications, which made the Single sign-on feature unavailable to some customers. After noticing the issue, corrective actions were made.

We apologize for any inconvenience it might have caused and thank you for your patience and understanding.