Marking incident as resolved to indicate that we believe things are working properly again.
- We will continue to monitory closely and open up a StatusPage incident should something occur. - We will share a longer postmortem when we've collected more information. - Please reach out to our support if you have any questions about a particular account.
Posted Dec 08, 2021 - 02:07 UTC
Update
All markets set to ONLINE (from POST_ONLY). We are continuing to monitor
Posted Dec 08, 2021 - 02:00 UTC
Update
Planning to set markets to `ONLINE` from `POST_ONLY` shortly.
Posted Dec 08, 2021 - 01:56 UTC
Update
The backlog looks to be fully processed, we are verifying things look good before setting markets back into full-trading mode (out of post-only mode)
Posted Dec 08, 2021 - 01:51 UTC
Update
We estimate 10 minutes remain to work through the post-processing of the matching-engine. Orders, fills, and positions may continue to not reflect the exact state of the matching-engine during this time
Posted Dec 08, 2021 - 01:40 UTC
Monitoring
Another instance of the matching-engine was brought up and the backlog is being processed
Trade timestamps are the time post-processed, not the time that the matching engine processed the trade
Posted Dec 08, 2021 - 01:37 UTC
Investigating
Experiencing issues bringing up another instance of the matching-engine. Continuing to investigate
Posted Dec 08, 2021 - 01:24 UTC
Update
AWS took down our matching-engine. Bringing another instance up and investigating
Posted Dec 08, 2021 - 01:12 UTC
Update
The expected time to process the backlog is approximately 15 minutes.
You may start to see orders update during this time. There will still be feedback latency when placing/canceling new orders as the backlog is worked through
Posted Dec 08, 2021 - 01:04 UTC
Update
We working through post-processing a large backlog of messages from the matching-engine
Posted Dec 08, 2021 - 01:03 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 08, 2021 - 01:00 UTC
Update
ECS/Fargate are still experiencing issues (in our experience and according to AWS statuspage).
EC2 seems to be up with degraded performance and we are looking into moving critical services there temporarily
Posted Dec 07, 2021 - 23:35 UTC
Update
We can now view information on our services from AWS but are still blocked from making any changes.
Posted Dec 07, 2021 - 22:52 UTC
Update
Things seem to slowly be recovering from the AWS side. We will keep additional information posted here
The issue with our system was the post-processing of order placement and cancelations. However, as soon as any successful order-cancel request was made, it was sent to the matching engine. I.e. from a risk perspective, it was either immediately canceled or was already filled. There should be no need to keep sending requests to cancel, even if the order is still in PENDING or OPEN status and you have already sent a cancel request for that order.
However, we are still unable to display the result of the post-processing (including if the order was filled or not).
Due to the outage and to prevent further issues, we expect that the post-processing will proceed "as normal", i.e. we cannot guarantee that all OPEN or PENDING orders are without fills until the post-processing is fixed. However, to reiterate, if you noticed an issue with the system and immediately sent cancel order requests for orders, then those requests were processed instantly by the matching-engine; they are no longer on the matching-engine's orderbook.
Posted Dec 07, 2021 - 22:06 UTC
Update
We are setting all markets to post-only mode. Orders placed going forwards must be post-only.
Orders that were placed (into PENDING) before this point may still execute as a taker order.
The post-processing of orders is delayed so it's hard to say whether orders are filled or canceled. We are looking into whether we can just have all of those orders cancel, however the AWS outage is preventing us from currently being sure if that's possible.
Posted Dec 07, 2021 - 21:20 UTC
Update
Due to an AWS outage, we are experiencing greater latency across services and impaired functionality with endpoints not working and the website not loading
Posted Dec 07, 2021 - 17:57 UTC
Update
Due to an AWS outage, we are experiencing greater latency across services and impaired functionality with endpoints not working and the website not loading
Posted Dec 07, 2021 - 17:32 UTC
Update
We are continuing to investigate this issue.
Posted Dec 07, 2021 - 17:32 UTC
Update
We are continuing to investigate this issue.
Posted Dec 07, 2021 - 17:31 UTC
Update
We are continuing to investigate this issue.
Posted Dec 07, 2021 - 17:29 UTC
Investigating
Due to an AWS outage, we are experiencing greater latency across services and impaired functionality with endpoints not working