The matching engine is back online and the metrics look better now. Verifying all fixes/upgrades
Posted Oct 26, 2021 - 18:14 UTC
Update
Redeploying our matching-engine (which doesn't rely on the database) which seemed to be having some latency issues processing messages
Posted Oct 26, 2021 - 18:11 UTC
Update
We are still seeing very slightly elevated latency in round-trip processing time for orders (on the order of 200ms). Will update when resolved
Posted Oct 26, 2021 - 18:01 UTC
Update
The upgrade looks successful but Cleanup is still ongoing. We will update here when latency looks normal again
Posted Oct 26, 2021 - 17:36 UTC
Verifying
Cleanup started. Summary Below:
10-15 Minutes: Cleanup - Orders and cancelations should start to proceed as normal again - We expect a small amount of slowness as the new database instance pages-in entries to its cache and background jobs tune the database tables
Posted Oct 26, 2021 - 17:12 UTC
Update
Failover started. Summary Below:
1-2 minutes: Failover and Service Restarts - The start time of this step is unknown and is triggered by the end of the database provisioning and upgrade mentioned above - The old instance will be taken down and new services will be changed to point to the new database instance. - Placing or canceling of orders will fail during this time - Triggerable (i.e. Stop-Limit or Take-Profit) orders will not trigger - Most other API requests will return 500 errors at this time
Posted Oct 26, 2021 - 17:11 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Oct 26, 2021 - 17:00 UTC
Update
Upgrade step started. Summary Below:
5-15 minutes: Upgrade - A new, upgraded database instance will be provisioned in the background - No service interruption or degradation at this time
Posted Oct 26, 2021 - 17:00 UTC
Update
Maintenance is still scheduled. A rough timeline of steps and what to expect starting at 17:00 UTC. The maintenance will be updated as we move through these steps.
5-15 minutes: Upgrade - A new, upgraded database instance will be provisioned in the background - No service interruption or degradation at this time
1-2 minutes: Failover and Service Restarts - The start time of this step is unknown and is triggered by the end of the database provisioning and upgrade mentioned above - The old instance will be taken down and new services will be changed to point to the new database instance. - Placing or canceling of orders will fail during this time - Triggerable (i.e. Stop-Limit or Take-Profit) orders will not trigger - Most other API requests will return 500 errors at this time
10-15 Minutes: Cleanup - Orders and cancelations should start to proceed as normal again - We expect a small amount of slowness as the new database instance pages-in entries to its cache and background jobs tune the database tables
Afterwards: Completion - Faster response times across the board when placing/canceling orders, especially during peak activity
Posted Oct 25, 2021 - 20:55 UTC
Update
We have moved the planned-maintenance day
From: Thursday, October 21 To: Tuesday, October 26
Posted Oct 20, 2021 - 19:02 UTC
Scheduled
We will be updating our database during this time. We expect about 2 minutes of downtime and possibly some general slowness.
Posted Oct 18, 2021 - 02:35 UTC
This scheduled maintenance affected: trade.dydx.exchange.