Emergency maintenance Ceph storage cluster (extended)

Emergency maintenance Ceph storage cluster (extended)

17-01-2020 17:09:37 - 21-01-2020 07:00:00

Subject: Emergency maintenance Ceph storage cluster
Urgency: Emergency
Affected services:
- Virtual machines
- Shared storage
- Shared hosting
- MSSQL instances
Expected impact:
- Multiple times short hindrance in performance of storage
Customer intervention required: No
Reference number: 166590

Summary:

Due to a recent change to our storage platform, we have noticed that the platform no longer performs optimally since then. To gain more insight, we need to make a number of changes.

Details:

During the maintenance of our storage platform on 19-12-2019, we moved meta-data from SSD drives to faster NVMe drives. We have therefore made a number of changes to our configuration and since then we have noticed that the platform no longer performs optimally. We will make a number of changes during this new maintenance to gain more insight into the issues surrounding this sub-optimal performance. These changes may also immediately resolve the problem. To ensure that the impact of these changes remains as low as possible, the duration of the maintenance is probably not going to be short. We foresee no problems with redundancy during this maintenance. It may be that the performance of the storage platform will temporarily be less good. We therefore keep a close eye on the situation and will intervene where possible to limit the inconvenience to customers as much as possible.
Update placed at 20/01/2020, 09:11
Following this change a lot of data needs to be moved. Because moving of the data is not ready yet we are forced to extend this maintenance. To limit the impact on the performance to the storage platform we have decided to stop moving the data and resume after 19:00.