Back to overview
Degraded

Service Stability Incident

Oct 29 at 10:24pm CET
Affected services
Ricardoneud.com
Billing Portal
Jobs Center
URL Forwarder
API
Oauth2 System
Geo IP Database Update
Geo IP Daily Limits
Account Verification's
Account Deletion's
Session's
Email Subscription's
API Session Secret's
API Rate Limit's
API Request Logs
AI Check
Exchangerate's
Subscription's Checks
Connections
SSL Provisioning
SSL Verification
SSL Renewal
DNS Validation

Resolved
Oct 30 at 10:22pm CET

All services are now fully operational again. We believe the underlying issues have been resolved and do not expect them to reoccur.

Our team will continue to monitor system performance closely to ensure ongoing stability, but for now, everything appears to be functioning as expected.

Updated
Oct 30 at 10:11pm CET

We have just received confirmation from our hosting provider that the new server will be ready in a few minutes. Once it is online, we will start all services and verify that everything is running correctly.

We are very pleased with the progress and expect this to fully resolve the ongoing issues. We will continue to actively monitor all systems to ensure long-term stability and consistent performance.

Updated
Oct 30 at 09:46pm CET

The server switch process has started and is causing a brief service interruption. Our team is actively working on the migration together with our hosting provider to ensure a smooth transition.

Updated
Oct 30 at 08:59pm CET

We will be switching the server in a few minutes, which may cause brief service interruptions. We are actively monitoring to ensure that all processes restart automatically.

No data loss is expected, as we have created full backups in advance. While the possibility of data loss is extremely low, it can be restored from these backups if necessary.

Updated
Oct 30 at 07:29pm CET

All cronjobs have now been updated and are running internally. The server switch is scheduled to take place in approximately 1/2 hours.

During the migration, there may be a brief service interruption. This will not be reflected on the status page due to the prolonged nature of this incident. Updates regarding this transition will be posted under this ongoing incident.

Updated
Oct 30 at 06:30pm CET

We have temporarily paused operations due to server instability. A bug in the AMD Ryzen processor was causing occasional server freezes. To resolve this, we are migrating to an AMD Epyc server, which, according to our hosting provider, should prevent these issues.

This migration may cause a temporary interruption, after which we will continue migrating the cronjobs to run fully internally. We expect this will restore all services to their normal functionality.

Please note that users are not affected. The impact is limited to our backend systems, causing brief interruptions where scheduled tasks may not run precisely on time, but no service downtime occurs.

Updated
Oct 30 at 05:16pm CET

We are currently seeing recovery across the majority of our services. The only remaining areas affected are the API and custom domains with SSL, including SSL requests, which are still marked as Degraded.

Users are not impacted by this, but we still need to migrate the cronjobs internally to achieve full stability.

Updated
Oct 30 at 04:55pm CET

We are seeing positive impact from the internal cronjob tests. We will proceed with running all cronjobs internally to fully resolve these issues.

Our team is actively working on this process, and we will provide an update as soon as everything is completed.

Updated
Oct 30 at 04:40pm CET

Our technical team is currently working on and testing the proposed fix. We are hopeful for a positive outcome that will prevent these issues from recurring.

Please note that this does not impact service availability for users. The issue affects backend systems, which may occasionally fail temporarily but recover almost immediately.

Updated
Oct 30 at 08:51am CET

We have identified a potential fix by running our cronjobs internally, as it appears that only BetterStack requests are being blocked. Our plan is to migrate one cronjob to run internally as a test.

If the internal cronjob continues to run successfully and the heartbeats to BetterStack function correctly, we will proceed with migrating all remaining cronjobs.

There is currently no ETA for this process, as our technical team will only be available later this evening to perform the implementation.

Updated
Oct 29 at 11:53pm CET

We have identified the root cause of the issue — it is related to the DDoS protection system. Our hosting provider is actively implementing a fix to resolve the problem.

There is currently no estimated time of resolution (ETA), but mitigation efforts are in progress. Services may remain unstable until the fix is fully applied.

Updated
Oct 29 at 11:06pm CET

Our investigation has confirmed that the issue originates outside of our own systems. The root cause lies elsewhere, and we are continuing to work with our hosting provider and relevant partners to identify and fully resolve it.

We will maintain the current monitoring and keep the status as “Degraded” until the issue is completely fixed and the root cause has been clearly identified.

Updated
Oct 29 at 10:50pm CET

We are observing signs of recovery across our main services. Our hosting provider has made adjustments to their filtering systems, which appear to have mitigated the issue.

As a precaution, we will keep the service status set to “Degraded” for a short period in case the issue reoccurs, though we do not expect further disruptions. Our team will continue to closely monitor performance to ensure full stability.

Created
Oct 29 at 10:24pm CET

We are currently experiencing ongoing issues affecting our main services. Our technical team is actively investigating the situation in collaboration with our hosting provider to identify the underlying cause and ensure full stability.

For transparency, we have set the service status to “Degraded”, although this does not necessarily mean that the services are currently degraded. This status is a precautionary measure in case any temporary outages occur during our investigation period, as we are not able to provide continuous 24/7 monitoring.

Our hosting provider will also assist in monitoring to help identify the root cause of the issue. We will post an update as soon as we receive further information from our hosting provider.