How to Monitor Cloud Migration and Data Transfer

Cloud migration is more than just a buzzword. According to several reports released at the beginning of 2019, almost 70% of enterprise organizations are moving their applications and infrastructure from local, self-managed hardware to one of the big cloud providers. Multiple case studies have been written about companies like Spotify, Dropbox, Gitlab, and Waze, all of which have replaced their core business infrastructures with cloud data centers. Migrating data from one data center to another is never simple, and the process is made especially difficult when the whole infrastructure has to be configured according to a set of principles and processes new to the DevOps staff. DevOps teams can find themselves even more challenged when they lose control over some, if not all, of the components the application depends on during the migration process. 

Failing to plan every last detail of the migration process can negatively impact a business and cause the loss of both money and customer trust. Monitoring the software and the migration process itself can prevent problems by allowing teams to detect changes caused by misconfigurations and discrepancies. This article will describe the considerations involved with monitoring cloud migration (such as best practices) and explain how monitoring can prevent future failures by revealing incorrect infrastructure setups early on.

Why Migrate Your Assets to the Cloud?

Migrating assets and resources from an on-premises infrastructure to a cloud provider’s infrastructure is a complicated and intense task. Often, big data requirements, application scale, or hardware refreshment processes prompt businesses to consider migrating from on-premises servers to cloud services. Just making the decision to perform the migration can be a long process, and it can and should involve many considerations. 

Migration Planning Considerations

When migrating assets to the cloud, security must be one of your top considerations. In a hybrid cloud migration process, assets are no longer hosted on the internal on-premises network. In order for an application to work well, the internal network must open some sort of connection to the outer world. This process invites security breaches. In a complete cloud infrastructure, security can be even more of a concern, since all of the company’s assets are under the provider’s security restrictions. 

Availability—both the system’s availability to customers and the infrastructure’s availability to R&D changes—is another important consideration for application backend deployment on cloud servers using managed services. As new network configurations are applied, the application might not be available in all of the regions in which it was previously. Moreover, when you use an external WAF or CDN provider, integration between that third-party provider and the cloud provider must be carefully inspected. Infrastructure and backend availability is also very important for deployment, investigation, and root cause analysis. Office and VPN access should be altered to work with the cloud provider environment according to relevant configurations and restrictions.

Scale and cost correlation should also be examined when considering cloud migration. It is much simpler to provision new services and instances in the cloud than on locally hosted hardware. In fact, some cloud providers offer automatic processes for provisioning which can impact the cost of cloud migration and usage. If they are not managed with clear thresholds and measurements, cloud migration costs can easily exceed their budgets, putting the whole migration in question. 

Adding monitoring to the migration process can simplify decision making around whether or not to perform various steps along the way. Monitoring can also help ensure that the process proceeds swiftly and correctly. 

What’s Monitoring Got to Do with It?

Monitoring minimizes risk. Every backend service and infrastructure component requires some level of monitoring in order to allow continuity, audit trail, and root cause analysis. During the various migration steps, different resources are being shut down in the on-premises environment and turned on (or having traffic redirected to them) in the cloud environment. There are several aspects of the migration process that need to be supervised and verified to ensure the desired results:

• Correct configuration. When starting the service in a different environment and a different infrastructure, all of the correct ports, permissions, and security rules should be configured to support access to the new service.
  • Availability. The outside world’s access to the application is obviously crucial and must be constantly validated. The response time of each service in the cloud infrastructure should be monitored as well.
• Billing. As new assets and components are spun up and managed services are put into action, the cost of running an application increases. R&D and FinOps (a position created for this exact purpose) teams need to pay close attention to projected costs during the migration process since unneeded resources will generate fees.
• Monitoring of managed services. Once the migration starts and some components are no longer under the R&D team’s control, monitoring should be set up to measure and track the functionality and performance of these components.

Tracking App Traffic

Tracking application traffic is the simplest way to validate both the proper configuration of all components and the application’s availability. Both during and after the migration, R&D teams must continuously verify that all requests arrive at the new servers, that error rate is maintained when compared with the previous deployment structure, and that current response times are similar to pre-migration response times without exceeding planned thresholds. This verification is done through service, web server, load balancer, and other infrastructure logs.

Another monitoring level can be performed on the API endpoint layer. R&D staff can compare the throughput of each of the pre-migration endpoints to the throughput of the new endpoints, and this data can provide insight into the current user experience and the usage of the new cloud infrastructure. Both R&D and IT teams can get detailed and accurate information about the application and about network usage by inspecting traffic from an internal point of view and by using another tool for the external inspection of the packets and headers. This combination provides the network utilization contemplation that can assist with making decisions about relevant network configurations. 

Tracking Costs

Tracking the costs of the migration steps, both during and after the migration process, is crucial. In some cases, cost savings is the primary motivation behind cloud migration. Incorrect usage of cloud services can lead to big losses and affect future plans for cloud usage. In order to verify that only required resources are being used and the cost of the migration process does not exceed the approved budget, sources like AWS Billing or Google’s billing system should be integrated, and their output should be part of the cloud monitoring system’s visualization. These tools allow you to display live information on predefined dashboards with alerts configured for every possible breach. Views like projected monthly costs and distribution of cost per cloud module are very helpful since they can immediately indicate incorrect or unconsented usage.

Application Security

Application security must also be monitored during the migration. All new components, managed or not, should be limited to permitted access only, and security restrictions should be applied to the environment as a whole. Monitoring is R&D’s only instrument for verifying the stability and strength of this configuration over time. The infrastructure configuration, including host open ports, internal traffic, and processes running on instances and accessing different services should be watched. Applications should also be monitored using metrics such as API hits per second and possible distributed denial of service attacks. Together, these monitoring procedures can help identify security incidents both during the migration process and after its completion. They can, therefore, help teams find out if the chosen infrastructure and configuration are correct and if they satisfy the security requirements of the application. Monitoring can also point out where DevOps staff should work to create higher standards of security and stability.

Conclusion

Cloud migration is a process that almost every company will go through at least once. High availability, scaling, and big data drive organizations to look for solutions that require less maintenance and improve efficiency. For many of them, the cloud is a very appealing option. However, the end goal should not automatically justify the means used to achieve it. The migration process should be thoroughly evaluated and planned. All of the steps in the process should be supervised, and strict boundaries should be enforced. Monitoring the migration process and its results can help to protect the R&D team and the company as a whole from unplanned downtime and unplanned expenses. This, in turn, can pave the way for cloud usage expansion and the improvement of day-to-day DevOps work.

Frequently Asked Questions