My Data Science Weekend: The Power of Using the Right Skills for the Job

Never Underestimate the Power of the Right Tools

I used to have a friend at university who would always say:

"Never underestimate the power of using the right tools for the job."

At the time, it sounded like simple advice, but over the years, I have come to appreciate just how true it is.

Imagine trying to slice a loaf of bread using a butter knife instead of a bread knife. The task itself is straightforward, but suddenly it becomes slower, more frustrating, and unnecessarily difficult. The problem isn't the bread—it's that you're using the wrong tool.

Data migration works in much the same way.

Moving data from one system to another is conceptually simple. Yet without the right skills, architecture, and tooling, it can quickly become an overwhelming exercise.

As a software developer, I've been in rooms where teams were effectively trying to cut bread with a butter knife. I've seen ETL (Extract, Transform and Load) processes become far more complicated than they needed to be.

Some of the most common mistakes I've encountered include:

Manually transferring data using JSON objects and relying on visual confirmation to verify success.
Creating inconsistent mappings between source and destination systems.
Spending hours manually moving large datasets that should have been processed automatically.

This weekend, I experienced first-hand just how transformative the right knowledge and tools can be.

Using a cloud-native ETL architecture, we successfully migrated more than 300,000 MobileGPT user records, together with multiple related tables, across two cloud platforms, two databases, and two completely different application architectures—in a single weekend.

Most importantly, the migration was completed securely, reliably, and without data loss.

The Problem We Were Facing

This weekend, we migrated our MobileGPT platform (https://mobile-gpt.io/) from DigitalOcean to Amazon Web Services (AWS).

Although the objective sounded simple, the migration itself was anything but.

Our previous platform was built as a monolithic application running on DigitalOcean Droplets.

The new platform was designed as a microservices architecture running on Amazon ECS, supported by Amazon RDS, Amazon ElastiCache, and a carefully planned data lifecycle that archives historical information into Amazon S3 for future analytical workloads.

This meant we weren't simply moving data.

We were moving it into an entirely different architecture.

Three major challenges immediately became apparent:

Data volume
Structural differences
Security

Data Volume

MobileGPT can onboard thousands of new users in a single day.

As a result, the production database had grown considerably.

Our previous attempts at migrating this data manually were unsuccessful.

Large migration jobs regularly timed out, forcing us to split the data into much smaller batches. Even then, each batch required more than five hours to complete the full ETL process.

At that rate, a complete migration simply wasn't practical.

Differences in Data Structure

The database schemas had evolved significantly.

Throughout the redesign of the platform:

Existing models had been improved.
Some fields had been renamed.
New entities had been introduced.
Other models had been removed entirely.

This meant the migration required transformation, not simply copying records from one database into another.

We needed application-specific transformation logic capable of understanding both database structures while validating every migrated record.

With more than 300,000 primary records and numerous related tables, manually verifying every row was no longer realistic.

The solution had to verify itself.

Security

Because the migration involved moving customer information between cloud providers, security became a top priority.

Some records contained personally identifiable information, including users' phone numbers.

The ETL pipeline therefore needed to ensure that data remained:

Encrypted in transit
Encrypted at rest
Fully traceable throughout the migration process

Security could never become a trade-off for speed.

You Cannot Know What You Do Not Know

Looking back, I realise that when I first approached this migration, I simply didn't know what I know today.

Like many developers, my first instinct was to attempt the migration manually.

It failed.

At one point, I even considered abandoning the historical data entirely and starting from scratch in the new environment.

Fortunately, I had recently completed a Data Science certification.

While studying both the certification material and AWS documentation, I developed a much deeper understanding of ETL architecture, cloud-native design, and scalable data pipelines.

That learning completely changed how I viewed the problem.

Certification exams often present multiple technically correct solutions.

The challenge is choosing the most efficient, reliable, secure, and scalable solution—not merely one that works.

Once I understood that principle, the migration architecture became obvious.

The problem hadn't changed.

My approach had.

Solution Requirements

Before building anything, I defined what success looked like.

The migration solution needed to:

Move large volumes of data efficiently.
Programmatically validate every migrated record.
Encrypt all data both in transit and at rest.
Run for as long as necessary without manual intervention.
Generate detailed audit reports for every migration batch.
Allow all temporary migration infrastructure to be removed afterward.

Only once these requirements were clear could the architecture be designed.

ETL Architecture

Building the Transformation Engine

The heart of the solution was a custom transformation engine.

This application-specific component understood both the source and destination database structures.

It knew:

Which fields had been renamed.
Which fields no longer existed.
Which new fields needed to be created.
How relationships between tables should be preserved.

Beyond transforming data, the engine also validated every migration.

It detected skipped rows, handled transformation errors, generated detailed audit reports, and verified that every batch had completed successfully.

Instead of relying on manual inspection, the migration validated itself.

Orchestrating the Pipeline

The overall ETL workflow was orchestrated using AWS Step Functions.

Each migration followed a controlled sequence:

Receive the migration file.
Transform the data.
Validate the records.
Load the transformed data into the destination database.
Generate audit reports.
Notify the team of the migration outcome.

After each execution, detailed reports were automatically shared with the team.

Monitoring and troubleshooting were handled through Amazon CloudWatch Logs and AWS X-Ray, providing complete visibility into every stage of the migration.

The Right Tools for the Job

AWS CDK and Infrastructure as Code

Because the migration infrastructure was temporary, I didn't want to leave unnecessary cloud resources running after the project had finished.

Using the AWS Cloud Development Kit (AWS CDK), I defined the entire migration environment as code and deployed it through AWS CloudFormation.

Once the migration was complete, the temporary infrastructure could be removed cleanly while retaining only the persistent Amazon S3 storage required for long-term use.

Event-Driven Cloud Architecture

Rather than keeping servers running continuously, we adopted an event-driven approach.

Amazon S3 acted as the secure landing zone for migration files.

Whenever a file was uploaded, an event automatically triggered the ETL pipeline.

No manual intervention was required.

The system simply responded whenever new work became available.

On-Demand Compute Resources

Processing was handled by Amazon ECS.

Instead of paying for servers that sat idle waiting for work, compute resources started automatically when migration files arrived and shut down once processing had completed.

This made the solution both efficient and cost-effective, allowing compute capacity to scale with demand rather than remain permanently allocated.