Skip links

How We Reduced Deployment Time by 80% with Automated Pipelines

In early 2023, deploying a WordPress site at Harbor Software took an average of 47 minutes. That included SSH-ing into the server, pulling the latest code from the Git repository, checking for file permission issues that seemed to appear randomly, running database migrations while holding your breath, clearing multiple layers of cache, testing a handful of critical pages manually in the browser, and posting in Slack that the deployment was done. By September 2023, the same deployment took 8 minutes, fully automated from git push to live site, with zero manual intervention and zero finger-crossing. The 80% reduction was not the result of a single clever tool or a magical CI/CD platform. It was the result of systematically identifying every manual step in our deployment process and replacing it with a scripted, tested, and monitored equivalent.

Article Overview

How We Reduced Deployment Time by 80% with Automated Pipe…

7 sections · Reading flow

01
The Before: Death by Manual Deployment
02
The Architecture: GitHub Actions + WP-CLI +…
03
Staging Environment: The Safety Net
04
Database Migrations: The Hard Part
05
Monitoring, Alerting, and Rollback Strategy
06
Results and Lessons Learned
07
Cost Analysis: What This Actually Takes to Build

HARBOR SOFTWARE · Engineering Insights

This is the story of that transformation: the problems we faced, the architecture we built, the tools we chose, and the mistakes we made along the way that you can avoid.

The Before: Death by Manual Deployment

Our deployment process in early 2023 looked like this, and if you run a small WordPress agency, yours probably looks uncomfortably similar:

  1. Engineer finishes feature branch, merges PR to main on GitHub
  2. Engineer SSHs into the production server using their personal credentials
  3. Navigates to the WordPress directory and runs git pull origin main
  4. Checks for file permission issues (happened approximately 20% of the time)
  5. Runs wp plugin update --all if plugin updates were queued
  6. Runs database migrations if there were schema changes in the release
  7. Clears LiteSpeed Cache from the command line
  8. Logs into Cloudflare dashboard and purges the CDN cache manually
  9. Manually visits the homepage, a product page, and the checkout in a browser to verify nothing is visually broken
  10. Posts in the team Slack channel: “Deployed to production. Looks good.”

The problems with this process were not theoretical concerns raised in a code review. They were actively costing us real time and causing real production incidents that affected client sites:

  • Step 4 (file permissions) wasted 5-10 minutes every time it failed, which was roughly one in five deployments. The root cause was that git pull sometimes created new files owned by the SSH user rather than the web server user, causing PHP to throw permission denied errors on those specific files. The fix was always the same (chown -R www-data:www-data) but diagnosing which files were affected varied each time.
  • Step 6 (database migrations) was the scariest step because we were running migrations on a live production database with active customer sessions and no automated rollback plan. If a migration failed halfway through, and this happened twice in six months, the database was left in an inconsistent state with some tables modified and others not, requiring manual intervention at 11 PM on a Tuesday.
  • Step 9 (manual testing) caught problems only about 30% of the time based on our incident log. The engineer doing the deploy was checking three pages out of hundreds. A broken product filter on category pages, a misaligned sidebar on the blog archive, or a JavaScript error on the contact form could go completely unnoticed for days until a client or customer reported it.
  • Step 10 (Slack notification) was forgotten approximately 40% of the time, leaving the rest of the team unaware that production had changed, which led to confusion when debugging issues that only appeared after a deployment nobody knew about.

We also had a serious bus-factor problem. Only two engineers on the team knew the full deployment procedure, including the specific server paths, the correct order of cache clearing, and the workarounds for known issues. If both were unavailable (vacation, sick, different timezone), deployment stalled and urgent client fixes waited. That is completely unacceptable for client sites that depend on timely updates.

The Architecture: GitHub Actions + WP-CLI + Health Checks

Our deployment pipeline runs on GitHub Actions and executes commands on the production server via SSH. The architecture is deliberately simple and uses only free, well-documented tools because complexity in deployment infrastructure is a liability, not an asset. Every additional tool in the pipeline is another thing that can break, another thing to learn, and another vendor dependency.

# .github/workflows/deploy-production.yml
name: Deploy to Production

on:
  push:
    branches: [main]

env:
  SERVER_HOST: ${{ secrets.PROD_HOST }}
  SERVER_USER: ${{ secrets.PROD_USER }}
  WP_PATH: /home/harborso/public_html

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup SSH key
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.SSH_PRIVATE_KEY }}" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key
          ssh-keyscan -H $SERVER_HOST >> ~/.ssh/known_hosts

      - name: Pre-deploy database backup
        run: |
          ssh -i ~/.ssh/deploy_key $SERVER_USER@$SERVER_HOST "
            cd $WP_PATH && 
            wp db export ~/backups/pre-deploy-$(date +%Y%m%d-%H%M%S).sql --quiet && 
            echo 'Database backup complete'
          "

      - name: Deploy files via rsync
        run: |
          rsync -avz --delete 
            --exclude='.git' 
            --exclude='wp-config.php' 
            --exclude='wp-content/uploads' 
            --exclude='.env' 
            --exclude='wp-content/cache' 
            --exclude='wp-content/upgrade' 
            --exclude='backups/' 
            -e "ssh -i ~/.ssh/deploy_key" 
            ./wp-content/ $SERVER_USER@$SERVER_HOST:$WP_PATH/wp-content/

      - name: Post-deploy cache flush and optimization
        run: |
          ssh -i ~/.ssh/deploy_key $SERVER_USER@$SERVER_HOST "
            cd $WP_PATH && 
            wp cache flush && 
            wp rewrite flush && 
            wp elementor flush-css 2>/dev/null; 
            wp cron event run --due-now && 
            echo 'Post-deploy commands complete'
          "

      - name: Health check - verify site responds
        run: |
          sleep 5
          HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' https://harborsoftware.com)
          if [ "$HTTP_CODE" != "200" ]; then
            echo "HEALTH CHECK FAILED: HTTP $HTTP_CODE"
            exit 1
          fi
          echo "Health check passed: HTTP $HTTP_CODE"

      - name: Notify team via Slack
        if: always()
        run: |
          STATUS=${{ job.status }}
          COMMIT_MSG=$(git log -1 --pretty=%B)
          AUTHOR=$(git log -1 --pretty=%an)
          curl -X POST ${{ secrets.SLACK_WEBHOOK }} 
            -H 'Content-Type: application/json' 
            -d "{"text": "Deploy to production: $STATUSnAuthor: $AUTHORnCommit: $COMMIT_MSG"}"

Every step in this pipeline directly replaces a manual step from our old process, with measurable improvements in reliability and speed:

  • Pre-deploy backup creates a database snapshot before any changes touch the server. If anything goes wrong during or after deployment, we can restore the database in under 2 minutes with wp db import. This replaced the terrifying practice of deploying without a safety net.
  • rsync with exclusions solves the file permission problem permanently. Files are transferred as the deploy user, who owns the web directory. No more git pull creating files with wrong ownership. The --delete flag ensures that files removed from the repository are also removed from the server, preventing orphaned files from accumulating.
  • Health check verifies the site is responding with HTTP 200 after deployment. This is a basic smoke test, not comprehensive, but it catches the most catastrophic failures: a fatal PHP error, a broken database connection, or a misconfigured web server. If it fails, the entire pipeline is marked as failed and the Slack notification shows the failure prominently.
  • Slack notification with if: always() runs regardless of whether the deployment succeeded or failed. The team always knows when a deployment happened, who triggered it, what was deployed, and whether it succeeded. No more forgotten notifications.

Staging Environment: The Safety Net

The automated pipeline was only safe to build and trust because we first established a proper staging environment. Without staging, automating deployment to production is automating the delivery of untested code to real users. You are not reducing risk; you are accelerating it.

Our staging setup mirrors production as closely as possible: same PHP version (8.2), same MySQL version (MariaDB 10.11), same plugins at the same versions, same theme. The only differences are the domain (staging.harborsoftware.com), the database content (a nightly automated copy from production), and the search engine visibility setting (blog_public = 0 to prevent Google from indexing staging content).

We maintain a separate GitHub Actions workflow for staging that triggers on push to the develop branch. The staging workflow is identical to production except it includes an additional step: automated visual regression testing.

# Visual regression testing on staging (not run on production)
- name: Visual regression test
  run: |
    npx backstopjs test --config=backstop.config.js
  env:
    BACKSTOP_BASE_URL: https://staging.harborsoftware.com

BackstopJS captures screenshots of 12 key pages at three viewport widths (mobile 375px, tablet 768px, desktop 1440px), resulting in 36 screenshot comparisons per deployment. Each screenshot is compared pixel-by-pixel against a reference image from the previous known-good deployment. A regression threshold of 0.1% flags visual changes that exceed normal rendering variation. This catches CSS regressions, layout shifts from updated plugins, missing elements, and broken responsive behavior that a quick manual check would miss entirely.

Database Migrations: The Hard Part

File deployment is the easy part of automation. Database migrations are the hard part because they are inherently risky, potentially destructive, and difficult to roll back cleanly. A dropped column cannot be un-dropped. A data transformation that changes values in place cannot be un-transformed without a backup. A migration that runs for 10 minutes on a large table locks that table for the duration, causing timeouts on the live site.

We solved this with a migration framework inspired by Laravel’s migration system, adapted for WordPress’s architecture:

<?php
// migrations/2024_01_15_add_product_custom_fields.php
// Each migration has three methods: up, down, and verify

class Migration_2024_01_15_Add_Product_Custom_Fields {

    public function up() {
        global $wpdb;
        $products = $wpdb->get_col(
            "SELECT ID FROM {$wpdb->posts}
             WHERE post_type = 'product'
             AND ID NOT IN (
                SELECT post_id FROM {$wpdb->postmeta}
                WHERE meta_key = '_custom_warranty_period'
             )"
        );

        foreach ($products as $product_id) {
            update_post_meta($product_id, '_custom_warranty_period', '12');
            update_post_meta($product_id, '_custom_warranty_unit', 'months');
        }

        return count($products) . ' products updated with warranty fields';
    }

    public function down() {
        global $wpdb;
        $wpdb->query(
            "DELETE FROM {$wpdb->postmeta}
             WHERE meta_key IN ('_custom_warranty_period', '_custom_warranty_unit')"
        );
        return 'Warranty fields removed from all products';
    }

    public function verify() {
        global $wpdb;
        $meta_count = $wpdb->get_var(
            "SELECT COUNT(DISTINCT post_id) FROM {$wpdb->postmeta}
             WHERE meta_key = '_custom_warranty_period'"
        );
        $product_count = $wpdb->get_var(
            "SELECT COUNT(*) FROM {$wpdb->posts}
             WHERE post_type = 'product' AND post_status = 'publish'"
        );
        return intval($meta_count) === intval($product_count);
    }
}

Every migration has three methods: up() applies the change, down() reverses it completely, and verify() confirms the migration produced the expected result. The deployment pipeline runs migrations in chronological order, calls verify() after each one, and aborts the entire deployment if verification fails. A failed migration triggers an automatic down() call followed by a database restore from the pre-deploy backup as a safety net. This two-layer protection (reversible migration plus backup restore) has prevented every potential database incident since we implemented it.

Monitoring, Alerting, and Rollback Strategy

An automated pipeline without monitoring is dangerous because it can deploy broken code faster than a manual process can. The speed advantage becomes a liability if you do not know when something goes wrong. Our monitoring stack ensures we know about problems within minutes, not hours:

  • Uptime monitoring (UptimeRobot): Checks the homepage every 60 seconds from multiple geographic locations. Alerts via Slack and SMS if the site returns a non-200 status for more than 2 consecutive checks. Cost: $7/month for 50 monitors, covering all client sites.
  • Error tracking (custom WP-CLI script): A cron job runs every 5 minutes, tails the PHP error log for new entries since the last check, and posts critical-level errors to a dedicated Slack channel. This catches fatal errors, deprecated function warnings from updated plugins, and memory exhaustion issues that do not cause a full outage but break specific features.
  • Performance monitoring (PageSpeed API): A weekly GitHub Action runs Google PageSpeed Insights tests on 5 key pages and records the scores in a spreadsheet. If any score drops below 75 compared to the previous week, it creates a GitHub issue automatically for investigation.
  • SSL certificate monitoring: A monthly check verifies certificate expiry is more than 30 days away. SSL expiry causes immediate browser warnings that destroy user trust and is entirely preventable with basic monitoring.

Our rollback strategy operates at three levels for different failure types:

Level 1: Code rollback. Git revert the problematic commit and push to main. The pipeline triggers automatically and deploys the reverted code within 8 minutes. This handles the most common failure: code that works on staging but breaks in production due to environment differences or timing issues.

Level 2: Database rollback. Import the pre-deploy SQL backup. Time: 1-2 minutes for databases under 500MB. This handles migration failures and data corruption.

Level 3: Full restore from off-site backup. When both code and database need to be restored to a known-good state. We keep 7 daily backups and 4 weekly backups on Amazon S3. Time: 5-10 minutes. We practice this restore quarterly on a test server to verify the backups are functional, because an untested backup is not a backup. It is a hope.

Results and Lessons Learned

After six months of running the automated pipeline across all client projects, here are the measurable outcomes compared to the previous six months of manual deployment:

  • Deployment time: 47 minutes average down to 8 minutes average (83% reduction)
  • Deployment frequency: Increased from 2-3 times per week to 8-12 times per week. We deploy smaller changes more frequently because the cost of each deployment is near zero.
  • Failed deployments: 3 failures in 6 months (versus 7 in the previous 6 months), all three caught by the health check and rolled back automatically before any user was affected
  • Time-to-rollback: Reduced from 15-30 minutes of panicked manual work to under 2 minutes of automated database restore
  • Bus factor: Expanded from 2 specific engineers who knew the process to anyone on the team who can merge a pull request to the main branch

The biggest lesson from this entire project: start with the backup automation. Before automating any other part of the deployment process, automate your backup creation and verify that you can restore from those backups reliably. Once you have tested, automated backups that you trust, every other automation step becomes dramatically safer because you always have a fallback. We spent our first two weeks of this initiative solely on backup automation and restore testing, and that investment de-risked every subsequent step.

The second lesson: do not automate the entire pipeline in one sprint. We added one step per week to the pipeline, tested that step on the staging environment for a full week of real deployments, then promoted it to the production pipeline. This incremental approach meant that when something went wrong (and it did, three times during the buildout), we knew exactly which step caused the failure because it was the only new addition.

The third lesson that is less obvious but equally important: invest in your staging environment before your pipeline. Our staging server costs $25/month on DigitalOcean, which feels like an unnecessary expense when you are a small agency watching every dollar. But that $25/month server prevented at least four production incidents during our pipeline buildout that would have cost us hours of downtime and significant client trust. The staging environment is not an expense; it is insurance with a 100x return on the first prevented incident.

A fourth lesson we learned through experience: do not optimize for deployment speed at the expense of deployment safety. Our initial pipeline design aimed for the fastest possible deployment by running steps in parallel. The backup ran simultaneously with the file transfer to save 30 seconds. Then one day the file transfer completed before the backup finished, the new code caused a database error, and we could not roll back because the backup was still being written. We immediately reverted to sequential execution where every safety step completes before the next step begins. The pipeline takes 8 minutes instead of 6, and it has never left us without a working rollback.

We also learned to separate the deployment mechanism from the deployment decision. Our pipeline triggers automatically on merge to main, but merging to main requires a pull request with at least one approval. This means the deployment automation is fast and reliable, but the decision to deploy is still made by a human who has reviewed the code and considered the timing. We do not deploy at 5 PM on Friday. We do not deploy during peak traffic hours. These timing decisions are made at the PR merge stage, not encoded into the pipeline, because the appropriate deployment window varies by project and client.

Cost Analysis: What This Actually Takes to Build

For agencies considering this transition, here is an honest accounting of what the investment looks like. The tools themselves are free or nearly free: GitHub Actions provides 2,000 minutes per month on the free tier, WP-CLI is open source, rsync is installed on every Linux server, and BackstopJS is open source. The real cost is engineering time to build, test, and iterate on the pipeline.

Our total investment was approximately 60 engineering hours spread over 8 weeks. That breaks down as: 12 hours for backup automation and testing, 8 hours for the rsync deployment script with proper exclusions, 6 hours for the GitHub Actions workflow configuration and secret management, 10 hours for the health check and Slack notification integration, 16 hours for the staging environment setup and visual regression testing, and 8 hours for documentation, team training, and runbook creation. At our internal cost rate, that is roughly $5,000-$7,000 in engineering time.

The return on that investment: 39 minutes saved per deployment multiplied by an average of 10 deployments per week equals 6.5 hours saved per week. At $100/hour billable rate, that is $650/week or $33,800 per year in recovered productive capacity. The pipeline paid for itself in the first month and will continue generating returns every week indefinitely. For any agency deploying WordPress sites more than twice a week, the ROI case is overwhelming.

Automated deployment is not a luxury reserved for teams building JavaScript applications on modern platforms. It works for WordPress. It works for WooCommerce stores processing real orders. It works for sites running on traditional hosting with SSH access. The tools are free (GitHub Actions, WP-CLI, rsync, BackstopJS), the patterns are straightforward and well-documented, and the return on investment is measured in hours saved per week, every week, compounding forever.

Leave a comment

Explore
Drag