Autoresize EBS root volume on Linux-based Amazon AMIs

By
Dimitrios Liappis

I can’t increase the size of my EBS root volume when I launch a new Debian instance!

Setting a larger than default (8GiB) root EBS device, when launching Debian AMIs, does not result in a similarly sized root filesystem.

In this article we are investigating why this doesn’t work out of the box, and how to create new AMIs that will be resized correctly after launch.

Example:

You want to launch a new Debian instance with the root device on Amazon EBS.

So you select an appropriate root EBS AMI, for example: Debian-jessie

In the “Add Storage” step, change the default size from 8 GiB to something larger; 33 GiB in this example:

DebianAMI_launch_instance_rootsize.png

After the instance has come up, ssh using your keypair1 and your root filesystem is still showing 8GiB: 

admin@ip-172-31-2-65:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  641M  6.8G   9% /


Where did my space go?

Digging a bit deeper with lsblk, we can see: 

admin@ip-172-31-2-65:~$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  33G  0 disk
└─xvda1 202:1    0   8G  0 part /

The Amazon EBS disk (/dev/xvda) presented to the OS has the right size: 33G

However the partition where the OS resides (/dev/xvda1) is stuck at 8G.

Linux AMIs come preinstalled with a customized version of the cloud-init package. This includes Python modules that can resize partitions and filesystems, if CLI tools like growpart and gpart are present. On standard Debian AMIs, they aren’t there.

CentOS AMIs/EPEL in comparison have the cloud-utils-growpart rpm package available providing the growpart tool.

cloud-init tools will also detect if the root file system (/) size differs from the partition size and utilize the appropriate filesystem expansion tool (e.g. resize2fs for ext4) to match the partition size.

Resizing an online partition on a linux AMI

Naturally we would like our AMIs to resize the root partition by themselves, to use all available space.

Since growpart is not available in Debian AMIs, we can accomplish this with parted and an init script that runs on the first boot (when the instance gets launched).

Making parted work non-interactively

One challenge here is that most tools will complain about resizing the partition containing a mounted [root] file system.

The version of parted shipped with Debian jessie allows us to grow a partition interactively and -- albeit not so well documented -- non interactively as well.

Normally, a command like this would suffice, but because the partition is in use, parted prompts for confirmation which would interrupt the flow in a script: 

root@ip-172-31-2-65:~# /sbin/parted /dev/xvda resizepart 1 100%
Warning: Partition /dev/xvda1 is being used. Are you sure you want to continue?
/sbin/parted: invalid token: 100%
Yes/No?

Luckily parted has an undocumented cli parameter that allows us force grow the partition without prompts: 

root@ip-172-31-2-65:~# /sbin/parted ---pretend-input-tty /dev/xvda resizepart 1 yes 100%
Warning: Partition /dev/xvda1 is being used. Are you sure you want to continue?
Information: You may need to update /etc/fstab.

The output may not be very encouraging, but it worked: 

root@ip-172-31-2-65:~# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  33G  0 disk
└─xvda1 202:1    0  33G  0 part /

The only thing remaining is to resize the filesystem. This will be done automatically by the cloud-init tools after a reboot or by running resize2fs /dev/xvda1

Putting it all together

If you are building your own custom AMI, it makes sense to add an init script that does the above steps immediately after a new instance gets launched and ensures it won’t rerun itself again.

An important requirement is to have parted installed; you can either do that inside the script or ensure parted gets preinstalled as part of the scripts building your custom AMI.

For Debian AMIs running the following script will create an init script linked to /etc/rc5.d/S05ami-resizerootpart.sh, that will take care of all the resizing. As the last step it will deactivate itself and report activities under /var/log/resizerrootfs.log

Are other AMIs affected by this problem?

Yes! CentOS6 AMIs are affected. Instead of parted though -- which is an old version that can’t resize online partitions -- make sure growpart is installed from the package cloud-utils-growpart. This package is part of the EPEL6 repo. The resize command is:

 /usr/bin/growpart /dev/xdva 1

Finally SLES11.x AMIs seem to suffer from the same issue.

Conclusion

Some Linux AMIs will fail to utilize the whole EBS volume size you've chosen when you launch them!

If you have been bitten by this issue, I'd love to hear about it and whether the ideas presented here were of help.

A few tips to keep in mind:

  • Use packer.io to build your custom AMIs.
  • Prefer growpart if it is available in your target distribution as it was built precisely for that purpose.
  • The cloud changes all the time and AMIs are no exception! The above issues may go away with newer releases.


Script for Debian AMIs

#!/bin/bash
# Run this to create init script for Debian 8 AMIs that autoresizes the root EBS partition
ROOTDEV="xvda"
ROOTPART="xvda1"
rootdevsize=$(blockdev --getsize /dev/${ROOTDEV})
rootpartsize=$(blockdev --getsize /dev/${ROOTPART})
if ! grep -q ${ROOTPART} /proc/partitions
then
    # EBS volume is unpartitioned ... no work for us
    exit 0
fi
source /etc/os-release
if [ $VERSION_ID -gt 8 ]
then
    # Exclude future Debian AMIs that hopefully will have this fixed
    exit 0
fi
cat >/etc/init.d/ami-resizerootpart.sh <<'EOF'
#! /bin/sh
### BEGIN INIT INFO
# Provides:          ami-resizerootpart
# Required-Start:    $local_fs $syslog
# Required-Stop:
# Should-Start:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Resize root partition on AWS if it smaller than the size of the device
# Description:       When using the Debian 8 AWS AMIs with root EBS fs, EBS is partitioned.
#                    As a result, when launching a new instance with a larger than default
#                    root EBS size, /dev/xvda has desired size but not /dev/xvda1 root
#                    partition.
#                    /etc/init.d/ami-resizerootpart detects this discrepancy and resizes
#                    the partition, triggering a reboot and fsck. cloud-init will resize the
#                    fs by itself after the reboot.
### END INIT INFO
. /lib/init/vars.sh
. /lib/init/mount-functions.sh
. /lib/lsb/init-functions
PATH=/sbin:/bin:/usr/sbin:/usr/bin
do_start() {
    ROOTDEV="xvda"
    ROOTPART="xvda1"
    LOGFILE="/var/log/resizerootfs.log"
    rootdevsize=$(blockdev --getsize /dev/${ROOTDEV})
    rootpartsize=$(blockdev --getsize /dev/${ROOTPART})
    if ! grep -q ${ROOTPART} /proc/partitions
    then
        # EBS volume is unpartitioned ... no work for us
        echo "Unpartitioned EBS volume no need to run" >>$LOGFILE
        return 0
    fi
    # Check if raw EBS device and partition differ by more than 10MB
    if [ ! $(($rootdevsize-$rootpartsize)) -ge 10485760 ]
    then
       # No need to resize
        echo "EBS partition is <=10MB of the device size; no need to act" >>$LOGFILE
       return 0
    fi
    # Resize root partition
    /sbin/parted ---pretend-input-tty /dev/${ROOTDEV} resizepart 1 yes 100%
    if [ ! $? -eq 0 ]
    then
        echo "Resize failed" >>$LOGFILE
        return 0
    fi
    echo "Successfully resized rootfs!" >>$LOGFILE
    # Force fsck on next reboot
    /usr/bin/touch /forcefsck
    # Disable myself job done
    rm /etc/rc5.d/S05ami-resizerootpart.sh
    sync
    reboot
}
case "$1" in
  start|"")
        do_start
        ;;
  stop|status)
        # No-op
        ;;
  *)
        echo "Usage: ami-resizerootpart.sh [start|stop]" >&2
        exit 3
        ;;
esac
EOF
chmod 755 /etc/init.d/ami-resizerootpart.sh
chown root:root /etc/init.d/ami-resizerootpart.sh
ln -s /etc/init.d/ami-resizerootpart.sh /etc/rc5.d/S05ami-resizerootpart.sh

1. ssh as admin on Debian AMIs. Back to text