Online Resizing of a Boot Partition on RHEL 6 or 7 with parted.
The Problem:
I've been getting away with using a 200MB boot partition for over a decade, but RHEL 7 has started pushing the limits of that scheme. Between GRUB2 and the automatic creation of a rescue kernel and initramfs that's twice the size as a normal initramfs, that 200MB gets consumed much more quickly. I can only make it through about two update cycles before having to uninstall the old kernels to avoid upgrade failures.
This is a handy command for removing all of the old kernels by the way:
yum erase $(rpm -qa kernel* | egrep -v $(uname -r))
I've updated my provisioning tools to make the boot partition larger, but obviously I still have to do something about the pre-existing systems.
All of my VMs are built with provisioning templates that create a boot partition on the first disk, then format the remainder of the first disk for LVM. In order to expand the boot partition, it's boundary needs to be moved into space occupied by the OS itself. Fortunately LVM is pretty flexible. Not only can I "move" the partition boundary, I can do it while the system is up and running.
Here's what we're starting off with:
[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 211MB 210MB primary xfs boot
2 211MB 53.7GB 53.5GB primary lvm
Two partitions on the first disk, the first partition is for /boot, the second is owned by LVM. As you can see there's no space between them: the second partition begins exactly where the first ends.
Here's what LVM looks like:
[root@azrst2ill001 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rootvg 1 6 0 wz--n- 49.80g 21.94g
[root@azrst2ill001 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 49.80g 21.94g
The primary volume group with all of the OS volumes is named "rootvg" and it is contained on a single PV - /dev/sda2. With some quick mental math you can see it's consuming just over 30GB.
Note:These instructions assume the machine was built with the default filesystem for each release, so the RHEL 6 instructions assume you built /boot with ext4, otherwise the instructions assume you're working with RHEL 7 and built /boot with xfs.
Warning:
This process is somewhat dangerous because if it is done incorrectly or the server is rebooted before it is complete, the system could be unbootable..
The Goal:
Expand the boot partition from 200MB to 1GB without re-imaging the system.
The Solution:
1. Verify the volume group has at least 1GB free because that's basically where the extra space for /boot is going to come from.
vgs
2. Add a temporary disk to the machine that's at least as large as the root volume group. In my case I attached a new virtual data disk in Azure and 128GB was the smallest option.
UDEV created the device as /dev/sdc:
[root@azrst2ill001 boot]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 953M 0 part /boot
└─sda2 8:2 0 49.1G 0 part
├─rootvg-rootlv (dm-0) 253:0 0 8G 0 lvm /
├─rootvg-localhomelv (dm-1) 253:1 0 2G 0 lvm /usr/local/home
├─rootvg-optlv (dm-2) 253:2 0 5G 0 lvm /opt
├─rootvg-swaplv (dm-3) 253:3 0 2G 0 lvm
├─rootvg-tmplv (dm-4) 253:4 0 5G 0 lvm /tmp
└─rootvg-varlv (dm-5) 253:5 0 4G 0 lvm /var
sdc 8:32 0 128G 0 disk
sdb 8:16 0 56G 0 disk
└─sdb1 8:17 0 56G 0 part /mnt/resource
3. Add the new disk to the primary VG
vgextend rootvg /dev/sdc
4. Move all of the physical extents (the data basically) off of /dev/sda2. This process will take some time.
pvmove /dev/sda2
5. Once all of the data is cleared off of /dev/sda2, delete it from the primary VG.
vgreduce rootvg /dev/sda2
6. Just for good measure remove /dev/sda2 from LVM altogether
pvremove /dev/sda2
7. (Optional) At this point I wanted to make sure I had some kind of fallback if I dorked something up, so I made a couple of quick backups.
I've been getting away with using a 200MB boot partition for over a decade, but RHEL 7 has started pushing the limits of that scheme. Between GRUB2 and the automatic creation of a rescue kernel and initramfs that's twice the size as a normal initramfs, that 200MB gets consumed much more quickly. I can only make it through about two update cycles before having to uninstall the old kernels to avoid upgrade failures.
This is a handy command for removing all of the old kernels by the way:
yum erase $(rpm -qa kernel* | egrep -v $(uname -r))
I've updated my provisioning tools to make the boot partition larger, but obviously I still have to do something about the pre-existing systems.
All of my VMs are built with provisioning templates that create a boot partition on the first disk, then format the remainder of the first disk for LVM. In order to expand the boot partition, it's boundary needs to be moved into space occupied by the OS itself. Fortunately LVM is pretty flexible. Not only can I "move" the partition boundary, I can do it while the system is up and running.
Here's what we're starting off with:
[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 211MB 210MB primary xfs boot
2 211MB 53.7GB 53.5GB primary lvm
Two partitions on the first disk, the first partition is for /boot, the second is owned by LVM. As you can see there's no space between them: the second partition begins exactly where the first ends.
Here's what LVM looks like:
[root@azrst2ill001 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rootvg 1 6 0 wz--n- 49.80g 21.94g
[root@azrst2ill001 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 rootvg lvm2 a-- 49.80g 21.94g
The primary volume group with all of the OS volumes is named "rootvg" and it is contained on a single PV - /dev/sda2. With some quick mental math you can see it's consuming just over 30GB.
Note:These instructions assume the machine was built with the default filesystem for each release, so the RHEL 6 instructions assume you built /boot with ext4, otherwise the instructions assume you're working with RHEL 7 and built /boot with xfs.
Warning:
This process is somewhat dangerous because if it is done incorrectly or the server is rebooted before it is complete, the system could be unbootable..
The Goal:
Expand the boot partition from 200MB to 1GB without re-imaging the system.
The Solution:
1. Verify the volume group has at least 1GB free because that's basically where the extra space for /boot is going to come from.
vgs
2. Add a temporary disk to the machine that's at least as large as the root volume group. In my case I attached a new virtual data disk in Azure and 128GB was the smallest option.
UDEV created the device as /dev/sdc:
[root@azrst2ill001 boot]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 953M 0 part /boot
└─sda2 8:2 0 49.1G 0 part
├─rootvg-rootlv (dm-0) 253:0 0 8G 0 lvm /
├─rootvg-localhomelv (dm-1) 253:1 0 2G 0 lvm /usr/local/home
├─rootvg-optlv (dm-2) 253:2 0 5G 0 lvm /opt
├─rootvg-swaplv (dm-3) 253:3 0 2G 0 lvm
├─rootvg-tmplv (dm-4) 253:4 0 5G 0 lvm /tmp
└─rootvg-varlv (dm-5) 253:5 0 4G 0 lvm /var
sdc 8:32 0 128G 0 disk
sdb 8:16 0 56G 0 disk
└─sdb1 8:17 0 56G 0 part /mnt/resource
3. Add the new disk to the primary VG
vgextend rootvg /dev/sdc
4. Move all of the physical extents (the data basically) off of /dev/sda2. This process will take some time.
pvmove /dev/sda2
5. Once all of the data is cleared off of /dev/sda2, delete it from the primary VG.
vgreduce rootvg /dev/sda2
6. Just for good measure remove /dev/sda2 from LVM altogether
pvremove /dev/sda2
7. (Optional) At this point I wanted to make sure I had some kind of fallback if I dorked something up, so I made a couple of quick backups.
Firstly, I backed up the whole boot partition:
dd if=/dev/sda1 of=/root/bootpart.img
Next I backed up the MBR
dd if=/dev/sda of=/root/mbr.img bs=512 count=1
These backups are only really helpful if you realize you screwed something up before trying to reboot.
8. Delete the LVM partition
parted -s /dev/sda rm 2
9. Put the starting position of the boot partition into a variable
START1=$(parted -s /dev/sda print | grep "^ 1" | awk '{print $2}')
10. (The scary part) remove the boot partition.
You can get away with this because the only thing parted is manipulating is the partition table. The actual filesystem data is unaffected by this. As long as you put the start of the boot partition in exactly the same place (which is why we saved that place in a variable before deleting it) everything will line up and work normally.
You can get away with this because the only thing parted is manipulating is the partition table. The actual filesystem data is unaffected by this. As long as you put the start of the boot partition in exactly the same place (which is why we saved that place in a variable before deleting it) everything will line up and work normally.
umount /boot
parted -s /dev/sda rm 1
11. Re-create the boot partition
parted -s /dev/sda mkpart primary xfs $START1 1GB
If you're doing this on RHEL 6, you probably want to go with ext4 instead:
parted -s /dev/sda mkpart primary ext4 $START1 1GB
12. Set the partition to bootable
parted -s /dev/sda set 1 boot on
13. Check the filesystem.
xfs_repair /dev/sda1
Note: if you get an error message "/dev/sda1 contains a mounted filesystem" just run "umount /boot" again right before you run the command. For some reason RHEL 7 kept re-mounting /boot automatically on me(autofs isn't even installed). I didn't take the time to figure out what was causing this behavior, but it's easy enough to work around.
If you're doing this on RHEL 6:
fsck -f /dev/sda1
14. Mount the /boot filesystem (if it didn't already mount itself)
mount /boot
15. Resize the /boot filesystem to fill the new larger partition.
xfs_growfs /dev/sda1
If you're doing this on RHEL 6:
resize2fs /dev/sda1
16. I want to use the rest of the space on /dev/sda for the second partition. To get the start and end positions of free space, I used parted to set some variables.
START2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $1}')
END2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $2}')
17. Create the new LVM partition (the -- tells parted to accept a blank value for filesystem because we're going to use the partition for LVM instead of installing a filesystem onto it.)
parted -s /dev/sda mkpart primary -- $START2 $END2
18. Set the partition flag to LVM
parted -s /dev/sda set 2 lvm on
19. Check to make sure everything looks right - most importantly that the boot partition is showing the desired new size, and that it's set to bootable, and that when you do an ls on /boot you can see the kernel and intramfs files.
[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 1075MB 1074MB primary xfs boot
2 1075MB 53.7GB 52.6GB primary lvm
[root@azrst2ill001 ~]# ls /boot
config-3.10.0-327.10.1.el7.x86_64 initrd-plymouth.img
config-3.10.0-327.28.3.el7.x86_64 symvers-3.10.0-327.10.1.el7.x86_64.gz
grub2 symvers-3.10.0-327.28.3.el7.x86_64.gz
initramfs-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3.img System.map-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64.img System.map-3.10.0-327.28.3.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64kdump.img vmlinuz-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3
initramfs-3.10.0-327.28.3.el7.x86_64.img vmlinuz-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.28.3.el7.x86_64kdump.img vmlinuz-3.10.0-327.28.3.el7.x86_64
20. Move the rootvg volume group back to the first disk, and remove the temporary disk from the VG.
vgextend rootvg /dev/sda2
pvmove /dev/sdc
vgreduce rootvg /dev/sdc
pvremove /dev/sdcAt this point it should be safe to remove the temporary disk you added in step 2, and to delete the backup .img files if you chose to create them in step 7.
Even though you can perform this whole operation with no downtime to the system, I strongly recommend rebooting at this point to verify everything is working as it should. Better to find out now than months from now when it's not fresh in your mind.
Comments
Post a Comment