- HEADLINE: Better handling of apt and dpkg locks during apt-get upgrade/update
- ROLE/AFFILIATION: Cloud hosting provider
- DESCRIPTION: Thanks for taking our feedback! I work in cloud services and deal with many Ubuntu 16.xx users of various skill levels. One of the most common issues that I see are folks reporting that the "apt-get update && apt-get upgrade" commands fail on newly deployed cloud servers. I wouldn't really classify this as a bug, but some internal changes to the apt/apt-get commands could make things a lot smoother. Here's what happens.
1. Cloud service providers periodically make a disk image of Ubuntu 16 with the latest packages an updates.
2. These images are used for several weeks (or sometimes even longer).
3. When customers deploy a new Ubuntu cloud server, the disk image is copied to a new machine.
4. The machine boots. Ubuntu realizes that it hasn't been booted for a while and performs some internal tasks. IIRC this is related to the "apt daily updates" service. This is the thread I always seem to reference ( http://unix.stackexchange.com/questions/315502/how-to-disabl... ). The apt tasks run, which can take serveral minutes.
5. Asynchronously, the cloud service reports that the server is ready.
6. The customer (or their scripts) will login and do some provisioning. A common first provisioning step is to do a package update: "apt-get update && apt-get upgrade". These may fail if the "apt daily updates" task is still running. I've seen a variety of errors, but usually look like this:
> E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
> E: Unable to lock directory /var/lib/apt/lists/
When running "apt-get update" during this state, it sleeps for a few moments, then reports an apt/dpkg lock error. Since most of the time this whole process happens autonomously, it seems like making "apt-get update" and "apt-get upgrade" block until a lock is available would be quite beneficial. Documentation on the web for upgrading ubuntu usually references these two commands, so I don't think adding new parameters would be that beneficial; changing the default behavior would be better.
Absolutely happy to have your feedback, so you're most welcome!
Is this Cloud Hosting Provider part of Canonical's Certified Public Cloud program? If so, I would be very surprised to see the kinds of issues you mention here. And if not, this is exactly the kinds of issues we routinely see with clouds which are not part of Canonical's CPC.
Currently, no, we're not in the CPC program. But that's something we will definitely consider.
Based on the CPC overview, it seems like the Ubuntu team can make specialized images for folks in the CPC program. Which is great, but a design change here would likely benefit the entire Ubuntu community as a whole. Many cloud providers allow users to save snapshots of their cloud instances - another area where this design issue conflicts with scripting.
Conflicts with the apt/dpkg locks weren't as common in the older Ubuntu server versions that used upstart. My team started to notice this more often after Ubuntu switched to systemd.
- HEADLINE: Better handling of apt and dpkg locks during apt-get upgrade/update
- ROLE/AFFILIATION: Cloud hosting provider
- DESCRIPTION: Thanks for taking our feedback! I work in cloud services and deal with many Ubuntu 16.xx users of various skill levels. One of the most common issues that I see are folks reporting that the "apt-get update && apt-get upgrade" commands fail on newly deployed cloud servers. I wouldn't really classify this as a bug, but some internal changes to the apt/apt-get commands could make things a lot smoother. Here's what happens.
1. Cloud service providers periodically make a disk image of Ubuntu 16 with the latest packages an updates. 2. These images are used for several weeks (or sometimes even longer). 3. When customers deploy a new Ubuntu cloud server, the disk image is copied to a new machine. 4. The machine boots. Ubuntu realizes that it hasn't been booted for a while and performs some internal tasks. IIRC this is related to the "apt daily updates" service. This is the thread I always seem to reference ( http://unix.stackexchange.com/questions/315502/how-to-disabl... ). The apt tasks run, which can take serveral minutes. 5. Asynchronously, the cloud service reports that the server is ready. 6. The customer (or their scripts) will login and do some provisioning. A common first provisioning step is to do a package update: "apt-get update && apt-get upgrade". These may fail if the "apt daily updates" task is still running. I've seen a variety of errors, but usually look like this:
> E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable) > E: Unable to lock directory /var/lib/apt/lists/
When running "apt-get update" during this state, it sleeps for a few moments, then reports an apt/dpkg lock error. Since most of the time this whole process happens autonomously, it seems like making "apt-get update" and "apt-get upgrade" block until a lock is available would be quite beneficial. Documentation on the web for upgrading ubuntu usually references these two commands, so I don't think adding new parameters would be that beneficial; changing the default behavior would be better.