I worked in a job with build scripts. Developers would list what they wanted in a drop-down menu on a website, with very few “fill in the blanks.” This would create a template, which was sanity-checked.
One of the “fill in the blanks” was “home directory of user, if not default /home/username.” Some people filled it in, some didn’t. A lot of “users” might be apps with /home being “/opt/appname” “/var/www/html” or something. We checked to make sure that directory existed, if not, create, and set permissions. Easy peasy, all automated. Ran this lots of times.
Then one day, the script failed. Borked the whole box. Sometimes the VM was corrupt, so delete VM and try again. Usually worked. But this time, the build kept failing. The box went down. Wasn’t even bootable. This happened several times with this one build. So we mounted the borked drive under a new VM and checked out the logs. Just like the dessert stage of Willy Wonka chewing gum, it always failed at the last stage: making /home directories.
It would create them, then halt that it could not find bash. We looked for bash on the bad drive, and it was the usual /bin/bash shortcut to /usr/bin/bash and we were truly puzzled. I did a chroot to the drive and NOTHING worked. It just hung. That was the first clue.
The second was looking through the build script (in bash, which we didn’t write) and checking the steps. Looked it the logs. Always died at creating some user named sapadm, the user for the HANA database. Eventually, I checked the configure file, and noticed it was the only user with the odd home directory “/usr/sap.” Then it hit me: the permissions.
The script, thinking it was a home directory, did a chmod - R 755 for all directories and chmod - R 644 for all files! That meant, while creating home, it made everything under /usr not executable anymore! Holy shit, no wonder nothing worked! So we commented out that user in the config, ran the build again, and we were good! We created the sapadm by hand, and then later fixed the bug in the script.
SANITIZE YOUR DATA. Or you might turn Violet Beauregarde into a blueberry.
I worked in a job with build scripts. Developers would list what they wanted in a drop-down menu on a website, with very few “fill in the blanks.” This would create a template, which was sanity-checked.
One of the “fill in the blanks” was “home directory of user, if not default /home/username.” Some people filled it in, some didn’t. A lot of “users” might be apps with /home being “/opt/appname” “/var/www/html” or something. We checked to make sure that directory existed, if not, create, and set permissions. Easy peasy, all automated. Ran this lots of times.
Then one day, the script failed. Borked the whole box. Sometimes the VM was corrupt, so delete VM and try again. Usually worked. But this time, the build kept failing. The box went down. Wasn’t even bootable. This happened several times with this one build. So we mounted the borked drive under a new VM and checked out the logs. Just like the dessert stage of Willy Wonka chewing gum, it always failed at the last stage: making /home directories.
It would create them, then halt that it could not find bash. We looked for bash on the bad drive, and it was the usual /bin/bash shortcut to /usr/bin/bash and we were truly puzzled. I did a chroot to the drive and NOTHING worked. It just hung. That was the first clue.
The second was looking through the build script (in bash, which we didn’t write) and checking the steps. Looked it the logs. Always died at creating some user named sapadm, the user for the HANA database. Eventually, I checked the configure file, and noticed it was the only user with the odd home directory “/usr/sap.” Then it hit me: the permissions.
The script, thinking it was a home directory, did a chmod - R 755 for all directories and chmod - R 644 for all files! That meant, while creating home, it made everything under /usr not executable anymore! Holy shit, no wonder nothing worked! So we commented out that user in the config, ran the build again, and we were good! We created the sapadm by hand, and then later fixed the bug in the script.
SANITIZE YOUR DATA. Or you might turn Violet Beauregarde into a blueberry.