Platforms like Kubernetes, Nomad, or any cloud-hosted platform-as-a-service (Paas) offer a variety of powerful capabilities. From scaling workloads to secrets management to deployment strategies, these workload orchestrators are optimized to help scale infrastructure in different ways. But do operators always need to pay the cost for maximal scalability? Sometimes, the cost of complexity and abstraction overcome their benefits. Many builders instead come to rely on radically simple deployment architectures for ease of management. Two virtual private servers behind a load balancer is a drastically simpler stack to manage in comparison to a sprawling cluster of microservers across a fleet of container hosts. This can start to pay dividends when there are fewer moving parts to debug when problems arise or upgrade when the time comes to maintain them. The foundation for many modern Linux distributions is , and it comes with a strong set of features that are often comparable to container orchestrators or PaaS systems. In this article, we’ll explore how you can leverage the latest systemd features to gain many of the abilities of those other large systems without the management headache and supercharge any ordinary server to be a very capable application platform. systemd Linux A systemd Primer On a single host, writing a systemd file is an ideal way to run a managed process. Most of the time, you don’t even need to change the application at all: systemd supports a variety of different kinds of services and can adapt accordingly. .service For example, consider this simple that defines how to run a simple web service: .service [Unit]
Description=a simple web service

[Service]
ExecStart=/usr/bin/python3 -m http.server 8080 Remember the defaults for systemd services: must be an absolute path, processes should not fork into the background, and you may need to set requisite environment variables with the option. ExecStart= Environment= When placed into a file like , this creates a service that you can control with : /etc/systemd/system/webapp.service systemctl will start the process. systemctl start webapp will display whether the service is running, its uptime, and output from and , as well as the process’s ID and other information. systemctl status webapp stderr stdout will end the service. systemctl stop webapp In addition, all output printed to and will be aggregated by journald and accessible via the system journal (with ) or targeted specifically using the flag: stderr stdout journalctl --unit journalctl --unit webapp Because journald rotates and manages its storage by default, collecting logs via the journal is a good strategy for managing log storage. The rest of this article will explore options to enhance a service like this one. Secrets Container support the ability to securely inject : values drawn from secure datastores and exposed to running workloads. Sensitive data like API keys or passwords require different treatment than environment variables or configuration files to avoid unintentional exposure. orchestrators like Kubernetes Secrets The supports loading sensitive values from files on-disk and exposing them to running services in a secure way. Like hosted platforms that manage secrets remotely, systemd will treat differently than values like environment variables to ensure that they’re kept safe. LoadCredential= systemd option Credentials To inject a secret into a systemd service, begin by placing a file containing the secret value into a path on the filesystem. For example, to expose an API key to a unit, create a file at to persist the file across reboots or at to avoid persisting the file permanently (the path can be arbitrary, but systemd will treat these paths as defaults). In either case, the file should have restricted permissions using a command like . .service /etc/credstore/api-key /run/credstore/api-key credstore chmod 400 /etc/credstore/api-key Under the section of the file, define the option and pass it two values separated by a colon ( ): the name of the credential and its path. For example, to call our file “token,” define the following systemd service option: [Service] .service LoadCredential= : /etc/credstore/api-key LoadCredential=token:/etc/credstore/api-key When systemd starts your service, the secret is exposed to the running service under a path of the form where is an environment variable populated by systemd. Your application code should read in each secret defined this way for use in libraries or code that require secure values like API tokens or passwords. For example, in Python, you can read this secret with code like the following: ${CREDENTIALS_DIRECTORY}/token ${CREDENTIALS_DIRECTORY} from os import environ
from pathlib import Path

credentials_dir = Path(environ["CREDENTIALS_DIRECTORY"])
with Path(credentials_dir / "token").open() as f:
    secret = f.read().strip() You can then use the variable with the contents of your secret for any libraries that may require an API token or password. secret Restarts Another capability of orchestrators like is the ability to automatically a workload that has crashed. Whether due to an unhandled application error or some other cause, restarting failed applications is a very useful capability that is often the first line of defense when designing an application to be resilient. Nomad restart The systemd controls whether systemd will automatically restart a running process. There are several potential values for this option, but for basic services, the setting is well-suited to satisfy most use cases. Restart= option on-failure Another setting to consider when configuring auto-restart is , which dictates how long systemd will wait before starting up the service again. Typically, this value should be customized to avoid restarting failed services in a tight loop and potentially consuming too much CPU time spent restarting processes. A short value that doesn’t wait long like is usually sufficient. the RestartSec= option too 5s Options like that accept duration periods or time-based values can parse a variety of formats like or depending on your needs. Reference for additional information. RestartSec= 5min 10s 1hour the manual for systemd.time Finally, two other options dictate how aggressively systemd will attempt to restart failed units before eventually giving up. and will control how often a unit is permitted to start within a given period of time. For example, the following settings: StartLimitIntervalSec= StartLimitBurst= StartLimitBurst=5
StartLimitIntervalSec=10 It will only permit a unit to try to start up for a maximum of 5 times over a period of 10 seconds. If the configured service attempts to start up a sixth time within a period of 10 seconds, systemd will stop attempting to restart the unit and mark it as instead. failed Combing all of these settings, you might include the following options for your unit to configure automatic restarts: .service [Unit]
StartLimitBurst=5
StartLimitIntervalSec=10

[Service]
Restart=on-failure
RestartSec=1 This configuration will restart a service if it fails - that is, it exits unexpectedly, such as with a nonzero exit code - after waiting for one second and will stop attempting to restart the service if it tries to start more than five times over the course of 10 seconds. Service Hardening One of the chief benefits of running within a container is security sandboxing. By segmenting an application process from the underlying operating system, any vulnerabilities that may be present in the service are much more difficult to escalate into full-blown compromise. Runtimes like Docker achieve this through a combination of cgroups and other security primitives. You may enable several systemd options to enforce similar restrictions that can help protect an underlying host against unpredictable workload behavior: can restrict write access to sensitive system paths like and . enumerates all the available options, but generally speaking, setting this option to is a reasonable default to protect these filesystem paths. ProtectSystem= /boot /usr The documentation for this option full can set the , , and directories to read-only with the setting or, when set to , mount them into the service’s filesystem as empty directories. Unless your application has a specific need to access these directories, setting this to can safely harden the system against illegitimate access to those directories. ProtectHome= /home /root /run/user read-only true true maintains a separate and for the configured service so that temporary files for this service and other processes remain private. Unless there is a compelling reason for processes to share information via temporary files, this is a useful option to enable. PrivateTmp= /tmp /var/tmp to harden a service by ensuring that the executed process cannot elevate its privileges. If you’re unsure about the ability to use other hardening options, this is generally one of the least problematic to enable. NoNewPrivileges= is another safe and straightforward way is a helpful resource for exploring the different options that apply to executable workloads like services. The manual page for systemd.exec Further Reading The are extensive and useful for learning about all the options available for running your own applications. Whether you’re running a persistent service like a webserver or a periodic unit to replace a cron job, the systemd documentation can offer helpful guidance. manual pages for the systemd project .timer

This story contains new, firsthand information uncovered by the writer.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Micro-DevOps With Systemd: Supercharge Any Ordinary Linux Server

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

20,000 Leagues Under Your Shell

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

20,000 Leagues Under Your Shell

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps