Running Flatcar Container Linux on Equinix Metal

    Equinix Metal (formerly known as Packet) is a bare metal cloud hosting provider. Flatcar Container Linux is installable as one of the default operating system options. You can deploy Flatcar Container Linux servers via the web portal or API. At the end of the document there are instructions for deploying with Terraform.

    Deployment instructions

    The first step in deploying any devices on Equinix Metal is to first create an account and decide if you’d like to deploy via our portal or API. The portal is appropriate for small clusters of machines that won’t change frequently. If you’ll be deploying a lot of machines, or expect your workload to change frequently it is much more efficient to use the API. You can generate an API token through the portal once you’ve set up an account and payment method.

    Projects

    Equinix Metal has a concept of ‘projects’ that represent a grouping of machines that defines several other aspects of the service. A project defines who on the team has access to manage the machines in your account. Projects also define your private network; all machines in a given project will automatically share backend network connectivity. The SSH keys of all team members associated with a project will be installed to all newly provisioned machines in a project. All servers need to be in a project, even if there is only one server in that project.

    Portal instructions

    Once logged into the portal you will be able to click the ‘New server’ button and choose Flatcar Container Linux from the menu of operating systems, and choose which region you want the server to be deployed in. If you choose to enter a custom Ignition config, you can enable ‘Add User Data’ and paste it there. The SSH key that you associate with your account and any other team member’s keys that are on the project will be added to your Flatcar Container Linux machine once it is provisioned.

    API instructions

    If you select to use the API to provision machines on Equinix Metal you should consider using one of the language libraries to code against. As an example, this is how you would launch a single Type 1 machine in a curl command. API Documentation .

    # Replace items in brackets (<EXAMPLE>) with the appropriate values.
    
    curl -X POST \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -H 'X-Auth-Token: <API_TOKEN>' \
    -d '{"hostname": "<HOSTNAME>", "plan": "c3.small.x86", "facility": "da11", "operating_system": "flatcar_stable", "userdata": "<USERDATA>"}' \
    https://api.equinix.com/metal/v1/projects/<PROJECT_ID>/devices
    

    Double quotes in the <USERDATA> value must be escaped such that the request body is valid JSON. See the Butane Config section below for more information about accepted forms of userdata.

    iPXE booting

    If you need to run a Flatcar Container Linux image which is not available through the OS option in the API, you can boot via ‘Custom iPXE’. This is the case for ARM64 images right now as they are not available via Equinix Metal’s API.

    Assuming you want to run boot an Alpha image via iPXE on a c2.large.arm machine, you have to provide this URL for ‘Custom iPXE Settings’:

    https://alpha.release.flatcar-linux.net/arm64-usr/current/flatcar_production_packet.ipxe
    

    Do not forget to provide an Ignition config with your SSH key because the PXE images don’t have any OEM packages which could fetch the Equinix Metal Project’s SSH keys after booting.

    If not configured elsewise, iPXE booting will only done at the first boot because you are expected to install the operating system to the hard disk yourself.

    Butane Configs

    Flatcar Container Linux allows you to configure machine parameters, configure networking, launch systemd units on startup, and more via Butane Configs. These configs are then transpiled into Ignition configs and given to booting machines. Head over to the docs to learn about the supported features . Note that Equinix Metal doesn’t allow an instance’s userdata to be modified after the instance has been launched. This isn’t a problem since Ignition only runs on the first boot.

    You can provide a raw Ignition JSON config to Flatcar Container Linux via Equinix Metal’s userdata field.

    As an example, this Butane YAML config will start an NGINX Docker container:

    variant: flatcar
    version: 1.0.0
    systemd:
      units:
        - name: nginx.service
          enabled: true
          contents: |
            [Unit]
            Description=NGINX example
            After=docker.service
            Requires=docker.service
            [Service]
            TimeoutStartSec=0
            ExecStartPre=-/usr/bin/docker rm --force nginx1
            ExecStart=/usr/bin/docker run --name nginx1 --pull always --log-driver=journald --net host docker.io/nginx:1
            ExecStop=/usr/bin/docker stop nginx1
            Restart=always
            RestartSec=5s
            [Install]
            WantedBy=multi-user.target        
    

    Transpile it to Ignition JSON:

    cat cl.yaml | docker run --rm -i quay.io/coreos/butane:latest > ignition.json
    

    Disabling/enabling autologin

    Beginning with Flatcar major version 3185 the kernelArguments directive in Ignition v3 allows to add/remove the flatcar.autologin kernel command line parameter that is set in grub.cfg. The following short Butane YAML config (to be transpiled to Ignition v3 JSON) ensures that the flatcar.autologin kernel parameter gets removed and then as part of the first boot it gets applied to an instant reboot before the instance comes up:

    variant: flatcar
    version: 1.0.0
    kernel_arguments:
      should_not_exist:
        - flatcar.autologin
    

    With should_exist instead of should_not_exist the argument would be added if it isn’t set in grub.cfg already.

    Read more about setting kernel command line parameters this way here .

    In case you want to disable the autologin on the console with Ignition v2 where no kernelArguments directive exists, you can use the following directive in your Container Linux Config YAML utilizing ct . To take effect it requires an additional reboot.

    storage:
      filesystems:
        - name: oem
          mount:
            device: /dev/disk/by-label/OEM
            format: btrfs
            label: OEM
      files:
        - path: /grub.cfg
          filesystem: oem
          mode: 0644
          append: true
          contents:
            inline: |
                        set linux_append=""
    

    To take effect directly on first boot, the alternative is to create a [email protected] drop-in, here a CLC snippet:

    systemd:
      units:
        - name: [email protected]
          dropins:
            - name: 10-autologin.conf
              contents: |
                [Service]
                ExecStart=
                ExecStart=-/sbin/agetty --noclear %I $TERM
    

    Using Flatcar Container Linux

    Now that you have a machine booted it is time to play around. Check out the Flatcar Container Linux Quickstart guide or dig into more specific topics .

    Terraform

    The metal Terraform Provider allows to deploy machines in a declarative way. Read more about using Terraform and Flatcar here .

    The following Terraform v0.13 module may serve as a base for your own setup.

    You can clone the setup from the Flatcar Terraform examples repository or create the files manually as we go through them and explain each one.

    git clone https://github.com/flatcar/flatcar-terraform.git
    # From here on you could directly run it, TLDR:
    cd equinix-metal-aka-packet
    export METAL_AUTH_TOKEN=...
    terraform init
    # Edit the server configs or just go ahead with the default example
    terraform plan
    terraform apply
    

    Start with a metal-machines.tf file that contains the main declarations:

    terraform {
      required_version = ">= 0.13"
      required_providers {
        metal = {
          source  = "equinix/metal"
          version = "3.3.0-alpha.1"
        }
        ct = {
          source  = "poseidon/ct"
          version = "0.7.1"
        }
        template = {
          source  = "hashicorp/template"
          version = "~> 2.2.0"
        }
      }
    }
    
    resource "metal_device" "machine" {
      for_each         = toset(var.machines)
      hostname         = "${var.cluster_name}-${each.key}"
      plan             = var.plan
      facilities       = var.facilities
      operating_system = "flatcar_stable"
      billing_cycle    = "hourly"
      project_id       = var.project_id
      user_data        = data.ct_config.machine-ignitions[each.key].rendered
    }
    
    data "ct_config" "machine-ignitions" {
      for_each = toset(var.machines)
      content  = data.template_file.machine-configs[each.key].rendered
    }
    
    data "template_file" "machine-configs" {
      for_each = toset(var.machines)
      template = file("${path.module}/machine-${each.key}.yaml.tmpl")
    
      vars = {
        ssh_keys = jsonencode(var.ssh_keys)
        name     = each.key
      }
    }
    

    Create a variables.tf file that declares the variables used above:

    variable "machines" {
      type        = list(string)
      description = "Machine names, corresponding to machine-NAME.yaml.tmpl files"
    }
    
    variable "cluster_name" {
      type        = string
      description = "Cluster name used as prefix for the machine names"
    }
    
    variable "ssh_keys" {
      type        = list(string)
      description = "SSH public keys for user 'core', only needed if you don't have it specified in the Equinix Metal Project"
    }
    
    variable "facilities" {
      type        = list(string)
      default     = ["sjc1"]
      description = "List of facility codes with deployment preferences"
    }
    
    variable "plan" {
      type        = string
      default     = "t1.small.x86"
      description = "The device plan slug"
    }
    
    variable "project_id" {
      type        = string
      description = "The Equinix Metal Project to deploy in (in the web UI URL after /projects/)"
    }
    

    An outputs.tf file shows the resulting IP addresses:

    output "ip-addresses" {
      value = {
        for key in var.machines :
        "${var.cluster_name}-${key}" => metal_device.machine[key].access_public_ipv4
      }
    }
    

    Now you can use the module by declaring the variables and a Container Linux Configuration for a machine. First create a terraform.tfvars file with your settings:

    cluster_name = "mycluster"
    machines     = ["mynode"]
    plan         = "t1.small.x86"
    facilities   = ["sjc1"]
    project_id   = "1...-2...-3...-4...-5..."
    ssh_keys     = ["ssh-rsa AA... [email protected]"]
    

    Create the configuration for mynode in the file machine-mynode.yaml.tmpl:

    passwd:
      users:
        - name: core
          ssh_authorized_keys: 
            - ${ssh_keys}
    storage:
      files:
        - path: /home/core/works
          filesystem: root
          mode: 0755
          contents:
            inline: |
              #!/bin/bash
              set -euo pipefail
              hostname="$(hostname)"
              echo My name is ${name} and the hostname is $${hostname}          
    
    

    Finally, run Terraform v0.13 as follows to create the machine:

    export METAL_AUTH_TOKEN=...
    terraform init
    terraform apply
    

    Log in via ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@IPADDRESS with the printed IP address.

    When you make a change to machine-mynode.yaml.tmpl and run terraform apply again, the machine will be replaced.

    It is recommended to register your SSH key in the Equinix Metal Project to use the out-of-band console. Since Flatcar will fetch this key, too, you can remove it from the YAML config.

    You can find this Terraform module in the repository for Flatcar Terraform examples .