Upgrade to Chef Automate 4.x

[edit on GitHub]

Chef Automate provides an entire suite of enterprise capabilities for node visibility and compliance. Chef Automate upgrades from one minor version to another automatically. However, Chef Automate will not automatically upgrade to a major version. See the instructions below for manually upgrading Chef Automate from date-based versions to Chef Automate 4.x.

Upgrade Journey

Please choose the following upgrade journey based on your current version of Chef Automate. All of these upgrades are manual upgrades.

For example, if today you are on version 2021201164433, your upgrade journey should be:

1. Manual upgrade to 20220329091442
2. Manual upgrade to 3.0.49
3. Manual upgrade to 4.0.x

Prerequisites

• Plan your downtime: This upgrade requires downtime. Before upgrading, set the environment to handle the downtime. Run sudo chef-automate maintenance on to turn on maintenance mode.
• Backup Chef Automate database: This Chef Automate version upgrades ElasticSearch. Backup your data before upgrading.
• Current version should be 3.0.49: If you are not on 3.0.49 version, do regular upgrades according to your topology.

Upgrade to Version 3.0.49

Check your current version:

sudo chef-automate version

Normal and airgapped upgrade to 3.0.49

To upgrade to automate 3.0.49 follow the link: upgrading to 3.0.49

Upgrade Path from 3.0.49 to 4.0.x

There are four possible scenarios to upgrade from 3.0.49 to 4.0.x version.

• Chef Automate with Embedded Elasticsearch

• Chef Automate with External Elasticsearch

• Chef Automate in Air-Gapped Environment With Embedded Elasticsearch

• Chef Automate in Air-Gapped Environment With External Elasticsearch

Chef Automate With Embedded Elasticsearch

To upgrade Chef Automate with embedded Elasticsearch, follow the steps given below:

Upgrade Chef Automate from version 3.0.49 to 4.0.x

1. Start a major version upgrade:

Here, you will be prompted to accept multiple Pre Upgrade checklist. Accept the actions before upgrade.

sudo chef-automate upgrade run --major

Once you are done with the upgrade, follow the steps post upgrade which are:

2. Check the upgrade status of Chef Automate:

sudo chef-automate upgrade status

3. Turn off maintenance mode:

sudo chef-automate maintenance off

4. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

For Example:

If Elasticsearch configuration was:

[elasticsearch]
  [elasticsearch.v1]
    [elasticsearch.v1.sys]
      [elasticsearch.v1.sys.cluster]
        max_shards_per_node = 6000
      [elasticsearch.v1.sys.indices]
        [elasticsearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [elasticsearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM```

Then add in OpenSearch configuration as:

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
        max_shards_per_node = 6000 # Refer the value from ElasticSearch Config, If this value is not there in elastic search config, then do not patch in openseaarch.
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [opensearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM

Apply this using the config patch command.

5. Migrate your data from ElasticSearch 6.8 to OpenSearch 1.2.4:

sudo chef-automate post-major-upgrade migrate --data=es

6. Verify whether all services are running:

sudo chef-automate status

7. Clear the old ElasticSearch 6.8 data if all the data is present in your upgraded Chef Automate.

sudo chef-automate post-major-upgrade clear-data --data=es

Chef Automate with External ElasticSearch

To upgrade Chef Automate with external Elasticsearch, follow the steps given below:

Upgrade Chef Automate from version 3.0.49 to 4.0.x

1. Start major version upgrade:

sudo chef-automate upgrade run --major

2. Check upgrade status is up-to-date

sudo chef-automate status

3. Upgrade your external ElasticSearch 6.8 to OpenSearch 1.2.4 manually. If you have configured Host, Port, Username or Password of ElasticSearch, patch the new configuration to use Chef Automate.

4. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

   Please refer to the elasticsearch.yml file to get the applied configuration on your external Elasticsearch. Add the releavent configuration from external Elasticsearch (elasticsearch.yml) to the opensearch.yml on your external OpenSearch.

5. Turn off maintenance mode:

sudo chef-automate maintenance off

Chef Automate in Air-Gapped Environment With Embedded ElasticSearch

Upgrade Chef Automate from version 3.0.49 to 4.0.x

To upgrade to 4.0.x, follow the steps below:

On Internet connected machine

1. Download latest CLI of Chef Automate.

curl https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef-automate && chmod +x chef-automate

2. Create an Airgap Installation Bundle (AIB).

sudo ./chef-automate airgap bundle create 

OR we can directly download via curl request

curl https://packages.chef.io/airgap_bundle/current/automate/latest.aib -o automate-4.x.y.aib

3. Copy the latest Chef Automate CLI (chef-automate) and AIB (automate-4.x.y.aib) to the air-gapped machine running Chef Automate.

On Air-Gapped machine running Chef Automate

1. Set your upgrade strategy to none in Chef Automate config. Confirm by the following code:

sudo ./chef-automate config show

2. Upgrade using new AIB and Chef Automate CLI:

sudo ./chef-automate upgrade run --airgap-bundle automate-4.x.y.aib --major

Post Upgrade

1. Check the upgrade status of Chef Automate, run the following command:

sudo chef-automate upgrade status

2. Turn off the maintenance mode.

sudo chef-automate maintenance off

3. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

For Example:

If Elasticsearch configuration was:

[elasticsearch]
  [elasticsearch.v1]
    [elasticsearch.v1.sys]
      [elasticsearch.v1.sys.cluster]
        max_shards_per_node = 6000
      [elasticsearch.v1.sys.indices]
        [elasticsearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [elasticsearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM```

Then add in OpenSearch configuration as:

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
        max_shards_per_node = 6000 # Refer the value from ElasticSearch Config, If this value is not there in elastic search config, then do not patch in openseaarch.
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"
      [opensearch.v1.sys.runtime]
        max_open_files = "65536"
        max_locked_memory = "unlimited"
        heapsize = "8g" # This should be the 50% of RAM

Apply this using config patch command.

4. Migrate your data from ElasticSearch 6.8 to OpenSearch 1.2.4:

sudo chef-automate post-major-upgrade migrate --data=es

5. Verify whether all services are running:

sudo chef-automate status

6. Clear the old ElasticSearch 6.8 data if all the data is present in your upgraded Chef Automate.

sudo chef-automate post-major-upgrade clear-data --data=es

Chef Automate in Air-Gapped Environment With External ElasticSearch

Upgrade Chef Automate from version 3.0.49 to 4.0.x

To upgrade to 4.0.x, follow the steps below:

On Internet connected machine

1. Download latest CLI of Chef Automate

curl https://packages.chef.io/files/current/latest/chef-automate-cli/chef-automate_linux_amd64.zip | gunzip - > chef-automate && chmod +x chef-automate

2. Create an Airgap Installation Bundle (AIB):

sudo ./chef-automate airgap bundle create

OR we can directly download via curl request

curl https://packages.chef.io/airgap_bundle/current/automate/latest.aib -o automate-4.x.y.aib

3. Copy the latest Chef Automate CLI (chef-automate) and AIB (automate-4.x.y.aib) to the air-gapped machine running Chef Automate.

On Air-Gapped machine running Chef Automate

1. Set your upgrade strategy to none in Chef Automate config. Confirm by the following code:

sudo ./chef-automate config show

2. Upgrade using new AIB and Chef Automate CLI:

sudo ./chef-automate upgrade run --airgap-bundle automate-4.x.y.aib --major

3. Check whether upgrade status is up-to-date

sudo chef-automate status

4. Upgrade your external ElasticSearch 6.8 to OpenSearch 1.2.4 manually. If you have configured Host, Port, Username or Password of ElasticSearch, patch the new configuration to use Chef Automate.

5. All relevant configuration fields of the Elasticsearch should be copied into the OpenSearch configuration.

   Please refer to the elasticsearch.yml file to get the applied configuration on your external Elasticsearch. Add the releavent configuration from external Elasticsearch (elasticsearch.yml) to the opensearch.yml on your external OpenSearch.

6. Turn off maintenance mode using following command:

sudo chef-automate maintenance off

Troubleshooting

Circuit Breaking Exception

{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [6126524880/5.7gb], which is larger than the limit of [5988548608/5.5gb], real usage: [6126524880/5.7gb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=74975/73.2kb, in_flight_requests=0/0b, accounting=89882860/85.7mb]","bytes_wanted":6126524880,"bytes_limit":5988548608,"durability":"PERMANENT"}]

• Update the Opensearch Config, using chef-automate config patch <config_patch.toml>

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.runtime]
         heapsize = "8g" # This should be Half of RAM
      [opensearch.v1.sys.indices]
        [opensearch.v1.sys.indices.breaker]
          total_limit = "95%"

Shard Failure

[ERROR] Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [997]/[1000] maximum shards open;]

To address the issue of shard limit hitting 1000, we need to increase the max_shards_per_node Update the Opensearch Config, using chef-automate config patch <config_patch.toml>

[opensearch]
  [opensearch.v1]
    [opensearch.v1.sys]
      [opensearch.v1.sys.cluster]
         max_shards_per_node = "<NUMBER_OF_SHARD>"

Migration Fails

If Chef Automate fails to migrate your data to OpenSearch 1.2.4 while running chef-automate post-major-upgrade migrate --data=es, restore the data using:

sudo chef-automate backup restore </path/to/backups/>BACKUP_ID

To restore your Air-Gapped bundle, run the following command:

sudo chef-automate backup restore  --airgap-bundle </path/to/bundle> </path/to/backups/>BACKUP_ID

Click here to know more about the Restore methods.

To start the upgrade, use the backup ID from the backup created. In case the restore fails even after upgrading the Chef Automate version, follow the steps given below:

1. Uninstall Chef Automate.

sudo chef-automate uninstall

2. Install the last major version (3.0.49) using the air-gapped installation process.

3. Restore the backup:

sudo chef-automate backup restore <backup_id>

Refer to the Chef Automate Restore documentation.