While working on collaborative environment of Terraform deployments, you might encounter an error of acquiring the tf state locking issue. In Terraform word when backend is configured to store state file on remote. It limits multiple users from making changes to environment at the same time to avoid conflicts and changes. In below example I set two terraform window side by side. on one screen I am running terraform apply, it acquired state lock and evaluated plan of resources and halted for my confirmation with yes as input to proceed. I didn't press the yes and kept it hanging.
At the same time on the another screen I ran terraform apply, but it was unable proceed with not able to acquire the state lock, as terraform doesn't allow simultaneous operation at the same time to avoid conflicts and further it provides lock information.
Generally you wait for the state lock to release unlock once the previous deployment is complete and finished. In case if there is any issue and its not self unlocking/released lock, it can be unlocked forcefully with below command as shown in below screenshot.
terraform force-unlock <ID in the Lock Info> -force
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
terraform apply Acquiring state lock. This may take a few moments... Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # azurerm_resource_group.dev will be created + resource "azurerm_resource_group" "dev" { + id = (known after apply) + location = "eastus" + name = "test-rg1" } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: ##################### From Another System ###################### PS D:\Projects\Terraform\\Azure_Backend> terraform apply ╷ │ Error: Error acquiring the state lock │ │ Error message: state blob is already locked │ Lock Info: │ ID: 8fb5e19f-866f-7d13-e5f0-be9329eb3cab │ Path: tfstate/example.terraform.tfstate │ Operation: OperationTypeApply │ Who: XBOXONE\janvi@XBOXONE │ Version: 1.9.2 │ Created: 2024-08-03 13:45:15.4760817 +0000 UTC │ Info: │ │ │ Terraform acquires a state lock to protect the state from being written │ by multiple users at the same time. Please resolve the issue above and try │ again. For most commands, you can disable locking with the "-lock=false" │ flag, but this is not recommended. ╵ PS D:\Projects\Terraform\\Azure_Backend> terraform force-unlock 8fb5e19f-866f-7d13-e5f0-be9329eb3cab Do you really want to force-unlock? Terraform will remove the lock on the remote state. This will allow local Terraform commands to modify this state, even though it may still be in use. Only 'yes' will be accepted to confirm. Enter a value: yes Terraform state has been successfully unlocked! The state has been unlocked, and Terraform commands should now be able to obtain a new lock on the remote state. |
Basically in the background this is what happens on Terraform configured backend on Azure Storage Account. This screenshot shows how tfstate file is locked on Azure Storage Account. Terraform acquires state lock using acquire lease on blob file inside container. You can break lease from portal by selecting terraform.tfstate file and clicking Break lease. Before breaking the state check the status and make sure you wait enough before making break decision.
Once lease is broken check for the Lease state, and now you can try the command terraform apply again.
This is additional for your knowledge. You can break lease (state unlock) on Storage Account >> Container blob file using below PowerShell script. You will need az module and connected to Azure with Connect-AzAccount cmdlet.
# Set up your storage account context $resourceGroupName = 'vcloud-lab.com' $storageAccountName = 'vcloudlabterraform' $containerName = 'tfstate' $filename = "example.terraform.tfstate" # Get the storage account key $storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0] # Create the storage context $storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey # Get the blob $blob = Get-AzStorageBlob -Context $storageContext -Container $containerName -Blob $filename # Break the Lease $blob.ICloudBlob.BreakLease() # Display the lease status $blob.ICloudBlob.Properties.LeaseStatus # # If you need to take a lease on the blob and get the lease ID # $leaseDuration = 60 # Duration in seconds; -1 for infinite lease # $leaseId = New-Guid # Generate a new GUID for the lease ID # # Acquire the lease # $lease = $blob.ICloudBlob.AcquireLease($leaseDuration, $leaseId.Guid) # # Output the lease ID # $lease.LeaseId
Typically lease has 3 properties - lease state, duration and id. While lease status and duration are exposed, lease id is not and is the responsibility of the lease holder to keep a note of that in case there's a need to renew, release or change the lease. Unfortunately you cannot see the locking ID on Azure Storage Account.
This is another PowerShell command, if you can see the lease id (State Lock ID) in terraform lock info, using the ID you can release/break the lock/lease.
# Set up your storage account context $resourceGroupName = 'vcloud-lab.com' $storageAccountName = 'vcloudlabterraform' $containerName = 'tfstate' $blobfilename = "example.terraform.tfstate" # Release the lease $leaseId = "Lease_ID_from_terraform_state_lock_info" # The lease ID obtained when the lease was acquired # Get the storage account key $storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName).Value[0] # Create the storage context $storageContext = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey # Get the blob $blob = Get-AzStorageBlob -Context $storageContext -Container $containerName -Blob $blobfilename # Release the lease $blob.ICloudBlob.ReleaseLease($leaseId)
Below is the another way to break the lease using AzureCLI / az cli command.
# Variables RESOURCE_GROUP="vcloud-lab.com" STORAGE_ACCOUNT_NAME="vcloudlabterraform" CONTAINER_NAME="tfstate" BLOB_NAME="example.terraform.tfstate" # Get the storage account key STORAGE_ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP --account-name $STORAGE_ACCOUNT_NAME --query "[0].value" --output tsv) # Break the lease on the blob az storage blob lease break \ --account-name $STORAGE_ACCOUNT_NAME \ --account-key $STORAGE_ACCOUNT_KEY \ --container-name $CONTAINER_NAME \ --blob-name $BLOB_NAME
This is what happens when someone is already running terraform apply / destroy command and it is in the middle of deployment. You try forcefully force unlock or break the lease of tfstate file. The person put the lock will see similar error message.
│ Error: checking for presence of existing resource group: resources.GroupsClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '2a5ffc9c-6805-4f55-985d-36bf3d4f4446' with object id '2a5ffc9c-6805-4f55-985d-36bf3d4f4446' does not have authorization to perform action 'Microsoft.Resources/subscriptions/resourcegroups/read' over scope '/subscriptions/9e22fba3-00a9-447c-b954-a26fec38e029/resourcegroups/test-rg1' or the scope is invalid. If access was recently granted, please refresh your credentials."
│
│ with azurerm_resource_group.dev,
│ on Terraform_Azure_Backend_Configure.tf line 39, in resource "azurerm_resource_group" "dev":
│ 39: resource "azurerm_resource_group" "dev" {
│
╵
Releasing state lock. This may take a few moments...
╷
│ Error: Error releasing the state lock
│
│ Error message: failed to retrieve lock info: blobs.Client#GetProperties: Failure responding to request:
│ StatusCode=412 -- Original Error: autorest/azure: error response cannot be parsed: {"" '\x00' '\x00'} error: EOF
│
│ Terraform acquires a lock when accessing your state to prevent others
│ running Terraform to potentially modify the state at the same time. An
│ error occurred while releasing this lock. This could mean that the lock
│ did or did not release properly. If the lock didn't release properly,
│ Terraform may not be able to run future commands since it'll appear as if
│ the lock is held.
│
│ In this scenario, please call the "force-unlock" command to unlock the
│ state manually. This is a very dangerous operation since if it is done
│ erroneously it could result in two people modifying state at the same time.
│ Only call this command if you're certain that the unlock above failed and
│ that no one else is holding a lock.
Just for note for local terraform.tfstate file force unlock doesn't work.
Useful Articles
Create storage account and Service Principal using PowerShell for Terraform Azure Backend
Terraform variable validation example
Terraform create Azure Virtual Machines from map of objects
Terraform refactoring moved block example
Terraform create Azure Virtual Machines from map of objects
Hashicorp Terraform map and object inside module and variable example
Terraform one module deploy null or multiple resources based on input
Terraform A reference to a resource type must be followed by at least one attribute access, specifying the resource name
Terraform fore_each for loop filter with if condition example
Terraform remote-exec provisioner with ssh connection in null_resource
Terraform count vs for_each for examples with map of objects
Terraform one module deploy null or multiple resources based on input (nested for loop) Example of Terraform
