Building for Windows Server 2019 yields IMAGE_STATE_UNDEPLOYABLE

[martingem]

Description

Hello,

I've been building the images for two years using the official code, the only difference is the publish part to an internal Azure compute galleries (formerly Shared image gallery).

azure-arm: [==========================99.8%==========================]
azure-arm: [==========================100.0%==========================]
azure-arm: [==========================100.0%==========================]
azure-arm: The operation completed successfully.
azure-arm: Clean up various directories
azure-arm: Removing C:\Recovery
azure-arm: Removing C:\Windows\logs
azure-arm: Removing C:\Windows\winsxs\manifestcache
azure-arm: Removing C:\Windows\Temp
azure-arm: Removing C:\Users\packer\AppData\Local\Temp
azure-arm: Finalize-VM.ps1 - completed

==> azure-arm: Restarting Machine
==> azure-arm: Waiting for machine to restart...
==> azure-arm: A system shutdown is in progress.(1115)

azure-arm: <MachineName> restarted.
==> azure-arm: Machine successfully restarted, moving on
==> azure-arm: Provisioning with Powershell...
==> azure-arm: Provisioning with powershell script: /tmp/powershell-provisioner173743804

azure-arm: IMAGE_STATE_COMPLETE
azure-arm: IMAGE_STATE_UNDEPLOYABLE
azure-arm: IMAGE_STATE_UNDEPLOYABLE
[IMAGE_STATE_UNDEPLOYABLE Repeating for hours until the 12-hour mark!]

Of course, I tried to rebuild (as suggested by others in TechNet forums) 4 times, no success.

It seems like the capture is working (Finalize VM), but the next step yields IMAGE_STATE_UNDEPLOYABLE. Using my googling skills, I found :

• https://medium.com/nerd-for-tech/packer-sysprep-error-image-state-undeployable-on-azure-b920ad18f6b0
  unattend.xml is being removed in the packer code
• Unable to deploy packer image MicrosoftDocs/azure-docs#12156 (comment) We do not use special disks, we use the same VM sizes as you.
• https://automatingops.com/error-imagestateundeployable-with-packer-and-azure: Tried the Set-Service RdAgent -StartupType Disabled and subsequent step, but Set-Service isn't recognized by Powershell at least as is.

Let me know what kind of details could help.

Platforms affected

• Azure DevOps
• GitHub Actions

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

I'm currently at 771b501

Is it regression?

e661113

Expected behavior

Image Deployment state to be "COMPLETE"

Actual behavior

Image Deployment state is "IMAGE_STATE_UNDEPLOYABLE"

Repro steps

Building using the same https://github.com/actions/runner-images/blob/main/images.CI/linux-and-win/azure-pipelines/image-generation.yml, except for Compute image gallery as a destination.

[simbo1984]

FYI: We are having the same problem and it seems to be a regression. It's also happening with Windows Server 2022.

[ioeifcss]

Same issue for us too, on Server 2019 images.

[Mikebeem]

Same for us with Windows 2019.
Yesterday the same issue with Windows 2022, but today we didn't got this for Windows 2022.

Update:
And now it is back again for Windows 2022:

==> vhd: Machine successfully restarted, moving on
==> vhd: Provisioning with Powershell...
==> vhd: Provisioning with powershell script: C:\Users\AzDevOps\AppData\Local\Temp\powershell-provisioner609395526
    vhd: IMAGE_STATE_COMPLETE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    vhd: IMAGE_STATE_UNDEPLOYABLE
    .......

[TLaborde]

We are having the same issue. Our SOC informed us that Microsoft Defender was finding suspicious script in the packer build. It was a false positive (pypy install script) but could that be putting the VM in a state that cannot be deployed?

[markatky46]

Had the exact same. For me my install/packer password used by WinRM to perform the build was not over 14 characters and did not include special characters. Changed in my script to match how MS are creating a password and passing it to packer and all is now good. Image builds in 4.5 hours.

[ioeifcss]

That's the install_password variable you're referring to, right?

[ioeifcss]

Looking at this repo, MS are generating the password as follows:

$InstallPassword = $env:UserName + [System.GUID]::NewGuid().ToString().ToUpper()

In my own repo, I'm generating the password as follows:

$InstallPassword = [System.GUID]::NewGuid().ToString().ToUpper()

The only difference is prepending the username at the start, so I'm honestly surprised that could affect the image state so drastically.

[markatky46]

I was surprised too. This was my issue though. The password being generated wasn't over 14 characters and it didn't meet the complexity of lowercase, uppercase and special chars,. I also found people saying that if the password contained % it was failing. I spent 2 weeks debugging and retrying this. The fix was to generate a random 20 char password with upper case, lowercase and !$ chars.

[SagiKovaliov]

Is there a solution for this? Getting the same error IMAGE_STATE_UNDEPLOYABLE on both, 2019 and 2022 images.
Getting this on the latest versions and previous tags as well.

[SagiKovaliov]

@martingem Have you found a solution ? I have the same issue.

[martingem]

Hello @SagiKovaliov, definitely not. I'm strongly considering changing this discussion back to an issue. It's not just me and should be considered as an issue, not a mere discussion.

Update 18-11-2022, I change my mind... obviously now we know it's Azure Defender, specifically MDE that's doing something.

[ioeifcss]

Not sure if it's related, but our subscriptions have Microsoft Defender for Servers (plan 2) enabled.

I turned this off and kicked off a new image build earlier this morning.

2022-09-26T08:49:30.1923126Z ==> vhd: Machine successfully restarted, moving on
2022-09-26T08:49:30.1934702Z ==> vhd: Provisioning with Powershell...
2022-09-26T08:49:30.1938884Z ==> vhd: Provisioning with powershell script: /tmp/powershell-provisioner987556087
2022-09-26T08:49:34.7810878Z     vhd: IMAGE_STATE_COMPLETE
2022-09-26T08:49:44.8020780Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:49:54.8361406Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:50:04.8489729Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:50:14.8604959Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:50:24.8600578Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:50:34.8608466Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:50:44.8710649Z     vhd: IMAGE_STATE_UNDEPLOYABLE
2022-09-26T08:51:54.5824585Z ==> vhd: Querying the machine's properties ...
2022-09-26T08:51:54.8206799Z ==> vhd: Querying the machine's additional disks properties ...
2022-09-26T08:51:54.8648290Z ==> vhd: Powering off machine ...
2022-09-26T08:52:41.2020544Z ==> vhd: Capturing image ...
2022-09-26T08:53:11.5269624Z ==> vhd: 
2022-09-26T08:53:11.5272313Z ==> vhd: Deleting individual resources ...

What's different here? Instead of the build screaming IMAGE_STATE_UNDEPLOYABLE for 12 hours like it's done in the past, it repeated itself only 7 times before moving on.

No idea if that means the image is still defective - I guess I'll find out soon?

---

UPDATE: Created a new agent pool which appears to run jobs without any issues with this new image.

Can everyone else on this thread check if they have Defender for Cloud enabled in their environment?

[martingem]

@ioeifcss I definitely can't test that, since in my - and the IT I worked with - understanding you need to disable it on a subscription level.
Also creating under a different agent pool is not really a good outcome. It'll work for two years, then what? I'd prefer to find the root cause.

[ioeifcss]

To be clear, I tested the image on a new test pool, and once I verified it was working there, I ensured my existing pools were updated to use that new image too.

[drazkiewicz]

I did disable Microsoft Defender for Cloud on subscription and it is working without any problem. It run already twice and - guess what - IMAGE_STATE_UNDEPLOYABLE appeared just once and image was captured afterwards without any issue.

[tonswart]

Just checked, Microsoft Defender for Cloud on subscription is enabled and build times out with IMAGE_STATE_UNDEPLOYABLE messages.

[tonswart]

I did disable Microsoft Defender for Cloud on the subscription and BOOM "IMAGE_STATE_COMPLETE". Still waiting for a better solution though.

[Mikebeem]

I found the following error in the sysprep logs:

Error                 SYSPRP ActionPlatform::DeleteValue: Error from RegDeleteValueW for value senseGuid under key HKEY_LOCAL_MACHINE\Software\Microsoft\Windows Advanced Threat Protection; dwRet = 0x5
Error                 SYSPRP SysprepSession::ExecuteAction: Failed during deleteValue operation; dwRet = 0x5
Error                 SYSPRP SysprepSession::ExecuteInternal: Error in executing action for Windows-SenseClient-Service; dwRet = 0x5
Error                 SYSPRP SysprepSession::Execute: Error in executing actions from C:\Windows\System32\Sysprep\ActionFiles\Generalize.xml; dwRet = 0x5
Error                 SYSPRP RunPlatformActions:Failed while executing Sysprep session actions; dwRet = 0x5
Error      [0x0f0070] SYSPRP RunDlls:An error occurred while running registry sysprep DLLs, halting sysprep execution. dwRet = 0x5
Error      [0x0f00a8] SYSPRP WinMain:Hit failure while processing sysprep generalize internal providers; hr = 0x80070005

Is there another way around this than disabling Defender for Cloud?

[drazkiewicz]

Good catch. I had the same error - just forget to post it here.

[McAdamz]

This issue is occurring for me too, Windows 2022.

==> vhd: Provisioning with powershell script: C:\Users\AzDevOps\AppData\Local\Temp\powershell-provisioner1816835113
vhd: IMAGE_STATE_COMPLETE
vhd: IMAGE_STATE_UNDEPLOYABLE
vhd: IMAGE_STATE_UNDEPLOYABLE
vhd: IMAGE_STATE_UNDEPLOYABLE
...

[markatky46]

Also seeing the same issues on Windows 2022

[angaaruriakhil]

Can confirm, having the same issue on Windows 2022.

[martingem]

@al-cheb I'll make a ticket with Azure to try to understand what can cause this, but I would think that the generation would fail as well for Microsoft, no?

Anyway, for all the others that replied, thanks. I'll post an update whenever I get a response and see where we go from here.

[martingem]

@markatky46 As @al-cheb said here: #6251 (reply in thread), they do not use it. At least they do not have this problem, but it's interesting that MS doesn't believe they need it.
In the meanwhile, I'd recommend that if you have a subscription for Visual Studio Pro (with GitHub Enterprise or not) to use it - or you could create a subscription for building agents. It's like 4$ to build an agent, so you can keep on going there while Defender solves their issues. If ever.

[markatky46]

@martingem how would a Visual Studio Pro sub help in this instance?

[martingem]

@markatky46 if Defender is not already disabled, to disable it for your own benefits-based subscription, which at "no cost", would allow you to work-around for now. I'm not saying it's a permanent solution, merely an option. You can also create or ask your Tenant owner to create a subscription with Defender disabled.
Again, these are just options meant for a work-around until this is resolved.

[martingem]

Note to everyone watching this, I learned how to use the MDE Analyzer and sent the results of UNDEPLOYABLE to Microsoft Support. Also (FYI), they asked me to try to "force" the state to complete in Regedit while the issue occurs, it didn't change much but I followed the instructions.

I'll keep you posted on my ticket.

[martingem]

Note to everyone watching this, I was suggested to use REST APIs to disable/offboard: https://learn.microsoft.com/en-us/microsoft-365/security/defender-endpoint/offboard-machine-api?view=o365-worldwide

Then you could potentially re-onboard it after the sysprep (which doesn't really makes sense - at that point the build is over).

Why I'm not suggesting this is the answer: if you guys feel that it's a valid option, you can apply it. On my end, I'll do it to diagnose, but I don't think I (the consumer of this repo), nor the maintainers of this repo should have something to do. Otherwise it can lead to more complicated setup (to have a authentication token, yet another setup to make this repo work for your needs).

So what I'd say is that as long as your IT department is understanding of the situation, I'd wait for me and the support (alongside others that are doing this also) to get through the diagnosing part.
If the offboard machine works and Microsoft deems this the solution, I'd have to see how this repo maintainers sees this.

It's either that the repo "can handle the case", or it won't.. and people will have to fork the repo always if they use MDE.

I'll keep you posted.

[simpleprovider]

We start having the same issues.

[ghost]

Is this still an issue with the WIN 2019 and WIN 2022 images?

[tomvo12]

yes, unfortunately it is, we are getting the same error for both, win19 and win22 in a subscription w/ cloud defender turned on

[elSagie]

all those that got "IMAGE_STATE_UNDEPLOYABLE" are having Defender for Cloud Servers plan turned on?

[ioeifcss]

That's right

[toddwalstad-eaton]

Yes, that is correct. It is from the Defender "Servers" setting. I was able to turn this off (not sure my company should allow this) and was able to get a build. Up till about a week ago, I was able to get past this by putting in some exceptions for the alerts it was finding. This stopped working and now I have to turn off this "Servers" setting to get a build.

For those saying use your own subscription, for some of us that is not allowed and could cause us to get fired.

[elSagie]

Could you verify that the image does not contain MMA/AMA or active MDE agent (SENSE)?

[McAdamz]

Could you verify that the image does not contain MMA/AMA or active MDE agent (SENSE)?

Where do I verify that exactly? In the Extensions + applications area?

[egg-r]

@elSagie I've started troubleshooting this as been seeing the described error for Win2022 image creations. I see the MDE.Windows extension being added. We have Defender for Cloud enabled and has been enabled for some time now. My last image generation was in August and that was successful.

[elSagie]

thank you all. we're checking this

[elSagie]

As @egg-r hinted, MDE integration is suspected to interfere with the process.
Please uncheck MDE integration: Microsoft Defender for Cloud - Environment settings - Settings | Integrations - Allow Microsoft Defender for Endpoint to access my data [ uncheck ] and retry the image generalization process

[PriceChild]

Hitting 'restart' on the VM when it's in this state seems to allow the packer build to complete successfully. I haven't confirmed whether the image is good yet.

[greha]

Tried this yesterday, image was not good.

[McAdamz]

@elSagie I have unticked as you specified and now the Win2022 build has finally completed.

Obviously this cannot be a permanent solution.

[elSagie]

@McAdamz Thanks for the repro and evidence, Fix WIP...

[egg-r]

I was able to get the image created too by disabling MDE. Just noting that MDE extension stopped being auto-added to new VMs the day after it was disabled on the subscription in my case.

[markatky46]

For now to resolve our issues we have disabled the DfC integration in Azure portal for our image build subscriptions

[toddwalstad-eaton]

@elSagie . @al-cheb any updates on this item, I know the workarounds are working, but I am getting pressure to re-enable these settings.

[martingem]

FYI the product team (Azure Support) is looking into my logs. Certainly it has more to do with Defender than this repo (i.e. @al-cheb ). But if the answer/resolution has to be documented, it would be good to then add it to the README.md if needed at all (if it's a bug that they resolve.. good. If we have to do special stuff, it's another).

[Neter56]

Hi people,
It appears that you are generalizing an image from a VM that is already onboarded to MDE. We believe that sysprep is failing when trying to remove MDE protected registry key or disable the agent, or both. It's not conclusive yet, engineering still looking for certain root-cause.
With that here's my advice: make sure to offboard the VM from MDE prior to generalizing its image!

To me it makes sense - there is no way you can deploy new VM to be onboarded to MDE upon deployment - MDE onboarding must be done by their script only.

[djh82uk]

Does anyone know of a programmatic way of disabling the integration (AZ CLI etc) so that it could be disabled at the start of a build, and re-enabled afterwards?

[martingem]

@djh82uk I was asked by MS Support to "do it" when I told them that I need it to be in a programmatic way, but I re-asked and said that I didn't know how to do it [today].

This shouldn't be the solution, it basically implies that anyone could disable it if they have the know-how. But for diagnosis purposes, I'll try that with them.

[elSagie]

Does anyone know of a programmatic way of disabling the integration (AZ CLI etc) so that it could be disabled at the start of a build, and re-enabled afterwards?

check below: #6251 (comment)

[vamshisiram]

I also have the same issue. Please assist...

[Tw1npe4ks]

We have same issue also with Windows Server images. It will repeat "IMAGE_STATE_UNDEPLOYABLE" for 12 hours.

[elSagie]

To disable the MDE integration (Setting - Integration - Allow Microsoft Defender for Endpoint to access my data) programmatically use REST API.
PUT https://management.azure.com/subscriptions/<Subscription ID>/providers/Microsoft.Security/settings/WDATP?api-version=2021-06-01
with json body:

{
	kind: "DataExportSettings",
    "properties": {
        "enabled": false
     }
}

[vamshisiram]

we have the same issue, any suggestions please other than what was posted above. We have Defender for cloud enabled since we started using this repo for more than a year, never face this issue until that last two months.

tagging @mikhailkoliada @igorboskovic3 @Alexey-Ayupov for some help with these, as these have been very helpful in resolving some of the issues in the past. Appreciate all the help.

[vamshisiram]

@PapaDers

i am creating a image, but i am pushing it to a Share Image Gallery. I was successful in the past, but not right now, when i try to do this for windows, i get image not prepared properly.

I use the shared image gallery option because we create VMSS from the Image in the Shared Image gallery across multiple devops orgs as agent pools.

should i not use the shared image gallery option?

I am unable to run the post generation scripts on the VM that was created by the build process. Is it mandatory to run the post generation scripts on the VM created by the build process before converting it to an image?

[PapaDers]

@vamshisiram - I am sorry. I assumed something else was happening.

I also use a shared image gallery to create image versions for our VMSS. I searched online and found a few others who have had similar issues creating a gallery image version from a VM. The solution was to generate a new image.

It has been a while, but I saw this once, and the solution for me was to choose to capture managed image instead. Then I used the managed image to generate the image version in the gallery and then deleted the managed image.

[vamshisiram]

Let me give that a try.

How did you run the post generate script?

Or

Can we skip that?

@PapaDers

[vamshisiram]

@PapaDers this is the error i get, no matter how i create the VMSS

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"OSProvisioningClientError","message":"OS Provisioning for VM 'vmss-winpool-jan2023_0' did not finish in the allotted time. However, the VM guest agent was detected running. This suggests the guest OS has not been properly prepared to be used as a VM image (with CreateOption=FromImage). To resolve this issue, either use the VHD as is with CreateOption=Attach or prepare it properly for use as an image:\r\n * Instructions for Windows: https://azure.microsoft.com/documentation/articles/virtual-machines-windows-upload-image/ \r\n * Instructions for Linux: https://azure.microsoft.com/documentation/articles/virtual-machines-linux-capture-image/ "}]}

i see no error when creating the managed image or the image version in the shared image gallery, but the VMSS just fails.

i did create a new build process, new VM and then captured it as a managed image, then added an image version from the managed image into the image gallery. Nothing works :-( . so, painful to create a windows image and get it working.

thanks for chatting it out with me. @PapaDers

[vamshisiram]

@PapaDers

any other suggestions, i get the same error for windows 2019 and windows 2022, no matter which way i try.

[simpleprovider]

Hi guys,

I also have this issue,
I have an open ticket with MS and currently this is what I am trying:
Add the following tag to your VM which is supposed to disable the MDE activation for the VM

Tag name: ExcludeMdeAutoProvisioning Tag value: True

You could achieve it by adding this in your packer json file in the azure-arm builder.
@martingem maybe this can be a solution for you as well as you don't need to change anything at subscription level.
"azure_tags": { "ExcludeMdeAutoProvisioning": "True" },

I tried until now one build and it worked. 👯‍♂️ 😄

[McAdamz]

I've also finally managed to get a win2022 build running by adding that tag to the build script.
I use Azure Dev ops to build the image and it runs on a schedule once a month to keep it up to date.

I haven't forked the repo as I didn't want to have to keep syncing, so I set up my YAML file to just clone it and added a new Powershell step to insert the ""azure_tags": { "ExcludeMdeAutoProvisioning": "True" }," tag to the build script.
Yes, it's a dirty hacky workaround, but simple to remove and clear as to what is going on:

 
  - task: PowerShell@2
    displayName: 'Workaround for issue: https://github.com/actions/runner-images/discussions/6251'
    inputs:
      targetType: 'inline'
      script: |        
        $TemplateFile = "virtual-environments/images/win/${{ parameters.image_type }}.json"
        Write-Host 'Attempting to modify $TemplateFile with workaround'
        ((Get-Content -path $TemplateFile -Raw) -replace '"type": "azure-arm",','"type": "azure-arm","azure_tags": { "ExcludeMdeAutoProvisioning": "True" },') | Set-Content -Path $TemplateFile               
        Write-Host "$TemplateFile modified"
        Get-Content -path $TemplateFile -Raw

I'm now happy to finally have the latest image successfully run!

[erik-de-bont]

@McAdamz If test your solution and it solved my issue too. I have made a pull request to have this settings added to the json files. #7043

[erik-de-bont]

Unfortunately the pull request has been rejected.

[jaswant-dnv]

McAdamz don't see Json but pkr.hcl and unable to find attributes.
Any suggestion how we can set azure_tags.

[baidarka]

PR #10884 : Adds parameter 'AzureTags' to 'build-image.ps1'. Depending on the way you build, enables you to pass Azure tags for the temporary Packer VM from within your pipeline.

[vamshisiram]

I am also getting successful build, but when I convert the vm into a managed image or a image in shared image gallery, it works fine. But the vm I create using the managed image or the shared gallery image is failing with the message "image not prepared properly use as is " I tried using it as is, i am not even running the post generation scripts on the vm and still getting same error, that image not prepared properly. Any help pls? Adam, can you please tell me how you opened an Ms ticket? I mean to which team , windows vm team or GitHub team or which one? Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: McAdamz ***@***.***> Sent: Thursday, February 2, 2023 12:06:12 PM To: actions/runner-images ***@***.***> Cc: vamshisiram ***@***.***>; Mention ***@***.***> Subject: Re: [actions/runner-images] Building for Windows Server 2019 yields IMAGE_STATE_UNDEPLOYABLE (Discussion #6251) Hi guys, I also have this issue, I have an open ticket with MS and currently this is what I am trying: Add the following tag to your VM which is supposed to disable the MDE activation for the VM Tag name: ExcludeMdeAutoProvisioning Tag value: True You could achieve it by adding this in your packer json file in the azure-arm builder. @martingem<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmartingem&data=05%7C01%7C%7Cbb475392d4d047f1321608db0558ef4a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638109651774990107%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1TMAHZC3J7bhHD7LbywstfZgD%2FEi9XSI9MYVBAeIjsU%3D&reserved=0> maybe this can be a solution for you as well as you don't need to change anything at subscription level. "azure_tags": { "ExcludeMdeAutoProvisioning": "True" }, I tried until now one build and it worked. 👯‍♂️ 😄 I've finally managed to get a win2022 build running by adding that tag to the build script. I use Azure Dev ops to build the image and it runs on a schedule once a month to keep it up to date. I haven't forked the repo as I didn't want to have to keep syncing, so I set up my YAML file to just clone it and added a new Powershell step to insert the ""azure_tags": { "ExcludeMdeAutoProvisioning": "True" }," tag to the build script. Yes, it's a dirty hacky workaround, but simple to remove and clear as to what is going on: - task: ***@***.*** displayName: 'Workaround for issue: #6251' inputs: targetType: 'inline' script: | $TemplateFile = "virtual-environments/images/win/${{ parameters.image_type }}.json" Write-Host 'Attempting to modify $TemplateFile with workaround' ((Get-Content -path $TemplateFile -Raw) -replace '"type": "azure-arm",','"type": "azure-arm","azure_tags": { "ExcludeMdeAutoProvisioning": "True" },') | Set-Content -Path $TemplateFile Write-Host "$TemplateFile modified" Get-Content -path $TemplateFile -Raw I'm now happy to finally have the latest image successfully run! — Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Factions%2Frunner-images%2Fdiscussions%2F6251%23discussioncomment-4855361&data=05%7C01%7C%7Cbb475392d4d047f1321608db0558ef4a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638109651775146325%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xA1f6i4Rg1GUEYkx3zGgdTq6fuenObdSo%2FIvDoMeaqc%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVU36WUVLUKNJZ3ZQTIS7V3WVQHTJANCNFSM6AAAAAAQRFHRAM&data=05%7C01%7C%7Cbb475392d4d047f1321608db0558ef4a%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638109651775146325%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=I7%2BvufU%2F7pI33rrl1QQc%2BiExlqjBUoRHxFNAl%2FMGiLc%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Alexey-Ayupov]

Hello @simpleprovider, obviously this tag ExcludeMdeAutoProvisioning belongs to one of the Azure policy definitions, assigned to your subscription. Can you check what exact policy is in charge of provision MDE and contains this tag?

[simpleprovider]

@Alexey-Ayupov : No, this tag was provided from Microsoft support as a workaround for now. It will be made public in case another solution will not be implemented, like sysprep to deal with the issue. I didn't create any policy for this.

[Alexey-Ayupov]

@simpleprovider, I got it the solution was provided by Microsoft support. However, I would like to understand the nature of this tag. Probably you have enabled a service plan for your VMs in Microsoft Defender for Cloud. And special Azure policy could be applied automatically to provision MDE on your VMs. I'm just asking you to check what policies are assigned and have the words "MDE" or "defender" in their names.

[simpleprovider]

@Alexey-Ayupov It was not enforced by Azure Policy , it was decided at company level that Microsoft Defender for Cloud should be enabled.
I could have uncheck the setting in the image below and the image creation would have worked.

By the way, now this option was removed from here and it is a new entry like in picture below "Endpoint protection".

[lzandman]

Add the following tag to your VM which is supposed to disable the MDE activation for the VM
Tag name: ExcludeMdeAutoProvisioning. Tag value: True.

Amazing hidden gem! Thanks! I wonder what other "special" tags there are that allow us to tame the magic of Azure... ;-)

Also, this seems like a rather big security issue. Anyone able to add this tag to a VM resource can bypass MS Defender for Cloud...