Azure Function App problems – Error indexing method

with No Comments

Today I came to work and noticed that one of our Azure Function Apps suddenly had stopped working. No code or any changes had been made to the app for several weeks, yet somehow the app failed to run.

 

The problem

In AppInsights we could see that our function app produced the following error for every function:

 

Ok so this was weird since we hadn’t made any changes to the code. There hadn’t been any changes to the functions app either. Utter dispair set in. I called in Anders Roos for help – and we started googling…

Turns out google wasn’t of so much help today. The only hint I could find was that someone had a similar issue a year ago with mismatching dependencies between Azure Storage assemblies and Azure Function SDK assemblies.

 

Troubleshooting

We tried to run the app locally. It worked.

 

I then fired up VS Code to inspect the code (which worked flawlessly yesterday) – it prompted me to upgrade the Azure Function Core Tools which is the toolset used to run and debug functions locally. Hmm…

It turns out that this new version included a new version of the Azure Function hosting runtime. Starting the function app locally with this new version produced a similar error!

 

We tried downgrading the tools the older version – the app worked.

So what we did was compare the output of the different tool versions. We could see that the new version explicitly checked the “extensionBundle”-property set in my hosts.json file. It then proceeded to downloaded version 1.1.1. This extensionBundle is just a .json-file with a list of references to other NuGet assemblies.
After some digging in the extension bundle repo, we found that version 1.1.0 of this release bumped minor versions of assembles we used. According to the internet, there seems to be a version mismatch between some of these assemblies (they do not play well together). I found one older GitHub issue where this was confirmed to be a regression in one of the packages.

 

The solution

The crux of the problem was that the older runtime (which ran yesterday) apparently did not check or at least did not download the new version of the extensionBundle. So the old version used 1.0.0 (which contains stable versions of the assembles) and the new version of the runtime downloaded 1.1.1 – which contains the regression.

 

After some laboration, our current work around is to fix the version of the extensionBundle to 1.0.0 and set the upper bound of the version range to 1.0.1. This prevents all versions except 1.0.0.

 

So before our hosts.json looked like this:

 

Now it looks like this:

 

I published an issue about this here: https://github.com/Azure/azure-webjobs-sdk/issues/2380

Leave a Reply