Managing large number of application settings in Azure Functions
up vote
0
down vote
favorite
I currently have an application that processes telemetry events from Azure IOT Hub. The processing of the events are done by triggering Azure Functions from the IOT Hub.
Several processing rules /settings (currently about 6) are set up per device. Settings are essentially just a set of simple (string/bool) properties. Seeing that there are several hundred devices, the qty of these settings explode over time. On-boarding new devices also creates a requirement for provisioning the settings and defaults.
When the Azure Function starts up to process events from a device, it needs to load the settings for that device and then process accordingly.
What is the best way to manage these settings? The primary focus is on efficient loading of the settings when the function starts up. Secondarily, the easy update of settings.
As far as I can think there are the following options:
Store individual settings in Azure Tables. I have implemented this but I am concerned that it will not scale well when hundreds of devices access the Azure Table. I have used PartitionKey|RowKey as DeviceId|SettingName. This will create a partition per device
Store all settings for a device as JSON in Azure tables. This will require parsing JSON to obtain the individual settings
Storing individual settings in ApplicationSettings is not really doable I think due to number of settings
Storing all settings for a device I think is also not doable due to the same reason as above.
azure-storage azure-functions asked 2 days ago
user437239
599
add a comment |
up vote
0
down vote
favorite
I currently have an application that processes telemetry events from Azure IOT Hub. The processing of the events are done by triggering Azure Functions from the IOT Hub.
Several processing rules /settings (currently about 6) are set up per device. Settings are essentially just a set of simple (string/bool) properties. Seeing that there are several hundred devices, the qty of these settings explode over time. On-boarding new devices also creates a requirement for provisioning the settings and defaults.
When the Azure Function starts up to process events from a device, it needs to load the settings for that device and then process accordingly.
What is the best way to manage these settings? The primary focus is on efficient loading of the settings when the function starts up. Secondarily, the easy update of settings.
As far as I can think there are the following options:
Store individual settings in Azure Tables. I have implemented this but I am concerned that it will not scale well when hundreds of devices access the Azure Table. I have used PartitionKey|RowKey as DeviceId|SettingName. This will create a partition per device
Store all settings for a device as JSON in Azure tables. This will require parsing JSON to obtain the individual settings
Storing individual settings in ApplicationSettings is not really doable I think due to number of settings
Storing all settings for a device I think is also not doable due to the same reason as above.
azure-storage azure-functions asked 2 days ago
user437239
599
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I currently have an application that processes telemetry events from Azure IOT Hub. The processing of the events are done by triggering Azure Functions from the IOT Hub.
Several processing rules /settings (currently about 6) are set up per device. Settings are essentially just a set of simple (string/bool) properties. Seeing that there are several hundred devices, the qty of these settings explode over time. On-boarding new devices also creates a requirement for provisioning the settings and defaults.
When the Azure Function starts up to process events from a device, it needs to load the settings for that device and then process accordingly.
What is the best way to manage these settings? The primary focus is on efficient loading of the settings when the function starts up. Secondarily, the easy update of settings.
As far as I can think there are the following options:
Store individual settings in Azure Tables. I have implemented this but I am concerned that it will not scale well when hundreds of devices access the Azure Table. I have used PartitionKey|RowKey as DeviceId|SettingName. This will create a partition per device
Store all settings for a device as JSON in Azure tables. This will require parsing JSON to obtain the individual settings
Storing individual settings in ApplicationSettings is not really doable I think due to number of settings
Storing all settings for a device I think is also not doable due to the same reason as above.
azure-storage azure-functions asked 2 days ago
user437239
599
I currently have an application that processes telemetry events from Azure IOT Hub. The processing of the events are done by triggering Azure Functions from the IOT Hub.
Several processing rules /settings (currently about 6) are set up per device. Settings are essentially just a set of simple (string/bool) properties. Seeing that there are several hundred devices, the qty of these settings explode over time. On-boarding new devices also creates a requirement for provisioning the settings and defaults.
When the Azure Function starts up to process events from a device, it needs to load the settings for that device and then process accordingly.
What is the best way to manage these settings? The primary focus is on efficient loading of the settings when the function starts up. Secondarily, the easy update of settings.
As far as I can think there are the following options:
Store individual settings in Azure Tables. I have implemented this but I am concerned that it will not scale well when hundreds of devices access the Azure Table. I have used PartitionKey|RowKey as DeviceId|SettingName. This will create a partition per device
Store all settings for a device as JSON in Azure tables. This will require parsing JSON to obtain the individual settings
Storing individual settings in ApplicationSettings is not really doable I think due to number of settings
Storing all settings for a device I think is also not doable due to the same reason as above.
azure-storage azure-functions azure-storage azure-functions asked 2 days ago
user437239
599
asked 2 days ago
user437239
599
asked 2 days ago
user437239
599
asked 2 days ago
user437239
599
asked 2 days ago
user437239
599
599
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday
add a comment |
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
It sounds like these settings may change, which means you either need to:
- Read them every time a function starts.
- Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
- Send the device settings with your messages -- is this a possibiliy, or are those managed separately?
As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.
answered 2 days ago
brettsam
1,875415
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
It sounds like these settings may change, which means you either need to:
- Read them every time a function starts.
- Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
- Send the device settings with your messages -- is this a possibiliy, or are those managed separately?
As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.
answered 2 days ago
brettsam
1,875415
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
add a comment |
up vote
1
down vote
accepted
It sounds like these settings may change, which means you either need to:
- Read them every time a function starts.
- Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
- Send the device settings with your messages -- is this a possibiliy, or are those managed separately?
As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.
answered 2 days ago
brettsam
1,875415
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
It sounds like these settings may change, which means you either need to:
- Read them every time a function starts.
- Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
- Send the device settings with your messages -- is this a possibiliy, or are those managed separately?
As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.
answered 2 days ago
brettsam
1,875415
It sounds like these settings may change, which means you either need to:
- Read them every time a function starts.
- Read them once, and store them in memory. Then you'd need an expiration policy to refresh them if they've changed (unless you have some way to be notified of changes). Perhaps something like MemoryCache.
- Send the device settings with your messages -- is this a possibiliy, or are those managed separately?
As for where they're stored -- using Azure Tables or even Azure Blobs would work. You could store JSON settings in a blob named by the device id and retrieve those settings very quickly. Storing them in something like CosmosDB (which is a JSON document store already) would also work. All of these should scale for you, especially if you're doing some kind of in-memory caching so you don't need to read settings every invocation.
answered 2 days ago
brettsam
1,875415
answered 2 days ago
brettsam
1,875415
answered 2 days ago
brettsam
1,875415
answered 2 days ago
brettsam
1,875415
1,875415
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
add a comment |
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
The Azure Functions are stateless so currently I read them with every invocation. I could make the settings static but then I would need to bounce the Function App when I change a setting. Settings are changed manually when the processing needs to deviate from the normal for diagnostics purposes. So I suppose that is not too bad. The settings pertain to backend processing so no real need for the settings on the devices. As long as Azure Table will scale properly it is my preference. The JSON is fairly small so parsing is likely not a big overhead at scale.
– user437239
yesterday
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53203835%2fmanaging-large-number-of-application-settings-in-azure-functions%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
How often do these settings change? Is it a problem if settings got stale for a while? You need some sort of caching there. Depending on the frequency of change and other parameters you can try to find the best fitting cache option for you.
– Sebastian Achatz
2 days ago
Settings do not change often. Mainly for diagnostics when needed
– user437239
yesterday