Celerity Workflow
Celerity resource type for workflows
celerity/workflow
Spec Version: v2026-02-27-draft
Ecosystem Compatibility: v1 (Preview)
blueprint transform: celerity-2026-02-27-draft
Ecosystem Compatibility
- v1 (Preview): Multi-cloud, with every workflow run by the Celerity workflow runtime as a containerised service on every target environment, across all cloud providers and all language runtimes.
Read more about Celerity versions here.
🚀 Planned for v1 - Workflow support is planned for v1, running on the Celerity workflow runtime as a containerised service across all cloud providers and all language runtimes.
The celerity/workflow resource type is used to define a workflow that orchestrates the execution of multiple handlers in a blueprint as a series of steps.
Every workflow is run by a single implementation, the Celerity workflow runtime, in every environment. Workflows are not mapped to cloud provider workflow services such as AWS Step Functions, Google Cloud Workflows or Azure Logic Apps. The workflow runtime is a long-running containerised service that exposes the workflow API. Workflows are deployed with the containerised aws, gcloud and azure deploy targets. As with other application resource types, compute configuration determines whether the runtime runs on a container orchestrator such as Amazon ECS on EC2, Amazon EKS, GKE or AKS, or on serverless container infrastructure such as AWS Fargate, Google Cloud Run or Azure Container Apps. On Google Cloud Run and Azure Container Apps, the runtime can also scale to zero, see Hosting Profiles for more information.
Because there is one implementation, a workflow behaves the same way on every target environment. Every state type, template function and condition function is available everywhere, and the error structure and JSON path behaviour documented on this page apply to every target.
Why workflows can't use serverless deploy targets
For celerity/api, celerity/consumer and celerity/schedule, a cloud provider's native service is a trigger that gets events to your handlers, which works much the same way on every platform. For workflows, the native service would be the execution engine, owning control flow, state persistence, retries, error handling and data transformation between steps. That is where the platforms differ most, so the same workflow would behave differently depending on where it was deployed. Running the Celerity workflow runtime everywhere avoids this, at the cost of requiring a container service and a PostgreSQL database.
Workflows can be used to define complex logic that requires multiple steps to be executed in a specific order. Handlers can be used for states which have the type executeStep. Workflows can also be used to define error handling and retries for each step to create robust and fault-tolerant applications.
Specification
The specification is the structure of the resource definition that comes under the spec field of the resource in a blueprint.
The rest of this section lists fields that are available to configure the celerity/workflow resource followed by examples of different configurations for the resource type, a section outlining the behaviour in supported target environments along with additional documentation.
The specification for a workflow is influenced by the concept of a workflow as a state machine inspired by Amazon's States Language created for AWS Step Functions.
startAt (required)
The name of the state used to begin execution of the workflow.
type
string
states (required)
A mapping of state names to state configurations that make up the state machine of the workflow. A state configuration resembles a state in a state machine which describes decisions, transitions or the step to be executed when the state is entered.
type
mapping[string, state]
Annotations
Annotations define additional metadata that can determine the behaviour of the resource in relation to other resources in the blueprint or to add behaviour to a resource that is not in its spec.
celerity/vpc 🔗 celerity/workflow
The following are a set of annotations that determine the behaviour of a workflow in relation to a VPC. Appropriate security groups are managed by the VPC to workflow link.
Workflows are always deployed as containerised services, so VPC annotations apply to every target environment that supports workflows. How each annotation is honoured varies by container platform; see the Target Environments section for details.
When a VPC is not defined, the default VPC for the account will be used.
Warning
When a VPC is not defined, annotations in the celerity/workflow will apply to the default VPC for the account.
celerity.workflow.vpc.subnetType
The kind of subnet that the workflow should be deployed to.
type
string
allowed values
public | private
default value
public - When a VPC links to a workflow, the workflow will be deployed to a public subnet.
celerity.workflow.vpc.lbSubnetType
The kind of subnet that the load balancer for the workflow should be deployed to. This is only relevant when the workflow is deployed to an environment that requires load balancers to interact with workflows via the API.
type
string
allowed values
public | private
default value
public - When a VPC links to a workflow, the load balancer for the workflow will be deployed to a public subnet.
Outputs
Outputs are computed values that are accessible via the {resourceName}.spec.* field accessor in a blueprint substitution.
For example, if the resource name is myWorkflow, the output would be accessible via ${myWorkflow.spec.id}.
id
The ID of the workflow deployed to the Celerity workflow runtime.
type
string
examples
691802b3-351e-4599-8785-2353ce970613
Data Types
state
FIELDS
type (required)
The type of state for the workflow.
executeStepis for a state that executes a handler.passis for a state that passes the input to the output without doing anything, a pass step can inject extra data into the output.parallelis for a state that executes multiple steps in parallel. Depending on the target environment, parallel execution may be limited by available compute resources. Read more about the limitations of parallel execution in the Celerity workflow runtime.waitis for a state that waits for a specific amount of time before transitioning to the next state.decisionis for a state that makes a decision on the next state based on the output of a previous state.failureis for a state that indicates a specific failure state in the workflow, this is a terminal state.successis for a state that indicates the end of the workflow, this is a terminal state.
field type
string
allowed values
executeStep | pass | parallel | wait | decision | failure | success
description
A human-readable description for the state.
field type
string
inputPath
The input path to use as the root input for the state. This is useful for extracting a subset of the input object to use in the current state. The supported path syntax is described in the Path Syntax section.
When not set, the input to the state is the entire input object from the previous state's output or from the initial input for the workflow.
Input path is only supported for executeStep, pass, wait, decision, success and parallel state types.
field type
string
examples
$.metadata
resultPath
The result path determines the location in the output object where the result of the state will be placed. This is useful for merging results into the input object instead of replacing it. The supported path syntax is described in the Path Syntax section.
The JSON Path Injections section provides more information as to how JSON paths behave in this context.
For example, for an input object:
{
"value": 10
}A resultPath of $.result would result in the following output object:
{
"value": 10,
"result": "some result"
}When not set, the output of the state will be the entire output object passed into the next state.
This field should not be set if the outputPath field is set.
Result path is only supported for executeStep, pass and parallel state types.
field type
string
examples
$.sum
$.result
outputPath
The output path determines the location of a value of the result to be passed to the next state. This is useful for extracting a subset of the result object to pass to the next state. The supported path syntax is described in the Path Syntax section.
For example, for an input object:
{
"id": "82612fc2-dc3f-4fee-a65b-a8721aec71b1",
"metadata": {
"value": 10
}
}An outputPath of $.metadata would result in the following output object:
{
"value": 10
}When not set, the output of the state will be the entire result object passed into the next state.
This field should not be set if the resultPath field is set.
Output path is only supported for executeStep, pass, wait, decision, success and parallel state types.
field type
string
examples
$.sum
$.result
payloadTemplate
A template for the payload to be passed into the handler configured to be executed for the current state.
When not set, the entire input object from the previous state's output or from the initial input for the workflow will be passed as the payload to the handler.
A payload template is a mapping structure that can be used to construct a payload. The payload template can contain literal values, nested mappings, arrays, paths to values in the input object and functions applied to input values.
See the Payload Template section for more information on how to construct a payload template.
This field is only supported for executeStep, pass, and parallel state types.
field type
mapping[string, any]
next (conditionally required)
The next state to transition to after the current state is executed.
This field is required for executeStep, pass, parallel and wait state types if end is not set to true.
This field can not be used for failure, decision or success states.
field type
string
end
Marks the state as the end of a workflow or parallel branch.
This is equivalent to adding a success or failure terminal state to a workflow or parallel branch.
The field can be used for any state type except for the failure, decision or success states.
field type
string
decisions (conditionally required)
A list of decision rules that determine the next state to transition to based on the output of a previous state.
Each decision rule will be evaluated in order until a rule is found where the output from the previous state matches the rule's condition.
This field is required for the decision state type.
The field can not be used for any other state types.
field type
array[decisionRule]
result
Result, when provided, is treated as the output as the result of the state,
as if it was the result from a handler execution.
When result is not set, the output is the input to the pass state that is subject to modification by the resultPath field.
This field is optional for the pass state type.
The field can not be used with any other state types.
field type
any
timeout
The maximum amount of time in seconds that the state is allowed to complete execution of a handler.
If the state runs for longer than the timeout, the state will fail with the Timeout error name.
This field is optional for the executeStep state type.
The field can not be used with any other state types.
field type
integer
default value
60
waitConfig (conditionally required)
State configuration specific to the wait state type.
This field is required for the wait state type, it can not be used with any other state types.
field type
failureConfig (conditionally required)
State configuration specific to the failure state type.
This field is required for the failure state type, it can not be used with any other state types.
field type
parallelBranches (conditionally required)
A list of branches to be executed in parallel.
The actual execution model will vary based on the target environment, see the limitations section for more information on how parallel execution is accomplished in the Celerity workflow runtime.
This field is required for the parallel state type, it can not be used for any other state types.
The output of each branch will be merged into an array with a single element for each branch. The order of the results in the output array will match the order of the branches defined in this field.
field type
array[parallelBranch]
retry
Retry configuration for the state when the handler execution fails.
This field is only supported for executeStep and parallel state types.
field type
array[retryConfiguration]
catch
Configuration to provide fallback states for specific error types.
This field is only supported for executeStep and parallel state types.
field type
array[catchConfiguration]
decisionRule
FIELDS
and (conditionally required)
A boolean expression consisting of an array of conditions where all must be true to advance to the state specified in the next field.
This field is required if the or, not, and condition fields are not set.
field type
array[condition]
or (conditionally required)
A boolean expression consisting of an array of conditions where only one has to be true to advance to the state specified in the next field.
This field is required if the and, not, and condition fields are not set.
field type
array[condition]
not (conditionally required)
A boolean expression consisting of a single condition for which the negation of the condition must be true to advance to the state specified in the next field.
This field is required if the and, or, and condition fields are not set.
field type
condition (conditionally required)
A single condition that must be true to advance to the state specified in the next field.
This field is required if the and, or, and not fields are not set.
field type
next (required)
The next state to transition to if the condition is met.
field type
string
condition
FIELDS
inputs (required)
A list of one or more inputs to be evaluated for the condition, the number of inputs provided will depend on the number of arguments expected by the function field.
An input can be a path or a literal value.
See the Path Syntax section for more information on how to specify paths.
field type
array[string | integer | float | boolean | array | object | null]
examples
Blueprint Language
inputs = ["$.value", 20]inputs = ["$.result.id", "someConstant"]inputs = [594.34, false, none]inputs = [
"someConstant",
20,
{ mapping = { key1 = "value1", key2 = "value2" } },
["$.value", 20],
[1, 3, 5, 10, 12, 15, 20]
]YAML
["$.value", 20]["$.result.id", "someConstant"][594.34, false, null]- "someConstant"
- 20
- mapping:
key1: "value1"
key2: "value2"
- ["$.value", 20]
- [1, 3, 5, 10, 12, 15, 20]function (required)
The function to be used to evaluate the condition.
Functions can take one or more input arguments and always return a boolean value.
The list of available functions for this version of the workflow resource type can be found in the Condition Functions section.
field type
string
examples
is_present
eq
parallelBranch
FIELDS
startAt (required)
The name of the state used to begin execution of the branch.
field type
string
states (required)
A mapping of state names to state configurations that make up a branch of the parallel state. A state configuration resembles a state in a state machine which describes decisions, transitions or the step to be executed when the state is entered.
States in a branch must only have a next field that points to a state within the same branch.
States outside of a branch can not transition to a state within a branch.
When there is a failure in any branch, due to an unhandled error or by transitioning to a failure state, the entire parallel state will fail.
When a branch transitions to a success state, only that branch will terminate as a result. A success state will pass its input through to the output that may be modified by the inputPath and outputPath fields.
State types that are allowed in a parallel branch are executeStep, pass, wait, decision, failure, and success. Nested parallel states are not supported.
State names
State names must be unique within the entire workflow definition so that handlers can be wired up to the correct states via workflow to handler link annotations.
type
mapping[string, state]
waitConfiguration
FIELDS
seconds (conditionally required)
Number of seconds to wait before transitioning to the next state.
This does not need to be hardcoded, it can be a path to a value in the input object.
This is required if the timestamp field is not set.
field type
string
examples
10
$.waitTime
timestamp (conditionally required)
The timestamp to wait until before transitioning to the next state. The timestamp must be in the format of the RFC3339 profile for the ISO 8601 standard.
The timestamp does not need to be hardcoded, it can be a path to a value in the input object.
This is required if the seconds field is not set.
field type
string
examples
2024-07-22T12:00:00Z
2024-07-22T12:00:00.123Z
$.expires
failureConfiguration
FIELDS
error
The error name to be used for the failure state. This can be a path or a literal value.
field type
string
examples
ResourceNotFound
$.error
cause
A description of the cause of the failure. This can be a path or a literal value.
field type
string
examples
Order not found in the system
$.cause
retryConfiguration
FIELDS
matchErrors (required)
A list of error names that should be matched in order to trigger a retry.
See the errors section for a list of built-in errors that can be matched against.
Custom error names may also be used to match specific errors thrown by handlers.
field type
array[string]
examples
Blueprint Language
matchErrors = ["Timeout", "ResourceNotFound"]matchErrors = ["*"] # Retry on all errorsYAML
["Timeout", "ResourceNotFound"]["*"] # Retry on all errorsinterval
A positive integer value representing the number of seconds to wait before the first attempt to retry a failed state.
field type
integer
default value
3
maxAttempts
A positive integer value representing the maximum number of attempts to retry a failed state.
A value of 0 is valid for this field; this indicates that the state should not be retried on a specific error.
field type
integer
default value
5
maxDelay
A positive integer value representing the maximum interval in seconds to wait between retries.
field type
integer
jitter
A boolean value that determines whether to apply jitter to the retry interval. This AWS blog post from 2015 provides a good insight into how works: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
The implementation of jitter varies across target environments.
field type
boolean
default value
false
backoffRate
A floating point value representing a multiplier that increases the retry interval on each attempt. This AWS blog post from 2015 provides a good insight into how exponential backoff works: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
field type
float
default value
2.0
catchConfiguration
FIELDS
matchErrors (required)
A list of error names that should be matched in order to trigger a retry.
See the errors section for a list of built-in errors that can be matched against.
Custom error names may also be used to match specific errors thrown by handlers.
field type
array[string]
examples
Blueprint Language
matchErrors = ["Timeout", "ResourceNotFound"]matchErrors = ["*"] # Retry on all errorsYAML
["Timeout", "ResourceNotFound"]["*"] # Retry on all errorsnext (required)
The name of the next state to transition to if the error matches any of the errors in the matchErrors field.
field type
string
resultPath
The path to the output field to inject the error information into the input object passed to the state defined
in the next field.
The JSON Path Injections section provides more information as to how JSON paths behave in this context.
For example, for an input object:
{
"value": 10
}A resultPath of $.errorInfo would result in the following output object:
{
"value": 10,
"errorInfo": {
"error": "Some error message",
"cause": "Some cause"
}
}When not set, the error information will be the entire output object passed into the next state.
field type
string
examples
$.errorInfo
Linked From
celerity/vpc
The VPC is used to place the workflow runtime service in the target environment. A workflow is always deployed as a containerised service, so a VPC link applies to every target environment, with platform-specific behaviour described in the Target Environments section.
celerity/handler
When a handler links out to a workflow, it will be configured with permissions and environment variables that enable the handler to trigger the workflow. If a secret store is associated with the handler or the application that it is a part of, the workflow configuration will be added to the secret store instead of environment variables. You can use guides and templates to get an intuition for how to source the configuration and trigger the workflow using the handlers SDK.
Read more about Celerity handlers
Links To
celerity/handler
Handlers are used to execute steps in a workflow. In the context of a workflow, a step is the state of the executeStep type. The handler is configured with annotations that determine the state in the workflow that will trigger the handler.
Read more about Celerity handlers
celerity/sqlDatabase
A workflow can link to a celerity/sqlDatabase resource to use that PostgreSQL database as its execution store, instead of having Celerity provision a dedicated database for the workflow. The workflow runtime creates its own tables in the linked database and applies its schema migrations on startup.
Read more about Celerity SQL databases
celerity/config
The celerity/config resource type can be used to store configuration and sensitive information such as API keys, database passwords, and other credentials that are used by the application.
A workflow can link to a celerity/config resource type to access secrets at runtime, linking an application to a secret and configuration store will automatically make secrets and configuration accessible to all handlers in the application without having to link each handler to the store.
Read more about Celerity config
Examples
Simple Workflow
Blueprint Language
version "2025-11-02"
transform "celerity-2026-02-27-draft"
resource videoIngestWorkflow: celerity/workflow {
select by label {
application = "videoIngest"
}
spec {
startAt = "downloadVideo"
states = {
downloadVideo = {
type = "executeStep",
description = "Download the video from the source URL provided in the input metadata.",
inputPath = "$.metadata",
payloadTemplate = {
videoUrl = "$.videoUrl"
},
next = "scanVideo"
},
scanVideo = {
type = "executeStep",
description = "Scans the video for any malicious content.",
resultPath = "$.scanResult",
next = "processVideo"
},
processVideo = {
type = "executeStep",
description = "Process the video to create a thumbnail and extract metadata.",
inputPath = "$.media",
payloadTemplate = {
videoFilePath = "$.downloaded.path"
},
next = "success"
},
success = {
type = "success",
description = "The video has been successfully processed."
}
}
}
}
resource downloadVideoHandler: celerity/handler {
metadata {
displayName = "Download Video Handler"
annotations = {
"celerity.handler.workflow.state" = "downloadVideo"
}
labels = {
application = "videoIngest"
}
}
spec {
handlerName = "DownloadVideoHandler-v1"
codeLocation = "handlers/video-ingest"
handler = "downloadVideoHandler"
runtime = "nodejs20.x"
memory = 1024
timeout = 60
tracingEnabled = true
}
}
resource scanVideoHandler: celerity/handler {
metadata {
displayName = "Scan Video Handler"
annotations = {
"celerity.handler.workflow.state" = "scanVideo"
}
labels = {
application = "videoIngest"
}
}
spec {
handlerName = "ScanVideoHandler-v1"
codeLocation = "handlers/video-ingest"
handler = "scanVideoHandler"
runtime = "nodejs20.x"
memory = 2048
timeout = 120
tracingEnabled = true
}
}
resource processVideoHandler: celerity/handler {
metadata {
displayName = "Process Video Handler"
annotations = {
"celerity.handler.workflow.state" = "processVideo"
}
labels = {
application = "videoIngest"
}
}
spec {
handlerName = "ProcessVideoHandler-v1"
codeLocation = "handlers/video-ingest"
handler = "processVideoHandler"
runtime = "nodejs20.x"
memory = 1024
timeout = 60
tracingEnabled = true
}
}YAML
version: 2025-11-02
transform: celerity-2026-02-27-draft
resources:
videoIngestWorkflow:
type: "celerity/workflow"
linkSelector:
byLabel:
application: "videoIngest"
spec:
startAt: "downloadVideo"
states:
downloadVideo:
type: "executeStep"
description: "Download the video from the source URL provided in the input metadata."
inputPath: "$.metadata"
payloadTemplate:
videoUrl: "$.videoUrl"
next: "scanVideo"
scanVideo:
type: "executeStep"
description: "Scans the video for any malicious content."
resultPath: "$.scanResult"
next: "processVideo"
processVideo:
type: "executeStep"
description: "Process the video to create a thumbnail and extract metadata."
inputPath: "$.media"
payloadTemplate:
videoFilePath: "$.downloaded.path"
next: "success"
success:
type: "success"
description: "The video has been successfully processed."
downloadVideoHandler:
type: "celerity/handler"
metadata:
displayName: "Download Video Handler"
annotations:
celerity.handler.workflow.state: "downloadVideo"
labels:
application: "videoIngest"
spec:
handlerName: DownloadVideoHandler-v1
codeLocation: "handlers/video-ingest"
handler: "downloadVideoHandler"
runtime: "nodejs20.x"
memory: 1024
timeout: 60
tracingEnabled: true
scanVideoHandler:
type: "celerity/handler"
metadata:
displayName: "Scan Video Handler"
annotations:
celerity.handler.workflow.state: "scanVideo"
labels:
application: "videoIngest"
spec:
handlerName: ScanVideoHandler-v1
codeLocation: "handlers/video-ingest"
handler: "scanVideoHandler"
runtime: "nodejs20.x"
memory: 2048
timeout: 120
tracingEnabled: true
processVideoHandler:
type: "celerity/handler"
metadata:
displayName: "Process Video Handler"
annotations:
celerity.handler.workflow.state: "processVideo"
labels:
application: "videoIngest"
spec:
handlerName: ProcessVideoHandler-v1
codeLocation: "handlers/video-ingest"
handler: "processVideoHandler"
runtime: "nodejs20.x"
memory: 1024
timeout: 60
tracingEnabled: trueComplex Workflow
Blueprint Language
version "2025-11-02"
transform "celerity-2026-02-27-draft"
resource docProcessingWorkflow: celerity/workflow {
select by label {
application = "docProcessor"
}
spec {
startAt = "fetchDocument"
states = {
fetchDocument = {
type = "executeStep",
description = "Fetch the document provided in the input data.",
inputPath = "$.document",
payloadTemplate = {
url = "$.url"
},
resultPath = "$.downloaded",
retry = [
{
matchErrors = ["Timeout"],
interval = 5,
maxAttempts = 3,
jitter = true,
backoffRate = 1.5
}
],
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
],
next = "scanDocument"
},
scanDocument = {
type = "executeStep",
description = "Scans the document for any malicious content.",
resultPath = "$.scanResult",
retry = [
{
matchErrors = ["Timeout"],
interval = 5,
maxAttempts = 3,
jitter = true,
backoffRate = 1.5
}
],
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
],
next = "scanResultDecision"
},
scanResultDecision = {
type = "decision",
description = "Decide the next state based on the scan result.",
decisions = [
{
condition = {
function = "eq",
inputs = ["$.scanResult", "Clean"]
},
next = "documentProcessingDecision"
},
{
condition = {
function = "eq",
inputs = ["$.scanResult", "Malicious"]
},
next = "maliciousContentFound"
}
]
},
documentProcessingDecision = {
type = "decision",
description = "Decide which document processing step to execute based on the document type.",
inputPath = "$.downloaded",
decisions = [
{
condition = {
function = "string_match",
inputs = ["$.path", "*.pdf"]
},
next = "processPDF"
},
{
condition = {
function = "string_match",
inputs = ["$.path", "*.docx"]
},
next = "processDOCX"
}
]
},
processPDF = {
type = "executeStep",
description = "Process the PDF document to extract text and metadata.",
inputPath = "$.downloaded",
payloadTemplate = {
filePath = "$.path"
},
resultPath = "$.extractedDataFile",
retry = [
{
matchErrors = ["Timeout"],
interval = 5,
maxAttempts = 3,
jitter = true,
backoffRate = 1.5
}
],
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
],
next = "waitForProcessing"
},
processDOCX = {
type = "executeStep",
description = "Process the word document to extract text and metadata.",
inputPath = "$.downloaded",
payloadTemplate = {
filePath = "$.path"
},
resultPath = "$.extractedDataFile",
retry = [
{
matchErrors = ["Timeout"],
interval = 5,
maxAttempts = 3,
jitter = true,
backoffRate = 1.5
}
],
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
],
next = "waitForProcessing"
},
waitForProcessing = {
type = "wait",
waitConfig = {
seconds = "120"
},
next = "uploadToSystem"
},
uploadToSystem = {
type = "executeStep",
description = "Upload the extracted data to the system.",
retry = [
{
matchErrors = ["Timeout"],
interval = 5,
maxAttempts = 3,
jitter = true,
backoffRate = 1.5
}
],
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
],
next = "success"
},
handleError = {
type = "executeStep",
description = "Handle any error that occurred during the workflow, persisting status and error information to the domain-specific database.",
inputPath = "$.errorInfo",
next = "failureDecision"
},
failureDecision = {
type = "decision",
description = "Choose the failure state to transition to based on the error type.",
inputPath = "$.errorInfo",
decisions = [
{
condition = {
function = "eq",
inputs = ["$.error", "DocumentFetchError"]
},
next = "fetchFailure"
},
{
condition = {
function = "eq",
inputs = ["$.error", "DocumentScanError"]
},
next = "scanFailure"
},
{
or = [
{ function = "eq", inputs = ["$.error", "ExtractPDFError"] },
{ function = "eq", inputs = ["$.error", "PDFLoadError"] }
],
next = "processPDFFailure"
},
{
or = [
{ function = "eq", inputs = ["$.error", "ExtractDOCXError"] },
{ function = "eq", inputs = ["$.error", "DOCXLoadError"] }
],
next = "processDOCXFailure"
},
{
condition = {
function = "eq",
inputs = ["$.error", "UploadToSystemError"]
},
next = "uploadToSystemFailure"
}
]
},
success = {
type = "success",
description = "The document has been successfully processed."
},
fetchFailure = {
type = "failure",
description = "The document could not be fetched.",
failureConfig = {
error = "DocumentFetchError",
cause = "The document could not be fetched from the provided URL."
}
},
scanFailure = {
type = "failure",
description = "An error occurred while scanning the document.",
failureConfig = {
error = "DocumentScanError",
cause = "An error occurred while scanning the document."
}
},
maliciousContentFound = {
type = "failure",
description = "Malicious content was found in the document.",
failureConfig = {
error = "MaliciousContentFound",
cause = "Malicious content was found in the document."
}
},
processPDFFailure = {
type = "failure",
description = "An error occurred while processing the PDF document.",
failureConfig = {
error = "PDFProcessingError",
cause = "An error occurred while processing the PDF document."
}
},
processDOCXFailure = {
type = "failure",
description = "An error occurred while processing the word document.",
failureConfig = {
error = "DOCXProcessingError",
cause = "An error occurred while processing the word document."
}
},
uploadToSystemFailure = {
type = "failure",
description = "An error occurred while uploading the extracted data to the system.",
failureConfig = {
error = "UploadToSystemError",
cause = "An error occurred while uploading the extracted data to the system."
}
}
}
}
}
resource fetchDocumentHandler: celerity/handler {
metadata {
displayName = "Fetch Document Handler"
annotations = {
"celerity.handler.workflow.state" = "fetchDocument"
}
labels = {
application = "documentProcessor"
}
}
spec {
handlerName = "FetchDocumentHandler-v1"
codeLocation = "handlers/doc-processor"
handler = "fetch_document"
runtime = "python3.12.x"
memory = 1024
timeout = 60
tracingEnabled = true
}
}
resource scanDocumentHandler: celerity/handler {
metadata {
displayName = "Scan Document Handler"
annotations = {
"celerity.handler.workflow.state" = "scanDocument"
}
labels = {
application = "documentProcessor"
}
}
spec {
handlerName = "ScanDocumentHandler-v1"
codeLocation = "handlers/doc-processor"
handler = "scan_document"
runtime = "python3.12.x"
memory = 1024
timeout = 60
tracingEnabled = true
}
}
resource processPDFHandler: celerity/handler {
metadata {
displayName = "PDF Processing Handler"
annotations = {
"celerity.handler.workflow.state" = "processPDF"
}
labels = {
application = "documentProcessor"
}
}
spec {
handlerName = "ProcessPDFHandler-v1"
codeLocation = "handlers/doc-processor"
handler = "process_pdf"
runtime = "python3.12.x"
memory = 4096
timeout = 60
tracingEnabled = true
}
}
resource processDOCXHandler: celerity/handler {
metadata {
displayName = "Word Document Processing Handler"
annotations = {
"celerity.handler.workflow.state" = "processDOCX"
}
labels = {
application = "documentProcessor"
}
}
spec {
handlerName = "ProcessDOCXHandler-v1"
codeLocation = "handlers/doc-processor"
handler = "process_docx"
runtime = "python3.12.x"
memory = 2048
timeout = 60
tracingEnabled = true
}
}
resource systemUploadHandler: celerity/handler {
metadata {
displayName = "System Upload Handler"
annotations = {
"celerity.handler.workflow.state" = "uploadToSystem"
}
labels = {
application = "documentProcessor"
}
}
spec {
handlerName = "SystemUploadHandler-v1"
codeLocation = "handlers/doc-processor"
handler = "upload_extracted_data"
runtime = "python3.12.x"
memory = 1024
timeout = 30
tracingEnabled = true
}
}YAML
version: 2025-11-02
transform: celerity-2026-02-27-draft
resources:
docProcessingWorkflow:
type: "celerity/workflow"
linkSelector:
byLabel:
application: "docProcessor"
spec:
startAt: "fetchDocument"
states:
fetchDocument:
type: "executeStep"
description: "Fetch the document provided in the input data."
inputPath: "$.document"
payloadTemplate:
url: "$.url"
resultPath: "$.downloaded"
retry:
- matchErrors: ["Timeout"]
interval: 5
maxAttempts: 3
jitter: true
backoffRate: 1.5
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"
next: "scanDocument"
scanDocument:
type: "executeStep"
description: "Scans the document for any malicious content."
resultPath: "$.scanResult"
retry:
- matchErrors: ["Timeout"]
interval: 5
maxAttempts: 3
jitter: true
backoffRate: 1.5
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"
next: "scanResultDecision"
scanResultDecision:
type: "decision"
description: "Decide the next state based on the scan result."
decisions:
- condition:
function: "eq"
inputs: ["$.scanResult", "Clean"]
next: "documentProcessingDecision"
- condition:
function: "eq"
inputs: ["$.scanResult", "Malicious"]
next: "maliciousContentFound"
documentProcessingDecision:
type: "decision"
description: "Decide which document processing step to execute based on the document type."
inputPath: "$.downloaded"
decisions:
- condition:
function: "string_match"
inputs: ["$.path", "*.pdf"]
next: "processPDF"
- condition:
function: "string_match"
inputs: ["$.path", "*.docx"]
next: "processDOCX"
processPDF:
type: "executeStep"
description: "Process the PDF document to extract text and metadata."
inputPath: "$.downloaded"
payloadTemplate:
filePath: "$.path"
resultPath: "$.extractedDataFile"
retry:
- matchErrors: ["Timeout"]
interval: 5
maxAttempts: 3
jitter: true
backoffRate: 1.5
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"
next: "waitForProcessing"
processDOCX:
type: "executeStep"
description: "Process the word document to extract text and metadata."
inputPath: "$.downloaded"
payloadTemplate:
filePath: "$.path"
resultPath: "$.extractedDataFile"
retry:
- matchErrors: ["Timeout"]
interval: 5
maxAttempts: 3
jitter: true
backoffRate: 1.5
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"
next: "waitForProcessing"
waitForProcessing:
type: "wait"
waitConfig:
seconds: "120"
next: "uploadToSystem"
uploadToSystem:
type: "executeStep"
description: "Upload the extracted data to the system."
retry:
- matchErrors: ["Timeout"]
interval: 5
maxAttempts: 3
jitter: true
backoffRate: 1.5
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"
next: "success"
handleError:
type: "executeStep"
description: "Handle any error that occurred during the workflow, persisting status and error information to the domain-specific database."
inputPath: "$.errorInfo"
next: "failureDecision"
failureDecision:
type: "decision"
description: "Choose the failure state to transition to based on the error type."
inputPath: "$.errorInfo"
decisions:
- condition:
function: "eq"
inputs: ["$.error", "DocumentFetchError"]
next: "fetchFailure"
- condition:
function: "eq"
inputs: ["$.error", "DocumentScanError"]
next: "scanFailure"
- or:
- function: "eq"
inputs: ["$.error", "ExtractPDFError"]
- function: "eq"
inputs: ["$.error", "PDFLoadError"]
next: "processPDFFailure"
- or:
- function: "eq"
inputs: ["$.error", "ExtractDOCXError"]
- function: "eq"
inputs: ["$.error", "DOCXLoadError"]
next: "processDOCXFailure"
- condition:
function: "eq"
inputs: ["$.error", "UploadToSystemError"]
next: "uploadToSystemFailure"
success:
type: "success"
description: "The document has been successfully processed."
fetchFailure:
type: "failure"
description: "The document could not be fetched."
failureConfig:
error: "DocumentFetchError"
cause: "The document could not be fetched from the provided URL."
scanFailure:
type: "failure"
description: "An error occurred while scanning the document."
failureConfig:
error: "DocumentScanError"
cause: "An error occurred while scanning the document."
maliciousContentFound:
type: "failure"
description: "Malicious content was found in the document."
failureConfig:
error: "MaliciousContentFound"
cause: "Malicious content was found in the document."
processPDFFailure:
type: "failure"
description: "An error occurred while processing the PDF document."
failureConfig:
error: "PDFProcessingError"
cause: "An error occurred while processing the PDF document."
processDOCXFailure:
type: "failure"
description: "An error occurred while processing the word document."
failureConfig:
error: "DOCXProcessingError"
cause: "An error occurred while processing the word document."
uploadToSystemFailure:
type: "failure"
description: "An error occurred while uploading the extracted data to the system."
failureConfig:
error: "UploadToSystemError"
cause: "An error occurred while uploading the extracted data to the system."
fetchDocumentHandler:
type: "celerity/handler"
metadata:
displayName: "Fetch Document Handler"
annotations:
celerity.handler.workflow.state: "fetchDocument"
labels:
application: "documentProcessor"
spec:
handlerName: FetchDocumentHandler-v1
codeLocation: "handlers/doc-processor"
handler: "fetch_document"
runtime: "python3.12.x"
memory: 1024
timeout: 60
tracingEnabled: true
scanDocumentHandler:
type: "celerity/handler"
metadata:
displayName: "Scan Document Handler"
annotations:
celerity.handler.workflow.state: "scanDocument"
labels:
application: "documentProcessor"
spec:
handlerName: ScanDocumentHandler-v1
codeLocation: "handlers/doc-processor"
handler: "scan_document"
runtime: "python3.12.x"
memory: 1024
timeout: 60
tracingEnabled: true
processPDFHandler:
type: "celerity/handler"
metadata:
displayName: "PDF Processing Handler"
annotations:
celerity.handler.workflow.state: "processPDF"
labels:
application: "documentProcessor"
spec:
handlerName: ProcessPDFHandler-v1
codeLocation: "handlers/doc-processor"
handler: "process_pdf"
runtime: "python3.12.x"
memory: 4096
timeout: 60
tracingEnabled: true
processDOCXHandler:
type: "celerity/handler"
metadata:
displayName: "Word Document Processing Handler"
annotations:
celerity.handler.workflow.state: "processDOCX"
labels:
application: "documentProcessor"
spec:
handlerName: ProcessDOCXHandler-v1
codeLocation: "handlers/doc-processor"
handler: "process_docx"
runtime: "python3.12.x"
memory: 2048
timeout: 60
tracingEnabled: true
systemUploadHandler:
type: "celerity/handler"
metadata:
displayName: "System Upload Handler"
annotations:
celerity.handler.workflow.state: "uploadToSystem"
labels:
application: "documentProcessor"
spec:
handlerName: SystemUploadHandler-v1
codeLocation: "handlers/doc-processor"
handler: "upload_extracted_data"
runtime: "python3.12.x"
memory: 1024
timeout: 30
tracingEnabled: truePath Syntax
Paths are used to access values in the input object to the current state in the workflow.
When evaluating a path as an input for a condition, a string beginning with $ is interpreted as a path, otherwise, the value is treated as a string literal.
The path to the input field from the previous state's output that the condition will be evaluated against.
The syntax for the path is $.*. $ represents the root input object and * represents any key in the object.
For example, $.value would match the value of the key value in the input object.
Nested keys can also be accessed using the dot notation, for example, $.result.id would match the value of the id key in the nested result object.
The JSONPath syntax which is implemented for paths in Celerity workflows.
Errors
There are a built-in set of errors that can be matched against in the retry and catch configuration for a state.
Built-in errors that can be retried include the following:
*- Matches all errors that can be retried.Timeout- The handler execution timed out.HandlerFailed- The handler execution failed.InvalidResultPath- The result path specified in the state configuration is invalid.InvalidOutputPath- The output path specified in the state configuration is invalid.BranchesFailed- One or more branches of a parallel state failed.
Built-in errors that can be caught include the following:
*- Matches all errors that can be caught.Timeout- The handler execution timed out.HandlerFailed- The handler execution failed.InvalidResultPath- The result path specified in the state configuration is invalid. It's important to note that an invalid result path error that occurs in a catcher can not be caught.InvalidOutputPath- The output path specified in the state configuration is invalid.BranchesFailed- One or more branches of a parallel state failed.InvalidInputPath- The input path specified in the state configuration is invalid.InvalidPayloadTemplate- The payload template specified in the state configuration is invalid.PayloadTemplateFailure- An error occurred while constructing the payload using the payload template, often an issue with a template function call.NoDecisionMatched- No decision rule matched in a decision state.InvalidCondition- The condition specified in a decision rule is invalid.
Error Output
When an error is caught by a state, the catcher will yield an object with the following structure:
{
"error": "ErrorName",
"cause": "Error cause message"
}This can be added to the input object passed to the next state using the resultPath field.
For example:
Blueprint Language
catch = [
{
matchErrors = ["*"],
next = "handleError",
resultPath = "$.errorInfo"
}
]YAML
catch:
- matchErrors: ["*"]
next: "handleError"
resultPath: "$.errorInfo"Will result in an object like the following:
{
"otherField": "some value",
"errorInfo": {
"error": "Some error message",
"cause": "Some cause"
}
}This will be passed into the next state as the input object.
JSON Path Injections
States and catchers can have a resultPath field that is used to inject the result of the state or error into the input object passed to the next state.
There are limitations around how JSON paths behave in this context;
JSON path injection only supports injecting values into fields of the root JSON object.
The $.<field> and $['<field>'] syntax is supported for injecting values into fields of the root object.
Payload Template
The payload template is a mapping structure that can be used to construct a payload to be passed into the handler configured to be executed for the current state.
If a field value starts with $, the value is treated as a path to a value in the input object to the current state in the workflow.
If a field value starts with func:, the value is treated as a template function that can take literals or JSON path expressions as arguments. See the Template Functions section for more information on the available template functions.
If neither of the above conditions are met, the value is treated as a literal value.
For example, given an input object:
{
"values": [10, 405, 304, 20, 304, 20],
"flag1": true
}A payload template of:
Blueprint Language
payloadTemplate = {
value1 = "$.values[0]",
restOfValues = "func:remove_duplicates($.values[-5:])",
nestedStructure = {
key1 = "some value",
key2 = 20,
flag = "$.flag1"
}
}YAML
payloadTemplate:
value1: "$.values[0]"
restOfValues: "func:remove_duplicates($.values[-5:])"
nestedStructure:
key1: "some value"
key2: 20
flag: "$.flag1"Will produce a payload object of:
{
"value1": 10,
"restOfValues": [405, 304, 20],
"nestedStructure": {
"key1": "some value",
"key2": 20,
"flag": true
}
}Template Function Syntax
The syntax for template functions can be simplified to func:<functionName>(<arg1>, <arg2>, ..., <argN>) in a concise form; func: is the prefix that indicates the preceding value is a function call.
<functionName> must consist of only alphanumeric characters and underscores.
<arg1>, <arg2>, ..., <argN> can consist of scalar literal values, JSON paths to values in the input object, or other function calls.
Scalar literal values refer to string, integer, float, boolean, or null values.
The formalised grammar for the template function syntax that comes after the func: prefix is as follows:
function call = name , "(" , function args , ")" ;
function args = [ function arg , { "," , function arg } ] ;
function arg = literal | json path | function call ;
literal = bool literal | null literal | float literal | int literal | string literal ;
json path = ? valid json path ? ;
# Lex tokens
string literal = '"' , string chars , '"' ;
bool literal = "true" | "false" ;
null literal = "null" ;
int literal = [ "-" ] , natural number ;
float literal = [ "-" ] , natural number , "." , natural number ;
natural number = { digit }- ;
string chars = { string char } ;
string char = ? utf-8 char excluding quote ? | escaped quote ;
escaped quote = "\" , '"' ;
name = start name char , name chars ;
name chars = { name char } ;
name char = letter | digit | "_" ;
start name char = letter | "_" ;
letter = ? [A-Za-z] ? ;
digit = ? [0-9] ? ;The above is a representation of the grammar in Extended Backus-Naur form notation.
JSON paths are parsed using the syntax defined in the JSONPath specification.
Template Functions
The following section describes the functions that can be used in payload templates, the parameters they accept and the return types.
format
This function formats a string using the provided arguments.
The use of {} in the format string will be replaced with the arguments in order.
Parameters:
string - The format string.
N variadic arguments - The values to replace {} in the format string.
Returns:
string - The formatted string.
Examples:
func:format("Hello, {}!", $.name)jsondecode
This function decodes a JSON string into an object, array or scalar value.
Parameters:
string- The JSON string to decode.
Returns:
string | integer | float | boolean | object | array - The decoded JSON value.
Examples:
func:jsondecode("{\"key\": \"value\"}")jsonencode
This function encodes a value as a JSON string.
Parameters:
string | integer | float | boolean | object | array- The value to encode as a JSON string.
Returns:
string - The JSON encoded string.
Examples:
func:jsonencode($.config)jsonmerge
This function merges 2 json objects together. Any duplicate keys in the second object will overwrite the keys in the first object.
Parameters:
object- The first object to merge.object- The second object to merge.
Returns:
object - The merged object.
Examples:
func:jsonmerge($.object1, $.object2)b64encode
This function base64 encodes a string.
Parameters:
string- The string to encode.
Returns:
string - The base64 encoded string.
Examples:
func:b64encode($.value)b64decode
This function decodes a base64 encoded string.
Parameters:
string- The base64 encoded string to decode.
Returns:
string - The decoded human-readable string.
Examples:
func:b64decode($.encodedValue)hash
This function hashes some input data using a specified algorithm.
The available hash algorithms are:
- SHA256
- SHA384
- SHA512
MD5 and SHA1 were considered in the original design but were not included due to the insecurity of these algorithms.
Parameters:
string- The input data to hash.string- The hash algorithm to use.
Returns:
string - The hashed data as a hex string.
Examples:
func:hash($.value, "SHA256")list
This function creates a list from a set of positional arguments.
Parameters:
N positional arguments - The values to create a list from.
Returns:
array - A list of values.
Examples:
func:list(10, $.values[0], 30, $.otherValue, "Some string")chunk_list
This function splits a list into chunks of a specified size.
Parameters:
array- The list to split into chunks.integer- The size of each chunk.
Returns:
array - A 2-dimensional array of chunks.
Examples:
func:chunk_list($.values, 5)list_elem
This function returns an element from a list at a specific index.
Parameters:
-
array- The list to extract the element from. -
integer- The index of the element to extract.
Returns:
string | integer | float | boolean | object | array | null - The element at the specified index.
Examples:
func:list_elem($.values, 2)remove_duplicates
This function removes duplicates from an array of values. The function will carry out deep equality checks for objects and arrays, performance may be significantly impacted when working with large and complex structures.
Parameters:
array- The array of values to remove duplicates from.
Returns:
array - An array of unique values.
Examples:
func:remove_duplicates($.values)contains
This function checks if a value is present in a list or a substring is present in a string. This will carry out deep equality checks for objects and arrays.
Parameters:
-
string | array- The list or string to check for the value or substring. -
string | integer | float | boolean | object | array- The value or substring to check for.
Returns:
boolean - true if the value or substring is present, false otherwise.
Examples:
func:contains($.numbers, 10)split
This function splits a string into an array of substrings based on a delimiter.
Parameters:
string- The string to split.string- The delimiter to split the string by.
Returns:
array - An array of substrings.
Examples:
func:split($.value, ",")math_rand
This function generates a random number between a minimum and maximum value. The random number generated is an integer and the provided parameters must be integers.
Warning
This will not generate a cryptographically secure random number,
math_rand should not be used in security-sensitive contexts.
Parameters:
integer- The minimum value for the random number. (inclusive)integer- The maximum value for the random number. (exclusive)
Returns:
integer - A random number between the minimum and maximum values.
Examples:
func:math_rand(0, 100)math_add
This function will add 2 numbers together.
Parameters:
integer | float- The first number to add.integer | float- The second number to add.
Returns:
float - The result of the addition as a floating point number.
Examples:
func:math_add(10, 20)math_sub
This function will subtract the second number from the first.
Parameters:
integer | float- The first number to subtract from.integer | float- The second number to subtract.
Returns:
float - The result of the subtraction as a floating point number.
Examples:
func:math_sub(20, 10)math_mult
This function will multiply 2 numbers together.
Parameters:
integer | float- The first number to multiply.integer | float- The second number to multiply.
Returns:
float - The result of the multiplication as a floating point number.
Examples:
func:math_mult(10, 20)math_div
This function will divide a number by another.
Parameters:
integer | float- The first number to divide.integer | float- The second number to divide by.
Returns:
float - The result of the division as a floating point number.
Examples:
func:math_div($.result[0].total, 10)len
This function will return the length of a string or an array.
Parameters:
string | array- The string or array to get the length of.
Returns:
integer - The length of the string or array.
Examples:
func:len($.values)uuid
This function generates a random UUID as per the UUID version 4 specification.
Parameters:
The uuid function does not take any parameters.
Returns:
string - An ID in the UUID version 4 format.
Examples:
func:uuid()nanoid
This function provides a way to generate an ID that is shorter than a UUID while maintaining guarantees of uniqueness.
See the nanoid project repo for more information on how the IDs are generated.
Parameters:
The nanoid function does not take any parameters.
Returns:
string - A unique ID in the nanoid format.
Examples:
func:nanoid()Condition Functions
The following section describes the functions that can be used in conditions for decision states and the parameters they accept. All functions return a boolean value.
is_present
This function checks if the input is present in the input object to the current state in the workflow.
Parameters:
string- The path to the input field to carry out a presence check. This must be a valid path of the$.*form laid out in the Path Syntax section.
is_null
This function checks if the input is null.
Parameters:
any- The value to check if it isnull. This can be a path or a literal value.
is_numeric
This function checks if the input is a numeric value. A numeric value can be an integer or a floating-point number.
Parameters:
any- The value to check if it is numeric. This can be a path or a literal value.
is_string
This function checks if the input is a string.
Parameters:
any- The value to check if it is a string. This can be a path or a literal value.
is_boolean
This function checks if the input is a boolean value.
Parameters:
any- The value to check if it is a boolean. This can be a path or a literal value.
is_timestamp
This function checks if the input is a timestamp. Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
any- The value to check if it is a timestamp. This can be a path or a literal value.
eq
This function checks if two values are equal.
If the values are not of the same type, the function will always return false.
For example, 1 is not considered equal to "1".
Parameters:
string | integer | float | boolean | null- The first value to compare, can be a path or a literal value.string | integer | float | boolean | null- The second value to compare, can be a path or a literal value.
lt
This function checks if the first value is less than the second value. The values must be numeric, either integers or floating-point numbers.
Parameters:
integer | float- The first value to compare, can be a path or a literal value.integer | float- The second value to compare, can be a path or a literal value.
gt
This function checks if the first value is greater than the second value. The values must be numeric, either integers or floating-point numbers.
Parameters:
integer | float- The first value to compare, can be a path or a literal value.integer | float- The second value to compare, can be a path or a literal value.
lte
This function checks if the first value is less than or equal to the second value. The values must be numeric, either integers or floating-point numbers.
Parameters:
integer | float- The first value to compare, can be a path or a literal value.integer | float- The second value to compare, can be a path or a literal value.
gte
This function checks if the first value is greater than or equal to the second value. The values must be numeric, either integers or floating-point numbers.
Parameters:
integer | float- The first value to compare, can be a path or a literal value.integer | float- The second value to compare, can be a path or a literal value.
timestamp_eq
This function checks if two timestamps are equal.
Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
string- The first timestamp to compare, can be a path or a literal value.string- The second timestamp to compare, can be a path or a literal value.
timestamp_lt
This function checks if the first timestamp is less than the second timestamp.
Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
string- The first timestamp to compare, can be a path or a literal value.string- The second timestamp to compare, can be a path or a literal value.
timestamp_gt
This function checks if the first timestamp is greater than the second timestamp.
Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
string- The first timestamp to compare, can be a path or a literal value.string- The second timestamp to compare, can be a path or a literal value.
timestamp_lte
This function checks if the first timestamp is less than or equal to the second timestamp.
Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
string- The first timestamp to compare, can be a path or a literal value.string- The second timestamp to compare, can be a path or a literal value.
timestamp_gte
This function checks if the first timestamp is greater than or equal to the second timestamp.
Timestamps are strings that must conform to the RFC3339 profile of the ISO 8601 standard.
Parameters:
string- The first timestamp to compare, can be a path or a literal value.string- The second timestamp to compare, can be a path or a literal value.
regex_match
This function checks if a string matches a regular expression pattern.
Parameters:
string- The string to match against the regular expression pattern, can be a path or a literal value.string- The regular expression pattern to match against.
string_match
This function checks if a string matches a template using * as a wildcard character.
The * wildcard character can match zero or more characters and can be used multiple times in a template.
Examples of valid templates would be:
*- Matches any string.prefix.*- Matches any string that starts withprefix..*.log- Matches any string that ends with.log.prefix*.log- Matches any string that starts withprefixand ends with.log.
Only the * character has special meaning in the template, all other characters are treated literally.
If the character * is required to be matched literally, it must be escaped with a backslash \*.
If the character \ is required to be matched literally, it must be escaped with another backslash \\.
If the workflow definition is a part of a blueprint defined in a JSON file, the escaped string \\* represents * and \\\\ represents \.
Parameters:
string- The string to match against the template, can be a path or a literal value.string- The template to match against.
Execution Guarantees
The Celerity workflow runtime provides exactly-once state transitions and at-least-once handler invocation.
State transitions are writes to the execution store, guarded by conditional updates and fencing leases, so each transition is committed once. Handler invocation for executeStep states is at-least-once; if the instance running an execution stops while a handler is running (for example, due to a crash, a deployment or scale-in), it is not possible to know whether the handler completed, so the handler is invoked again when the execution is resumed.
Handlers must be idempotent
Handlers used for executeStep states can be invoked more than once for the same step of the same execution. Design them so that running them more than once has the same effect as running them once, for example by using the execution ID and state name as an idempotency key for side effects such as payments or outbound messages.
Executions are durable and survive crashes, deployments and scale-in. Any instance of the workflow runtime can resume an execution that another instance was running. When an execution is resumed:
| Prior state status | Behaviour on resume |
|---|---|
| Succeeded | Skipped, its output is read from the execution store. |
| Failed with retries remaining | Resumed at the next retry, preserving the attempt count. |
In progress on an executeStep state | The handler is invoked again. |
In progress on a wait state | Resumed from the stored deadline. |
In progress on a parallel state | Only branches that have not reached a terminal status are run again. |
An execution is always resumed against the version of the workflow definition it started with, so deploying a change to a workflow does not affect executions that are already in progress, including those suspended in a long wait state.
Triggering an execution with an execution name that is already in use returns the existing execution rather than starting a second one, so a trigger can be safely retried.
Hosting Profiles
The workflow runtime is a long-running service that can be deployed under one of two hosting profiles. The hosting profile is deploy configuration, not a property of the workflow; workflow behaviour is the same under both, and a workflow does not need to change when switching between them.
| Profile | Runs on | Idle cost |
|---|---|---|
| Always-on (default) | Amazon ECS (Fargate or EC2), Amazon EKS, Google Cloud Run, GKE, Azure Container Apps, AKS | At least one running instance |
| Scale-to-zero | Google Cloud Run, Azure Container Apps | No compute cost while idle |
Always-on
One or more instances of the workflow runtime are always running. An execution suspended in a wait state sleeps in the running process and costs nothing extra. If an instance crashes or is replaced during a deployment, the executions it was running are resumed by another instance.
Scale-to-zero
The container platform stops all instances of the workflow runtime when it is idle. A periodic schedule (Cloud Scheduler for Google Cloud, a scheduled job for Azure Container Apps) wakes the service every few minutes so that it can resume executions that are due, such as those that have reached the end of a wait state. This is one schedule per deployment, not one per execution.
Under this profile, wait states shorter than a threshold (60 seconds by default) sleep in the running process. Longer wait states store their deadline in the execution store and release the instance, and the execution is resumed by the first instance to run after the deadline has passed. This means that a wait state can resume up to one schedule interval after its deadline.
Scale-to-zero does not remove all idle cost
The execution store is a PostgreSQL database that is always running, so scale-to-zero removes the idle cost of the workflow runtime's compute but not of its database. See Execution Store for ways to reduce the cost of the database.
Execution Store
The workflow runtime stores execution state in PostgreSQL on every target environment. The same schema and queries are used everywhere, which is part of what keeps workflow behaviour identical across target environments.
| Environment | Execution store |
|---|---|
| Local development | A PostgreSQL container in the local development stack |
| AWS | Amazon RDS for PostgreSQL or Aurora PostgreSQL, fronted by RDS Proxy |
| Google Cloud | Cloud SQL for PostgreSQL |
| Azure | Azure Database for PostgreSQL |
There are three ways to provide the execution store for a workflow:
- Provisioned by Celerity (default) - Celerity creates a PostgreSQL database for the workflow in your cloud account.
- A linked
celerity/sqlDatabase- Link the workflow to acelerity/sqlDatabaseresource in the same blueprint to use that database as the execution store. - A connection string - Supply the connection string of a PostgreSQL database you already run through the
{cloud}.workflow.store.connectionStringdeploy configuration option. Any PostgreSQL service can be used, including serverless and free-tier offerings.
The workflow runtime creates the tables and indexes it needs and applies its schema migrations on startup.
Cost and operation of the execution store
A PostgreSQL database has an idle cost, even when no workflows are running, and needs backups, version upgrades and storage growth to be looked after. Celerity sets up the database when it is provisioned by Celerity, but it runs in your cloud account and is billed to you. To reduce this cost, use a linked celerity/sqlDatabase that your application already uses, or supply a connection string for an existing database or a lower-cost PostgreSQL service.
Target Environments
Local Development
🚀 Planned for v1
In the local development environment, a workflow is deployed as a containerised version of the Celerity workflow runtime in a local container orchestrator.
The workflow API will be exposed to the host machine on a configurable port, allowing manually testing your workflow application and enabling other applications to interact with the workflow API.
In the local environment, the runtime is backed by a PostgreSQL container in the local development stack for storing workflow execution state, the same postgres:17-alpine container used for celerity/sqlDatabase resources. Using PostgreSQL locally means that workflow execution behaves the same way on a developer machine as it does when deployed.
Links from VPCs to APIs are ignored for this environment as the workflow is deployed to a local container network on a developer or CI machine.
AWS
🚀 Planned for v1 - This target environment will be available in a future v0 evolution.
In the AWS environment, workflows are deployed as a containerised version of the Celerity workflow runtime under the always-on hosting profile.
Workflows can be deployed to ECS or EKS backed by Fargate or EC2 using compute configuration that is shared between compute resource types in the app deploy configuration file. To run the workflow runtime on serverless container infrastructure, set aws.compute.launchType to fargate. Handlers for executeStep states are loaded into the workflow runtime's container.
The workflow runtime stores execution state in PostgreSQL. By default, an Amazon RDS for PostgreSQL database fronted by RDS Proxy is created when the workflow is deployed, provisioned in the same way as a celerity/sqlDatabase. You can use a linked celerity/sqlDatabase or an existing database instead; see Execution Store for more information.
ECS
When a workflow is first deployed to ECS, a new cluster is created for the workflow. A service is provisioned within the cluster to run the application. A public-facing application load balancer is created to route traffic to the service, if you require private access to the workflow API, the load balancer can be configured to be internal. When domain configuration is provided and the load balancer is public-facing, an ACM certificate is created for the domain and attached to the load balancer, you will need to verify the domain ownership before the certificate can be used.
The service is deployed with an auto-scaling group that will scale the number of tasks running the workflow based on the CPU and memory usage of the tasks. The auto-scaling group will scale the desired task count with a minimum of 1 task and a maximum of N tasks depending on the app environment.
The default maximum number of tasks is 3 for development app environments and 6 for production app environments. Deploy configuration can be used to override this behaviour.
If backed by EC2, the auto-scaling group will scale the number of instances based on CPU utilisation of the instances with a minimum of 1 instance and a maximum of N instances depending on the app environment.
The default maximum number of EC2 instances is 3 for development app environments and 6 for production app environments. Deploy configuration can be used to override this behaviour.
When it comes to networking, ECS services need to be deployed to VPCs; if a VPC is defined in the blueprint and linked to the workflow, it will be used, otherwise the default VPC for the account will be used. The load balancer will be placed in the public subnet by default, but can be configured to be placed in a private subnet by setting the celerity.workflow.vpc.lbSubnetType annotation to private. The service for the application will be deployed to a public subnet by default, but can be configured to be deployed to a private subnet by setting the celerity.workflow.vpc.subnetType annotation to private.
By default, 2 private subnets and 2 public subnets are provisioned for the associated VPC, the subnets are spread across 2 availability zones, the ECS resources that need to be associated with a subnet will be associated with these subnets, honouring the workflow subnet type defined in the annotations.
The CPU to memory ratio used by default for AWS deployments backed by EC2 is that of the t3.* instance family. The auto-scaling launch configuration will use the appropriate instance type based on the requirements of the application, these requirements will be taken from the deploy configuration or derived from the handlers configured for the workflow. If the requirements can not be derived, a default instance type will be selected. For production app environments, the default instance type will be t3.medium with 2 vCPUs and 4GB of memory. For development app environments, the default instance type will be t3.small with 2 vCPUs and 2GB of memory.
Fargate-backed ECS deployments use the same CPU to memory ratios allowed for Fargate tasks as per the task definition parameters.
If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. For an EC2-backed cluster, the task housing the containers that make up the service for the workflow will be deployed with less than 50 percent of memory and 0.8 vCPUs. Less than half of the memory and CPU is allocated to the EC2 instance to allow for smooth deployments of new versions of the workflow, this is done by making sure there is enough memory and CPU available to the ECS agent. The exact memory usage values for the defaults would be 1,792MB for production app environments and 870MB for development app environments.
For a Fargate-backed cluster, in production app environments, the task housing the containers that make up the service for the workflow application will be deployed with 2GB of memory and 1 vCPU. In development app environments, the task for the workflow application will be deployed with 1GB of memory and 0.5 vCPUs.
A sidecar ADOT collector container is deployed with the workflow to collect traces and metrics for the application, this will take up a small portion of the memory and CPU allocated to the application. Traces are always collected for workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
EKS
When a workflow is first deployed to EKS, a new cluster is created for the workflow unless you specify an existing cluster to use in the deploy configuration.
using existing clusters
When using an existing cluster, the cluster must be configured in a way that is compatible with the VPC annotations configured for the workflow as well as the target compute type.
For example, a cluster without a Fargate profile can not be used to deploy a workflow that is configured to use Fargate. Another example would be a cluster with a node group only associated with public subnets not being compatible with a workflow with the celerity.workflow.vpc.subnetType annotation set to private.
You also need to make sure there is enough memory and CPU allocated for node group instances to run the application in addition to other workloads running in the cluster.
Cost of running on EKS
Running a Celerity application on EKS will often not be the most cost-effective option for APIs with low traffic or applications that are not expected to use a lot of resources. All EKS clusters have a fixed cost of $74 per month for the control plane, in addition to the cost of the EC2 instances or Fargate tasks that are used to run the application along with cost of the load balancer, data transfer and networking components. If you are looking for a cost-effective solution for low-traffic/low-load applications on AWS, consider using ECS backed by Fargate instead.
The cluster is configured with a private endpoint for the Kubernetes API server by default, this can be overridden in the deploy configuration. (VPC links will be required to access the Kubernetes API server with the default configuration)
For an EKS cluster backed by EC2, a node group is configured with auto-scaling configuration to have a minimum size of 2 nodes and a maximum size of 6 nodes by default for production app environments. For development app environments, the minimum size of a node group is 1 with a maximum size of 3 by default. Auto-scaling is handled by the Kubernetes Cluster Autoscaler. The instance type configured for node groups is determined by the CPU and memory requirements defined in the deploy configuration or derived from the handlers of the workflow, if the requirements can not be derived, a default instance type will be selected. For production app environments, the default instance type will be t3.medium with 2 vCPUs and 4GB of memory. For development app environments, the default instance type will be t3.small with 2 vCPUs and 2GB of memory.
For an EKS cluster backed by Fargate, a Fargate profile is configured to run the workflow.
The Kubernetes Horizontal Pod Autoscaler is used to scale the number of pods running the workflow application based on CPU utilisation and average memory utilisation. In development app environments, the minimum number of pods is set to 1 and the maximum number of pods is set to 3 by default. In production app environments, the minimum number of pods is set to 2 and the maximum number of pods is set to 6 by default. The minimum and maximum number of pods can be overridden in the deploy configuration.
Once the cluster is up and running, Kubernetes services are provisioned to run the application, an Ingress service backed by an application load balancer is created to route traffic to the service, if you require private access to the workflow, the load balancer can be configured to be internal. When domain configuration is provided and the load balancer is public-facing, an ACM certificate is created for the domain and attached to the ingress service via annotations, you will need to verify the domain ownership before the certificate can be used.
When it comes to networking, EKS services need to be deployed to VPCs; if a VPC is defined in the blueprint and linked to the workflow, it will be used, otherwise the default VPC for the account will be used.
The ingress service backed by an application load balancer will be placed in the public subnet by default, but can be configured to be placed in a private subnet by setting the celerity.workflow.vpc.lbSubnetType annotation to private.
By default, 2 private subnets and 2 public subnets are provisioned for the associated VPC, the subnets are spread across 2 availability zones. For EC2-backed clusters, the EKS node group will be associated with all 4 subnets when celerity.workflow.vpc.subnetType is set to public; when celerity.api.vpc.subnetType is set to private, the node group will only be associated with the 2 private subnets. The orchestrator will take care of assigning one of the subnets defined for the node group.
For Fargate-backed clusters, the Fargate profile will be associated with the private subnets due to the limitations with Fargate profiles. For Fargate, the workflow will be deployed to one of the private subnets associated with the profile.
Warning
celerity.workflow.vpc.subnetType has no effect for Fargate-based EKS deployments, the workflow application will always be deployed to a private subnet.
If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. For an EC2-backed cluster, the containers that make up the service for the workflow will be deployed with less than 50 percent of memory and 0.8 vCPUs. Less than half of the memory and CPU is allocated to a node that will host the containers to allow for smooth deployments of new versions of the workflow, this is done by making sure there is enough memory and CPU available to the Kubernetes agents. The exact memory usage values for the defaults would be 1,792MB for production app environments and 870MB for development app environments.
For a Fargate-backed cluster, in production app environments, the pod for the application will be deployed with 2GB of memory and 0.5 vCPUs. In development app environments, the pod for the workflow application will be deployed with 1GB of memory and 0.5 vCPUs. Fargate has a fixed set of CPU and memory configurations that can be used.
A sidecar ADOT collector container is deployed in the pod with the workflow application to collect traces and metrics for the application, this will take up a small portion of the memory and CPU allocated to the workflow. Traces are always collected for workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
Google Cloud
🚀 Planned for v1 - This target environment will be available in a future v0 evolution.
In the Google Cloud environment, workflows are deployed as a containerised version of the Celerity workflow runtime.
Workflows can be deployed to Cloud Run, as well as Google Kubernetes Engine (GKE) in Autopilot or Standard mode using deploy configuration for the Google Cloud target environment. Cloud Run, a serverless container platform, is used by default, set by the gcloud.compute.containerService option. Handlers for executeStep states are loaded into the workflow runtime's container.
The workflow runtime stores execution state in PostgreSQL. By default, a Cloud SQL for PostgreSQL database is created when the workflow is deployed. You can use a linked celerity/sqlDatabase or an existing database instead; see Execution Store for more information.
Cloud Run
Cloud Run is a relatively simple environment to deploy workflows to, the workflow is deployed as a containerised application that is fronted by either an internal or external load balancer.
Cloud Run supports both hosting profiles. The always-on profile is used by default. When the gcloud.workflow.hostingProfile deploy configuration option is set to scaleToZero, the minimum number of instances is set to 0 and a Cloud Scheduler job is created to wake the service periodically so that it can resume executions that are due.
Autoscaling is configured with the use of Cloud Run annotations through autoscaling.knative.dev/minScale and autoscaling.knative.dev/maxScale annotations. The knative autoscaler will scale the number of instances based on the number of requests and the CPU and memory usage of the instances. By default, under the always-on hosting profile, for production environments, the application will be configured to scale the number of instances with a minimum of 2 instances and a maximum of 5 instances. The default values for development app environments are a minimum of 1 instance and a maximum of 3 instances. Deploy configuration can be used to override this behaviour.
When domain configuration is provided and the load balancer is public-facing and Google-managed, a managed TLS certificate is created for the domain and attached to the load balancer, you will need to verify the domain ownership before the certificate can be used.
For Cloud Run, the workflow will not be associated with a VPC, defining custom VPCs for Cloud Run applications is not supported. Creating and linking a VPC to the workflows will enable the Internal networking mode in the network ingress settings. celerity.workflow.vpc.subnetType has no effect for Cloud Run deployments, the application will always be deployed to a network managed by Google Cloud. Setting celerity.workflow.vpc.lbSubnetType to private will have the same affect as attaching a VPC to the application, making the application load balancer private. Setting celerity.workflow.vpc.lbSubnetType to public will have the same effect as not attaching a VPC to the workflow, making the application load balancer public. public is equivalent to the "Internal and Cloud Load Balancing" ingress setting.
Memory and CPU resources allocated to the workflow can be defined in the deploy configuration, when not defined, memory and CPU will be derived from the handlers configured for the workflow. If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. The Cloud Run service will be allocated a limit of 2GB of memory and 1 vCPU per instance in production app environments. For development app environments, the service will be allocated a limit of 1GB of memory and 0.5 vCPUs per instance.
A sidecar OpenTelemetry Collector container is deployed in the service with the workflow to collect traces and metrics, this will take up a small portion of the memory and CPU allocated to the workflow. Traces will always be collected for Workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
GKE
When a workflow is first deployed to GKE, a new cluster is created for the workflow unless you specify an existing cluster to use in the deploy configuration.
Using existing clusters
When using an existing cluster, the cluster must be configured in a way that is compatible with the VPC annotations configured for the application as well as the target compute type.
Cost of running on GKE
Running a Celerity application on GKE will often not be the most cost-effective option for APIs with low traffic or applications that are not expected to use a lot of resources. All GKE clusters have a fixed cost of around $72 per month for cluster management (control plane etc.), in addition to the cost of the nodes (VMs) that are used to run the application pods along with cost of the load balancer, data transfer and networking components. If you are looking for a cost-effective solution for low-traffic/low-load applications on Google Cloud, consider using Cloud Run with the scale-to-zero hosting profile instead.
When in standard mode, for production app environments, the cluster will be regional with 2 zones for better availability guarantees. A node pool is created with autoscaling enabled, by default, for production app environments, the pool will have a minimum of 1 node and a maximum of 3 nodes per zone. As the cluster has 2 zones, this will be a minimum of 2 nodes and a maximum of 6 nodes overall. The cluster autoscaler is used to manage scaling and choosing the appropriate instance type to use given the requirements of the workflow service. For development app environments, the cluster will be regional with 1 zone. The node pool will have a minimum of 1 node and a maximum of 3 nodes in the signle zone for development app environments. The minimum and maximum number of nodes can be overridden in the deploy configuration.
For standard mode, the machine type configured for node pools is determined by the CPU and memory requirements defined in the deploy configuration or derived from the handlers of the workflow application, if the requirements can not be derived, a default machine type will be selected. For production app environments, the default machine type will be n2-highcpu-4 with 4 vCPUs and 4GB of memory. For development app environments, the default machine type will be n1-highcpu-2 with 2 vCPUs and 2GB of memory. The machine type for node pools can be overridden in the deploy configuration.
When in autopilot mode, Google manages scaling, security and node pools. Based on memory and CPU limits applied at the pod-level, appropriate node instance types will be selected and will be scaled automatically. There is no manual autoscaling configuration when running in autopilot mode, GKE Autopilot is priced per pod request rather than provisioned infrastructure, depending on the nature of your workloads, it could be both a cost-effective and convenient way to run your applications. Read more about autopilot mode pricing.
In standard mode, the Kubernetes Horizontal Pod Autoscaler is used to scale the number of pods running the workflow application based on CPU utilisation and average memory utilisation. In development app environments, the minimum number of pods is set to 1 and the maximum number of pods is set to 3 by default. In production app environments, the minimum number of pods is set to 2 and the maximum number of pods is set to 6 by default. The minimum and maximum number of pods can be overridden in the deploy configuration.
When domain configuration is provided and the load balancer that powers the Ingress service is public-facing and Google-managed, a managed TLS certificate is created for the domain and attached to the Ingress object, you will need to verify the domain ownership before the certificate can be used.
When it comes to networking, a GKE cluster is deployed as a private cluster, nodes that the pods for the workflow run on only use internal IP addresses, isolating them from the public internet. The Control plane has both internal and external endpoints, the external endpoint can be disabled from the Google Cloud/Kubernetes side. When celerity.workflow.vpc.lbSubnetType is set to public, an Ingress service is provisioned using an external Application Load Balancer. When celerity.workflow.vpc.lbSubnetType is set to private, an Ingress service is provisioned using an internal Application Load Balancer. The Ingress service is used to route traffic from the public internet to the service running the workflow.
Warning
celerity.workflow.vpc.subnetType has no effect for GKE clusters, the application will always be deployed to a private network, the workflow API is exposed through the ingress service if the ingress is configured to be public.
If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. Limits of 1,792MB of memory and 1.6 vCPUs will be set for the pods that run the workflow in production app environments. For development app environments, the pods will be deployed with 870MB of memory and 0.8 vCPUs.
The OpenTelemetry Operator is used to configure a sidecar collector container for the workflow to collect traces and metrics. Traces will always be collected for workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
Azure
🚀 Planned for v1 - This target environment will be available in a future v0 evolution.
In the Azure environment, workflows are deployed as a containerised version of the Celerity workflow runtime.
Workflows can be deployed to Azure Container Apps or Azure Kubernetes Service (AKS) using deploy configuration for the Azure target environment. Azure Container Apps, a serverless container platform, is used by default, set by the azure.compute.containerService option. Handlers for executeStep states are loaded into the workflow runtime's container.
The workflow runtime stores execution state in PostgreSQL. By default, an Azure Database for PostgreSQL flexible server is created when the workflow is deployed. You can use a linked celerity/sqlDatabase or an existing database instead; see Execution Store for more information.
Container Apps
Container Apps is a relatively simple environment to deploy applications to, the workflow is deployed as a containerised application that is fronted by either external HTTPS ingress or internal ingress.
Container Apps supports both hosting profiles. The always-on profile is used by default. When the azure.workflow.hostingProfile deploy configuration option is set to scaleToZero, the minimum number of replicas is set to 0 and a scheduled Container Apps job is created to wake the service periodically so that it can resume executions that are due.
Autoscaling is determined based on the number of concurrent HTTP requests for public APIs, for private workflow APIs KEDA-supported CPU and memory scaling triggers are used. By default, under the always-on hosting profile, in production app environments, the scale definition is set to scale from 2 to 5 replicas per revision. For development app environments, the default scale definition is set to scale from 1 to 3 replicas per revision. The minimum and maximum number of replicas can be overridden in the deploy configuration.
When domain configuration is provided and the workflow API is configured to be public-facing, a managed TLS certificate is provisioned and attached to the Container App's HTTP ingress configuration. You will need to verify the domain ownership before the certificate can be used.
Container Apps will not be associated with a private network by default, a VNet is automatically generated for you and generated VNets are publicly accessible over the internet. Read about networking for Container Apps. When you define a VPC and link it to the workflow, a custom VNet will be provisioned and the workflow will be deployed to either a private or public subnet based on the celerity.workflow.vpc.subnetType annotation, defaulting to a public subnet if not specified. When the celerity.workflow.vpc.lbSubnetType is set to public, a public HTTPS ingress is provisioned for the API; when set to private, an internal HTTP ingress is provisioned for the workflow. Azure's built-in zone redundancy is used to ensure high availability of the workflow API.
Memory and CPU resources allocated to the workflow can be defined in the deploy configuration, when not defined, memory and CPU will be derived from the handlers configured for the workflow. If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. For production app environments, the Container App service will be allocated a limit of 2GB of memory and 1 vCPU per instance in the consumption plan, see allocation requirements. For development app environments, the Container App service will be allocated a limit of 1GB of memory and 0.5 vCPUs per instance in the consumption plan.
The OpenTelemetry Data Agent is used to collect traces and metrics for the application. Traces will always be collected for workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
AKS
When a workflow is first deployed to AKS, a new cluster is created for the workflow unless you specify an existing cluster to use in the deploy configuration.
Using existing clusters
When using an existing cluster, it must be configured in a way that is compatible with the VPC annotations configured for the workflow as well as the target compute type.
Cost of running on AKS
Running a Celerity application on AKS will often not be the most cost-effective option for APIs with low traffic or applications that are not expected to use a lot of resources. The default configuration uses instances that meet the minimum requirements to run Kubernetes in AKS that will cost hundreds of US dollars a month to run. If you are looking for a cost-effective solution for low-traffic/low-load applications on Azure, consider using Azure Container Apps with the scale-to-zero hosting profile instead.
The cluster is created across 2 availability zones for better availability guarantees. Best effort zone balancing is used with Azure VM Scale Sets. 2 separate node pools will be configured for the cluster, 1 for the Kubernetes system components and 1 for your workflow application. When using an existing cluster, a new node pool will be created specifically for this application.
The cluster is configured with an autoscaler for each node pool. For production app environments, by default, the autoscaler for the application node pool will be configured with a minimum of 3 nodes and a maximum of 6 nodes distributed across availability zones as per Azure's zone balancing. For development app environments, the autoscaler for the application node pool will be configured with a minimum of 1 node and a maximum of 3 nodes by default.
The autoscaler for the system node pool will be configured with a minimum of 2 nodes and a maximum of 3 nodes for production app environments. For development app environments, the autoscaler for the system node pool will be configured with a minimum of 1 node and a maximum of 2 nodes by default.
For both production and development app environments, the default node size for the system node pool is Standard_D4d_v5 with 4 vCPUs and 16GB of memory. This size has been chosen because of the minimum requirements for system Node Pools.
For the application node pool, the default node size differs based on the app environment. For production app environments, the default node size is Standard_D4ls_v6 with 4 vCPUs and 8GB of memory. For development app environments, the default node size is Standard_D2ls_v6 with 2 vCPUs and 4GB of memory.
If the CPU or memory requirements of the workflow application defined in the app blueprint cause the default node size to not be able to comfortably run 2 instances of the application, a larger node size will be selected.
Min and max node count along with the node size for both system and application node pools can be overridden in the deploy configuration.
The Kubernetes Horizontal Pod Autoscaler is used to scale the number of pods running the workflow application based on CPU utilisation and average memory utilisation. In development app environments, the minimum number of pods is set to 1 and the maximum number of pods is set to 6 by default. In production app environments, the minimum number of pods is set to 2 and the maximum number of pods is set to 12 by default. The minimum and maximum number of pods can be overridden in the deploy configuration.
When domain configuration is provided and the load balancer that powers the Ingress service is public-facing, a TLS certificate is generated with Let's Encrypt via cert-manager to provision certificates for domains associated with Ingress resources in Kubernetes. The certificate is used by the Ingress object to terminate TLS traffic. You will need to verify the domain ownership before the certificate can be used.
When it comes to networking, the workflow will be deployed with the overlay network model in a public network as per the default AKS access mode. Read about private and public clusters for AKS.
When you define a VPC and link it to the workflow, the application will be deployed as a private cluster using the VNET integration feature of AKS where the control plane will not be made available through a public endpoint. The celerity.workflow.vpc.subnetType annotation has no effect for AKS deployments as the networking model for Azure with it's managed Kubernetes offering is different from other cloud providers and all services running on a cluster are private by default, exposed to the internet through a load balancer or ingress controller. When celerity.workflow.vpc.lbSubnetType is set to public, an Ingress service is provisioned using the nginx ingress controller that uses an external Azure Load Balancer under the hood. When celerity.workflow.vpc.lbSubnetType is set to private, the nginx Ingress controller is configured to use an internal Azure Load Balancer, read more about the private ingress controller.
Memory and CPU resources allocated to the workflow pod can be defined in the deploy configuration, if not specified, memory and CPU will be derived from handlers configured for the application. If memory and CPU is not defined in the deploy configuration and can not be derived from the handlers, some defaults will be set. For production app environments, the pod that runs the application will be allocated a limit of 1,792MB of memory and 0.8 vCPUs. For development app environments, the pod that runs the workflow will be allocated a limit of 870MB of memory and 0.4 vCPUs.
The OpenTelemetry Operator is used to configure a sidecar collector container for the workflow application to collect traces and metrics. Traces will always be collected for workflow executions, however, they are only collected for handlers when tracing is enabled for the handler.
Serverless Target Environments
Workflows can not be deployed to the aws-serverless, gcloud-serverless and azure-serverless deploy targets. Workflows are not mapped to AWS Step Functions, Google Cloud Workflows or Azure Logic Apps, and the Celerity workflow runtime is always deployed as a containerised service.
To run a workflow on serverless infrastructure, deploy it with the containerised deploy target for your cloud provider and choose a serverless container service with compute configuration, as you would for other application resource types:
| Cloud provider | Deploy target | Serverless container service |
|---|---|---|
| AWS | aws | ECS with aws.compute.launchType set to fargate |
| Google Cloud | gcloud | Cloud Run, the default gcloud.compute.containerService |
| Azure | azure | Container Apps, the default azure.compute.containerService |
App Deploy Configuration
Configuration specific to a target environment can be defined for celerity/workflow resources in the app deploy configuration file.
This section lists the configuration options that can be set in the deployTarget.config object in the app deploy configuration file.
Compute Configuration
Compute configuration that can be used for the celerity/api, celerity/consumer, celerity/schedule and the celerity/workflow resource types is documented here.
AWS Configuration Options
aws.workflow.store.connectionString
The connection string of an existing PostgreSQL database to use as the execution store for the workflow runtime. When set, Celerity does not provision a database for the workflow in Amazon RDS. The workflow runtime creates its own tables in the database and applies its schema migrations on startup.
The connection string contains credentials, so avoid committing a deploy configuration file that contains it to version control.
Type
string
Deploy Targets
aws
Default Value
There is no default value for this option. When it is not set and the workflow is not linked to a celerity/sqlDatabase, Celerity provisions a database for the workflow.
Example
{
"deployTarget": {
"name": "aws",
"appEnv": "production",
"config": {
"aws.workflow.store.connectionString": "postgresql://workflow_user:password@my-db.example.us-east-1.rds.amazonaws.com:5432/workflows"
}
}
}Google Cloud Configuration Options
gcloud.workflow.hostingProfile
The hosting profile for the workflow runtime. This only applies when the workflow runtime is deployed to Cloud Run, which is the default gcloud.compute.containerService.
Type
string
Deploy Targets
gcloud
Allowed Values
alwaysOn | scaleToZero
Default Value
alwaysOn
Example
{
"deployTarget": {
"name": "gcloud",
"appEnv": "production",
"config": {
"gcloud.workflow.hostingProfile": "scaleToZero"
}
}
}gcloud.workflow.store.connectionString
The connection string of an existing PostgreSQL database to use as the execution store for the workflow runtime. When set, Celerity does not provision a database for the workflow in Cloud SQL. The workflow runtime creates its own tables in the database and applies its schema migrations on startup.
The connection string contains credentials, so avoid committing a deploy configuration file that contains it to version control.
Type
string
Deploy Targets
gcloud
Default Value
There is no default value for this option. When it is not set and the workflow is not linked to a celerity/sqlDatabase, Celerity provisions a database for the workflow.
Example
{
"deployTarget": {
"name": "gcloud",
"appEnv": "production",
"config": {
"gcloud.workflow.store.connectionString": "postgresql://workflow_user:password@10.0.0.5:5432/workflows"
}
}
}Azure Configuration Options
azure.workflow.hostingProfile
The hosting profile for the workflow runtime. This only applies when the workflow runtime is deployed to Azure Container Apps, which is the default azure.compute.containerService.
Type
string
Deploy Targets
azure
Allowed Values
alwaysOn | scaleToZero
Default Value
alwaysOn
Example
{
"deployTarget": {
"name": "azure",
"appEnv": "production",
"config": {
"azure.workflow.hostingProfile": "scaleToZero"
}
}
}azure.workflow.store.connectionString
The connection string of an existing PostgreSQL database to use as the execution store for the workflow runtime. When set, Celerity does not provision a database for the workflow in Azure Database for PostgreSQL. The workflow runtime creates its own tables in the database and applies its schema migrations on startup.
The connection string contains credentials, so avoid committing a deploy configuration file that contains it to version control.
Type
string
Deploy Targets
azure
Default Value
There is no default value for this option. When it is not set and the workflow is not linked to a celerity/sqlDatabase, Celerity provisions a database for the workflow.
Example
{
"deployTarget": {
"name": "azure",
"appEnv": "production",
"config": {
"azure.workflow.store.connectionString": "postgresql://workflow_user:password@my-db.postgres.database.azure.com:5432/workflows"
}
}
}⚠️ Limitations
Celerity workflows are run by the Celerity workflow runtime on every target environment, not by a cloud provider's workflow service, so features specific to those services are not available.
- Workflows always require a container service and a PostgreSQL database, and can not be deployed to the
aws-serverless,gcloud-serverlessandazure-serverlesstarget environments. See Execution Store for ways to reduce the cost of the database. - Execution history is not visible in a cloud provider's workflow console, such as the AWS Step Functions console. Executions are accessed through the workflow API of the Celerity workflow runtime.
- Parallel execution of steps is supported; however, parallel execution is limited by the compute resources available to the workflow runtime. In the Celerity workflow runtime, parallel execution is achieved by running multiple handlers concurrently using the tokio runtime, tokio is configured to make use of multiple threads but is primarily an asynchronous runtime that uses a concurrency model optimised for efficiently using a single core to carry out I/O bound tasks. Using parallel execution for CPU-bound tasks may not have the desired effect when using the Celerity workflow runtime.
- Only handlers can be executed as steps in a workflow, not containers or third-party services, a workflow needs to be able to run end-to-end within the Celerity workflow runtime process.
- Nested workflows are not supported. A workflow can only contain handlers as steps, to get around this, you can trigger another workflow from a handler used for one of the steps and wait for the result.
- Mapping or iterating over a list of items as a part of a previous state's output is not supported in the Celerity workflow runtime. You will have to operate in the list of items within the application code of a handler that is executed as a step in the workflow.
- Celerity workflows only support handlers of the
celerity/handlerresource type for theexecuteStepstate type. There are no direct service integrations like those available in cloud services such as AWS Step Functions, where a state can call a cloud service API without a handler. To integrate with a cloud provider service, use a handler that interacts with the service's API. Built-in integration steps for Celerity resource types are being considered for a future iteration of workflows.
Last updated on