Testing custom extensions
The typical development loop for custom extensions would be as follows:
-
Write the extension code
-
Start a Synapse instance with the extension loaded
-
Send requests to Synapse using a Cerbos SDK or a tool like cURL to exercise the extension
Synapse includes a testing framework to simplify the above process. It enables test-driven development and automated testing of custom extensions under various conditions. Tests are written as Starlark scripts but they can be used to test any kind of WASM or Starlark extension. For each test suite, the test runner starts a fresh instance of Synapse using the configuration defined in the suite and executes the test functions against that instance. There is a rich library of utility functions available to craft and send requests to exercise the extensions under test.
A simple test suite
The test runner expects test suites to be defined in files that have names ending with _test.star such as my_awesome_test_suite_test.star. Test cases must be functions that begin with the prefix test_ and accept a single context parameter. Any function that doesn’t start with test_ will be ignored by the test runner so you are free to define other functions that don’t begin with test_ to help you organize your test code.
The following example shows a simple test suite with a test case named always_pass that always passes and another one called always_fail that always fails.
# This test case always passes
def test_always_pass(context):
return testing.ok("I always pass")
# This test case always fails
def test_always_fail(context):
return testing.fail("I always fail")
# This function is ignored by the test runner because it doesn't start with 'test_'
def ignored():
pass
Save the contents of the above as example_test.star and execute the test suite.
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
test /tests
It should produce output similar to the following:
== FAIL:example (/tests/example_test.star) Passed=1 Failed=1 Skipped=0 --- FAIL:always_fail | I always fail --- PASS:always_pass | I always pass
The above example is a bare minimum test suite to illustrate how the test runner works. Read on to learn how to write more useful tests that actually exercise your extensions.
Defining test suite parameters
The test runner provisions a fresh Synapse instance for each test suite. By default it is a basic server with no extensions and an in-process PDP configured to read policies from a directory named policies in the current working directory. You can provide your own Synapse configuration to use in the test suite by defining a test_suite object.
test_suite = struct(
synapse_config = testing.load_synapse_config("synapse.yaml"),
)
The above example instructs the test runner to start a Synapse instance using the configuration read from the file named synapse.yaml. It should be a valid configuration file as documented in Configuration.
It is also possible to provide Synapse configuration inline as a Starlark dictionary.
test_suite = struct(
synapse_config = {
"pdp": {
"inProcess": {
"storage": {
"driver": "disk",
"disk": {
"directory": "${SYNAPSE_ROOT}/policies"
}
}
}
}
}
)
The test runner sets the SYNAPSE_ROOT environment variable to the directory of the test file. You can use this in the configuration file to reference other files that are stored relative to the test file.
|
By default the test suite name is derived from the file name. You can provide a more descriptive name using the name attribute.
test_suite = struct(
name = "Tests for extension foo",
synapse_config = testing.load_synapse_config("synapse.yaml")
)
Set the skip attribute to make the test runner skip executing the tests defined in this test suite.
test_suite = struct(
name = "Tests for extension foo",
synapse_config = testing.load_synapse_config("synapse.yaml"),
skip = True
)
If an attribute named test_cases is defined, it specifies the test functions that belong to this test suite. It overrides the implicit test discovery mechanism and the test runner will only run the functions that are listed in the test_cases attribute.
def always_fail(context):
return testing.fail("I always fail")
test_suite = struct(
name = "Tests for extension foo",
synapse_config = testing.load_synapse_config("synapse.yaml"),
test_cases = {
"always_pass": lambda context: testing.ok("I always pss"), (1)
"always_fail": always_fail (2)
}
)
| 1 | A test function defined inline as a lambda |
| 2 | Reference to a test function defined elsewhere in the file |
In the example above, the always_fail function needs to be declared first before it’s referenced by the test_suite object because Starlark is a dialect of Python that always evaluates top to bottom. Things must be declared before they are referenced.
|
The testing module
Every test suite has access to the built-in testing module which provides utilty functions for writing Synapse tests.
| Name | Description |
|---|---|
|
Assert that the expression evaluates to true |
|
Load a Synapse configuration file from the given path |
|
Load a test data file from the give path (see reusing test data section for more details) |
|
Return this to explicitly mark the test as passed |
|
Return this to explicitly mark the test as failed |
|
Return this to explicitly mark the test as skipped |
Setting the status for a test
The return value of the test function determines its status. If the function returns nothing, a True value or testing.ok(), it’s considered to have passed. Triggering a runtime error, returning False or testing.fail() marks the test as failed. Returning testing.skip() marks the test as skipped.
The test case context
The context object passed to each test case function contains helper methods to make requests to the Synapse instance of that test suite.
test_suite = struct(
name = "Illustrate how to exercise different extension types",
synapse_config = testing.load_synapse_config("synapse.yaml"),
)
# Send a Cerbos CheckResources request to exercise configured proxy extensions
def test_check_resources(context):
request = struct(
request_id = "test",
principal = struct(id = "john", roles = ["employee"]),
resources = [struct(
actions = ["view"],
resource = struct(
kind = "invoice",
id = "XX125",
attr = { "owner": "john", "department": "IT", "geography": "GB" }
)
)]
)
have = context.check_resources(request)
return testing.assert(have.results[0].actions["view"] == "EFFECT_ALLOW")
# Send a Cerbos PlanResources request to exercise configured proxy extensions
def test_plan_resources(context):
request = struct(
request_id = "test",
principal = struct(id = "john", roles = ["employee"]),
actions = ["view"],
resource = struct(kind = "invoice")
)
have = context.plan_resources(request)
return testing.assert(set(have.actions) == set(["view"]) and have.filter.kind == "KIND_CONDITIONAL")
# Send an AuthZEN access evaluation request to exercise configured proxy extensions
def test_access_evaluation(context):
request = struct(
subject = struct(
type = "user",
id = "john",
properties = {"cerbos.roles": ["employee"]}
),
resource = struct(
type = "invoice",
id = "XX125",
properties = { "owner": "john", "department": "IT", "geography": "GB" }
),
action = struct(name = "view")
)
have = context.access_evaluation(request)
return testing.assert(have.decision)
# Send an AuthZEN access evaluation batch request to exercise configured proxy extensions
def test_access_evaluation_batch(context):
request = struct(
subject = struct(
type = "user",
id = "john",
properties = {"cerbos.roles": ["employee"]}
),
resource = struct(
type = "invoice",
id = "XX125",
properties = { "owner": "john", "department": "IT", "geography": "GB" }
),
evaluations = [struct(action = struct(name = "view"))]
)
have = context.access_evaluation_batch(request)
return testing.assert(have.evaluations[0].decision)
# Send an extenal authz request to the Envoy extension
def test_envoy_check(context):
request = struct(
attributes = struct(
source = struct(
principal = "spiffe://cerbos.dev/cerbie"
),
request = struct(
http = struct(
id = "foo",
method = "GET",
path = "/foo?op=check_response"
),
),
),
)
have = context.envoy_check(request)
return testing.assert(have.status.code == 0)
# Send an HTTP request to a configured route extension
def test_route_extension(context):
have = context.http_get("/ext/foo", {"op": "http_response"})
return testing.assert(have.status_code == 200 and have.body() == "Welcome")
| Name | Description |
|---|---|
|
Send an AuthZEN access evaluation request |
|
Send an AuthZEN access evaluation batch request |
|
Send a Cerbos CheckResources request |
|
Send an Envoy Check request |
|
Send an HTTP DELETE request to the path. |
|
Send an HTTP GET request to the path. |
|
Send an HTTP HEAD request to the path. |
|
Send an HTTP OPTIONS request to the path. |
|
Send an HTTP PATCH request to the path. |
|
Send an HTTP POST request to the path. |
|
Send an HTTP PUT request to the path. |
|
Send a Cerbos PlanResources request |
Reusing test data
You can reuse common requests and responses by defining them in a test data file. They can be loaded into the test suite using the testing.load_testdata function. The test data file must be a YAML or JSON file in the following format:
testData: (1)
req_john: (2)
checkResourcesRequest: (3)
requestId: "john"
principal:
id: "john"
roles: ["employee"]
resources:
- resource:
id: "XX125"
kind: "invoice"
attr:
owner: "john"
department: "Music"
geography: "GB"
actions: ["view"]
resp_john:
checkResourcesResponse:
requestId: "john"
results:
- resource:
id: "XX125"
kind: "invoice"
actions:
view: EFFECT_ALLOW
| 1 | Required. The file’s root object must be named testData. |
| 2 | Identifier for the data object. This is the name you use to reference different test data objects. |
| 3 | Type of the data. See below for acceptable values. |
The following types of data can be included in a test data file.
| Type | Schema |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
testdata = testing.load_testdata("check-resources-test-data.yaml") (1)
def test_external_test_data(context):
request = testdata["req_john"] (2)
have = context.check_resources(request)
want = testdata["resp_john"]
return testing.assert(have.results[0].actions["view"] == want.results[0].actions["view"])
| 1 | Load test data from file. This can be done inside the test function as well but creating a global variable allows it to be shared between all test functions. |
| 2 | Retrieve one of the named test data objects by its identifier. |
Parameterized tests
To avoid repetition, you can generate test cases within the test suite itself. The following example illustrates generating test cases using a test data file. The test data in this case is organized into request and response pairs (IDs prefixed with req_ and resp_). The script iterates through all the requests and generates test cases for each that checks whether the actual response matches the expected response.
def build_test_case(request, expected):
def test_case(context):
actual = context.check_resources(request)
for (i, expected_result) in enumerate(expected.results):
for (action, expected_effect) in dict(expected_result.actions).items():
if actual.results[i].actions[action] != expected_effect:
return testing.fail("Action {} of result {} should be {} but is {}".format(action, i, expected_effect, actual.results[i].actions[action]))
return test_case
def build_test_case_list():
testdata = testing.load_testdata("check-resources-test-data.yaml")
test_cases = {}
for (name, data) in testdata.items():
if name.startswith("req_"):
test_name = name.removeprefix("req_")
expected_resp = testdata.get("resp_" + test_name)
if expected_resp:
test_cases[test_name] = build_test_case(data, expected_resp)
return test_cases
test_suite = struct(
name = "CheckResources tests",
synapse_config = testing.load_synapse_config("synapse.yaml"),
test_cases = build_test_case_list() (1)
)
| 1 | Call the build_test_case_list() function to generate the set of test cases for this suite. |
Loading modules
You can load any of the supported modules for use in tests.
load("re", "re")
def test_regex(context):
return testing.assert(re.match("f.*", "foo"))
Running a subset of tests
Multiple directories and files can be passed as arguments to the test command. When an argument is a directory, it’s recursively searched to discover all files that have the _test.star suffix. If you want to run just a single test suite, pass the full path to the file. If you want to run only test cases that match a particular pattern, use the --match argument to pass a regular expression.
# Discover all the test suites under /tests and run them
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
test /tests
# Run only extensionx_test.star and extensiony_test.star suites
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
test /tests/extensionx/extensionx_test.star /tests/extensiony/extensiony_test.star
# Run only test cases whose names begin with "enrich"
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
test --match='^enrich' /tests
If you need to further process the test results (using a script, for instance) pass --output=jsonl to make the test runner produce output in JSONLines format.
Debugging failures
Pass --verbose to see the logs from Synapse instance and any output produced by the test cases through print().
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
test --verbose /tests
You can use the Starlark REPL to help debug issues with your test scripts. Try out tricky expressions in the REPL to figure out how they behave or load the test script in and invoke its functions by hand to test them.
$ docker run \
-v $(pwd):/tests:ro \
CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
starlark repl /tests/fail_test.star
The above command loads the test script into the REPL. Any functions defined in the script will be available for you to call from the REPL. Please note that some elements such as the full context object passed to tests during actual test runs won’t be available here to use. You may need to decouple the code that you want to debug from the test framework and pass in mock data to simulate the conditions during the actual test run.
Further reading
Even though you might be developing extensions in WASM, in order to do integration testing with Synapse you need to be aware of Starlark basics.
-
Most of the scripting information in Developing Starlark extensions is applicable for testing as well
-
Refer to the Starlark spec for language primitives