Testing custom extensions

The typical development loop for custom extensions would be as follows:

  1. Write the extension code

  2. Start a Synapse instance with the extension loaded

  3. Send requests to Synapse using a Cerbos SDK or a tool like cURL to exercise the extension

Synapse includes a testing framework to simplify the above process. It enables test-driven development and automated testing of custom extensions under various conditions. Tests are written as Starlark scripts but they can be used to test any kind of WASM or Starlark extension. For each test suite, the test runner starts a fresh instance of Synapse using the configuration defined in the suite and executes the test functions against that instance. There is a rich library of utility functions available to craft and send requests to exercise the extensions under test.

Other ways of testing

Using the testing framework provided by Synapse is not mandatory. Most programming languages have support for starting a Docker container programmatically for tests. The Testcontainers project is one such example. You can author tests in your preferred language and use a container testing library to spin up Synapse containers you need.

Most CI systems also have a concept of "services" where you can configure the CI runner to start a container of your choice and make it available for the duration of the test run. Refer to the documentation of your CI system for more information.

A simple test suite

The test runner expects test suites to be defined in files that have names ending with _test.star such as my_awesome_test_suite_test.star. Test cases must be functions that begin with the prefix test_ and accept a single context parameter. Any function that doesn’t start with test_ will be ignored by the test runner so you are free to define other functions that don’t begin with test_ to help you organize your test code.

The following example shows a simple test suite with a test case named always_pass that always passes and another one called always_fail that always fails.

example_test.star
# This test case always passes
def test_always_pass(context):
    return testing.ok("I always pass")

# This test case always fails
def test_always_fail(context):
    return testing.fail("I always fail")

# This function is ignored by the test runner because it doesn't start with 'test_'
def ignored():
    pass

Save the contents of the above as example_test.star and execute the test suite.

$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test /tests

It should produce output similar to the following:

== FAIL:example (/tests/example_test.star)
   Passed=1 Failed=1 Skipped=0
   --- FAIL:always_fail | I always fail
   --- PASS:always_pass | I always pass

The above example is a bare minimum test suite to illustrate how the test runner works. Read on to learn how to write more useful tests that actually exercise your extensions.

Defining test suite parameters

The test runner provisions a fresh Synapse instance for each test suite. By default it is a basic server with no extensions and an in-process PDP configured to read policies from a directory named policies in the current working directory. You can provide your own Synapse configuration to use in the test suite by defining a test_suite object.

example_test.star
test_suite = struct(
   synapse_config = testing.load_synapse_config("synapse.yaml"),
)

The above example instructs the test runner to start a Synapse instance using the configuration read from the file named synapse.yaml. It should be a valid configuration file as documented in Configuration.

It is also possible to provide Synapse configuration inline as a Starlark dictionary.

example_test.star
test_suite = struct(
   synapse_config = {
        "pdp": {
            "inProcess": {
                "storage": {
                    "driver": "disk",
                    "disk": {
                        "directory": "${SYNAPSE_ROOT}/policies"
                    }
                }
            }
        }
   }
)
The test runner sets the SYNAPSE_ROOT environment variable to the directory of the test file. You can use this in the configuration file to reference other files that are stored relative to the test file.

By default the test suite name is derived from the file name. You can provide a more descriptive name using the name attribute.

example_test.star
test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml")
)

Set the skip attribute to make the test runner skip executing the tests defined in this test suite.

example_test.star
test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
   skip = True
)

If an attribute named test_cases is defined, it specifies the test functions that belong to this test suite. It overrides the implicit test discovery mechanism and the test runner will only run the functions that are listed in the test_cases attribute.

example_test.star
def always_fail(context):
    return testing.fail("I always fail")

test_suite = struct(
   name = "Tests for extension foo",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
   test_cases = {
      "always_pass": lambda context: testing.ok("I always pss"), (1)
      "always_fail": always_fail (2)
   }
)
1 A test function defined inline as a lambda
2 Reference to a test function defined elsewhere in the file
In the example above, the always_fail function needs to be declared first before it’s referenced by the test_suite object because Starlark is a dialect of Python that always evaluates top to bottom. Things must be declared before they are referenced.

The testing module

Every test suite has access to the built-in testing module which provides utilty functions for writing Synapse tests.

Table 1. Functions available in the testing module
Name Description

testing.assert(expression)

Assert that the expression evaluates to true

testing.load_synapse_config(path)

Load a Synapse configuration file from the given path

testing.load_testdata(path)

Load a test data file from the give path (see reusing test data section for more details)

testing.ok(message)

Return this to explicitly mark the test as passed

testing.fail(message)

Return this to explicitly mark the test as failed

testing.skip(message)

Return this to explicitly mark the test as skipped

Setting the status for a test

The return value of the test function determines its status. If the function returns nothing, a True value or testing.ok(), it’s considered to have passed. Triggering a runtime error, returning False or testing.fail() marks the test as failed. Returning testing.skip() marks the test as skipped.

The test case context

The context object passed to each test case function contains helper methods to make requests to the Synapse instance of that test suite.

example_test.star
test_suite = struct(
   name = "Illustrate how to exercise different extension types",
   synapse_config = testing.load_synapse_config("synapse.yaml"),
)

# Send a Cerbos CheckResources request to exercise configured proxy extensions
def test_check_resources(context):
    request = struct(
        request_id = "test",
        principal = struct(id = "john", roles = ["employee"]),
        resources = [struct(
            actions = ["view"],
            resource = struct(
                kind = "invoice",
                id = "XX125",
                attr = { "owner": "john", "department": "IT", "geography": "GB"  }
            )
        )]
    )

    have = context.check_resources(request)
    return testing.assert(have.results[0].actions["view"] == "EFFECT_ALLOW")

# Send a Cerbos PlanResources request to exercise configured proxy extensions
def test_plan_resources(context):
    request = struct(
        request_id = "test",
        principal = struct(id = "john", roles = ["employee"]),
        actions = ["view"],
        resource = struct(kind = "invoice")
    )

    have = context.plan_resources(request)
    return testing.assert(set(have.actions) == set(["view"]) and have.filter.kind == "KIND_CONDITIONAL")

# Send an AuthZEN access evaluation request to exercise configured proxy extensions
def test_access_evaluation(context):
    request = struct(
        subject = struct(
            type = "user",
            id = "john",
            properties = {"cerbos.roles": ["employee"]}
        ),
        resource = struct(
            type = "invoice",
            id = "XX125",
            properties =  { "owner": "john", "department": "IT", "geography": "GB"  }
        ),
        action = struct(name = "view")
    )

    have = context.access_evaluation(request)
    return testing.assert(have.decision)

# Send an AuthZEN access evaluation batch request to exercise configured proxy extensions
def test_access_evaluation_batch(context):
    request = struct(
        subject = struct(
            type = "user",
            id = "john",
            properties = {"cerbos.roles": ["employee"]}
        ),
        resource = struct(
            type = "invoice",
            id = "XX125",
            properties =  { "owner": "john", "department": "IT", "geography": "GB"  }
        ),
        evaluations = [struct(action = struct(name = "view"))]
    )

    have = context.access_evaluation_batch(request)
    return testing.assert(have.evaluations[0].decision)

# Send an extenal authz request to the Envoy extension
def test_envoy_check(context):
    request = struct(
        attributes = struct(
            source = struct(
                principal = "spiffe://cerbos.dev/cerbie"
            ),
            request = struct(
                http = struct(
                   id = "foo",
                   method = "GET",
                   path = "/foo?op=check_response"
                ),
            ),
        ),
    )

    have = context.envoy_check(request)
    return testing.assert(have.status.code == 0)

# Send an HTTP request to a configured route extension
def test_route_extension(context):
   have = context.http_get("/ext/foo", {"op": "http_response"})
   return testing.assert(have.status_code == 200 and have.body() == "Welcome")
Table 2. Functions available in the context object
Name Description

context.access_evaluation(request)

Send an AuthZEN access evaluation request

context.access_evaluation_batch(request)

Send an AuthZEN access evaluation batch request

context.check_resources(request)

Send a Cerbos CheckResources request

context.envoy_check(request)

Send an Envoy Check request

context.http_delete(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP DELETE request to the path.

context.http_get(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP GET request to the path.

context.http_head(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP HEAD request to the path.

context.http_options(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP OPTIONS request to the path.

context.http_patch(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP PATCH request to the path.

context.http_post(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP POST request to the path.

context.http_put(path, params=None, headers=None, auth=(), body=None, json_body=None, form_body=None, form_encoding="", timeout=30, allow_redirects=True, verify=True)

Send an HTTP PUT request to the path.

context.plan_resources(request)

Send a Cerbos PlanResources request

Reusing test data

You can reuse common requests and responses by defining them in a test data file. They can be loaded into the test suite using the testing.load_testdata function. The test data file must be a YAML or JSON file in the following format:

testdata.yaml
testData: (1)
  req_john: (2)
    checkResourcesRequest: (3)
      requestId: "john"
      principal:
        id: "john"
        roles: ["employee"]
      resources:
        - resource:
            id: "XX125"
            kind: "invoice"
            attr:
              owner: "john"
              department: "Music"
              geography: "GB"
          actions: ["view"]

  resp_john:
    checkResourcesResponse:
      requestId: "john"
      results:
        - resource:
            id: "XX125"
            kind: "invoice"
          actions:
            view: EFFECT_ALLOW
1 Required. The file’s root object must be named testData.
2 Identifier for the data object. This is the name you use to reference different test data objects.
3 Type of the data. See below for acceptable values.

The following types of data can be included in a test data file.

Type Schema

authzenEvaluationBatchRequest

authzen.authorization.v1.AccessEvaluationBatchRequest

authzenEvaluationBatchResponse

authzen.authorization.v1.AccessEvaluationBatchResponse

authzenEvaluationRequest

authzen.authorization.v1.AccessEvaluationRequest

authzenEvalationResponse

authzen.authorization.v1.AccessEvaluationResponse

checkResourcesRequest

cerbos.request.v1.CheckResourcesRequest

checkResourcesResponse

cerbos.response.v1.CheckResourcesResponse

envoyCheckRequest

envoy.service.auth.v3.CheckRequest

envoyCheckResponse

envoy.service.auth.v3.CheckResponse

planResourcesRequest

cerbos.request.v1.PlanResourcesRequest

planResourcesResponse

cerbos.response.v1.PlanResourcesResponse

example_test.star
testdata = testing.load_testdata("check-resources-test-data.yaml") (1)

def test_external_test_data(context):
    request = testdata["req_john"] (2)
    have = context.check_resources(request)
    want = testdata["resp_john"]
    return testing.assert(have.results[0].actions["view"] == want.results[0].actions["view"])
1 Load test data from file. This can be done inside the test function as well but creating a global variable allows it to be shared between all test functions.
2 Retrieve one of the named test data objects by its identifier.

Parameterized tests

To avoid repetition, you can generate test cases within the test suite itself. The following example illustrates generating test cases using a test data file. The test data in this case is organized into request and response pairs (IDs prefixed with req_ and resp_). The script iterates through all the requests and generates test cases for each that checks whether the actual response matches the expected response.

example_test.star
def build_test_case(request, expected):
    def test_case(context):
        actual = context.check_resources(request)
        for (i, expected_result) in enumerate(expected.results):
            for (action, expected_effect) in dict(expected_result.actions).items():
                if actual.results[i].actions[action] != expected_effect:
                    return testing.fail("Action {} of result {} should be {} but is {}".format(action, i, expected_effect, actual.results[i].actions[action]))


    return test_case

def build_test_case_list():
    testdata = testing.load_testdata("check-resources-test-data.yaml")
    test_cases = {}
    for (name, data) in testdata.items():
        if name.startswith("req_"):
            test_name = name.removeprefix("req_")
            expected_resp = testdata.get("resp_" + test_name)
            if expected_resp:
                test_cases[test_name] = build_test_case(data, expected_resp)

    return test_cases


test_suite = struct(
    name = "CheckResources tests",
    synapse_config = testing.load_synapse_config("synapse.yaml"),
    test_cases = build_test_case_list() (1)
)
1 Call the build_test_case_list() function to generate the set of test cases for this suite.

Loading modules

You can load any of the supported modules for use in tests.

example_test.star
load("re", "re")

def test_regex(context):
    return testing.assert(re.match("f.*", "foo"))

Running a subset of tests

Multiple directories and files can be passed as arguments to the test command. When an argument is a directory, it’s recursively searched to discover all files that have the _test.star suffix. If you want to run just a single test suite, pass the full path to the file. If you want to run only test cases that match a particular pattern, use the --match argument to pass a regular expression.

# Discover all the test suites under /tests and run them
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test /tests

# Run only extensionx_test.star and extensiony_test.star suites
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test /tests/extensionx/extensionx_test.star /tests/extensiony/extensiony_test.star

# Run only test cases whose names begin with "enrich"
$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test --match='^enrich' /tests

If you need to further process the test results (using a script, for instance) pass --output=jsonl to make the test runner produce output in JSONLines format.

Debugging failures

Pass --verbose to see the logs from Synapse instance and any output produced by the test cases through print().

$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    test --verbose /tests

You can use the Starlark REPL to help debug issues with your test scripts. Try out tricky expressions in the REPL to figure out how they behave or load the test script in and invoke its functions by hand to test them.

$ docker run \
    -v $(pwd):/tests:ro \
    CERBOS_DISTRIBUTION_REPO/synapse/synapse:latest \
    starlark repl /tests/fail_test.star

The above command loads the test script into the REPL. Any functions defined in the script will be available for you to call from the REPL. Please note that some elements such as the full context object passed to tests during actual test runs won’t be available here to use. You may need to decouple the code that you want to debug from the test framework and pass in mock data to simulate the conditions during the actual test run.

Further reading

Even though you might be developing extensions in WASM, in order to do integration testing with Synapse you need to be aware of Starlark basics.

  1. Most of the scripting information in Developing Starlark extensions is applicable for testing as well

  2. Refer to the Starlark spec for language primitives