Break It

The organizers will make the source code of every build-it team’s submission available to the break-it teams. Included with that code will be some standard logfiles and information about how they were generated. Break-it teams will submit test cases that are evidence of a bug in a particular submission. Test cases demonstrate bugs in one of four categories: correctness, crash, integrity, confidentiality. The test case format differs depending on the category. All submissions of test cases are done via git. For breaker’s reference, this page describes the build-it specification.

Here are the salient details of the rest of this page:

• You may submit up to 10 bug reports, in the form of test cases or other evidence, for each build-it submission.
• Of these, at most one can be a privacy violation, one can be an integrity violation, and the rest can be either crashes or correctness bugs. (In essence, all integrity bugs are treated as equivalent, and likewise all privacy bugs.) Correctness and crash bug reports must be accompanied by a textual justification (which may argue that the bug is security-relevant). Different sorts of bugs are scored differently.
• Correctness bugs are vetted by the Oracle – a test case that fails on the Oracle will not be counted for points against any submission. However, those cases that succeed on the Oracle but fail on a submission may still be invalid due to conflicts between the specification and the Oracle (some of which were found during the Build-it round). Build-it teams may petition for points back during the Fix-it round in these cases.

Next we discuss the submission mechanism and format, the per-submission logfiles that we will provide, and finally discuss the different kinds of test cases.

Submission

Break submissions are made via git. Create a break folder in the top-level folder of your team’s repository (which is the same repository used to submit code during the build-it phase if you participated in that phase). Within that folder, you will specify test cases. Each test case consists of a directory within your break folder, and in that folder should be a single file test.json. Any changes to a test case file will be ignored once it has already been commited. You must create a new test directory with a fresh test.json file if you want to make modifications.

Submission file format

JSON files that specify test cases will look like the following (the format is explained below):

{
  "target_team": 9,
  "type": "integrity",
  "commands": [
    {
      "program": "logread",
      "args": ["-R", "-G", "GERDA"]
    }
  ],
  "logfile": "integ9",
  "replacement": "LVQgMCAtSyBCRlVYVU1YVSAtRyBPREEgLUEgQUlDVUNMTUQKLVQgMSAtSyBCRlVYVU1YVSAtRyBPREEgLUwgQUlDVUNMTUQK"
}

The type of the test is specified in the type attribute with either correctness, crash, integrity, or confidentiality. The team whose submission this test proves is buggy is identified in the target_team attribute, and is specified with an integer team ID from the team’s source code. The logfile attribute specifies the input log to use; it will name a file provided by the organizers, as described in detail shortly. Input log files are used for integrity and confidentiality breaks.

Each command under commands will, under the args parameter, specify command lines to logappend or logread, as specified by the program tag, as applied to the already-specified logfile. The output parameter of a command specifies its expected output to stdout (if any). Such output is not newline (and whitespace) sensitive.

The replacement parameter is used to provide substitute logfile contents, i.e., instead of the contents originally in the specified logfile. These contents are base64-encoded. This parameter is used in integrity tests (only), discussed below.

Correctness violations should also be accompanied by a bug report, described below.

Supplied Log Files

Prior to each break-it team receiving access to it, a build-it team’s submission will be modified to contain top-level directories called integ and conf. These directories will contain several logfiles that were generated by that team’s implementation. Those in integ will be accompanied with complete transcripts of the logappend commands that generated the file, excluding the authentication tokens.

These logs will be named by an alphanumeric string, for example:

• a941f06a58b361fdc279e75b29ad34227f53f8b8 is the log produced by a submission
• a941f06a58b361fdc279e75b29ad34227f53f8b8.logappend is the transcript of the command line invocations of logappend used to create the log. This file might not be present for some logs, and in some transcripts the token used might be redacted.

When submitting tests that are based on supplied logs, you will specify the name of the log in the logfile parameter.

There are four types of break-it tests you can submit, described next.

Correctness Violations

A correctness violation represents a deviation from the specification. This could take the form of invalid output, return codes, or incorrect responses to queries. To demonstrate a correctness violation, submit a test with the type attribute set to correctness and commands attributes to demonstrate the correctness error. The elements of commands can describe steps to build a log. The attribute batch may optionally be included, which is a base64 encoded batch file. It will be called “batch” when uploaded to the test machine.

To test a correctness violation, the commands will be run on the oracle and then the target build-it team. If any standard output or return codes differ, the correctness violation is confirmed. Remember that multiple break submissions against the same bug will gain you no additional points.

As an example, if you wanted to submit a testcase that is evidence of a correctness violation in team 10’s submission, you could create a file break/orangelemon1/test.json like so:

{
  "target_team": 10,
  "type": "correctness",
  "commands": [
    {
      "program": "logappend",
      "args": ["-K", "secret", "-T", "0", "-G", "GERDA", "-A", "log"]
    },
    {
      "program": "logappend",
      "args": ["-K", "secret", "-T", "1", "-G", "GERDA", "-A", "-R", "16", "log"]
    },
    {
      "program": "logread",
      "args": [ "-K", "secret", "-R", "-G", "GERDA", "log"],
    }
  ],
  "batch": "LUsgc2VjcmV0IC1FIGVtcGwgLUEgbG9nCg=="
}

These examples show how to call logappend and generate log files.

For correctness violations, provide a textual justification for why you think the test case you are submitting is indeed a bug. Do this in a file ending with .txt with the same name as your .json file. This file should be no more than 250 words. It may refer to relevant portions of this spec (by HTML file name and line/section number) and/or line numbers in the submitted code, if you like.

Correctness violations against ambiguities or differences between the specification and oracle are invalid and will be rejected. Build-it teams will have the opportunity to dispute invalid break submissions during the fix-it round.

Crash Violations

A crash occurs when a program unexpectedly terminates due to violations of memory safety. To demonstrate a crash, submit a test with the type attribute set to crash and commands attributes to demonstrate the crash. The elements of commands can describe steps to build a log.

{
  "target_team": 10,
  "type": "crash",
  "commands": [
    {
      "program": "logappend",
      "args": ["-K", "secret", "-T", "0", "-G", "GERDA", "-A", "log"]
    },
    {
      "program": "logappend",
      "args": ["-K", "secret", "-T", "1", "-G", "GERDA", "-A", "-R", "16", "log"]
    },
    {
      "program": "logread",
      "args": [ "-K", "secret", "-R", "-G", "GERDA", "log"],
    }
  ],
  "batch": "LUsgc2VjcmV0IC1FIGVtcGwgLUEgbG9nCg=="
}

For crash violations, provide a textual justification for why you think the test case you are submitting is indeed a crash caused by memory violation. Do this in a file ending with .txt with the same name as your .json file. This file should be no more than 250 words. It may refer to relevant portions of this spec (by HTML file name and line/section number) and/or line numbers in the submitted code, if you like.

Integrity Violations

An integrity violation occurs when an attacker can successfully modify a log without knowledge of the token used to produce that file. Evidence of successful modification is that the modified file is interpreted correctly by logread (when using the token that was used to create the original log). The tokens for supplied log files will not be provided.

To demonstrate that this is possible, the replacement attribute should be a modified version of one of the supplied logs for which there is a transcript. The commands attribute should contain a single query, with the token and name of the log omitted. A correct implementation should always exit with the output “integrity violation”; an incorrect one will produce some other output.

Here is an example demonstrating an integrity violation against team 9.

{
  "target_team": 9,
  "type": "integrity",
  "commands": [
    {
      "program": "logread",
      "args": ["-R", "-G", "GERDA"]
    }
  ],
  "logfile": "integ9",
  "replacement": "LVQgMCAtSyBCRlVYVU1YVSAtRyBPREEgLUEgQUlDVUNMTUQKLVQgMSAtSyBCRlVYVU1YVSAtRyBPREEgLUwgQUlDVUNMTUQK"
}

This tells the testing framework to run team 9’s logread against both our “integ9” file and the contestant-provided “replacement” file, whose contents are defined by inputfile; in both cases logread is run using the correct authentication token. If the query command against “integ9” logfile and against the “replacement” file produce different non-zero lenth standard outs and no errors, then an integrity violation has been detected.

Integrity breaks may also include multiple replacement log files. This is done by providing an array of replacement inputs. Each object in the array should have the name and content attributes. Here is an example.

{
  "target_team": 9,
  "type": "integrity",
  "commands": [
    {
      "program": "logread",
      "args": ["-R", "-G", "GERDA"]
    }
  ],
  "logfile": "integ9",
  "replacement": [
    {
      "name": "integ9",
      "content": "LVQgMCAtSyBCRlVYVU1YVSAtRyBPREEgLUEgQUlDVUNMTUQKLVQgMSAtSyBCRlVYVU1YVSAtRyBPREEgLUwgQUlDVUNMTUQK"
    }
  ]
}

Confidentiality Violations

A confidentiality violation occurs when an attacker can infer information about the contents of a log without knowledge of the token or transcript. Evidence of such a violation is a successful guess of an outcome of a query on the file, despite not knowing the token or the transcript.

To demonstrate that this is possible, the commands attribute should contain a single query provided with the token and name of the log omitted, along with a output attribute that contains the expected answer, had the token been given. Our testing infrastructure will run with the token, and thus if the expected output is indeed produced, it is clear that the break-it team has reverse-engineered the contents of the file.

Here is an example of submitting a confidentiality test against team 9. In this test, the submitting team knows the room information for GERDA, even though the logfile “conf8” was provided with neither token nor transcript.

{
  "target_team": 9,
  "type": "confidentiality",
  "logfile": "conf8",
  "commands": [
    {
      "program": "logread",
      "args": ["-R", "-G", "GERDA"],
      "output": "16"
    }
  ]
}

Scoring

Both integrity and confidentiality violations are considered vulnerabilities by the scoring system. Crashes receive their own category by the scoring system. All other correctness violations are not scored as vulnerabiltiies. If you have reason to argue that other correctness violations should be considered security relevant, include the justification in your error report.

Limitations

You may only submit up to 10 submissions against a single team. You will only be allowed to make one integrity submission and one confidentiality submission against a single team.