Antenna configuration

Introduction

Configuration Antenna is not too hard. You have some basic choices to make. You codify the configuration into an env file. Then set ANTENNA_ENV environment variable to the path to the env file and you’re done.

Here’s an example. This uses Datadog installed on the EC2 node for metrics and also IAM bound to the EC2 node that Antenna is running on so it doesn’t need S3 credentials for crashstorage.

# Metrics things
METRICS_CLASS=antenna.metrics.DogStatsdMetrics
STATSD_NAMESPACE=mcboatface

# BreakdpadSubmitterResource settings
CRASHSTORAGE_CLASS=antenna.ext.s3.crashstorage.S3CrashStorage

# S3CrashStorage and S3Connection settings
CRASHSTORAGE_BUCKET_NAME=org-myorg-mybucket

Application

First, you need to configure the application-scoped variables.

Configuration

These all have sane defaults, so you don’t have to configure any of this.

Options:
  • BASEDIR (str) –

    The root directory for this application to find and store things.

    Defaults to '/home/docs/checkouts/readthedocs.org/user_builds/antenna/checkouts/latest'.

  • LOGGING_LEVEL (str) –

    The logging level to use. DEBUG, INFO, WARNING, ERROR or CRITICAL

    Defaults to 'DEBUG'.

  • METRICS_CLASS (<ListOf(everett.manager.parse_class)>) –

    (‘Comma-separated list of metrics backends to use. Possible options: “antenna.metrics.LoggingMetrics” and “antenna.metrics.DatadogMetrics”’,)

    Defaults to 'antenna.metrics.LoggingMetrics'.

  • SECRET_SENTRY_DSN (str) –

    Sentry DSN to use. See https://docs.sentry.io/quickstart/#configure-the-dsn for details. If this is not set an unhandled exception logging middleware will be used instead.

    Defaults to ''.

  • HOST_ID (str) –

    Identifier for the host that is running Antenna. This identifies this Antenna instance in the logs and makes it easier to correlate Antenna logs with other data. For example, the value could be a public hostname, an instance id, or something like that. If you do not set this, then socket.gethostname() is used instead.

    Defaults to ''.

Metrics

LoggingMetrics

component antenna.metrics.LoggingMetrics

Configuration for LoggingMetrics backend.

DogStatsd metrics

component antenna.metrics.DogStatsdMetrics

Configuration for DatadogMetrics backend.

Options:
  • STATSD_HOST (str) –

    Hostname for the statsd server

    Defaults to 'localhost'.

  • STATSD_PORT (int) –

    Port for the statsd server

    Defaults to '8125'.

  • STATSD_NAMESPACE (str) –

    Namespace for these metrics

    Defaults to ''.

Breakpad crash resource

component antenna.breakpad_resource.BreakpadSubmitterResource

Handles incoming breakpad crash reports and saves to crashstorage

This handles incoming HTTP POST requests containing breakpad-style crash reports in multipart/form-data format.

It can handle compressed or uncompressed POST payloads.

It parses the payload from the HTTP POST request, runs it through the throttler with the specified rules, generates a crash_id, returns the crash_id to the HTTP client and then saves the crash using the configured crashstorage class.

Note

From when a crash comes in to when it’s saved by the crashstorage class, the crash is entirely in memory. Keep that in mind when figuring out how to scale your Antenna nodes.

The most important configuration bit here is choosing the crashstorage class.

For example:

CRASHSTORAGE_CLASS=antenna.ext.s3.crashstorage.S3CrashStorage
Options:
  • DUMP_FIELD (str) –

    the name of the field in the POST data for dumps

    Defaults to 'upload_file_minidump'.

  • DUMP_ID_PREFIX (str) –

    the crash type prefix

    Defaults to 'bp-'.

  • CRASHSTORAGE_CLASS (everett.manager.parse_class) –

    the class in charge of storing crashes

    Defaults to 'antenna.ext.crashstorage_base.NoOpCrashStorage'.

  • CONCURRENT_CRASHMOVERS (int) –

    the number of crashes concurrently being saved to s3

    Defaults to '2'.

Throttler

component antenna.throttler.Throttler

Accepts/rejects incoming crashes based on specified rule set

The throttler can throttle incoming crashes using the content of the crash. To throttle, you set up a rule set which is a list of Rule instances. That goes in a Python module which is loaded at run time.

If you don’t want to throttle anything, use this:

THROTTLE_RULES=antenna.throttler.accept_all

To set up a rule set, put it in a Python file and define the rule set there. For example, you could have file myruleset.py with this in it:

from antenna.throttler import Rule

rules = [
    Rule('ProductName', 'Firefox', 100),
    # ...
]

then set THROTTLE_RULES to the path for that. For example, depending on the current working directory and PYTHONPATH, the above could be:

THROTTLE_RULES=myruleset.rules

FIXME(willkg): Flesh this out.

Options:THROTTLE_RULES (antenna.throttler.parse_attribute) –

Python dotted path to ruleset

Defaults to 'antenna.throttler.mozilla_rules'.

Crash storage

For crash storage, you have three options one of which is a no-op for debugging.

NoOpCrashStorage

The NoOpCrashStorage class is helpful for debugging, but otherwise shouldn’t be used.

component antenna.ext.crashstorage_base.NoOpCrashStorage

This is a no-op crash storage that logs crashes it would have stored

It keeps track of the last 10 crashes in .crashes instance attribute with the most recently stored crash at the end of the list. This helps when writing unit tests for Antenna.

Filesystem

The FSCrashStorage class will save crash data to disk. If you choose this, you’ll want to think about what happens to the crash after Antenna has saved it and implement that.

component antenna.ext.fs.crashstorage.FSCrashStorage

Saves raw crash files to the file system.

This generates a tree something like this which mirrors what we do on S3:

<FS_ROOT>/
    <YYYYMMDD>/
        raw_crash/
            <CRASHID>.json
        dump_names/
            <CRASHID>.json
        <DUMP_NAME>/
            <CRASHID>

Couple of things to note:

  1. This doesn’t ever delete anything from the tree. You should run another process to clean things up.
  2. If you run out of disk space, this component will fail miserably. There’s no way to recover from a full disk–you will lose crashes.

FIXME(willkg): Can we alleviate or reduce the likelihood of the above?

When set as the BreakpadSubmitterResource crashstorage class, configuration for this class is in the CRASHSTORAGE namespace.

Example:

CRASHSTORAGE_FS_ROOT=/tmp/whatever
Options:CRASHSTORAGE_FS_ROOT (str) –

path to where files should be stored

Defaults to '/tmp/antenna_crashes'.

AWS S3

The S3CrashStorage class will save crash data to AWS S3. You might be able to use this to save to other S3-like systems, but that’s not tested or supported.

component antenna.ext.s3.connection.S3Connection

Connection object for S3.

Credentials and permissions

When configuring this connection object, you can do one of two things:

  1. provide ACCESS_KEY and SECRET_ACCESS_KEY in the configuration, OR
  2. use one of the other methods described in the boto3 docs http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials

The AWS credentials that Antenna is configured with must have the following Amazon S3 permissions:

  • s3:ListBucket

    When Antenna starts up, S3Connection will call HEAD on the bucket verifying the bucket exists, the endpoint url is good, it’s accessible and the credentials are valid. This means that the credentials you use must have “list” permissions on the bucket.

    If that fails, then this will raise an error and will halt startup.

    Warning

    This does not verify that it has write permissions to the bucket. Make sure to test your configuration by sending a test crash and watch your logs at startup!

  • s3:PutObject

    This permission is used to save items to the bucket.

Retrying saves

When saving crashes, this connection will retry saving several times. Then give up. The crashmover coroutine will put the crash back in the queue to retry later. Crashes are never thrown out.

When set as the BreakpadSubmitterResource crashstorage class, configuration for this class is in the CRASHSTORAGE namespace.

Example:

CRASHSTORAGE_BUCKET_NAME=mybucket
CRASHSTORAGE_REGION=us-west-2
CRASHSTORAGE_ACCESS_KEY=somethingsomething
CRASHSTORAGE_SECRET_ACCESS_KEY=somethingsomething
Options:
  • CRASHSTORAGE_ACCESS_KEY (str) –

    AWS S3 access key. You can also specify AWS_ACCESS_KEY_ID which is the env var used by boto3.

    Defaults to ''.

  • CRASHSTORAGE_SECRET_ACCESS_KEY (str) –

    AWS S3 secret access key. You can also specify AWS_SECRET_ACCESS_KEY which is the env var used by boto3.

    Defaults to ''.

  • CRASHSTORAGE_REGION (str) –

    AWS S3 region to connect to. For example, us-west-2

    Defaults to 'us-west-2'.

  • CRASHSTORAGE_ENDPOINT_URL (str) –

    endpoint_url to connect to; None if you are connecting to AWS. For example, http://localhost:4569/.

    Defaults to ''.

  • CRASHSTORAGE_BUCKET_NAME (str) – AWS S3 bucket to save to. Note that the bucket must already have been created and must be in the region specified by region.
component antenna.ext.s3.crashstorage.S3CrashStorage

Saves raw crash files to S3.

This will save raw crash files to S3 in a pseudo-tree something like this:

<BUCKET>
   v1/
       dump_names/
           <CRASHID>
       <DUMPNAME>/
           <CRASHID>
   v2/
       raw_crash/
           <ENTROPY>/
               <YYYYMMDD>/
                   <CRASHID>

When set as the BreakpadSubmitterResource crashstorage class, configuration for this class is in the CRASHSTORAGE namespace.

Generally, if the default connection class is fine, you don’t need to do any configuration here.

Options:CRASHSTORAGE_CONNECTION_CLASS (everett.manager.parse_class) –

S3 connection class to use

Defaults to 'antenna.ext.s3.connection.S3Connection'.