Configuration

Introduction

Antenna uses environment configuration to define its behavior.

The local development environment is configured in the my.env and docker/config/local_dev.env env files and that configuration is pulled in when you run Antenna using docker compose.

In a server environment, configuration is pulled in from the process environment.

Here’s an example. This uses Datadog installed on the EC2 node for metrics and also IAM bound to the EC2 node that Antenna is running on so it doesn’t need S3 credentials for crashstorage.

# Metrics things
STATSD_NAMESPACE=mcboatface

# BreakdpadSubmitterResource settings
CRASHMOVER_CRASHSTORAGE_CLASS=antenna.ext.s3.crashstorage.S3CrashStorage

# S3CrashStorage and S3Connection settings
CRASHMOVER_CRASHSTORAGE_BUCKET_NAME=org-myorg-mybucket

Gunicorn configuration

For Gunicorn configuration, see Dockerfile. You’ll want to set the following:

GUNICORN_WORKERS
Parser:

str

Default:

“1”

Required:

No

The number of Antenna processes to spin off. We use 2x+1 where x is the number of processors on the machine we’re using.

This is the workers Gunicorn configuration setting.

https://docs.gunicorn.org/en/stable/settings.html#workers

This is used in bin/run_web.sh.

GUNICORN_WORKER_CLASS
Parser:

str

Default:

“sync”

Required:

No

This is the worker-class Gunicorn configuration setting.

https://docs.gunicorn.org/en/stable/settings.html#worker-class

This is used in bin/run_web.sh.

GUNICORN_MAX_REQUESTS
Parser:

str

Default:

“0”

Required:

No

If set to 0, this does nothing.

For a value greater than 0, the maximum number of requests for the worker to serve before Gunicorn restarts the worker.

This is the ma-requests Gunicorn configuration setting.

https://docs.gunicorn.org/en/stable/settings.html#max-requests

This is used in bin/run_web.sh.

GUNICORN_MAX_REQUESTS_JITTER
Parser:

str

Default:

“0”

Required:

No

Maximum jitter to add to GUNICORN_MAX_REQUESTS setting.

This is the ma-requests-jitter Gunicorn configuration setting.

https://docs.gunicorn.org/en/stable/settings.html#max-requests-jitter

This is used in bin/run_web.sh.

CMD_PREFIX
Parser:

str

Default:

“”

Required:

No

Specifies a command prefix to run the Gunicorn process in.

This is used in bin/run_web.sh.

Application

First, you need to configure the application-scoped variables.

Configuration

These are defaults appropriate for a server environment, so you may not have to configure any of this.

Configuration summary:

Setting

Parser

Required?

BASEDIR

str

LOGGING_LEVEL

str

LOCAL_DEV_ENV

bool

STATSD_HOST

str

STATSD_PORT

int

SECRET_SENTRY_DSN

str

HOSTNAME

str

Configuration options:

BASEDIR
Parser:

str

Default:

“/home/docs/checkouts/readthedocs.org/user_builds/antenna/checkouts/latest”

Required:

No

The root directory for this application to find and store things.

LOGGING_LEVEL
Parser:

str

Default:

“INFO”

Required:

No

The logging level to use. DEBUG, INFO, WARNING, ERROR or CRITICAL

LOCAL_DEV_ENV
Parser:

bool

Default:

“False”

Required:

No

Whether or not this is a local development environment.

STATSD_HOST
Parser:

str

Default:

“localhost”

Required:

No

Hostname for statsd server.

STATSD_PORT
Parser:

int

Default:

“8125”

Required:

No

Port for statsd server.

SECRET_SENTRY_DSN
Parser:

str

Default:

“”

Required:

No

Sentry DSN to use. See https://docs.sentry.io/quickstart/#configure-the-dsn for details. If this is not set an unhandled exception logging middleware will be used instead.

HOSTNAME
Parser:

str

Default:

“build-24580004-project-52135-antenna”

Required:

No

Identifier for the host that is running Antenna. This identifies this Antenna instance in the logs and makes it easier to correlate Antenna logs with other data. For example, the value could be a an instance id, pod id, or something like that.

If you do not set this, then socket.gethostname() is used instead.

Breakpad crash resource

component antenna.breakpad_resource.BreakpadSubmitterResource

Handles incoming breakpad-style crash reports.

This handles incoming HTTP POST requests containing breakpad-style crash reports in multipart/form-data format.

It can handle compressed or uncompressed POST payloads.

It parses the payload from the HTTP POST request, runs it through the throttler with the specified rules, generates a crash_id, returns the crash_id to the HTTP client, and passes the crash report data to the crashmover.

Configuration summary:

Setting

Parser

Required?

BREAKPAD_DUMP_FIELD

str

Configuration options:

BREAKPAD_DUMP_FIELD
Parser:

str

Default:

“upload_file_minidump”

Required:

No

The name of the field in the POST data for dumps.

Throttler

component antenna.throttler.Throttler

Accept or reject incoming crashes based on specified rule set.

The throttler can throttle incoming crashes using the content of the crash. To throttle, you set up a rule set which is a list of Rule instances. That goes in a Python module which is loaded at run time.

If you don’t want to throttle anything, use this:

BREAKPAD_THROTTLER_RULES=antenna.throttler.ACCEPT_ALL

If you want to support all products, use this:

BREAKPAD_THROTTLER_PRODUCTS=antenna.throttler.ALL_PRODUCTS

To set up a rule set, put it in a Python file and define the rule set there. For example, you could have file myruleset.py with this in it:

from antenna.throttler import Rule

rules = [
    Rule('ProductName', 'Firefox', 100),
    # ...
]

then set BREAKPAD_THROTTLER_RULES to the path for that. For example, depending on the current working directory and PYTHONPATH, the above could be:

BREAKPAD_THROTTLER_RULES=myruleset.rules

Configuration summary:

Setting

Parser

Required?

BREAKPAD_THROTTLER_RULES

antenna.throttler.parse_attribute

BREAKPAD_THROTTLER_PRODUCTS

antenna.throttler.parse_attribute

BREAKPAD_THROTTLER_PRODUCT_PACKAGENAMES

antenna.throttler.parse_attribute

Configuration options:

BREAKPAD_THROTTLER_RULES
Parser:

antenna.throttler.parse_attribute

Default:

“antenna.throttler.MOZILLA_RULES”

Required:

No

Python dotted path to ruleset

BREAKPAD_THROTTLER_PRODUCTS
Parser:

antenna.throttler.parse_attribute

Default:

“antenna.throttler.MOZILLA_PRODUCTS”

Required:

No

Python dotted path to list of supported products

BREAKPAD_THROTTLER_PRODUCT_PACKAGENAMES
Parser:

antenna.throttler.parse_attribute

Default:

“antenna.throttler.MOZILLA_PRODUCT_PACKAGENAMES”

Required:

No

Python dotted path to map of product -> list of supported packagenames

Crash mover

component antenna.crashmover.CrashMover

Handles saving and publishing crash reports.

The crashmover saves the crash using the configured crashstorage class and publishes it using the configured crashpublish class.

Note

From when a crash comes in to when it’s saved by the crashstorage class, the crash is entirely in memory. Keep that in mind when figuring out how to scale your Antenna nodes.

The most important configuration bit here is choosing the crashstorage class.

For example:

CRASHMOVER_CRASHSTORAGE_CLASS=antenna.ext.s3.crashstorage.S3CrashStorage

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHSTORAGE_CLASS

everett.manager.parse_class

CRASHMOVER_CRASHPUBLISH_CLASS

everett.manager.parse_class

CRASHMOVER_MAX_ATTEMPTS

int

CRASHMOVER_RETRY_SLEEP_SECONDS

int

Configuration options:

CRASHMOVER_CRASHSTORAGE_CLASS
Parser:

everett.manager.parse_class

Default:

“antenna.ext.crashstorage_base.NoOpCrashStorage”

Required:

No

The class in charge of storing crashes.

CRASHMOVER_CRASHPUBLISH_CLASS
Parser:

everett.manager.parse_class

Default:

“antenna.ext.crashpublish_base.NoOpCrashPublish”

Required:

No

The class in charge of publishing crashes.

CRASHMOVER_MAX_ATTEMPTS
Parser:

int

Default:

“5”

Required:

No

Maximum number of attempts to save or publish a crash before giving up.

CRASHMOVER_RETRY_SLEEP_SECONDS
Parser:

int

Default:

“2”

Required:

No

Seconds to sleep between attempts to save or publish a crash.

Crash storage

For crash storage, you have three options one of which is a no-op for debugging.

NoOpCrashStorage

The NoOpCrashStorage class is helpful for debugging, but otherwise shouldn’t be used.

component antenna.ext.crashstorage_base.NoOpCrashStorage

This is a no-op crash storage that logs crashes it would have stored.

It keeps track of the last 10 crashes in .saved_things instance attribute with the most recently stored crash at the end of the list. This helps when writing unit tests for Antenna.

Configuration summary:

Setting

Parser

Required?

Configuration options:

Filesystem

The FSCrashStorage class will save crash data to disk. If you choose this, you’ll want to think about what happens to the crash after Antenna has saved it and implement that.

component antenna.ext.fs.crashstorage.FSCrashStorage

Save raw crash files to the file system.

This generates a tree something like this which mirrors what we do on S3:

<FS_ROOT>/
    <YYYYMMDD>/
        raw_crash/
            <CRASHID>.json
        dump_names/
            <CRASHID>.json
        <DUMP_NAME>/
            <CRASHID>

Couple of things to note:

  1. This doesn’t ever delete anything from the tree. You should run another process to clean things up.

  2. If you run out of disk space, this component will fail miserably. There’s no way to recover from a full disk–you will lose crashes.

FIXME(willkg): Can we alleviate or reduce the likelihood of the above?

When set as the CrashMover crashstorage class, configuration for this class is in the CRASHMOVER_CRASHSTORAGE namespace.

Example:

CRASHMOVER_CRASHSTORAGE_FS_ROOT=/tmp/whatever

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHSTORAGE_FS_ROOT

str

Configuration options:

CRASHMOVER_CRASHSTORAGE_FS_ROOT
Parser:

str

Default:

“/tmp/antenna_crashes”

Required:

No

path to where files should be stored

AWS S3

The S3CrashStorage class will save crash data to AWS S3. You might be able to use this to save to other S3-like systems, but that’s not tested or supported.

component antenna.ext.s3.connection.S3Connection

Connection object for S3.

Credentials and permissions

When configuring credentials for this connection object, you can do one of two things:

  1. provide ACCESS_KEY and SECRET_ACCESS_KEY in the configuration, OR

  2. use one of the other methods described in the boto3 docs https://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials

The AWS credentials that Antenna is configured with must have the following Amazon S3 permissions:

  • s3:ListBucket

    Antenna periodically checks its health and during that health check, it will HEAD the S3 Bucket. This requires s3:ListBucket permissions.

  • s3:PutObject

    This permission is used to save items to the bucket.

    Additionally, at startup, Antenna will attempt to save a test file to the bucket. If that fails, then this will raise an error and will halt startup.

Retrying saves

When saving crashes, this connection will retry saving several times. Then give up.

When set as the CrashMover crashstorage class, configuration for this class is in the CRASHMOVER_CRASHSTORAGE namespace.

Example:

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME=mybucket
CRASHMOVER_CRASHSTORAGE_REGION=us-west-2
CRASHMOVER_CRASHSTORAGE_ACCESS_KEY=somethingsomething
CRASHMOVER_CRASHSTORAGE_SECRET_ACCESS_KEY=somethingsomething

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHSTORAGE_ACCESS_KEY

str

CRASHMOVER_CRASHSTORAGE_SECRET_ACCESS_KEY

str

CRASHMOVER_CRASHSTORAGE_REGION

str

CRASHMOVER_CRASHSTORAGE_ENDPOINT_URL

str

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME

str

Yes

Configuration options:

CRASHMOVER_CRASHSTORAGE_ACCESS_KEY
Parser:

str

Default:

“”

Required:

No

AWS access key. You can also specify AWS_ACCESS_KEY_ID which is the env var used by boto3.

CRASHMOVER_CRASHSTORAGE_SECRET_ACCESS_KEY
Parser:

str

Default:

“”

Required:

No

AWS secret access key. You can also specify AWS_SECRET_ACCESS_KEY which is the env var used by boto3.

CRASHMOVER_CRASHSTORAGE_REGION
Parser:

str

Default:

“us-west-2”

Required:

No

AWS region to connect to. For example, us-west-2

CRASHMOVER_CRASHSTORAGE_ENDPOINT_URL
Parser:

str

Default:

“”

Required:

No

endpoint_url to connect to; None if you are connecting to AWS. For example, http://localhost:4569/.

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME
Parser:

str

Required:

Yes

AWS S3 bucket to save to. Note that the bucket must already have been created and must be in the region specified by region.

component antenna.ext.s3.crashstorage.S3CrashStorage

Save raw crash files to S3.

This will save raw crash files to S3 in a pseudo-tree something like this:

<BUCKET>
   v1/
       dump_names/
           <CRASHID>
       <DUMPNAME>/
           <CRASHID>
       raw_crash/
           <YYYYMMDD>/
               <CRASHID>

When set as the CrashMover crashstorage class, configuration for this class is in the CRASHMOVER_CRASHSTORAGE namespace.

Generally, if the default connection class is fine, you don’t need to do any configuration here.

CRASHMOVER_CRASHSTORAGE_CONNECTION_CLASS
Parser:

everett.manager.parse_class

Default:

“antenna.ext.s3.connection.S3Connection”

Required:

No

S3 connection class to use

Google Cloud Storage

The GcsCrashStorage class will save crash data to Google Cloud Storage.

component antenna.ext.gcs.crashstorage.GcsCrashStorage

Save raw crash files to GCS.

This will save raw crash files to GCS in a pseudo-tree something like this:

<BUCKET>
   v1/
       dump_names/
           <CRASHID>
       <DUMPNAME>/
           <CRASHID>
       raw_crash/
           <YYYYMMDD>/
               <CRASHID>

Authentication

The google cloud sdk will automatically detect credentials as described in https://googleapis.dev/python/google-api-core/latest/auth.html:

  • If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.

  • If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:

    $ gcloud auth application-default login
    
  • If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable:

    $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
    

Local emulator

If you set the environment variable STORAGE_EMULATOR_HOST=http://host:port, then this will connect to a local GCS emulator.

When set as the CrashMover crashstorage class, configuration for this class is in the CRASHMOVER_CRASHSTORAGE namespace.

Example:

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME=mybucket

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME

str

Yes

Configuration options:

CRASHMOVER_CRASHSTORAGE_BUCKET_NAME
Parser:

str

Required:

Yes

Google Cloud Storage bucket to save to. Note that the bucket must already have been created.

Crash publish

For crash publishing, you have two options one of which is a no-op.

NoOpCrashPublish

The NoOpCrashPublish class is helpful for debugging and also if you don’t want Antenna to be publishing crash ids somewhere.

component antenna.ext.crashpublish_base.NoOpCrashPublish

No-op crash publish class that logs crashes it would have published.

It keeps track of the last 10 crash ids in .published_things instance attribute with the most recently published crash id at the end of the list. This helps when writing unit tests for Antenna.

Google Pub/Sub

The PubSubCrashPublish class will publish crash ids to a Google Pub/Sub topic.

component antenna.ext.pubsub.crashpublish.PubSubCrashPublish

Publisher to Pub/Sub.

Required GCP things

To use this, you need to create:

  1. Google Compute project

  2. topic in that project

  3. subscription for that topic so you can consume queued items

If something in the above isn’t created, then Antenna may not start.

Verification

This component verifies that it can publish to the topic by publishing a fake crash id of test. Downstream consumer should throw this out.

Authentication

The google cloud sdk will automatically detect credentials as described in https://googleapis.dev/python/google-api-core/latest/auth.html:

  • If you’re running in a Google Virtual Machine Environment (Compute Engine, App Engine, Cloud Run, Cloud Functions), authentication should “just work”.

  • If you’re developing locally, the easiest way to authenticate is using the Google Cloud SDK:

    $ gcloud auth application-default login
    
  • If you’re running your application elsewhere, you should download a service account JSON keyfile and point to it using an environment variable:

    $ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"
    

Local emulator

If you set the environment variable PUBSUB_EMULATOR_HOST=host:port, then this will connect to a local Pub/Sub emulator.

When set as the BreakpadSubmitterResource crashpublish class, configuration for this class is in the CRASHMOVER_CRASHPUBLISH namespace.

You need to set the project id and topic name.

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHPUBLISH_PROJECT_ID

str

Yes

CRASHMOVER_CRASHPUBLISH_TOPIC_NAME

str

Yes

CRASHMOVER_CRASHPUBLISH_TIMEOUT

float

Configuration options:

CRASHMOVER_CRASHPUBLISH_PROJECT_ID
Parser:

str

Required:

Yes

Google Cloud project id.

CRASHMOVER_CRASHPUBLISH_TOPIC_NAME
Parser:

str

Required:

Yes

The Pub/Sub topic name to publish to.

CRASHMOVER_CRASHPUBLISH_TIMEOUT
Parser:

float

Default:

“5”

Required:

No

The amount of time in seconds individual rpc calls to pubsub are allowed to take before giving up. This is passed to PublisherOptions. From 2018 through 2023 Mozilla’s internal data platform’s use of pubsub generally saw publish response times less than 150ms, and had one incident where response times were approximately 1 second. Based on that experience this has a default of 5 seconds.

AWS SQS

The SQSCrashPublish class will publish crash ids to an AWS SQS queue.

component antenna.ext.sqs.crashpublish.SQSCrashPublish

Publisher to AWS SQS.

Required AWS SQS things

When configuring credentials for this crashpublish object, you can do one of two things:

  1. provide ACCESS_KEY and SECRET_ACCESS_KEY in the configuration, OR

  2. use one of the other methods described in the boto3 docs https://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials

You also need to create an AWS SQS standard queue with the following settings:

Setting

Value

Default Visibility Timeout

5 minutes

Message Retention Period

default

Maximum Message Size

default

Delivery Delay

default

Receive Message Wait Time

default

The AWS credentials that Antenna is configured with must have the following Amazon SQS permissions on the SQS queue you created:

  • sqs:GetQueueUrl

    Antenna needs to convert a queue name to a queue url. This requires the sqs:GetQueueUrl

  • sqs:SendMessage

    Antenna sends messages to a queue–this is how it publishes crash ids. This requires the sqs:SendMessage permission.

If something isn’t configured correctly, then Antenna may not start.

Verification

This component verifies that it can publish to the queue by publishing a fake crash id of test. Downstream consumers should ignore these.

When set as the CrashMover crashpublish class, configuration for this class is in the CRASHMOVER_CRASHPUBLISH namespace.

Configuration summary:

Setting

Parser

Required?

CRASHMOVER_CRASHPUBLISH_ACCESS_KEY

str

CRASHMOVER_CRASHPUBLISH_SECRET_ACCESS_KEY

str

CRASHMOVER_CRASHPUBLISH_REGION

str

CRASHMOVER_CRASHPUBLISH_ENDPOINT_URL

str

CRASHMOVER_CRASHPUBLISH_QUEUE_NAME

str

Yes

Configuration options:

CRASHMOVER_CRASHPUBLISH_ACCESS_KEY
Parser:

str

Default:

“”

Required:

No

AWS SQS access key. You can also specify AWS_ACCESS_KEY_ID which is the env var used by boto3.

CRASHMOVER_CRASHPUBLISH_SECRET_ACCESS_KEY
Parser:

str

Default:

“”

Required:

No

AWS SQS secret access key. You can also specify AWS_SECRET_ACCESS_KEY which is the env var used by boto3.

CRASHMOVER_CRASHPUBLISH_REGION
Parser:

str

Default:

“us-west-2”

Required:

No

AWS region to connect to. For example, us-west-2

CRASHMOVER_CRASHPUBLISH_ENDPOINT_URL
Parser:

str

Default:

“”

Required:

No

endpoint_url to connect to; None if you are connecting to AWS. For example, http://localhost:4569/.

CRASHMOVER_CRASHPUBLISH_QUEUE_NAME
Parser:

str

Required:

Yes

The AWS SQS queue name.