faustian

Faustian

Faustian is an Apache Mesos framework for launching and managing Faust workers. This service was envisioned to provide a tighter integration between Mesos, Faust, monitoring and our cloud infrastructure.

Considerations

The functionality currently provided by this framework may be better achieved by using Marathon + Pods and moving the autoscaling & monitoring portions to a separate service. However, we continued developing this framework as Marathon does not yet support multiple Mesos roles, which mean we would have to run another marathon cluster as we wanted to keep our data processing and our general purpose Mesos resources separate.

Configuration

Faustian can be configure from the command line or using environment variables.

Pipeline Definition

Faustian treats your consumer group of worker processes as a pipeline. Each worker in a pipeline is expected to be a docker image with an ENTRYPOINT directly into the faust worker. This means your dockerfile must have an ENTRYPOINT of either ENTRYPOINT ["faust", "-A", "application"] or if your python application calls app.main() ENTRYPOINT ["python", "app.py"]

{
  "id": "string", // must be unique per pipeline
  "container": {
    "image": "string", // Docker image if your faust worker
    "network": "HOST" // Networking type. Either HOST or BRIDGE. If using BRIDGE, Faustian uses the `mesos-bridge` network
  },
  "environment": {
  }, // environment variables applied to all processes
  "processes": { // Faustian uses TASK_GROUPS (aka pods in marathon) to run a group of workers at the same time. This is useful if you want to run multiple agents in different processes.
    "parse": {
      "id": "parse", // id for this worker, must match the key name
      "arguments": ["worker", "-l", "info"], // command line arguments for this worker
      "environment": {
      }, // environment variables for just this work
      "resources": {
        "cpu": 0.29,
        "mem": 512
      }, // resources for this task
      "port_mappings": [{
        "host_port": 0,
        "container_port": 6066,
        "protocol": "tcp"
      }], // Port mappings, like in Marathon. This only applies when the network mode is BRIDGE.
      "kill_grace_period": 30 // kill grace period, this is how long we will wait for the application to quit before sending a KILL
    }
  },
  "executor_resources": {
    "cpu": 0.01,
    "mem": 32,
    "disk": 1
  }, // executor resourcess
  "instances": 5 // number of worker instances
}

HTTP API

POST /pipelines

Create a new pipeline. Returns a 204 on success.

PATCH /pipelines/{pipeline-id}

Update a pipeline. If only the number of instances are changed, then new workers will be added/removed. If anything else changes all the old workers are killed and relaunched.

License

MIT