ACHIEVING DEVSECOPS BY INSTRUMENTING MICROSERVICES ORCHESTRATION IN VAMP

With DevOps, we have a higher return on our investment in code, by making it possible to release new features to production in realtime. We do this by automating our tests, which is something that is hard to do for security.

Now, you have to choose: lower returns on investment in features and losing customers by delaying deployments, or risking data breaches in functionality that went live untested.

With realtime instrumentation, we propose a solution that isolates attackers from actual customers transparently, moving the attack traffic to the audited version of an application while actual customers have access to the latest features.

In this blogpost, we will use VAMP, BitSensor and Elastalert to achieve DevSecOps

A screencast about this post is available: YouTube - Microservices meets DevSecOps with VAMP

To deploy the application that we use in our demo, we’re using Vamp.

VAMP is an open source and self-hosted platform for managing (micro)service oriented architectures that rely on container technology. Vamp takes care of complex, multi-step actions like canary releases, route updates, metrics collection and service discovery.

Vamp is functionality agnostic, but functions well in an API centric, event driven and stateless environment. Vamp is not a strict container platform, but uses the power of container platforms under the hood.

Vamp is written in Scala, Go and ReactJS and comes with a server installation, a command line interface, a graphical UI and a RESTful API.

Vamp Dashboard

There is a Docker quick start and DC/OS installation available. I have not been able to use the Docker quick start, so I had to setup a small DC/OS cluster using Vagrant.

This is a tedious process, as you currently have to be picky about the version numbers. I have also experienced issues with the DC/OS a2 instance default memory size, so I have changed VagrantConfig.yaml to give that instance an additional 2048 Mb RAM. Please make sure the machine you are using for deployment has about 10 Gb available memory with this configuration.

DC/OS will now be available on port 80 on m1.dcos.

You can now continue installing VAMP. VAMP should now be available on a2.dcos:9090.

Lets add a new blueprint, that contains both hackable-audited as well as hackable-devops. If you are not running BitSensor, you can replace hackable-devops:1.1.0 by rubenvanvreeland/dvwa:latest. This container includes hackable, ElasticSearch and Kibana.

Port our Hackable application will be available on

    ---
    name: hackable
    gateways:
      9051: hackable/port

We want to send 100% of our traffic to the latest version of our application, in this case hackable:1.1.0
    clusters:
        hackable:

            routing:
                routes:
                    hackable:1.0.0:
                        weight: 0%  
                    hackable:1.1.0:

                        weight: 100%
        services:
            -
                breed: # an audited version of our service
                    name: hackable:1.0.0 

                    deployable: rubenvanvreeland/hackable-audited:latest
                    ports:
                        port: 80/http
                scale:
                    cpu: 0.2

                    memory: 512MB
                    instances: 1
            -
                breed: # a new version of our service
                    name: hackable:1.1.0 

                    deployable: rubenvanvreeland/hackable-devops:latest
                    ports:
                        port: 80/http
                scale:
                    cpu: 0.2

                    memory: 512MB
                    instances: 1

Deploy your blueprint, and move to the Vamp deployments tab. Edit your deployment, and make sure to set filter_strength to 100% on your routes in routing. This way we ensure our filters are always evaluated.

Please, go ahead and checkout hackable on a2.dcos:9051! You might need to setup the database, this just takes a single click.

Feel free to start hacking around! When you are done, lets setup alerting. Clone elastalert and install it on your machine. You also need Jsawk:

    curl -L http://github.com/micha/jsawk/raw/master/jsawk > jsawk
    chmod 755 jsawk && mv jsawk ~/bin/

You might use the following rule in _rules/yourrule.yaml

Change this to the ES host and index that you are using
    es_host: a2.dcos
    es_port: 9200

    # Index to search, wildcard supported
    index: bitsensor
    timestamp_field: endpoint.localtime

Rule name

    name: Every Detection

Rule type, any for an alert every time a document matches. Realert 0 to get an alert every single time.

    # (Required)
    # Type of alert.
    # the frequency rule type alerts when num_events events occur with timeframe time
    type: any
    realert: 
    minutes: 0

Filter documents to match "union", indicating a SQL injection attack

    # (Required)
    # A list of elasticsearch filters used for find events
    # These filters are joined with AND and nested in a filtered query
    # For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
    filter: 
        - query
            query_string:
                query: "union"

In the alert, include basic information such as IP address, user-agent or tool used, and session information.

    include:
        - endpoint.location
        - endpoint.name
        - context.http.userAgent
        - context.ip
        - context.php.session.sessionId
        - detections.type
        - detections.name   


This part describes the content of the alert that you will see in Slack

    alert_subject: "Attack on {}"
    alert_subject_args:
        - endpoint.name

    alert_text_type: alert_text_only
    alert_text: "Detection triggered at {}\nIP: {} \nUA: {}"

    alert_text_args:
        - endpoint.location
        - context.ip
        - context.http.userAgent

Important: change deployment UUID! Twice! Slack also needs an API key for alerts.

Reminder: change deployment UUID in the curl call as well!

    # (Required)

    # The alert is use when a match is found
    alert:
      - slack
      - command
    slack_webhook_url: https://hooks.slack.com/services/xxxx

    command: > 
        echo " {
            \"vamp\":
                `curl -s http://localhost:9090/api/v1/deployments/9f6fde51-33cd-474a-b855-b628cbde22d8`,
            \"alert\":

                $(cat -)
    }" \
        | ~/bin/jsawk '
            var session = this.alert[0].context.php.session.sessionId; 
            var routes = this.vamp.clusters.hackable.routing.port.routes["hackable:1.0.0"];

            var session_filter = {}; 
            session_filter.condition = "cookie PHPSESSID has " + session; 
            routes.filters.push(session_filter); 
            out(this.vamp);' -n \
        | curl \

            -X PUT \
            -H "Content-Type: application/json" -d @- \
            http://localhost:9090/api/v1/deployments/9f6fde51-33cd-474a-b855-b628cbde22d8
    pipe_match_json: true

Run elastalert by executing elastalert. Make sure you are in the elastalert directory.

Now, lets do a SQL injection attack in your browser by inserting the following query in the login form under the Bruteforce tab:

    union all select 1,2 from INFORMATION_SCHEMA.tables

Slack will notify you about an attack

Vamp will now update it’s filters, redirecting the session that is used by the attacker to hackable-audited.

If you now refresh hackable in your browser, you will see “Level: impossible”, indicating that the filters actually kicked in.

We have now contained the attack, by isolating the attacker to a safe version of your deployment.

Feel free to watch and learn from the attacker as he executes his attacks at your sandboxed/honeypotted application. Might he accidentally find a vulnerability there, you can use the trace BitSensor gives to pinpoint this attack to the line of code. You can deploy this change quickly using Dev(Sec)Ops, and complete the circle.