Monasca installation and configuration guide

In this tutorial, i will describe how to setup monasca components in detail. Before we start, something needs to be confirmed:

  • All the components of monasca can be installed on one node, such as on openstack controller node, or you can deploy it in multi-nodes. In this tutorial, i will install monasca-api in a new VM created in my openstack cluster, which has a floating ip associated. Monasca-agent was installed on controller node. The agent node post metrics to api node through floating ip. the are in the same sub net.
  • All the user name and password in this tutorial is monasca and qydcos. change it to yours.
  • the installation will be performed on ubuntu 14.04 openstack Mitaka version, for liberty, special settings must be made and be described later.
  • All the file related in this tutorial are here, clone it before you start.

    1, install packages and tools we needed.

    apt-get install -y git
    apt-get install openjdk-7-jre-headless python-pip python-dev
    

    2, install mysql database
    if you install monasca-api in openstack controller node, you can skip installing it, use the msyql already installed for openstack services.

    apt-get install -y mysql-server
    

    create monasca database schema, download mon_mysql here, the schema file in github has a bug, and it can not create notification, i have fixed it here. remember to modify the user name and password in line 234,235 of mon_mysql.sql to yours.

    mysql -uroot -ppassword < mon_mysql.sql
    

    3, install zookeeper
    install zookeeper and restart it. i use localhost interface and only one zookeeper, so the default configuration file needs nothing to be configured.

    apt-get install -y zookeeper zookeeperd zookeeper-bin
    service zookeeper restart
    

    4, install and configure kafka

    wget http://apache.mirrors.tds.net/kafka/0.8.1.1/kafka_2.9.2-0.8.1.1.tgz
    mv kafka_2.9.2-0.8.1.1.tgz /opt
    cd /opt
    tar zxf kafka_2.9.2-0.8.1.1.tgz
    ln -s /opt/kafka_2.9.2-0.8.1.1/ /opt/kafka
    ln -s /opt/kafka/config /etc/kafka
    

    create kafka system user, kafka service will started as this user

    useradd kafka -U -r
    

    create kafka startup scripts in /etc/init/kafka.conf, copy following contents into /etc/init/kafka.conf and save it.

    description "Kafka"
    
    start on runlevel [2345]
    stop on runlevel [!2345]
    
    respawn
    
    limit nofile 32768 32768
    
    # If zookeeper is running on this box also give it time to start up properly
    pre-start script
        if [ -e /etc/init.d/zookeeper ]; then
            /etc/init.d/zookeeper restart
        fi
    end script
    
    # Rather than using setuid/setgid sudo is used because the pre-start task must run as root
    exec sudo -Hu kafka -g kafka KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" JMX_PORT=9997 /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties
    

    configure kafka, vim /etc/kafka/server.properties, make sure the following contents is configured

     host.name=localhost
     advertised.host.name=localhost
     log.dirs=/var/kafka
    

    create kafka log dirs.

    mkdir /var/kafka
    mkdir /var/log/kafka
    chown -R kafka. /var/kafka/
    chown -R kafka. /var/log/kafka/
    

    start kafka service

    service kafka start
    

    the next step is to create kafka topics.

    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 64 --topic metrics
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic events
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic raw-events
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic transformed-events
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic stream-definitions
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic transform-definitions
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic alarm-state-transitions
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic alarm-notifications
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 12 --topic stream-notifications
    /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic retry-notifications
    

    5, install and configure influxdb

    curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
    echo "deb https://repos.influxdata.com/ubuntu trusty stable" > /etc/apt/sources.list.d/influxdb.list
    apt-get update
    apt-get install -y apt-transport-https
    apt-get install -y influxdb
    
    service influxdb start
    

    create influxdb database, user, password, retention policy, change password to yours.

    influx
    CREATE DATABASE mon
    CREATE USER monasca WITH PASSWORD 'qydcos'
    CREATE RETENTION POLICY persister_all ON mon DURATION 90d REPLICATION 1 DEFAULT
    exit
    

    6, install and configure storm

    wget http://apache.mirrors.tds.net/storm/apache-storm-0.9.6/apache-storm-0.9.6.tar.gz
    mkdir /opt/storm
    cp apache-storm-0.9.6.tar.gz /opt/storm/
    cd /opt/storm/
    tar xzf apache-storm-0.9.6.tar.gz
    ln -s /opt/storm/apache-storm-0.9.6 /opt/storm/current
    
    useradd storm -U -r
    mkdir /var/storm
    mkdir /var/log/storm
    chown -R storm. /var/storm/
    chown -R storm. /var/log/storm/
    

    modify storm.yaml as follow, vim current/storm/conf/storm.yaml

    ### base
    java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"
    storm.local.dir: "/var/storm"
    
    ### zookeeper.*
    storm.zookeeper.servers:
        - "localhost"
    storm.zookeeper.port:  2181
    storm.zookeeper.retry.interval: 5000
    storm.zookeeper.retry.times: 29
    storm.zookeeper.root: "/storm"
    storm.zookeeper.session.timeout: 30000
    
    ### supervisor.* configs are for node supervisors
    supervisor.slots.ports:
        - 6701
        - 6702
        - 6703
        - 6704
    supervisor.childopts: "-Xmx1024m"
    
    ### worker.* configs are for task workers
    worker.childopts: "-Xmx1280m -XX:+UseConcMarkSweepGC -Dcom.sun.management.jmxremote"
    
    ### nimbus.* configs are for the masteri
    nimbus.host: "localhost"
    nimbus.thrift.port: 6627
    mbus.childopts: "-Xmx1024m"
    
    ### ui.* configs are for the master
    ui.host: 127.0.0.1
    ui.port: 8078
    ui.childopts: "-Xmx768m"
    
    ### drpc.* configs
    
    ### transactional.* configs
    transactional.zookeeper.servers:
        - "localhost"
    transactional.zookeeper.port: 2181
    transactional.zookeeper.root: "/storm-transactional"
    
    ### topology.* configs are for specific executing storms
    topology.acker.executors: 1
    topology.debug: false
    
    logviewer.port: 8077
    logviewer.childopts: "-Xmx128m"
    

    create storm supervisor startup scripts, vim /etc/init/storm-supervisor.conf

    # Startup script for Storm Supervisor
    
    description "Storm Supervisor daemon"
    start on runlevel [2345]
    
    console log
    respawn
    
    kill timeout 240
    respawn limit 25 5
    
    setgid storm
    setuid storm
    chdir /opt/storm/current
    exec /opt/storm/current/bin/storm supervisor
    

    create storm nimbus scripts.vim /etc/init/storm-nimbus.conf

    # Startup script for Storm Nimbus
    
    description "Storm Nimbus daemon"
    start on runlevel [2345]
    
    console log
    respawn
    
    kill timeout 240
    respawn limit 25 5
    
    setgid storm
    setuid storm
    chdir /opt/storm/current
    exec /opt/storm/current/bin/storm nimbus
    

    start storm supervisor and nimbus

    service storm-supervisor start
    service storm-nimbus start
    

    7, install monasca api python packages
    some monasca components have both python and java code available, mainly i choose python code to deploy.

    pip install monasca-common
    pip install gunicorn
    pip install greenlet  # Required for both
    pip install eventlet  # For eventlet workers
    pip install gevent    # For gevent workers
    pip install monasca-api
    pip install influxdb
    

    vim /etc/monasca/api-config.ini , modify host to your ip address

    [DEFAULT]
    name = monasca_api
    
    [pipeline:main]
    # Add validator in the pipeline so the metrics messages can be validated.
    pipeline = auth keystonecontext api
    
    [app:api]
    paste.app_factory = monasca_api.api.server:launch
    
    [filter:auth]
    paste.filter_factory = keystonemiddleware.auth_token:filter_factory
    
    [filter:keystonecontext]
    paste.filter_factory = monasca_api.middleware.keystone_context_filter:filter_factory
    
    [server:main]
    use = egg:gunicorn#main
    host = 192.168.2.23
    port = 8082
    workers = 1
    proc_name = monasca_api
    

    vim /etc/monasca/api-config.conf, modify the following contents

    [DEFAULT]
    # logging, make sure that the user under whom the server runs has permission
    # to write to the directory.
    log_file = monasca-api.log
    log_dir = /var/log/monasca/api/
    debug=False
    region = RegionOne
    [security]
    # The roles that are allowed full access to the API.
    default_authorized_roles = admin, user, domainuser, domainadmin, monasca-user
    
    # The roles that are allowed to only POST metrics to the API. This role would be used by the Monasca Agent.
    agent_authorized_roles = admin
    
    # The roles that are allowed to only GET metrics from the API.
    read_only_authorized_roles = admin
    
    # The roles that are allowed to access the API on behalf of another tenant.
    # For example, a service can POST metrics to another tenant if they are a member of the "delegate" role.
    delegate_authorized_roles = admin
    
    [kafka]
    # The endpoint to the kafka server
    uri = localhost:9092
    
    [influxdb]
    # Only needed if Influxdb database is used for backend.
    # The IP address of the InfluxDB service.
    ip_address = localhost
    
    # The port number that the InfluxDB service is listening on.
    port = 8086
    
    # The username to authenticate with.
    user = monasca
    
    # The password to authenticate with.
    password = qydcos
    
    # The name of the InfluxDB database to use.
    database_name = mon
    
    [database]
    url = "mysql+pymysql://monasca:[email protected]/mon"
    
    
    [keystone_authtoken]
    identity_uri = http://192.168.1.11:35357
    auth_uri = http://192.168.1.11:5000
    admin_password = qydcos
    admin_user = monasca
    admin_tenant_name = service
    cafile =
    certfile =
    keyfile =
    insecure = false
    

    comment out [mysql] section, others keeps on default.
    create monasca system user and log dirs

    useradd monasca -U -r
    mkdir /var/log/monasca
    mkdir /var/log/monasca/api
    chown -R monasca. /var/log/monasca/
    

    on openstack controller node, create monasca user password, assign admin role for user monasca in tenant service.

    openstack user create --domain default --password qydcos monasca 
    openstack role add --project service --user monasca admin
    
    openstack service create --name monasca --description "Monasca monitoring service" monitoring
    
    create endpoint 
    openstack endpoint create --region RegionOne monasca public http://192.168.1.143:8082/v2.0
    openstack endpoint create --region RegionOne monasca internal http://192.168.1.143:8082/v2.0
    openstack endpoint create --region RegionOne monasca admin http://192.168.1.143:8082/v2.0
    

    192.168.1.143 is the floating ip of my api vm address, change it to yours.
    create monasca api startup scripts,vim /etc/init/monasca-api.conf

    # Startup script for the Monasca API
    
    description "Monasca API Python app"
    start on runlevel [2345]
    
    console log
    respawn
    
    setgid monasca
    setuid monasca
    exec /usr/local/bin/gunicorn -n monasca-api -k eventlet --worker-connections=2000 --backlog=1000 --paste /etc/monasca/api-config.ini
    

    start monasca-api service

    service monasca-api start
    

    if you get mysql connection error, modify monasca-common python file, and restart monasca-api service, the python code has bug reading mysql configuration. here is a quick hack,
    vim /usr/local/lib/python2.7/dist-packages/monasca_common/repositories/mysql/mysql_repository.py

                self.conf = cfg.CONF
                #self.database_name = self.conf.mysql.database_name
                #self.database_server = self.conf.mysql.hostname
                #self.database_uid = self.conf.mysql.username
                #self.database_pwd = self.conf.mysql.password
    
                self.database_name = 'mon'
                self.database_server = 'localhost'
                self.database_uid = 'monasca'
                self.database_pwd = 'qydcos'
    
    

    8, install monasca-persister
    the monasca-persister java code has a bug writting data into influxdb, i fixed it and rebuild a jar file and upload it into monasca.git. the monasca-persister python code also has a bug writting data into influxdb, i have no time to fix it.

    copy monasca-persister.jar file into /opt/monasca/
    copy persister-config.yml into /etc/monasca/

    create monasca-persister startup script
    vim /etc/init/monasca-persister.conf

    # Startup script for the Monasca Persister
    
    description "Monasca Persister Python app"
    start on runlevel [2345]
    
    console log
    respawn
    
    setgid monasca
    setuid monasca
    exec /usr/bin/java -Dfile.encoding=UTF-8 -cp /opt/monasca/monasca-persister.jar monasca.persister.PersisterApplication server /etc/monasca/persister-config.yml
    

    start monasca-persister

    service monasca-persister start
    

    9, install monasca-notificatoin

    pip install --upgrade monasca-notification
    apt-get install sendmail
    

    copy notification.yaml into /etc/monasca/
    create startup script, vim /etc/init/monasca-notification.conf

    # Startup script for the monasca_notification
    
    description "Monasca Notification daemon"
    start on runlevel [2345]
    
    console log
    respawn
    
    setgid monasca
    setuid monasca
    exec /usr/bin/python /usr/local/bin/monasca-notification
    

    start notification service

    service monasca-notification start
    

    10, install monasca-thresh
    copy monasca-thresh into /etc/init.d/
    copy monasca-thresh.jar into /opt/monasca-thresh/
    copy thresh-config.yml into /etc/monasca/ and modify host, database to yours.
    start monasca-thresh

    service monasca-thresh start
    

    11, install monasca-agent
    install monasca-agent on openstack controller node, so that it can monitor openstack service process.

    sudo pip install --upgrade monasca-agent
    

    setup monasca-agent, if you are on liberty, change user domain id and project domain id to default, for mitaka, use default domain id,

    monasca-setup -u monasca -p qydcos --user_domain_id e25e0413a70c41449d2ccc2578deb1e4 --project_domain_id e25e0413a70c41449d2ccc2578deb1e4 --user monasca \
     --project_name service -s monitoring --keystone_url http://192.168.1.11:35357/v3 --monasca_url http://192.168.1.143:8082/v2.0 --config_dir /etc/monasca/agent --log_dir /var/log/monasca/agent --overwrite
    

    source admin-rc.sh, run monasca metric-list

  • 128 comments

    1. Hello Shaun,

      Any idea how to resolve this issue?

      2017-03-06 01:58:19,624.624 4586 INFO keystonemiddleware.auth_token [-] Starting Keystone auth_token middleware
      2017-03-06 01:58:19,627.627 4586 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
      2017-03-06 01:58:19,628.628 4586 WARNING keystonemiddleware.auth_token [-] Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.

    2. Hello Shaun,

      To enable monasca ui on openstack dashboard:
      1,git clone https://git.openstack.org/openstack/monasca-ui.git # clone monasca-ui
      2,git clone https://github.com/hpcloud-mon/grafana.git # clone grafana
      3,cp monasca-ui/monitoring/enabled/_50_admin_add_monitoring_panel.py /usr/share/openstack-dashboard/openstack_dashboard/local/enabled/
      4, ln -s monasca-ui/monitoring /usr/share/openstack-dashboard/monitoring
      5, ln -s grafana /var/www/html/grafana
      6,vim /etc/apache2/sites-enabled/000-default.conf
      add
      Alias /grafana /var/www/html/grafana
      in section, usually under CustomLog line
      7, restart apache
      then you can see monasca tab on dashboard.

      Just need to clone the git , no need to install it, right.

    3. Hello , Any idea how to resolve this issue on horizon..

      Error: Unable to retrieve notifications: Failed to connect to http://10.48.151.148:8082/v2.0, error was HTTPConnectionPool(host=’10.48.151.148′, port=8082): Max retries exceeded with url: /v2.0/notification-methods?limit=10000&offset=0 (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’,))

    4. Could you help to guide to resolve this issue..

      monasca metric-list , shows nothing

      _send_broker_aware_request /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/client.py:207
      2017-03-17 04:34:44,471.471 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reinitializing socket connection for localhost:9092 reinit /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:214
      2017-03-17 04:34:44,476.476 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to send 36373 bytes to Kafka, request 50 send /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:135
      2017-03-17 04:34:44,480.480 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reading response 50 from Kafka recv /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:157
      2017-03-17 04:34:44,481.481 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-17 04:34:44,482.482 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 4/4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-17 04:34:44,483.483 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-17 04:34:44,484.484 13489 DEBUG monasca_common.kafka_lib.conn [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 35/35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-17 04:34:44,486.486 13489 DEBUG monasca_common.kafka_lib.client [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Response 50: [ProduceResponse(topic=’metrics’, partition=32, error=0, offset=2836)] _send_broker_aware_request /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/client.py:265
      2017-03-17 04:34:44,489.489 13489 DEBUG gunicorn.error [req-9ee42f2f-cb7a-4156-8c73-ded603a9d42e 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Closing connection. debug /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:250

    5. 2017-03-17 04:36:14,458.458 13489 DEBUG keystonemiddleware.auth_token [-] Authenticating user token process_request /usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py:401
      2017-03-17 04:36:14,461.461 13489 DEBUG keystonemiddleware.auth_token [-] Received request from user: user_id 5b462a4db4324487af33b6f4647dd1f3, project_id 6062fb6b11a648cf8e9616fcdedc479f, roles admin process_request /usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py:672
      2017-03-17 04:36:14,466.466 13489 DEBUG monasca_common.kafka_lib.client [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Request 53 to BrokerMetadata(nodeId=0, host=’localhost’, port=9092): [ProduceRequest(topic=’metrics’, partition=43, messages=[Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540846.991, “value_meta”: null, “name”: “jvm.memory.total.max”, “value”: 1145569280, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.125, “value_meta”: null, “name”: “alarm-state-transitions-added-to-batch-counter[0]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.6108, “value_meta”: null, “name”: “metrics-added-to-batch-counter[3]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.2231, “value_meta”: null, “name”: “alarm-state-transitions-added-to-batch-counter[1]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.487, “value_meta”: null, “name”: “metrics-added-to-batch-counter[1]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.549, “value_meta”: null, “name”: “metrics-added-to-batch-counter[2]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.302, “value_meta”: null, “name”: “metrics-added-to-batch-counter[0]”, “value”: 0.0, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540847.061, “value_meta”: null, “name”: “jvm.memory.total.used”, “value”: 59874632, “dimensions”: {“url”: “http://localhost:8091/metrics”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.587, “value_meta”: null, “name”: “net.in_errors_sec”, “value”: 0.0, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.484, “value_meta”: null, “name”: “net.in_packets_sec”, “value”: 9.066666666666666, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.4211, “value_meta”: null, “name”: “net.out_bytes_sec”, “value”: 755.9333333333333, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.76, “value_meta”: null, “name”: “net.out_packets_dropped_sec”, “value”: 0.0, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.633, “value_meta”: null, “name”: “net.out_errors_sec”, “value”: 0.0, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.698, “value_meta”: null, “name”: “net.in_packets_dropped_sec”, “value”: 0.4, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.3372, “value_meta”: null, “name”: “net.in_bytes_sec”, “value”: 645.8333333333334, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570793.546, “value_meta”: null, “name”: “net.out_packets_sec”, “value”: 0.8666666666666667, “dimensions”: {“device”: “eth0”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750540860.986, “value_meta”: null, “name”: “http_status”, “value”: 0, “dimensions”: {“url”: “http://localhost:8091/healthcheck”, “hostname”: “lvpalkvm207”, “component”: “monasca-persister”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.538, “value_meta”: null, “name”: “mem.total_mb”, “value”: 3951, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.727, “value_meta”: null, “name”: “mem.used_cache”, “value”: 1126, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.7112, “value_meta”: null, “name”: “mem.used_buffers”, “value”: 168, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.741, “value_meta”: null, “name”: “mem.used_real_mb”, “value”: 846, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.604, “value_meta”: null, “name”: “mem.used_mb”, “value”: 2142, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.648, “value_meta”: null, “name”: “mem.swap_total_mb”, “value”: 4095, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.57, “value_meta”: null, “name”: “mem.free_mb”, “value”: 513, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.6802, “value_meta”: null, “name”: “mem.swap_free_mb”, “value”: 3838, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.697, “value_meta”: null, “name”: “mem.swap_free_perc”, “value”: 93.7, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.754, “value_meta”: null, “name”: “mem.used_shared”, “value”: 0, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.59, “value_meta”: null, “name”: “mem.usable_mb”, “value”: 1590, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.626, “value_meta”: null, “name”: “mem.usable_perc”, “value”: 40.2, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570837.6611, “value_meta”: null, “name”: “mem.swap_used_mb”, “value”: 257, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.446, “value_meta”: null, “name”: “disk.space_used_perc”, “value”: 35.4, “dimensions”: {“device”: “ubuntu–vg-root”, “hostname”: “lvpalkvm207”, “mount_point”: “/”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570857.044, “value_meta”: null, “name”: “io.write_kbytes_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.8699, “value_meta”: null, “name”: “io.read_req_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.927, “value_meta”: null, “name”: “io.write_req_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.7988, “value_meta”: null, “name”: “disk.inode_used_perc”, “value”: 0.49, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570857.095, “value_meta”: null, “name”: “io.read_time_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.75, “value_meta”: null, “name”: “disk.space_used_perc”, “value”: 33.5, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570857.142, “value_meta”: null, “name”: “io.write_time_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.997, “value_meta”: null, “name”: “io.read_kbytes_sec”, “value”: 0.0, “dimensions”: {“device”: “vda1”, “hostname”: “lvpalkvm207”, “mount_point”: “/boot”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570856.514, “value_meta”: null, “name”: “disk.inode_used_perc”, “value”: 6.25, “dimensions”: {“device”: “ubuntu–vg-root”, “hostname”: “lvpalkvm207”, “mount_point”: “/”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.757, “value_meta”: null, “name”: “cpu.wait_perc”, “value”: 0.0, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.7808, “value_meta”: null, “name”: “cpu.idle_time”, “value”: 13359320.99, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.929, “value_meta”: null, “name”: “cpu.percent”, “value”: 39.7, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.796, “value_meta”: null, “name”: “cpu.user_time”, “value”: 3071255.5100000002, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.841, “value_meta”: null, “name”: “cpu.user_perc”, “value”: 33.4, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.896, “value_meta”: null, “name”: “cpu.wait_time”, “value”: 6941.16, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.959, “value_meta”: null, “name”: “cpu.system_perc”, “value”: 5.6, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.99, “value_meta”: null, “name”: “cpu.system_time”, “value”: 470672.79000000004, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.827, “value_meta”: null, “name”: “cpu.stolen_perc”, “value”: 0.7, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.812, “value_meta”: null, “name”: “cpu.idle_perc”, “value”: 60.2, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570884.865, “value_meta”: null, “name”: “cpu.frequency_mhz”, “value”: 2933.436, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570912.96, “value_meta”: null, “name”: “load.avg_1_min”, “value”: 0.338, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570913.004, “value_meta”: null, “name”: “load.avg_15_min”, “value”: 0.307, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570912.988, “value_meta”: null, “name”: “load.avg_5_min”, “value”: 0.295, “dimensions”: {“hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570931.9949, “value_meta”: null, “name”: “zookeeper.max_latency_sec”, “value”: 2.78, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570931.9739, “value_meta”: null, “name”: “zookeeper.avg_latency_sec”, “value”: 0.0, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.097, “value_meta”: null, “name”: “zookeeper.zxid_epoch”, “value”: 0, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.136, “value_meta”: null, “name”: “zookeeper.node_count”, “value”: 200, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.052, “value_meta”: null, “name”: “zookeeper.connections_count”, “value”: 13, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.015, “value_meta”: null, “name”: “zookeeper.in_bytes”, “value”: 19223406, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570931.942, “value_meta”: null, “name”: “zookeeper.min_latency_sec”, “value”: 0.0, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.079, “value_meta”: null, “name”: “zookeeper.outstanding_bytes”, “value”: 0, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.034, “value_meta”: null, “name”: “zookeeper.out_bytes”, “value”: 19276621, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570932.115, “value_meta”: null, “name”: “zookeeper.zxid_count”, “value”: 2358068, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “zookeeper”, “mode”: “standalone”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570969.398, “value_meta”: null, “name”: “kafka.consumer_lag”, “value”: 0, “dimensions”: {“topic”: “metrics”, “hostname”: “lvpalkvm207”, “component”: “kafka”, “service”: “kafka”, “consumer_group”: “thresh-metric”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750570981.0332, “value_meta”: null, “name”: “kafka.consumer_lag”, “value”: 0, “dimensions”: {“topic”: “events”, “hostname”: “lvpalkvm207”, “component”: “kafka”, “service”: “kafka”, “consumer_group”: “thresh-event”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571129.876, “value_meta”: null, “name”: “process.mem.rss_mbytes”, “value”: 502.0, “dimensions”: {“process_name”: “kafka.Kafka”, “hostname”: “lvpalkvm207”, “service”: “kafka”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571209.481, “value_meta”: null, “name”: “process.mem.rss_mbytes”, “value”: 1033.0, “dimensions”: {“process_name”: “monasca-thresh”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “storm”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571152.345, “value_meta”: null, “name”: “process.thread_count”, “value”: 16, “dimensions”: {“process_name”: “influxd”, “component”: “influxd”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571163.209, “value_meta”: null, “name”: “process.pid_count”, “value”: 0, “dimensions”: {“process_name”: “monasca-notification”, “component”: “monasca-notification”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “mon-notification”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.884, “value_meta”: null, “name”: “process.mem.rss_mbytes”, “value”: 156.0, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571128.754, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “kafka.Kafka”, “hostname”: “lvpalkvm207”, “service”: “kafka”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.8499, “value_meta”: null, “name”: “process.open_file_descriptors”, “value”: 125.0, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571174.4731, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571229.762, “value_meta”: null, “name”: “process.pid_count”, “value”: 0, “dimensions”: {“process_name”: “rabbitmq”, “hostname”: “lvpalkvm207”, “service”: “rabbitmq”, “process_user”: “rabbitmq”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571151.325, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “influxd”, “component”: “influxd”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571129.8, “value_meta”: null, “name”: “process.thread_count”, “value”: 48, “dimensions”: {“process_name”: “kafka.Kafka”, “hostname”: “lvpalkvm207”, “service”: “kafka”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.81, “value_meta”: null, “name”: “process.io.write_kbytes”, “value”: 1221368.0, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571185.751, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “storm.daemon.nimbus”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571209.457, “value_meta”: null, “name”: “process.cpu_perc”, “value”: 1.2, “dimensions”: {“process_name”: “monasca-thresh”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “storm”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571152.4329, “value_meta”: null, “name”: “process.mem.rss_mbytes”, “value”: 35.0, “dimensions”: {“process_name”: “influxd”, “component”: “influxd”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.6511, “value_meta”: null, “name”: “process.io.write_count”, “value”: 78517971, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571129.835, “value_meta”: null, “name”: “process.cpu_perc”, “value”: 2.5, “dimensions”: {“process_name”: “kafka.Kafka”, “hostname”: “lvpalkvm207”, “service”: “kafka”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571152.387, “value_meta”: null, “name”: “process.cpu_perc”, “value”: 0.3, “dimensions”: {“process_name”: “influxd”, “component”: “influxd”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.7742, “value_meta”: null, “name”: “process.io.read_count”, “value”: 267140020, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571119.14, “value_meta”: null, “name”: “process.thread_count”, “value”: 23, “dimensions”: {“process_name”: “org.apache.zookeeper.server”, “hostname”: “lvpalkvm207”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571219.835, “value_meta”: null, “name”: “process.pid_count”, “value”: 0, “dimensions”: {“process_name”: “epmd”, “hostname”: “lvpalkvm207”, “service”: “rabbitmq”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571119.175, “value_meta”: null, “name”: “process.cpu_perc”, “value”: 0.7, “dimensions”: {“process_name”: “org.apache.zookeeper.server”, “hostname”: “lvpalkvm207”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571197.713, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “storm.daemon.supervisor”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.697, “value_meta”: null, “name”: “process.thread_count”, “value”: 94, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571119.2131, “value_meta”: null, “name”: “process.mem.rss_mbytes”, “value”: 124.0, “dimensions”: {“process_name”: “org.apache.zookeeper.server”, “hostname”: “lvpalkvm207”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.7341, “value_meta”: null, “name”: “process.cpu_perc”, “value”: 2.0, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571175.924, “value_meta”: null, “name”: “process.io.read_kbytes”, “value”: 12608.0, “dimensions”: {“process_name”: “monasca-persister”, “component”: “monasca-persister”, “hostname”: “lvpalkvm207”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571209.429, “value_meta”: null, “name”: “process.thread_count”, “value”: 236, “dimensions”: {“process_name”: “monasca-thresh”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “storm”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571117.9968, “value_meta”: null, “name”: “process.pid_count”, “value”: 1, “dimensions”: {“process_name”: “org.apache.zookeeper.server”, “hostname”: “lvpalkvm207”, “service”: “zookeeper”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571207.4238, “value_meta”: null, “name”: “process.pid_count”, “value”: 5, “dimensions”: {“process_name”: “monasca-thresh”, “component”: “apache-storm”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “storm”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571141.326, “value_meta”: null, “name”: “process.pid_count”, “value”: 0, “dimensions”: {“process_name”: “monasca-agent”, “component”: “monasca-agent”, “hostname”: “lvpalkvm207”, “service”: “monitoring”, “process_user”: “mon-agent”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571253.072, “value_meta”: null, “name”: “monasca.collection_time_sec”, “value”: 0.4812037944793701, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “monasca-agent”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’), Message(magic=0, attributes=0, key=’1489750574466′, value='{“metric”: {“timestamp”: 1489750571267.263, “value_meta”: null, “name”: “monasca.thread_count”, “value”: 13, “dimensions”: {“hostname”: “lvpalkvm207”, “component”: “monasca-agent”, “service”: “monitoring”}}, “meta”: {“region”: “RegionOne”, “tenantId”: “6062fb6b11a648cf8e9616fcdedc479f”}, “creation_time”: 1489750574}’)])] _send_broker_aware_request /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/client.py:207
      2017-03-17 04:36:14,471.471 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reinitializing socket connection for localhost:9092 reinit /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:214
      2017-03-17 04:36:14,476.476 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to send 36354 bytes to Kafka, request 53 send /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:135
      2017-03-17 04:36:14,478.478 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reading response 53 from Kafka recv /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:157
      2017-03-17 04:36:14,479.479 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-17 04:36:14,479.479 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 4/4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-17 04:36:14,480.480 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-17 04:36:14,481.481 13489 DEBUG monasca_common.kafka_lib.conn [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 35/35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-17 04:36:14,482.482 13489 DEBUG monasca_common.kafka_lib.client [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Response 53: [ProduceResponse(topic=’metrics’, partition=43, error=0, offset=2051)] _send_broker_aware_request /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/client.py:265
      2017-03-17 04:36:14,486.486 13489 DEBUG gunicorn.error [req-e89e5161-7abf-4dfb-ad4e-2560117c8fe3 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Closing connection. debug /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:250

    6. Hello Shaun,

      still having the same issue..

      on dashboard I can see error like
      Error: Unable to retrieve notifications: (pymysql.err.OperationalError) (2003, “Can’t connect to MySQL server on ‘10.48.151.148’ ([Errno 111] ECONNREFUSED)”)

    7. mysql -umonasca -p
      Enter password:
      Welcome to the MySQL monitor. Commands end with ; or \g.
      Your MySQL connection id is 283
      Server version: 5.5.54-0ubuntu0.14.04.1 (Ubuntu)

      Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.

      Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

      mysql> quit
      Bye
      [email protected]:/tmp# mysql -umonasca -p 10.48.151.148
      Enter password:
      ERROR 1044 (42000): Access denied for user ‘monasca’@’localhost’ to database ‘10.48.151.148’

    8. mysql -umonasca -pqydcos -h10.48.151.148
      ERROR 2003 (HY000): Can’t connect to MySQL server on ‘10.48.151.148’ (111)

      1. in step 2, have you modified the db name and user, password in mon_mysql file, in line 234, 235, if you did, recheck step 7, modify the db user, db name, password to yours.

    9. I didn’t change the password in mon_mysql file. i changed the conf file to update use and password.. however i still have issue. i can log into mysql with localhost but not with 10.48.151.148..

      monasca metric-list show blank..
      now I dont see issue on dashboard.. and api log.. except these as below;

      2017-03-20 02:51:25,471.471 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reinitializing socket connection for localhost:9092 reinit /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:214
      2017-03-20 02:51:25,475.475 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to send 36361 bytes to Kafka, request 279 send /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:135
      2017-03-20 02:51:25,487.487 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Reading response 279 from Kafka recv /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:157
      2017-03-20 02:51:25,488.488 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-20 02:51:25,489.489 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 4/4 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-20 02:51:25,490.490 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] About to read 35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:88
      2017-03-20 02:51:25,491.491 2334 DEBUG monasca_common.kafka_lib.conn [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Read 35/35 bytes from Kafka _read_bytes /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/conn.py:110
      2017-03-20 02:51:25,491.491 2334 DEBUG monasca_common.kafka_lib.client [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Response 279: [ProduceResponse(topic=’metrics’, partition=10, error=0, offset=13266)] _send_broker_aware_request /usr/local/lib/python2.7/dist-packages/monasca_common/kafka_lib/client.py:265
      2017-03-20 02:51:25,494.494 2334 DEBUG gunicorn.error [req-028c151c-1068-4301-b026-2d31876f5f65 5b462a4db4324487af33b6f4647dd1f3 6062fb6b11a648cf8e9616fcdedc479f – default default] Closing connection. debug /usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py:250

    Leave a Reply

    Your email address will not be published. Required fields are marked *