Posts

Showing posts from November, 2013

Logstash - Configure logstash-indexer the right way to avoid high CPU usage

Image
In the previous blog post about setting logstash with ElasticSearch (here), I was output like this:

input {
        redis {
                host => "127.0.0.1"
                data_type => "list"
                key => "logstash"
                codec => json
        }
}
output {
        elasticsearch {
embedded => true
        }
}

As you can see, what logstash does is only move logs from redis to elasticsearch. There is no way it needs a lot of cpu power as I pointed out in the previous blog post.



What's wrong? After a while searching around Google and digging the logstash documentation, I found out the reason. It is the "embbedded" option in my configuration.


According to the logstash documentation about storing logs to elasticsearch:

embedded

Value type is boolean
Default value is false

Run the elasticsearch server embedded in this process. This option is useful if you want to run a single logstash process that handles log processing and in…

Ubuntu - Run Logstash as a service

In the previous blog post, I had setup a log management system with logstash, elasticsearch, and redis. But I started logstash by running the java command for demonstration purpose. So, to run Logstash as a service in a Ubuntu machine:


1. Create a init script for logstash at /etc/init.d/logstash: (This will also start the Kibana web interface)

This script is for logstash in the indexer server. For the shipper machine, change "mode", "logstash_conf" and "command" definition:

#! /bin/sh

### BEGIN INIT INFO
# Provides:          logstash-indexer
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO

. /lib/lsb/init-functions

mode="indexer"
name="logstash-$mode"
logstash_bin="/usr/bin/java -- -jar /opt/logstash/logstash-1.2.2-flatjar.jar"
logstash_…

Proxmox - Set number of CPUs available in the container (OpenVZ)

Image
To set number of CPUs for a container (OpenVZ) in Proxmox VE:

+ Open the Proxmox console by click the Shell link in the web interface of proxmox:








+ In the opened console window, run vzlist to get CTID of containers:



+ Run the following command to set number of CPU for the container with CTID 102:

# vzctl set 102 --cpus 2 --save




References: http://pve.proxmox.com/wiki/Vzctl_manual

Centralized Logs Management with Logtash, ElasticSearch, and Redis

Image
Deploying a Centralized Logs Management System seems very easy these days with such these great tools:

+ Logtash: collect logs, index logs, process logs, and ship logs
+ Redis: receive logs from logs shippers
+ ElasticSearch: store logs
+ Kibana: web interface with graphs, tables...

We will implement the logs management system as the following architecture:


(based on this article: http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/)
In  this tutorial, I only deploy one shipper (nginx logs of my Django app) on one machine, and one server to play as logs indexer (redis, logstash, elasticsearch, kibana):


1. On the indexer server, install and run Redis: 

http://iambusychangingtheworld.blogspot.com/2013/11/install-redis-and-run-as-service.html

2. On the indexer server, install and run ElasticSearch:

$ sudo aptitude install openjdk-6-jre
$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.7.deb
$ sudo dpk…

Install Redis and run as a service

To install and run Redis as a service in Ubuntu, following these steps:


1. Download and build Redis:


$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make
$ make test


After the compilation the src directory inside the Redis distribution is populated with the different executables that are part of Redis:

redis-server is the Redis Server itself.redis-cli is the command line interface utility to talk with Redis.redis-benchmark is used to check Redis performances.redis-check-aof and redis-check-dump are useful in the rare event of corrupted data files.
It is a good idea to copy both the Redis server than the command line interface in proper places using the following commands:

$ sudo cp redis-server /usr/local/bin/
$ sudo cp redis-cli /usr/local/bin/


2. Install Redis:



Create a directory where to store your Redis config files and your data:

$ sudo mkdir /etc/redis
$ sudo mkdir /var/redis

Copy the init script that you'll find in the Redis distri…

Python - Reverse Cipher

This is the most basic and weakest encrypting method: Reverse Cipher

def reverse_cipher(msg):
    translated = ''
    for i in reversed(range(len(msg))):
        translated += msg[i]
    return translated


>>> reverse_cipher('This is so genius')
'suineg os si sihT'



"Effective Django" by Nathan Yergler

"Effective Django" is a great tutorial made by Nathan Yergler which was presented at PyCon2013.

All the texts: http://effectivedjango.com/

Video: https://www.youtube.com/watch?v=NfsJDPm0X54

This maybe the best thing to do this weekend. "Practice makes perfect", you know.

Linux - List files filtered by owner and permissions

This is quite an interesting and common question in linux:

List all the files in a folder that are owned by a the user "trinh", with permissions "-rw-r--r--"



1. The easiest way is to use ls command and chain filter with grep:

ls -lt . | grep trinh | grep '\-rw\-r\-\-r\-\-'


2. But, if you don't want to escape these permission characters, you can use awk to translate them into number. So, we will translate "-rw-r--r--" as following:

+ "-" : the first character indicates whether it is a file or a directory
+ "rw-" : the next 3 characters is the owner's right, can be translated to 110 which means 6 in decimal system.
+ "r--": next 3 characters is the group's right, is 100 in binary and 4 in decimal.
+ "r--": the last 3 characters is the other users's right, 100 in binary, and 4 in decimal.

So, "-rw-r--r--" is 644 in decimal system.


ls -lt . | grep trinh | awk '{k=0;for(i=0;i<=8;i++)k+=(…

Using git to deploy/update source code to production server

I've just tried the we-all-know-what-it-is tool, git, to deploy and update changes to production server:


1. On my development computer, commit all changes and push to the repository:

$ git add -A .
$ git commit -m "Before deploying"
$ git push origin master


2. On the production server:

$ git clone myaccount@my.repo.address


3. In the development machine, update code, commit, and push to repository:

$ git add -A .
$ git commit -m "Updates"
$ git push origin master


4. In production server, pull code from master:


$ git pull origin master



...and so on


It's quite interesting!

Django - Using Memcached

Image
The easiest way to use Memcached as a cache backend for django is to enable it to cache the entire site:

1. Install Memcached:

$ sudo apt-get install memcached

The memcached service will run at 127.0.0.1:11211 by default.


2. Install python API for memcached:

(myvirtualenv)$ pip install python-memcached


3. Enable Django to use Memcached by modified settings.py:

MIDDLEWARE_CLASSES = (
'django.middleware.cache.UpdateCacheMiddleware', # should be the first line
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.cache.FetchFromCacheMiddleware', # should be the last line
)

#memcache
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
   …

Supervisord and django-celery: Stop the process properly

Just noted when I stopped the django-celery app which monitored by supervisord with:

# supervisorctl stop myappcelery

The status show myappcelery had stopped:

# supervisorctl status

myappcelery                         STOPPED    Nov 20 10:48 AM


But, the python processed of django-celery were still running:

# ps aux | grep celery

root     25729  9.0  1.8 225676 38188 ?        S    11:07   0:00 python /mydjangoapp/manage.py celeryd --pool=eventlet -v 2 -B -s celery -E -l info
root     25748  1.0  1.7 164640 36596 ?        S    11:07   0:00 python /mydjangoapp/manage.py celeryd --pool=eventlet -v 2 -B -s celery -E

Then, I take a look at the supervisord configuration for my django app:

[program:ptccele]
directory=/mydjangoapp/
command=/mydjangoapp/shellscripts/production/ptccele.sh
user=root
autostart=true
autorestart=true
stdout_logfile=/var/log/cele.log
stderr_logfile=/var/log/cele_error.log
redirect_stderr=true

I used a shellscript to run the celery worker. Supervisord was managing sh script…

Supervisord - Using the built-in web interface to monitor processes

Image
Supervisord, a process control system, has a simple built-in web interface to help you manage processes. It is just so great!!!

To enable it:

1. Add these line to /etc/supervisor/supervisord.conf to enable supervisord web interface in port 9001 (localhost, domain: my.domain.com):

...
[inet_http_server]
port=127.0.0.1:9001
username=guest
password=mysecret
...


+Restart supervisord:

$ sudo supervisorctl reload



2. Pass requests of my.domain.com to 127.0.0.1:9001:

+ Create a nginx configuration file for supervisord /etc/nginx/sites-available/supervisord:

upstream supervisord {
server localhost:9001 fail_timeout=0;
}


server {
        listen 80;
server_name my.domain.com;
access_log /var/log/access_supervisor.log;
error_log /var/log/error_supervisor.log;

location / {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;

proxy_pass http://supervisord;
}

}

+ Create a symlink to that file in /etc/nginx/sites-enable/:

$ sudo ln -s /etc/nginx/sites-ava…

Bash script and Crontab

There is one thing I want to take note about running a bash script with crontab:

If I use the bash script alone, this will work:

$ /path/to/my/script.sh


#! /bin/bash
export DISPLAY=:0
/path/to/my/pythonGUIapp.py &



But if I run script.sh with crontab, It will not work:

$ sudo crontab -u trinh -e

5 5  * * *  /path/to/my/script.sh


To make it work, I have to change directory to the pythonGUIapp.py before executing it. So, change the script.sh as below:

#! /bin/bash
export DISPLAY=:0
cd /path/to/my/
./pythonGUIapp.py &

To Read Is To Fly

Image
Here are some great resources for readers or book lovers like me to read online:


The Public Domain Review: http://publicdomainreview.org/

"The Public Domain Review is a not-for-profit project dedicated to showcasing the most interesting and unusual out-of-copyright works available online.

All works eventually fall out of copyright – from classic works of art, music and literature, to abandoned drafts, tentative plans, and overlooked fragments. In doing so they enter the public domain, a vast commons of material that everyone is free to enjoy, share and build upon without restriction.

We believe the public domain is an invaluable and indispensable good, which – like our natural environment and our physical heritage – deserves to be explicitly recognised, protected and appreciated.

The Public Domain Review aims to help its readers to explore this rich terrain – like a small exhibition gallery at the entrance of an immense network of archives and storage rooms that lie beyond."




So…

Xubuntu - Remotely lock/unlock your monitor using an Android phone

Image
Today, I looked for a way to lock a Xubuntu machine remotely, and after a while of googling, I figured out this method:


1. I created 2 alias command to lock and unlock the monitor's screen (xscreensaver) in /home/myuser/.bash_aliases:

alias lock='export DISPLAY=:0.0; xflock4;'
alias unlock='export DISPLAY=:0.0; kill -9 "$(pidof xscreensaver)"; xscreensaver -no-splash &'



2. Install JuiceSSH (a ssh client) app in my Android phone:

https://play.google.com/store/apps/details?id=com.sonelli.juicessh&hl=en



3. From my Android, connect to the Xubuntu machine via SSH with username 'myuser', and then:

To lock the screen, type this in my phone terminal:

$ lock

And unlock the screen with:

$ unlock



It's just fascinating! Feel like a hacker!!! \m/\m/\m/

Bash script - Using alias

Recently, whenever I open my computer's terminal, I always have to change directory to my projects folder. So, to avoid repeating that action, I add an alias to my Xubuntu's bash an alias:

~/.bash_aliases:

alias projects='cd /var/projects/'


Restart, and then I can only need to type 'projects' to change my current directory to the project base.


For more information:

[0] http://www.ibm.com/developerworks/aix/library/au-getstartedbash/
[1] http://www.tldp.org/LDP/abs/abs-guide.pdf


Shell script - Kill a specific program if its CPU load exceeded a certain threshold

This shell script I wrote in order to kill the python process if its CPU usage percentage exceeded 96%:


#! /bin/bash
THRESHOLD=96
while [ 1 ];
do
    MYPROGRAM_ID=`pidof python`
    # if the result is not empty, or there is a python process
    if [ ! -z "$
MYPROGRAM_ID" ] ; then
        CPU_LOAD=`ps -o %C -p $
MYPROGRAM_ID | tail -n 1`
        echo $CPU_LOAD
        # bc will return 0 for false and 1 for true
        if [ $(echo "$CPU_LOAD > $THRESHOLD" | bc) -ne 0 ] ; then
            kill -9 $GATEKEEPER_ID
        else
            echo "Normal"
        fi
    fi
done



Useful!


Shell script - Get CPU usage percentage of a process

To get the CPU usage percentage of a specific process, run the following command:

$ ps -o %C -p <PID> | tail -n 1


Example: to get CPU load of /usr/sbin/mysqld process which PID is 32241:

$ ps -o $C -p 32241 | tail -n 1

0.1

Xubuntu - Add an autostart program without GUI

To add an autostart program in Xubuntu without using GUI tools:


1. Create a file in ~/.config/autostart/newautostartapp.desktop


2. Fill the file with:


[Desktop Entry]
Encoding=UTF-8
Version=0.9.4
Type=Application
Name=New Autostart App
Comment=New Autostart App
Exec=<my application command>
StartupNotify=false
Terminal=false
Hidden=false



Restart and log-in again to see.

Xubuntu - Disable Mouse's Right Click

To disable right click for just desktop :

xmodmap -e "pointer = 1 2 99"

To apply for system wide:

xmodmap -e 'pointer = 1 2 0 4 5 6 7 8 9'

If you want to reverse it:

xmodmap -e 'pointer = default'

But, notice that it will restore to defaults upon rebooting.So, create an autostart script as following in ~/.config/autostart/disablerightclick.desktop:

[Desktop Entry]
Encoding=UTF-8
Version=0.9.4
Type=Application
Name=Disable Right Click
Comment=Disable Right Click
Exec=xmodmap -e "pointer = 1 2 99"
StartupNotify=false
Terminal=false
Hidden=false



Restart and log-in again to see it work.

wxPython - ENTER/RETURN key binding for a wx.Dialog

To set ENTER / RETURN key binding for a wx.Dialog, add the following to a button of that dialog:

self.mybutton.SetDefault()


For example:

class LoginDialog(wx.Dialog):

    def __init__(self, parent, id=-1, title='Login',
                 text='Please type your username and password:',
                 username=''):

        size = (400, 200)
        wx.Dialog.__init__(self, parent, id, title, size=size)

        self.panel = LoginPanel(self)

        sizer = wx.GridSizer(1, 1)
        sizer.Add(self.panel, flag=wx.ALIGN_CENTER | wx.ALIGN_CENTER_VERTICAL)
        self.SetSizer(sizer)

        self.Bind(wx.EVT_CLOSE, self.OnClose, self)
        self.Show()

    def OnClose(self, event):
        self.Destroy()


class LoginPanel(wx.Panel):

    def __init__(self, parent):
        wx.Panel.__init__(self, parent)

        self.setup_login_form()

        self.do_layout()
        self.setup_bindings()


    def setup_login_form(self):
        self.usernametxt = wx.StaticText(self, label="Username:…

Ubuntu - How to launch a GUI program in a remote Ubuntu box via SSH

To launch a GUI program in a xUbuntu machine from a remote xUbuntu box, following these steps:

1. Assuming someone had logged-in to the remote machine (let call it R) with user 'remoteuser' in graphical mode (init 5)


2. From my computer, ssh to R machine with user 'remoteuser' (!important, if you login to R with another user, this won't work):

trinh@local-pc$ ssh remoteuser@R.IP.Address

3. Run the following command before launch any graphical program:

remoteuser@remote-pc$ export DISPLAY=:0

4. Finally, launch whatever GUI program you want, for example: Firefox:

remoteuser@remote-pc$ firefox &

Xubuntu 12.04 - Fix low screen solution issue

Early this morning, I installed a fresh Xubuntu 12.04 in a machine. After the computer booted up, the screen displays at 1024x768 mode. And I cannot changed to higher mode (actually, there was higher mode). So, I do the following steps to fix this issue:


1.Install this tool in a good machine (which is Xubuntu and does not have this problem) to get EDID (Extended Display Identification Data) of the monitor:

$ sudo apt-get install read-edid


2. Run the following command to get the  monitor information (EDID):

$ sudo get-edid | parse-edid


The results will be something like:

        # EDID version 1 revision 3
Section "Monitor"
        # Block type: 2:0 3:ff
        # Block type: 2:0 3:fc
        Identifier "DELL E1912H"
        VendorName "DEL"
        ModelName "DELL E1912H"
        # Block type: 2:0 3:ff
        # Block type: 2:0 3:fc
        # Block type: 2:0 3:fd
        HorizSync 30-83
        VertRefresh 56-75
        # Max dot clock (video bandwidth) 90 MHz
  …

SSH - Allow only some specific users/groups use SSH

To allow only some specific users/groups access the SSH server:

$ sudo nano /etc/ssh/sshd_config


Add the following line to allow access only for user 'trinh':


AllowUsers trinh

Or this line to allow only group 'mygroup':

AllowGroups mygroup



Restart ssh service:

$ sudo service ssh restart


More options (source: http://knowledgelayer.softlayer.com/learning/how-do-i-permit-specific-users-ssh-access):

AllowGroups
This keyword can be followed by a list of group name patterns, separated by spaces.If specified, login is allowed only for users whose primary group or supplementary group list matches one of the patterns.`*' and `?' can be used as wildcards in the patterns.Only group names are valid; a numerical group ID is not recognized.By default, login is allowed for all groups.AllowUsers
This keyword can be followed by a list of user name patterns, separated by spaces.If specified, login is allowed only for user names that match one of the patterns.`*' and `?' can be …

Install MariaDB in Ubuntu13.10

Ubuntu 13.10 is the latest release when I post this blog. I tried to install MariaDB following these instructions:


sudo apt-get install software-properties-common sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db sudo add-apt-repository 'deb http://ftp.yz.yamagata-u.ac.jp/pub/dbms/mariadb/repo/5.5/ubuntu saucy main'sudo apt-get update sudo apt-get install mariadb-server

(source: https://downloads.mariadb.org/mariadb/repositories/)


And I got this error message:

The following packages have unmet dependencies: mariadb-server-5.5 : Depends: mariadb-client-5.5 (>= 5.5.33a+maria-1~saucy) but it is not going to be installed Depends: mariadb-server-core-5.5 (>= 5.5.33a+maria-1~saucy) but it is not going to be installed E: Unable to correct problems, you have held broken packages.

The reason for this issue is the version of MariaDB is mismatch with the version in the Ubuntu repositories (click to read more about this).


So, to fix that:


1. Yo…

wxPython + Django - Basic authentication in wxPython with Django

The beauty of Python and any other OOP programing languages is the reusable functionality. So, I reuse the authentication module of Django to add a basic security layer for my wxPython application:

+ Authenticate a user with Django (django.contrib.auth.authenticate)
+ Save the session to a file names 'cred'
+ Read the 'cred' file to verify if that user has the privilege.


* The main frame:

class HomeFrame(wx.Frame):

    def __init__(self):
        wx.Frame.__init__(
            self,
            parent=None,
            title='My wxPython application',
            size=wx.DisplaySize(),
        )

        self.panel = HomePanel(self)

        sizer = wx.GridSizer(1, 1)
        sizer.Add(self.panel, flag=wx.ALIGN_CENTER)
        self.SetSizer(sizer)

        self.Bind(wx.EVT_CLOSE, self.OnClose, self)
        self.Show()


    def show_login_dialog(self):
        login_dialog = LoginDialog(self)



    def OnClose(self, event):
        self.Destroy()



class HomePanel(wx.Panel):

    search_he…

wxPython - Using Save File Dialog

I use this snippet to create a dialog for saving file using wxPython library:

class ReportDialog(wx.Dialog):

    def __init__(self, parent, id=-1, title='Report',
                 text='Select Type Of Report'):

        size = (350, 200)
        wx.Dialog.__init__(self, parent, id, title, size=size)

        self.panel = ReportPanel(self)

        sizer = wx.GridSizer(1, 1)
        sizer.Add(self.panel, flag=wx.ALIGN_CENTER | wx.ALIGN_CENTER_VERTICAL)
        self.SetSizer(sizer)

        self.Show()


REPORT_TYPES = ['All', 'Teachers', 'Students', 'Parents']

class ReportPanel(wx.Panel):

    def __init__(self, parent):
        wx.Panel.__init__(self, parent)

        self.setup_report_panel()

        self.do_layout()
        self.setup_bindings()

    def setup_report_panel(self):
        self.errmsg = wx.StaticText(self, label="")
        self.errmsg.SetForegroundColour((255,0,0))

        self.reporttypeselectbox = wx.ComboBox(self, choices=REPORT_TYPES, …

Just a great quote from "On Liberty" by John Stuart Mill

Image
"No society in which these liberties are not, on the whole, respected, is free, whatever may be its form of government; and none is completely free in which they do not exist absolute and unqualified. The only freedom which deserves the name, is that of pursuing our own good in our own way, so long as we do not attempt to deprive others of theirs, or impede their efforts to obtain it. Each is the proper guardian of his own health, whether bodily, or mental and spiritual. Mankind are greater gainers by suffering each other to live as seems good to themselves, than by compelling each to live as seems good to the rest."

~"On Liberty" by John Stuart Mill.



John  Stuart Mill (1806-1873)
(source: http://en.wikipedia.org/wiki/John_Stuart_Mill)