Posts

Showing posts from December, 2013

Google App Engine SDK for Python - Using datastore query at local for testing

When I tried to do some test of datastore query in the python console:

query = MyModel.all()
query.filter("username =", "myusername")
result = query.get()


I got this error"

BadArgumentError: app must not be empty.


After googling around, I found this solution: before execute the query, run these following commands:

import os

os.environ['APPLICATION_ID'] = 'dangtrinhnt'                               
datastore_file = '/dev/null'
from google.appengine.api import apiproxy_stub_map,datastore_file_stub
apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
stub = datastore_file_stub.DatastoreFileStub('dangtrinhnt', datastore_file, '/')
apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)


Then, execute any query I want, for example the query at the beginning of this blog post (the blue lines).


Reference: http://einaregilsson.com/unit-testing-model-classes-in-google-app-engine/

Google App Engine - Running GAE inside a virtualenv

To be able to work with Google App Engine inside a virtualenv environment, we have to do some tricks:

1. Download and extract the Google App Engine SDK for Python:

+ Link: https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python

+ Extract the zip file to /opt directory. So, the directory path for GAE is something like:

/opt/gae


2. Open the activate script of the virtualenv environment at /home/.venv/myenv/bin/activate:

+ Find the line:

PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH


+ Insert the GAE path:

PATH="$VIRTUAL_ENV/bin:$PATH"
PATH="/opt/gae:$PATH"
export PATH

Save & exit

3. Create a path configuration file for GAE inside the virtualenv environment name /home/.venv/myenv/lib/python2.7/site-packages/gae.pth with the following content:

/opt/gae
import dev_appserver; dev_appserver.fix_sys_path()


Now you can run your GAE application inside a virtualenv environment.


If you get trouble with GAE does not regconize any python libraries, modif…

Python - Sorting a list of objects

Assuming I have a list of JSON objects:

obj1 = {'title': 'Super genius', 'modifiedDate': 1334014208.0, 'user_id': 123}

obj2 = {'title': 'Awesome', 'modifiedDate': 1333962645.0, 'user_id': 345}

obj3 = {'title': 'Gorgeous', 'modifiedDate': 1333894106.0, 'user_id': 531}

obj4 = {'title': 'Cool', 'modifiedDate': 1333968061.0, 'user_id': 214}

obj5 = {'title': 'Badass', 'modifiedDate': 1334016506.0, 'user_id': 678}


unorder_list = [obj4, obj5, obj1, obj2, obj3]


And I want to create a list contains those objects in ascending order of modifiedDate.


1. Solution #1: using lambda to sort by 'modifiedDate'

def order_list():
    order_list = unorder_list
    order_list.sort(key = lambda x: x['modifiedDate'])
    return order_list


2. Solution #2: do not use lambda

def order_list():
    order_list = []
    tmp = {}
    for i in unorder_list:
 …

Python - ROT13 encode

"ROT13 ("rotate by 13 places", sometimes hyphenated ROT-13) is a simple letter substitution cipher that replaces a letter with the letter 13 letters after it in the alphabet. ROT13 is an example of the Caesar cipher, developed in ancient Rome."

~Wikipedia.


Here are 2 solutions to implement ROT13 in Python:

0. 1st solution:

import string

ROT13 = string.maketrans(\
         "ABCDEFGHIJKLMabcdefghijklmNOPQRSTUVWXYZnopqrstuvwxyz",\
         "NOPQRSTUVWXYZnopqrstuvwxyzABCDEFGHIJKLMabcdefghijklm")

def rot13_encode(user_text):
    return string.translate(user_text, ROT13)


This method works smoothly in the console but when using with the web it will raise error because of some special characters in the input:

File"/usr/lib/python27/string.py", line 498,in translate return s.translate(table + s[:0])UnicodeDecodeError:'ascii' codec can't decode byte 0x80 in position 128: ordinal not in range(128)
The Unicode version of translate requir…

Python - Check if a string is convertible to an integer

To check if a string is convertible to an integer, use this following built-in function:

mystring.isdigit()

=> True: mystring can be converted to an integer.
=> False: mystring cannot be converted to an integer.

Reference:http://stackoverflow.com/questions/5606585/python-test-if-value-can-be-converted-to-an-int-in-a-list-comprehension

Python - Change case with built-in string functions

Case conversion in Python is quite easy with those built-in string functions:

swapcase()
Return a copy of the string with uppercase characters converted to lowercase and vice versa.

upper()
Return a copy of the string converted to uppercase.

title()
Return a titlecased version of the string: words start with uppercase characters, all remaining cased characters are lowercase.

lower()
Return a copy of the string converted to lowercase.

capitalize()
Return a copy of the string with only its first character capitalized.



References:http://docs.python.org/2/library/stdtypes.html#string-methods

Google App Engine - How to download source code

To download source code from a Google App Engine application, run the following command:


appcfg.py download_app -A <your_app_id> -V <your_app_version> <output_dir>


Reference: https://developers.google.com/appengine/docs/python/tools/uploadinganapp?csw=1#Python_Downloading_source_code

Happiness - Just a great quote from Jean-Jacques Rousseau

Image
"If there is a state where the soul can find a resting-place secure enough to establish itself and concentrate its entire being there, with no need to remember the past or reach into the future, where time is nothing to it, where the present runs on indefinitely but this duration goes unnoticed, with no sign of the passing of time, and no other feeling of deprivation or enjoyment, pleasure or pain, desire or fear than the simple feeling of existence, a feeling that fills our soul entirely, as long as this state lasts, we can call ourselves happy, not with a poor, incomplete and relative happiness such as we find in the pleasures of life, but with a sufficient, complete and perfect happiness which leaves no emptiness to be filled in the soul."

~Jean-Jacques Rousseau



(source: https://d202m5krfqbpi5.cloudfront.net/authors/1182863711p8/7994.jpg)

Google Drive API - Rename duplicate files / folders

Here are the case:

You create an empty document. You forgot to name the file and just close the window after typing something. The file will stay in your Google Drive with the name "Untitle Docutment". You repeat that action some other times. You will end up having so many duplicate files (or folders) in your Google Drive with the same name.

...

Someday you will (I will) need to re-organize all the documents and folders in your Drive account. The first thing to do is to rename all the duplicate files or folders by modified date. The newest file will have the name (e.g. "My document"), and the older files will have number after the name ("My document (1)", "My document (2)",...)

The following snippet I wrote to do the renaming task and make the world alot easier:


from dateutil import parser


def rename_file(service, file_id, new_title):
    """Rename a file.

    Args:
    service: Drive API service instance.
    file_id: ID of the file t…

Google Drive - Files or folders lost after moving around

If you move files or folders around your Google Drive in the web interface, you may lose them. They just disappear mysteriously.

According to a discussion thread on Google Group, this issue has been arround for almost 2 years. And there are some ways may help you to get those files/folders back:


0. Wait for 24h and your files/folders may be back (from nowhere).

1. Search in the Trash.

2. Search with "<name of the files or folder>*" keywords.



References:http://productforums.google.com/forum/#!msg/drive/wiXPYZR_DdA/zXtpOJXDBukJ

Python csv - Error when read data from csv file

Image
I wrote a utility function to read data from a csv file and add them to a dictionary (read here). But, you will get the following error when using that function:

Traceback (most recent call last):
...
    sniffdialect = csv.Sniffer().sniff(csv_file.read(10000), delimiters='\t,;')
  File "/usr/lib/python2.7/csv.py", line 184, in sniff
    raise Error, "Could not determine delimiter"
_csv.Error: Could not determine delimiter


It is because the csv file you are using is not in Unicode (UTF-8) format. So, to fix this error, please make sure the input csv file is formatted with UTF-8 standard (Not only Unicode).

* Not good:



* Good:



Google Drive API - A better way to get files you own

In the previous blog post about how to search your own files I used 'isAuthenticated' key to see if a file is mine (here). Today, I found a better and more accurate way to get files you own in Google Drive using the API:

+ Use the following query to search for files:

query = "'%s' in owners and trashed = false" % user_email

or

query = "'me' in owners and trashed = false"

def search_files(service, query_string):
    result = []
    page_token = None
    while True:
        try:
            param = {}
            if page_token:
                param['pageToken'] = page_token
            param['q'] = query_string
            files = service.files().list(**param).execute()

            result.extend(files['items'])
            page_token = files.get('nextPageToken')
            if not page_token:
                break
        except BadStatusLine, badstatus:
            print 'Error when searching files: %s' % badstatus
     …

Google App API - Migrate all users's emails from old domain to the new one

I'm using the following procedure to migrate all users's emails from old google domain to the new one:


1. Enable IMAP on all user accounts in the old domain. Use the enable_imap.py script from my previous blog post:

http://iambusychangingtheworld.blogspot.com/2013/12/google-email-settings-api-enable-imap.html

2. Loop through each user and executing these following steps:

a. Run the gyb.py script to backup emails from old domain account:

def backup_emails(email, service_account_email, email_folder):
    command = [
        "python", "gyb.py",
        "--email", email,
        "--action", "backup",
        "--local-folder", email_folder,
        "--service-account", service_account_email
    ]
    print "Executing commnad: %s" % command
    subprocess.call(command)

b. Run gyb.py script to restore backup-ed emails to new domain account:

def restore_emails(email, service_account_email, email_folder):
    command = [
   …

Google Email Settings API - Enable IMAP for all users in your domain

I wrote the following python script to enable IMAP for all users in my Google App domain:


#! /usr/bin/env python

import gdata.apps.emailsettings.client
from gdata.client import BadAuthentication
from gdata.client import RequestError
import csv
import socket
if socket.gethostname() in ['trinh-pc',]: # add your hostname here
    from settings_local import *
else:
    from settings import *
import sys



DOMAIN = 'mydomain.com'
DOMAIN_ADMIN = 'domain_admin_username'
DOMAIN_ADMIN_P = 'domain_admin_password'


def get_username_list_from_text(username_file_path):
    # username_list = []
    text_file = open(username_file_path, 'rb')
    username_list = text_file.read()
    text_file.close()
    username_list = username_list.split('\n')
    return username_list


def create_gdata_client(domain, admin_email, admin_password, appname='GEM'):
    try:
        client = gdata.apps.emailsettings.client.EmailSettingsClient(domain=domain)
        client.ClientLogin(email=admi…

Android - How to see saved wifi password

In Android, to see the saved password of all the wifi networks:

1. Install a root browser such as ROM Toolbox Lite' s root browser (you may need to root your phone to be able to use this tool)


2. Using root browser navigate to /data/misc/wifi/. You will see a file name wpa_supplicant.conf. All the wifi networks's information are stored in this file.


3. Open that file with a text editor in your phone. And you will see something like:

network={
    ssis="MyWifiNetwork"
psk="MyWifiP@ssword"
    key_mgmt=WPA-PSK
    sim_slot="-1"
    imsi="none"
    pcsc="none"
    priority=6
}

The value "MyWifiP@ssword" of "psk" key is the password you need, in plain text.

Excel - Split string by a delimiter

To split a string by a delimiter (for example: "@") using the following function:

For example, we have a string at Cell #A2:

myusername@mydomain.com


+ Get the left part of the string, myusername:

=LEFT(A2, FIND("@", A2)-1)

+ Get the right part of the string, mydomain.com:

=RIGHT(A2, LEN(A2) - FIND("@", A2))





Just bought a BeagleBone Black

Image
I've just bought a BeagleBone Black this morning. So many interesting projects waiting for me!!!! I'm so excited!!!!

I  bought it at a Vietnamese's online shop: http://www.hshopvn.com/BeagleBone Black's Specifications (http://beagleboard.org/Products/BeagleBone+Black):
What is BeagleBone Black?BeagleBone Black is a $45 MSRP community-supported development platform for developers and hobbyists. Boot Linux in under 10 seconds and get started on development in less than 5 minutes with just a single USB cable.

Processor: AM335x 1GHz ARM® Cortex-A8512MB DDR3 RAM2GB 8-bit eMMC on-board flash storage3D graphics acceleratorNEON floating-point accelerator2x PRU 32-bit microcontrollers
ConnectivityUSB client for power & communicationsUSB hostEthernetHDMI2x 46 pin headers

Software Compatibility
Ångström LinuxAndroidUbuntuCloud9 IDE on Node.js w/ BoneScript libraryplus much more

Awesomeness!!!!







M$ Active Directory - Delete multiple AD users through command line

To delete multiple AD users through command line in a Active Directory Domain Controller:

1. Prepare the list of users in a text file (users.txt), one user per line:

user.txt:

user1
user2
user3


2. Open command prompt as administrator, run the following command:

for /f %i in (users.txt) do dsquery user -samid %i | dsrm -noprompt

Django - Create a form with read-only field

To Create a read-only field for a form in Django:


somefield = forms.CharField( widget=forms.TextInput(attrs={'readonly':'readonly'}))

Example: 

class AddStudentAccountForm(forms.Form):
         first_name = forms.CharField(max_length=100)
         last_name = forms.CharField(max_length=100)
         unique_id = forms.CharField(max_length=10)
description = forms.CharField(max_length=255, initial='Students <Grad Year>', widget=forms.TextInput(attrs={'readonly':'readonly'}))
department = forms.CharField(max_length=255, initial='SSIS Students', widget=forms.TextInput(attrs={'readonly':'readonly'}))
         grad_year = forms.ChoiceField(choices=YEARS)

Windows Active Directory Domain Services - All "dsadd" parameters

"dsadd" is a command line tool to add an account into the MS Windows Active Directory :

dsadd user <UserDN> [-samid <SAMName>] [-upn <UPN>] [-fn <FirstName>] [-mi <Initial>] \ [-ln <LastName>] [-display <DisplayName>] [-empid <EmployeeID>] [-pwd {<Password> | *}] \ [-desc <Description>] [-memberof <Group> ...] [-office <Office>] [-tel <PhoneNumber>] \ [-email <Email>] [-hometel <HomePhoneNumber>] [-pager <PagerNumber>] [-mobile <CellPhoneNumber>] \ [-fax <FaxNumber>] [-iptel <IPPhoneNumber>] [-webpg <WebPage>] [-title <Title>] \ [-dept <Department>] [-company <Company>] [-mgr <Manager>] [-hmdir <HomeDirectory>] \ [-hmdrv <DriveLetter>:][-profile <ProfilePath>] [-loscr <ScriptPath>] [-mustchpwd {yes | no}] \ [-canchpwd {yes | no}] [-reversiblepwd {ye…

Google Drive API - Rename a file

We can use the patch function of Google Drive's Files API to rename a file / folder:


def rename_file(service, file_id, new_title):
    """Rename a file.

    Args:
    service: Drive API service instance.
    file_id: ID of the file to rename.
    new_title: New title for the file.
    Returns:
    Updated file metadata if successful, None otherwise.
    """
    try:
        file = {'title': new_title}

        # Rename the file.
        updated_file = service.files().patch(
                            fileId=file_id,
                            body=file,
                            fields='title').execute()

        return updated_file
    except errors.HttpError, error:
        print 'An error occurred: %s' % error

    return None


References: https://developers.google.com/drive/v2/reference/files/patch

Python - A hack to process a long list of data faster

I've learned a hack to process a long list of data faster from a colleague. It's just great. Here is the case:


Assuming I have list of thousands of users info to send email and do some other stuffs with those users.

1. First, when I loop through each user, translate the username to number with:

def str_to_num(input_str):
    return_num = 0
    for ch in input_str:
        return_num += ord(ch)

    return return_num



2. Get the result of:

num = translated_username % 10


3. If the result of the previous calculation is in the input condition (when I run the script), then execute the main function (email,...)


if num in condition_number:
      main_function()



The main function:

...

def get_dict_data_from_csv_file(csv_file_path):
    csv_file = open(csv_file_path, 'rb')
    csv_file.seek(0)
    sniffdialect = csv.Sniffer().sniff(csv_file.read(10000), delimiters='\t,;')
    csv_file.seek(0)
    dict_reader = csv.DictReader(csv_file, dialect=sniffdialect)
    csv_file.seek(0)
    dict_d…

Google Drive API - Search only your own files

Today, when I tried to search all the files of a user in the domain, I found some files that have no parents. I figured out that those files are shared files from another users. So, to search all files that belong to me I 've just use the normal search function (here), and then remove files which have no parents out of the results:


def remove_no_parent_files(files):
    tmp = files
    for file in files:
        if not file['parents']:
            tmp.remove(file)
    return tmp



Update 12/06/2013: Just verify this method, it's wrong (files of a shared folder has parents too). So, user 'owners' attribute instead to check the owners is the authenticated user:


def retrieve_own_files(service):
    tmp = []
    allfiles = search_files(service, "trashed = false")
    if allfiles:
        for file in allfiles:
            if file['owners'][0]['isAuthenticatedUser']:
                tmp.append()

    return tmp


This is how I solve the issue. Maybe there are m…

Google Drive API - Search file in root folder

To search for files in root folder, using the following query:

query = "'root' in parents"

and then you can search with this function:

def search_files(service, query_string):
    result = []
    page_token = None
    while True:
        try:
            param = {}
            if page_token:
                param['pageToken'] = page_token
            param['q'] = query_string
            files = service.files().list(**param).execute()

            result.extend(files['items'])
            page_token = files.get('nextPageToken')
            if not page_token:
                break
        except errors.HttpError, error:
            print 'An error occurred: %s' % error
            break

    return result

Google Drive API - Copy folder

There is no usual way to copy a folder in Google Drive using the API. But, We can still create folders and have the copy file api. So, following these steps we can copy folders:


1. Insert a new folder
2. Make a copy of all files inside the source folder
3. Assign the new folder as the parent of the copied files
4. For sub folders, use recursion to copy.



I've implemented that flow:


from apiclient import errors

def copy_file(service, origin_file_id, copy_title, parentid=None):
    """Copy an existing file.

    Args:
        service: Drive API service instance.
        origin_file_id: ID of the origin file to copy.
        copy_title: Title of the copy.

    Returns:
        The copied file if successful, None otherwise.
    """
    copied_file = {'title': copy_title}
    if parentid:
        copied_file['parents'] = [{'id': parentid}]
    try:
        file = service.files().copy(
                    fileId=origin_file_id, body=copied_file).execut…

Google Drive API - How to work with the domain-wide service account

Image
I was trying to use the service account of my app to access to all documents of all users in my domain following this instruction:

https://developers.google.com/drive/delegation


But it didn't work.


This afternoon, my college showed me the old interface of the Google API Console which is different from the above one. So, I tried to create a service account using this page:

https://code.google.com/apis/console/b/0/?noredirect


The old interface allows me to create a service account which results in these following information:

+ A private key file (to download right after I created the service account)
+ Client ID
+ Email adress



Next, I will add the above Client ID to the Admin Console of my domain admin interface, and assign google drive api access for my app's service account with scope:

https://www.googleapis.com/auth/drive


After that, I can use the service account (private key and email address) to access to all the documents of all users in my domain.

import httplib2

from api…

Fix error "openssl/ssl.h: No such file or directory"

If you got this error message when installing some packages such as pyopenssl or pycrypto: “openssl/ssl.h: No such file or directory”


Install these following packages will fix:

$ sudo apt-get install libssl-dev

Fix error: "fatal error: Python.h: No such file or directory"

To fix the error: "fatal error: Python.h: No such file or directory"

install the python development packages:

$ sudo apt-get install python-dev

Google Drive API - To use SignedJwtAssertionCredentials function

In order to use this function:

SignedJwtAssertionCredentials
of the google-api-python-client, you have to install pyopenssl modules:

(env)$ pip install pyopenssl


For more information about SignedJwtAssertionCredentials, please read: http://google-api-python-client.googlecode.com/hg/docs/epy/oauth2client.client.SignedJwtAssertionCredentials-class.html



Google Drive API - How to create a folder

Creating a file in Google Drive using the API is well documented:

https://developers.google.com/drive/v2/reference/files/insert

But, I can't find anywhere in that Google Drive SDK documentations a way to create a folder. After reading carefully the docs, I see that a folder is just a file in Google Drive. The different here is the mimeType. The mimeType of a folder is:

'application/vnd.google-apps.folder'


So, I wrote a function to create a folder in an authorized drive account, based on the docs example:


def create_a_folder(service, title, desc, parentid=None):
    body = {
        'title': title,
        'description': desc,
        'mimeType': 'application/vnd.google-apps.folder'
    }
    if parentid:
        body['parents'] = [{'id': parentid}]

    folder = service.files().insert(body=body).execute()
    pprint.pprint(folder)

    return folder['id']


"service" parameter is a drive service instance which can be retriev…

Google Drive API - Feeling the openness of Google

I'm currently working on a project that need to migrate all my organization documents (of all users) which hosted on Google App to a new domain (also on Google App)

So, the process is:

1. Using a Service Account of the old domain to read all files and folders of all user in the old domain along with their current sharing permissions.
2. For each file / folder, share with the relevant new accounts in the new domain.
3. In the new domain, make a copy of each shared file / folder by another Service Account of this new domain.
4. Apply the old sharing permissions to the same file in the new domain.
5. In the old domain, remove shared permissions to new domain accounts.



Service Account is a special account used by the application (this) with the domain-wide privileges. It can manipulate all files and folders of all accounts in the domain. (https://developers.google.com/drive/delegation)


I've just played around with the api documentations and have some utility functions. The job have …