Posts

Showing posts from August, 2014

PHP cURL with HTTPS

When using PHP cURL against a SSL powered website, you need to add this line (the red one) before calling curl_exec:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://my.domain.com');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'mycookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'mycookie.txt');
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$buffer = curl_exec($ch);


(Notes that this is just a workaround, for proper fix please read this article: http://unitstep.net/blog/2009/05/05/using-curl-in-php-to-access-https-ssltls-protected-sites/)

Mass reset ActiveDirectory accounts password using the "LTB Self Service Password"

LTB Self Service Password is a small web applications written in php to delegate administrative tasks to users. It is a part of the LTB project (http://ltb-project.org/wiki/). The main functionality of this app is to help the users to self-reset their own Active Directory account's password, but you can leverage its utility functions (php script) to access the Active Directory service.

One of the advantages of using the LTB Self Service Password functions is mass resetting a group of AD accounts's password. Here is the working snippet (I used Python to load the accounts list because of the easiness of the csv module):

Firstly, make sure that you have installed php5-ldap lib:

$ sudo apt-get install php5-ldap


massreset.php



wrapper.py



accounts.csv

student_number,lastname,firstname,username
123,Super, Cool,username1
456,Ultra,Awesome,username2
789,Wonder,Who,username3


So, to mass reset password of all the users in accounts.csv, run the following command:

$ python wrapper.py



How to fix RSS Feed parsing error in Wordpress

From last night, whenever I tried to access the feed url of my Wordpress blog, I saw this error:

This page contains the following errors: error on line 12 at column 49: Entity 'ndash' not defined Below is a rendering of the page up to the first error.
So, here is how I fixed it:

1. First, go to the online feed validator tool to see what was wrong:

http://feedvalidator.org/

Enter my blog feed url and see the result. It turned out that there is something wrong with the feed title tag. It includes a strange character called –

2. Go to the template's include folder, open the template file which defines the feed's title and remove that character. I'm using genbu theme, so I will edit genbu/includes/tamatebako.php, the  tamatebako_wp_title function as following line:

...
        /* Add Site Description */
        if( is_front_page() ){
                if ( $site_description ){
                        $doctitle = "{$doctitle} {$site_description}";
           …

Why the Celery task was not executed in my Django app

Yesterday, after deploying my Django apps stack to a new Ubuntu server (14.04 from 12.04), I found that the Celery task was not executed. What happened?

Finally, I managed to make it work again following these changes:

1. Upgrade these python modules to the latest version (come with Ubuntu 14.04): Django 1.6.6, django-celery 3.1.10, celery 3.1.13, librabbitmq 1.5.2 (to avoid warning about rabbitmq compatibility)

2. Add these CELERY settings (in red color) in my Django app's settings.py:

...
CELERY_IMPORTS = ("myapp.tasks",)
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
CELERY_ACCEPT_CONTENT = ['pickle', 'json'] # new in celery 3.1
CELERY_TRACK_STARTED = True
...

(remember to restart the celery beat after making those changes)

SQLAlchemy's equivalent to Django's auto_now, auto_now_add

To define a datetime column table in SQLAlchemy with auto_now or auto_now_add default value which are used in Django models:


from sqlalchemy import Column, DateTime
import datetime

...
first_created = Column(DateTime(), default=datetime.datetime.now)

last_modified = Column(DateTime(), onupdate=datetime.datetime.now)
...


Run celery as a daemon using root user in Ubuntu

In the latest stable version of celery (3.1.13), you cannot run a celery beat as a daemon (I used supervisord) using root user without a special environment variable. That special variable is C_FORCE_ROOT. So, in your supervisord configuration file, add this:

...
environment=C_FORCE_ROOT="yes"
...
This is not encouraged but in some cases like mine, I just need the system works, and avoiding any extra configuration.

How to solve the djcelery's "DatabaseError: no such table: myapp.celery_taskmeta" error

When I executed a celery task in my django project (South installed), I got this error:

...
Programming Error ... DatabaseError: no such table: myapp.celery_taskmeta...
...

The problem here is actually that South manages the djcelery tables. So, I need to migrate djcelery to it's new schema

python manage.py migrate djcelery 0001 --fake
python manage.py migrate djcelery

python manage.py migrate djcelery 0002 --fake
python manage.py migrate djcelery

...

(try to do the fake migration until the error disappears. I had to do it 4 times (0004) in my case)


Reference: http://stackoverflow.com/questions/6959702/why-are-celery-taskmeta-and-other-tables-not-being-created-when-running-a-syncdb

How to be able to send form data with the '@' character at the beginning of the data string in PHP cURL

If you post form data using cURL in php, you cannot use the '@' character at the beginning of a data string value (e.g: password=@@mypassword). '@' is the indicator for file path in POST operation of cURL. You need to disable the '@' prefix to be able to send data string start with '@' character.

Here is the description of PHP cURL's CURLOPT_POSTFIELDS option:
CURLOPT_POSTFIELDS

The full data to post in a HTTP "POST" operation. To post a file, prepend a filename with @ and use the full path. The filetype can be explicitly specified by following the filename with the type in the format ';type=mimetype'. This parameter can either be passed as a urlencoded string like 'para1=val1&para2=val2&...' or as an array with the field name as key and field data as value. If value is an array, the Content-Type header will be set to multipart/form-data. As of PHP 5.2.0, value must be an array if files are passed to this option with…

Avoid crontabs overlapping in Ubuntu using run-one

Last Saturday, I created a crontab to execute a long running script which supposed to run every 45 minutes. I went home and realized I got hundreds of error messages notifications in my mailbox. What's the problem? The script passed the test and everything was OK back then.

The next day, I had to go to myh office and investigated the situation. There were 5 instances of the script running at the same time. What? What the hell was going on with the shitty cron? The server which my script connects to does not allow multiple session of the same account. So, that was why I got a lot of error messages. But, why were there 5 running tasks? It was because the script needs more than 45 minutes to complete. And then after 45 minutes, when the first task hadden't finished yet, the second task was execute, and so on...

So, to solve this issue, I need to find a way to make sure that there is only one instance of the script running at a time. There are many ways to do that, such as: implem…

Get all parent accounts from PowerSchool using PHP

You can use this php snippet to get all the parent accounts from your PowerSchool SIS's Oracle database:




$ php get_parent_accounts.php

edx-platform - Default ElasticSearch configuration caused me trouble

A couple weeks ago, our OpenEdX instance's network interface which was deployed in a DigitalOcean's VPS was blocked by the company. They said that there was a large flood of traffic coming out from our server that was disrupting the normal traffic flow for other users. We were still able to ssh to the server using the web interface of DO to troubleshoot the issue but It was really annoying. I tried many things such as turn off or reconfigure services of the OpenEdX instance but it seemed like the problem was still there and our server was still blocked.

After a while, the DO engineer suggested us to check our ElasticSearch configuration. There is a vulnerability of ElasticSearch if we use the default configuration:

http://bouk.co/blog/elasticsearch-rce/#how_to_secure_against_this_vulnerability

I configure our ElasticSearch service as the article suggested, add these two line in /etc/elasticsearch/elasticsearch.yml:

1. Disable dynamic scripting and prevent remote code execution:
s…

edx-platform - Strange error when generating certificates

I got this strange error when trying to generating certificates for all students in a course:

$: sudo -u www-data /edx/bin/python.edxapp ./manage.py lms --settings aws ungenerated_certs -c UNS/Py_001/201405 --insecure

2014-08-03 22:02:20,229 INFO 21338 [dd.dogapi] dog_stats_api.py:66 - Initializing dog api to use statsd: localhost, 8125
Fetching enrolled students for UNS/Py_001/201405
2014-08-03 22:02:21,494 ERROR 21338 [xmodule.x_module] x_module.py:265 - Unable to load item i4x://UNS/Py_001/vertical/8401ba9a15ee4da098ab580680e3d544, skipping
Traceback (most recent call last):
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 262, in get_children
    child = self.runtime.get_block(child_loc)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/x_module.py", line 1041, in get_block
    return self.load_item(usage_id)
  File "/edx/app/edxapp/edx-platform/common/lib/xmodule/xmodule/modulestore/mongo/base.py", line 162, in l…