How to install django-celery in windows (python, django, celery, erlang, rabbitmq)

A project of mine goes through a realistic simulation that doesn't take too long (15 seconds), but that, unfortunately is about 15 seconds too long. So I had to find a workaround for it. Threads was the first thing that came to mind, but Python is worthless for this....

Anyhow, let's get straight to the point:

***************************************
Install erlang

Install RabbitMQ

See: http://www.rabbitmq.com/install-windows.html

pip install kombu

pip install celery

pip install django-celery

the above commands might cause errors for sub-installation of kombu and billiard. If this occurs:

install MinGQ

in the directory for billiard (ex: C:\Users\YOUR_USER_NAME\build\billiard)

run: setup.py install build --compiler=mingw32

See: http://blog.eddsn.com/2010/05/unable-to-find-vcvarsall-bat/

then rerun:

pip install kombu

pip install celery

pip install django-celery

then update settings.py according to following link and run SyncDB:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

***************************************

Credits: I have Dan (http://www.linkedin.com/pub/dan-ziembienski/17/720/679) to thank for this.

Scala 2.9.2, Play! 2.0.2, and Amazon EC2

Summary: I describe how to setup a domain on amazon's EC2 (using my personal named server), and the tricks on getting scala + play (2.0.2) to run on a micro instance.

---

Introduction: So recently I got an iphone and while out and about, I wanted to look at the source code of a page on safari. It so happens that I can't do that. (Yes, I know, I took it for granted.) So while there are some apps out there, I refuse to buy an app that I'll use only once. As a matter of luck, it so happened that this weekend there was a "$.99 .com" sale, so I hastily grabbed "www.seesster.com", and considered building a site that "sees what your browser sees".

It so happens that I've been been reading about Scala as well as the Play framework, and Friday night started to code seesster up. I won't go into details of the Scala/Play code and what I'm doing (long story short: scala + phantomjs + pygments), but I will describe how I was able to set this up (running!) on my free amazon EC2 micro instance.

DNS and sub-domains: So when I login to my AWS management console, there are two things that need to be done:

1) I need to name my instance with the exact subdomain I am planning on redirecting the service to. In this case, I've named the instance www.seesster.com

2) I also need to jot down the Public DNS (which I'll use for the CNAME entry of my named config file). In this case, it is ec2-174-129-174-216.compute-1.amazonaws.com

I've highlighted the entries I'm talking about in red....

Next up, I need to update the config file. While I won't show you the whole file, as long as you know *anything* about setting up your own DNS server, this should suffice... In a nutshell, we need to setup a CNAME entry that points to the public DNS. Note the "." after the url string.

www    CNAME    ec2-174-129-174-216.compute-1.amazonaws.com.

At this point, when I tried "www.seesster.com" I would correctly end up on amazon's server. However, this was the easiest part of my weekend.

Play won't run: My coding environment has plenty of memory, so I never thought once on how this could impact me. That is, until I tried to run play on it... "play run", "play dist", "play start" and even "play compile" would end up with memory allocation problems. I was between a rock and a hard place.

Solving this involved:

1) Manually downloading and installing java 1.7 (jdk1.7.0_05).
2) Updating the symbolic links:

$ ll /usr/lib/jvm/jre
lrwxrwxrwx 1 root root 31 Jul  2 03:09 /usr/lib/jvm/jre -> /home/ec2-user/jdk1.7.0_05/jre/

$ ll /usr/bin/java
lrwxrwxrwx 1 root root 35 Jul  2 03:10 /usr/bin/java -> /home/ec2-user/jdk1.7.0_05/bin/java

3) on my dev box, running "play dist" and uploading that stand-alone file to amazon
4) updating the "start" script in the stand-alone folder to:

$ cat start
#!/usr/bin/env sh

exec java -Xms64M -Xmx128M -Dhttp.port=80 $* -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`

What this allows me to do is run my play application with limited memory usage on port 80. (Reminder: this is on Play 2.0.2).

Happy ending: Cool or not cool? As of today, www.seesster.com is running and working. I will be updating code during the week for sure. If you are reading this in the future (wow!), and the site doesn't work, I am probably aware of it. After all, a year from now the domain will expire. And unless it is making me money, I'll probably *not* renew it.

:)

mysqldump with wildcard

Today I had to dump an existing database, but only wanted a subset of tables. Currently, mysql does not have a built-in option to do this, but with awk I got this to work:

1) filter out the table you are looking for. connect to you mysql server and test you wildcard combination

> mysql -u user -p test_db
> password: **********
> show tables like 'iesna%'

2) now combine that with awk and mysqldump command

> mysqldump -u user -p test_db 'mysql -DN test_db -e "show tables like 'iesna%'" | awk '{printf $1" "}''

:)

Install MySQL for Python (MySQLdb) on Windows 7

I loved it how people praise python as being so simple, but lack to explain the fact that they either download pre-compiled packages and libraries, and/or use linux. Lackluster is what comes to mind when I refuse to download this (http://code.google.com/p/soemin/downloads/detail?name=MySQL-python-1.2.3.win32-py2.7.exe) and rather try to compile from scratch on windows 7 (using python 2.7, mysql community 5.5).

Given that I've put in more than 15 minutes to get this rolling, I thought I would blog about it. There are other options online, but nothing better than documenting your failures, right? :)

1) First step is installing mysql *with* the development tools. Also the Mysql Connector C (http://dev.mysql.com/downloads/connector/c/).
2) Next up is installation of the setuptools (http://pypi.python.org/pypi/setuptools). And while there is a windows executable, that didn't work for me. So download the ez_setup.py and it should download/update your lib.
3) Check if your site.cfg file has the correct registry key for your version of mysql server (registry_key = SOFTWARE\MySQL AB\MySQL Server 5.5).
4) Also double-check if your mingw path is updated in your environment variables (computer >> properties >> advanced system settings >> environment variables >> ADD to your path "C:\MinGW\bin")
5) You need to comment out the lines regarding to the static build in setup_windows.py, as well as instructing the compiler to use the dynamic lib version (add 'client = "libmysql"').

    # XXX static doesn't actually do anything on Windows
    #if enabled(options, 'embedded'): B
    #    client = "mysqld"
    #else:
    #    client = "mysqlclient"
    client = "libmysql"

6) If you are lucky, running python setup.py build –compile=mingw32 will work. I wasn't as lucky since I got a missing 'config-win.h'. (Check this out: http://stackoverflow.com/questions/1972259/mysql-python-install-problem-using-virtualenv-windows-pip).

Determining Linux Distro/Version

# Determine the Linux distribution and version that is being run.
   #
   # Check for GNU/Linux distributions
   if [ -f /etc/SuSE-release ]; then
     DISTRIBUTION="suse"
   elif [ -f /etc/UnitedLinux-release ]; then
     DISTRIBUTION="united"
  elif [ -f /etc/debian_version ]; then
    DISTRIBUTION="debian"
  elif [ -f /etc/redhat-release ]; then
    a=`grep -i 'red.*hat.*enterprise.*linux' /etc/redhat-release`
    if test $? = 0; then
      DISTRIBUTION=rhel
    else
      a=`grep -i 'red.*hat.*linux' /etc/redhat-release`
      if test $? = 0; then
        DISTRIBUTION=rh
      else
        a=`grep -i 'cern.*e.*linux' /etc/redhat-release`
        if test $? = 0; then
          DISTRIBUTION=cel
        else
          a=`grep -i 'scientific linux cern' /etc/redhat-release`
          if test $? = 0; then
            DISTRIBUTION=slc
          else
            DISTRIBUTION="unknown"
          fi
        fi
      fi
    fi
  else
    DISTRIBUTION="unknown"
  fi

  ###    VERSION=`rpm -q redhat-release | sed -e 's#redhat[-]release[-]##'`

  case ${DISTRIBUTION} in
  rh|cel|rhel)
      VERSION=`cat /etc/redhat-release | sed -e 's#[^0-9]##g' -e 's#7[0-2]#73#'`
      ;;
  slc)
      VERSION=`cat /etc/redhat-release | sed -e 's#[^0-9]##g' | cut -c1`
      ;;
  debian)
      VERSION=`cat /etc/debian_version`
      if [ ${VERSION} = "testing/unstable" ]; then
          # The debian testing/unstable version must be translated into
          # a numeric version number, but no number makes sense so just
          # remove the version all together.
          VERSION=""
      fi
      ;;
  suse)
      VERSION=`cat /etc/SuSE-release | grep 'VERSION' | sed  -e 's#[^0-9]##g'`
      ;;
  united)
      VERSION=`cat /etc/UnitedLinux-release`
      ;;
  *)
      VERSION='00'
      ;;
  esac;

  echo ${DISTRIBUTION}${VERSION}

Fork off a process and specify a timeout value for it to complete in Java

I have a personal preference to code in C/C++ or Java. However, I'll swap between any other programming language I know if I think it will solve my problem. For what it's worth, people often get frustrated when they ask me what language is the best (or worse, or most important, etc..) because I'll often ask them what they know best, and state that as my answer.

Anyhow, I've had this need to interact with a couple of programs that have no API whatsoever. My solution has been to run the suckers and grab their output. For that reason, I got to learn a bit about how to execute a command in a separate process in Java. Through a class called ProcessBuilder (java.lang package), it allows one to get the input or output stream of the subprocess, as well as the exit status (among other available information).

Quick example:

import java.io.*;
import java.util.*;
public class MyExample {
  public static void main(String args[]) throws IOException {
    if (args.length <= 0) {
      System.err.println("What should I run?");
      System.exit(-1);
    }
    Process process = new ProcessBuilder(args).start();
    InputStream is = process.getInputStream();
    InputStreamReader reader = new InputStreamReader(is);
    BufferedReader buffer = new BufferedReader(reader);
    String line;
    System.out.printf("Output of running %s is:", Arrays.toString(args));
    while ((line = buffer.readLine()) != null) {
      System.out.println(line);
    }
  }
} 

And while I won't go into details of how to customize your ProcessBuilder inputs, you can try this out with > java MyExample ls.

My problem is that the software I'm executing in the subprocess, forks other threads and processes. Which screws up the process' signaling in Java. So I actually do the process forking in another C-based software I coded, and use Java to call that one. (Complicated tongue-twister, right?) But, this requires that the Java process waits until my C code finishes, which has to wait until the other programs finishes. For that reason, I had to include a  process.waitFor(); right after the start command.

This leads to the title of this post: How to specify a time out value to the waitFor()? Apparently waitFor() has no timeout. Which means that I need to create yet another Thread just to wait for it, and join both of the after the timeout expires.

1. Thread thread = new Thread(new Runnable()  
2. {  
3.     public void run()  
4.     {  
5.         try  
6.         {  
7.             process.waitFor();  
8.         }  
9.         catch (Exception e) { e.printStackTrace(); }  
10.     }  
11. });  
12. thread.start();  
13. thread.join(15000); // 15 seconds  

It's one of those things that I'm not necessarily bragging about, but not ashamed either. And most importantly, I don't want to forget. :)

Overlay plots in R (or "Is there a command equivalent to 'hold on' of MATLAB in R?")

So used to the simple method of plotting graphs in MATLAB, I've run into a couple of quirky situations while using R. As the tile of this post describes it, the latest problem I've had was when I was trying to overlay different plots on the same graph.

In MATLAB, this is easy: we just plot, then issue a 'hold on', then plot again. Done!

In R, I've finally found a solution: between two calls to plot, I set a 'par(new=T)'. Additionally one should specify the xlim as well as the ylim=c(y1,y2) in each call to plot() to get proper matching of the overlaid plots.

Quick example (YMMV):

f1.range = range(f1.results)

f2.range = range(f2.results)

plot(f1, broken = TRUE, bcontrol = list(style = "gap"), ylim = range(f1.range, f2.range))

par(new=T)

plot(f2, ylim = range(f1.range, f2.range), pch=24)