How to install django-celery in windows (python, django, celery, erlang, rabbitmq)

A project of mine goes through a realistic simulation that doesn't take too long (15 seconds), but that, unfortunately is about 15 seconds too long. So I had to find a workaround for it. Threads was the first thing that came to mind, but Python is worthless for this....

Anyhow, let's get straight to the point:

***************************************
Install erlang

Install RabbitMQ

See: http://www.rabbitmq.com/install-windows.html

pip install kombu

pip install celery

pip install django-celery

the above commands might cause errors for sub-installation of kombu and billiard. If this occurs:

install MinGQ

in the directory for billiard (ex: C:\Users\YOUR_USER_NAME\build\billiard)

run: setup.py install build --compiler=mingw32

See: http://blog.eddsn.com/2010/05/unable-to-find-vcvarsall-bat/

then rerun:

pip install kombu

pip install celery

pip install django-celery

then update settings.py according to following link and run SyncDB:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

***************************************

Credits: I have Dan (http://www.linkedin.com/pub/dan-ziembienski/17/720/679) to thank for this.

Scala 2.9.2, Play! 2.0.2, and Amazon EC2

Summary: I describe how to setup a domain on amazon's EC2 (using my personal named server), and the tricks on getting scala + play (2.0.2) to run on a micro instance.

---

Introduction: So recently I got an iphone and while out and about, I wanted to look at the source code of a page on safari. It so happens that I can't do that. (Yes, I know, I took it for granted.) So while there are some apps out there, I refuse to buy an app that I'll use only once. As a matter of luck, it so happened that this weekend there was a "$.99 .com" sale, so I hastily grabbed "www.seesster.com", and considered building a site that "sees what your browser sees".

It so happens that I've been been reading about Scala as well as the Play framework, and Friday night started to code seesster up. I won't go into details of the Scala/Play code and what I'm doing (long story short: scala + phantomjs + pygments), but I will describe how I was able to set this up (running!) on my free amazon EC2 micro instance.

DNS and sub-domains: So when I login to my AWS management console, there are two things that need to be done:

1) I need to name my instance with the exact subdomain I am planning on redirecting the service to. In this case, I've named the instance www.seesster.com

2) I also need to jot down the Public DNS (which I'll use for the CNAME entry of my named config file). In this case, it is ec2-174-129-174-216.compute-1.amazonaws.com

I've highlighted the entries I'm talking about in red....

Next up, I need to update the config file. While I won't show you the whole file, as long as you know *anything* about setting up your own DNS server, this should suffice... In a nutshell, we need to setup a CNAME entry that points to the public DNS. Note the "." after the url string.

www    CNAME    ec2-174-129-174-216.compute-1.amazonaws.com.

At this point, when I tried "www.seesster.com" I would correctly end up on amazon's server. However, this was the easiest part of my weekend.

Play won't run: My coding environment has plenty of memory, so I never thought once on how this could impact me. That is, until I tried to run play on it... "play run", "play dist", "play start" and even "play compile" would end up with memory allocation problems. I was between a rock and a hard place.

Solving this involved:

1) Manually downloading and installing java 1.7 (jdk1.7.0_05).
2) Updating the symbolic links:

$ ll /usr/lib/jvm/jre
lrwxrwxrwx 1 root root 31 Jul  2 03:09 /usr/lib/jvm/jre -> /home/ec2-user/jdk1.7.0_05/jre/

$ ll /usr/bin/java
lrwxrwxrwx 1 root root 35 Jul  2 03:10 /usr/bin/java -> /home/ec2-user/jdk1.7.0_05/bin/java

3) on my dev box, running "play dist" and uploading that stand-alone file to amazon
4) updating the "start" script in the stand-alone folder to:

$ cat start
#!/usr/bin/env sh

exec java -Xms64M -Xmx128M -Dhttp.port=80 $* -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`

What this allows me to do is run my play application with limited memory usage on port 80. (Reminder: this is on Play 2.0.2).

Happy ending: Cool or not cool? As of today, www.seesster.com is running and working. I will be updating code during the week for sure. If you are reading this in the future (wow!), and the site doesn't work, I am probably aware of it. After all, a year from now the domain will expire. And unless it is making me money, I'll probably *not* renew it.

:)

mysqldump with wildcard

Today I had to dump an existing database, but only wanted a subset of tables. Currently, mysql does not have a built-in option to do this, but with awk I got this to work:

1) filter out the table you are looking for. connect to you mysql server and test you wildcard combination

> mysql -u user -p test_db
> password: **********
> show tables like 'iesna%'

2) now combine that with awk and mysqldump command

> mysqldump -u user -p test_db 'mysql -DN test_db -e "show tables like 'iesna%'" | awk '{printf $1" "}''

:)

Install MySQL for Python (MySQLdb) on Windows 7

I loved it how people praise python as being so simple, but lack to explain the fact that they either download pre-compiled packages and libraries, and/or use linux. Lackluster is what comes to mind when I refuse to download this (http://code.google.com/p/soemin/downloads/detail?name=MySQL-python-1.2.3.win32-py2.7.exe) and rather try to compile from scratch on windows 7 (using python 2.7, mysql community 5.5).

Given that I've put in more than 15 minutes to get this rolling, I thought I would blog about it. There are other options online, but nothing better than documenting your failures, right? :)

1) First step is installing mysql *with* the development tools. Also the Mysql Connector C (http://dev.mysql.com/downloads/connector/c/).
2) Next up is installation of the setuptools (http://pypi.python.org/pypi/setuptools). And while there is a windows executable, that didn't work for me. So download the ez_setup.py and it should download/update your lib.
3) Check if your site.cfg file has the correct registry key for your version of mysql server (registry_key = SOFTWARE\MySQL AB\MySQL Server 5.5).
4) Also double-check if your mingw path is updated in your environment variables (computer >> properties >> advanced system settings >> environment variables >> ADD to your path "C:\MinGW\bin")
5) You need to comment out the lines regarding to the static build in setup_windows.py, as well as instructing the compiler to use the dynamic lib version (add 'client = "libmysql"').

    # XXX static doesn't actually do anything on Windows
    #if enabled(options, 'embedded'): B
    #    client = "mysqld"
    #else:
    #    client = "mysqlclient"
    client = "libmysql"

6) If you are lucky, running python setup.py build –compile=mingw32 will work. I wasn't as lucky since I got a missing 'config-win.h'. (Check this out: http://stackoverflow.com/questions/1972259/mysql-python-install-problem-using-virtualenv-windows-pip).