Skip to content

Setup

Jump to bottom
StephenChan edited this page Apr 21, 2013 · 53 revisions

This page is about setting up your own machine to develop and run the CoralNet project (or a fork of the project).

Contents

Git and GitHub

Setting up Git and GitHub

Follow GitHub's guide for setting up Git and GitHub. Make sure to select your operating system at the top of the page.

  • As of 2012 October 5, the top of that guide suggests using GitHub's native app for setting up and using Git and GitHub. This hasn't yet been tried by anyone on the CoralNet team, but you're welcome to try it out. It's available for Windows and Mac.

Contributing

  • Core collaborators, who can push directly to the main CoralNet repository, can only be added manually by the repository admin. In general, we don't accept requests for core team members.
  • Anyone is welcome to fork the CoralNet repository and submit pull requests. For more information, see here.
  • Lighter ways of contributing to the project include creating Issues and submitting patches.

Getting the code repository onto your machine

  • Run a git clone command on the repository you're going to work on. For example, for the main CoralNet project, it's git clone https://github.com/DevangS/CoralNet.git.

MySQL

Setup guide on the MySQL site

Download

  • Here
  • We use MySQL 5.1 (as of 2012 October 5). All CoralNet developers should use the same version of MySQL.
  • Some of the Windows packages are labelled as "Essentials" and some aren't (Essentials is a smaller package). Essentials will suffice for CoralNet development.
  • If your have a 32-bit machine, get 32-bit MySQL. If you have a 64-bit machine, either 64 or 32 bit MySQL should be fine. 32 vs. 64 bit shouldn't raise any compatibility issues between developers.
  • After picking a package to download, skip the registration stuff; just scroll down and click "No thanks, just start my download!".

Database server setup

There's a setup wizard for Windows, and other setup methods for other OSes. Whichever way you install, make sure you do these things:

  • Enable InnoDB tables. It's very important that the database tables support transactions.
  • Enable TCP/IP networking. This will make it possible (or at least much easier) for Django to connect to MySQL.
  • When you set the root password, remember what that password is.

Create a database and a user

  • Now that you've installed and configured MySQL, start the MySQL Server command line.
  • You'll be asked for the root password, so enter the root password. Then you'll be able to start entering MySQL commands.
  • Create a database to use for your development server. If you wanted to name the database coralnet, then use: CREATE DATABASE coralnet;
  • Create a user for Django to log in as to access the database. If you wanted the username usernamehere and the password passwordhere, then use: CREATE USER 'usernamehere'@'localhost' IDENTIFIED BY 'passwordhere';
  • Give that user permission to do anything with the database: GRANT ALL ON coralnet.* TO 'usernamehere'@'localhost';
  • When running Django unit tests, a separate test database is created, used for the duration of the test, and then destroyed after tests are finished. That test database has the same name as your main database, except it's prefixed with "test_". You'll want to grant permission to do anything with this test database, too: GRANT ALL ON test_coralnet.* TO 'usernamehere'@'localhost';

Python

Install Python

  • CoralNet uses Python 2.7.1 (as of 2012 October 6). Anything 2.7.x should be fine for development, so if you already have Python and the version number is 2.7.0 through 2.7.3, then use that.
  • 2.6.x could run into issues due to not supporting everything that 2.7.x has. For example, 2.6.x doesn't have the argparse module, and doesn't have the curly-brace set literal syntax.
  • 3.x should be avoided for now, as some common functions will not work in 3 the same way it did in 2. See here for examples of Python 2 vs. 3 differences. Notable differences include print 'hello' becoming print('hello'), and dict.keys() returning a "view" instead of a list. Another reason to avoid 3.x for now is that some of the Python packages we use may outright refuse to install when seeing your Python is 3.x, because the package was coded for 2.x.
  • Note that you may have multiple versions of Python on the same machine without problems. For example, on Windows you can have C:\Python26 and C:\Python27 side by side, or perhaps usr/lib/python27 and usr/lib/python33 on Linux. Just be sure to set your system path so that when you call python, easy_install, pip, etc., you're actually referring to the Python installation that you think you are.
  • For Windows, if your have a 32-bit machine, get 32-bit Python. If you have a 64-bit machine, it's recommended to still get 32-bit Python. 64 bit is workable, but can make things tricky; also, certain Python utilities like PyYAML don't seem to have 64-bit Windows binaries.

Python packages

Packages to download and install manually

  • pip
  • MySQL-Python adapter
    • Install from source (.tar.gz), or via Windows binary. If from a Windows binary, make sure to get 32 bit if your Python installation is 32 bit, or 64 bit if your Python is 64 bit. Note for OS X 10.8: the installation path is wrong for Mountain Lion. You'll have to resolve the path for where your installation of mySQL is in order for python to see it.
  • PyYAML
    • As of 2012 October 7, this is required to run most of the Django unit tests, because the fixtures (which specify initial database data) are written in YAML.
  • PIL (Python Imaging Library)
    • On Windows, you can just download the binary of the latest version, found in the link above.
    • On Mac OS X (10.8 Mountain Lion): You will need to install PIL and libjpeg. Install homebrew and install libjpeg BEFORE installing pil. Once that is finished, run pip install pil. If you see the confirmation that jpeg files are supported, then the installation worked. If not, uninstall pil and try again.
    • On Linux, you might have better luck installing Pillow instead of plain PIL.
  • NumPy

Virtualenv (optional)

virtualenv - if you have other Python projects that may require different versions of the same packages that CoralNet uses, then you should strongly consider using virtualenv. It will allow you to keep separate sets of Python packages under separate directories.

  • You can download virtualenv using pip (described below).
  • Create an env for CoralNet, and be sure to activate that env before pip-installing any packages that you will use only for CoralNet.
  • For packages that can't be pip-installed, first install the package normally, into your global Python installation (e.g. something like C:\Python27 on Windows). Then go to your global Python installation's Lib/site-packages directory and locate the package's directory (and .pth file, if any). Copy these into the CoralNet env's Lib/site-packages directory. This should "install" the packages into that env.
  • If you use the PyCharm IDE (described later), you may have to:
    • Point it toward the Scripts/python.exe of your CoralNet env. You would set this in Settings -> Project Settings -> Project Interpreter -> Python Interpreters.
    • Manually make Run/Debug configurations to make PyCharm find and run your unit tests.

Packages to download and install with pip

Read abot using pip - here are some commonly useful commands:

  • Install the latest version of my-package: pip install my-package
  • Upgrade an existing installation: pip install -U my-package
  • Upgrade an installation, without touching dependency packages: pip install -U --no-deps my-package
  • Install version 1.2.3 of my-package: pip install my-package==1.2.3
  • Install the latest version of my-package that's less than 1.2: pip install my-package<1.2
  • Uninstall my-package: pip uninstall my-package

For CoralNet, get the following:

  • django==1.3.0
  • django-userena==1.0.1
    • This will probably auto-install django-guardian and easy_thumbnails (listed below) as dependencies. This is fine.
  • django-guardian
  • easy_thumbnails
  • South
  • django-reversion==1.5.0
    • This should be updated if django is updated. See here.
  • django-sentry<1.13
    • Make sure you install 'django-sentry', not 'sentry'.
    • 1.13 and above obsolete sentry.client, which we use, hence the <1.13.
    • If < is a special character in your shell, try: pip install "django-sentry<1.13"
  • django-dajaxice==0.2
  • django-dajax==0.8.4
  • django-celery
  • GChartWrapper
  • django-supervisor

Notes:

  • Many of the above packages will also auto-install other packages as dependencies, particularly django-sentry.
  • On Windows, if you get an error saying "Unable to find vcvarsall.bat", try this workaround. (Yes, unfortunately, it requires installing MinGW if you don't have it already.)

Development Environment

Here are some recommended programs to use for Django coding.

PyCharm IDE

Get it here.

It's not freeware (although you can get a 30-day free trial to try it out). See here for licensing options. In particular:

  • If you're a student, you may be able to apply for a Classroom License for free, or an Academic License for $29. Or, check your school's computing services (or similar) to see if they already have a PyCharm license available for students to use.
  • You could just get a Personal License for $99.
  • As of 2012 October 5, we're not sure if CoralNet developers could get a free Open Source License, but it's something we should look into later.

Pros:

  • Great support for Django, such as server debugging, template syntax highlighting, relevant code suggestions, and generally understanding the structure of a Django project.
  • Syntax highlighting and code suggestions for client-side languages such as Javascript and CSS.
  • Integration with version control, particularly Git, and a very good Diff tool.

Cons:

  • Uses a lot of resources, so can be slow on less powerful machines (such as a netbook with an Atom processor and 2 GB memory).
  • Code suggestions are occasionally a bit overzealous. However, the kinds of suggestions can be turned off individually.

Other IDEs

Wingware, Komodo IDE, Aptana Studio, PyScripter

Text Editors

Komodo Edit, Notepad++, JEdit, Vim, Emacs, Sublime Text

  • Just make sure you have these configured so that Python indentation isn't a chore. Python doesn't like tab characters, and standard indentation is 4 spaces per indent. Here is an example of a possible Vim setup for working with Python.

Final setup steps for Django

Required files that aren't in the Git repository

The following files aren't in the Git repository, but must be created on each machine running CoralNet:

  • settings_2.py: Create this on the project root level, and use settings_2_example.py as an example. This file is for Django settings that should differ between the production server and development machines, which is why it's not in the repository.
  • images/task_helpers.py: Use images/task_helpers_example.py as an example. This is for calling system-specific external scripts/programs to perform certain asynchronous tasks.
  • templates/maintenance_notice.html: Use templates/maintenance_notice_example.html as an example. This is for putting up a "under maintenance" message at the top of the website. maintenance_notice.html will be changed whenever site maintenance begins or ends, and these changes shouldn't be considered code updates, so the file isn't in the repository.
  • Some directories and their sub-directories may need to be created prior to Django creating files under them. These directories include PROCESSING_ROOT, TEST_PROCESSING_ROOT, MEDIA_ROOT, TEST_MEDIA_ROOT, and STATIC_ROOT. An example of sub-directories that may need to be created: data/original/ under MEDIA_ROOT and TEST_MEDIA_ROOT. You'll know later if you need to create these directories if Django gets a "permission denied" error when trying to create files under them.

Creating the database tables

  • Make sure you've installed South (the Python package) before proceeding.
  • Make sure that in settings_2.py, the DATABASES setting's NAME, USER, and PASSWORD correspond to the database name, username, and user password that you created when configuring MySQL. Using the example given in the MySQL setup section, NAME would be 'coralnet', USER would be 'usernamehere', and PASSWORD would be 'passwordhere'. If you don't configure DATABASES correctly, you'll probably get some kind of "permission denied" error on the following steps.
  • From the project root directory, run python manage.py syncdb to create the database tables that aren't under South control.
    • At this point, you'll probably be asked to create a superuser, which is a website user with all available privileges. Create one, and write down the username and password. Later, you can log in to your website as this superuser.
  • Then run python manage.py migrate to create the database tables that are under South control. Note that this is done by creating some initial tables and then applying a bunch of changes to them (these changes corresponded to how CoralNet's database tables changed over time), so it can take a little while to complete.

Collecting static files

From the project root directory, run python manage.py collectstatic and reply 'yes' to the prompt. This will collect static files (CSS, Javascript, etc.) to the STATIC_ROOT directory. When DEBUG is False (this includes all runs of unit tests), static files must be collected to this directory in order to be served to the web browser.

For more info, see Django notes - Media and static files.

Running

Try running the server.

  • From the project root directory, run python manage.py runserver and wait for the server to start. (Or run the server in your IDE.)
  • Visit http://127.0.0.1:8000/ in a web browser. If things work, that should take you to the website homepage.
  • Browse the website to see if the various pages work.
  • Try creating labels, sources, labelsets, uploading images, and annotating images.

Try running some unit tests.

Celery

TODO

  • Setting up Celery probably won't be necessary immediately. It'll only be necessary if you are working on anything related to asynchronous tasks.

Clone this wiki locally