Merge branch 'main' into cc-payment-display

Author: benjaminjkraft

.travis.yml

@@ -9,7 +9,7 @@ before_install:
 install: 
   - sudo apt-get install -y $(cat esp/packages_base.txt | grep -v ^memcached | grep -v ^postgres) -qq
   - esp/packages_base_manual_install.sh
-  - (pip install -r esp/requirements.txt --use-mirrors || tail pip.log 1>&2 2>/dev/null) | tee pip.log | grep '^Downloading/unpacking'
+  - pip install -r esp/requirements.txt --use-mirrors -q --log pip.log || (tail pip.log && exit 1)
 before_script:
   - cp esp/esp/local_settings.py.travis esp/esp/local_settings.py
   - ln -s `pwd`/esp/public/media/default_images esp/public/media/images

docs/dev/contributing.rst

@@ -0,0 +1,80 @@
+Workflow
+========
+
+Recommended one-time setup
+--------------------------
+
+Remove ``--global`` if you don't want it to apply to other git repos on your computer: ::
+
+  git config --global user.name "Your Name"
+  git config --global user.email [email protected]
+
+Other git config you might find useful: ::
+
+  git config --global push.default simple # makes git only push the current branch, which is useful for not accidentally messing things up
+  git config --global color.{ui,status,branch,diff,interactive,grep} auto # makes various interfaces usefully colorful
+  git config --global log.decorate true # shows branch information in `git log` by default
+  git config --global core.pager "less -R"
+  git config --global pager.{status,branch,diff,show,log} true
+  git config --global color.pager true
+  git config --global core.editor vim # or emacs, nano, your favorite text editor, etc.
+  git config --global grep.lineNumber true
+
+For normal, non-urgent features and bug fixes
+---------------------------------------------
+
+The following workflow applies if you've already been added as a collaborator to the repository.  If not, you should fork it using the button on Github, add it as a remote, and replace all of the ``git push``/``git push origin`` steps with ``git push remote-name``.
+
+From the directory ``/esp``: ::
+
+  git checkout main
+  git pull
+  ./update_deps.sh # on vagrant: see vagrant docs; no need to bother if deps haven’t changed
+  ./manage.py update # on vagrant: fab manage:cmd=update
+  git checkout -b new-branch-name
+
+Write some code!
+Test your code!
+
+Look at what you’ve changed (``git status`` and/or ``git diff``), and then run ``git commit -a``, and type a commit message (see `<http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html>` for good commit message style).  Repeat a few times from “write some code!” if you want to make multiple commits.
+
+When you’re ready to make a pull request, or want other people to be able to pull your code: ::
+
+  git push --set-upstream origin new-branch-name
+
+Now go to `<https://github.com/learning-unlimited/ESP-Website>`_. If you’re logged in, you should see a banner near the top suggesting that you make a pull request. Click the button, write up a summary of your pull request, and submit it.
+
+Now wait for someone to review your pull request. If it takes a long time, poke a friend to review it.
+
+When someone reviews your pull request, they will probably have some comments. If they have a lot to say, or suggest some major changes, don’t feel bad! This is a normal part of the code review process. Now you need to address their comments. If you disagree with what they’ve said, or want to discuss more, feel free to do that on the pull request. To change your code: ::
+
+  git checkout new-branch-name # only if you’d switched to another branch while waiting
+  Make some changes
+  Test your changes
+  git commit -a -m "Fixed foo, bar, and baz"
+  git push
+
+The reviewer will look at your changes. In some cases, you might go through a few more cycles of this. Once everything is resolved, they’ll merge your pull request. Congratulations!
+
+For urgent features and fixes
+-----------------------------
+
+The following is MIT-specific, although by using a different prod branch a similar thing might work for other sites. ::
+
+  git pull
+  git checkout $(git merge-base main mit-prod)
+  git checkout -b urgent-branch-name
+
+Write some code!
+Test your code! Be careful, since you’re putting this on our live server without full review. ::
+
+  git commit -a -m "Did something important"
+  git push --set-upstream origin urgent-branch-name
+
+  git checkout mit-prod
+  git merge urgent-branch-name
+  git push
+
+Pull code as described in `<https://esp.mit.edu/wiki/index.php/The_Server#Pulling_New_Code>`_.
+
+Make a pull request for ``urgent-branch-name`` as described above

docs/dev/directory_structure.rst

@@ -0,0 +1,30 @@
+Directory Structure
+===================
+
+This is not intended to be an exhaustive list, just a summary with some comments, to help new devs find things.  Starting in the root of the repository:
+
+* ``docs``: Fairly new and therefore very incomplete; add to it!
+
+  * ``admin``
+  * ``dev``
+
+* ``esp``: The root of the django project.
+
+  * ``esp``: The root of the python code.  Every folder from here on down needs to have an ``__init__.py``.  Contains a directory for each django app we use, along with some other files and directories.  There are a whole lot of these, but the most important are below.  Each app contains a file or directory for models, views, and tests; most also contain a directory for migrations.
+
+    * ``accounting``: The website’s accounting system (``accounting_core`` and ``accounting_docs`` are old and deprecated).
+    * ``program``: This app contains most of the site’s logic.  For the most part it’s a fairly normal Django app, but it also contains all the program modules.
+
+      * ``models``: The main models for ``Program``, ``ClassSubject``, ``ClassSection``; many of the other models used by programs live in other apps, such as ``resources``.
+      * ``modules``: This directory contains all the program modules.  The python code for the individual modules is in ``handlers/<name>module.py``; most of the remainder of the directory won’t be relevant to you.  See `<program_modules.rst>`_ for details on the program module system.
+
+    * ``qsd``: This app contains all of the code for QSD (which is the website’s system for admin-editable "quasi-static" text).
+    * ``users``: This app contains important models like ``ESPUser``, ``Permission``, ``Record``, ``ContactInfo``, and so on.
+    * ``*_settings.py``: Various files in which our django settings are kept.  ``django_settings.py`` contains settings used by all sites; ``local_settings.py`` is generated by the setup script and contains site-specific settings.
+    * ``urls.py``: Master URL routing table.
+
+  * ``manage.py``: Runs django commands like ``./manage.py shell_plus``.
+  * ``public``: Contains things that will be served statically.
+    * ``media``: Includes our javascript, CSS, images, etc.  Served statically at ``/media/``, so ``/esp/public/media/scripts/foo.js`` would be served at ``doma.in/media/scripts/foo.js``.
+  * ``templates``: Contains our django templates, in a directory structure roughly mirroring that of ``/esp/esp``.
+  * ``useful_scripts``: Contains various scripts intended to be run by devs, of varying usefulness, which may be site-specific, terribly-coded, completely broken, etc.

docs/dev/models.png

@@ -0,0 +1,18 @@
+Commonly Used Models
+====================
+Since `this picture <models.png>`_ may not be the most useful way to figure out what model does what you want, here’s a list of the most commonly used ones:
+
+* ``esp.users.models.ESPUser``: This is a proxy model over ``django.contrib.auth.User`` (the standard user account model). Proxy model basically means it attaches extra methods, but not extra fields.
+* ``django.contrib.auth.models.Group``: Keeps track of types of users, like Student and Teacher.
+* ``esp.users.models.Permission``: Permissions for various parts of the website; can be applied to a particular ESPUser or Group.
+* ``esp.users.models.{StudentInfo,TeacherInfo,ContactInfo}``: Various types of extra information associated with various types of users.
+* ``esp.program.models.Program``: An instance of a program, like Splash Fall 2014.
+* ``esp.program.models.class_.ClassSubject``: A class, like "X9002: Steak: Theory and Practice".
+* ``esp.program.models.class_.ClassSection``: A section of a class, like X9002s1.
+* ``esp.program.models.RegistrationProfile``: An instance of the profile information that students fill out before registering for a program.
+* ``esp.program.models.StudentRegistration``: A student’s registration for or interest in a ClassSection, like "benkraft marked X9002s1 as Priority/1".
+* ``esp.program.models.StudentSubjectInterest``: A student’s interest in a ClassSubject (correspond to stars in the two-phase lottery).
+* ``esp.cal.models.Event``: A timeblock (or sometimes another timed event such as a teacher training), associated with a program.
+* ``esp.resources.models.Resource``: A resource, like a classroom or the LCD projector in a classroom.  Note``: the resources models are a bit of a mess, and we’re working on rewriting them.
+* ``esp.accounting.models.LineItemType``: A type of payment, such as "Saturday Lunch for Splash Fall 2014 ($5)"
+* ``esp.tagdict.models.Tag``: A generic key/value pair, optionally associated to another object in the database (most commonly a Program).  Commonly used for settings.

docs/dev/useful_models.rst

@@ -0,0 +1,24 @@
+Other Utilities and Customizations
+==================================
+
+The ESP website has a bunch of other things in it that are useful to know.  Here are some of them:
+
+Rendering and Templates
+-----------------------
+* We use a modified ``esp.web.util.main.render_to_response`` instead of django's; it takes the template path, the request, and the context, and inserts useful things into the context, like information needed to render navbars, and the request.
+* To render a block of text that will be editable by admins, put a ``{% load render_qsd %}`` somewhere, and then put a ``{% render_inline_qsd name %}`` or ``{% render_inline_program_qsd program name %}`` in where you want the block.  If you want it to have default text, put it between the tags ``{% inline_qsd_block name %}`` and ``{% end_inline_qsd_block %}`` or ``{% inline_program_qsd_block program name %}`` and ``{% end_inline_program_qsd_block %}``.
+* You almost always want your template to begin with ``{% extends "main.html" %}``, to inherit from the main site layout; then put the content between ``{% block content %}`` and ``{% endblock content %}`` tags.  Even if you don't want the main site layout, you may want to inherit ``{% extends "elements/html" %}`` to get our standard JS and CSS and such.
+* We have a model in the database, ``esp.utils.TemplateOverride``, that makes it possible to override a template on a particular site by saving a new one in the database.  It's terrifying but also kind of useful. The theme editor uses these, and we often use them to customize printables.
+
+Caching
+-------
+* We have a caching library, ``esp.cache``, which does a lot of magical and terrifying things to make the website not have to recompute data all the time.
+* You can cache any function by putting an ``@cache_function`` decorator on top.  The hard part is of caching is making sure it always gets expired correctly, when any data that it depended on changes.  You do this by, after the function, putting ``function_name.depend_on_row()`` or similar calls, to make it depend on certain rows of the database.  The syntax is powerful but long, and probably best learned by looking at examples.
+* Don't cache things unless you’re sure you're expiring the cache at the correct times!
+* More details and the internals of the caching library are described in `<cache.rst>`_.
+
+Testing
+-------
+* You should always test features manually on your dev server before you push them to a server.  Seriously, do it.
+* We use the `Django Debug Toolbar <//django-debug-toolbar.readthedocs.org>`_, which will show up automatically on the right of all pages on a dev server.  It has useful things to tell you how slow your page is, and help debug errors.
+* We also have automated tests; they don't cover as much of the codebase as we'd like, but they do exist.  They get automatically run by Travis when you push to Github; you can also run some locally, although you probably don't want to run the whole test suite locally.  You should write more tests!  See the Django testing docs for details on how.

docs/dev/utils.rst

@@ -6,7 +6,6 @@
 
 from esp.accounting.controllers import GlobalAccountingController, ProgramAccountingController
 from esp.program.models import Program
-from esp.accounting_core.models import Transaction, LineItem
 
 class Migration(SchemaMigration):
 

esp/esp/accounting/migrations/0002_accounting_data.py

@@ -5,7 +5,6 @@
 from django.db import models
 from django import db as django_db
 
-from esp.accounting_docs.models import Document
 from esp.program.models import Program
 from esp.users.models import ESPUser
 
@@ -15,154 +14,7 @@
 
 def migrate_program(program):
 
-	docs = Document.objects.filter(anchor=program.anchor, doctype=2)
-	num_uncategorized = 0
-	num = 0
-	found_for_program = False
-	student_dict = {}
-
-	#   Clear financial data for this program
-	pac = ProgramAccountingController(program)
-	pac.clear_all_data()
-	lineitem_types = {'one': {}, 'multi': {}, 'select': []}
-	lineitem_choices = {}
-		
-	#   Build a database of each student's financial transactions for the program
-	for doc in docs:
-		student_id = doc.user.id
-
-		if student_id not in student_dict:
-			student_dict[student_id] = {
-				'admission': {},
-				'finaid_amt': Decimal('0'),
-				'items': [],
-				'items_select': [],
-				'payment': [],
-			}
-		lineitems = doc.txn.lineitem_set.all()
-		"""
-		if lineitems.count() > 0:
-			try:
-				print '\nStudent: %s' % doc.user.name()
-			except:
-				print '\nStudent: %s' % doc.user.username
-		"""
-		for li in lineitems:
-			found_for_program = True
-			base_txt = '[%5d] %s: %.2f' % (li.id, li.text, li.amount)
-			#   if li.anchor.uri == splash.anchor.uri + '/LineItemTypes/Required':
-			if 'admission' in li.text.lower() or 'cost of attending' in li.text.lower():
-				base_txt = '(Admission) ' + base_txt
-				if li.text not in student_dict[student_id]['admission']:
-					student_dict[student_id]['admission'][li.text] = li.amount.copy_abs()
-			elif 'financial aid' in li.text.lower():
-				base_txt = '(Finaid) ' + base_txt
-				student_dict[student_id]['finaid_amt'] += li.amount
-			elif 'payment received' in li.text.lower():
-				base_txt = '(Payment) ' + base_txt
-				student_dict[student_id]['payment'].append(li.amount.copy_abs())
-			elif 'expecting on-site payment' in li.text.lower():
-				base_txt = '(Payment expected) ' + base_txt
-				student_dict[student_id]['payment'].append(li.amount.copy_abs())
-			elif 'BuyMultiSelect' in li.anchor.uri:
-				base_txt = '(Select: field "%s", choice "%s") ' % (li.anchor.name, li.text) + base_txt
-				student_dict[student_id]['items_select'].append((li.anchor.name, li.text, li.amount.copy_abs()))
-				if li.anchor.name not in lineitem_types['select']:
-					lineitem_types['select'].append(li.anchor.name)
-					lineitem_choices[li.anchor.name] = []
-				if (li.text, li.amount.copy_abs()) not in lineitem_choices[li.anchor.name]:
-					lineitem_choices[li.anchor.name].append((li.text, li.amount.copy_abs()))
-			elif 'BuyMany' in li.anchor.uri or 'shirt' in li.text.lower():
-				base_txt = '(Multi: field "%s") ' % (li.anchor.name) + base_txt
-				student_dict[student_id]['items'].append((li.text, li.amount.copy_abs()))
-				if li.text not in lineitem_types['multi']:
-					lineitem_types['multi'][li.text] = li.amount.copy_abs()
-			elif 'BuyOne' in li.anchor.uri or 'lunch' in li.text.lower() or 'dinner' in li.text.lower() or 'photo' in li.text.lower():
-				base_txt = '(Single: field "%s") ' % (li.anchor.name) + base_txt
-				student_dict[student_id]['items'].append((li.text, li.amount.copy_abs()))
-				if li.text not in lineitem_types['one']:
-					lineitem_types['one'][li.text] = li.amount.copy_abs()
-			else:
-				num_uncategorized += 1
-				print 'WARNING: Uncategorized line item: %s' % base_txt
-				#   raise Exception('Uncategorized line item: %s' % base_txt)
-				
-			num += 1
-			#   print '-- %s' % base_txt 
-		"""
-		if student_dict[student_id]['finaid_amt'] > 0:
-			print student_dict[student_id]
-		elif len(student_dict[student_id]['items_multi']) > 0:
-			print student_dict[student_id]
-		"""
-		
-	if found_for_program:
-		num_programs = 1
-	else:
-		num_programs = 0
-	
-	#   Populate line item types for the program
-	optional_items = []
-	for item in lineitem_types['one']:
-		optional_items.append((item, lineitem_types['one'][item], 1))
-	for item in lineitem_types['multi']:
-		optional_items.append((item, lineitem_types['multi'][item], 10))
-	select_items = []
-	for item in lineitem_types['select']:
-		select_items.append((item, lineitem_choices[item]))
-	
-	#   print optional_items
-	#   print select_items
-	pac.setup_accounts()
-	pac.setup_lineitemtypes(0.0, optional_items, select_items)
-	
-	#   Create new transfer records for this student
-	for student_id in student_dict:
-		user = ESPUser.objects.get(id=student_id)
-		iac = IndividualAccountingController(program, user)
-		rec = student_dict[student_id]
-		
-		#   Admission fee
-		admission_total_cost = 0
-		for key in rec['admission']:
-			admission_total_cost += rec['admission'][key]
-		if admission_total_cost > 0:
-			initial_transfer = iac.add_required_transfers()[0]
-			initial_transfer.amount_dec = admission_total_cost
-			initial_transfer.save()
-			
-		#   Financial aid
-		if rec['finaid_amt'] > 0:
-			if rec['finaid_amt'] >= admission_total_cost:
-				iac.grant_full_financial_aid()
-			else:
-				iac.set_finaid_params(rec['finaid_amt'], None)
-		
-		#   Optional items
-		prefs = []
-		for item in rec['items']:
-			prefs.append((item[0], 1, item[1]))
-		for item in rec['items_select']:
-			prefs.append((item[0], 1, item[2]))
-		iac.apply_preferences(prefs)
-		
-		#   Payments
-		for payment in rec['payment']:
-			iac.submit_payment(payment)
-	
-	try:
-		attended_ids = program.students()['attended'].values_list('id', flat=True)
-
-		#   Execute transfers for the students that are marked as attended
-		pac.execute_pending_transfers(ESPUser.objects.filter(id__in=attended_ids))
-
-		#   Clear transfers for the students that are marked as not attended
-		pac.remove_pending_transfers(ESPUser.objects.exclude(id__in=attended_ids))
-	except:
-		print 'Unable to determine which students attended the program; all transfers remain unexecuted'
-
-	print 'Converted %d line items for %s; %d uncategorized' % (num, program.niceName(), num_uncategorized)
-	return num_programs
+	return 0
 
 class Migration(SchemaMigration):
 
@@ -323,4 +175,4 @@ def backwards(self, orm):
         }
     }
 
-    complete_apps = ['accounting']
+    complete_apps = ['accounting']