CS 51 Week 5, L1: Designing Your Own Advisor


HARVARD UNIVERSITY
Faculty of Arts and Sciences 


Dear Radhika,

As faculty, we believe, first and foremost, in achieving excellence in
everything we undertake. We seek to make Harvard College the premier
college of liberal arts and sciences in this country, and to maintain
its central position at the heart of this great research
university. We aim to ensure Harvard's role as a leader in the arts
and sciences. And we envision a Harvard that is more than the sum of
its many parts, across both sides of the Charles River, and, in many
fields, spanning the globe. In our review of undergraduate education,
we have continued to refine our proposals for change, but we have also
begun to implement new measures in the curriculum. I salute this
Faculty for persisting in doing the business of this institution.

However I am getting rather sick and tired of trying to coax faculty
into serving as freshman advisors.  I've tried sending cute memos
through departmental mail, offering free dinners, and even mild
threats. But your colleagues, it seems, don't want to have anything to
do with students. Even when they do volunteer to advise freshmen, the
advice they give isn't that good anyway.

Therefore as part of our curriculum review efforts, our education
quality team has recommended that we get rid of all this freshman
advising stuff and replace it with a a computer program.  Actually I
recommended it. I know you've got smart students in CS51. I'm sure
they could write something that will be at least as useful as our
current crop of advisors.  Do you think you could ask some of them to
try?  That would be a wonderful service to our great college.

BTW We really need this by the end of the week in order to plan the
budget for next year.

Sincerely yours,

Michael D. Smith

DEAN


Designing a Freshman Advisor

Today we will design a simple Freshman Advisor program. The goal of the Freshman Advisor is to listen to students and give an appropriate reply. This advisor program is based on the famous Eliza program written in the 1960's by Prof. Joseph Weizenbaum [1]. Eliza carried on ``conversations'' similar to those in non-directive psychiatric analysis.

A basic "advisor" program can be designed in four basic parts.

  1. A driver loop that takes input and prints output.

  2. A method for specifying a set of rules. A rule consist of a pattern and an associated reply-procedure. The set is most likely ranked in priority.

  3. A try-rule procedure that takes a rule and an input and sees if the input matches the rule pattern. If so it uses the reply-procedure to construct an appropriate reply.

  4. A construct-reply loop that takes a ruleset and an input and tries every rule in order. It returns the first successful reply or 'fail.

Nice Features: The basic design separates domain knowledge from the "advisor". So we can create diferent advisor structures, or extend an advisor by just providing new rule sets. In fact the original ELIZA program had German and English rule sets (he called them "scripts"). However a good advisor is still one that can reply intelligently to what a freshman says. So coming up with good rule sets is critical! Even there "design" is helpful because many types of questions end up having a similar structures.

Order of implementation

  1. Requirements: We will start with brainstorming (what does advisor do?)
  2. Basic Infrastructure Then we will build infrastructure, starting with simple less powerful versions, and then REFINE as we go along. This will make testing easier too.
  3. Domain Knowledge We will tackle the domain knowledge last.

Iterated Refinement: Most importantly however, we will think of the design in terms of ITERATIONS. We will incrementally build and test a first iteration advisor that achieves some subset of the requirements. In doing so we may find reasons to modify our design. We will make a list of improvements (extensions/optimizations) for next time around rather than try and build a perfect advisor in one shot. The goal of the first iteration (this lecture) is to demonstrate that the (simplified) design modules can work together as an integrated whole.

Week 4 Pithy Design Quote

Rules of Thumb for scheduling a software task (Brooks, The Mythical Man-Month)
  • 1/3 planning
  • 1/6 coding
  • 1/4 component test and early system test
  • 1/4 system test, all components in hand

[1] Weizenbaum, Joseph. "ELIZA - A Computer Program for the Study of Natural Language Communication between Man and Machine," Communications of the Assoc. for Computing Machinery 9 (1966): 36-45. Weizenbaum was quite disturbed by the lay persons response to this program, and the fact that some people chose to confide in Eliza rather than another human. This led him to write a book -- "Computer Power and Human Reason", 1976.

Basic Infrastructure

This is iteration one of the advisor. It will be able to answer a subset of questions, and demonstrate the basic input-output capabilities. The first thing we will build is the simplest part: the driver loop. 

;; DRIVER LOOP
;;-------------

;; Top-level procedures

(define (see-advisor myname)
  (display-intro-speech myname)
  (advisor-driver-loop myname))

(define (advisor-driver-loop name)
  (let ((user-response (prompt-for-input)))
    (cond ((equal? user-response '(goodbye))   ;; keyword to exit the advisor loop
	   (display-goodbye-speech name))
	  (else 
	   (display-reply-to user-response)
	   (advisor-driver-loop name))
	  )))



(define (display-intro-speech name)
  (newline)
  (display-line (list 'Hi name))
  (display-line '(I am your freshman advisor))
  (display-line '(What are your plans for the semester)))

(define (display-goodbye-speech name)
  (display-line (list 'Goodbye name))
  (display-line '(Have a good semester!)))

(define (display-reply-to input)
  (display-line (construct-reply input advisor-ruleset)))



;; Helpers

(define (display-line myline)
  (display myline)
  (newline))

(define (prompt-for-input)
  (display "** ")
  (read))





;; TESTS
;;--------

(display-line '(hello world))
(prompt-for-input)

;; This is a filler, for our wishful thinking
(define advisor-ruleset '())
(define (construct-reply input ruleset) (list 'Thats 'nice))	   
(advisor-driver-loop 'radhika)


(see-advisor 'radhika)


;; Example output
;;-----------------

(Hi radhika)
(I am your freshman advisor)
(What are your plans for the semester)
** (not much)
(Thats nice)
** (how about you?)
(Thats nice)
** (you don't say much do you?)
(Thats nice)
** bye
(Thats nice)
** goodbye
(Thats nice)
** (goodbye)
(Goodbye radhika)
(Have a good semester!)



;; Things to notice (worth fixing later!)
;; - Input right now must be surrounded by parens 
;; - if not this causes an error or misundertsanding 
;;   (like goodbye vs (goodbye))
;; Also the advisor doesn't say much


Reply To

Our next step is to provide a means for the advisor to make knowledgeable and seemingly intelligent remarks. This encompasses parts (2), (3) and (4) of our basic infrastructure. Problem is that unlike the driver loop, these are mutually dependent parts that are much harder to attack in a modular fashion. Still we can do a first version of the design with some wishful thinking 

;; CONSTRUCT-REPLY
;;------------------

;; Construct reply takes an input and a ruleset
;; Its purpose is to try each rule in order
;;     If the rule SUCCEEDS, then return the reply
;;     If the rule FAILS, then move on to the next rule.
;;     If you run out of rules - that's not good.

(define (construct-reply input rules)
  (if (ruleset-empty? rules)
      'construct-reply-failed
      (let ((thereply (try-rule input (ruleset-first rules))))
	(if (equal? thereply 'fail)
	    (construct-reply input (ruleset-rest rules))
	    thereply))
      ))



;; RULESET ABSTRACTION

(define make-ruleset list)
(define ruleset-first car)
(define ruleset-rest cdr)
(define ruleset-empty? null?)




;; TESTS
;;-------

;; Problem is there is no real ruleset yet 
;; -- or really any sense of what a rule is!
;; Still its V. important to be able to incrementally test.
;; We can create stubs to help us test this part

(define (try-rule input rule) rule)

(construct-reply 'hello (make-ruleset 'bye 'fail 'fail 'fail 'goodbye))
(construct-reply 'hello (make-ruleset 'fail 'fail 'fail 'goodbye))

(define (try-rule input rule) (if (equal? rule 'fail) 'fail input))

(construct-reply 'hello (make-ruleset 'bye 'fail 'fail 'fail 'goodbye))
(construct-reply 'hello (make-ruleset 'fail 'fail 'fail 'goodbye))

;; INTEGRATION TEST
;; Given this I can test integration with the previous part
;; (I expect this advisor to reflect my input)

(define advisor-ruleset (make-ruleset 'fail 'fail 'fail 'success))

(see-advisor 'radhika)


(Hi radhika)
(I am your freshman advisor)
(What are your plans for the semester)
** (nothing)
(nothing)
** (so what)
(so what)
** (thats annoying)
(thats annoying)
** (goodbye)
(Goodbye radhika)
(Have a good semester!)


Try-Rule

What is a rule?

What should try-rule do? 

;; TRY-RULE
;;----------

(define (try-rule input arule)
  (if (match? (rule-pattern arule) input)
      ((rule-procedure arule) input)
      'fail))


;; RULE ABSTRACTION

(define make-rule cons)
(define rule-pattern car)
(define rule-procedure cdr)


;; It may seem unnecessary or wasteful to make such simple renames. After all
;; this causes a level of redirect, correct? Not necessarily. The scheme interpreter
;; can bind the name directly to the primitive-procedure-obj that the name CONS points to.
;;
;; The reason why this renaming is important is because THIS IS A COMPLEX PROGRAM
;; - it has many types of data sprinkled throughout, and many levels of computation
;; - in such a program READABILITY is the ultimate road to efficiency.
;;   (its better to have a READABLE program than an efficient non-working one)




;; Test
;;------

(define (match? a b) #t)

(try-rule
 (list 'hello 'professor) ;; input
 (make-rule 'foopattern  (lambda (input) (dispense-random-advice)))) ;; output


Pattern Matching

Now the rubber hits the road! We have arrived at the real execution machine.

A rule consists of two parts

But,

Lets start by looking at a few simple question-answers. For now in the context of CS51.. 

 Ques: blah blah blah CS51 blah blah blah
 Ans: CS51 is a great course, Its about abstraction and
      design... but the workload is high. The CUE guide says...

 Ques: blah blah PREREQ blah blah CS51 blah blah blah
 Ans: CS 50 is the only prereq for CS51. However if you have taken
      AP exam and secured an A, then ... the thing to watch out for is...

 Ques: blah blah CS51 blah PREREQ FOR blah blah CS124
 Ans: The prereq for CS124 is programming experience. 
      CS51 is a preq for CS 143, CS161 and CS152.


We can go from simple to complicated pretty darn fast. And we might want to have a similar set of questions for every class. That's quite a bit of "data" to manage. For now, we will focus on a few patterns and think about how to write match? 


;;
;; Ques 1: blahblah CS51 blahblah
;; Ques 2: blahblah PREREQ FOR blahblah CS51 blahblah
;; Ques 3: blahblah CS51 blahblah PREREQ OF blahblah CS124 blahblah
;;          where blahblah is zero or more words
;;
;; General pattern -> some sequence of "*" and keywords
;; Match           -> keywords occur in correct order


MATCH (POINT) 1 

;; MATCH (POINT) 1
;; pattern = list of keywords that should occur in order

(define (match1? pattern data)
  ;; (display pattern)             ; debugging
  ;; (display data) (newline)
  (cond ((null? pattern) #t)       ; everything in my pattern matched something
        ((null? data) #f)          ; still have pattern left over, so failed
        ((equal? (car pattern) (car data))     ; matched, move to the next keyword
         (match1? (cdr pattern) (cdr data)))
        (else (match1? pattern (cdr data)))   ; didn't match, keep looking in the data
        ))

#|
;; TESTS
;;-------

(match1? '() '(some random statement))
(match1? '(some random pattern) '())
(match1? '() '())

(match1? '(hello) '(hello how are you?))
;; expect #t
(match1? '(hello how) '(hello how are you?))
;; expect #t
(match1? '(hello how) '(hello so how are you?))
;; expect #t
(match1? '(hello how) '(Jasmine hello so how are you?))
;; expect #t
(match1? '(hello how) '(Jasmine hello whats up?))
;; expect #f
(match1? '(hello how) '(Jasmine how is hello doing?))
;; expect #f

;; If we add code to match1 to display its arguments at the start
;; of each iteration (this is an iterative process), then we can
;; see how it works.

> (match1? '(hello how) '(Jasmine hello whats up?))
(hello how)(Jasmine hello whats up?)
(hello how)(hello whats up?)
(how)(whats up?)
(how)(up?)
(how)()
#f

>(match1? '(hello how) '(hello so how are you?))
(hello how)(hello so how are you?)
(how)(so how are you?)
(how)(how are you?)
()(are you?)
#t

|#


MATCH (POINT) 2 

;; MATCH (POINT) 2
;; pattern = list of *'s and keywords
;;         = *s can match zero or one words, keyword should occur in order
;; Match 2 allows us to look for multiple keywords that must
;; appear in order with no intervening words
;; Arbitrary behavior: match assumes a wildcard (*) at the end.

(define (match2? pattern data skip)
  ;; (display pattern)          ; debugging
  ;; (display data) (newline)
  (cond ((null? pattern) #t)                      ; Everything in my pattern matched something, 
	                                          ;     (allow unmatched data at the end)
        ((null? data) #f)                         ; Still have pattern left over, so failed

	((equal? (car pattern) '*)                ; Wildcard
	 (match2? (cdr pattern) data #t))         ; Set skip to #t

        ((equal? (car pattern) (car data))        ; Matched keyword, move to the next keyword
         (match2? (cdr pattern) (cdr data) #f))   ; Reset skip to #f

        (skip (match2? pattern (cdr data) #t))    ; Didn't match keyword, if skip=#t keep looking
       
	(else #f)))                               ; if skip=#f, then pattern match failed

#|
;; Tests
;;--------

(match2? '() '(hello how are you?) #f)
;; expect true

(match2? '(*) '(hello how are you?) #f)
;; expect true

(match2? '(hello) '(hello so how are you?) #f)
(match2? '(* hello) '(hello so how are you?) #f)
(match2? '(hello *) '(hello so how are you?) #f)
(match2? '(* hello *) '(hello so how are you?) #f)
;; expect true for all cases

(match2? '(hello) '(Jasmine hello so how are you?) #f)
;; expect false (no * assumed at start of line)

(match2? '(hello how) '(hello so how are you?) #f)
;; expect false

(match2? '(hello * how) '(hello so how are you?) #f)
;; expect true

|#


A SIMPLE ADVISOR 

;; A SIMPLE ADVISOR
;;-----------------------
;; Given this basic structure, We can already test our advisor's abilities
;; This is essentially an integration test (again!)
;; But a far more interesting one than before

(define (match? pattern data) (match2? pattern data #f))


(define advisor-ruleset
  (make-ruleset
   (make-rule '(* prereq of * CS51) 
	      (lambda (input)
		'(The only prereq of CS51 is CS50. 
		  However if you have a high opinion
                  of your skills with programming then 
                  you may consider skipping CS50)))

   (make-rule '(* CS51 * prereq for) 
	      (lambda (input)
		'(CS51 is a prereq for CS152 and CS182)))

   (make-rule '(* skip * CS51) 
	      (lambda (input)
		'(CS51 is a wonderful class. We do not recommend skipping it.
		  You miss out on some fabulous prizes)))

   (make-rule '(* CS51) 
	      (lambda (input)
		'(CS51 is a wonderful class. They give you pens and stuff
                  But generally we dont recommend that Freshman take it.
                  Freshman already have too many pens.)))

   (make-rule '()
	      (lambda (input) (dispense-random-advice)))
   ))



(define (dispense-random-advice)
  (list-ref general-advice (random (length general-advice))))

(define general-advice
  '((Make sure to take some core subjects)
    (Harvard CS has a lot of interesting classes - what are you interests?)
    (Make sure to get time to explore the Boston area)
    (How about a freshman seminar?)
    ))



;; Tests
;;-------

;; Constructing tests bottom up usually reveals many different kinds of bugs
;; for renaming procedures midway to more subtle issues
(construct-reply '(I was thinking of taking CS51) advisor-ruleset)
(construct-reply '(I am a sophomore) advisor-ruleset)
(construct-reply '(Is xyz a prereq of CS51) advisor-ruleset)  
(construct-reply '(Is xyz a prereq of CS51 ?) advisor-ruleset)
(construct-reply '(Is CS51 a prereq for rock climbing ?) advisor-ruleset)
(construct-reply '(Why should I take CS51 - I am not a CS concentrator)  advisor-ruleset)
(construct-reply '(My roommates suggested I should skip CS51 altogether) advisor-ruleset)

;; Now we run our advisor


;; SAMPLE RUN
;;-----------
(see-advisor 'radhika)


(Hi radhika)
(I am your freshman advisor)
(What are your plans for the semester)
** (I am not sure)
(How about a freshman seminar?)

** (Well I was thinking about taking CS51 )
(CS51 is a wonderful class. They give you pens and stuff 
But generally we dont recommend that Freshman take it. Freshman already have too many pens.)

** (What are the prereqs for CS51)
(CS51 is a wonderful class. They give you pens and stuff 
But generally we dont recommend that Freshman take it. Freshman already have too many pens.)

** (Is CS50 a prereq of CS51 because I haven't taken it yet)
(The only prereq of CS51 is CS50. However if you have a high opinion 
of your skills with programming then you may consider skipping CS50)

** (Well I do have a high opinion of myself how else would i be here?)
(Harvard CS has a lot of interesting classes - what are you interests?)

** (I like rock climbing)
(Make sure to get time to explore the Boston area)

** (goodbye)
(Goodbye radhika)
(Have a good semester!)



;; Notice that we do still have "bugs" but they are more related to our domain knowledge
;; For example "prereqs" vs "prereq" and "prereq of" vs "prereq for"
;; Still we aren't doing too bad
;; But we wouldn't fool a single student.

|#


MATCH (POINT) 3 

;; MATCH (POINT) 3
;; pattern = list of *'s and keywords
;;         = *s can match zero or one words, keyword should occur in order
;; match returns a LIST OF PHRASES that matched the *s

;; Consider the following case
;;-----------------------------
;;
;; Ques: (I am considering doing a double concentration in CS and rock-climbing)
;; Ans: (CS is fascinating and you can make a living doing 
;;       it if rock-climbing doesnt work out)
;;
;;
;; One thing to notice from the previous cases, is that we almost never needed
;; to use pieces of the students input in our reply - we just reacted to key phrases
;; and ignored the rest (how surprising...)
;;
;; In this case however our pattern is
;; PATTERN (*1 double concentration in *2 and *3)
;; REPLY (*2 is fascinating and you can make a living doing it if *3 doesnt work out)
;;

(define (match3? pattern data skip curphrase phraselist)
  ;; (display pattern)                             ; debugging
  ;; (display data)
  ;; (display curphrase) (display phraselist)
  ;; (newline)

  (cond ((null? pattern)                          ; Everything in my pattern matched something
	 (reverse (cons data phraselist)))

        ((null? data) #f)                         ; Still have pattern left over, so failed

	((equal? (car pattern) '*)                ; Wildcard,
	 (match3? (cdr pattern) data              ; Set skip=#t and curphrase to '()
		  #t '()
		  phraselist))

        ((equal? (car pattern) (car data))        ; Matched keyword, move to the next keyword
         (match3? (cdr pattern) (cdr data)        ; Set skip to #f and curphase '()
		  #f '()                          ; 
		  (if skip                        ; If skip was true, then we were collecting
		      (cons (reverse curphrase) phraselist)  ; Put collection sofar onto phraselist
		      phraselist)))

        (skip                                     ; Didn't match keyword
	 (match3? pattern (cdr data)              ; if skip=#t keep looking and collecting
		  skip 
		  (cons (car data) curphrase)
		  phraselist))
                                                  ; Didn't match keyword
	(else #f)                              ; if skip=#f, then pattern match failed
	))


#|

;; Tests
;;-------

(match3? '(* CS51) '(Hello i like CS51 alot) #f '() '())
((Hello i like) (alot))

(match3? '(* weather * cold) '(I hate the fact that the weather is so cold every day) #f '() '())
((I hate the fact that the) (is so) (every day))

(match3? '(* weather * cold) '(I hate the fact that the weather is so hot every day) #f '() '())
#f

(match3? '(* weather * cold) '(weather is so cold) #f '() '())
(() (is so) ())


;; Notice however that this particular change percolates higher than just match!
;; 
;; TRY-RULE keeps the matching process separate from the output process.
;; So we would need to change the way we implemented try-rule to accomodate this new behavior
;;
;; RULE-PROCEDURES now need to be procedures that take TWO arguments!
;; So we would need to change our rulesets as well

(define (match? pattern data)
  (match3? pattern data #f '() '()))

(define (try-rule input arule)
  (let ((phraselist (match? (rule-pattern arule) input)))
    (if (equal? phraselist #f)
	'fail
	((rule-procedure arule) input phraselist))))


(define doublerule
  (make-rule '(* double concentration in * and)
	     (lambda (input phraselist)
	       (append (cadr phraselist)
		       '(is fascinating and you can make a living doing it if)
		       (caddr phraselist)
		       '(doesnt work out)))))

(try-rule '(I am considering doing a double concentration in CS and rock-climbing)
	  doublerule)

;; (CS is fascinating and you can make a living doing it if rock-climbing doesnt work out)

(try-rule '(My roommate said I should do a double concentration in Economics and CS)
	  doublerule)

;; (Economics is fascinating and you can make a living doing it if CS doesnt work out)

;; DESIGN FLAW!
;;--------------
;; So as we got ambitious some design decision we made before is no longer valid
;; This is why design is a PROCESS!
;; - It is not possible to determine every contigency ahead of time
;; - It is not good to design for every contigency the first time around!
;; 
;; This is why a considerable amount of time is spent in the design phase and includes
;; making prototypes to hash out design decisions that are too narrow 
;; for the expected future
;;
;; Our new match is considerably more complex, we could make it even more complex
;; But if all we wanted to do was simple responses like before that would be a waste
;; of time. Instead it might be be better to focus on building up the domain knowledge.


Domain Knowledge

Now that we have a basic infrastructure for the advisor we can start thinking about domain knowledge. What all do we want our system to know? What all natural language constructs should it recognize?

And most critically, how do we manage all this domain knowledge?

In the freshman advisor case we might have

Each of these is not nearly as hard to "program" as what we have already done. Each of these is a data type that may already exist in some electronic format. We may want to interface to these sources. For example: 
(make-rule 
 '(* CS124)
 (lambda (input) (append '(Glad to see you've thinkign about your courses)
			 (list 'CS124 'is (catalog-lookup-description 'CS124)))) )


(make-rule 
 '(* prereq of * CS182)
 (lambda (input) (append (list 'The 'prereqs 'of 'CS182 'are (catalog-lookup-prereq 'CS182))
			 (list 'Also 'the 'CUE 'guide 'says (cue-lookup-description 'CS182)))))

There are alot of courses, so we may even generalize rule construction 

(define (make-prereq-rule COURSENAME)
  (make-rule 
   (list '* 'prereq 'of '* COURSENAME)
   (lambda (input) 
     (append (list 'The 'prereqs 'of COURSENAME 'are (catalog-lookup-prereq COURSENAME))
	     (list 'Also 'the 'CUE 'guide 'says (cue-lookup-description COURSENAME))))
   ))

But it may make more sense to just have multiple rulesets, instead of one humongous ruleset to end all rulesets. How much of our design would we need to change?

Improvements and Extensions

  1. Output to Other Programs: For example. voice input and output (think how automated phone systems are specified), aol instant message input and output (think of an automated help desk)

  2. Input from a File: allow administrators to easily update and extend the set of rules.

  3. Access to Student Records: Then the advisor could give much more specific advise based on what the background the student has.

  4. Add state: This would allow the advisor to act a bit more intelligently and not repeat itself, or forget what the student dsaid earlier.

  5. Use a more flexible pattern language: This would allow us to quickly capture many more ways of saying the same thing and capture patterns that are very similar. 
    For example,
= (* [OR prereq prereqs prequisites] [OR 'of 'for] * CS51)

Pattern recognition is a rich and old area and people have thought long and hard about pattern description languages, ways of automatically creating match functions (parsers), and the complexity of matching. Pattern description languages are used ALL THE TIME. 

GOOGLE SEARCH: grep OR sed NOT awk, DEFINE competence, IMAGE blah blah

Other Similar Programs

If we take away the vaneer of the freshman advisor and natural language, what we have designed is a system that takes in a set of rules (patterns and corresponding actions) and applies them to data. Lots of problems and programs look like this.

Here are some examples:

Surprisingly, writing programs that parse natural language is in fact really hard. Most sentences are ambiguous and require a significant amount of "real-world" knowledge.

CS51, Spring 2008, Radhika Nagpal