Breakpoints Revisited  

Bob Hinkle and Ralph E. Johnson

539 N. Santa Anna
Mesa, AZ  85201

Department of Computer Science
University of Illinois at Urbana-Champaign
1308 W. Springfield Ave., Urbana, IL  61801

hinkle@primenet.com
johnson@cs.uiuc.edu

<Notes to Ralph:  passing this thing back and forth electronically does make it harder for you to add comments (nothing is easier than a red pen!), but maybe you could just type them in and indicate with your initials or other easy string for me to search on.
Also, I’ve highlighted the major Smalltalk code sections with a different font, but still need to do that for most references in the midst of narrative text.  That’s something I’ll get to soon.
Also also, the current word count is around 7000, so I assume it will have to be a two-parter.  I’ll work it into that shape after we’ve polished it up for a round or two.  Bob>

Our previous article [1] described how to debug the behavior of individual objects using lightweight classes.  To make debugging easier, and as something of a side project, we also introduced breakpoints, a common feature of modern programming environments that has so far been represented in Smalltalk-80 only by its poor cousin, the halt statement.  However, while the breakpoints we introduced had several benefits over self halt--notably their independence from the changes file and their ease of addition and removal--they also had several limitations.  In particular, these breakpoints could only be set at the very beginning of a method, and they were unconditional.  This paper shows how to implement breakpoints that can be either absolute or conditional and that can be inserted between any two statements in any block of code.  As before, this work has an additional purpose, which is to expose some of the more arcane inner workings of Smalltalk, and to show why it is both useful and powerful to allow programmers reflective access to these parts of the system.  The solution described here is based on VisualWorks version 2.0, though it can be adapted to previous versions of Smalltalk-80.
BreakpointsNew and Improved!
What features should we expect from a breakpoint system?  We still want breakpoints that can be set without affecting the ChangeSet and the changes file, we want them to be inserted and removed as easily as possible, and we want the Browser to indicate which methods have breakpoints.  We also want breakpoints that can be either absolute, meaning they (like self halt) always cause a NotifierView to open, or conditional, meaning that they open a NotifierView only if some expression evaluates to true.  Furthermore, this expression should be a general block of Smalltalk code that will be evaluated in the context of the breakpointed method, allowing access to method and block parameters and temporaries.  We want to be able to insert breakpoints between any two statements in any block.  This means that the browser will have to indicate their position in the method’s text graphically.  Finally, breakpoints should be manipulable in debuggers as well as in browsers.
These new requirements impose several difficulties, dictating a change in tactics from our previous implementation.  The possibility of breakpoints being conditional means that we must be able to insert an arbitrary block of code anywhere breakpoints are allowed, and, in addition, we need to provide an interface for users to enter and edit breakpoint condition strings.  More significantly, allowing breakpoints in the middle of a method requires a more general mechanism for installing the breakpoint code:  previously, we wrapped a new breakpoint method around the unchanged original method, but now we will have to insert breakpoint code in the body of the breakpointed method.  Furthermore, we must perform this insertion in a way that not only works correctly, but that also respects the source code map, so that stepping through breakpointed methods in the debugger works as the user expects.  Finally, indicating breakpoints graphically in a method’s source, combined with our continued desire to insert breakpoints without affecting the change log, implies the need for two levels of source code for breakpointed methods; one to display in browsers, and the other to store and retrieve from disk.
The Beginning and the End
The life of a breakpoint has three phases.  First, a breakpoint is created by a user interacting with a programming tool to insert a breakpoint into the text of a method.  Second, the breakpoint is implemented when the source text is compiled into a new method.  Third, the breakpoint is activated when the new method is executed and draws attention to itself in some useful way.  While the most difficult and intrusive work for this project is concentrated on that middle phase, it’s helpful to discuss what happens in the beginning and in the end to motivate our choices and to establish a context for our discussion.
Breakpoints are inserted and removed in the text views of the various kinds of browsers, including the debugger.  The text view menu will provide options to insert a breakpoint, either absolute or conditional, at the cursor point, to convert breakpoints from absolute to conditional and back again, and to remove all breakpoints within the selection.  We’ll explain in detail how to insert a conditional breakpoint, since the other options are mostly minor variations on this.
When the user asks to insert a breakpoint, we would like to insert an instance of a new class called Breakpoint into the edited text to mark the breakpoint’s position.  Unfortunately, the text view of a browser edits ComposedText, which contains an instance of Text, whose content is defined by a String, which is defined to contain only Characters.  To support breakpoints, therefore, we need to create a new variety of editable text that can contain things other than characters.  For this purpose, we create a new object called StructuredText, which will be used to represent the source code of a breakpointed method.  StructuredText objects will satisfy enough of the interface of Text objects that they can be stored in ComposedText objects and edited by ParagraphEditor and its subclasses.  The definition of StructuredText is:

Object subclass: #StructuredText
	instanceVariableNames: ‘children displayText'
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Enhancements-Text'

A StructuredText has two instances variables.  The variable children holds an ordered collection of text-like objects, including Breakpoints (which we will define in a moment).  The variable displayText holds the result of concatenating the elements in children into an actual Text object.  This variable is lazily initialized, and if ever set to nil (which is done by the message #resetDisplayText), its value will be recomputed on the next access.
Many of the messages understood by Text objects are implemented in StructuredText by delegating to this displayText component, simplifying the programming of the class:  these messages include #at:, #copyFrom:to:, #emphasisAt:, #hasChangeOfEmphasis, #runLengthFor:, #size, #string, and #=.  The other methods in the class revolve around manipulating its children:  adding new children, removing existing children, enumerating children, and replacing or inserting text at a specified index.  Before we look at the implementation of these methods, we must first introduce the two kinds of objects that can be stored as children in a StructuredText.  The first, called SubText, is a simple refinement of Text:

Text subclass: #SubText
	instanceVariableNames: 'parent start stop '
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Enhancements-Text'

The parent instance variable will point back to the StructuredText that contains the SubText instance.  The variables start and stop are used in enumerating over the StructuredText and record respectively the index in the StructuredText’s displayText of the initial and final characters in the SubText, thus providing a convenient way to convert between local string coordinates in the SubText and global string coordinates in its containing StructuredText.  SubText provides methods to get and set all three of these variables.  Also, to facilitate creation of SubTexts, we implemented a new method Text>>asContainableText, which converts the receiver to a SubText with the same contents and emphasis, and a similar conversion message, #asStructuredText, converts a Text to StructuredText.  Since Breakpoints will also be stored in StructuredText, we define them as a subclass of SubText:

SubText subclass: #Breakpoint
	instanceVariableNames: 'isAbsolute conditionString '
	classVariableNames: 'BreakpointSignal '
	poolDictionaries: ''
	category: 'Projects-Breakpoints'

Breakpoint class implements two messages, #absoluteBreakpointCharacter and #conditionalBreakpointCharacter, that define the characters used to indicate absolute and conditional breakpoints, respectively.  They are currently the empty-set character  and the upside-down question-mark character À, though these choices are easily changed.  The class also provides two new instance creation methods:

newAbsolute
	^(self new: 1) beAbsolute

newConditional
	^(self new: 1) beConditional

These each create a Breakpoint with a one-character string.  The instance messages #beAbsolute and #beConditional set the value of isAbsolute to true and false, respectively, and set the value of the string using the character-defining methods in the class.  The conditionString instance variable is only relevant if isAbsolute is false.  In that case, it is used to generate the source string for the Breakpoint as follows:

sourceString
	| condString |
	condString := self isConditional
		ifTrue: [conditionString]
		ifFalse: ['true'].
	^'[', condString, '] value ifTrue: [self breakpoint]'

For an absolute Breakpoint, the return value of #sourceString will be the string ‘[true] value ifTrue: [self breakpoint]’thus, when this code is executed, the message #breakpoint will always be sent.  However, for a conditional Breakpoint, the value of its conditionString will be used as the body of the initial test block instead of ‘true’.
With SubText and Breakpoint defined, we can return to the implementation of StructuredText, and in particular to the key method allowing insertion of a new Breakpoint at a specified index within the StructuredText.  The method StructuredText>>addChild:at: is defined as:

addChild: aChild at: anIndex
	| localIndex textChild target |
	anIndex > self size
		ifTrue: [^self addLast: aChild].
	anIndex <= 1
		ifTrue: [^self addFirst: aChild].

If the indicated index is either before the beginning or after the end of the StructuredText, the method adds it accordingly to the collection of children.

	self computeTextPositions.
	self resetDisplayText.
	textChild := aChild asText asContainableText parent: self.
	target := self children select: [ :text | (text start <= anIndex) & (anIndex <= text stop)].

Otherwise, the method uses #computeTextPositions to set the start and stop variables of all of its children, resets its displayText (since this will be out of synch once the new child is added, and converts the new child into something containable (SubText or a subclass).  The targeted child is the one whose position in global coordinates brackets the insertion point.  This insertion point is then converted into the local coordinates of the targeted SubText.  If its local coordinate is 1, the new child can simply be added before the targeted child.

	(localIndex := anIndex - target start + 1) = 1
		ifTrue: [self children add: textChild before: target]
		ifFalse: [
			self children
				add: (target copyFrom: localIndex to: target size) asContainableText
				after: target;
				add: textChild after: target;
				add: (target copyFrom: 1 to: localIndex - 1) asContainableText
				after: target;
				remove: target].
	^textChild

If, however, the local coordinate of the insertion point is not 1, the targeted child must be broken up into two parts, with the new textChild added to the collection of children in between.  The other messages involving manipulation of the StructuredText’s children behave similarly, with a wrinkle added when removing a child:  if removal results in two newly adjacent SubTexts, they are combined into a single SubText to reduce fragmentation.
Now that we have StructuredText to use, it is fairly easy to implement Browser manipulation of breakpoints.  When the menu item for inserting a conditional breakpoint is chosen, the controller of the Browser’s TextView sends the message #addConditionalBreakpoint:from: to the Browser:

addConditionalBreakpoint: fullText from: aController
	| newSelector producer  start  bp text str |
	str := DialogView 
		request: 'Enter the condition code -- must evaluate to a Boolean' 
		numLines: 10
		initialAnswer: ''
		onCancel: [^self].
	str isEmpty ifTrue: [str := 'true'].

The browser first prompts the user for the conditionString to be associated with the new breakpoint.  To do this, we created a new class method for DialogView that opens a multi-line text dialog.

	producer := self selectedClass methodProducerFor: fullText.
	start := aController view startBlock stringIndex.
	bp := Breakpoint newConditional conditionString: str.
	(text := fullText asStructuredText copy) addChild: bp at: start.

Next, the Browser finds a MethodProducer appropriate to the input text and selected class.  MethodProducers will be explained in detail in the next section, but for now think of them as a new kind of compiler. A new conditional breakpoint is created with the previously input conditionString.  The Browser ensures that the edited text is an instance of StructuredText using the conversion message, and adds the new breakpoint to the StructuredText at the insertion point indicated by the TextView’s controller.

	newSelector := self selectedClass
		compile: text
		classified: protocol
		notifying: aController
		producer: producer.
	newSelector == nil ifTrue: [^false].
	self newSelectorList: newSelector.
	^true

Finally, the MethodProducer is passed along to the selected class in a new variation of the compilation messages.  We’ll explain what happens during compilation in the next section.  Afterward, though, a new breakpointed method will have been placed into the selected class’ method dictionary with the selector returned in newSelector.  The Browser updates its selector list because the list provides an indicator of breakpointed methods, which may have changed with the insertion of this new breakpoint.  When the selector list is updated, the TextView will also be updated, but what will its contents be?  Remember that one requirement for this project was that breakpoint changes are not written to disk, so without work on our part, the Browser would retrieve the newly compiled method’s source from the change log and display it, but would not indicate where this breakpoint (and any others) occur.  We need to change the way Browsers access method source code to allow for this possibility.  First, we add a few simple methods to CompiledMethod:

getSource
	^self getDisplaySource

getDisplaySource
	^self getStoredSource

getStoredSource
	^Cursor read
		showWhile: 
			[SourceFileManager default stringAt: self sourcePointer ifAbsent: [nil]]

When asked for its source using the normal message #getSource, a CompiledMethod replies with result of its own #getDisplaySource message, since the #getSource request is sent to obtain the source code that should be displayed to the user.  For a CompiledMethod, though, the displayed source is the same as its stored source, which is accessed from the sources file or change log as appropriate.  A method with breakpoints will have a different answer for the #getDisplaySource message, and so we create a new class:

CompiledMethod variableSubclass: #BreakpointMethod
	instanceVariableNames: 'displaySource '
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Projects-Breakpoints'

getDisplaySource
	^displaySource

displaySource: aTextOrString
	displaySource := aTextOrString

When a BreakpointMethod receives the #getSource message, it still responds with the value of its own #getDisplaySource message, but this now returns the value of the BreakpointMethod’s similarly named instance variable.  The StructuredText produced in the Browser when a breakpoint is added will have been stored in this variable during the compilation process, so when the Browser updates the source code for a breakpointed method, it will retrieve this StructuredText to present and edit, with all indications of breakpoint characters included.
Once a breakpoint has been inserted in a method, what should happen when it is encountered during execution?  An activated breakpoints should draw the user’s attention, and we use NotifierView to achieve this.  The NotifierView will alert the user that a breakpoint has been encountered and give them a chance to open a debugger at that location.  However, if the user is already debugging a process, we don’t want to open a new NotifierView with the potential to open a new debugger, but rather we’d simply like the debugger to announce the fact and switch to the breakpointed context.  This is just how self halt works, so we will pattern the breakpoint alert mechanism after it.  We need to implement a new Signal, a way to access it, a way to raise it, and something to catch it when it occurs.  The signal itself will be stored in a class variable of Breakpoint, and we’ll call it BreakpointSignal.  The new methods to support it can then be implemented as follows: 

Breakpoint class>>initialize
	"self initialize"
	BreakpointSignal := Object controlInterruptedSignal newSignal 
		notifierString: 'Breakpoint in';
		nameClass: self message: #breakpointSignal

Breakpoint class>>breakpointSignal
	^BreakpointSignal

The class initialization method creates a new signal whose parent is the controlInterruptedSignal defined in Object; Object haltSignal’s parent is the same.  We make this signal available to the rest of the image in the class method, so that BreakpointSignal can be raised by any object and a handler for it can be defined by any object.  A default way to raise the signal is provided in:

Object>>breakpoint
	| sender |
	sender := thisContext sender.
	Breakpoint breakpointSignal 
		raiseRequestWith: thisContext
		errorString: ' ', sender mclass printString, '>>', sender selector

When a signal is raised, a new exception is created, and the virtual machine looks for an exception handler in the current process stack.  (For more details on the exception handling mechanism, consult chapter 8 of [2], or our previous article on exception handling [3, 4].)  The VM finds the handler in the most recent send of the message #handle:do: to the raised signal or to any one of its parents.  If such a send exists, the first parameter is a block that will be evaluated to handle the exception.  However, if no handler exists, which will often be the case for breakpoints (and self halt, for that matter) during development, the block used to handle the exception will be Exception class>>emergencyHandler.  The body of this block is actually defined in Exception class>>emergency:from:, and so we need to modify that method to provide default handling for a BreakpointSignal-based exception.

Exception class>>emergency: exception from: context
	"Define a default emergency handler.
	This one is appropriate for development
	environments, but not for runtime."

	(Object controlInterruptedSignal accepts: exception signal)
		ifTrue: [
			exception signal == Breakpoint breakpointSignal
				ifTrue: [NotifierView openBreakpoint: exception]
				ifFalse: [NotifierView openException: exception]]
		ifFalse: [NotifierView
					openContext: context
					label: 'Unhandled exception: ' , exception errorString
					proceedable: exception willProceed]

The method Signal>>accepts: returns true if the receiver is equal to the parameter or is a parent of the parameter.  We created BreakpointSignal as a descendant of Object>>controlInterruptedSignal, so the #accepts: message send will evaluate to true for an exception generated from BreakpointSignal.  Then, we check whether the signal for this particular exception is BreakpointSignal:  if it is we open a NotifierView using the message #openBreakpoint:, and if not we open a NotifierView using the message #openException:.  This customized method for viewing breakpoint exceptions allows us to specialize for the user’s convenience, which primarily involves adjusting the NotifierView to open on the context in which the breakpoint occurred rather than the context of Object>>breakpoint itself.
The Smalltalk Compiler
Most of the changes in this project center around the process of compiling code to produce new methods.  Therefore, we will examine how the compilation process currently works to see how to extend it to support breakpoints.  This process involves six major steps:
1.	The user selects accept in the TextView of a Browser.
2.	The Browser passes its current text to the currently selected class using the message #compile:classified:, which forwards to ClassDescription>>compile:classified:notifying:.
3.	The class creates a new Compiler--the class is actually defined in the class method #compilerClass, but this by default always returns Compiler--and passes the source text using the message #compile:in:notifying:ifFail:.
4.	The Compiler initializes itself and eventually calls the method that does the work, #translate:noPattern:ifFail:needSourceMap:handler:.
a)	The Compiler creates a Parser, which it uses to create a parse tree from the source text.  The Parser works with a ProgramNodeBuilder to create this tree, which consists of instances of the various subclasses of ProgramNode.
b)	The compiler creates a new CodeStream and tells the MethodNode at the root of the parse tree, using the #emitEffect: message, to generate byte codes for its effect into the CodeStream.
c)	The resulting CompiledMethod is packaged into a MethodNodeHolder, which is returned as the result.
5.	The Class obtains a CompiledMethod from the MethodNodeHolder using the message #generate (which is currently trivial, returning the CompiledMethod from step 4c).
6.	The Class updates the ChangeSet, writes the source text out to the change file, sets the CompiledMethod’s sourcePointer, and adds the new method to its MethodDictionary.

Most of the work is done in step 4.  Each sub-step of 4 is a major production point, a place where an important object in the compilation is produced and returned to be used in the next step:  step 4a creates the parse tree, step 4b creates the byte code stream, and step 4c creates the method itself.  Eight classes of objects are created in steps 4a through 4c, but only the Compiler and the Parser are parametrized.  The class of the compiler is specified in  ClassDescription>>compilerClass and the class of the parser in Compiler>>preferredParserClass. This parametrization is currently trivial, since these methods are never (yet) over-ridden.  The other six classes are hard-coded into various methods used in the process, so that Parsers always create ProgramNodeBuilders, which in turn always create instances of ProgramNode; and Compilers always create CodeStreams, which only make CompiledMethods, which Compilers always store in MethodNodeHolders.  Making this process more flexible, so that programmers can define new behavior at each production point, requires that all classes used in the process be parametrized, so that new kinds of collaborators can be easily introduced into the interaction of compiling.  
If we view the three production points of step 4 as partitioning compilation into its major stages, we can think of the four points that lie before, between, and after these production points as intervention points, places where it may (and will) be convenient to massage data to affect the result of compilation.  The first intervention point, before step 4a, can be used to manipulate the input source.  The next can be used to transform the parse tree before it is encoded.  The next one, between steps 4b and 4c, can manipulate the bytecode stream, while the final intervention point can alter the compiled method that is the result of this whole process.  This project only needs to alter two of the production points and one of the intervention points, but in the conclusion we will summarize uses for specializing the other four production and intervention points.
The Smalltalk Compiler Revisited
As you may have anticipated, the new compiler we will describe allows programmers to over-ride behavior at each of the seven production and intervention points, allowing great flexibility in the compilation process.  Our new compiler, ParametrizedCompiler, achieves this flexibility by parametrizing each of the seven classes of objects used during the process of compilation, allowing any subset of these seven classes to be replaced with new specialized classes for user-defined purposes.  Also, the ParametrizedCompiler allows user action at each of the four intervention points by sending messages to its client.  To support this flexibility, the ParametrizedCompiler requires a new collaborator, an object that will supply it with the classes it should use, and that will respond (even if trivially) to the intervention-point messages.  This collaborator will be an instance of the new class MethodProducer.  The class definitions for these two cooperative classes are as follows:

Object subclass: #MethodProducer
	instanceVariableNames: 'sourceCode '
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Projects-Breakpoints'

SmalltalkCompiler subclass: #ParametrizedCompiler
	instanceVariableNames: 'producer parserClass builderClass codeStreamClass methodClass holderClass '
	classVariableNames: ''
	poolDictionaries: ''
	category: 'Projects-Breakpoints'

Every class will respond to the message #methodProducer by returning the default MethodProducer for building the class’ methods, just as they now define their default compiler class.   This class-specific MethodProducer will be responsible for creating a new ParametrizedCompiler, initializing the instance variables that decide what classes are used during compilation, and responding to messages from the compiler at each intervention point.  
MethodProducer and ParametrizedCompiler are introduced into the standard compilation process by changing the compiling messages in ClassDescription (and Behavior, if desired).  We redefine the method ClassDescription>>compile:classified:notifying: (mentioned in step 2 above) and add a new method it will forward to:

compile: code classified: heading notifying: requestor 
	^self
		compile: code
		classified: heading
		notifying: requestor
		producer: (self methodProducerFor: code)

compile: code classified: heading notifying: requestor producer: producer
	^producer
		compile: code
		in: self
		classified: heading
		notifying: requestor

The actual MethodProducer created in the first of the two above methods is provided by implementing two additional methods.

methodProducer
	^MethodProducer new

methodProducerFor: aTextOrString
	^self methodProducer

The first method instantiates a MethodProducer of a type suitable for the class, and classes can over-ride this method to use new subclasses of MethodProducer.  The second method, while trivial now, is an opportunity for a class to use different MethodProducers depending on the method and/or source code to be compiled.
The MethodProducer responds to #compile:in:classified:notifying: with a method similar to the old ClassDescription>>compile:classified:notifying:

compile: code in: aClass classified: aProtocol notifying: aRequestor
	| methodNode selector method |
	sourceCode := code.
	Cursor execute showWhile: [
		methodNode := (self newCompilerFor: aClass)
			compile: code
			in: aClass
			notifying: aRequestor
			ifFail: [^nil].
		selector := methodNode selector.
		method := methodNode generate.
		self storeSource: code method: method class: aClass classified: aProtocol selector: selector.
		aClass addSelector: selector withMethod: method].
	^selector

The differences between this method and the old ClassDescription compilation method are the two self messages, which create the actual compiler and store the newly compiled source, respectively.  The first of these methods is implemented by MethodProducer as:

newCompilerFor: aClass
	^sourceCode hasBreakpoints
		ifTrue: [
			ParametrizedCompiler new
				producer: self;
				parserClass: BreakpointParser;
				builderClass: ProgramNodeBuilder;
				codeStreamClass: ParametrizedCodeStream;
				methodClass: BreakpointMethod;
				holderClass: MethodNodeHolder]
		ifFalse: [
			aClass compilerClass new]

In addition to the new class ParametrizedCompiler, this method references three new classes:  BreakpointParser, ParametrizedCodeStream, and BreakpointMethod.  These three classes are used to change the outcome of two of the production points identified earlier.  BreakpointParser will be used in step 4a to affect the parsing process, while ParametrizedCodeStream and BreakpointMethod will be used in step 4c to change the kind of method produced.  BreakpointParser is a very simple subclass of Parser that re-defines only a single method:  BreakpointParser>>initScanner.  This new method uses a super send to initialize its token type table as usual, and then adds two new entries to treat as white space the two special characters used to denote breakpoints in the method’s display source;.  Thus, the BreakpointParser will be able to scan source that includes breakpoints, treating them like any normal delimiter.  Similarly, ParametrizedCodeStream is a simple extension of CodeStream that adds one instance variable, methodClass, which determines the class of any method produced by the code stream.  By default, this class is set to CompiledMethod at initialization, but can be changed using the message #methodClass:.  BreakpointMethods were discussed previously in connection with the need to distinguish methods’ display source from their stored source.
ParametrizedCompiler is also quite similar to its superclass, SmalltalkCompiler.  It provides accessor methods to set the value of each of the classes used in the compilation process.  In addition, it uses four methods to actually instantiate its co-workers, providing a locus for over-riding in future subclasses.  These four instantiation methods are:

newBuilder
	^builderClass new

newCodeStream
	^codeStreamClass new methodClass: methodClass

newMethodHolder
	^holderClass new

newParser
	^parserClass new

The newCodeStream method assumes that codeStreamClass is either ParametrizedCodeStream or some subtype, since it uses the #methodClass: message to set the desired kind of method to create.  The most significant feature of ParametrizedCompiler is its over-riding of the key worker method of step 4, #translate:noPattern:ifFail:needSourceMap:handler:, as follows:

translate: aStream noPattern: noPattern ifFail: failBlock needSourceMap: mapFlag handler: handler
	| methodNode holder codeStream method  |
	methodNode :=
		self newParser
			parse: aStream
			class: class
			noPattern: noPattern
			context: context
			notifying: handler
			builder: self newBuilder
			saveComments: mapFlag
			ifFail: [^failBlock value].
	methodNode := producer transformTree: methodNode in: class.
	handler selector: methodNode selector.  "save selector in case of error"
	codeStream := self newCodeStream.
	codeStream class restartSignal
		handle: [:ex |
				codeStream := self newCodeStream.
				ex restart]
		do: [codeStream class: targetClass outerScope: self scopeForClass;
					requestor: handler.
			mapFlag ifTrue: [codeStream saveSourceMap].
			noPattern
				ifTrue: [methodNode emitValue: codeStream inContext: context]
				ifFalse: [methodNode emitEffect: codeStream].
			method := codeStream makeMethod: methodNode].
	holder := self newMethodHolder.
	holder node: methodNode.
	holder method: method.
	mapFlag ifTrue: [holder sourceInfo: codeStream sourceInfo].
	^holder

There are two things to note here in contrast to the superclass’ implementation of this method.  First, no classes are referred to by name--they’re all accessed by the instantiation methods shown earlier, so that they can be easily changed and specialized.  Second, there is a new message send just after parsing:  our second intervention point.  (In a similar way, messages can be added at the other three intervention points to allow the compiler’s producer to alter the objects flowing through the production points.)  The producer working with this ParametrizedCompiler is given the newly obtained parse tree and the class it’s being compiled in with the #transformTree:in: message.  The producer is expected to return the parse tree that should be used for code generation.  In the default case, when nothing unusual is needed, this will simply be the input parse tree.  However, if there are breakpoints in this method, the producer uses the opportunity to alter the parse tree accordingly.  We will see how this is achieved after first looking at the last difference between MethodProducer>>compile:in:classified:notifying: and the old ClassDescription>>compile:classified:notifying:.  Where the latter method directly added the newly compiled source code to the change log, our new MethodProducer makes a separate call to store source, which gives us an opportunity to do some special handling, as well as providing an easy way for future specializations of MethodProducer to extend or over-ride.

storeSource: code method: method class: aClass classified: aProtocol selector: selector
	| newSource oldMethod oldSource |
	method displaySource: code.
	newSource := self asStoredSource: code.
	oldMethod := aClass compiledMethodAt: selector ifAbsent: [nil].
	oldSource := oldMethod isNil
		ifTrue: ['']
		ifFalse: [oldMethod getStoredSource].
	newSource ~= oldSource
		ifTrue: [oldMethod notNil
				ifTrue: [ChangeSet current changeSelector: selector class: aClass]
				ifFalse: [ChangeSet current addSelector: selector class: aClass].
			method sourcePointer: 
				(SourceFileManager default
					storeMethodSource: newSource
					class: aClass
					category: aProtocol
					safely: true)]
		ifFalse: [method sourcePointer: oldMethod sourcePointer].
	aClass organization classify: selector under: aProtocol

This code is very similar to the post-compilation processing of ClassDescription>>compile:classified:notifying: and ClassDescription>>compile:classified:.  The two differences are that the source string is written to disk only if it is different from the previous source string, and that the stored source is derived from the display source using the message #asStoredSource:, which strips out any occurrences of Breakpoints.  The result of these two differences is that if a method’s display source is changed only by added or removing breakpoints, its stored source will not have been changed at all, so no change will be entered in the ChangeSet and change log.
To finish off the compilation process, we need finally to look at MethodProducer’s response to the #transformTree:in: message.  The goal is to add new code at every spot where a breakpoint occurs.  This code will be the breakpoint’s source string, which, as we saw before, will either be the fixed message self breakpoint, or an evaluation of the breakpoint’s condition string followed by the same message if that evaluation returns true.  

transformTree: methodNode in: aClass
	| statements eval |
	(self breakpoints asSortedCollection: [ :x :y | x start <= y start]) reverseDo: [ :bp |
		ProgramNodeEvaluator new 
			do: [ :n | 
				((n  isKindOf: SequenceNode) 
				and: [bp start >= n absoluteSourcePosition
				and: [bp stop <= n absoluteEndSourcePosition]])
					ifTrue: [statements := n statements]]
			on: methodNode.
		statements isNil
			ifTrue: [statements := methodNode block body statements].
		(self add: bp toStatements: statements in: aClass) isNil
			ifTrue: [^nil]].
	^methodNode

A parse tree, as returned from the Parser/BreakpointParser in step 4a, is referenced by its root node, which is an instance of class MethodNode.  That node, and all others in the tree, are subinstances of the abstract superclass ProgramNode, which defines the common behavior of all members of a parse tree.  For our purposes, there are three important aspects of this common behavior.
The first is that each ProgramNode has an instance variable named sourcePosition, which is used to indicate the substring of the parsed string to which each ProgramNode corresponds.  For example, the source position of a VariableNode is the index in the source string of the first character in the variable’s name.  The source position of a MessageNode is an Interval from the first character of the selector to the last character in the selector (for a unary message) or the last character defining the argument(s) (for a binary or keyword message).  Finally, the source position of a SequenceNode is an Interval from the first non-whitespace character after any temporaries  (or after the method’s pattern or block opening character if there are no temporaries) to the last character before the method’s end or block closing character.  Using these source position intervals, we can find the nearest SequenceNode enclosing each breakpoint, in order to determine which statement the breakpoint should follow.
While the source position is all you need to determine the right sequence node, it is not enough to determine which statement in the sequence node contains a newly inserted breakpoint.  That is because the source position doesn’t run from statement beginning to end, but just over those characters that will be highlighted during stepping in the debugger.  We overcame this shortcoming by adding a second positional instance variables to all ProgramNodes, which we call statementPosition.  This variable is set during the parsing of a SequenceNode, and it includes all characters in each statement, from the first non-whitespace character to the last one, including the terminating period if present.  Knowing this complete interval for each statement makes it easy to reposition breakpoints to a canonical end position if they are inserted in the middle of a statement.
Finally, each subclass of ProgramNode has a different instance layout. For example, a MethodNode contains a single block as its child in the tree, while a MessageNode has both a receiver and an array of arguments as its children.  This makes it hard to enumerate over a parse tree.  Fortunately, help is provided in the form of ProgramNodeEnumerator, an abstract class that outlines the procedure for enumerating over a parse tree.  We have provided a concrete subclass in the form of ProgramNodeEvaluator, a class that provides the normal block-based enumerations used in the Collection classes, including #do: and #select:.   The ProgramNodeEvaluator and the ProgramNodes interact using a technique called double-dispatching; the evaluator sends the generic #nodeDo: message to each node with the evaluator itself as a parameter, which in turn sends back a class-specific message to the evaluator, such as #doMessage:receiver:selector:arguments: from a MessageNode and #doAssignment:variable:value: from an AssignmentNode.  The evaluator implements these messages by sending itself #doNode: for each node passed to it; in the case of the AssignmentNode, that would mean the variable node and the value node, while in the case of the MessageNode it includes the receiver node and all of the argument nodes.  Finally, the evaluator implements #doNode: by evaluating the block with the ProgramNode parameter as an argument, and then sending #nodeDo: back to that node to continue the enumeration.  The enumeration terminates because the various leaf nodes, such as LiteralNode or VariableNode, have no ProgramNode children for the evaluator to #doNode: on.
With that understanding, we can understand the first half of #transformTree:in:, which is implemented as:

ProgramNodeEvaluator new 
	do: [ :n | 
		((n  isKindOf: SequenceNode) 
		and: [bp start >= n absoluteSourcePosition
		and: [bp stop <= n absoluteEndSourcePosition]])
			ifTrue: [statements := n statements]]
	on: methodNode.
statements isNil
	ifTrue: [statements := methodNode block body statements].

This will iterate over all nodes in the tree rooted at methodNode, evaluating the block for each node.  The condition in the block will evaluate to true if and only if the node is a SequenceNode that contains the breakpoint within its source text, and in that case the temporary variable statements will be set to the OrderedCollection of statements in the SequenceNode.  Furthermore, since sourcePositions never overlap, if there is more than one SequenceNode that contains a breakpoint, they will be nested, and the most deeply nested one, which is to say the most closely enclosing one, will be encountered last in the numeration since the block evaluation is applied in pre-order.  Finally, if there is no enclosing SequenceNode, which could happen if a breakpoint is inserted before or in the midst of the definition of tempories, the statements in the outermost SequenceNode are used to define the breakpoints position.
The latter half of MethodProducer>>transformTree:in: sends another message to actually add the breakpoint code to the collection of statements determined by the ProgramNodeEvaluator.  We will explain the implementation of #add:toStatements:in: a piece at a time.  The method first determines which individual statement contains the breakpoint to be added.

add: bp toStatements: statements in: class
	|  last c stmt index eval stmts |
	last := (c := statements select: [ :s | s statementPosition first < bp start]) isEmpty
		ifTrue: [nil]
		ifFalse: [c last].

It would be nice to simply test which statement’s statementPosition contains the breakpoint’s position, but unfortunately the statementPosition does not include white spaces between statements (and furthermore it would be quite difficult to make it do so).  So, instead, we find all statements that start before the breakpoint is defined, and choose the last of these.  Thus, the breakpoint is either placed in the midst of the statement or in the whitespace immediately following.

	last notNil
		ifTrue: [
			index := last statementPosition last + 1.
			index <= bp start
				ifFalse: [
					bp topParent moveChild: bp toIndex: index.
					bp topParent computeTextPositions]]
		ifFalse: [
			index := statements first statementPosition first.
			index = bp start
				ifFalse: [
					bp topParent moveChild: bp toIndex: index.
					bp topParent computeTextPositions]]

The second step moves any breakpoints that are in the middle of a statement to the end of the statement.  We do this for the sake of consistent debugging:  the breakpoint code will be inserted and evaluated after all of the statement’s code, so it is consistent for the breakpoint to appear after the statement’s text.  If we found a statement that contains the breakpoint (indicated by last being non-nil), we want the breakpoint to be inserted at the statement’s end,  or one more than the last member of its sourcePosition’s Interval.  Then we test to see if the breakpoint’s position is already at or beyond this point:  if it is, the breakpoint is already in the whitespace between this statement and the next; if not, we change the breakpoint’s position within its parent StructuredText and have the StructuredText recalculate text positions accordingly.  However, if last was nil, the breakpoint is being inserted before any statements at all, and so we choose the start of the very first statement as the breakpoint’s canonical position.

	stmts := (self parserClassFor: class) new
				parseBody: bp sourceString readStream
				class: class
				notifying: nil
				ifFail: [^nil].

Now we parse the breakpoint’s source string, use the same parser as for the code that contains this breakpoint.  The class was passed as a parameter in #transformTree:in: and add:toStatements:in: for this purpose, since this parser can depend on the class and its preferred compiler.  If there is a problem parsing this code, which is possible for conditional breakpoints, the method returns nil to MethodProducer>>transformTree:in:, which can indicate an appropriate error condition.  We assume most condition code will be relatively simple, and therefore this situation will not arise frequently.  If it does, you can add more elaborate error handling, using a non-nil requestor to the parsing, to be able to report more fully whatever error occurs.

	stmt := stmts at: 1.
	ProgramNodeEvaluator new 
		do: [ :n | 
			n sourcePosition: (bp start to: bp stop);
				statementPosition: (bp start to: bp stop)].
		on: stmt.
	stmt receiver
		sourcePosition: nil.

This step adjusts the sourcePosition and statementPosition of the ProgramNodes generated from the breakpoint’s source string so that they point only to the breakpoint’s character position in the method’s source string.  Otherwise, these nodes would refer to indexes that don’t make sense in the method’s source string.  Also, we set the sourcePosition of a conditional breakpoint’s condition block to nil, so that it won’t be present in the sourceMap used for debugging.  The result of this will be that every breakpoint, whether conditional or absolute, will only require a single step to process in the debugger, making the interface uniform.

	last isNil
		ifTrue: [statements addFirst: stmt]
		ifFalse: [statements add: stmt after: last]

The final step in #add:toStatements:in: is to add the newly produced ProgramNode into the collection of statements that was passed in.  We add it after the statement that contained the breakpoint’s insertion point, if there was one, or else before any other statement.  This mechanism will let you add breakpoint code after a return node, since that error is only detected during parsing.  This will not cause any errors, though of course the breakpoint will never be activated.  It would be possible to alert the user if this happens, so they won’t be surprised by breakpoints that don’t fire when it seems they should.
Conclusion
These new breakpoints improve significantly on the old version, and, when combined with the lightweight classes of our previous article, significantly extend the programmer’s options and opportunities when debugging.  The improvements provided can best be summarized as locality enhancements:  the ability to place breakpoints anywhere throughout a method, and to have them occur only under certain conditions, makes it easier for the programmer to investigate close to the locality of a suspected problem, without having to wade through a lot of extraneous debugging.  (The same thing is true of the usefulness of lightweight classes in debugging, since they help narrow the focus of debugging to the locality of individual objects.)
In addition to this help in debugging, this project again demonstrates the large degree of openness in the Smalltalk-80 environment, and the usefulness of allowing programmers reflective access to the environment.  The central work described in this article is the creation of the new tandem of MethodProducer and ParametrizedCompiler, and our ability to create them and use them to produce breakpointed methods relies on the existence of all objects used in the compilation process as first-class objects, ones that can be examined, extended, and changed by the programmer.  The pair of producer and parametrized compiler have many uses beyond breakpoints, and we intend to bring those out in future articles, starting with (as we in fact promised in our last article) a look at the creation of active variables and their use in defining watchpoints in the debugger.

References

[1] Bob Hinkle, Vicki Jones, and Ralph E. Johnson. Debugging Objects. The Smalltalk Report, Vol. 2, Number 9, July-August 1993.

[2] VisualWorks User’s Guide.  ParcPlace Systems, Inc., 1994.

[3] Bob Hinkle and Ralph E. Johnson. Taking Exception to Smalltalk, Part 1. The Smalltalk Report, Vol. 2, Number 3, November/December 1992.

[4] Bob Hinkle and Ralph E. Johnson. Taking Exception to Smalltalk, Part 2. The Smalltalk Report, Vol. 2, Number 4, January 1993.