This is my solution to the fourth part of "A Game of Tokens", which can be found here.
You can find the tests and the code presented here in this repository in the branch called part4.
Level 15 - Exponentiation
Lexer
The lexer can process the exponentiation operator ^ out of the box as a LITERAL token, so no changes to the code are needed.
Parser
The test test_parse_exponentiation() can be passed adding a PowerNode class
classPowerNode(BinaryNode):node_type='exponentiation'and a parse_exponentiation() method to the parser
defparse_exponentiation(self):left=self.parse_factor()next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.value=='^':operator=self._parse_symbol()right=self.parse_exponentiation()returnPowerNode(left,operator,right)returnleftThis allows the parser to explicitly parse the exponentiation operation, but when the operation is mixed with others the parser doesn't know how to deal with it, as parse_exponentiation() is not called by any other method.
To pass the test_parse_exponentiation_with_other_operators() test we need to change the parse_term() method and call parse_exponentiation() instead of parse_factor()
defparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
andnext_token.valuein['*','/']:operator=self._parse_symbol()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)the full code of the CalcParser class is now
classCalcParser:def__init__(self):self.lexer=clex.CalcLexer()def_parse_symbol(self):t=self.lexer.get_token()returnLiteralNode(t.value)defparse_integer(self):t=self.lexer.get_token()returnIntegerNode(t.value)def_parse_variable(self):t=self.lexer.get_token()returnVariableNode(t.value)defparse_factor(self):next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.valuein['-','+']:operator=self._parse_symbol()factor=self.parse_factor()returnUnaryNode(operator,factor)ifnext_token.type==clex.LITERALandnext_token.value=='(':self.lexer.discard_type(clex.LITERAL)expression=self.parse_expression()self.lexer.discard_type(clex.LITERAL)returnexpressionifnext_token.type==clex.NAME:t=self.lexer.get_token()returnVariableNode(t.value)returnself.parse_integer()defparse_exponentiation(self):left=self.parse_factor()next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.value=='^':operator=self._parse_symbol()right=self.parse_exponentiation()returnPowerNode(left,operator,right)returnleftdefparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
andnext_token.valuein['*','/']:operator=self._parse_symbol()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleftdefparse_expression(self):left=self.parse_term()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
andnext_token.valuein['+','-']:operator=self._parse_symbol()right=self.parse_term()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleftdefparse_assignment(self):variable=self._parse_variable()self.lexer.discard(token.Token(clex.LITERAL,'='))value=self.parse_expression()returnAssignmentNode(variable,value)defparse_line(self):try:self.lexer.stash()returnself.parse_assignment()exceptclex.TokenError:self.lexer.pop()returnself.parse_expression()Visitor
The given test test_visitor_exponentiation() requires the CalcVisitor to parse nodes of type exponentiation. The code required to do this is
ifnode['type']=='exponentiation':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])returnlvalue**rvalue,ltypeas a final case for the CalcVisitor class. The full code of the class is is now
classCalcVisitor:def__init__(self):self.variables={}defisvariable(self,name):returnnameinself.variablesdefvalueof(self,name):returnself.variables[name]['value']deftypeof(self,name):returnself.variables[name]['type']defvisit(self,node):ifnode['type']=='integer':returnnode['value'],node['type']ifnode['type']=='variable':returnself.valueof(node['value']),self.typeof(node['value'])ifnode['type']=='unary':operator=node['operator']['value']cvalue,ctype=self.visit(node['content'])ifoperator=='-':return-cvalue,ctypereturncvalue,ctypeifnode['type']=='binary':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])operator=node['operator']['value']ifoperator=='+':returnlvalue+rvalue,rtypeelifoperator=='-':returnlvalue-rvalue,rtypeelifoperator=='*':returnlvalue*rvalue,rtypeelifoperator=='/':returnlvalue//rvalue,rtypeifnode['type']=='assignment':right_value,right_type=self.visit(node['value'])self.variables[node['variable']]={'value':right_value,'type':right_type}returnNone,NoneLevel 16 - Float numbers
Lexer
The first thing the lexer need is a label to identify FLOAT tokens
FLOAT='FLOAT'then the method _process_integer() cna be extended to process float numbers as well. To do this the method is renamed to _process_number(), the regular expression is modified, and the token_type is managed according to the presence of the dot.
def_process_number(self):regexp=re.compile('[\d\.]+')match=regexp.match(self._text_storage.tail)ifnotmatch:returnNonetoken_string=match.group()token_type=FLOATif'.'intoken_stringelseINTEGERreturnself._set_current_token_and_skip(token.Token(token_type,token_string))Remember that the get_token() function has to be modified to use the new name of the method. The new code is
defget_token(self):eof=self._process_eof()ifeof:returneofeol=self._process_eol()ifeol:returneolself._process_whitespace()name=self._process_name()ifname:returnnameinteger=self._process_number()ifinteger:returnintegerliteral=self._process_literal()ifliteral:returnliteralParser
First we need to add a new type of node
classFloatNode(ValueNode):node_type='float'The new version of parse_integer(), renamed parse_number(), shall deal with both cases but also raise the TokenError exception if the parsing fails
defparse_number(self):t=self.lexer.get_token()ift.type==clex.INTEGER:returnIntegerNode(int(t.value))elift.type==clex.FLOAT:returnFloatNode(float(t.value))raiseclex.TokenErrorVisitor
The change to support float nodes is trivial, we just need to include it alongside with the integer case
defvisit(self,node):ifnode['type']in['integer','float']:returnnode['value'],node['type']To fix the missing type promotion when dealing with integers and floats it's enough to add
ifltype=='float':rtype=ltypejust before evaluating the operator in the binary nodes. The full code of the visit() method is then
defvisit(self,node):ifnode['type']in['integer','float']:returnnode['value'],node['type']ifnode['type']=='variable':returnself.valueof(node['value']),self.typeof(node['value'])ifnode['type']=='unary':operator=node['operator']['value']cvalue,ctype=self.visit(node['content'])ifoperator=='-':return-cvalue,ctypereturncvalue,ctypeifnode['type']=='binary':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])operator=node['operator']['value']ifltype=='float':rtype=ltypeifoperator=='+':returnlvalue+rvalue,rtypeelifoperator=='-':returnlvalue-rvalue,rtypeelifoperator=='*':returnlvalue*rvalue,rtypeelifoperator=='/':returnlvalue//rvalue,rtypeifnode['type']=='assignment':right_value,right_type=self.visit(node['value'])self.variables[node['variable']]={'value':right_value,'type':right_type}returnNone,Noneifnode['type']=='exponentiation':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])returnlvalue**rvalue,ltypeFinal words
I bet at this point of the challenge the addition of exponentiation and float numbers wasn't that hard. The refactoring might have been a bit thougher, however, but I hope that it showed you the real power of TDD.
Feedback
Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.