evaluate a model More...

Inheritance diagram for Python.Markopy.Evaluation.ModelEvaluator:

Collaboration diagram for Python.Markopy.Evaluation.ModelEvaluator:

Public Member Functions
None	__init__ (self, str filename)
	default constructor for evaluator More...

def	evaluate (self)

def	check_dangling (self)

def	check_structure (self)

def	check_weight_deviation (self)

def	check_min (self)

def	check_min_10percent (self)

def	check_lean (self)

def	check_distrib (self)

def	success (self, checkname)
	pass a test More...

def	fail (self, checkname)
	fail a test More...

def	finalize (self)

Public Attributes
	lnodes

	rnodes

	ews

	edge_count

	lnode_count

	rnode_count

	stdev

	filename

	checks

	TEST_PASS_SYMBOL

	TEST_FAIL_SYMBOL

	all_checks_passed

	files

	check_funcs

Private Member Functions
list	_evaluate (self, file)
	internal evaluation function for a single file More...

Detailed Description

evaluate a model

Definition at line 90 of file evaluate.py.

Constructor & Destructor Documentation

◆ init()

None Python.Markopy.Evaluation.ModelEvaluator.__init__	(		self,
		str	filename
	)

default constructor for evaluator

Parameters

filename filename to evaluate. Can be a pattern

Reimplemented from Python.Markopy.Evaluation.Evaluator.

Definition at line 96 of file evaluate.py.

     def __init__(self, filename: str) -> None:
         "! @brief default constructor"
         valid = super().__init__(filename)
         
         if not valid:
             return False
  

Member Function Documentation

◆ _evaluate()

list Python.Markopy.Evaluation.Evaluator._evaluate	(	self,
		file
	)

privateinherited

internal evaluation function for a single file

Parameters

file	filename to evaluate

Reimplemented in Python.Markopy.Evaluation.CorpusEvaluator.

Definition at line 52 of file evaluate.py.

     def _evaluate(self, file) -> list:
         """! 
         @brief internal evaluation function for a single file
         @param file filename to evaluate
         """
         if(not os.path.isfile(file)):
             logging.pprint(f"Given file {file} is not a valid filename")
             return False
         else:
              return open(file, "rb").read().split(b"\n")
         
         
  

Referenced by Python.Markopy.Evaluation.Evaluator.evaluate(), and Python.Markopy.Evaluation.ModelEvaluator.evaluate().

Here is the caller graph for this function:

◆ check_dangling()

def Python.Markopy.Evaluation.ModelEvaluator.check_dangling ( self )

Definition at line 151 of file evaluate.py.

     def check_dangling(self):
         "! @brief check if model has dangling nodes"
         if(self.lnode_count == self.rnode_count):
             self.success("No dangling nodes")
         else:
             logging.pprint(f"Dangling nodes found, lnodes and rnodes do not match", 0)
             self.fail("No dangling nodes")
     

References Python.Markopy.Evaluation.Evaluator.fail(), Python.Markopy.Evaluation.ModelEvaluator.lnode_count, Python.Markopy.Evaluation.ModelEvaluator.rnode_count, and Python.Markopy.Evaluation.Evaluator.success().

Here is the call graph for this function:

◆ check_distrib()

def Python.Markopy.Evaluation.ModelEvaluator.check_distrib ( self )

Definition at line 228 of file evaluate.py.

     def check_distrib(self):
         "! @deprecated"
         sorted_ews = copy(self.ews)
         sorted_ews.sort(reverse=True)
         ratio1 = sorted_ews[0]/sorted_ews[int(self.edge_count/2)]
         ratio2 = sorted_ews[int(self.edge_count/2)]/sorted_ews[int(self.edge_count*0.1)]
         #print(ratio1)
         #print(ratio2)
  
  

References Python.Markopy.Evaluation.ModelEvaluator.edge_count, and Python.Markopy.Evaluation.ModelEvaluator.ews.

◆ check_lean()

def Python.Markopy.Evaluation.ModelEvaluator.check_lean ( self )

Definition at line 207 of file evaluate.py.

     def check_lean(self):
         "! @brief check which way model is leaning. Left, or right"
         sample = self.ews[int(self.edge_count*0.1)]
         avg = sum(self.ews) / len(self.ews)
         med = statistics.median(self.ews)
         
         if(med*10<sample):
             logging.pprint("Median is too left leaning and might indicate high entropy")
             self.fail("Median too left leaning")
         else:
             self.success("Median in expected ratio")
         pass
  
         if(sample*5>avg):
             logging.pprint("Least probable 10% too close to average, might indicate inadequate training")
             self.fail("Bad bottom 10%")
         else:
             self.success("Good bottom 10%")
         pass
         
  

References Python.Markopy.Evaluation.ModelEvaluator.edge_count, Python.Markopy.Evaluation.ModelEvaluator.ews, Python.Markopy.Evaluation.Evaluator.fail(), and Python.Markopy.Evaluation.Evaluator.success().

Here is the call graph for this function:

◆ check_min()

def Python.Markopy.Evaluation.ModelEvaluator.check_min ( self )

Definition at line 186 of file evaluate.py.

     def check_min(self):
         "! @brief check 0 edge weights distribution"
         count = 0
         for ew in self.ews:
             if ew==0:
                 count+=1
         if(count > self.rnode_count*0.8):
             self.fail("Too many 0 edges")
             logging.pprint(f"0 weighted edges are dangerous and may halt the model.", 0)
         else:
             self.success("0 edges below threshold")
     

References Python.Markopy.Evaluation.ModelEvaluator.ews, Python.Markopy.Evaluation.Evaluator.fail(), Python.Markopy.Evaluation.ModelEvaluator.rnode_count, and Python.Markopy.Evaluation.Evaluator.success().

Here is the call graph for this function:

◆ check_min_10percent()

def Python.Markopy.Evaluation.ModelEvaluator.check_min_10percent ( self )

Definition at line 198 of file evaluate.py.

     def check_min_10percent(self):
         "! @brief check minimum 10% of the edges"
         sample = self.ews[int(self.edge_count*0.1)]
         #print(f"10per: {sample}")
         avg = sum(self.ews) / len(self.ews)
         #print(f"avg: {avg}")
         med = statistics.median(self.ews)
         #print(f"med: {med}")
  

References Python.Markopy.Evaluation.ModelEvaluator.edge_count, and Python.Markopy.Evaluation.ModelEvaluator.ews.

◆ check_structure()

def Python.Markopy.Evaluation.ModelEvaluator.check_structure ( self )

Definition at line 159 of file evaluate.py.

     def check_structure(self):
         "! @brief check model structure for validity"
         if((self.lnode_count-1) * (self.rnode_count-1) + 2*(self.lnode_count-1)):  
             self.success("Model structure")
         else:
             logging.pprint(f"Model did not satisfy structural integrity check (lnode_count-1) * (rnode_count-1) + 2*(lnode_count-1)", 0)
             self.fail("Model structure")
  

References Python.Markopy.Evaluation.Evaluator.fail(), Python.Markopy.Evaluation.ModelEvaluator.lnode_count, Python.Markopy.Evaluation.ModelEvaluator.rnode_count, and Python.Markopy.Evaluation.Evaluator.success().

Here is the call graph for this function:

◆ check_weight_deviation()

def Python.Markopy.Evaluation.ModelEvaluator.check_weight_deviation ( self )

Definition at line 167 of file evaluate.py.

     def check_weight_deviation(self):
         "! @brief check model standart deviation between edge weights"
         mean = sum(self.ews) / len(self.ews)
         variance = sum([((x - mean) ** 2) for x in self.ews]) / len(self.ews)
         res = variance ** 0.5
         self.stdev = res
         if(res==0):
             logging.pprint(f"Model seems to be untrained", 0)
             self.fail("Model has any training")
         else:
             self.success("Model has any training")
         if(res<3000):
             logging.pprint(f"Model is not adequately trained. Might result in inadequate results", 1)
             self.fail("Model has training")
             self.fail(f"Model training score: {round(self.stdev,2)}")
         else:
             self.success("Model has training")
             self.success(f"Model training score: {round(self.stdev)}")
  

References Python.Markopy.Evaluation.ModelEvaluator.ews.

◆ evaluate()

def Python.Markopy.Evaluation.ModelEvaluator.evaluate ( self )

Reimplemented from Python.Markopy.Evaluation.Evaluator.

Definition at line 103 of file evaluate.py.

     def evaluate(self):
         "! @brief evaluate a model"
         logging.VERBOSITY=2
         logging.SHOW_STACK_THRESHOLD=3
         super().evaluate()
         for file in self.files:
             logging.pprint(f"Model: {file.split('/')[-1]}: ",2)
             edges = super()._evaluate(file)
             if not edges:
                 continue
             self.lnodes = {}
             self.rnodes = {}
             self.ews = []
             self.edge_count = len(edges)
             for edge in edges:
                 if(edge ==b''):
                     self.edge_count-=1
                     continue
                 try:
                     e = edge.split(b',')
                     self.ews.append(int(edge[2:-2:1]))
                     if(e[0] not in self.lnodes):
                         self.lnodes[e[0]]=1
                     else:
                         self.lnodes[e[0]]+=1
                     if(e[-1] not in self.rnodes):
                         self.rnodes[e[-1]]=1
                     else:
                         self.rnodes[e[-1]]+=1
                 except Exception as e:
                     print(e)
                     logging.pprint(f"Model file is corrupted.", 0)
                     continue
         
             self.lnode_count = len(self.lnodes)
             self.rnode_count = len(self.rnodes)
             logging.pprint(f"total edges: {self.edge_count}", 1)
             logging.pprint(f"unique left nodes: {self.lnode_count}", 1)
             logging.pprint(f"unique right nodes: {self.rnode_count}", 1)
  
             for check in self.check_funcs:
                 try:
                     self.__getattribute__(check)()
                 except Exception as e:
                     print(e)
                     self.fail(f"Exceptionn in {check}")
             self.finalize()
         

References Python.Markopy.Evaluation.Evaluator._evaluate(), and Python.Markopy.Evaluation.Evaluator.files.

Referenced by Python.Markopy.MarkopyCLI.parse().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ fail()

def Python.Markopy.Evaluation.Evaluator.fail	(	self,
		checkname
	)

inherited

fail a test

Parameters

checkname text to display with the check

Definition at line 72 of file evaluate.py.

     def fail(self, checkname):
         """!
         @brief fail a test
         @param checkname text to display with the check
         """
  
         self.all_checks_passed = False
         self.checks.append((checkname, self.TEST_FAIL_SYMBOL))
  

References Python.Markopy.Evaluation.Evaluator.all_checks_passed, Python.Markopy.Evaluation.Evaluator.checks, and Python.Markopy.Evaluation.Evaluator.TEST_FAIL_SYMBOL.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_dangling(), Python.Markopy.Evaluation.ModelEvaluator.check_lean(), Python.Markopy.Evaluation.ModelEvaluator.check_min(), Python.Markopy.Evaluation.ModelEvaluator.check_structure(), and Python.Markopy.Evaluation.CorpusEvaluator.evaluate().

Here is the caller graph for this function:

◆ finalize()

def Python.Markopy.Evaluation.Evaluator.finalize ( self )

inherited

Definition at line 81 of file evaluate.py.

     def finalize(self):
         "! @brief finalize an evaluation and print checks"
         print("\n################ Checks ################ ")
         for test in self.checks:
             logging.pprint(f"{test[0]:30}:{test[1]} ")
         print("\n")
         self.checks = []
         return self.all_checks_passed
  

References Python.Markopy.Evaluation.Evaluator.all_checks_passed, and Python.Markopy.Evaluation.Evaluator.checks.

Referenced by Python.Markopy.Evaluation.CorpusEvaluator.evaluate().

Here is the caller graph for this function:

◆ success()

def Python.Markopy.Evaluation.Evaluator.success	(	self,
		checkname
	)

inherited

pass a test

Parameters

checkname text to display with the check

Definition at line 65 of file evaluate.py.

     def success(self, checkname):
         """!
         @brief pass a test
         @param checkname text to display with the check
         """
         self.checks.append((checkname, self.TEST_PASS_SYMBOL))
  

References Python.Markopy.Evaluation.Evaluator.checks, and Python.Markopy.Evaluation.Evaluator.TEST_PASS_SYMBOL.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_dangling(), Python.Markopy.Evaluation.ModelEvaluator.check_lean(), Python.Markopy.Evaluation.ModelEvaluator.check_min(), Python.Markopy.Evaluation.ModelEvaluator.check_structure(), and Python.Markopy.Evaluation.CorpusEvaluator.evaluate().

Here is the caller graph for this function:

Member Data Documentation

◆ all_checks_passed

Python.Markopy.Evaluation.Evaluator.all_checks_passed

inherited

Definition at line 36 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.Evaluator.fail(), and Python.Markopy.Evaluation.Evaluator.finalize().

◆ check_funcs

Python.Markopy.Evaluation.Evaluator.check_funcs

inherited

Definition at line 49 of file evaluate.py.

◆ checks

Python.Markopy.Evaluation.Evaluator.checks

inherited

Definition at line 33 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.Evaluator.fail(), Python.Markopy.Evaluation.Evaluator.finalize(), and Python.Markopy.Evaluation.Evaluator.success().

◆ edge_count

Python.Markopy.Evaluation.ModelEvaluator.edge_count

Definition at line 116 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_distrib(), Python.Markopy.Evaluation.ModelEvaluator.check_lean(), and Python.Markopy.Evaluation.ModelEvaluator.check_min_10percent().

◆ ews

Python.Markopy.Evaluation.ModelEvaluator.ews

Definition at line 115 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_distrib(), Python.Markopy.Evaluation.ModelEvaluator.check_lean(), Python.Markopy.Evaluation.ModelEvaluator.check_min(), Python.Markopy.Evaluation.ModelEvaluator.check_min_10percent(), and Python.Markopy.Evaluation.ModelEvaluator.check_weight_deviation().

◆ filename

Python.Markopy.Evaluation.Evaluator.filename

inherited

Definition at line 32 of file evaluate.py.

◆ files

Python.Markopy.Evaluation.Evaluator.files

inherited

Definition at line 37 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.Evaluator.evaluate(), Python.Markopy.Evaluation.ModelEvaluator.evaluate(), and Python.Markopy.Evaluation.CorpusEvaluator.evaluate().

◆ lnode_count

Python.Markopy.Evaluation.ModelEvaluator.lnode_count

Definition at line 137 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_dangling(), and Python.Markopy.Evaluation.ModelEvaluator.check_structure().

◆ lnodes

Python.Markopy.Evaluation.ModelEvaluator.lnodes

Definition at line 113 of file evaluate.py.

◆ rnode_count

Python.Markopy.Evaluation.ModelEvaluator.rnode_count

Definition at line 138 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.ModelEvaluator.check_dangling(), Python.Markopy.Evaluation.ModelEvaluator.check_min(), and Python.Markopy.Evaluation.ModelEvaluator.check_structure().

◆ rnodes

Python.Markopy.Evaluation.ModelEvaluator.rnodes

Definition at line 114 of file evaluate.py.

◆ stdev

Python.Markopy.Evaluation.ModelEvaluator.stdev

Definition at line 172 of file evaluate.py.

◆ TEST_FAIL_SYMBOL

Python.Markopy.Evaluation.Evaluator.TEST_FAIL_SYMBOL

inherited

Definition at line 35 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.Evaluator.fail().

◆ TEST_PASS_SYMBOL

Python.Markopy.Evaluation.Evaluator.TEST_PASS_SYMBOL

inherited

Definition at line 34 of file evaluate.py.

Referenced by Python.Markopy.Evaluation.Evaluator.success().

The documentation for this class was generated from the following file:

Markopy/Markopy/src/CLI/evaluate.py

Public Member Functions

Public Attributes

Private Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ _evaluate()

◆ check_dangling()

◆ check_distrib()

◆ check_lean()

◆ check_min()

◆ check_min_10percent()

◆ check_structure()

◆ check_weight_deviation()

◆ evaluate()

◆ fail()

◆ finalize()

◆ success()

Member Data Documentation

◆ all_checks_passed

◆ check_funcs

◆ checks

◆ edge_count

◆ ews

◆ filename

◆ files

◆ lnode_count

◆ lnodes

◆ rnode_count

◆ rnodes

◆ stdev

◆ TEST_FAIL_SYMBOL

◆ TEST_PASS_SYMBOL

◆ init()