Python.Markopy.ModelMatrixCLI Class Reference

Extension of Python.Markopy.Base.BaseCLI for Markov::API::ModelMatrix. More...

Inheritance diagram for Python.Markopy.ModelMatrixCLI:

Collaboration diagram for Python.Markopy.ModelMatrixCLI:

Public Member Functions
def	__init__ (self, bool add_help=True)
	initialize base CLI More...
def	add_arguments (self)
def	init_post_arguments (self)
def	help (self)
def	parse (self)
def	parse_arguments (self)
def	import_model (self, str filename)
	Import a model file. More...
def	train (self, str dataset, str seperator, str output, bool output_forced=False, bool bulk=False)
	Train a model via CLI parameters. More...
def	export (self, str filename)
	Export model to a file. More...
def	generate (self, str wordlist, bool bulk=False)
	Generate strings from the model. More...
def	process (self)
	Process parameters for operation. More...
def	FastRandomWalk (int count, str wordlist, int minlen, int maxlen)
int	FastRandomWalk (unsigned long int n, const char *wordlistFileName, int minLen=6, int maxLen=12, int threads=20, bool bFileIO=true)
	Random walk on the Matrix-reduced Markov::Model. More...
bool	ConstructMatrix ()
	Construct the related Matrix data for the model. More...
void	DumpJSON ()
	Debug function to dump the model to a JSON file. More...
void	Import (const char *filename)
	Open a file to import with filename, and call bool Model::Import with std::ifstream. More...
bool	Import (std::ifstream *)
	Import a file to construct the model. More...
void	Train (const char *datasetFileName, char delimiter, int threads)
	Train the model with the dataset file. More...
std::ifstream *	OpenDatasetFile (const char *filename)
	Open dataset file and return the ifstream pointer. More...
std::ofstream *	Save (const char *filename)
	Export model to file. More...
void	Generate (unsigned long int n, const char *wordlistFileName, int minLen=6, int maxLen=12, int threads=20)
	Call Markov::Model::RandomWalk n times, and collect output. More...
void	Buff (const char *str, double multiplier, bool bDontAdjustSelfLoops=true, bool bDontAdjustExtendedLoops=false)
	Buff expression of some characters in the model. More...
char *	RandomWalk (Markov::Random::RandomEngine *randomEngine, int minSetting, int maxSetting, char *buffer)
	Do a random walk on this model. More...
void	AdjustEdge (const char *payload, long int occurrence)
	Adjust the model with a single string. More...
bool	Export (std::ofstream *)
	Export a file of the model. More...
bool	Export (const char *filename)
	Open a file to export with filename, and call bool Model::Export with std::ofstream. More...
Node< char > *	StarterNode ()
	Return starter Node. More...
std::vector< Edge< char > * > *	Edges ()
	Return a vector of all the edges in the model. More...
std::map< char, Node< char > * > *	Nodes ()
	Return starter Node. More...
void	OptimizeEdgeOrder ()
	Sort edges of all nodes in the model ordered by edge weights. More...

Static Public Member Functions
def	check_import_path (str filename)
	check import path for validity More...
def	check_corpus_path (str filename)
	check import path for validity More...
def	check_export_path (str filename)
	check import path for validity More...

Public Attributes
	model
	fileIO
	parser
	print_help
	args

Protected Member Functions
int	FastRandomWalk (unsigned long int n, std::ofstream *wordlist, int minLen=6, int maxLen=12, int threads=20, bool bFileIO=true)
	Random walk on the Matrix-reduced Markov::Model. More...
void	FastRandomWalkPartition (std::mutex *mlock, std::ofstream *wordlist, unsigned long int n, int minLen, int maxLen, bool bFileIO, int threads)
	A single partition of FastRandomWalk event. More...
void	FastRandomWalkThread (std::mutex *mlock, std::ofstream *wordlist, unsigned long int n, int minLen, int maxLen, int id, bool bFileIO)
	A single thread of a single partition of FastRandomWalk. More...
bool	DeallocateMatrix ()
	Deallocate matrix and make it ready for re-construction. More...

Protected Attributes
char **	edgeMatrix
	2-D Character array for the edge Matrix (The characters of Nodes) More...
long int **	valueMatrix
	2-d Integer array for the value Matrix (For the weights of Edges) More...
int	matrixSize
	to hold Matrix size More...
char *	matrixIndex
	to hold the Matrix index (To hold the orders of 2-D arrays') More...
long int *	totalEdgeWeights
	Array of the Total Edge Weights. More...
bool	ready
	True when matrix is constructed. False if not. More...

Private Member Functions
def	_generate (self, str wordlist)
	wrapper for generate function. More...
void	TrainThread (Markov::API::Concurrency::ThreadSharedListHandler *listhandler, char delimiter)
	A single thread invoked by the Train function. More...
void	GenerateThread (std::mutex *outputLock, unsigned long int n, std::ofstream *wordlist, int minLen, int maxLen)
	A single thread invoked by the Generate function. More...

Private Attributes
std::ifstream *	datasetFile
std::ofstream *	modelSavefile
	Dataset file input of our system More...
std::ofstream *	outputFile
	File to save model of our system More...
std::map< char, Node< char > * >	nodes
	Map LeftNode is the Nodes NodeValue Map RightNode is the node pointer. More...
Node< char > *	starterNode
	Starter Node of this model. More...
std::vector< Edge< char > * >	edges
	A list of all edges in this model. More...

Detailed Description

Extension of Python.Markopy.Base.BaseCLI for Markov::API::ModelMatrix.

adds -st/–stdout arguement to the command line.

Definition at line 18 of file mmx.py.

Constructor & Destructor Documentation

__init__()

def Python.Markopy.ModelMatrixCLI.__init__	(		self,
		bool	add_help = True
	)

initialize base CLI

Parameters

add_help

decide to overload the help function or not

Reimplemented from Python.Markopy.BaseCLI.

Definition at line 27 of file mmx.py.

    def __init__(self, add_help:bool=True):
        "! @brief initialize model with Markov::API::ModelMatrix"
        super().__init__(add_help)
        self.model = markopy.ModelMatrix()
 

Member Function Documentation

_generate()

def Python.Markopy.ModelMatrixCLI._generate (   self, str  wordlist  )

private

wrapper for generate function.

This can be overloaded by other models

Parameters

wordlist

filename to generate to

Reimplemented from Python.Markopy.BaseCLI.

Reimplemented in Python.CudaMarkopy.CudaModelMatrixCLI.

Definition at line 40 of file mmx.py.

    def _generate(self, wordlist : str, ):
        self.model.FastRandomWalk(int(self.args.count), wordlist, int(self.args.min), int(self.args.max), int(self.args.threads), self.fileIO)
 

References Python.CudaMarkopy.CudaMarkopyCLI.args, Python.Markopy.BaseCLI.args, Python.Markopy.MarkopyCLI.args, Python.Markopy.ModelMatrix.FastRandomWalk(), Python.Markopy.ModelMatrixCLI.fileIO, Python.CudaMarkopy.CudaModelMatrixCLI.model, Python.Markopy.BaseCLI.model, Python.Markopy.ModelMatrixCLI.model, Python.Markopy.MarkovPasswordsCLI.model, and Markov::GUI::MarkovPasswordsGUI.model().

Referenced by Python.Markopy.BaseCLI.generate().

Here is the call graph for this function:

Here is the caller graph for this function:

add_arguments()

def Python.Markopy.ModelMatrixCLI.add_arguments

(

self

)

Reimplemented from Python.Markopy.AbstractGenerationModelCLI.

Reimplemented in Python.Markopy.MarkopyCLI, and Python.CudaMarkopy.CudaModelMatrixCLI.

Definition at line 32 of file mmx.py.

    def add_arguments(self):
        super().add_arguments()
        self.parser.add_argument("-st", "--stdout", action="store_true", help="Stdout mode")
    

References Python.Markopy.BaseCLI.parser.

Referenced by Python.Markopy.BaseCLI.parse(), and Python.Markopy.MarkopyCLI.parse().

Here is the caller graph for this function:

AdjustEdge()

void Markov::Model< char >::AdjustEdge ( const NodeStorageType *  payload, long int  occurrence  )

inherited

Adjust the model with a single string.

Start from the starter node, and for each character, AdjustEdge the edge EdgeWeight from current node to the next, until NULL character is reached.

Then, update the edge EdgeWeight from current node, to the terminator node.

This function is used for training purposes, as it can be used for adjusting the model with each line of the corpus file.

Example Use: Create an empty model and train it with string: "testdata"

Markov::Model<char> model;
char test[] = "testdata";
model.AdjustEdge(test, 15); 

Parameters

string	- String that is passed from the training, and will be used to AdjustEdge the model with
occurrence	- Occurrence of this string.

Definition at line 109 of file model.h.

                                                                                                 {
    NodeStorageType p = payload[0];
    Markov::Node<NodeStorageType>* curnode = this->starterNode;
    Markov::Edge<NodeStorageType>* e;
    int i = 0;
 
    if (p == 0) return;
    while (p != 0) {
        e = curnode->FindEdge(p);
        if (e == NULL) return;
        e->AdjustEdge(occurrence);
        curnode = e->RightNode();
        p = payload[++i];
    }
 
    e = curnode->FindEdge('\xff');
    e->AdjustEdge(occurrence);
    return;
}

Buff()

void Markov::API::MarkovPasswords::Buff ( const char *  str, double  multiplier, bool  bDontAdjustSelfLoops = true, bool  bDontAdjustExtendedLoops = false  )

inherited

Buff expression of some characters in the model.

Parameters

str	A string containing all the characters to be buffed
multiplier	A constant value to buff the nodes with.
bDontAdjustSelfEdges	Do not adjust weights if target node is same as source node
bDontAdjustExtendedLoops	Do not adjust if both source and target nodes are in first parameter

Definition at line 153 of file markovPasswords.cpp.

                                                                                                                               {
    std::string buffstr(str);
    std::map< char, Node< char > * > *nodes;
    std::map< char, Edge< char > * > *edges;
    nodes = this->Nodes();
    int i=0;
    for (auto const& [repr, node] : *nodes){
        edges = node->Edges();
        for (auto const& [targetrepr, edge] : *edges){
            if(buffstr.find(targetrepr)!= std::string::npos){
                if(bDontAdjustSelfLoops && repr==targetrepr) continue;
                if(bDontAdjustExtendedLoops){
                    if(buffstr.find(repr)!= std::string::npos){
                        continue;
                    }
                }
                long int weight = edge->EdgeWeight();
                weight = weight*multiplier;     
                edge->AdjustEdge(weight);
            }
 
        }
        i++;
    }
 
    this->OptimizeEdgeOrder();
}

References Markov::Edge< NodeStorageType >::AdjustEdge(), Markov::Node< storageType >::Edges(), Markov::Edge< NodeStorageType >::EdgeWeight(), Markov::Model< NodeStorageType >::Nodes(), and Markov::Model< NodeStorageType >::OptimizeEdgeOrder().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

check_corpus_path()

def Python.Markopy.BaseCLI.check_corpus_path ( str  filename )

staticinherited

check import path for validity

Parameters

filename

filename to check

Definition at line 181 of file base.py.

    def check_corpus_path(filename : str):
        """!
        @brief check import path for validity
        @param filename filename to check
        """
 
        if(not os.path.isfile(filename)):
            return False
        return True
 

Referenced by Python.Markopy.BaseCLI.train().

Here is the caller graph for this function:

check_export_path()

def Python.Markopy.BaseCLI.check_export_path ( str  filename )

staticinherited

check import path for validity

Parameters

filename

filename to check

Definition at line 192 of file base.py.

    def check_export_path(filename : str):
        """!
        @brief check import path for validity
        @param filename filename to check
        """
 
        if(filename and os.path.isfile(filename)):
            return True
        return True
 

Referenced by Python.Markopy.BaseCLI.train().

Here is the caller graph for this function:

check_import_path()

def Python.Markopy.BaseCLI.check_import_path ( str  filename )

staticinherited

check import path for validity

Parameters

filename

filename to check

Definition at line 169 of file base.py.

    def check_import_path(filename : str):
        """!
        @brief check import path for validity
        @param filename filename to check
        """
        
        if(not os.path.isfile(filename)):
            return False
        else:
            return True
 

Referenced by Python.Markopy.BaseCLI.import_model().

Here is the caller graph for this function:

ConstructMatrix()

bool Markov::API::ModelMatrix::ConstructMatrix ( )

inherited

Construct the related Matrix data for the model.

This operation can be used after importing/training to allocate and populate the matrix content.

this will initialize: char** edgeMatrix -> a 2D array of mapping left and right connections of each edge. long int **valueMatrix -> a 2D array representing the edge weights. int matrixSize -> Size of the matrix, aka total number of nodes. char* matrixIndex -> order of nodes in the model long int *totalEdgeWeights -> total edge weights of each Node.

Returns

True if constructed. False if already construced.

Definition at line 31 of file modelMatrix.cpp.

                                          {
    if(this->ready) return false;
    this->matrixSize = this->StarterNode()->edgesV.size() + 2;
 
    this->matrixIndex = new char[this->matrixSize];
    this->totalEdgeWeights = new long int[this->matrixSize];
 
    this->edgeMatrix = new char*[this->matrixSize];
    for(int i=0;i<this->matrixSize;i++){
        this->edgeMatrix[i] = new char[this->matrixSize];
    }
    this->valueMatrix = new long int*[this->matrixSize];
    for(int i=0;i<this->matrixSize;i++){
        this->valueMatrix[i] = new long int[this->matrixSize];
    }
    std::map< char, Node< char > * > *nodes;
    nodes = this->Nodes();
    int i=0;
    for (auto const& [repr, node] : *nodes){
        if(repr!=0) this->matrixIndex[i] = repr;
        else this->matrixIndex[i] = 199;
        this->totalEdgeWeights[i] = node->TotalEdgeWeights();
        for(int j=0;j<this->matrixSize;j++){
            char val = node->NodeValue();
            if(val < 0){
                for(int k=0;k<this->matrixSize;k++){
                    this->valueMatrix[i][k] = 0;
                    this->edgeMatrix[i][k] = 255;
                }
                break;
            }
            else if(node->NodeValue() == 0 && j>(this->matrixSize-3)){
                this->valueMatrix[i][j] = 0;
                this->edgeMatrix[i][j] = 255;
            }else if(j==(this->matrixSize-1)) {
                this->valueMatrix[i][j] = 0;
                this->edgeMatrix[i][j] = 255;
            }else{
                this->valueMatrix[i][j] = node->edgesV[j]->EdgeWeight();
                this->edgeMatrix[i][j]  = node->edgesV[j]->RightNode()->NodeValue();
            }
 
        }
        i++;
    }
    this->ready = true;
    return true;
    //this->DumpJSON();
}

References Markov::API::ModelMatrix::edgeMatrix, Markov::Edge< NodeStorageType >::EdgeWeight(), Markov::API::ModelMatrix::matrixIndex, Markov::API::ModelMatrix::matrixSize, Markov::Model< NodeStorageType >::Nodes(), Markov::Node< storageType >::NodeValue(), Markov::API::ModelMatrix::ready, Markov::Edge< NodeStorageType >::RightNode(), Markov::Model< NodeStorageType >::StarterNode(), Markov::API::ModelMatrix::totalEdgeWeights, Markov::Node< storageType >::TotalEdgeWeights(), and Markov::API::ModelMatrix::valueMatrix.

Referenced by Markov::Markopy::CUDA::BOOST_PYTHON_MODULE(), Markov::Markopy::BOOST_PYTHON_MODULE(), Markov::API::ModelMatrix::Import(), and Markov::API::ModelMatrix::Train().

Here is the call graph for this function:

Here is the caller graph for this function:

DeallocateMatrix()

bool Markov::API::ModelMatrix::DeallocateMatrix ( )

protectedinherited

Deallocate matrix and make it ready for re-construction.

Returns

True if deallocated. False if matrix was not initialized

Definition at line 81 of file modelMatrix.cpp.

                                           {
    if(!this->ready) return false;
    delete[] this->matrixIndex;
    delete[] this->totalEdgeWeights;
 
    for(int i=0;i<this->matrixSize;i++){
        delete[] this->edgeMatrix[i];
    }
    delete[] this->edgeMatrix;
 
    for(int i=0;i<this->matrixSize;i++){
        delete[] this->valueMatrix[i];
    }
    delete[] this->valueMatrix;
 
    this->matrixSize = -1;
    this->ready = false;
    return true;
}

References Markov::API::ModelMatrix::edgeMatrix, Markov::API::ModelMatrix::matrixIndex, Markov::API::ModelMatrix::matrixSize, Markov::API::ModelMatrix::ready, Markov::API::ModelMatrix::totalEdgeWeights, and Markov::API::ModelMatrix::valueMatrix.

Referenced by Markov::API::ModelMatrix::Import(), and Markov::API::ModelMatrix::Train().

Here is the caller graph for this function:

DumpJSON()

void Markov::API::ModelMatrix::DumpJSON ( )

inherited

Debug function to dump the model to a JSON file.

Might not work 100%. Not meant for production use.

Definition at line 101 of file modelMatrix.cpp.

                                   {
 
    std::cout << "{\n   \"index\": \"";
    for(int i=0;i<this->matrixSize;i++){
        if(this->matrixIndex[i]=='"') std::cout << "\\\"";
        else if(this->matrixIndex[i]=='\\') std::cout << "\\\\";
        else if(this->matrixIndex[i]==0) std::cout << "\\\\x00";
        else if(i==0) std::cout << "\\\\xff";
        else if(this->matrixIndex[i]=='\n') std::cout << "\\n";
        else std::cout << this->matrixIndex[i];
    }
    std::cout << 
    "\",\n"
    "   \"edgemap\": {\n";
 
    for(int i=0;i<this->matrixSize;i++){
        if(this->matrixIndex[i]=='"') std::cout << "      \"\\\"\": [";
        else if(this->matrixIndex[i]=='\\') std::cout << "      \"\\\\\": [";
        else if(this->matrixIndex[i]==0) std::cout << "      \"\\\\x00\": [";
        else if(this->matrixIndex[i]<0) std::cout << "      \"\\\\xff\": [";
        else std::cout << "      \"" << this->matrixIndex[i] << "\": [";
        for(int j=0;j<this->matrixSize;j++){
            if(this->edgeMatrix[i][j]=='"') std::cout << "\"\\\"\"";
            else if(this->edgeMatrix[i][j]=='\\') std::cout << "\"\\\\\"";
            else if(this->edgeMatrix[i][j]==0) std::cout << "\"\\\\x00\"";
            else if(this->edgeMatrix[i][j]<0) std::cout << "\"\\\\xff\"";
            else if(this->matrixIndex[i]=='\n') std::cout << "\"\\n\"";
            else std::cout << "\"" << this->edgeMatrix[i][j] << "\"";
            if(j!=this->matrixSize-1) std::cout << ", ";
        }
        std::cout << "],\n";
    }
    std::cout << "},\n";
 
    std::cout << "\"   weightmap\": {\n";
    for(int i=0;i<this->matrixSize;i++){
        if(this->matrixIndex[i]=='"') std::cout << "      \"\\\"\": [";
        else if(this->matrixIndex[i]=='\\') std::cout << "      \"\\\\\": [";
        else if(this->matrixIndex[i]==0) std::cout << "      \"\\\\x00\": [";
        else if(this->matrixIndex[i]<0) std::cout << "      \"\\\\xff\": [";
        else std::cout << "      \"" << this->matrixIndex[i] << "\": [";
 
        for(int j=0;j<this->matrixSize;j++){
            std::cout << this->valueMatrix[i][j];
            if(j!=this->matrixSize-1) std::cout << ", ";
        }
        std::cout << "],\n";
    }
    std::cout << "  }\n}\n";
}

References Markov::API::ModelMatrix::edgeMatrix, Markov::API::ModelMatrix::matrixIndex, Markov::API::ModelMatrix::matrixSize, and Markov::API::ModelMatrix::valueMatrix.

Referenced by Markov::Markopy::CUDA::BOOST_PYTHON_MODULE(), and Markov::Markopy::BOOST_PYTHON_MODULE().

Here is the caller graph for this function:

Edges()

std::vector<Edge<char >*>* Markov::Model< char >::Edges ( )

inlineinherited

Return a vector of all the edges in the model.

Returns

vector of edges

Definition at line 176 of file model.h.

{ return &edges;}

Export() [1/2]

bool Markov::Model< char >::Export ( const char *  filename )

inherited

Open a file to export with filename, and call bool Model::Export with std::ofstream.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Export file to filename

Markov::Model<char> model;
model.Export("test.mdl");

Definition at line 166 of file model.h.

                                                              {
    std::ofstream exportfile;
    exportfile.open(filename);
    return this->Export(&exportfile);
}

export()

def Python.Markopy.BaseCLI.export (   self, str  filename  )

inherited

Export model to a file.

Parameters

filename

filename to export to

Definition at line 138 of file base.py.

    def export(self, filename : str):
        """! 
        @brief Export model to a file
        @param filename filename to export to
        """
        self.model.Export(filename)
 

References Python.CudaMarkopy.CudaModelMatrixCLI.model, Python.Markopy.BaseCLI.model, Python.Markopy.ModelMatrixCLI.model, Python.Markopy.MarkovPasswordsCLI.model, and Markov::GUI::MarkovPasswordsGUI.model().

Referenced by Python.Markopy.BaseCLI.train().

Here is the call graph for this function:

Here is the caller graph for this function:

Export() [2/2]

bool Markov::Model< char >::Export ( std::ofstream *  f )

inherited

Export a file of the model.

File contains a list of edges. Format is: Left_repr;EdgeWeight;right_repr. For more information on the format, check out the project wiki or github readme.

Iterate over this vertices, and their edges, and write them to file.

Returns

True if successful, False for incomplete models.

Example Use: Export file to ofstream

Markov::Model<char> model;
std::ofstream file("test.mdl");
model.Export(&file);

Definition at line 155 of file model.h.

                                                        {
    Markov::Edge<NodeStorageType>* e;
    for (std::vector<int>::size_type i = 0; i != this->edges.size(); i++) {
        e = this->edges[i];
        //std::cout << e->LeftNode()->NodeValue() << "," << e->EdgeWeight() << "," << e->RightNode()->NodeValue() << "\n";
        *f << e->LeftNode()->NodeValue() << "," << e->EdgeWeight() << "," << e->RightNode()->NodeValue() << "\n";
    }
 
    return true;
}

FastRandomWalk() [1/3]

def Python.Markopy.ModelMatrix.FastRandomWalk ( int  count, str  wordlist, int  minlen, int  maxlen  )

inherited

Definition at line 48 of file mm.py.

    def FastRandomWalk(count : int, wordlist : str, minlen : int, maxlen : int):
        pass

Referenced by Python.Markopy.ModelMatrixCLI._generate().

Here is the caller graph for this function:

FastRandomWalk() [2/3]

int Markov::API::ModelMatrix::FastRandomWalk ( unsigned long int  n, const char *  wordlistFileName, int  minLen = 6, int  maxLen = 12, int  threads = 20, bool  bFileIO = true  )

inherited

Random walk on the Matrix-reduced Markov::Model.

This has an O(N) Memory complexity. To limit the maximum usage, requests with n>50M are partitioned using Markov::API::ModelMatrix::FastRandomWalkPartition.

If n>50M, threads are going to be synced, files are going to be flushed, and buffers will be reallocated every 50M generations. This comes at a minor performance penalty.

While it has the same functionality, this operation reduces Markov::API::MarkovPasswords::Generate runtime by %96.5

This function has deprecated Markov::API::MarkovPasswords::Generate, and will eventually replace it.

Parameters

n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
threads	- number of OS threads to spawn
bFileIO	- If false, filename will be ignored and will output to stdout.

Markov::API::ModelMatrix mp;
mp.Import("models/finished.mdl");
mp.FastRandomWalk(50000000,"./wordlist.txt",6,12,25, true);

Definition at line 217 of file modelMatrix.cpp.

                                                                                                                                            {
    std::ofstream wordlist; 
    if(bFileIO)
        wordlist.open(wordlistFileName);
    this->FastRandomWalk(n, &wordlist, minLen, maxLen, threads, bFileIO);
    return 0;
}

References Markov::API::ModelMatrix::FastRandomWalk().

Referenced by Markov::Markopy::BOOST_PYTHON_MODULE().

Here is the call graph for this function:

Here is the caller graph for this function:

FastRandomWalk() [3/3]

int Markov::API::ModelMatrix::FastRandomWalk ( unsigned long int  n, std::ofstream *  wordlist, int  minLen = 6, int  maxLen = 12, int  threads = 20, bool  bFileIO = true  )

protectedinherited

Random walk on the Matrix-reduced Markov::Model.

This has an O(N) Memory complexity. To limit the maximum usage, requests with n>50M are partitioned using Markov::API::ModelMatrix::FastRandomWalkPartition.

If n>50M, threads are going to be synced, files are going to be flushed, and buffers will be reallocated every 50M generations. This comes at a minor performance penalty.

While it has the same functionality, this operation reduces Markov::API::MarkovPasswords::Generate runtime by %96.5

This function has deprecated Markov::API::MarkovPasswords::Generate, and will eventually replace it.

Parameters

n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
threads	- number of OS threads to spawn
bFileIO	- If false, filename will be ignored and will output to stdout.

Markov::API::ModelMatrix mp;
mp.Import("models/finished.mdl");
mp.FastRandomWalk(50000000,"./wordlist.txt",6,12,25, true);

Definition at line 204 of file modelMatrix.cpp.

                                                                                                                                     {
    
 
    std::mutex mlock;
    if(n<=50000000ull) this->FastRandomWalkPartition(&mlock, wordlist, n, minLen, maxLen, bFileIO, threads);
    else{
        int numberOfPartitions = n/50000000ull;
        for(int i=0;i<numberOfPartitions;i++)
            this->FastRandomWalkPartition(&mlock, wordlist, 50000000ull, minLen, maxLen, bFileIO, threads);
    }
    return 0;
}

References Markov::API::ModelMatrix::FastRandomWalkPartition().

Referenced by Markov::API::ModelMatrix::FastRandomWalk().

Here is the call graph for this function:

Here is the caller graph for this function:

FastRandomWalkPartition()

void Markov::API::ModelMatrix::FastRandomWalkPartition ( std::mutex *  mlock, std::ofstream *  wordlist, unsigned long int  n, int  minLen, int  maxLen, bool  bFileIO, int  threads  )

protectedinherited

A single partition of FastRandomWalk event.

Since FastRandomWalk has to allocate its output buffer before operation starts and writes data in chunks, large n parameters would lead to huge memory allocations. Without Partitioning:

• 50M results 12 characters max -> 550 Mb Memory allocation
• 5B results 12 characters max -> 55 Gb Memory allocation
• 50B results 12 characters max -> 550GB Memory allocation

Instead, FastRandomWalk is partitioned per 50M generations to limit the top memory need.

Parameters

mlock	- mutex lock to distribute to child threads
wordlist	- Reference to the wordlist file to write to
n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
threads	- number of OS threads to spawn
bFileIO	- If false, filename will be ignored and will output to stdout.

Definition at line 225 of file modelMatrix.cpp.

                                                                                                                                                                {
    
    int iterationsPerThread = n/threads;
    int iterationsPerThreadCarryOver = n%threads;
 
    std::vector<std::thread*> threadsV;
    
    int id = 0;
    for(int i=0;i<threads;i++){
        threadsV.push_back(new std::thread(&Markov::API::ModelMatrix::FastRandomWalkThread, this, mlock, wordlist, iterationsPerThread, minLen, maxLen, id, bFileIO));
        id++;
    }
 
    threadsV.push_back(new std::thread(&Markov::API::ModelMatrix::FastRandomWalkThread, this, mlock, wordlist, iterationsPerThreadCarryOver, minLen, maxLen, id, bFileIO));
 
    for(int i=0;i<threads;i++){
        threadsV[i]->join();
    }
}

References Markov::API::ModelMatrix::FastRandomWalkThread().

Referenced by Markov::API::ModelMatrix::FastRandomWalk().

Here is the call graph for this function:

Here is the caller graph for this function:

FastRandomWalkThread()

void Markov::API::ModelMatrix::FastRandomWalkThread ( std::mutex *  mlock, std::ofstream *  wordlist, unsigned long int  n, int  minLen, int  maxLen, int  id, bool  bFileIO  )

protectedinherited

A single thread of a single partition of FastRandomWalk.

A FastRandomWalkPartition will initiate as many of this function as requested.

This function contains the bulk of the generation algorithm.

Parameters

mlock	- mutex lock to distribute to child threads
wordlist	- Reference to the wordlist file to write to
n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
id	- DEPRECATED Thread id - No longer used
bFileIO	- If false, filename will be ignored and will output to stdout.

Definition at line 153 of file modelMatrix.cpp.

                                                                                                                                                        {
    if(n==0) return;
 
    Markov::Random::Marsaglia MarsagliaRandomEngine;
    char* e;
    char *res = new char[(maxLen+2)*n];
    int index = 0;
    char next;
    int len=0;
    long int selection;
    char cur;
    long int bufferctr = 0;
    for (int i = 0; i < n; i++) {
        cur=199;
        len=0;
        while (true) {
            e = strchr(this->matrixIndex, cur);
            index = e - this->matrixIndex;
            selection = MarsagliaRandomEngine.random() % this->totalEdgeWeights[index];
            for(int j=0;j<this->matrixSize;j++){
                selection -= this->valueMatrix[index][j];
                if (selection < 0){
                    next = this->edgeMatrix[index][j];
                    break;
                }
            }
 
            if (len >= maxLen)  break;
            else if ((next < 0) && (len < minLen)) continue;
            else if (next < 0) break;  
            cur = next;
            res[bufferctr + len++] = cur;
        }
        res[bufferctr + len++] = '\n';
        bufferctr+=len;
        
    }
    if(bFileIO){
        mlock->lock();
        *wordlist << res;
        mlock->unlock();
    }else{
        mlock->lock();
        std::cout << res;
        mlock->unlock();
    }
    delete res;
 
}

References Markov::API::ModelMatrix::edgeMatrix, Markov::API::ModelMatrix::matrixIndex, Markov::API::ModelMatrix::matrixSize, Markov::Random::Marsaglia::random(), Markov::API::ModelMatrix::totalEdgeWeights, and Markov::API::ModelMatrix::valueMatrix.

Referenced by Markov::API::ModelMatrix::FastRandomWalkPartition().

Here is the call graph for this function:

Here is the caller graph for this function:

generate()

def Python.Markopy.BaseCLI.generate (   self, str  wordlist, bool  bulk = False  )

inherited

Generate strings from the model.

Parameters

model	model instance
wordlist	wordlist filename
bulk	marks bulk operation with directories

Definition at line 145 of file base.py.

    def generate(self, wordlist : str, bulk : bool=False):
        """! 
            @brief Generate strings from the model
            @param model: model instance
            @param wordlist wordlist filename
            @param bulk marks bulk operation with directories
        """
        if not (wordlist or self.args.count):
            logging.pprint("Generation mode requires -w/--wordlist and -n/--count parameters. Exiting.")
            return False
    
        if(bulk and os.path.isfile(wordlist)):
            logging.pprint(f"{wordlist} exists and will be overwritten.", 1)
        self._generate(wordlist)
 

References Python.CudaMarkopy.CudaModelMatrixCLI._generate(), Python.Markopy.BaseCLI._generate(), Python.Markopy.ModelMatrixCLI._generate(), Python.Markopy.MarkovPasswordsCLI._generate(), Python.CudaMarkopy.CudaMarkopyCLI.args, Python.Markopy.BaseCLI.args, and Python.Markopy.MarkopyCLI.args.

Referenced by Python.Markopy.BaseCLI.process().

Here is the call graph for this function:

Here is the caller graph for this function:

Generate()

void Markov::API::MarkovPasswords::Generate ( unsigned long int  n, const char *  wordlistFileName, int  minLen = 6, int  maxLen = 12, int  threads = 20  )

inherited

Call Markov::Model::RandomWalk n times, and collect output.

Generate from model and write results to a file. a much more performance-optimized method. FastRandomWalk will reduce the runtime by %96.5 on average.

Deprecated:

See Markov::API::MatrixModel::FastRandomWalk for more information.

Parameters

n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
threads	- number of OS threads to spawn

Definition at line 118 of file markovPasswords.cpp.

                                                                                                                               {
    char* res;
    char print[100];
    std::ofstream wordlist; 
    wordlist.open(wordlistFileName);
    std::mutex mlock;
    int iterationsPerThread = n/threads;
    int iterationsCarryOver = n%threads;
    std::vector<std::thread*> threadsV;
    for(int i=0;i<threads;i++){
        threadsV.push_back(new std::thread(&Markov::API::MarkovPasswords::GenerateThread, this, &mlock, iterationsPerThread, &wordlist, minLen, maxLen));
    }
 
    for(int i=0;i<threads;i++){
        threadsV[i]->join();
        delete threadsV[i];
    }
 
    this->GenerateThread(&mlock, iterationsCarryOver, &wordlist, minLen, maxLen);
    
}

References Markov::API::MarkovPasswords::GenerateThread().

Referenced by Markov::Markopy::BOOST_PYTHON_MODULE(), and Markov::GUI::Generate::generation().

Here is the call graph for this function:

Here is the caller graph for this function:

GenerateThread()

void Markov::API::MarkovPasswords::GenerateThread ( std::mutex *  outputLock, unsigned long int  n, std::ofstream *  wordlist, int  minLen, int  maxLen  )

privateinherited

A single thread invoked by the Generate function.

DEPRECATED: See Markov::API::MatrixModel::FastRandomWalkThread for more information. This has been replaced with a much more performance-optimized method. FastRandomWalk will reduce the runtime by %96.5 on average.

Parameters

outputLock	- shared mutex lock to lock during output operation. Prevents race condition on write.
n	number of lines to be generated by this thread
wordlist	wordlistfile
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate

Definition at line 140 of file markovPasswords.cpp.

                                                                                                                                       {
    char* res = new char[maxLen+5];
    if(n==0) return;
 
    Markov::Random::Marsaglia MarsagliaRandomEngine;
    for (int i = 0; i < n; i++) {
        this->RandomWalk(&MarsagliaRandomEngine, minLen, maxLen, res); 
        outputLock->lock();
        *wordlist << res << "\n";
        outputLock->unlock();
    }
}

References Markov::Model< NodeStorageType >::RandomWalk().

Referenced by Markov::API::MarkovPasswords::Generate().

Here is the call graph for this function:

Here is the caller graph for this function:

help()

def Python.Markopy.BaseCLI.help (   self )

inherited

Reimplemented in Python.Markopy.MarkopyCLI, and Python.CudaMarkopy.CudaMarkopyCLI.

Definition at line 51 of file base.py.

    def help(self):
        "! @brief Handle help strings. Defaults to argparse's help"
        self.print_help()
 

References Python.Markopy.BaseCLI.print_help.

Referenced by Python.Markopy.MarkopyCLI.add_arguments().

Here is the caller graph for this function:

Import() [1/2]

void Markov::API::ModelMatrix::Import ( const char *  filename )

inherited

Open a file to import with filename, and call bool Model::Import with std::ifstream.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Import a file with filename

Markov::Model<char> model;
model.Import("test.mdl");

Construct the matrix when done.

Definition at line 19 of file modelMatrix.cpp.

                                                     {
    this->DeallocateMatrix();
    this->Markov::API::MarkovPasswords::Import(filename);
    this->ConstructMatrix();
}

References Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), and Markov::Model< NodeStorageType >::Import().

Referenced by Markov::Markopy::CUDA::BOOST_PYTHON_MODULE(), Markov::Markopy::BOOST_PYTHON_MODULE(), and main().

Here is the call graph for this function:

Here is the caller graph for this function:

Import() [2/2]

bool Markov::Model< char >::Import ( std::ifstream *  f )

inherited

Import a file to construct the model.

File contains a list of edges. For more info on the file format, check out the wiki and github readme pages. Format is: Left_repr;EdgeWeight;right_repr

Iterate over this list, and construct nodes and edges accordingly.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Import a file from ifstream

Markov::Model<char> model;
std::ifstream file("test.mdl");
model.Import(&file);

Definition at line 126 of file model.h.

                                                        {
    std::string cell;
 
    char src;
    char target;
    long int oc;
 
    while (std::getline(*f, cell)) {
        //std::cout << "cell: " << cell << std::endl;
        src = cell[0];
        target = cell[cell.length() - 1];
        char* j;
        oc = std::strtol(cell.substr(2, cell.length() - 2).c_str(),&j,10);
        //std::cout << oc << "\n";
        Markov::Node<NodeStorageType>* srcN;
        Markov::Node<NodeStorageType>* targetN;
        Markov::Edge<NodeStorageType>* e;
        if (this->nodes.find(src) == this->nodes.end()) {
            srcN = new Markov::Node<NodeStorageType>(src);
            this->nodes.insert(std::pair<char, Markov::Node<NodeStorageType>*>(src, srcN));
            //std::cout << "Creating new node at start.\n";
        }
        else {
            srcN = this->nodes.find(src)->second;
        }
 
        if (this->nodes.find(target) == this->nodes.end()) {
            targetN = new Markov::Node<NodeStorageType>(target);
            this->nodes.insert(std::pair<char, Markov::Node<NodeStorageType>*>(target, targetN));
            //std::cout << "Creating new node at end.\n";
        }
        else {
            targetN = this->nodes.find(target)->second;
        }
        e = srcN->Link(targetN);
        e->AdjustEdge(oc);
        this->edges.push_back(e);
 
        //std::cout << int(srcN->NodeValue()) << " --" << e->EdgeWeight() << "--> " << int(targetN->NodeValue()) << "\n";
 
 
    }
 
    this->OptimizeEdgeOrder();
 
    return true;
}

import_model()

def Python.Markopy.BaseCLI.import_model (   self, str  filename  )

inherited

Import a model file.

Parameters

filename

filename to import

Definition at line 77 of file base.py.

    def import_model(self, filename : str):
        """! 
        @brief Import a model file
        @param filename filename to import
        """
        logging.pprint("Importing model file.", 1)
 
        if not self.check_import_path(filename):
            logging.pprint(f"Model file at {filename} not found. Check the file path, or working directory")
            return False
 
        self.model.Import(filename)
        logging.pprint("Model imported successfully.", 2)
        return True
 
 
 

References Python.Markopy.BaseCLI.check_import_path(), Python.CudaMarkopy.CudaModelMatrixCLI.model, Python.Markopy.BaseCLI.model, Python.Markopy.ModelMatrixCLI.model, Python.Markopy.MarkovPasswordsCLI.model, and Markov::GUI::MarkovPasswordsGUI.model().

Referenced by Python.Markopy.BaseCLI.process().

Here is the call graph for this function:

Here is the caller graph for this function:

init_post_arguments()

def Python.Markopy.ModelMatrixCLI.init_post_arguments

(

self

)

Reimplemented from Python.Markopy.BaseCLI.

Reimplemented in Python.Markopy.MarkopyCLI, Python.CudaMarkopy.CudaModelMatrixCLI, and Python.Markopy.MarkopyCLI.

Definition at line 36 of file mmx.py.

    def init_post_arguments(self):
        super().init_post_arguments()
        self.fileIO = not self.args.stdout
        

Referenced by Python.Markopy.BaseCLI.parse(), and Python.Markopy.MarkopyCLI.parse().

Here is the caller graph for this function:

Nodes()

std::map<char , Node<char >*>* Markov::Model< char >::Nodes ( )

inlineinherited

Return starter Node.

Returns

starter node with 00 NodeValue

Definition at line 181 of file model.h.

{ return &nodes;}

OpenDatasetFile()

std::ifstream * Markov::API::MarkovPasswords::OpenDatasetFile ( const char *  filename )

inherited

Open dataset file and return the ifstream pointer.

Parameters

filename

- Filename to open

Returns

ifstream* to the the dataset file

Definition at line 51 of file markovPasswords.cpp.

                                                                          {
 
    std::ifstream* datasetFile;
 
    std::ifstream newFile(filename);
 
    datasetFile = &newFile;
 
    this->Import(datasetFile);
    return datasetFile;
}

References Markov::Model< NodeStorageType >::Import().

Here is the call graph for this function:

OptimizeEdgeOrder()

void Markov::Model< char >::OptimizeEdgeOrder

inherited

Sort edges of all nodes in the model ordered by edge weights.

Definition at line 186 of file model.h.

                                                    {
    for (std::pair<unsigned char, Markov::Node<NodeStorageType>*> const& x : this->nodes) {
        //std::cout << "Total edges in EdgesV: " << x.second->edgesV.size() << "\n"; 
        std::sort (x.second->edgesV.begin(), x.second->edgesV.end(), [](Edge<NodeStorageType> *lhs, Edge<NodeStorageType> *rhs)->bool{
            return lhs->EdgeWeight() > rhs->EdgeWeight();
        });
        //for(int i=0;i<x.second->edgesV.size();i++)
        //  std::cout << x.second->edgesV[i]->EdgeWeight() << ", ";
        //std::cout << "\n";
    }
    //std::cout << "Total number of nodes: " << this->nodes.size() << std::endl;
    //std::cout << "Total number of edges: " << this->edges.size() << std::endl;
}

parse()

def Python.Markopy.BaseCLI.parse (   self )

inherited

Reimplemented in Python.Markopy.MarkopyCLI, and Python.CudaMarkopy.CudaMarkopyCLI.

Definition at line 55 of file base.py.

    def parse(self):
        "! @brief add, parse and hook arguements"
        self.add_arguments()
        self.parse_arguments()
        self.init_post_arguments()
 

References Python.CudaMarkopy.CudaModelMatrixCLI.add_arguments(), Python.Markopy.BaseCLI.add_arguments(), Python.Markopy.AbstractGenerationModelCLI.add_arguments(), Python.Markopy.AbstractTrainingModelCLI.add_arguments(), Python.Markopy.MarkopyCLI.add_arguments(), Python.Markopy.ModelMatrixCLI.add_arguments(), Python.CudaMarkopy.CudaModelMatrixCLI.init_post_arguments(), Python.Markopy.BaseCLI.init_post_arguments(), Python.Markopy.MarkopyCLI.init_post_arguments(), Python.Markopy.ModelMatrixCLI.init_post_arguments(), and Python.Markopy.BaseCLI.parse_arguments().

Here is the call graph for this function:

parse_arguments()

def Python.Markopy.BaseCLI.parse_arguments (   self )

inherited

Definition at line 73 of file base.py.

    def parse_arguments(self):
        "! @brief trigger parser"
        self.args = self.parser.parse_known_args()[0]
 

Referenced by Python.Markopy.BaseCLI.parse(), and Python.Markopy.MarkopyCLI.parse().

Here is the caller graph for this function:

process()

def Python.Markopy.BaseCLI.process (   self )

inherited

Process parameters for operation.

Reimplemented in Python.Markopy.MarkopyCLI.

Definition at line 202 of file base.py.

    def process(self):
        """!
        @brief Process parameters for operation
        """
        if(self.args.bulk):
            logging.pprint(f"Bulk mode operation chosen.", 4)
            if (self.args.mode.lower() == "train"):
                if (os.path.isdir(self.args.output) and not os.path.isfile(self.args.output)) and (os.path.isdir(self.args.dataset) and not os.path.isfile(self.args.dataset)):
                    corpus_list = os.listdir(self.args.dataset)
                    for corpus in corpus_list:
                        self.import_model(self.args.input)
                        logging.pprint(f"Training {self.args.input} with {corpus}", 2)
                        output_file_name = corpus
                        model_extension = ""
                        if "." in self.args.input:
                            model_extension = self.args.input.split(".")[-1]
                        self.train(f"{self.args.dataset}/{corpus}", self.args.seperator, f"{self.args.output}/{corpus}.{model_extension}", output_forced=True, bulk=True)
                else:
                    logging.pprint("In bulk training, output and dataset should be a directory.")
                    exit(1)
 
            elif (self.args.mode.lower() == "generate"):
                if (os.path.isdir(self.args.wordlist) and not os.path.isfile(self.args.wordlist)) and (os.path.isdir(self.args.input) and not os.path.isfile(self.args.input)):
                    model_list = os.listdir(self.args.input)
                    print(model_list)
                    for input in model_list:
                        logging.pprint(f"Generating from {self.args.input}/{input} to {self.args.wordlist}/{input}.txt", 2)
                        self.import_model(f"{self.args.input}/{input}")
                        model_base = input
                        if "." in self.args.input:
                            model_base = input.split(".")[1]
                        self.generate(f"{self.args.wordlist}/{model_base}.txt", bulk=True)
                else:
                    logging.pprint("In bulk generation, input and wordlist should be directory.")
 
        else:
            self.import_model(self.args.input)
            if (self.args.mode.lower() == "generate"):
                self.generate(self.args.wordlist)
 
 
            elif (self.args.mode.lower() == "train"):
                self.train(self.args.dataset, self.args.seperator, self.args.output, output_forced=True)
 
 
            elif(self.args.mode.lower() == "combine"):
                self.train(self.args.dataset, self.args.seperator, self.args.output)
                self.generate(self.args.wordlist)
 
 
            else:
                logging.pprint("Invalid mode arguement given.")
                logging.pprint("Accepted modes: 'Generate', 'Train', 'Combine'")
                exit(5)
 

References Python.CudaMarkopy.CudaMarkopyCLI.args, Python.Markopy.BaseCLI.args, Python.Markopy.MarkopyCLI.args, Python.Markopy.BaseCLI.generate(), Python.Markopy.BaseCLI.import_model(), Markov::GUI::Generate.train(), Markov::GUI::Train.train(), and Python.Markopy.BaseCLI.train().

Here is the call graph for this function:

RandomWalk()

char * Markov::Model< char >::RandomWalk ( Markov::Random::RandomEngine *  randomEngine, int  minSetting, int  maxSetting, NodeStorageType *  buffer  )

inherited

Do a random walk on this model.

Start from the starter node, on each node, invoke RandomNext using the random engine on current node, until terminator node is reached. If terminator node is reached before minimum length criateria is reached, ignore the last selection and re-invoke randomNext

If maximum length criteria is reached but final node is not, cut off the generation and proceed to the final node. This function takes Markov::Random::RandomEngine as a parameter to generate pseudo random numbers from

This library is shipped with two random engines, Marsaglia and Mersenne. While mersenne output is higher in entropy, most use cases don't really need super high entropy output, so Markov::Random::Marsaglia is preferable for better performance.

This function WILL NOT reallocate buffer. Make sure no out of bound writes are happening via maximum length criteria.

Example Use: Generate 10 lines, with 5 to 10 characters, and print the output. Use Marsaglia

Markov::Model<char> model;
Model.import("model.mdl");
char* res = new char[11];
Markov::Random::Marsaglia MarsagliaRandomEngine;
for (int i = 0; i < 10; i++) {
    this->RandomWalk(&MarsagliaRandomEngine, 5, 10, res); 
    std::cout << res << "\n";
 }

Parameters

randomEngine	Random Engine to use for the random walks. For examples, see Markov::Random::Mersenne and Markov::Random::Marsaglia
minSetting	Minimum number of characters to generate
maxSetting	Maximum number of character to generate
buffer	buffer to write the result to

Returns

Null terminated string that was generated.

Definition at line 86 of file model.h.

                                                                                                                                                         {
    Markov::Node<NodeStorageType>* n = this->starterNode;
    int len = 0;
    Markov::Node<NodeStorageType>* temp_node;
    while (true) {
        temp_node = n->RandomNext(randomEngine);
        if (len >= maxSetting) {
            break;
        }
        else if ((temp_node == NULL) && (len < minSetting)) {
            continue;
        }
 
        else if (temp_node == NULL){
            break;
        }
            
        n = temp_node;
 
        buffer[len++] = n->NodeValue();
    }
 
    //null terminate the string
    buffer[len] = 0x00;
 
    //do something with the generated string
    return buffer; //for now
}

Save()

std::ofstream * Markov::API::MarkovPasswords::Save ( const char *  filename )

inherited

Export model to file.

Parameters

filename

- Export filename.

Returns

std::ofstream* of the exported file.

Definition at line 106 of file markovPasswords.cpp.

                                                                {
    std::ofstream* exportFile;
 
    std::ofstream newFile(filename);
 
    exportFile = &newFile;
    
    this->Export(exportFile);
    return exportFile;
}

References Markov::Model< NodeStorageType >::Export().

Here is the call graph for this function:

StarterNode()

Node<char >* Markov::Model< char >::StarterNode ( )

inlineinherited

Return starter Node.

Returns

starter node with 00 NodeValue

Definition at line 171 of file model.h.

{ return starterNode;}

Train()

void Markov::API::ModelMatrix::Train ( const char *  datasetFileName, char  delimiter, int  threads  )

inherited

Train the model with the dataset file.

Parameters

datasetFileName	- Ifstream* to the dataset. If null, use class member
delimiter	- a character, same as the delimiter in dataset content
threads	- number of OS threads to spawn

Markov::API::MarkovPasswords mp;
mp.Import("models/2gram.mdl");
mp.Train("password.corpus");

Construct the matrix when done.

Definition at line 25 of file modelMatrix.cpp.

                                                                                        {
    this->DeallocateMatrix();
    this->Markov::API::MarkovPasswords::Train(datasetFileName,delimiter,threads);
    this->ConstructMatrix();
}

References Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), and Markov::API::MarkovPasswords::Train().

Referenced by Markov::Markopy::CUDA::BOOST_PYTHON_MODULE(), and Markov::Markopy::BOOST_PYTHON_MODULE().

Here is the call graph for this function:

Here is the caller graph for this function:

train()

def Python.Markopy.BaseCLI.train (   self, str  dataset, str  seperator, str  output, bool  output_forced = False, bool  bulk = False  )

inherited

Train a model via CLI parameters.

Parameters

model	Model instance
dataset	filename for the dataset
seperator	seperator used with the dataset
output	output filename
output_forced	force overwrite
bulk	marks bulk operation with directories

Definition at line 94 of file base.py.

    def train(self, dataset : str, seperator : str, output : str, output_forced : bool=False, bulk : bool=False):
        """! 
            @brief Train a model via CLI parameters 
            @param model Model instance
            @param dataset filename for the dataset
            @param seperator seperator used with the dataset
            @param output output filename
            @param output_forced force overwrite
            @param bulk marks bulk operation with directories
        """
        logging.pprint("Training.")
 
        if not (dataset and seperator and (output or not output_forced)):
            logging.pprint(f"Training mode requires -d/--dataset{', -o/--output' if output_forced else''} and -s/--seperator parameters. Exiting.")
            return False
 
        if not bulk and not self.check_corpus_path(dataset):
            logging.pprint(f"{dataset} doesn't exists. Check the file path, or working directory")
            return False
 
        if not self.check_export_path(output):
            logging.pprint(f"Cannot create output at {output}")
            return False
 
        if(seperator == '\\t'):
            logging.pprint("Escaping seperator.", 3)
            seperator = '\t'
        
        if(len(seperator)!=1):
            logging.pprint(f'Delimiter must be a single character, and "{seperator}" is not accepted.')
            exit(4)
 
        logging.pprint(f'Starting training.', 3)
        self.model.Train(dataset,seperator, int(self.args.threads))
        logging.pprint(f'Training completed.', 2)
 
        if(output):
            logging.pprint(f'Exporting model to {output}', 2)
            self.export(output)
        else:
            logging.pprint(f'Model will not be exported.', 1)
 
        return True
 

References Python.CudaMarkopy.CudaMarkopyCLI.args, Python.Markopy.BaseCLI.args, Python.Markopy.MarkopyCLI.args, Python.Markopy.BaseCLI.check_corpus_path(), Python.Markopy.BaseCLI.check_export_path(), Python.Markopy.BaseCLI.export(), Python.CudaMarkopy.CudaModelMatrixCLI.model, Python.Markopy.BaseCLI.model, Python.Markopy.ModelMatrixCLI.model, Python.Markopy.MarkovPasswordsCLI.model, and Markov::GUI::MarkovPasswordsGUI.model().

Referenced by Python.Markopy.BaseCLI.process().

Here is the call graph for this function:

Here is the caller graph for this function:

TrainThread()

void Markov::API::MarkovPasswords::TrainThread ( Markov::API::Concurrency::ThreadSharedListHandler *  listhandler, char  delimiter  )

privateinherited

A single thread invoked by the Train function.

Parameters

listhandler	- Listhandler class to read corpus from
delimiter	- a character, same as the delimiter in dataset content

Definition at line 85 of file markovPasswords.cpp.

                                                                                                                  {
    char format_str[] ="%ld,%s";
    format_str[3]=delimiter;
    std::string line;
    while (listhandler->next(&line) && keepRunning) {
        long int oc;
        if (line.size() > 100) {
            line = line.substr(0, 100);
        }
        char* linebuf = new char[line.length()+5];
#ifdef _WIN32
        sscanf_s(line.c_str(), "%ld,%s", &oc, linebuf, line.length()+5); //<== changed format_str to-> "%ld,%s"
#else
        sscanf(line.c_str(), format_str, &oc, linebuf);
#endif
        this->AdjustEdge((const char*)linebuf, oc); 
        delete linebuf;
    }
}

References Markov::Model< NodeStorageType >::AdjustEdge(), keepRunning, and Markov::API::Concurrency::ThreadSharedListHandler::next().

Referenced by Markov::API::MarkovPasswords::Train().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

args

Python.Markopy.BaseCLI.args

inherited

Definition at line 75 of file base.py.

Referenced by Python.CudaMarkopy.CudaModelMatrixCLI._generate(), Python.Markopy.BaseCLI._generate(), Python.Markopy.ModelMatrixCLI._generate(), Python.Markopy.MarkovPasswordsCLI._generate(), Python.Markopy.BaseCLI.generate(), Python.Markopy.MarkopyCLI.help(), Python.Markopy.BaseCLI.init_post_arguments(), Python.Markopy.MarkopyCLI.parse(), Python.CudaMarkopy.CudaMarkopyCLI.parse_fail(), Python.Markopy.BaseCLI.process(), and Python.Markopy.BaseCLI.train().

datasetFile

std::ifstream* Markov::API::MarkovPasswords::datasetFile

privateinherited

Definition at line 123 of file markovPasswords.h.

edgeMatrix

char** Markov::API::ModelMatrix::edgeMatrix

protectedinherited

2-D Character array for the edge Matrix (The characters of Nodes)

Definition at line 175 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), Markov::API::ModelMatrix::DumpJSON(), and Markov::API::ModelMatrix::FastRandomWalkThread().

edges

std::vector<Edge<char >*> Markov::Model< char >::edges

privateinherited

A list of all edges in this model.

Definition at line 204 of file model.h.

fileIO

Python.Markopy.ModelMatrixCLI.fileIO

Definition at line 38 of file mmx.py.

Referenced by Python.CudaMarkopy.CudaModelMatrixCLI._generate(), and Python.Markopy.ModelMatrixCLI._generate().

matrixIndex

char* Markov::API::ModelMatrix::matrixIndex

protectedinherited

to hold the Matrix index (To hold the orders of 2-D arrays')

Definition at line 190 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), Markov::API::ModelMatrix::DumpJSON(), and Markov::API::ModelMatrix::FastRandomWalkThread().

matrixSize

int Markov::API::ModelMatrix::matrixSize

protectedinherited

to hold Matrix size

Definition at line 185 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), Markov::API::ModelMatrix::DumpJSON(), Markov::API::ModelMatrix::FastRandomWalkThread(), and Markov::API::CUDA::CUDAModelMatrix::FlattenMatrix().

model

Python.Markopy.ModelMatrixCLI.model

Definition at line 30 of file mmx.py.

Referenced by Python.CudaMarkopy.CudaModelMatrixCLI._generate(), Python.Markopy.BaseCLI._generate(), Python.Markopy.ModelMatrixCLI._generate(), Python.Markopy.MarkovPasswordsCLI._generate(), Python.Markopy.BaseCLI.export(), Python.Markopy.BaseCLI.import_model(), and Python.Markopy.BaseCLI.train().

modelSavefile

std::ofstream* Markov::API::MarkovPasswords::modelSavefile

privateinherited

Dataset file input of our system

Definition at line 124 of file markovPasswords.h.

nodes

std::map<char , Node<char >*> Markov::Model< char >::nodes

privateinherited

Map LeftNode is the Nodes NodeValue Map RightNode is the node pointer.

Definition at line 193 of file model.h.

outputFile

std::ofstream* Markov::API::MarkovPasswords::outputFile

privateinherited

File to save model of our system

Definition at line 125 of file markovPasswords.h.

parser

Python.Markopy.BaseCLI.parser

inherited

Definition at line 25 of file base.py.

Referenced by Python.CudaMarkopy.CudaMarkopyCLI.__init__(), Python.CudaMarkopy.CudaModelMatrixCLI.add_arguments(), Python.Markopy.BaseCLI.add_arguments(), Python.Markopy.AbstractGenerationModelCLI.add_arguments(), Python.Markopy.AbstractTrainingModelCLI.add_arguments(), Python.Markopy.MarkopyCLI.add_arguments(), Python.Markopy.ModelMatrixCLI.add_arguments(), Python.CudaMarkopy.CudaMarkopyCLI.help(), and Python.Markopy.MarkopyCLI.help().

print_help

Python.Markopy.BaseCLI.print_help

inherited

Definition at line 39 of file base.py.

Referenced by Python.Markopy.BaseCLI.help(), and Python.Markopy.MarkopyCLI.help().

ready

bool Markov::API::ModelMatrix::ready

protectedinherited

True when matrix is constructed. False if not.

Definition at line 200 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), and Markov::API::ModelMatrix::ModelMatrix().

starterNode

Node<char >* Markov::Model< char >::starterNode

privateinherited

Starter Node of this model.

Definition at line 198 of file model.h.

totalEdgeWeights

long int* Markov::API::ModelMatrix::totalEdgeWeights

protectedinherited

Array of the Total Edge Weights.

Definition at line 195 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), and Markov::API::ModelMatrix::FastRandomWalkThread().

valueMatrix

long int** Markov::API::ModelMatrix::valueMatrix

protectedinherited

2-d Integer array for the value Matrix (For the weights of Edges)

Definition at line 180 of file modelMatrix.h.

Referenced by Markov::API::ModelMatrix::ConstructMatrix(), Markov::API::ModelMatrix::DeallocateMatrix(), Markov::API::ModelMatrix::DumpJSON(), and Markov::API::ModelMatrix::FastRandomWalkThread().

---

The documentation for this class was generated from the following file:

• Markopy/Markopy/src/CLI/mmx.py