Python.Markopy.MarkovModel Class Reference

Abstract representation of a markov model. More...

Inheritance diagram for Python.Markopy.MarkovModel:

Collaboration diagram for Python.Markopy.MarkovModel:

Public Member Functions
def	Import (str filename)
def	Export (str filename)
def	Train (str dataset, str seperator, int threads)
def	Generate (int count, str wordlist, int minlen, int maxlen, int threads)
std::ifstream *	OpenDatasetFile (const char *filename)
	Open dataset file and return the ifstream pointer. More...
void	Train (const char *datasetFileName, char delimiter, int threads)
	Train the model with the dataset file. More...
std::ofstream *	Save (const char *filename)
	Export model to file. More...
void	Generate (unsigned long int n, const char *wordlistFileName, int minLen=6, int maxLen=12, int threads=20)
	Call Markov::Model::RandomWalk n times, and collect output. More...
void	Buff (const char *str, double multiplier, bool bDontAdjustSelfLoops=true, bool bDontAdjustExtendedLoops=false)
	Buff expression of some characters in the model. More...
char *	RandomWalk (Markov::Random::RandomEngine *randomEngine, int minSetting, int maxSetting, char *buffer)
	Do a random walk on this model. More...
void	AdjustEdge (const char *payload, long int occurrence)
	Adjust the model with a single string. More...
bool	Import (std::ifstream *)
	Import a file to construct the model. More...
bool	Import (const char *filename)
	Open a file to import with filename, and call bool Model::Import with std::ifstream. More...
bool	Export (std::ofstream *)
	Export a file of the model. More...
bool	Export (const char *filename)
	Open a file to export with filename, and call bool Model::Export with std::ofstream. More...
Node< char > *	StarterNode ()
	Return starter Node. More...
std::vector< Edge< char > * > *	Edges ()
	Return a vector of all the edges in the model. More...
std::map< char, Node< char > * > *	Nodes ()
	Return starter Node. More...
void	OptimizeEdgeOrder ()
	Sort edges of all nodes in the model ordered by edge weights. More...

Private Member Functions
void	TrainThread (Markov::API::Concurrency::ThreadSharedListHandler *listhandler, char delimiter)
	A single thread invoked by the Train function. More...
void	GenerateThread (std::mutex *outputLock, unsigned long int n, std::ofstream *wordlist, int minLen, int maxLen)
	A single thread invoked by the Generate function. More...

Private Attributes
std::ifstream *	datasetFile
std::ofstream *	modelSavefile
	Dataset file input of our system More...
std::ofstream *	outputFile
	File to save model of our system More...
std::map< char, Node< char > * >	nodes
	Map LeftNode is the Nodes NodeValue Map RightNode is the node pointer. More...
Node< char > *	starterNode
	Starter Node of this model. More...
std::vector< Edge< char > * >	edges
	A list of all edges in this model. More...

Detailed Description

Abstract representation of a markov model.

To help with the python-cpp gateway documentation.

Definition at line 13 of file mm.py.

Member Function Documentation

AdjustEdge()

void Markov::Model< char >::AdjustEdge ( const NodeStorageType *  payload, long int  occurrence  )

inherited

Adjust the model with a single string.

Start from the starter node, and for each character, AdjustEdge the edge EdgeWeight from current node to the next, until NULL character is reached.

Then, update the edge EdgeWeight from current node, to the terminator node.

This function is used for training purposes, as it can be used for adjusting the model with each line of the corpus file.

Example Use: Create an empty model and train it with string: "testdata"

Markov::Model<char> model;
char test[] = "testdata";
model.AdjustEdge(test, 15); 

Parameters

string	- String that is passed from the training, and will be used to AdjustEdge the model with
occurrence	- Occurrence of this string.

Definition at line 109 of file model.h.

                                                                                                 {
    NodeStorageType p = payload[0];
    Markov::Node<NodeStorageType>* curnode = this->starterNode;
    Markov::Edge<NodeStorageType>* e;
    int i = 0;
 
    if (p == 0) return;
    while (p != 0) {
        e = curnode->FindEdge(p);
        if (e == NULL) return;
        e->AdjustEdge(occurrence);
        curnode = e->RightNode();
        p = payload[++i];
    }
 
    e = curnode->FindEdge('\xff');
    e->AdjustEdge(occurrence);
    return;
}

Buff()

void Markov::API::MarkovPasswords::Buff ( const char *  str, double  multiplier, bool  bDontAdjustSelfLoops = true, bool  bDontAdjustExtendedLoops = false  )

inherited

Buff expression of some characters in the model.

Parameters

str	A string containing all the characters to be buffed
multiplier	A constant value to buff the nodes with.
bDontAdjustSelfEdges	Do not adjust weights if target node is same as source node
bDontAdjustExtendedLoops	Do not adjust if both source and target nodes are in first parameter

Definition at line 153 of file markovPasswords.cpp.

                                                                                                                               {
    std::string buffstr(str);
    std::map< char, Node< char > * > *nodes;
    std::map< char, Edge< char > * > *edges;
    nodes = this->Nodes();
    int i=0;
    for (auto const& [repr, node] : *nodes){
        edges = node->Edges();
        for (auto const& [targetrepr, edge] : *edges){
            if(buffstr.find(targetrepr)!= std::string::npos){
                if(bDontAdjustSelfLoops && repr==targetrepr) continue;
                if(bDontAdjustExtendedLoops){
                    if(buffstr.find(repr)!= std::string::npos){
                        continue;
                    }
                }
                long int weight = edge->EdgeWeight();
                weight = weight*multiplier;     
                edge->AdjustEdge(weight);
            }
 
        }
        i++;
    }
 
    this->OptimizeEdgeOrder();
}

References Markov::Edge< NodeStorageType >::AdjustEdge(), Markov::Node< storageType >::Edges(), Markov::Edge< NodeStorageType >::EdgeWeight(), Markov::Model< NodeStorageType >::Nodes(), and Markov::Model< NodeStorageType >::OptimizeEdgeOrder().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

Edges()

std::vector<Edge<char >*>* Markov::Model< char >::Edges ( )

inlineinherited

Return a vector of all the edges in the model.

Returns

vector of edges

Definition at line 176 of file model.h.

{ return &edges;}

Export() [1/3]

bool Markov::Model< char >::Export ( const char *  filename )

inherited

Open a file to export with filename, and call bool Model::Export with std::ofstream.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Export file to filename

Markov::Model<char> model;
model.Export("test.mdl");

Definition at line 166 of file model.h.

                                                              {
    std::ofstream exportfile;
    exportfile.open(filename);
    return this->Export(&exportfile);
}

Export() [2/3]

bool Markov::Model< char >::Export ( std::ofstream *  f )

inherited

Export a file of the model.

File contains a list of edges. Format is: Left_repr;EdgeWeight;right_repr. For more information on the format, check out the project wiki or github readme.

Iterate over this vertices, and their edges, and write them to file.

Returns

True if successful, False for incomplete models.

Example Use: Export file to ofstream

Markov::Model<char> model;
std::ofstream file("test.mdl");
model.Export(&file);

Definition at line 155 of file model.h.

                                                        {
    Markov::Edge<NodeStorageType>* e;
    for (std::vector<int>::size_type i = 0; i != this->edges.size(); i++) {
        e = this->edges[i];
        //std::cout << e->LeftNode()->NodeValue() << "," << e->EdgeWeight() << "," << e->RightNode()->NodeValue() << "\n";
        *f << e->LeftNode()->NodeValue() << "," << e->EdgeWeight() << "," << e->RightNode()->NodeValue() << "\n";
    }
 
    return true;
}

Export() [3/3]

def Python.Markopy.MarkovModel.Export

(

str

filename

)

Definition at line 26 of file mm.py.

    def Export(filename : str):
        pass
 

Generate() [1/2]

def Python.Markopy.MarkovModel.Generate	(	int	count,
		str	wordlist,
		int	minlen,
		int	maxlen,
		int	threads
	)

Definition at line 34 of file mm.py.

    def Generate(count : int, wordlist : str, minlen : int, maxlen: int, threads : int):
        pass
 
 

Referenced by Python.Markopy.MarkovPasswordsCLI._generate().

Here is the caller graph for this function:

Generate() [2/2]

void Markov::API::MarkovPasswords::Generate ( unsigned long int  n, const char *  wordlistFileName, int  minLen = 6, int  maxLen = 12, int  threads = 20  )

inherited

Call Markov::Model::RandomWalk n times, and collect output.

Generate from model and write results to a file. a much more performance-optimized method. FastRandomWalk will reduce the runtime by %96.5 on average.

Deprecated:

See Markov::API::MatrixModel::FastRandomWalk for more information.

Parameters

n	- Number of passwords to generate.
wordlistFileName	- Filename to write to
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate
threads	- number of OS threads to spawn

Definition at line 118 of file markovPasswords.cpp.

                                                                                                                               {
    char* res;
    char print[100];
    std::ofstream wordlist; 
    wordlist.open(wordlistFileName);
    std::mutex mlock;
    int iterationsPerThread = n/threads;
    int iterationsCarryOver = n%threads;
    std::vector<std::thread*> threadsV;
    for(int i=0;i<threads;i++){
        threadsV.push_back(new std::thread(&Markov::API::MarkovPasswords::GenerateThread, this, &mlock, iterationsPerThread, &wordlist, minLen, maxLen));
    }
 
    for(int i=0;i<threads;i++){
        threadsV[i]->join();
        delete threadsV[i];
    }
 
    this->GenerateThread(&mlock, iterationsCarryOver, &wordlist, minLen, maxLen);
    
}

References Markov::API::MarkovPasswords::GenerateThread().

Referenced by Markov::Markopy::BOOST_PYTHON_MODULE(), and Markov::GUI::Generate::generation().

Here is the call graph for this function:

Here is the caller graph for this function:

GenerateThread()

void Markov::API::MarkovPasswords::GenerateThread ( std::mutex *  outputLock, unsigned long int  n, std::ofstream *  wordlist, int  minLen, int  maxLen  )

privateinherited

A single thread invoked by the Generate function.

DEPRECATED: See Markov::API::MatrixModel::FastRandomWalkThread for more information. This has been replaced with a much more performance-optimized method. FastRandomWalk will reduce the runtime by %96.5 on average.

Parameters

outputLock	- shared mutex lock to lock during output operation. Prevents race condition on write.
n	number of lines to be generated by this thread
wordlist	wordlistfile
minLen	- Minimum password length to generate
maxLen	- Maximum password length to generate

Definition at line 140 of file markovPasswords.cpp.

                                                                                                                                       {
    char* res = new char[maxLen+5];
    if(n==0) return;
 
    Markov::Random::Marsaglia MarsagliaRandomEngine;
    for (int i = 0; i < n; i++) {
        this->RandomWalk(&MarsagliaRandomEngine, minLen, maxLen, res); 
        outputLock->lock();
        *wordlist << res << "\n";
        outputLock->unlock();
    }
}

References Markov::Model< NodeStorageType >::RandomWalk().

Referenced by Markov::API::MarkovPasswords::Generate().

Here is the call graph for this function:

Here is the caller graph for this function:

Import() [1/3]

bool Markov::Model< char >::Import ( const char *  filename )

inherited

Open a file to import with filename, and call bool Model::Import with std::ifstream.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Import a file with filename

Markov::Model<char> model;
model.Import("test.mdl");

Definition at line 137 of file model.h.

                                                              {
    std::ifstream importfile;
    importfile.open(filename);
    return this->Import(&importfile);
 
}

Import() [2/3]

bool Markov::Model< char >::Import ( std::ifstream *  f )

inherited

Import a file to construct the model.

File contains a list of edges. For more info on the file format, check out the wiki and github readme pages. Format is: Left_repr;EdgeWeight;right_repr

Iterate over this list, and construct nodes and edges accordingly.

Returns

True if successful, False for incomplete models or corrupt file formats

Example Use: Import a file from ifstream

Markov::Model<char> model;
std::ifstream file("test.mdl");
model.Import(&file);

Definition at line 126 of file model.h.

                                                        {
    std::string cell;
 
    char src;
    char target;
    long int oc;
 
    while (std::getline(*f, cell)) {
        //std::cout << "cell: " << cell << std::endl;
        src = cell[0];
        target = cell[cell.length() - 1];
        char* j;
        oc = std::strtol(cell.substr(2, cell.length() - 2).c_str(),&j,10);
        //std::cout << oc << "\n";
        Markov::Node<NodeStorageType>* srcN;
        Markov::Node<NodeStorageType>* targetN;
        Markov::Edge<NodeStorageType>* e;
        if (this->nodes.find(src) == this->nodes.end()) {
            srcN = new Markov::Node<NodeStorageType>(src);
            this->nodes.insert(std::pair<char, Markov::Node<NodeStorageType>*>(src, srcN));
            //std::cout << "Creating new node at start.\n";
        }
        else {
            srcN = this->nodes.find(src)->second;
        }
 
        if (this->nodes.find(target) == this->nodes.end()) {
            targetN = new Markov::Node<NodeStorageType>(target);
            this->nodes.insert(std::pair<char, Markov::Node<NodeStorageType>*>(target, targetN));
            //std::cout << "Creating new node at end.\n";
        }
        else {
            targetN = this->nodes.find(target)->second;
        }
        e = srcN->Link(targetN);
        e->AdjustEdge(oc);
        this->edges.push_back(e);
 
        //std::cout << int(srcN->NodeValue()) << " --" << e->EdgeWeight() << "--> " << int(targetN->NodeValue()) << "\n";
 
 
    }
 
    this->OptimizeEdgeOrder();
 
    return true;
}

Import() [3/3]

def Python.Markopy.MarkovModel.Import

(

str

filename

)

Definition at line 22 of file mm.py.

    def Import(filename : str):
        pass
 

Nodes()

std::map<char , Node<char >*>* Markov::Model< char >::Nodes ( )

inlineinherited

Return starter Node.

Returns

starter node with 00 NodeValue

Definition at line 181 of file model.h.

{ return &nodes;}

OpenDatasetFile()

std::ifstream * Markov::API::MarkovPasswords::OpenDatasetFile ( const char *  filename )

inherited

Open dataset file and return the ifstream pointer.

Parameters

filename

- Filename to open

Returns

ifstream* to the the dataset file

Definition at line 51 of file markovPasswords.cpp.

                                                                          {
 
    std::ifstream* datasetFile;
 
    std::ifstream newFile(filename);
 
    datasetFile = &newFile;
 
    this->Import(datasetFile);
    return datasetFile;
}

References Markov::Model< NodeStorageType >::Import().

Here is the call graph for this function:

OptimizeEdgeOrder()

void Markov::Model< char >::OptimizeEdgeOrder

inherited

Sort edges of all nodes in the model ordered by edge weights.

Definition at line 186 of file model.h.

                                                    {
    for (std::pair<unsigned char, Markov::Node<NodeStorageType>*> const& x : this->nodes) {
        //std::cout << "Total edges in EdgesV: " << x.second->edgesV.size() << "\n"; 
        std::sort (x.second->edgesV.begin(), x.second->edgesV.end(), [](Edge<NodeStorageType> *lhs, Edge<NodeStorageType> *rhs)->bool{
            return lhs->EdgeWeight() > rhs->EdgeWeight();
        });
        //for(int i=0;i<x.second->edgesV.size();i++)
        //  std::cout << x.second->edgesV[i]->EdgeWeight() << ", ";
        //std::cout << "\n";
    }
    //std::cout << "Total number of nodes: " << this->nodes.size() << std::endl;
    //std::cout << "Total number of edges: " << this->edges.size() << std::endl;
}

RandomWalk()

char * Markov::Model< char >::RandomWalk ( Markov::Random::RandomEngine *  randomEngine, int  minSetting, int  maxSetting, NodeStorageType *  buffer  )

inherited

Do a random walk on this model.

Start from the starter node, on each node, invoke RandomNext using the random engine on current node, until terminator node is reached. If terminator node is reached before minimum length criateria is reached, ignore the last selection and re-invoke randomNext

If maximum length criteria is reached but final node is not, cut off the generation and proceed to the final node. This function takes Markov::Random::RandomEngine as a parameter to generate pseudo random numbers from

This library is shipped with two random engines, Marsaglia and Mersenne. While mersenne output is higher in entropy, most use cases don't really need super high entropy output, so Markov::Random::Marsaglia is preferable for better performance.

This function WILL NOT reallocate buffer. Make sure no out of bound writes are happening via maximum length criteria.

Example Use: Generate 10 lines, with 5 to 10 characters, and print the output. Use Marsaglia

Markov::Model<char> model;
Model.import("model.mdl");
char* res = new char[11];
Markov::Random::Marsaglia MarsagliaRandomEngine;
for (int i = 0; i < 10; i++) {
    this->RandomWalk(&MarsagliaRandomEngine, 5, 10, res); 
    std::cout << res << "\n";
 }

Parameters

randomEngine	Random Engine to use for the random walks. For examples, see Markov::Random::Mersenne and Markov::Random::Marsaglia
minSetting	Minimum number of characters to generate
maxSetting	Maximum number of character to generate
buffer	buffer to write the result to

Returns

Null terminated string that was generated.

Definition at line 86 of file model.h.

                                                                                                                                                         {
    Markov::Node<NodeStorageType>* n = this->starterNode;
    int len = 0;
    Markov::Node<NodeStorageType>* temp_node;
    while (true) {
        temp_node = n->RandomNext(randomEngine);
        if (len >= maxSetting) {
            break;
        }
        else if ((temp_node == NULL) && (len < minSetting)) {
            continue;
        }
 
        else if (temp_node == NULL){
            break;
        }
            
        n = temp_node;
 
        buffer[len++] = n->NodeValue();
    }
 
    //null terminate the string
    buffer[len] = 0x00;
 
    //do something with the generated string
    return buffer; //for now
}

Save()

std::ofstream * Markov::API::MarkovPasswords::Save ( const char *  filename )

inherited

Export model to file.

Parameters

filename

- Export filename.

Returns

std::ofstream* of the exported file.

Definition at line 106 of file markovPasswords.cpp.

                                                                {
    std::ofstream* exportFile;
 
    std::ofstream newFile(filename);
 
    exportFile = &newFile;
    
    this->Export(exportFile);
    return exportFile;
}

References Markov::Model< NodeStorageType >::Export().

Here is the call graph for this function:

StarterNode()

Node<char >* Markov::Model< char >::StarterNode ( )

inlineinherited

Return starter Node.

Returns

starter node with 00 NodeValue

Definition at line 171 of file model.h.

{ return starterNode;}

Train() [1/2]

void Markov::API::MarkovPasswords::Train ( const char *  datasetFileName, char  delimiter, int  threads  )

inherited

Train the model with the dataset file.

Parameters

datasetFileName	- Ifstream* to the dataset. If null, use class member
delimiter	- a character, same as the delimiter in dataset content
threads	- number of OS threads to spawn

Markov::API::MarkovPasswords mp;
mp.Import("models/2gram.mdl");
mp.Train("password.corpus");

Definition at line 65 of file markovPasswords.cpp.

                                                                                               {
    signal(SIGINT, intHandler);
    Markov::API::Concurrency::ThreadSharedListHandler listhandler(datasetFileName);
    auto start = std::chrono::high_resolution_clock::now();
 
    std::vector<std::thread*> threadsV;
    for(int i=0;i<threads;i++){
        threadsV.push_back(new std::thread(&Markov::API::MarkovPasswords::TrainThread, this, &listhandler, delimiter));
    }
 
    for(int i=0;i<threads;i++){
        threadsV[i]->join();
        delete threadsV[i];
    }
    auto finish = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double> elapsed = finish - start;
    std::cout << "Elapsed time: " << elapsed.count() << " s\n";
 
}

References intHandler(), Markov::API::Concurrency::ThreadSharedListHandler::ThreadSharedListHandler(), and Markov::API::MarkovPasswords::TrainThread().

Referenced by Markov::Markopy::BOOST_PYTHON_MODULE(), Markov::GUI::Generate::train(), Markov::GUI::Train::train(), and Markov::API::ModelMatrix::Train().

Here is the call graph for this function:

Here is the caller graph for this function:

Train() [2/2]

def Python.Markopy.MarkovModel.Train	(	str	dataset,
		str	seperator,
		int	threads
	)

Definition at line 30 of file mm.py.

    def Train(dataset: str, seperator : str, threads : int):
        pass
 

TrainThread()

void Markov::API::MarkovPasswords::TrainThread ( Markov::API::Concurrency::ThreadSharedListHandler *  listhandler, char  delimiter  )

privateinherited

A single thread invoked by the Train function.

Parameters

listhandler	- Listhandler class to read corpus from
delimiter	- a character, same as the delimiter in dataset content

Definition at line 85 of file markovPasswords.cpp.

                                                                                                                  {
    char format_str[] ="%ld,%s";
    format_str[3]=delimiter;
    std::string line;
    while (listhandler->next(&line) && keepRunning) {
        long int oc;
        if (line.size() > 100) {
            line = line.substr(0, 100);
        }
        char* linebuf = new char[line.length()+5];
#ifdef _WIN32
        sscanf_s(line.c_str(), "%ld,%s", &oc, linebuf, line.length()+5); //<== changed format_str to-> "%ld,%s"
#else
        sscanf(line.c_str(), format_str, &oc, linebuf);
#endif
        this->AdjustEdge((const char*)linebuf, oc); 
        delete linebuf;
    }
}

References Markov::Model< NodeStorageType >::AdjustEdge(), keepRunning, and Markov::API::Concurrency::ThreadSharedListHandler::next().

Referenced by Markov::API::MarkovPasswords::Train().

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

datasetFile

std::ifstream* Markov::API::MarkovPasswords::datasetFile

privateinherited

Definition at line 123 of file markovPasswords.h.

edges

std::vector<Edge<char >*> Markov::Model< char >::edges

privateinherited

A list of all edges in this model.

Definition at line 204 of file model.h.

modelSavefile

std::ofstream* Markov::API::MarkovPasswords::modelSavefile

privateinherited

Dataset file input of our system

Definition at line 124 of file markovPasswords.h.

nodes

std::map<char , Node<char >*> Markov::Model< char >::nodes

privateinherited

Map LeftNode is the Nodes NodeValue Map RightNode is the node pointer.

Definition at line 193 of file model.h.

outputFile

std::ofstream* Markov::API::MarkovPasswords::outputFile

privateinherited

File to save model of our system

Definition at line 125 of file markovPasswords.h.

starterNode

Node<char >* Markov::Model< char >::starterNode

privateinherited

Starter Node of this model.

Definition at line 198 of file model.h.

---

The documentation for this class was generated from the following file:

• Markopy/Markopy/src/CLI/mm.py