Supervised machine learning algo for text matching across two files

[Anonymous Monk]

Hey monks,

I need some help thinking through what type of algorithm I need to build and some suggestions on the best way to do it in Perl.

I've got two files and the objective is to match the rows in file 1 with an ID in a row in file 2.

File 1 has millions of rows and about 55 fields, but there is no one field that is capable of matching to anything in file 2 directly, despite in the real-world there being a very real 1:1 mapping for each row. From manual matching efforts of content experts I have a list of about 15% matched but the method to do this cannot be used any further thus I would like to use this 13% as a training set to build a machine learning algorithm that could help match the rest. I am thinking the right algorithm type is a multi-class classifier.

My hypothesis is that throughout the 15% matched there are patterns that are not easy to see that an algorithm like a decision forest or something could help to tease out and help get the match rate from 15% up to 30% etc...

Any comments or suggestions are tremendously helpful. Cheers!

[zentara]

In order to get that extra sorting difference, which only a trained eye can see, you need AI, Artificial::Intelligence,

I think you need to run all your data thru TensorFlow and see where the AI deduced sets are, and where the eigenvectors point. :-) There are alot of good AI tutorials on google, just search for "AI TensorFlow" and do the same on youtube. There are a few quick to watch tutorials on the topic of analyzing data into sets and vectors. There is a start of a Perl layer to TensorFlow, but generally people are using Python, C, or what have you. See Perl Tensorflow .

Otherwise in terms of whats available in Perl right now, it's PDL and it's superfast matrix routines. But you would need to manually setup matrix operations and analysing them probably requires the consultation of a good linear algebra scientist, not a Perl module.:-)

It is interesting to think, what the approach would be with Perl, to do real machine AI learning from huge data sets. Xs and assembly based routines could probably speed things up. Everyone seems to be using Python however I think Perl could do many of the sorts and column and row operations needed to sort data. Actually, learning to run TensorFlow, or one of it's competing variants, is the biggest software business there is going right now. It's funny you should even ask the question. :-)

[etj]

For PDL matrix operations, I don't think it's possible to get better performance than using PDL::LinearAlgebra, which wraps LAPACK (and the best-performing version of that is probably OpenBLAS).

[LanX]

My neural pattern matching says "tf-idf" could be the answer ( ... probably???)

You can improve the answer by "training" us with more data. ;-)

[thanos1983]

Hello Anonymous Monk,

Your description unfortunately is not enough for us (or just me) to provide you an answer. We need more data, sample of file1 and sample of matching on file2.

As a solution (if I understand correctly) you have millions lines on file 1 but some how 55 fields, maybe the rows contain some key words that can match with file 2? If so you can write a regex that will match the key word with line number. As a next step you can have a hash containing the previously collected data that you will compare they keys (key words file 1, values would be line numbers) with fields on file 2.

Maybe I am really far out for answering your question, but as I said I do not know what are your data on your files.

With more information it would be easier to answer.

Help us to help you. :)

[Anonymous Monk]

Hey there thanks for the responses !!

There are definitely key words that can match but they are not possible to match with regex. A random example i made up would be: file 1: HCBS_max, file 2: National healthcare basketball society . In this case there is an acronym and intuitively I can google both and then decide that ok these are the same let me do the match manually. I could make a regex rule that would search for acronyms sure, but there is no obvious patterning like this... its all human entered data and thus all over the place with no standardization.

What I am thinking is that I can use the other 50 columns in the file 1 and search for patterns and associations that are not intuitive but none the less help me to classify some of the matches. Is this what a random forest can do potentially, utilizing the 15% of "ground truth" data I have as a training set?

[thanos1983]

Hello again Anonymous Monk,

Well to be honest I do not see any way out of my mind on matching HCBS_max and National healthcare basketball society. So I can not really say that this could be done with comparing data.

Are you able to manipulate this files while the data are populated inside of them?

If so you could add based on conditions abbreviations.

[KurtZ]

intuitively I can google both and then decide

I think that's the only way IF goggle allowed automatic queries.

The point is you don't have enough data for automatic associations, but Google does.

[NetWallah]

NOT a perl solution, and I don't know enough about it to be sure it is applicable, but this tool may be of interest:

http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/

[zentara]

Hi again! As it just so happens in the synchronicity of things, today, the very next day, I get turned on to the R language, which as it happens has a Perl module at Statistics::R. The R homepage even has a useful set of recipes at CRAN.... Smart name. :-) It does what you need it to do. Process information into sets.