Re: Supervised machine learning algo for text matching across two files

Hello Anonymous Monk,

Your description unfortunately is not enough for us (or just me) to provide you an answer. We need more data, sample of file1 and sample of matching on file2.

As a solution (if I understand correctly) you have millions lines on file 1 but some how 55 fields, maybe the rows contain some key words that can match with file 2? If so you can write a regex that will match the key word with line number. As a next step you can have a hash containing the previously collected data that you will compare they keys (key words file 1, values would be line numbers) with fields on file 2.

Maybe I am really far out for answering your question, but as I said I do not know what are your data on your files.

With more information it would be easier to answer.

Help us to help you. :)

Seeking for Perl wisdom...on the process of learning...not there...yet!

Comment on Re: Supervised machine learning algo for text matching across two files Select or Download Code

Replies are listed 'Best First'.
Re^2: Supervised machine learning algo for text matching across two files by Anonymous Monk on May 24, 2017 at 20:28 UTC
Hey there thanks for the responses !! There are definitely key words that can match but they are not possible to match with regex. A random example i made up would be: file 1: HCBS_max, file 2: National healthcare basketball society . In this case there is an acronym and intuitively I can google both and then decide that ok these are the same let me do the match manually. I could make a regex rule that would search for acronyms sure, but there is no obvious patterning like this... its all human entered data and thus all over the place with no standardization. What I am thinking is that I can use the other 50 columns in the file 1 and search for patterns and associations that are not intuitive but none the less help me to classify some of the matches. Is this what a random forest can do potentially, utilizing the 15% of "ground truth" data I have as a training set?	[reply]
Re^3: Supervised machine learning algo for text matching across two files by thanos1983 (Parson) on May 24, 2017 at 20:37 UTC
Hello again Anonymous Monk, Well to be honest I do not see any way out of my mind on matching `HCBS_max` and `National healthcare basketball society`. So I can not really say that this could be done with comparing data. Are you able to manipulate this files while the data are populated inside of them? If so you could add based on conditions abbreviations. Seeking for Perl wisdom...on the process of learning...not there...yet!	[reply] [d/l] [select]
Re^4: Supervised machine learning algo for text matching across two files by AnomalousMonk (Archbishop) on May 24, 2017 at 21:08 UTC
I'm with thanos1983 on this one. `'National healthcare basketball society'` maps to `'HCBS_max'`?!? Wow! If anyone figures out a solution to this one, please let me know; I'd sure like to go in with you on patenting/exploiting it! Give a man a fish: `<%-{-{-{-<`	[reply] [d/l] [select]
Re^5: Supervised machine learning algo for text matching across two files by choroba (Cardinal) on May 24, 2017 at 21:15 UTC
Re^4: Supervised machine learning algo for text matching across two files by Anonymous Monk on May 24, 2017 at 20:46 UTC
Since I have 15% matched to use as ground truth... can't I somehow use the other 50 columns in the file ( all of which has various data fields) to train some sort of supervised approach that uses all the data to suggest a match statistically? I want to say the answer has something to do with Expectation Maximization type approach but I'm way out of my depth here.	[reply]
Re^3: Supervised machine learning algo for text matching across two files by KurtZ (Friar) on May 24, 2017 at 23:40 UTC
intuitively I can google both and then decide I think that's the only way IF goggle allowed automatic queries. The point is you don't have enough data for automatic associations, but Google does.	[reply]


Just another Perl shrine
	PerlMonks