Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Regexp to read through log file and convert into pipe delimited file?.

by laknar (Initiate)
on Nov 25, 2012 at 00:41 UTC ( #1005442=perlquestion: print w/ replies, xml ) Need Help??
laknar has asked for the wisdom of the Perl Monks concerning the following question:

Im trying to grep a set of strings to find each record in a log file and grep the fields for the corresponding string. the scenario is to find the string and read the above line for date and also grep a field OrderID from the below lines for each search string.The search strings are "Authenticate RequestXML", "Authenticate ResponseXML", "Authorize RequestXML", "Authorize ResponseXML"

Below is the sample log file.
Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() FINER: ENTRY Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : FINER: ENTRY Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : INFO: Authenticate RequestXML : <CardinalMPI> <Version>1.7</Version> <SourceVersion>2.8</SourceVersion> <TransactionPwd>fhsdkjvdxfv</TransactionPwd> <MsgType>cmpi_authenticate</MsgType> <ProcessorId>202</ProcessorId> <Source>JTC</Source> <MerchantId>50515</MerchantId> <TransactionType>HC</TransactionType> <OrderId>8585273913266697</OrderId> <PAResPayload>202&#124;50515&#124;0einiSheyXIFqE4KYrlD&#124;HC</PA +ResPayload> </CardinalMPI> Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : FINER: ENTRY Nov 16, 2012 5:17:54 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() SEVERE: Authenticate ResponseXML : <CardinalMPI> <MerchantData></MerchantData> <ShippingAmount>1762</ShippingAmount> <MerchantReferenceNumber></MerchantReferenceNumber> <ErrorNo>0</ErrorNo> <CustomData></CustomData> <TransactionId>0einiSheyXIFqE4KYrlD</TransactionId> <ErrorDesc></ErrorDesc> <OrderNumber>5207694</OrderNumber> <ShippingMethod></ShippingMethod> <CardType>Visa</CardType> <PAResStatus>Y</PAResStatus> <ProcessorOrderNumber></ProcessorOrderNumber> <TaxAmount>543</TaxAmount> <PaymentType>CC</PaymentType> <PaymentProcessorOrderNumber></PaymentProcessorOrderNumber> <NotificationId>CCCDFHFCIGIILHEIT32Z89NPWO5S8A</NotificationId> <ActionCode>AUTHORIZE</ActionCode> <ReasonDesc></ReasonDesc> <CardLastFour>6061</CardLastFour> <Amount>8303</Amount> <AuthorizationCode></AuthorizationCode> <ReasonCode></ReasonCode> <OrderId>8585273913266697</OrderId> <SignatureVerification>Y</SignatureVerification> </CardinalMPI> Nov 16, 2012 5:17:54 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() SEVERE: : Authentication Success for for ORDER : 5207694 Nov 16, 2012 5:17:54 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() FINER: ENTRY Nov 16, 2012 5:17:56 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean checkHealth() INFO: checkHealth() Nov 16, 2012 5:17:56 AM com.sitapp01.commerce.payments.plugin.cybersou +rce.bean.AYCyberSourcePaymentsPluginBean checkHealth() INFO: checkHealth() Nov 16, 2012 5:17:56 AM com.sitapp01.commerce.payments.plugin.giftcard +.bean.AYGiftCardPaymentsPluginBean checkHealth() INFO: checkHealth() Nov 16, 2012 5:18:44 AM com.sitapp01.commerce.checkout.commands.OrderR +eviewCmdImpl performExecute INFO: OrderReview submmitted : Nov 16, 2012 5:18:48 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean approve(pluginContext, approveTran +saction, retry )false FINER: ENTRY Nov 16, 2012 5:18:48 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean setAuthorizeRequests FINER: ENTRY Nov 16, 2012 5:18:48 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean setAuthorizeRequests INFO: Authorize RequestXML : <CardinalMPI> <Version>1.7</Version> <SourceVersion>2.8</SourceVersion> <TransactionPwd>dfdskfjdskfjc</TransactionPwd> <MsgType>cmpi_authorize</MsgType> <ProcessorId>202</ProcessorId> <Source>JTC</Source> <Amount>8303</Amount> <CurrencyCode>840</CurrencyCode> <MerchantId>50515</MerchantId> <MerchantData>50515</MerchantData> <TransactionType>HC</TransactionType> <OrderId>8585273913266697</OrderId> <OrderDescription>Test Desc</OrderDescription> </CardinalMPI> Nov 16, 2012 5:18:48 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean setAuthorizeRequests FINER: EXIT Nov 16, 2012 5:18:48 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getAuthorizeResponse() : FINER: ENTRY Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getAuthorizeResponse() : INFO: Authorize ResponseXML : <CardinalMPI> <AuthorizationCode>016618</AuthorizationCode> <StatusCode>Y</StatusCode> <ReasonCode></ReasonCode> <ErrorDesc></ErrorDesc> <ProcessorRawResponse_CC_Authorize>AuthRecord=0110322001810E900002 +000000000000008303111603185000842700015912000000000001593234595939443 +457484D4E303136363138303030323834363038300001590083004831345730303233 +323134303733303039393250465244202030303030303030303030303030303030303 +030303030303000183232415050524F56414C20202020202020200003343820000656 +4943524320&ProcessorResponse=00&ReconciliationId=Y24YY9D4WHMN&Request +Id=3530647306590176056425</ProcessorRawResponse_CC_Authorize> <MerchantReferenceNumber></MerchantReferenceNumber> <ErrorNo>0</ErrorNo> <ProcessorRawResponse>interface=OK&lp_execute_status=PENDING&ACCT_ +NUM=XXXXXXXXXXXX6061&ACT_CD=OA&AMT=8303&CARD_SEC_IND_CD=1&CUST_ADDR1= +2607+Whittier+Place&CUST_CITY=Hephzibah&CUST_CNTRY_CD=USA&CUST_EMAIL= +eugenec30%40cs.com&CUST_FNAME=William&CUST_HOME_PHONE=7067937577&CUST +_ID=130668178&CUST_LNAME=Cutler&CUST_POSTAL_CD=308156120&CUST_STPR_CD +=GA&CUST_TYPE_CD=B&D_ACCT_NUM=e7daea2735ed4296363388954468550b46e026d +f&DIV_NUM=cardinal&FRAUD_REC_ID=000151010001GB920121116062534707&FRAU +D_RSP_CD=0100&FRAUD_STAT_CD=ACCEPT&FRAUD_USE_CD=C&MOP_TYPE_CD=VI&ORD_ +ID=5207694&PROC_CD=EBT&REQ_ID=780078748245&REQ_TYPE_CD=E&RSP_AVS_CD=Y +&RSP_CD=00&RSP_SEC_CD=P&S_ACCT_NUM=MDEyOI8ptWYFiREeP37gqZdSL9SRlRumMJ +LuEsAV5sT5eDKsnOkEx1VOVTLaPyTdr%2FFxd4%2BBOD1fceoDgwv4kA0q1Xd7GulPwgC +KMqR1sTcvKKhC1DYSJVS5Sman9R0GyFQr%2F3QBHhFA%2BGrLq7GbjgBbNSMs8dq%2BeC +49pIwIlpEJgwtM&S_CARD_EXP_DT=MDEyOGb4%2BEILRMAysKxbyGGm6%2BscLClDCHB3 +rTZ4wlRhwdKoNxLFLo%2BbOfyEgWuA80qbnn0vyKk1tl5Z37HENlh1CmW2cxqjxknooiX +JcdDMwQxlXievOvmSBGMh6XxP0W2cfaCydkI9tsuoqvOn949iCJv4aBDIAy0yDFcGBicz +UCYI&S_KEY_ID=129&STAT_CD=PENDING</ProcessorRawResponse> <CardCodeResult>P</CardCodeResult> <TransactionId>un7Y71kayZdgiVCsak86</TransactionId> <OrderNumber>5207694</OrderNumber> <ReasonDesc></ReasonDesc> <MerchantData>50515</MerchantData> <AVSResult>P</AVSResult> <OrderId>8585273913266697</OrderId> <ProcessorOrderNumber>3530647306590176056425</ProcessorOrderNumber +> </CardinalMPI> Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getRequestAuthCode SEVERE: : Authorization Code for Order : 5207694 Is 780078748245 Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getRedStatus() SEVERE: : RED status for Order : 5207694 Is ACCEPT Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getAuthorizeResponse() : SEVERE: : Payment Success for for ORDER : 5207694 Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean getAuthorizeResponse() : FINER: EXIT Nov 16, 2012 5:18:52 AM com.sitapp01.commerce.payments.plugin.centinel +.bean.AYCentinelPaymentsPluginBean approve(pluginContext, approveTran +saction, retry )false FINER: EXIT Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setCentinelRequest FINER: ENTRY Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setCommodityCodeAndShipId: SEVERE: Entry Point:orders ID 5210188 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setCommodityCodeAndShipId SEVERE: Attribute Id ::219055 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setMFPartNumber: SEVERE: Entry Point:orders ID 5210188 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setItemDesc: SEVERE: Entry Point:orders ID 5210188 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setItemDesc: SEVERE: Item_Desc_Wilson Championship Nylon Shuttlecocks 6-Pack Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setItemShippingNumber: SEVERE: Entry Point:orders ID 5210188 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getReDShipMethod FINER: ENTRY Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getReDShipMethod FINE: shippingModeId: 10551 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getReDShipMethod FINE: Shipping Code: UPS Ground Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getReDShipMethod FINE: returnReDShipMethod: C Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setItemShippingNumber FINE: Attribute Id ::219051 Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setItemShippingNumber SEVERE: Item_Shipping_Number_:S&L Nov 16, 2012 5:27:33 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setCentinelRequest

the expected output would be like below.

Column Names. Start_Date|Message|Request_Type|OrderNumber|OrderID Nov 16, 2012 5:17:53 AM|INFO|Authenticate RequestXML||8585273913266697 Nov 16, 2012 5:17:53 AM|SEVERE|Authenticate ResponseXML|5207694|858527 +3913266697 Nov 16, 2012 5:18:48 AM|INFO|Authorize RequestXML||8585273913266697 Nov 16, 2012 5:18:52 AM|INFO|Authorize ResponseXML|5207694|85852739132 +66697
I need to use perl script to acheive this scenario. Any help will be greatly appreciated.

Comment on Regexp to read through log file and convert into pipe delimited file?.
Select or Download Code
Re: Regexp to read through log file and convert into pipe delimited file?.
by mbethke (Hermit) on Nov 25, 2012 at 00:59 UTC

    Hi laknar,

    Could you please add some formatting (i.e. code tags) so the log is actually readable? And it would be nice if you could show where exactly you're having problems---crafting the necessary regex, merging lines or whatever.

Re: Regexp to read through log file and convert into pipe delimited file?.
by AnomalousMonk (Abbot) on Nov 25, 2012 at 01:40 UTC
Re: Regexp to read through log file and convert into pipe delimited file?.
by roboticus (Canon) on Nov 25, 2012 at 02:40 UTC

    laknar:

    We're not a code-writing service. We'll be glad to help you with problems in your code, but you haven't provided any yet. You may want to read through perlintro and perlre to get started.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

Re: Regexp to read through log file and convert into pipe delimited file?.
by CountZero (Bishop) on Nov 25, 2012 at 10:06 UTC
    Two nested while loops and some regex:
    use Modern::Perl; while (<DATA>) { my ($month, $day, $year, $time, $ampm) = (split ' ')[0 .. 4]; next unless $ampm and $ampm =~ m/AM|PM/i; my $datetime = join ' ', ($month, $day, $year, $time, $ampm); my $xml; my $level; my $orderID; my $orderNumber; while (<DATA>) { if (/([^:]+)\s*:\s*((Authenticate RequestXML)|(Authenticate Re +sponseXML)|(Authorize ResponseXML))\s*:\s*<CardinalMPI>/i){ $level = $1; $xml = $2; } if (m|<OrderId>(\d+)</OrderId>|) { $orderID = $1; } if (m|<OrderNumber>(\d+)</OrderNumber>|) { $orderNumber = $1; } if (m|^</CardinalMPI>$|) { say "$datetime - $level - $xml - $orderNumber - $orderID" +if $xml; last; } } } __DATA__ Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() FINER: ENTRY Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : FINER: ENTRY Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : INFO: Authenticate RequestXML : <CardinalMPI> <Version>1.7</Version> <SourceVersion>2.8</SourceVersion> <TransactionPwd>fhsdkjvdxfv</TransactionPwd> <MsgType>cmpi_authenticate</MsgType> <ProcessorId>202</ProcessorId> <Source>JTC</Source> <MerchantId>50515</MerchantId> <TransactionType>HC</TransactionType> <OrderId>8585273913266697</OrderId> <PAResPayload>202&#124;50515&#124;0einiSheyXIFqE4KYrlD&#124;HC</PA +ResPayload> </CardinalMPI> Nov 16, 2012 5:17:53 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til setAuthenticateRequest() : FINER: ENTRY Nov 16, 2012 5:17:54 AM com.sitapp01.commerce.CentinelUtil.AYCentinelU +til getAuthenticateResponse() ETC ETC ETC ...
    Output:
    Nov 16, 2012 5:17:53 AM - INFO - Authenticate RequestXML - - 85852739 +13266697 Nov 16, 2012 5:17:53 AM - SEVERE - Authenticate ResponseXML - 5207694 +- 8585273913266697 Nov 16, 2012 5:18:48 AM - INFO - Authorize ResponseXML - 5207694 - 858 +5273913266697
    Proper formatting of the output is left as an exercise for the reader.

    Note that you will get some "Use of uninitialized value ..." warnings. This is considered a feature not a bug! It points you to some records which possibly have incomplete data.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1005442]
Approved by mbethke
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (8)
As of 2014-09-19 06:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (132 votes), past polls