850 EDI Help

[the_Don]

Does anyone know of any modules that assist in the parsing and data mining, if you will, of a file that is formatted in compliance of the EDI 850 specification for purchase orders?

There are various places on the web that have information regarding its use. A google search should bring up something if you do not know what it is by name.

This might be a niche-type thing, so if no one knows - oh well. :-)

the_Don
...making offers others can't rufuse.

[diskcrash]

Yo the_Don,

I had to build a general purpose X12 EDI header parser and store and forward system. I just put a version of it on code snippets at

EDI Store and Forward System

If you get heavily into EDI you will find a great deal of customization past what is in the DISA references. In particular there will be Trading Partner Agreements that establish very specific nuances. It is most often the case that you will work with one big player, like a large retailer, in concert with hundreds or thousands of little guys. The big guy publishes a TPA and the other guys follow it. So... you really need to get the TPA that is relavant to your problem and then... thank heavens for perl as it will mung right into parsing the thing. Even if you got a good generic 850 parser you will need to go line by line (pardon me char by char)to assure a good result. Then... ouch many TPAs and their users will adhere to some older standards that don't comply with the current methods and your perl may be the transformer for their future. I LOVE EDI, but like a marriage it takes custom work.

Best Regards,

Diskcrash

[lhoward]

I don't know of any X12 EDI perl modules, but I used to work for a company that did EDI (I even wrote a perl program that colorized/pretty-formatted X12 EDI). The following is purely from memory and may not be quoite right, but I hope you find it helpful.

X12 EDI documents are basically deimited files (i.e. split works great on them). Their are record separators, element separators, and sub-element separators. The first record in any X12 document is the ISA header, and this is fixed-length, so you can use it to find what all the separators are.

Once you know the separators, if you split the document on the record separator, you'll see its structured as follows (indentation added for clarity). If you then separate each record into elements by splitting on the element separator, the first element of each record is its record type:

ISA
  GS
    ST
      .. one transaction in here
    SE
    ST
      ..
    SE
  GE
  GS
    ...
  GE
IEA

i.e. the ISA/IEA records are the "document header and footer", it can contain one or more groups (delimited by GS/GE), each group contains one or more transactions (delimited by ST/SE). These ST/SE are the individual transactions (your 850, among others)

The 2 outer envelopes (ISA/IEA and GS/GE) contain routing information that indicates how to deliver the transcations. Within the ST/SE section are all the records that make up the individual 850. I don't remember what the exact records are.

Within the ST/SE block each record corresponds to specific information about the transaction.

That's about all I can remember for now. I'll post more if I can remember any other information in the structure of 850s....

[jordanh]

++ Good overview. A couple of additional points.

With Revision 004030, they've added a new separator, the Repetition Separator character, which is located at ISA11. ISA11 is the 11th element in the ISA segment. You'll see that terminology a lot. GS02 (the second element in the GS), for example, is the sender's Application ID and GS03 is the receiver's Application ID. You can split a given segment up into elements using the Element Separator, which just happens to be the 4th character of the ISA segment.

The two important separators are the segment terminator, with is the 106th character of the ISA (the ISA is the only fixed size segment... hmmm... maybe IEA is fixed size too) and the element separator is the 4th character. The component separator is the 105th character of the ISA, but it's rarely used (not used at all in 850 document IIRC). I don't know where the Repetition separator is used, but it must be even rarer than the Component separator as I've never seen it in use.

The ISA/GS segments contain information pertaining to addressing of the document and other header type information like EDI version and date/time stamps, etc.

If you find standards documents for your partner, these will typically explain what the standards are for what goes in between the ST/SE pairs.

Hope this helps.

[foxops]

God I'd love some EDI modules, I evaluate 856s for correct shipment segment population (among other things) and I find myself reinventing the wheel far to many times. I'd post my own scripts but they lack any sort of optimization (in one task I open the same 160+ mg file several times just to do another "while search").

[bart]

I gather this must at least be somewhat relevant:
• CPAN: XML::EDIfact
  
• http://www.xml-edifact.org/TR/XML-Edifact.html

[jordanh]

I don't think this is relevant at all. These modules and documentation pertain to the UN EDIfact standard.

EDI 850 probably refers to ASC X12 EDI Transaction Set Identifier 850 (Purchase Order), published by DISA.

I don't believe there are any publicly available modules or even really publicly available documentation for ASC X12.

There's been a discussion on this subject over on Use Perl.

I've used Perl for manipulation of X12 documents, but I don't have anything releasable. The hard part isn't parsing the documents, it's getting the information about all the fields and coordinating with your partners on acceptable use of the standards (what they call Implementation Conventions or ICs).

If there were to be any Open Source development in X12, I think that I'd be able to help test and advise. I'd have to check with my employer if I could contribute otherwise.

[bart]

I don't think this is relevant at all.

Well, this post (second message in the digest) surely makes the connection between EDI 850 and EDIfact.

Maybe it's just the new loose search engine on CPAN, but XML::EDI doesn't seem to find anything relevant.

[the_Don]

Perhaps the filenames of the incoming data are not named properly because there is no fixed header or footers. The file is ASCII text, but it seems to to be delimited by *. I am weary of giving examples because it is real data that should not be shared, and I am one of the last people that can authorize its release.

However, I can say that there are tags that start the lines and determine the data that will follow. Some examples of tags are 'BEG', 'N1', 'N3', 'N4', 'PO1', 'PID'. All I can gather thus far is that the BEG always starts the file and the N_ tags describe where products will be shipped to, billed to, etc..

My problem is that a common distributor has various clients (some of which are large, i.e national). My company accepts files that detail the orders because we are in charge of shipping. I was wondering if there were absolutes regarding where information is stored within these 850 files.

As a result of reading, it appears that the 850 guys only created a set of tags and formats to be used, not how to use them. It also appears as if each client has their own way... so it seems up to me to make some calls and determine if a set of characteristics and/or formats can be developed to make my job of determing who to ship stuff to easier.

Thanks to all who have offered their opinions.

the_Don
...making offers others can't rufuse.

[jordanh]

Can't be sure, but it sounds like what you are looking at are files using \n as the segment terminator and '*' as the element separator. With \n as the segment terminator, it's natural to just read the segments as separate lines.

These are good choices for the terminator, even recommended by the standards, IIRC. You're in luck in that your partners haven't used non-printable characters so this data is easy to read by just eyeball inspection. Oh, if you're trying to parse out the elements (the data between the '*'s), remember to quote the '*' properly in the split command. That's one pitfall of using '*', but an easy one to overcome.

The 'BEG', 'N1', 'N3', 'PO1' are called Segment Identifiers. Each field in those segments is defined in terms of allowable values (alpha, numeric, special tables of values, etc.), but there is a lot of latitude on how to use these standards. This latitude is typically nailed down in ICs (Implementation Conventions). In this discussion I have some links to some sample ICs used by the Government and others. These Sample ICs are actually based on the ASC X12 documents.

Ask around and see if your trading partners have written up ICs. It may well be the case that they haven't, in which case you have to reverse engineer them from their use. Getting this right can be complicated.