Guru: Web Services, DATA-INTO and DATA-GEN, Part 1 – IT Jungle

April 5, 2021
Many of the “Can you help me with. . . ” communications that cross my desk these days include reference to JSON. Sometimes the questioner is receiving JSON in a file, or has to retrieve it from a web service, or needs to generate JSON in response to a query. While there are many ways to handle these requirements, RPG’s built-in DATA-INTO and DATA-GEN can do a lot of the heavy lifting for you and are quite simple once you understand the basics.
In this series of tips, I am going to start with a basic example that uses both DATA-INTO and DATA-GEN to generate the request data and processes the response from a simple REST web service. Subsequent tips in the series will delve further into the capabilities and quirks of these opcodes by working through more complex examples. In addition to the RPG opcodes, I will be using Scott Klement’s YAJL library – specifically the YAJLDTAGEN and YAJLINTO routines. I will also be using Scott’s HTTPAPI library for communicating with the web service. While I will give some details of using HTTPAPI, that is not my primary purpose in these tips, so if you are interested in knowing more about this fabulous tool let me know.
Time to introduce the web service. In these early examples I will be using the free “fake” REST services provided by the website JSONPlaceholder. I chose this site because it is likely to still be there in a year or two when those who read this in the future may want to try the code. I also like the fact that you do not need an account or any authorization code to use it. As a result, you should be able to copy my code and make your own explorations at any time.
This story contains code, which you can download here.
This initial example uses a web service that will create a new blog post (consisting of a title and the blog entry content) for a specified author (the userid). The service will return a confirmation which includes a unique ID assigned by the service to the article. This service uses the POST method and the data must be sent in JSON format in the HTTP request body. The actual request needs to be in the format shown below.
Below is a sample of the JSON response that the web service returns. Note that it basically contains just the data that I submitted together with the unique ID that the service assigned to the new post. I should point out at this stage that, because this is a “fake” service, the ID it returns is always the same and the data is not actually being stored. So don’t be alarmed if you test this and it doesn’t seem to be working as you might expect in the real world!
Obviously, in a simple case like this I could form the JSON request by simply stringing the content together. But that’s messy and error prone so let’s look at how DATA-GEN can do the work for us. Besides, it’s rare that you’ll have a request this simple in real life.
The first thing to do is to define a data structure (DS) that maps to the desired JSON data. In this case it is just a simple single level DS containing the three required fields in their correct sequence. That DS will then be processed by DATA-GEN, with the help of Scott’s YAJLDTAGEN routine, to produce the required JSON. Here’s the DS definition.
Note that the sequence of fields in the DS and their names must match the sequence and exact spelling of the JSON elements to be created. RPG would treat the field names Title, TITLE and title as being the same, JSON does not! Names in JSON are case-sensitive! So you must use the exact spelling, including case.
<A> The DS is specified as QUALIFIED so that I can use the same field names in the response DS, as you will see in a moment.
<B> The fields in the DS have the names required by the web service. title and body are both defined as Varchar to ensure that we don’t generate a large number of trailing spaces in the fields. The userid (think customer number) is specified as int but could be any numeric data type.
Once we have the DS in place we can tell DATA-GEN to generate JSON from it. Here’s how that looks:
<C> Identifies our requestData DS as the source of the data.
<D> Names the target field to receive the generated JSON. In my example it is a varchar field with a length of 200 characters.
<E> %GEN identifies that the YAJLDTAGEN generator is to be used. It also allows for the specification of additional optional parameters that are passed to the generator, but none are needed in this example. You will see them in action in future tips.
That’s all we need in this simple example. If you study the complete code (which you can download here) you will see that I used the DSPLY opcode to obtain the data that is to be passed to the web service.
Once the JSON request has been built it is just a matter of sending it to the web service with HTTPAPI. I am using the HTTP_string routine to do this as it is the easiest one to use and the only data I am interested in is the response ID. If the web service call fails, it is sufficient to know there was an error. I am not concerned with the cause. In fact this particular service probably wouldn’t tell me anything useful anyway.
Because HTTP_string causes an exception to be thrown, as opposed to returning an error response code, it is normal to run it in a MONITOR group <F>. Any errors detected will cause an exit to the ON-ERROR code which you can see if you study the full example code.
At <G> we identify the variable response as the recipient of the data and that this web service uses the POST HTTP method.
<H> Next we identify the variable that contains the URL for the service we are using.
<I> The next parameter specifies that request is the variable containing the HTTP request body. In this case it is the JSON data that DATA-GEN constructed for us back at <C>.
<J> The final parameter identifies that the request body contains JSON. If the web service additionally supported XML as an input type, we could construct the request in XML format and code this as ‘application/xml’. It is the web service that determines the formats of data that are acceptable. In this case JSON is the only option.
Once HTTP_string has successfully run, we have the JSON data from the web service in the response variable. We can now process it using DATA-INTO. As was the case with DATA-GEN we will need to define a DS. This time though it will receive the values parsed out of the JSON response stream. Here is the DS I am using:
As you can see this DS is basically the same as the one used earlier to generate the JSON request. The exception is the addition of the id field, which the web service generated to uniquely identify the new post <K>. With the DS in place and the web service response in hand we can now use DATA-INTO to parse the JSON into the DS for subsequent processing by our RPG code.
Note that I have deliberately defined the id field in a different position to where it appears in the JSON. This demonstrates that, just like with XML-INTO, DATA-INTO does not require the field sequence to match. It is storing values based on the field names, not on their sequence.
<L> Identifies the responseData DS as the target for the DATA-INTO operation.
<M> %DATA identifies the source of the data to be processed. In this case that is the field response that was the result of the HTTP_string operation at <G>. The ‘case=any’ option specifies that the field names in the JSON should be converted to upper case before attempting to match them against names of the fields in the target DS. Those of you familiar with XML-INTO will recognize this option as being identical in effect to the same option on the %XML BIF. As you will see in future examples, this is true of many of the options used with DATA-INTO.
<N> Last, but not least, we use %PARSER to specify that we will use the YAJLINTO parser to handle the JSON. Just as with %GEN, %PARSER also allows for additional parameters which we will see in action later in the series.
And that is all there is to it.
I thought I would take a couple of minutes to address two questions that may have already popped into your mind.
1) “Can’t I use SQL to do all of this kind of JSON parsing for me?”
2) “Doesn’t IBM offer a web services client as part of the Integrated Web Services software?”
The answer to the first question is “Yes, but I’ll leave that explanation to someone else.” I will add though that while I find SQL easy enough to use for simple examples like this, I find that it often becomes horribly complicated and hard to maintain for more complex examples. I am not a fan of the “SQL is the answer — now what was the question?” mentality that seems to have flourished in recent years. The RPG tooling in this area is good and fast.
The answer to the second question is “Yes, but I don’t like it.” It would take far too long for me to explain in detail, but to me using it just doesn’t “feel” like RPG. HTTPAPI was written by an RPG user and it shows. IBM’s offering is a port of a Unix-style Apache tool and involves creating and compiling C stubs to interface to the system components. HTTPAPI also has a huge user following and an active forum where users constantly trade tips and techniques. The same is not true of the IBM offering. Just my perspective — your mileage may vary.
In the next tip in this series, I will be taking a look at an example that uses the GET method to retrieve information about blog posts.
In the meantime, if you have any questions or if there are any particular aspects of this very broad topic you would like me to delve into in future tips please let me know.
Jon Paris is one of the world’s foremost experts on programming on the IBM i platform. A frequent author, forum contributor, and speaker at User Groups and technical conferences around the world, he is also an IBM Champion and a partner at Partner400 and System i Developer. Until Covid-19 messed everything up he hosted the RPG & DB2 Summit with partners Susan Gantner and Paul Tuohy. These days he has to be content with everything being done over Zoom. That includes the free Summit Lunch & Learn series, the Summit Hands-On Live! Workshops, and the Virtual RPG & DB2 Summit.
RPG’s New DATA-INTO
Tags: Tags: , , , , , , ,
Summit Hands-On Live! Workshop Series
April 27 – May 20, 2021
In-Person, Interactive, In-Depth — Online!
Master critical IBM i development skills at record speed with the Summit’s full-day Hands-On Live! workshops.
Teaching live online via video meeting, instructors Paul Tuohy, Jon Paris, Susan Gantner, and Mike Pavlak take you deep into the best ways to use your favorite technologies, cementing your knowledge with guided, hands-on labs and personal attention.
Choose from 7 different workshops to hone your skills.
Check out the workshop abstracts and prepare to take your productivity to new heights!


Jon,
Excellent piece. In reviewing the article and code, I see where you said: “The ‘case=any’ option specifies that the field names in the JSON should be converted to upper case before attempting to match them against names of the fields in the target DS”. I assume that you do this because, even though we may code in lower or mixed case, the RPG compiler has generated the DS and Field names as UPPER case? Thanks for clarifying this matter.
Also, do the same DATA-GEN, DATA-INTO and YAJL tools support similar capabilities for CSV files? I find that although CSV is the “weak sister” to JSON or XML, plenty of people still see it as their go-to strategy for transferring/interfacing data systems. I get frustrated by the error prone nature of CSV for situations where fields include embedded commas, quotes, tabs or other characters that tend to confound the translation. I find myself hoping CSV will go the way of the Dodo. I would appreciate you thoughts on this.
regards,
Rick
Rick, first of all my sincere apologies for the delay in responding. I was not notified by the system that you had commented. ITJungle thought they had fixed the problem but …
So – first of all – your assumption about what RPG does with names under the covers is correct. In fact the reason I happened read this post today is that in the next part of this series (part 3) I will be dealing with the problems that RPG’s action with names can cause.
As to the CSV question. Scott Klement has added a CSV parser to his CSVR4 library (http://scottklement.com/csv/) so that is one option. Personally I normally use an Open Access CSV reader or writer handler since it is an easy retrofit to a current set up that uses CPYxxxIMPF and a regular RPG read/write scenario. If you are interested in this approach you can find my articles and the associated source code here: https://authory.com/JonParisAndSusanGantner?collection=Open-Access
Once again my apologies for the delay.

Copyright © 2021 IT Jungle

source