Notes about using Pentaho Data Integration and Apache Hop as Non-Programmer
provided a sentence like:
neue Technologien zur Verkehrsoptimierung
should get a inital capital like:
Neue Technologien zur Verkehrsoptimierung
This can be achieved in Pentaho / Apache Hop by a java expression
text == null ? null : text.substring(0,1).toUpperCase() + text.substring(1)
from: https://forums.pentaho.com/threads/218295-Capitalize-first-letter-of-a-string/
To get a first impression about the content and size I was using the Crossref Rest API facet search:
So for 2020 there are 542k entries. In order to download metadata data (or in my case just the DOI, date of DOI registration and the Elsevier internal ID) I’m using again the REST API of Crossref.
Actually a Crossref query is limited to provide max 1000 DOIs. Using cursors, it’s however to possible to loop further and get about 100k DOIs before the API times out. So in order to get all publications from 2020 I created a monthly batch based on the created date like:
https://api.crossref.org/works?filter=member:78,from-created-date:2020-01-01,until-created-date:2020-01-31&select=DOI,created,alternative-id,&rows=1000&cursor=*
In PDI I’ve created a job, that handles the cursor and repeats the transformation with the REST Query as long there is a new cursor coming back from Crossref.
So I get a list with all alternative IDs (like: S0960982219315106) from Elsevier articles. With that I can create an URL to all abstract pages of this article: https://www.sciencedirect.com/science/article/abs/pii/S0960982219315106
Using the Pentaho HTTP Client, it’s now possible to get the HTML of the this abstract page.
It’s important that there’s a HTTP-Header like: “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5)”, otherwise the Pentaho Client will be routed to a error page.
Elsevier makes it very easy to extract information about the article without going deep into the HTML-Structure of ScienceDirect. Within the HTML there’s a whole section with structured data in JSON:
Loading the whole HTML in one field, the following Regex can be used to extract the json block:
*<script type="application/json" data-iso-key="_0">(.*)</script> <iframe style="display: none.*
Having the JSON data block separated from the rest of the HTML, we now can pass it to the JSON-input step. There is really plenty of information and much more than you would see via front end.
For the moment I’m mostly interested in the author and affiliation data:
In the JSON we can extract:
extracting the authors works similar like the affiliation.
An author can have multiple labels/refids, emails, contributor-roles. In order to get those flat in a row separated by a coma, a group step can be used, while getting it back into the stream using the “stream lookup”-step:
In a third step authors and affiliations are merged by the the label:
in order to get a list with authors and affiliations
Alternatively to the “Replace in string”-Step, the “Modified JavaScript value”-Step can be used to replace something in a string. eg. here to replace a carriage return with a white space:
var Author = replace(Author_seperated, “\r”,” “)
Using the “Modified JavaScript value”-step you can call functions in java script, like Math.max().
How to extract the email of the corresponding author of a publication, like: https://doi.org/10.1039/C7CS00709D with Pentaho Data integration?
The first email appearing in the HTML will put into the filed email.
Alternatively the Online Service https://www.convertcsv.com/email-extractor.htm also provides a nice possibility to extract emails from several websites:
You can extract a pattern as an additional field with Pentaho using the “Regex evaluation” step.
Example to extract the Arxiv-ID with this Regex: .*(\d{4}\.\d{4,5}).*
To trigger a shell script or a terminal command after a transformation, you have to create a job (it’s not available in a transformation). In the following scenario I wanted to transform a HTML-File to XML using tidy.
So I define a job, where the file is created and use the step “Execute a shell script…”
then enter the tidy command:
tidy -asxhtml -numeric < file_old.html > file_new.xml
in the next tab “Script”:
Provided tidy hasn’t failed, the “file_old.html” has been converted to “file_new.xml” in your job directory.
When you want to query a REST API but have values that need url-encoding first, you can use a User Defined Java Expression:
In this particular example it is necessary as PDI would throw the error (“Illegal character in path”) if you query the following URL unmodified from a string:
There are several possibilities in Pentaho Data Integration to put together several fields:
This step is very straightforward if you want to concat fields using the same separator:
Similar to Excel you can create a new string using your fields and ad hoc defined strings in the Formula step:
Pentaho & Apache Hop Hints 